An interpretable image recognition method based on visual transformer and prototype learning

Abstract

The application discloses an interpretable image recognition method based on visual Transformer and prototype learning, and comprises the following steps: image preprocessing, normalizing and dividing a to-be-recognized image in training data into a plurality of non-overlapping image patches, then flattening each image patch into a one-dimensional vector; using different types of visual Transformer encoders to extract long-range dependence feature representation of the image patch; designing a prototype branch and composing a double-branch prototype layer and a single-branch prototype layer; according to the type of the visual Transformer encoder, calculating the similarity through the double-branch prototype layer or the single-branch prototype layer, obtaining an activation map through transformation, and processing the activation map; constructing a linear classifier to obtain an image recognition result; and interpreting the acquisition process of the image recognition result to complete the interpretable image recognition based on the visual Transformer and the prototype learning.

Description

TECHNICAL FIELD

[0001] The present application relates to an image recognition method, in particular to an interpretable image recognition method based on visual Transformer and prototype learning. BACKGROUND

[0002] The part provided in this part is only background information related to the present disclosure, which is not necessarily prior art.

[0003] Deep learning has been widely used in computer vision, natural language processing, reinforcement learning and recommendation systems, and has surpassed traditional machine learning methods and human experts in many fields. However, deep learning models are often considered as "black box" models, which are difficult to explore and mine their internal operation mechanism due to their massive model parameters and high nonlinearity, thereby limiting users' trust in these models. In order to improve the transparency of deep learning models and promote users to understand, trust and manage the models, interpretable methods of deep learning models have emerged. Such methods aim to provide model interpretation to users in understandable terms from the aspects of modelability, decomposability and algorithm transparency. Among these methods, the Prototypical Part Networks (PPN) has shown great potential in enhancing the transparency of model decision.

[0004] The existing prototype learning method based on convolutional neural network (CNN) has some limitations, such as when directly applying prototype learning to visual Transformer (ViTs), resulting in scattered and noisy prototype activations.

[0005] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0006] The present application aims to solve the technical problems of the prior art, and provides an interpretable image recognition method based on visual Transformer and prototype learning.

[0007] In order to solve the above technical problems, the present application discloses an interpretable image recognition method based on visual Transformer and prototype learning, comprising the following steps:

[0008] Step 1, image preprocessing, normalizing and dividing the to-be-identified image in the training data into a plurality of non-overlapping image patches, and then flattening each image patch into a one-dimensional vector;

[0009] Step 2, long-range dependency feature representations of the image patches are extracted using different types of visual Transformer encoders, and the long-range dependency feature representations include non-local feature representations or local feature representations;

[0010] Step 3, a prototype branch is designed and composed into a double-branch prototype layer and a single-branch prototype layer, and the prototype branch includes a non-local prototype branch and a local prototype branch, each of which maps the training data to a metric space and defines a set of prototypes representing the non-local feature representations or the local feature representations;

[0011] Step 4, in the metric space, the similarity between the non-local feature representations and the local feature representations in step 3 and the prototypes is calculated in turn through the double-branch prototype layer or the single-branch prototype layer according to the type of the visual Transformer encoder in step 2;

[0012] Step 5, the activation map is obtained by converting the similarity in step 4, the activation map is processed using a class-aware activation refinement method, and the dispersion degree of the activated region in the activation map is quantified using a diffusion pointer;

[0013] Step 6, a linear classifier is constructed, and the input is the similarity between the non-local feature representations or the local feature representations and the prototypes, and the output is the probability distribution of the image class; the probability is calculated according to the probability to obtain the predicted class of the image to be identified, i.e. the image recognition result;

[0014] Step 7, the prototype is mapped to the original image space according to the class-aware activation refinement method in step 5 to generate a visual representation, and the input and weight of the linear classifier in step 6 are combined to explain the acquisition process of the image recognition result, and the interpretable image recognition based on visual Transformer and prototype learning is completed.

[0015] Advantages:

[0016] 1、The long-range dependency relationship between image representations is introduced in the application, so that global features can be captured more effectively, and the recognition accuracy of the model is improved.

[0017] 2、The application designs a special prototype branch for different Transformer backbone networks, and designs orthogonality loss, clustering loss and dispersion loss to ensure the diversity and accuracy of the prototypes.

[0018] 3、The application uses a class-aware activation refinement mechanism to constrain the prototype to effectively focus on the most recognizable part of the image, reducing the interference of irrelevant information, thereby significantly improving the interpretability of the model. BRIEF DESCRIPTION OF DRAWINGS

[0019] The above and / or other aspects of the present application will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings in which:

[0020] Figure 1 A schematic diagram of the overall workflow of an embodiment of the present application.

[0021] Figure 2 A flowchart of the calculation of a dual-branch network structure of an embodiment of the present application.

[0022] Figure 3 A flowchart of the calculation of a single-branch network structure of an embodiment of the present application.

[0023] Figure 4 A schematic diagram of a prototype subspace of an embodiment of the present application.

[0024] Figure 5 An interpretable prototype-based classification decision process of an embodiment of the present application.

[0025] Figure 6 A schematic diagram of an image recognition process of an embodiment of the present application. DETAILED DESCRIPTION

[0026] The present application proposes a new prototype partial refinement Transformer, which reduces attention dispersion and improves the interpretability of image recognition models by introducing a prototype branch and a class-aware activation refinement mechanism.

[0027] In view of the limitations of existing prototype methods based on convolutional neural networks, the present application introduces a visual Transformer backbone network to extract prototypes and uses a class-aware activation map refinement method to solve the prototype activation dispersion problem, significantly improving the performance and interpretability of image recognition models. The specific technical solutions are as follows:

[0028] The present application discloses an interpretable image recognition method based on visual Transformer and prototype learning, which specifically comprises the following steps:

[0029] Step 1, data preparation and preprocessing: normalize the to-be-recognized images in the training set to a uniform size, divide the to-be-recognized images into a plurality of non-overlapping image patches, and then flatten each patch into a one-dimensional vector. Specifically, step 1 includes:

[0030] The to-be-recognized images in the training set are normalized to a size of HxW=224x224, the image patch size is set to LxL, and the original image is divided into non-overlapping image patches, and then flatten each patch into a one-dimensional vector (Reference: Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., & Jégou, H. (2021, July). Training data-efficient image transformers & distillation through attention. In International conference on machine learning (pp. 10347-10357). PMLR.); the flattened vector is mapped to a d-dimensional feature space through linear projection to generate an initial feature representation (Reference: Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T.,... & Houlsby, N. (2020, October). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations.).

[0031] Step 2: Feature extraction based on Transformer encoder: the flattened patch vector in step 1 is mapped to a feature space through linear projection to generate an initial feature embedding and input into the Transformer encoder, which performs self-attention calculation through stacked Transformer blocks to capture long-range dependencies of the image patch, and obtains the feature representation; step 2 specifically includes the following steps:

[0032] Step 2-1: Feature extraction using a visual Transformer encoder (Reference: Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T.,... & Houlsby, N. (2020, October). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations.) on the data preprocessed in Step 1; the encoder consists of multiple visual Transformer blocks (the number is not fixed, depending on the specific backbone network used, such as DeiT-Ti / DeiT-S with 12 Transformer blocks, CaiT-XXS / CaiT-S with 24, and Swin-T with 12), each block includes a multi-head self-attention mechanism and a feedforward neural network; self-attention computation is performed on the image blocks to capture long-range dependencies, and the encoded feature embeddings are passed to the next Transformer block; after all the Transformer blocks, the final image block feature representation is obtained, which is used as the input to the prototype part. The operation of this step is consistent with the processing flow of the visual Transformer encoder on image data in the prior art (see Dosovitskiy et al., 2020)

[0033] Step 2-2: To match the output feature representation of the visual Transformer encoder and the dimension of the prototype branch in Step 3, a single-layer 1x1 convolution layer is added after the visual Transformer encoder, and finally the feature representation t of the input image is obtained, which contains local feature representation and non-local feature representation.

[0034] The visual encoder configuration includes DeiT, CaiT, and Swin Transformer network models.

[0035] Step 3: Building the prototype branch: As shown in Figure 2 and 3 , the prototype branch is designed according to the structural characteristics of the visual Transformer model and consists of a double-branch prototype layer and a single-branch prototype layer; Step 3 specifically includes the following steps:

[0036] Step 3-1: Design the prototype branch, each local prototype branch contains C linear subspaces, each subspace is dedicated to representing the feature mode of a class. Each subspace is composed of M learnable local prototype vectors wherein, denotes the m-th local prototype vector, ρ1 denotes the height of the local prototype vector, ρ2 denotes the width of the local prototype vector, and E denotes the channel dimension of the local prototype vector. The local prototype vector can be

[0037] feature maps generated from the visual Transformer encoder to extract local feature representation; set the size of the prototype vector and wherein, denotes the height of the feature map, denotes the width of the feature map. The default values of ρ1 and ρ2 are 1 to ensure that the prototype can match and cover the corresponding local features.

[0038] Each non-local prototype branch also contains C linear subspaces, each of which is composed of M learnable non-local prototype vectors wherein, denotes the m-th non-local prototype vector; the non-local prototype branch learns by comparing with the pre-defined base prototype vector through the classification label, captures and obtains the global feature representation.

[0039] Step 3-2: Assemble the double-branch prototype layer and the single-branch prototype layer:

[0040] In the double-branch prototype layer, the local prototype branch and the non-local prototype branch are included. The non-local prototype branch is used to process the global feature representation containing the classification label, and the local prototype branch is used to process the local feature representation containing the visual label.

[0041] In the double-branch prototype layer, the feature representation of step 2-2 is divided according to the label and is respectively taken as the input of the local prototype branch and the non-local prototype branch; the local prototype vector and the non-local prototype vector are optimized so that they can effectively represent the local features and non-local features of each class;

[0042] In the single-branch prototype layer, the local prototype branch is included. The local feature representation of step 2-2 is directly taken as the input of the single-branch prototype layer, and the local prototype vector is optimized so that it can effectively represent the local features of each class;

[0043] Step 3-3: Determine the type of the visual Transformer encoder in step 2. Since the DeiT and CaiT encoders contain not only visual labels but also classification labels, the double-branch prototype layer is used. Since the Swin Transformer only contains visual labels, the single-branch prototype layer is used.

[0044] Step 3-4: Optimize the local prototype vectors and non-local prototype vectors described in step 3-1, design the total loss function Specifically as follows:

[0045] Wherein, λ1, λ2 and λ3 are weight coefficients of different loss terms, is the cross-entropy loss, is the orthogonality loss, is the clustering loss, is the separation loss.

[0046] Compare the local feature representation with the local prototype, and compare the non-local feature representation with the non-local prototype to make class assignment; Specifically, in the local prototype branch, compare the local feature representation of the image to be recognized with the local prototype vector; In the non-local prototype branch, compare the non-local feature representation of the image to be recognized with the non-local prototype vector; The image to be recognized is assigned to the class composed of the prototype most similar to its local feature representation and non-local feature representation:

[0047]

[0048] In turn, calculate the probability distribution of the image to be recognized in the local prototype branch and the non-local prototype branch belonging to class y i , the calculation formula is:

[0049]

[0050] Wherein, the probability calculated by the local prototype branch depends on the similarity between the image to be recognized and the local prototype vector of each class, and the probability calculated by the non-local prototype branch depends on the similarity between the image to be recognized and the non-local prototype vector of each class.

[0051] Optimize by minimizing the cross-entropy loss of the probability of the input sample belonging to the prototype of its real class:

[0052]

[0053] Connect the local prototype vectors described in step 3-1 into a local prototype matrix P l , and connect the non-local prototype vectors into a non-local prototype matrix P n . In turn, calculate the orthogonality loss on the local prototype matrix P i and the non-local prototype matrix P n of each class. This loss term is calculated by Frobenius norm, which is used to constrain the orthogonality between prototype vectors, so as to ensure the semantic diversity of prototype vectors of different classes:

[0054] ​

[0055] where, is the Frobenius norm, I M denotes an identity matrix of size M x M. In the local prototype branch, P (c) denotes the local prototype matrix of class c; in the non-local prototype branch, P (c) denotes the non-local prototype matrix of class c.

[0056] The clustering loss of local prototype vectors and non-local prototype vectors is calculated in turn and separation loss to facilitate meaningful prototype learning; specifically, and The loss terms respectively make the following constraints on each batch of training images: contain image patches similar to at least one prototype in the same class, and far away from prototypes not belonging to its class:

[0057]

[0058] where, denotes the number of training images, and T denotes the image patches represented by local feature representations.

[0059] The local prototype space constructed by the method is shown in Figure 4 The input samples of the same class are gathered around the corresponding class prototype and separated from other classes.

[0060] Step 4: Calculate the similarity between local feature representation and non-local feature representation and prototype in turn: up to step 3, each prototype layer has mapped the training data to a metric space, in which the classification can be carried out according to the similarity between feature representation and prototype. Specifically, it includes the following steps:

[0061] Step 4-1: for each input sample feature representation t, by calculating the distance between it and all prototype vectors in each class to determine the similarity between the feature and the class prototype; this distance measure represents the minimum rotation required to rotate one subspace to another; the principal angle is used to quantify the distance between subspaces, and the calculation formula is the dot product of the prototype and each feature vector:

[0062]

[0063] where, denotes the matrix transpose;

[0064] The distance is calculated by the dot product of the prototype and each feature vector, which represents the minimum rotation angle from one subspace to another.

[0065] Step 4-2: Convert the distance between local features and local prototypes into a similarity score according to the following formula to form an initial activation map S m which reflects the spatial location of prototypes p m in the image, which can then be indicated by an up-sampling operation to indicate the intensity of the prototype's presence:

[0066]

[0067] According to the similarity score generate an initial activation map S m .

[0068] Step 5: Class-aware activation refinement: calculate the diffusion index (DI) of each prototype activation map according to the activation map, and use the index to quantify the dispersion degree of the activation region; use the class-aware activation refinement mechanism to process the generated prototype activation map to reduce noise and make the activation region of the prototype more concentrated in the image region related to the target class; Specifically, step 5 mainly includes the following steps:

[0069] Step 5-1: Calculate the Gini coefficient to measure the dispersion degree of the pixel activation value in the prototype activation map, defined as follows:

[0070]

[0071] where S i and S j represent the absolute value of the i-th and j-th dimension in the prototype activation map, n represents the size of the activation map, and S l represents the average value of the activation map; the Gini coefficient G has a value range of [0, 1], when G = 0, it means that all dimensions of the pixel activation value are the same, when G = 1, it means that only one dimension of the pixel activation value is the strongest, while other dimensions of the pixel are not activated.

[0072] Step 5-2: Convert the Gini coefficient G to a diffusion index (Diffusion Index) through the formula DI = (1-G) x 100, the diffusion index has a range of [0, 100], a larger DI value indicates that the pixel activation value in the activation map is more concentrated in a few dimensions, representing the dispersion of attention; use the calculated diffusion index to evaluate the effectiveness of the prototype activation in the model; by controlling the DI value, the attention allocation of the model can be optimized, reducing the negative impact of attention dispersion on model performance, thereby improving the interpretability of the prototype.

[0073] Step 5-3: Calculate the attention weight A l of the multi-head self-attention module in the l-th Transformer block:

[0074]

[0075] where Q l and K l are the projected key and value representations from the input of the l-th Transformer block, with dimension D and H attention heads; to integrate the semantic context captured by all heads, the average of the attention vectors of all heads is computed as To obtain a single attention map, these attention vectors are aggregated over the Transformer block as

[0076]

[0077] and reshaped as This matrix captures the affinity of spatial features.

[0078] Step 5-4: Since the neurons of the visual Transformer have polyhedral properties, the prototypes usually respond to a wide distribution of image patches, resulting in noise in the activation map; for this purpose, the attention map obtained in step 3-6 is refined by class-aware activation, which is element-wise multiplied with the original activation map S to generate the refined activation map S ′ , so that the response area of the prototype is more concentrated:

[0079]

[0080] Step 5-5: The similarity score of each prototype is reduced to a compact maximum value by global max-pooling Specifically as follows:

[0081]

[0082] It is worth noting that global max-pooling is not applied in the non-local branch because its size is already 1; if t is the image patch with the highest similarity to the prototype p m , the function monotonically decreases with distance; therefore, the maximum value indicates that there is an image patch that is projected near the m-th prototype in the prototype space.

[0083] Step 6: Construct a prototype-based linear classifier: calculate the similarity between the feature representation of the test image and the prototype, and the classifier outputs the predicted class of the image according to the weighted sum of the similarity scores of different class prototypes; specifically, step 6 mainly includes:

[0084] In the form of a sparse score table, the similarity scores calculated in step 4 are input as and the final classification probability z is obtained according to the corresponding weights; for prototypes of the same class, the weights of the linear classifier are initialized as​ to ensure that these prototypes make positive contributions to the classification results of this category; for prototypes of other categories, the weights are initialized as so that these prototypes do not contribute to the scores of this category; for models containing double-prototype branches (such as simultaneously learning non-local and local prototypes), the non-local probability z n and the local probability z l are calculated respectively, where z n is usually calculated based on the CLS token, and z l is calculated based on the visual token; the non-local probability z n and the local probability z l are integrated to generate the final output z c :

[0085] z c = λ l z l + λ n z n

[0086] where λ n and λ l are weight coefficients of the double-branch, used to balance the influence of the non-local and local branches on the final classification result. For single-branch prototype layers, the local probability z l based on the visual token is directly taken as the final classification probability z c .

[0087] Step 7: Prototype visualization: In the test stage, the prototype is mapped to the image space by matching the feature activation map with the closest training image, generating a visualized prototype representation. The visualization of the prototype and the weights of the linear classifier constructed in step 6 are combined to explain the decision-making process of the model, showing how the model classifies by identifying key feature regions, making the decision-making process of the model more transparent. The whole process is shown in Figure 5 , specifically, step 7 mainly includes the following steps:

[0088] For a given prototype p m and an image to be recognized x, the visualization of the prototype in the original pixel space is represented by selecting the long-range dependent feature representation t of the image patch most similar to the prototype belonging to the category c, and the projection process can be represented as:

[0089]

[0090] The activation map S' is upsampled to match the size of the input image, determining the exact location of the prototypes and providing a basis for subsequent prototype visualization; by identifying and calibrating the smallest rectangular region containing more than 95 percent of the activation values in the image x, this rectangular region represents the most representative prototype activation pattern in the image; through the visualization of each prototype, it is interpreted as a certain activation pattern in the latent feature space, and it is mapped to a specific image block in the original pixel space; this process enhances the interpretability of the model, making the specific meaning of the prototype more intuitive visually.

[0091] The self-explaining prototype partial network model designed by the application based on visual Transformer backbone network not only significantly improves the interpretability and performance of the model, but also ensures its wide applicability and stability in different application scenarios. The method mainly includes six steps of data preparation and preprocessing, feature extraction based on Transformer encoder, construction of prototype branch, construction of linear classifier based on prototype, class-aware activation refinement, and prototype visualization and model explanation. Data preparation and preprocessing is to normalize the original image and divide it into multiple flattened one-dimensional vector patches; feature extraction based on Transformer encoder is to extract the long-range dependency feature representation of the image through linear projection and Transformer encoder; constructing the prototype branch is to design non-local and local prototype branches, and to optimize the prototype vector to represent the feature representation of each class; constructing a linear classifier based on the prototype is to output the predicted class of the image according to the similarity score of the feature representation and the prototype; class-aware activation refinement is to calculate the diffusion index and refine the prototype activation map to reduce noise and focus on the relevant image area; prototype visualization and model explanation is to visualize the prototype and combine the classifier weight to explain the decision-making process of the model.

[0092] Embodiment:

[0093] The following takes the bird data fine-grained classification task as an example to further illustrate the application. The application aims to learn a set of interpretable base prototypes to establish the connection between the feature representation from the visual Transformer encoder and the 200 output classes. These prototypes construct a transparent embedding space for representing the training samples. The application generates output predictions by comparing the representation of unseen test images with a set of prototypes, not only providing classification probabilities, but also providing the prototypes on which the predictions rely, thereby improving the interpretability of the model. As shown in Figure 1 The application constructs an interpretable image recognition method based on visual Transformer and prototype learning, which mainly includes seven steps:

[0094] Step 1, data preparation and preprocessing:

[0095] To facilitate image processing by the model, the original images are scaled, and the resolution of all images is adjusted to 224×224. Therefore, the size of the batch input images is B×3×224×224, where B is the batch size. The input images are processed through an image patch embedding layer, which involves the following two cases:

[0096] For DeiT and CaiT encoders, the following steps are included:

[0097] S1: Divide the image X into multiple non-overlapping image patches with a side length of 16, and flatten each image patch into a one-dimensional vector. The resulting sequence of one-dimensional vectors is X = {x1, x2, ..., x...} 196}

[0098] S2: Then it is linearly mapped to a higher-dimensional feature space to represent the local features of the image, generating an embedding vector z0 = [x1ε; x2ε; ...; x 196 ε}, where ε is the projection matrix.

[0099] A special classification label, a learnable vector, is added to the beginning of the input sequence to represent the global information of the entire image, resulting in the sequence X. embed =[cls;x1ε;x2ε;…;x 196 [ε]. The classification token is expanded to match the batch size of the input images. Specifically, the CLS token has a shape of 1×1×E, where E is the feature dimension, and is expanded to a shape of B×1×E to accommodate batch processing.

[0100] S3: Add positional embeddings to the sequence containing classification and visual labels. Positional embeddings are used to encode information at each location in the input image to preserve the spatial structure of the image. A position drop (pos_drop) operation is applied to regularize the embedding representation to reduce the risk of overfitting.

[0101] S4: After the above steps, the embedding representation is divided into two parts: the first part is the classification label, with a shape of B×1×E; the second part is the visual label, with a shape of B×196×D, where N represents the number of image patches. The processed classification label and visual label are concatenated along dimension 1 to generate a new embedding representation with a shape of B×197×E, which is then passed to the subsequent Transformer block.

[0102] For the Swing Transformer encoder, the following steps are included:

[0103] S1: Since Swin Transformer is different from the standard vision Transformer, it introduces a hierarchical local window attention mechanism, while introducing global information through sliding windows. First, the image is divided into multiple non-overlapping image patches with a side length of 4, and each image patch is flattened into a one-dimensional vector. The image patch is further divided into local windows with a size of 7x7, each window containing multiple adjacent image patches.

[0104] S2: In each local window, the relative position between image blocks is encoded into a learnable two-dimensional relative position offset table using relative position encoding. The size of the table is where is the window size. For each pair of image patches within the window, the relative position difference between them in the two-dimensional plane is calculated. This offset table can cover all possible relative position relationships from the top left to the bottom right of the window. Each window has a position difference in the horizontal and vertical directions of the image blocks, and these offsets are used to generate relative position encoding.

[0105] S3: Swin Transformer adopts a hierarchical structure, as the network layer deepens, the size of the window remains unchanged, but the resolution of the feature map gradually decreases, while the dimension of the feature is increased. After the above steps, the shape of the embedding representation is Bx49x E, and it is passed to the subsequent Transformer block.

[0106] Step 2, feature extraction based on Transformer encoder:

[0107] A Transformer encoder pre-trained on the ImageNet dataset is used for feature extraction, such as: deit_tiny_patch16_224, deit_small_patch16_224, cait_xxs24_224, cait_s24_224 and swin_tiny_patch4_window7_224. Their parameter and computational amounts are different, and the sizes of the divided image patches, embedding dimensions and network depths are also different.

[0108] Each Transformer block in these vision Transformer models contains layer normalization, multi-head attention module and feed-forward neural network. Among them, the multi-head attention module calculates the attention distribution between the entire input sequence to capture the global relationship between different parts of the image. The feed-forward neural network further processes the features of each position after each self-attention operation. Usually, these feed-forward networks contain two linear layers and a ReLU activation function. After each self-attention module and feed-forward network, residual connection and layer normalization are used to stabilize the training process and enhance the representation ability of the model.

[0109] For DeiT and CaiT, the following steps are mainly included:

[0110] S1: For each input vector, calculate the query Query, key Key and value Value vectors, Q = X embed W Q , K = X embed W k , V = X embed W V , where W Q , W K , W V are learnable weight matrices.

[0111] S2: Calculate the attention weight of each input position with other positions through the self-attention mechanism to capture long-range dependencies: Multiple attention heads are calculated at the same time, and then their results are spliced together to enrich the expression ability of the model: MultiHead(Q, K, V) = Concat(head1, …, head H )W O , where H is the number of attention heads, W O is the output weight matrix.

[0112] S3: Add residual connection to the output of multi-head self-attention and perform layer normalization:

[0113] X embed ′ = LayerNorm(X embed + MultiHead(Q, K, V));

[0114] S4: The feed-forward neural network includes two linear layers and a ReLU activation function:

[0115] FFN(X embed ′) = ReLU(X embed ′W1 + b1) + b2;

[0116] Residual connection and layer normalization are used again: Xembed " = LayerNorm(X embed ' + FFN(X embed ' ) ) ;

[0117] S5: Repeat steps S1-S4 until the end of the last Transformer block. Meanwhile, the attention weights of the last Transformer block need to be returned for class-aware activation refinement in step 5.

[0118] Step 3, build prototype branch:

[0119] Different prototype layers are designed for different visual Transformer encoders; for DeiT and CaiT encoders, a double-branch prototype layer is used to learn global prototypes and local prototypes, while for Swin Transformer, a single-branch prototype layer is used. Figure 2 and Figure 3 The structure diagrams of double-branch and single-branch models are given respectively.

[0120] Local prototype branch: learn a set of prototype vectors P n = {p1, p2, …, p 1000}, which form 200 linear subspaces, each corresponding to a class of the dataset. Each subspace is composed of 5 learnable prototype vectors, which are dedicated to representing the feature patterns of a class, and can extract local feature representations from the feature maps generated by the Transformer encoder, performing the following steps:

[0121] S1: Construct prototype vectors on the Grassmann manifold with size Bx1000x1x1;

[0122] S2: Compare the local prototype vector p l with the local feature t l = T l [i,j,:] at the spatial position (i,j), first use cosine convolution, then normalize and project the calculated distance to get the distance δ(t l , p l );

[0123] S3: Convert the distance between each position of the local feature and the prototype to a similarity to get the activation map S m , which records the response degree of each region of the image to the prototype. The closer to the prototype, the higher the activation value;

[0124] S4: Aggregate the activation map into a single value by global max pooling max_pool2d;

[0125] Global prototype branch: learn a set of prototype vectors P n = {p1, p2, …, p 1000} Each prototype branch also contains 200 linear subspaces, each consisting of 5 learnable prototype vectors that can extract global feature representations from cls_token, performing the following steps:

[0126] S1: Construct prototype vectors on the Grassmann manifold, size B x 1000 x 1 x 1;

[0127] S2: Compare the global prototype vector p n with the global feature t n = T n [1, 1, :], calculate the distance δ(t n , p n ), and convert it to similarity;

[0128] S3: Since the sequence length of the CLS token is 1, this step omits the step of integrating the similarity into a single value using global max pooling max_pool2d;

[0129] In order to learn meaningful prototype space, the invention designs orthogonality loss, clustering loss and separation loss to optimize prototype vectors in addition to the basic cross-entropy loss.

[0130] S1: Cross-entropy loss ensures that the model can correctly classify, minimize the cross-entropy of the probability that the input sample belongs to the prototype of its true category, the calculation formula is

[0131] S2: Orthogonality loss is used to ensure that the redundancy between prototypes in the same category is minimized, so that the prototype is representative and has distinguishability, the calculation formula is where is the Frobenius regularization term, I M represents a unit matrix of size M x M.

[0132] S3: Clustering loss encourages prototypes of the same category to be as close as possible to image features of the corresponding category, the calculation formula is where is the number of images in the dataset.

[0133] S4: Separation loss ensures that prototypes of different categories maintain a distance from image features of other categories, the calculation formula is

[0134] S5: Weighted sum of all loss terms, get λ1, λ2, and λ3 are the coefficients of different loss terms, set to 0.001, 0.8, and -0.2, respectively. S1 to S5 are executed iteratively until the loss is calculated for all batches of samples.

[0135] Step 4: Calculate the similarity between the feature representation and the prototype:

[0136] Up to step 3, the prototype layer has mapped the training data to a metric space, in which classification can be performed based on the similarity between the feature representation and the prototype. Perform the following steps:

[0137] S1: For each input sample's feature representation t, calculate its distance to all prototype vectors in each class. This determines the similarity between features and prototypes of each category; this distance metric represents the minimum amount of rotation required to rotate from one subspace to another; the principal angle is used to quantify the distance between subspaces, calculated as the dot product of the prototype and each feature vector:

[0138] δ(t,p m )=t . p m ;

[0139] This distance is calculated by the dot product of the prototype and each eigenvector, representing the minimum rotation angle from one subspace to another.

[0140] S2: Calculate the distance between the local feature and the prototype according to the following formula. Convert to similarity scores to form the initial activation map D. m This activation map reflects the prototype p in the image. m The spatial location of the prototype can then be used to indicate the intensity of its presence through upsampling:

[0141]

[0142] Based on similarity score Generate the initial activation graph S m .

[0143] Step 5, Refine class-aware activation:

[0144] Because ViT neurons are multifaceted, the prototype typically responds to various scattered image patches, resulting in high noise in the activation map. To address this issue, this invention utilizes category-aware activation refinement to force the prototype to narrow its response region and extract category information from noise-free latent features, mainly including the following steps:

[0145] S1: To evaluate the dispersion of pixel activation values in the prototype activation map, the Gini coefficient is introduced. The Gini coefficient is a commonly used indicator to measure the degree of distribution imbalance. In this method, it is applied to measure the imbalance of pixel values in the activation map, and the calculation formula is:

[0146]

[0147] where S i represents the absolute value of the i-th dimension in the prototype activation map, represents the average value of the activation map. This formula measures the uniformity of the activation map by comparing the activation value difference of each pair of pixels. The Gini coefficient ranges from 0 to 1, and its physical meaning is: when G = 0, it means that the pixel activation values of all dimensions are exactly the same, indicating that the model pays equal attention to all parts of the image, and the distribution of the activation map is very uniform; when G = 1, it means that only one dimension of pixel activation value is the highest, while all other dimensions of pixel are not activated, indicating that the model focuses attention on a specific area. By calculating the Gini coefficient, we can understand the attention concentration or dispersion of the prototype activation map, helping model designers adjust the attention distribution of the model.

[0148] S2: To further quantify the attention dispersion of the model, the Gini coefficient is converted to a diffusion index. The definition of the diffusion index is DI = (1-G) x 100, and its value range is [0, 100]. When DI is larger, it means that the activation value is concentrated in a few dimensions, and the attention is dispersed, and the model may be too focused on some specific areas while ignoring other areas; when DI is smaller, it means that the attention of the model is more evenly distributed on the entire image.

[0149] S3: In the Transformer model, each Transformer block captures the long-range dependencies between different parts of the input image through self-attention mechanisms. For the l-th Transformer block, the attention weight of the multi-head self-attention module is:

[0150]

[0151] Through this step, the model can calculate the dependency between different positions in the input, thereby capturing the global context information of the image.

[0152] S4: The multi-head attention mechanism in the Transformer model processes different parts of the input in parallel through multiple heads. To obtain more complete image semantic information, the average value of the attention vectors of all heads needs to be calculated: After integrating the multi-head attention vectors, further aggregate these attention vectors over all the Transformer blocks: Finally, reshape the matrix as Capture the spatial affinity between different regions of the image.

[0153] S5: Introduce class-aware activation refinement, multiply the attention map with the original activation map S element-wise to generate a more refined activation map: Through this element-wise multiplication operation, the model can filter out irrelevant activation information, making the response area of the prototype more focused on the image region related to the target class, reducing the interference of background noise.

[0154] S6: For the generated refined activation map, use global max pooling to extract the maximum response value of each prototype unit: Through this operation, redundant information in the activation map can be reduced, and only the most representative activation region is retained. It should be noted that global max pooling is not applied in the non-local branch because its size is already 1.

[0155] Step 6, construct a prototype-based linear classifier:

[0156] In the form of a sparse score table, use the similarity scores of the prototype vectors as input. These similarity scores are obtained by calculating the similarity between the input image features and the prototype vectors in step 3. The main steps include:

[0157] S1: Initialize the weight matrix W based on the forward reasoning principle cls where the weights of prototypes of the same class are initialized to These weights ensure that prototypes of the same class make a positive contribution to the classification result of that class. For prototypes of other classes, the initialization weight is to ensure that these prototypes do not contribute to the score of the current class. In this way, the classification layer can only calculate the classification score based on the similarity of prototypes of the same class as the input image, thereby ensuring the accuracy of classification.

[0158] S2: For models with dual prototype branches, calculate the classification probability of non-local prototypes and local prototypes respectively. The probability of the non-local branch is usually calculated based on the classification label, which is a representation of the global features of the input image, and the similarity of the non-local prototype determines the non-local classification probability z n . The probability of the local branch z l is calculated based on the visual label, which represents the similarity of the local features of the input image and the local prototype. To generate the final classification output, the two are weighted and summed. Use weight coefficients λ n and λ lto balance the influence of non-local and local information on the classification result: z c = λ l z l + λ n z n , the hyperparameter is set to λ n = λ l = 0.5.

[0159] Step 7, visualization and interpretation:

[0160] This step contains how to enhance the interpretability of the model by visualizing the prototype after the end of model training, given a training image.

[0161] Given a prototype p and a training image x, the model will find the most similar image patch t in the feature space to the prototype, which comes from the image sample of the same category as the prototype. The mapping of the prototype can be represented by the following formula: In order to determine the specific location of the prototype in the input image, the activation map is upsampled to match the size of the original input image. The upsampled activation map can show the exact location of the prototype in the original image, helping to find the activation area of the prototype in the visualization process. In the upsampled activation map, calculate the activation value of each pixel, and box out the area with activation value higher than 95 percentile with a rectangular frame to form the smallest bounding box. This rectangular area represents the most representative prototype activation pattern in the image, which is the most concerned part of the model.

[0162] Through the above process, the prototype is mapped to the original pixel space, and the specific image block corresponding to the prototype is displayed through the high activation value area. The visualization of each prototype can be explained as a certain activation pattern in the latent feature space, which is intuitively displayed through the image patch in the original pixel space. Through this prototype visualization method, the decision-making process of the model becomes more transparent and intuitive, allowing users to see how the model makes classification decisions by identifying key areas in the image.

[0163] For example Figure 6The illustrated is the explainable reasoning process of the present application, when a test image with a category label of "chestnut-sided warbler" is input, the method of the present application can provide an explainable reasoning process according to the learned prototypes and visualizations in the training samples, which are given in columns 2 to 4. The first prototype of the chestnut-sided warbler class obtains the maximum similarity score of 7.47 in the feature combination of yellow crown, black eyes and chestnut side pattern, and the second prototype obtains a similarity score of 2.93 in the yellow and black wing pattern. The original test image is decomposed into blocks, and the learned prototypes are identified as the evidence of prediction. The most active image blocks in the original test image are marked in the bounding box of the first column and highlighted in the activation map.

[0164] In a specific implementation, the present application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium can store a computer program, and the computer program can run the invention content of the explainable image recognition method based on visual Transformer and prototype learning and part or all steps in each embodiment of the present application when executed by the data processing unit. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0165] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present application can be realized by means of a computer program and its corresponding general hardware platform. Based on such understanding, the technical solutions in the embodiments of the present application or the part that essentially contributes to the prior art can be embodied in the form of a computer program, i.e. a software product, which can be stored in a storage medium, including a plurality of instructions for causing a device (which can be a personal computer, a server, a single-chip microcomputer, an MCU or a network device, etc.) containing a data processing unit to execute the method described in each embodiment or some part of the embodiments of the present application.

[0166] The present application provides a thought and method of an explainable image recognition method based on visual Transformer and prototype learning. There are many methods and ways to realize the technical solutions, and the above description is only the preferred embodiment of the present application. It should be pointed out that, for ordinary skilled in the art, without departing from the principles of the present application, some improvements and refinements can be made, which should also be regarded as the protection scope of the present application. The components not explicitly described in the embodiments can be realized by using the existing technology.

Claims

1. An interpretable image recognition method based on visual Transformer and prototype learning, characterized in that, Includes the following steps: Step 1: Perform image preprocessing. Normalize the images to be identified in the training data and divide them into multiple non-overlapping image patches. Then flatten each image patch into a one-dimensional vector. Step 2: Extract long-range dependency feature representations of the image patch using different types of visual Transformer encoders. The long-range dependency feature representations include non-local feature representations or local feature representations. Step 3: Design prototype branches and form a two-branch prototype layer and a single-branch prototype layer. The prototype branches include non-local prototype branches and local prototype branches. Each prototype branch maps the training data to a metric space and defines a set of prototypes representing the non-local feature representation or the local feature representation. Step 4: In the metric space, depending on the type of visual Transformer encoder described in Step 2, the similarity between the non-local feature representation and the local feature representation described in Step 3 and the prototype is calculated sequentially through a two-branch prototype layer or a single-branch prototype layer. Step 5: The activation map is obtained by similarity transformation described in Step 4. The activation map is processed using the class-aware activation refinement method, and the diffusion index is used to quantify the dispersion of the activation regions in the activation map. Step 6: Construct a linear classifier. The input is the similarity between the non-local feature representation or the local feature representation and the prototype. The output is the probability distribution of the image category. Calculate the predicted category of the image to be identified based on the probabilities, which is the image recognition result. Step 7: Map the prototype to the original image space according to the class-aware activation refinement method described in Step 5 to generate a visual representation. Combine the input and weights of the linear classifier described in Step 6 to explain the acquisition process of the image recognition result, and complete the interpretable image recognition based on visual Transformer and prototype learning. Specifically, step 5 involves processing the activation map using a class-aware activation refinement method and quantifying the dispersion of activation regions in the activation map using a diffusion index. Step 5-1: Calculate the Gini coefficient G, which is used to measure the activation map S. m The dispersion of the activation values ​​of the mid-pixels is as follows: Among them, S i and S j Indicates activation graph S m The absolute values ​​of the i-th and j-th dimensions, where n represents the size of the activation map. Indicates activation graph S m The average value of the Gini coefficient G is [0,1]. When G=0, it means that the pixel activation values ​​of all dimensions are exactly the same. When G=1, it means that only one dimension of the pixel has the strongest activation value, while the pixels of other dimensions are not activated. Step 5-2: Convert the Gini coefficient G to the diffusion index DI. The calculation method is as follows: DI = (1-G) × 100 The diffusion index DI ranges from [0, 100]. The larger the DI value, the higher the dispersion of attention. The diffusion index DI is used to optimize the attention distribution of the visual Transformer encoder. Step 5-3: Calculate the attention weight A of the multi-head self-attention module in the l-th visual Transformer block of the visual Transformer encoder. l The method is as follows: Among them, Q l and K l It is the query and key of the l-th visual Transformer block, with dimension D and number of attention heads H; calculate the average attention vector of all attention heads. aggregate them into a matrix The method is as follows: Where L represents the number of visual Transformer blocks; matrix Remodeling into a matrix Obtain attention map Step 5-4: Use class-aware activation refinement to refine the attention map. With activation graph S m Element-wise multiplication generates a fine activation graph S. ′ The details are as follows: Where ⊙ represents element-wise multiplication; Step 5-5: Reduce the similarity score of each prototype to a compact maximum value using global max pooling. Specifically as follows:

2. The interpretable image recognition method based on visual Transformer and prototype learning according to claim 1, characterized in that, The image preprocessing described in step 1 specifically includes: Step 1-1: Collect the training set and normalize the images to be recognized in the training set to a preset size; Step 1-2: Set the image patch size to L×L, and divide the image to be identified into R non-overlapping image patches, as follows; Where H and W represent the height and width of the image to be recognized, respectively; Steps 1-3: Flatten each image patch into a one-dimensional vector; Steps 1-4 involve mapping the flattened one-dimensional vector to the d-dimensional feature space via linear projection to generate an initial feature representation.

3. The interpretable image recognition method based on visual Transformer and prototype learning according to claim 2, characterized in that, Step 2, which describes extracting long-range dependency feature representations of the image patch using different types of visual Transformer encoders, specifically includes: Step 2-1: Use a visual Transformer encoder to extract features from the initial feature representation described in Steps 1-4, as follows: The visual Transformer encoder is used to perform self-attention computation on the initial feature representation, capture long-range dependencies, and obtain the feature representation of the image patch. The visual Transformer encoder is composed of visual Transformer blocks, each of which includes a multi-head self-attention mechanism and a feedforward neural network. The visual Transformer block encodes the initial feature representation of the input and embeds the encoded feature representation into the next layer of visual Transformer blocks. After passing through all visual Transformer blocks, the feature representation of the image patch is obtained. Step 2-2: After the visual Transformer encoder, add a single 1×1 convolutional layer to obtain the long-range dependency feature representation t of the image patch, which includes non-local feature representation or local feature representation.

4. The interpretable image recognition method based on visual Transformer and prototype learning according to claim 3, characterized in that, Step 3, which involves designing prototype branches and forming a dual-branch prototype layer and a single-branch prototype layer, specifically includes: Step 3-1, design the prototype branch, as follows: Each local prototype branch contains C linear subspaces, where C represents the number of classes. Each subspace is dedicated to representing the feature patterns of one class; each subspace consists of M learnable local prototype vectors. Composition, among which, Let P represent the m-th local prototype vector, ρ1 represent the height of the local or non-local prototype vector, ρ2 represent the width of the local or non-local prototype vector, and E represent the channel dimension of the local prototype vector; the local feature representation obtained by the visual Transformer encoder is reconstructed into a feature map, and the local prototype vector P... l By comparing and learning with the feature map, the local prototype vector P l Size set to and Indicates the height of the feature map. Indicates the width of the feature map; Each nonlocal prototype branch also contains C linear subspaces, each subspace consisting of M learnable nonlocal prototype vectors. Composition, among which, This represents the m-th nonlocal prototype vector; the nonlocal prototype branch learns by comparing the classification label with the predefined base prototype vector, capturing and obtaining the global feature representation. Step 3-2 involves constructing a dual-branch prototype layer and a single-branch prototype layer, as detailed below: The dual-branch prototype layer includes local prototype branches and non-local prototype branches; The single-branch prototype layer includes local prototype branches; Step 3-3: Determine the type of the visual Transformer encoder mentioned in Step 2. For a visual Transformer encoder that includes classification tags, select to use a two-branch prototype layer; for a visual Transformer encoder that only includes visual tags, select to use a single-branch prototype layer. Each prototype branch in the dual-branch prototype layer and the single-branch prototype layer maps the training data to a metric space and defines a set of prototypes that can represent the non-local or local feature representations of the training data. Step 3-4: Optimize the local prototype vectors and non-local prototype vectors described in Step 3-1, and design the total loss function. Specifically as follows: Where λ1, λ2, and λ3 are the weighting coefficients for different loss terms. For cross-entropy loss, Orthogonality loss For clustering loss, This represents the separation loss.

5. The interpretable image recognition method based on visual Transformer and prototype learning according to claim 4, characterized in that, The cross-entropy loss described in steps 3-4 Specifically as follows: The image to be identified is assigned to the category most similar to its local and non-local features, as shown below: Where x represents the image to be identified; in the local prototype branch, t represents the local feature representation of x; in the non-local prototype branch, t represents the non-local feature representation of x; δ(·,·) is the distance formula, that is, in the local prototype branch, the local feature representation of the image to be identified is compared with the local prototype vector, and in the non-local prototype branch, the non-local feature representation of the image to be identified is compared with the non-local prototype vector. Calculate the category y of the image to be identified in the local prototype branch and the non-local prototype branch in turn. i The probability distribution is calculated as follows: Where, x i Representing the i-th image to be recognized, in the local prototype branch, t i x represents i Local feature representation; in the nonlocal prototype branch, t i x represents i Nonlocal feature representation; The probability calculated by the local prototype branch depends on the similarity between the local feature representation of the image to be identified and the local prototype vectors of each category, while the probability calculated by the non-local prototype branch depends on the similarity between the non-local feature representation of the image to be identified and the non-local prototype vectors of each category. The cross-entropy is the negative logarithm of the probabilities, calculated as follows: in, This indicates the number of images to be identified in the training data; Orthogonality loss Specifically as follows: Using the Frobenius norm, successively apply the local prototype vector p l The local prototype matrix P is composed of l and by the nonlocal prototype vector p n The nonlocal prototype matrix P is formed n Apply orthogonality constraints: in, For Frobenius regularization, I M P represents an identity matrix of size M×M; in the local prototype branch, P (c) The local prototype matrix represents category c; in the non-local prototype branch, P (c) The nonlocal prototype matrix represents category c; Calculate the clustering loss for local prototype vectors and non-local prototype vectors in turn. and separation loss Specifically as follows: Where T represents an image patch represented by local features.

6. The interpretable image recognition method based on visual Transformer and prototype learning according to claim 5, characterized in that, Step 4, which involves sequentially calculating the similarity between the local feature representation and the non-local feature representation and the prototype, specifically includes: Step 4-1: Calculate the similarity between the local feature representation t described in Step 2-2 and the local prototype vector described in Step 3-1. The calculation method is as follows: in, Indicates matrix transpose; Step 4-2, the similarity δ(t,p) is calculated. m Convert to similarity score Specifically as follows: Based on similarity score Generate the initial activation graph S m .

7. The interpretable image recognition method based on visual Transformer and prototype learning according to claim 6, characterized in that, Step 6, which describes constructing a linear classifier, includes the following steps: Step 6-1: Using a sparse scoring table, the similarity score between the prototype and the image to be identified is used as input; for prototypes of the same category, the weights in the linear classifier are initialized to 1; for prototypes of other categories, the weights are initialized to 0. Step 6-2: For the dual-branch prototype layer, calculate the nonlocal probability z based on the classification label. n And visually labeled local probability z l Among them, nonlocal probability is used to capture global features of the image, while local probability is used to capture features of local image patches; The final classification odds z are generated by weighting the nonlocal and local odds. c The output is calculated as follows: With c =λ l With l +λ n With n Where, λ n and λ l These are the weighting coefficients; For a single-branch prototype layer, the local probability z based on visual labels is directly applied. l As the final classification probability z c .

8. The interpretable image recognition method based on visual Transformer and prototype learning according to claim 7, characterized in that, Step 7, mapping the prototype to the original image space to generate a visual representation, specifically includes: For a given prototype p m Given the image to be identified x, the long-range dependency feature representation t of the image patch most similar to the category c to which the prototype belongs is used to represent the visualization of the prototype in the original pixel space. This process is expressed as: Among them, T (c) This represents all image patches of category c; The activation image S′ is upsampled to match the size of the input image to be identified, thereby determining the exact location of the prototype in the image to be identified; By identifying the smallest rectangular region in the image to be identified with an activation value higher than a preset ratio, the most representative prototype activation region in the image to be identified is determined; the visualization process of each prototype maps its activation mode in the latent feature space to an image block in the original pixel space, thus completing the interpretation of the acquisition process of the image recognition result.

9. The interpretable image recognition method based on visual Transformer and prototype learning according to claim 8, characterized in that, The visual Transformer encoder mentioned in step 2-2 is a DeiT network model, a CaiT network model, or a Swing Transformer network model.

Images