Explanatable image recognition method based on visual Transform and prototype learning

Abstract

The invention discloses an interpretable image recognition method based on visual Transform and prototype learning, and the method comprises the following steps: carrying out the image preprocessing, carrying out the normalization of a to-be-recognized image in training data, dividing the to-be-recognized image into a plurality of non-overlapping image patches, and flattening each image patch into a one-dimensional vector; using different types of visual Transform encoders to extract the long-range dependent feature representation of the image patch; prototype branches are designed, and a double-branch prototype layer and a single-branch prototype layer are formed; according to the type of a visual Transform encoder, through a double-branch prototype layer or a single-branch prototype layer, calculating similarity and converting to obtain an activation graph, and processing the activation graph; constructing a linear classifier to obtain an image recognition result; and explaining the acquisition process of the image recognition result to complete interpretable image recognition based on visual Transform and prototype learning.

Description

Technical Field

[0001] The invention relates to an image recognition method, in particular to an explainable image recognition method based on visual Transformer and prototype learning. Background Art

[0002] This section merely provides background information related to the present disclosure and is not necessarily prior art.

[0003] Deep learning has been widely used in computer vision, natural language processing, reinforcement learning, and recommendation systems, and has surpassed the performance of traditional machine learning methods and human experts in many fields. However, deep learning models are often regarded as "black box" models. Due to the massive model parameters and high nonlinearity, it is often difficult to deeply explore and mine their internal operating mechanisms, which limits users' trust in these models. In order to improve the transparency of deep learning models and promote users' understanding, trust, and management of models, interpretable methods for deep learning models have emerged. Such methods aim to provide users with model interpretation in understandable terms from the aspects of simulatability, decomposability, and algorithm transparency. Among these methods, prototypical part networks have shown great potential in enhancing the transparency of model decisions.

[0004] Existing prototype learning methods based on convolutional neural networks (CNNs) have some limitations, such as resulting in scattered and noisy prototype activations when directly applying prototype learning to Visual Transformers (ViTs).

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] Purpose of the invention: The technical problem to be solved by the present invention is to provide an interpretable image recognition method based on visual Transformer and prototype learning in view of the shortcomings of the prior art.

[0007] In order to solve the above technical problems, the present invention discloses an interpretable image recognition method based on visual Transformer and prototype learning, comprising the following steps:

[0008] Step 1: perform image preprocessing, normalize the image to be identified in the training data and divide it into multiple non-overlapping image patches, and then flatten each image patch into a one-dimensional vector;

[0009] Step 2, using different types of visual Transformer encoders to extract long-range dependent feature representations of the image patch, wherein the long-range dependent feature representations include non-local feature representations or local feature representations;

[0010] Step 3, designing prototype branches and forming a double-branch prototype layer and a single-branch prototype layer, wherein the prototype branches include a non-local prototype branch and a local prototype branch, each prototype branch maps the training data to a metric space and defines a set of prototypes representing the non-local feature representation or the local feature representation;

[0011] Step 4, in the metric space, according to the type of the visual Transformer encoder described in step 2, through a double-branch prototype layer or a single-branch prototype layer, sequentially calculating the similarity between the non-local feature representation and the local feature representation described in step 3 and the prototype;

[0012] Step 5, obtaining an activation map by the similarity conversion in step 4, processing the activation map using a class-aware activation refinement method, and quantifying the dispersion of activation areas in the activation map using a diffusion index;

[0013] Step 6, constructing a linear classifier, the input is the similarity between the non-local feature representation or the local feature representation and the prototype, and the output is the probability distribution of the image category; according to the probabilities, calculations are performed respectively to obtain the predicted category of the image to be recognized, that is, the image recognition result;

[0014] Step 7, mapping the prototype to the original image space according to the class-aware activation refinement method described in step 5, generating a visual representation, and combining the input and weight of the linear classifier described in step 6 to explain the process of obtaining the image recognition result, thereby completing interpretable image recognition based on visual Transformer and prototype learning.

[0015] Beneficial effects:

[0016] 1. The present invention introduces long-range dependencies between image representations, thereby more effectively capturing global features and improving the recognition accuracy of the model.

[0017] 2. The present invention designs special prototype branches for different Transformer backbone networks, and designs orthogonality loss, clustering loss and dispersion loss to ensure the diversity and accuracy of the prototypes.

[0018] 3. The present invention activates a refinement mechanism through class perception, constrains the prototype to effectively focus on the most recognizable part of the image, reduces the interference of irrelevant information, and thus significantly improves the interpretability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0020] Figure 1 Schematic diagram of the overall workflow of an embodiment of the present invention.

[0021] Figure 2 This is a calculation flow chart of a dual-branch network structure according to an embodiment of the present invention.

[0022] Figure 3 This is a single-branch network structure calculation flow chart of an embodiment of the present invention.

[0023] Figure 4 Schematic diagram of the prototype subspace of an embodiment of the present invention.

[0024] Figure 5 This is an explainable prototype classification-based decision process of an embodiment of the present invention.

[0025] Figure 6 FIG. 4 is a schematic diagram of an image recognition process according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] This paper proposes a novel prototype partial refinement Transformer, which reduces attention distraction and improves the interpretability of image recognition models by introducing prototype branches and class-aware activation refinement mechanisms.

[0027] In view of the limitations of existing prototype methods based on convolutional neural networks, this paper introduces a visual Transformer backbone network to extract prototypes, and adopts a class-aware activation map refinement method to solve the problem of prototype activation dispersion, significantly improving the performance and interpretability of the image recognition model. The specific technical solution is as follows:

[0028] The present invention discloses an interpretable image recognition method based on visual Transformer and prototype learning, which specifically comprises the following steps:

[0029] Step 1, data preparation and preprocessing: normalize the images to be identified in the training set to a uniform size, divide the images to be identified into multiple non-overlapping image patches, and then flatten each patch into a one-dimensional vector. Specifically, step 1 includes:

[0030] Normalize the image to be recognized in the training set to a size of H×W=224×224, set the image patch size to L×L, and divide the original image into non-overlapping image patches, and then flatten each patch into a one-dimensional vector (reference: Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., & Jégou, H. (2021, July). Training data-efficient image transformers & distillation through attention. In International conference on machine learning (pp. 10347-10357). PMLR.); the flattened vector is mapped to the d-dimensional feature space by linear projection to generate the initial feature representation (reference: Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., ... & Houlsby, N. (2020, October). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations.).

[0031] Step 2: Feature extraction based on Transformer encoder: The flattened patch vector in step 1 is mapped to the feature space through linear projection, the initial feature embedding is generated and input into the Transformer encoder, and self-attention calculation is performed on it through the stacked Transformer blocks to capture the long-range dependencies of the image patches and obtain the feature representation; Step 2 specifically includes the following steps:

[0032] Step 2-1: Use the visual Transformer encoder (reference: Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., ... & Houlsby, N. (2020, October). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations.) Feature extraction is performed on the data preprocessed in step 1; the encoder consists of multiple visual Transformer blocks (the number is not fixed, depending on the specific backbone network used, such as DeiT-Ti / DeiT-S is 12 Transformer blocks, CaiT-XXS / CaiT-S is 24, and Swin-T is 12), each block includes a multi-head self-attention mechanism and a feedforward neural network; self-attention calculation is performed on the image block to capture long-range dependencies, and the encoded feature embedding is passed to the next layer of Transformer blocks. After passing through all Transformer blocks, the final image block feature representation is obtained, which is used as the input of the prototype part. The operation of this step is consistent with the processing flow of the visual Transformer encoder on image data in the prior art (see Dosovitskiy et al., 2020)

[0033] Step 2-2: In order to match the output feature representation of the visual Transformer encoder and the dimension of the prototype branch in step 3, a single 1×1 convolution layer is added after the visual Transformer encoder, and finally the feature representation t of the input image is obtained, including local feature representation and non-local feature representation.

[0034] The visual encoder configuration includes DeiT, CaiT and Swin Transformer network models.

[0035] Step 3: Build a prototype branch: Figure 2 and 3 As shown in the figure, according to the structural characteristics of the visual Transformer model, the prototype branches are designed to form a double-branch prototype layer and a single-branch prototype layer; step 3 specifically includes the following steps:

[0036] Step 3-1: Design prototype branches. Each local prototype branch contains C linear subspaces, each of which is dedicated to representing the characteristic pattern of a category. Each subspace consists of M learnable local prototype vectors Composition, among which, represents the mth local prototype vector, ρ1 represents the height of the local prototype vector, ρ2 represents the width of the local prototype vector, and E represents the channel dimension of the local prototype vector. The local prototype vector can

[0037] Feature maps generated from the visual Transformer encoder Extract local feature representation from; Set the size of the prototype vector and in, represents the height of the feature map, Represents the width of the feature map. The default value of ρ1 and ρ2 is 1 to ensure that the prototype can match and cover the corresponding local features.

[0038] Each non-local prototype branch also contains C linear subspaces, each of which consists of M learnable non-local prototype vectors Composition, among which, Represents the mth non-local prototype vector; the non-local prototype branch captures and obtains the global feature representation by comparing the classification label with the predefined basic prototype vector.

[0039] Step 3-2: Composition of dual-branch prototype layer and single-branch prototype layer:

[0040] The dual-branch prototype layer includes a local prototype branch and a non-local prototype branch, wherein the non-local prototype branch is used to process the global feature representation including the classification mark, and the local prototype branch is used to process the local feature representation including the visual mark.

[0041] In the dual-branch prototype layer, the feature representation of step 2-2 is divided according to the label and used as the input of the local prototype branch and the non-local prototype branch respectively; the local prototype vector and the non-local prototype vector are optimized so that they can effectively represent the local features and non-local features of each category;

[0042] The single-branch prototype layer includes a local prototype branch. The local feature representation of step 2-2 is directly used as the input of the single-branch prototype layer, and the local prototype vector is optimized so that it can effectively represent the local features of each category;

[0043] Step 3-3: Determine the type of the visual Transformer encoder described in step 2. Since the DeiT and CaiT encoders contain not only visual tags but also classification tags, a dual-branch prototype layer is used; since the Swin Transformer only contains visual tags, a single-branch prototype layer is used.

[0044] Step 3-4: Optimize the local prototype vector and non-local prototype vector described in step 3-1 and design the total loss function The details are as follows:

[0045] Among them, λ1, λ2 and λ3 are weight coefficients of different loss terms, is the cross entropy loss, is the orthogonality loss, is the clustering loss, For separation loss.

[0046] The local feature representation is compared with the local prototype, and the non-local feature representation is compared with the non-local prototype to perform category assignment; specifically, in the local prototype branch, the local feature representation of the image to be identified is compared with the local prototype vector; in the non-local prototype branch, the non-local feature representation of the image to be identified is compared with the non-local prototype vector; the image to be identified is assigned to the category consisting of the prototype that is most similar to its local feature representation and non-local feature representation:

[0047]

[0048] Calculate the category y of the image to be identified in the local prototype branch and the non-local prototype branch in turn i The probability distribution of is calculated as:

[0049]

[0050] Among them, the probability calculated by the local prototype branch depends on the similarity between the image to be identified and the local prototype vectors of each category, and the probability calculated by the non-local prototype branch depends on the similarity between the image to be identified and the non-local prototype vectors of each category.

[0051] The optimization is performed by minimizing the cross entropy loss of the probability that an input sample belongs to its true class prototype:

[0052]

[0053] Connect the local prototype vectors described in step 3-1 to form the local prototype matrix P l , the non-local prototype vectors are connected to form the non-local prototype matrix P n . In each category, the local prototype matrix P i and the non-local prototype matrix P n Orthogonality loss is calculated on This loss term is calculated using the Frobenius norm and is used to constrain the orthogonality between prototype vectors, thereby ensuring that prototype vectors of different categories have semantic diversity:

[0054]

[0055] in, is the Frobenius regularization term, I M represents the identity matrix of size M×M. In the local prototype branch, P (c) Represents the local prototype matrix of category c; in the non-local prototype branch, P (c) Represents the non-local prototype matrix of category c.

[0056] Calculate the clustering loss of local prototype vectors and non-local prototype vectors in turn and separation loss To promote meaningful prototype learning; specifically, and The loss term imposes the following constraints on each batch of training images: containing image patches similar to at least one prototype in the same category, and staying away from prototypes that do not belong to their category:

[0057]

[0058] in, denotes the number of training images and T denotes the image patch represented by the local feature representation.

[0059] The local prototype space constructed by this method is as follows Figure 4 As shown, input samples of the same category are clustered around the corresponding category prototype and separated from other categories.

[0060] Step 4: Calculate the similarity between local feature representation and non-local feature representation and the prototype in turn: Up to step 3, each prototype layer has mapped the training data to a metric space, in which classification can be performed based on the similarity between the feature representation and the prototype. Specifically, the following steps are included:

[0061] Step 4-1: For each input sample’s feature representation t, calculate the distance between it and all prototype vectors in each class To determine the similarity between the features and the prototypes of each category; this distance metric represents the minimum rotation required to rotate one subspace to another subspace; the main angle is used to quantify the distance between subspaces, and the calculation formula is the dot product of the prototype and each feature vector:

[0062]

[0063] in, Represents matrix transpose;

[0064] The distance is calculated by taking the dot product of the prototype and each eigenvector and represents the minimum rotation angle from one subspace to another.

[0065] Step 4-2: The distance between the local feature and the local prototype is calculated according to the following formula Converted into similarity scores to form the initial activation map S m , the activation map reflects the prototype p in the image m The spatial position of can then be upsampled to indicate the strength of the prototype presence:

[0066]

[0067] Based on similarity score Generate initial activation map S m .

[0068] Step 5: Class-aware activation refinement: Calculate the diffusion index (DI) of each prototype activation map according to the activation map, and use the index to quantify the dispersion of the activation area; Process the generated prototype activation map using the class-aware activation refinement mechanism to reduce noise and make the activation area of ​​the prototype more concentrated in the image area related to the target category; Specifically, step 5 mainly includes the following steps:

[0069] Step 5-1: Calculate the Gini coefficient to measure the dispersion of pixel activation values ​​in the prototype activation map, which is defined as follows:

[0070]

[0071] Among them, S i and S j represents the absolute value of the i-th and j-th dimensions in the prototype activation map, n represents the size of the activation map, Represents the average value of the activation map; the value range of the Gini coefficient G is [0,1]. When G=0, it means that the pixel activation values ​​of all dimensions are exactly the same. When G=1, it means that only one dimension has the strongest pixel activation value, and the pixels of other dimensions are not activated.

[0072] Step 5-2: Convert the Gini coefficient G into a diffusion index (Diffusion Index) using the formula DI = (1-G) × 100. The range of the diffusion index is [0,100]. A larger DI value indicates that the distribution of pixel activation values ​​in the activation map is more concentrated in a few dimensions, representing the dispersion of attention. Use the calculated diffusion index to evaluate the effectiveness of prototype activation in the model. By controlling the DI value, the attention allocation of the model can be optimized, reducing the negative impact of attention dispersion on model performance, thereby improving the interpretability of the prototype.

[0073] Step 5-3: Calculate the attention weight A of the multi-head self-attention module in the lth Transformer block l :

[0074]

[0075] Among them, Q l and K l is the key and value representation projected from the input of the lth Transformer block, with dimension D and number of attention heads H; in order to integrate the semantic context captured by all heads, the average of the attention vectors of all heads is calculated, and we get To obtain a single attention map, these attention vectors are aggregated over the Transformer block as

[0076]

[0077] , and reshape it into This matrix captures the affinity of spatial features.

[0078] Step 5-4: Due to the multifaceted nature of the neurons in the visual Transformer, the prototypes usually respond to widely distributed image patches, resulting in noise in the activation map; for this reason, class-aware activation refinement is used to transform the attention map obtained in step 3-6 Multiply the original activation map S element by element to generate a refined activation map S ′ , so that the prototype's responsive area is more focused:

[0079]

[0080] Step 5-5: Reduce the similarity score of each prototype to a compact maximum through global maximum pooling The details are as follows:

[0081]

[0082] It is worth noting that the global maximum pooling is not applied in the non-local branch because its size is already 1; if t is the same as the prototype p m The image patch with the highest similarity, function It decreases monotonically with distance; therefore, the maximum value indicates that there is an image patch that is projected near the mth prototype in the prototype space.

[0083] Step 6: Build a prototype-based linear classifier: Calculate the similarity between the feature representation of the test image and the prototype. The classifier performs a weighted sum based on the similarity scores of prototypes of different categories and outputs the predicted category of the image. Specifically, step 6 mainly includes:

[0084] Using the form of a sparse scoring table, the similarity scores calculated in step 4 are used as the input, and the final classification probability z is obtained according to the corresponding weights; for prototypes of the same category, the weights of the initialized linear classifier are To ensure that these prototypes make a positive contribution to the classification results of this category; for prototypes of other categories, the initial weights are So that these prototypes do not contribute to the score of this category; for models with dual prototype branches (such as learning non-local and local prototypes at the same time), the non-local probability z is calculated separately n and the local probability z l , where z n Usually calculated based on CLS token, z l Based on the visual token calculation, the non-local probability z n and the local probability z l Integration is performed to generate the final output z by weighted sum c :

[0085] z c =λ l z l +λ n z n

[0086] Among them, λ n and λ l is the weight coefficient of the two branches, which is used to balance the impact of non-local and local branches on the final classification result. For the single-branch prototype layer, the local probability z based on the visual mark is directly l As the final classification probability z c .

[0087] Step 7: Prototype visualization: During the test phase, the prototype is mapped to the image space by matching the feature activation map with the closest training image, generating a visual representation of the prototype. The visualization of the prototype is combined with the weights of the linear classifier built in step 6 to explain the decision-making process of the model, showing how the model classifies by identifying key feature areas, making the decision-making process of the model more transparent. The whole process is as follows: Figure 5 As shown, specifically, step 7 mainly includes the following steps:

[0088] For a given prototype p m and the image to be recognized x, the prototype is visualized in the original pixel space by selecting the long-range dependent feature representation t of the image patch that is most similar to the category c to which the prototype belongs. The projection process can be expressed as:

[0089]

[0090] The activation map S′ is upsampled to match the size of the input image to determine the exact location of the prototype and provide a basis for subsequent prototype visualization; by identifying and calibrating the smallest rectangular area in the image x that contains activation values ​​above the 95th percentile, the rectangular area represents the most representative prototype activation pattern in the image; by visualizing each prototype, it is interpreted as a certain activation pattern in the latent feature space and corresponds to a specific image block in the original pixel space; this process enhances the interpretability of the model and makes the specific meaning of the prototype more visually intuitive.

[0091] The present invention designs a self-interpreting prototype partial network model with visual Transformer as the backbone network, which not only significantly improves the interpretability and performance of the model, but also ensures its wide applicability and stability in different application scenarios. The method mainly includes six steps: data preparation and preprocessing, feature extraction based on Transformer encoder, construction of prototype branch, construction of prototype-based linear classifier, class-aware activation refinement, prototype visualization and model interpretation. Data preparation and preprocessing is to normalize the original image and divide it into multiple flattened one-dimensional vector patches; feature extraction based on Transformer encoder is to extract the long-range dependent feature representation of the image through linear projection and Transformer encoder; construction of prototype branch is to design non-local and local prototype branches, and optimize the prototype vector to represent the feature representation of each category; construction of prototype-based linear classifier is to output the predicted category of the image according to the similarity score between the feature representation and the prototype; class-aware activation refinement is to calculate the diffusion index and refine the prototype activation map to reduce noise and focus on the relevant image area; prototype visualization and model interpretation is to explain the decision process of the model by visualizing the prototype and combining the classifier weight.

[0092] Example:

[0093] The present invention is further illustrated below by taking the bird data fine-grained classification task as an example. The present invention aims to learn a set of interpretable basic prototypes to establish the connection between the feature representation from the visual Transformer encoder and the 200 output categories. These prototypes construct a transparent embedding space for characterizing training samples. The present invention generates output predictions by comparing the representation of unseen test images with a set of prototypes, which not only provides the classification probability but also provides the prototypes on which the prediction depends, thereby improving the interpretability of the model. Figure 1 As shown, the interpretable image recognition method based on visual Transformer and prototype learning constructed by the present invention is mainly divided into seven steps:

[0094] Step 1, data preparation and preprocessing:

[0095] In order to facilitate the model to process the image, the original image is scaled here, and the resolution of all images is adjusted to 224×224, so the size of the batch input image is B×3×224×224, where B is the batch size. The input image is processed through the image patch embedding layer, which can be divided into the following two cases:

[0096] For DeiT and CaiT encoders, the following steps are involved:

[0097] S1: Divide the image X into multiple non-overlapping image patches with a side length of 16, and flatten each image patch into a one-dimensional vector. The resulting sequence of one-dimensional vectors is X = {x1, x2, ..., x 196}.

[0098] S2: Then linearly map it to a higher dimensional feature space to represent the local features of the image and generate an embedding vector z0 = [x1ε; x2ε; ...; x 196 ε}, ε is the projection matrix.

[0099] At the front end of the input sequence, a special classification tag is added, which is a learnable vector, which is ultimately used to represent the global information of the entire image. The resulting sequence is X embed =[cls;x1ε;x2ε;…;x 196 ε]. The classification tokens are expanded to match the batch size of the input images. Specifically, the shape of the CLS token is 1×1×E, where E is the feature dimension, which is expanded to a shape of B×1×E to accommodate batch processing.

[0100] S3: Add position embeddings to the sequence containing classification and visual tags. Position embeddings are used to encode the information of each position in the input image to maintain the spatial structure of the image. A position drop (pos_drop) operation is applied to regularize the embedded representation to reduce the risk of overfitting.

[0101] S4: After the above steps, the embedded representation is divided into two parts: the first part is the classification mark, whose shape is B×1×E; the second part is the visual mark, whose shape is B×196×D, where N represents the number of image patches. The processed classification mark and visual mark are concatenated along dimension 1 to generate a new embedded representation with a shape of B×197×E, which is passed to the subsequent Transformer block.

[0102] For the Swin Transformer encoder, the following steps are involved:

[0103] S1: Since the Swin Transformer is different from the standard visual Transformer, it introduces a hierarchical local window attention mechanism and introduces global information through a sliding window. First, the image is divided into multiple non-overlapping image patches with a side length of 4, and each image patch is flattened into a one-dimensional vector. The image patch is further divided into local windows of size 7×7, each of which contains multiple adjacent image patches.

[0104] S2: In each local window, relative position encoding is used, and the relative positions between image blocks are encoded into a learnable two-dimensional relative position offset table. The size of this table is in is the window size. For each pair of image patches in the window, their relative position difference in the two-dimensional plane is calculated. This offset table can cover all possible relative position relationships from the upper left corner to the lower right corner in the window. The position difference of the image blocks in each window has an offset in the horizontal and vertical directions, and these offsets are used to generate relative position encoding.

[0105] S3: Swin Transformer adopts a layered structure. As the number of network layers increases, the window size remains unchanged, but the resolution of the feature map gradually decreases, while increasing the dimension of the feature. After the above steps, the shape of the embedded representation is B×49×E, and is passed to the subsequent Transformer block.

[0106] Step 2, feature extraction based on Transformer encoder:

[0107] Transformer encoders pre-trained on the ImageNet dataset are used for feature extraction, such as: deit_tiny_patch16_224, deit_small_patch16_224, cait_xxs24_224, cait_s24_224, and swin_tiny_patch4_window7_224. They have different parameter and computational amounts, and different image patch sizes, embedding dimensions, and network depths.

[0108] Each Transformer block of these visual Transformer models basically contains layer normalization, multi-head attention module and feed-forward neural network. Among them, the multi-head attention module calculates the attention distribution across the entire input sequence to capture the global relationship between different parts of the image. The feed-forward neural network further processes the features of each position after each self-attention operation. Typically, these feed-forward networks contain two linear layers and a ReLU activation function. After each self-attention module and feed-forward network, residual connections and layer normalization are used to stabilize the training process and enhance the representation ability of the model.

[0109] For DeiT and CaiT, the following steps are mainly included:

[0110] S1: For each input vector, calculate the query, key, and value vector, Q = X embed W Q , K = X embed W k , V = X embed W V , where W Q ,W K ,W V is a learnable weight matrix.

[0111] S2: The self-attention mechanism is used to calculate the attention weights of each input position with respect to other positions, capturing long-range dependencies: Multiple attention heads are calculated simultaneously, and then their results are concatenated together to enrich the expressive power of the model: MultiHead(Q,K,V) = Concat(head1,…,head H )W O , where H is the number of attention heads and W O is the output weight matrix.

[0112] S3: Add residual connections to the output of multi-head self-attention and perform layer normalization:

[0113] X embed ′=LayerNorm(X embed +MultiHead(Q,K,V));

[0114] S4: The feedforward neural network consists of two linear layers and a ReLU activation function:

[0115] FFN(X embed ′) = ReLU(X embed ′W1+b1)+b2;

[0116] Again using residual connections and layer normalization: X embed = LayerNorm(X embed ′+FFN(X embed ′));

[0117] S5: Repeat steps S1 to S4 until the end of the last Transformer block is reached. At the same time, the attention weights of the last Transformer block need to be returned for class-aware activation refinement in step 5.

[0118] Step 3, build the prototype branch:

[0119] Different prototype layers are designed for different visual Transformer encoders; for DeiT and CaiT encoders, a dual-branch prototype layer is used to learn global and local prototypes, while for Swin Transformer, a single-branch prototype layer is used. Figure 2 and Figure 3 The structural diagrams of the double-branch and single-branch models are given respectively.

[0120] Local prototype branch: learn a set of prototype vectors P n ={p1,p2,…,p 1000}, forming 200 linear subspaces, each of which corresponds to a category in the dataset. Each subspace consists of 5 learnable prototype vectors, which are dedicated to representing the characteristic pattern of a category and can extract local feature representations from the feature map generated by the Transformer encoder. Perform the following steps:

[0121] S1: construct a prototype vector on the Grassmann manifold with a size of B×1000×1×1;

[0122] S2: The local prototype vector p l and the local feature t of the spatial position (i, j) l =T l [i,j,:] is compared, first using cosine convolution, then the calculated distance is normalized and projected to obtain the distance δ(t l ,p l );

[0123] S3: Convert the distance between each position of the local feature and the prototype into similarity Get the activation map S m ,The activation map records the response degree of each area of ​​the image to the prototype. The closer it is to the prototype, the higher the activation value;

[0124] S4: Aggregate the activation map into a single value through the global maximum pooling max_pool2d;

[0125] Global prototype branch: learn a set of prototype vectors P n ={p1,p2,…,p 1000 Each prototype branch also contains 200 linear subspaces, each of which consists of 5 learnable prototype vectors that can extract global feature representations from cls_token. Perform the following steps:

[0126] S1: construct a prototype vector on the Grassmann manifold with a size of B×1000×1×1;

[0127] S2: The global prototype vector p n With the global feature t n =T n [1,1,:] to compare and calculate the distance δ(t n ,p n ), and converted into similarity;

[0128] S3: Since the sequence length of CLS token is 1, this step omits the step of using global maximum pooling max_pool2d to integrate the similarity into a single value;

[0129] In order to learn a meaningful prototype space, the present invention designs orthogonality loss, clustering loss and separation loss in addition to the basic cross entropy loss to optimize the prototype vector.

[0130] S1: Cross entropy loss ensures that the model can correctly classify and minimizes the cross entropy of the probability that the input sample belongs to the prototype of its true category. The calculation formula is

[0131] S2: Orthogonality loss is used to ensure that the redundancy between prototypes in the same category is minimized, making the prototypes representative and discriminative. The calculation formula is in, is the Frobenius regularization term, I M Represents the identity matrix of size M×M.

[0132] S3: Clustering loss encourages prototypes of the same category to be as close as possible to the image features of the corresponding category. The calculation formula is: in is the number of images in the dataset.

[0133] S4: Separation loss ensures that prototypes of different categories keep a distance from image features of other categories. The calculation formula is

[0134] S5: Weighted sum of all loss terms to get λ1, λ2, and λ3 are coefficients of different loss terms, which are set to 0.001, 0.8, and -0.2, respectively. S1 to S5 are executed in a loop until the loss is calculated for all batches of samples.

[0135] Step 4, calculate the similarity between the feature representation and the prototype:

[0136] By step 3, the prototype layer has mapped the training data into a metric space where it can be classified based on the similarity of the feature representation to the prototype. Perform the following steps:

[0137] S1: For each input sample’s feature representation t, calculate the distance between it and all prototype vectors in each class To determine the similarity between the features and the prototypes of each category; this distance metric represents the minimum rotation required to rotate one subspace to another subspace; the main angle is used to quantify the distance between subspaces, and the calculation formula is the dot product of the prototype and each feature vector:

[0138] δ(t,p m ) = t . p m

[0140] The distance is calculated by taking the dot product of the prototype and each eigenvector and represents the minimum rotation angle from one subspace to another.

[0141] S2: The distance between the local feature and the prototype is calculated according to the following formula Converted into similarity scores to form the initial activation map D m , the activation map reflects the prototype p in the image m The spatial position of can then be upsampled to indicate the strength of the prototype presence:

[0142]

[0143] Based on similarity score Generate initial activation map S m .

[0144] Step 5, class-aware activation refinement:

[0145] Since the neurons of ViT are multifaceted, the prototype usually responds to various scattered image patches, resulting in noisy activation maps. To solve this problem, the present invention uses category-aware activation refinement to force the prototype to narrow the response area and extract category information from noise-free latent features, which mainly includes the following steps:

[0146] S1: In order to evaluate the dispersion of pixel activation values ​​in the prototype activation map, the Gini coefficient is introduced. The Gini coefficient is a common indicator for measuring the degree of distribution imbalance. In this method, it is used to measure the imbalance of pixel values ​​in the activation map. The calculation formula is:

[0147]

[0148] Among them, S i represents the absolute value of the i-th dimension in the prototype activation map, Represents the average value of the activation map. This formula measures the uniformity of the activation map by comparing the difference in activation values ​​of each pair of pixels. The value range of the Gini coefficient is [0,1]. Its physical meaning is: when G = 0, it means that the pixel activation values ​​of all dimensions are exactly the same, indicating that the model pays equal attention to all parts of the entire image, and the distribution of the activation map is very uniform; when G = 1, it means that only one dimension has the highest pixel activation value, while the pixels of all other dimensions are not activated, indicating that the model focuses on a specific area. By calculating the Gini coefficient, we can understand the concentration or dispersion of the prototype activation map, which helps model designers adjust the attention distribution of the model.

[0149] S2: In order to further quantify the model's attention dispersion, the present invention converts the Gini coefficient into a diffusion index. The diffusion index is defined as: DI = (1-G) × 100, and its value range is [0, 100]. When DI is large, it means that the activation values ​​are concentrated in a few dimensions, the attention is dispersed, and the model may be too focused on certain specific areas and ignore other areas; when DI is small, it means that the model's attention is more evenly distributed over the entire image.

[0150] S3: In the Transformer model, each Transformer block captures the long-range dependencies between different parts of the input image through the self-attention mechanism. For the lth Transformer block, the attention weight of the multi-head self-attention module is:

[0151]

[0152] Through this step, the model is able to compute dependencies between different positions in the input, thereby capturing the global contextual information of the image.

[0153] S4: The multi-head attention mechanism in the Transformer model processes different parts of the input in parallel through multiple heads. In order to obtain more complete image semantic information, it is necessary to calculate the average of the attention vectors of all heads: After integrating the multi-head attention vectors, these attention vectors are further aggregated over all Transformer blocks: Finally, reshape the matrix into Capture the spatial feature affinity between different regions of the image.

[0154] S5: Introducing class-aware activation refinement, the attention map Multiply element-wise with the original activation map S to generate a more refined activation map: Through this element-by-element multiplication operation, the model is able to filter out irrelevant activation information, making the prototype's response area more concentrated on the image area related to the target category and reducing the interference of background noise.

[0155] S6: For the resulting refined activation map, global max pooling is used to extract the maximum response value of each prototype unit: Through this operation, the redundant information in the activation map can be reduced, and only the most representative activation areas are retained. It should be noted that the global maximum pooling is not applied in the non-local branch because its size is already 1.

[0156] Step 6: Build a prototype-based linear classifier:

[0157] The similarity scores of the prototype vectors are used as input in the form of a sparse scoring table. These similarity scores are obtained by calculating the similarity between the input image features and the prototype vectors in step 3. The main steps include:

[0158] S1: Initialize the weight matrix W based on the forward reasoning principle cls , where the weights of prototypes of the same category are initialized as These weights ensure that prototypes of the same category make a positive contribution to the classification results of that category. For prototypes of other categories, the initial weights are Make sure these prototypes do not contribute to the score of the current category. In this way, the classification layer can calculate the classification score only based on the similarity with the prototypes of the same category of the input image, thereby ensuring the accuracy of the classification.

[0159] S2: For models with dual prototype branches, the classification probabilities of the non-local prototype and the local prototype are calculated separately. The probability of the non-local branch is usually calculated based on the classification mark, which is a representation of the global features of the input image. Its similarity with the non-local prototype determines the non-local classification probability z n The probability of local branching is z l Based on the visual markup calculation, it represents the similarity between the local features of the input image and the local prototype. In order to generate the final classification output, the two are weighted and summed. The weight coefficient λ is used n and λ lTo balance the impact of non-local and local information on the classification results: c =λ l z l +λ n z n , the hyperparameter is set to λ n =λ l =0.5.

[0160] Step 7, Visualization and Interpretation:

[0161] This step involves enhancing the interpretability of the model by visualizing the prototype given the training images after the model training is completed.

[0162] Given a prototype p and a training image x, the model will find the image patch t that is most similar to the prototype in the feature space. This image patch comes from an image sample that belongs to the same category as the prototype. The mapping method of the prototype can be expressed by the following formula: In order to determine the specific location of the prototype in the input image, the activation map Upsample to match the size of the original input image. The upsampled activation map can show the exact location of the prototype in the original image, helping to find the activation area of ​​the prototype during visualization. In the upsampled activation map, the activation value of each pixel is calculated, and the area with an activation value higher than the 95th percentile is selected with a rectangular box to form the smallest calibration box. This rectangular area represents the most representative prototype activation pattern in the image and is the part that the model pays most attention to.

[0163] The above process maps the prototype to the original pixel space, and displays the specific image block corresponding to the prototype through the high activation value area. The visualization of each prototype can be interpreted as a certain activation pattern in the latent feature space, which is intuitively displayed in the original pixel space through image patches. Through this prototype visualization method, the decision-making process of the model becomes more transparent and intuitive, allowing users to see how the model makes classification decisions by identifying key areas in the image.

[0164] like Figure 6As shown, the interpretable reasoning process of the present invention is shown. When a test image with a class label of "Chestnut-sided Warbler" is input, the method of the present invention can provide an interpretable reasoning process based on the learned prototypes and visualizations in the training samples, which are given in columns 2 to 4. The first prototype of the Chestnut-sided Warbler class obtains the maximum similarity score of 7.47 in the feature combination of yellow crown, black eyes and chestnut side pattern, and the second prototype obtains a similarity score of 2.93 in the yellow and black wing pattern. The original test image is decomposed into blocks, and the learned prototypes are identified as evidence for the prediction. The most active image blocks in the original test image are marked in the bounding box in the first column and highlighted in the activation map.

[0165] In a specific implementation, the present application provides a computer storage medium and a corresponding data processing unit, wherein the computer storage medium can store a computer program, and when the computer program is executed by the data processing unit, the invention content of an interpretable image recognition method based on visual Transformer and prototype learning provided by the present invention and some or all of the steps in each embodiment can be run. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0166] Those skilled in the art can clearly understand that the technical solutions in the embodiments of the present invention can be implemented by means of computer programs and their corresponding general hardware platforms. Based on such an understanding, the technical solutions in the embodiments of the present invention can be essentially or partly contributed to the prior art in the form of computer programs, i.e., software products, which can be stored in a storage medium and include several instructions for enabling a device including a data processing unit (which can be a personal computer, a server, a single-chip microcomputer, an MCU or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0167] The present invention provides an idea and method for interpretable image recognition based on visual Transformer and prototype learning. There are many methods and ways to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. An interpretable image recognition method based on visual Transformer and prototype learning, characterized in that: The following steps are involved: Step 1: perform image preprocessing, normalize the image to be identified in the training data and divide it into multiple non-overlapping image patches, and then flatten each image patch into a one-dimensional vector; Step 2, using different types of visual Transformer encoders to extract long-range dependent feature representations of the image patch, wherein the long-range dependent feature representations include non-local feature representations or local feature representations; Step 3, designing prototype branches and forming a double-branch prototype layer and a single-branch prototype layer, wherein the prototype branches include a non-local prototype branch and a local prototype branch, each prototype branch maps the training data to a metric space and defines a set of prototypes representing the non-local feature representation or the local feature representation; Step 4, in the metric space, according to the type of the visual Transformer encoder described in step 2, through a double-branch prototype layer or a single-branch prototype layer, sequentially calculating the similarity between the non-local feature representation and the local feature representation described in step 3 and the prototype; Step 5, obtaining an activation map by the similarity conversion in step 4, processing the activation map using a class-aware activation refinement method, and quantifying the dispersion of activation areas in the activation map using a diffusion index; Step 6, constructing a linear classifier, the input is the similarity between the non-local feature representation or the local feature representation and the prototype, and the output is the probability distribution of the image category; according to the probabilities, calculations are performed respectively to obtain the predicted category of the image to be recognized, that is, the image recognition result; Step 7, mapping the prototype to the original image space according to the class-aware activation refinement method described in step 5, generating a visual representation, and combining the input and weight of the linear classifier described in step 6 to explain the process of obtaining the image recognition result, thereby completing interpretable image recognition based on visual Transformer and prototype learning.

2. The interpretable image recognition method based on visual Transformer and prototype learning according to claim 1, characterized in that: The image preprocessing described in step 1 specifically includes: Step 1-1, collect a training set and normalize the images to be identified in the training set to a preset size; Step 1-2, setting the image patch size to L×L, dividing the image to be identified into R non-overlapping image patches, as follows; Where H and W represent the height and width of the image to be recognized respectively; Steps 1-3, flatten each image patch into a one-dimensional vector; Steps 1-4, map the flattened one-dimensional vector to the d-dimensional feature space through linear projection to generate the initial feature representation.

3. The interpretable image recognition method based on visual Transformer and prototype learning according to claim 2, characterized in that: The use of different types of visual Transformer encoders in step 2 to extract long-range dependency feature representations of the image patch specifically includes: Step 2-1, use the visual Transformer encoder to extract features from the initial feature representation described in step 1-4, as follows: Using the visual Transformer encoder to perform self-attention calculation on the initial feature representation, capture long-range dependencies, and obtain feature representations of image patches; The visual Transformer encoder is composed of visual Transformer blocks, each of which includes a multi-head self-attention mechanism and a feed-forward neural network; the visual Transformer block encodes the initial feature representation of the input, and embeds the encoded feature representation and passes it to the next layer of visual Transformer blocks, and after passing through all visual Transformer blocks, the feature representation of the image patch is obtained; Step 2-2, adding a single 1×1 convolutional layer after the visual Transformer encoder to obtain a long-range dependency feature representation t of the image patch, including a non-local feature representation or a local feature representation.

4. The interpretable image recognition method based on visual Transformer and prototype learning according to claim 3, characterized in that: The design prototype branches described in step 3 form a double-branch prototype layer and a single-branch prototype layer, specifically including: Step 3-1, design the prototype branch, as follows: Each local prototype branch contains C linear subspaces, where C represents the number of categories. Each subspace is dedicated to representing the characteristic pattern of a category. Each subspace consists of M learnable local prototype vectors. Composition, among which, represents the mth local prototype vector, ρ1 represents the height of the local prototype vector or the non-local prototype vector, ρ2 represents the width of the local prototype vector or the non-local prototype vector, and E represents the channel dimension of the local prototype vector; the local feature representation obtained by the visual Transformer encoder is reshaped into a feature map, and the local prototype vector P l Compared with the feature map, the local prototype vector P l The size is set to and represents the height of the feature map, Indicates the width of the feature map; Each non-local prototype branch also contains C linear subspaces, each of which consists of M learnable non-local prototype vectors Composition, among which, Represents the mth non-local prototype vector; the non-local prototype branch learns by comparing the classification label with the predefined basic prototype vector to capture and obtain the global feature representation; Step 3-2, compose the double-branch prototype layer and the single-branch prototype layer, as follows: The dual-branch prototype layer includes a local prototype branch and a non-local prototype branch; The single-branch prototype layer includes a local prototype branch; Step 3-3, determine the type of the visual Transformer encoder described in step 2, and for a visual Transformer encoder containing classification marks, choose to use a dual-branch prototype layer; for a visual Transformer encoder containing only visual marks, choose to use a single-branch prototype layer; Each prototype branch in the dual-branch prototype layer and the single-branch prototype layer maps the training data to a metric space and defines a set of prototypes that can represent the non-local feature representation or local feature representation of the training data; Step 3-4: Optimize the local prototype vector and non-local prototype vector described in step 3-1 and design the total loss function The details are as follows: Among them, λ1, λ2 and λ3 are weight coefficients of different loss terms, is the cross entropy loss, Orthogonality loss is the clustering loss, For separation loss.

5. The interpretable image recognition method based on visual Transformer and prototype learning according to claim 4, characterized in that: The cross entropy loss described in steps 3-4 The details are as follows: Assign the image to be identified to the category that is most similar to its local and non-local features, as follows: Where x represents the image to be recognized; in the local prototype branch, t represents the local feature representation of x; in the non-local prototype branch, t represents the non-local feature representation of x; δ(·,·) is the distance formula, that is, in the local prototype branch, the local feature representation of the image to be recognized is compared with the local prototype vector, and in the non-local prototype branch, the non-local feature representation of the image to be recognized is compared with the non-local prototype vector; Calculate the category y of the image to be identified in the local prototype branch and the non-local prototype branch in turn i The probability distribution of is calculated as follows: Among them, x i Represents the i-th image to be recognized. In the local prototype branch, t i Represents x i The local feature representation of t i Represents x i Non-local feature representation of Among them, the probability calculated by the local prototype branch depends on the similarity between the local feature representation of the image to be identified and the local prototype vectors of each category, and the probability calculated by the non-local prototype branch depends on the similarity between the non-local feature representation of the image to be identified and the non-local prototype vectors of each category; the cross entropy is the negative logarithm of the probability, and the calculation method is as follows: in, Represents the number of images to be recognized in the training data; Orthogonality loss The details are as follows: Use the Frobenius norm to sequentially transform the local prototype vector p l The local prototype matrix P l and by the non-local prototype vector p n The non-local prototype matrix P n To impose orthogonality constraints: in, is the Frobenius regularization term, I M represents the identity matrix of size M×M; in the local prototype branch, P (c) Represents the local prototype matrix of category c; in the non-local prototype branch, P (c) Represents the non-local prototype matrix of category c; Calculate the clustering loss of local prototype vectors and non-local prototype vectors in turn and separation loss The details are as follows: Where T represents the image patch represented by the local feature representation.

6. The interpretable image recognition method based on visual Transformer and prototype learning according to claim 5, characterized in that: The similarity between the local feature representation and the non-local feature representation and the prototype is calculated in sequence as described in step 4, specifically: Step 4-1, calculate the similarity between the local feature representation t described in step 2-2 and the local prototype vector described in step 3-1, and the calculation method is as follows: δ(t,p m )=t T p m Where T represents matrix transpose; Step 4-2: convert the similarity δ(t,p m ) is converted to a similarity score The details are as follows: Based on similarity score Generate initial activation map S m .

7. The interpretable image recognition method based on visual Transformer and prototype learning according to claim 6, characterized in that: The activation map is processed using the class-aware activation refinement method described in step 5, and the degree of dispersion of the activation area in the activation map is quantified using the diffusion index, specifically including: Step 5-1, calculate the Gini coefficient G, which is used to measure the activation map S m The dispersion of pixel activation values ​​in is as follows: Among them, S i and S j Represents the activation map S m The absolute values ​​of the i-th and j-th dimensions in , n represents the size of the activation map, Represents the activation map S m The value range of the Gini coefficient G is [0,1]. When G=0, it means that the pixel activation values ​​of all dimensions are exactly the same. When G=1, it means that only one dimension has the strongest pixel activation value, and the pixels of other dimensions are not activated. Step 5-2, convert the Gini coefficient G into the diffusion index DI, the calculation method is as follows: DI=(1-G)×100 The range of the diffusion index DI is [0,100], and the larger the DI value is, the higher the degree of attention dispersion is; the diffusion index DI is used to optimize the attention distribution of the visual Transformer encoder; Step 5-3: Calculate the attention weight A of the multi-head self-attention module in the lth visual Transformer block of the visual Transformer encoder l , the method is as follows: Among them, Q l and K l is the query Query and key Key of the lth visual Transformer block, with dimension D and number of attention heads H; calculate the average attention vector of all attention heads And aggregate it into a matrix Here’s how: Where L represents the number of visual Transformer blocks; The matrix Reshape into a matrix Get the attention map Step 5-4, use class-aware activation refinement to convert the attention map With the activation map S m Multiply element by element to generate a refined activation map S ′ , as follows: Among them, ⊙ represents element-by-element multiplication; Step 5-5: Reduce the similarity score of each prototype to a compact maximum value through global maximum pooling The details are as follows:

8. The interpretable image recognition method based on visual Transformer and prototype learning according to claim 7, characterized in that: The construction of the linear classifier described in step 6 includes the following steps: Step 6-1, using a sparse scoring table, takes the similarity score between the prototype and the image to be identified as input; for prototypes of the same category, the weight in the linear classifier is initialized to 1; for prototypes of other categories, the weight is initialized to 0; Step 6-2: For the dual-branch prototype layer, calculate the non-local probability z based on the classification label respectively n and the local probability z based on visual markings l , where the non-local probability is used to capture the global features of the image, and the local probability is used to capture the features of the local image block; The non-local probability and the local probability are weighted and summed to generate the final classification probability z c And output, the calculation method is as follows: With c =λ l With l +λ m With n Among them, λ n and λ l is the weight coefficient; For the single-branch prototype layer, the local probability z based on the visual mark is directly l As the final classification probability z c .

9. The interpretable image recognition method based on visual Transformer and prototype learning according to claim 8, characterized in that: Step 7 describes mapping the prototype to the original image space to generate a visual representation, specifically including: For a given prototype p m And the image to be recognized x, by selecting the long-range dependent feature representation t of the image patch that is most similar to the category c to which the prototype belongs to represent the visualization of the prototype in the original pixel space, the process is expressed as: Among them, T (c) represents all image patches of category c; Upsample the activation map S′ to match the size of the input image to be recognized, thereby determining the exact position of the prototype in the image to be recognized; By identifying the smallest rectangular area in the image to be identified whose activation value is higher than a preset ratio, the most representative prototype activation area in the image to be identified is determined; the visualization process of each prototype maps its activation pattern in the latent feature space to the image block in the original pixel space, completing the explanation of the acquisition process of the image recognition result.

10. The interpretable image recognition method based on visual Transformer and prototype learning according to claim 9, characterized in that: The visual Transformer encoder described in step 2-2 is a DeiT network model, a CaiT network model, or a Swin Transformer network model.

Images