Prototype-based classification method and device based on dynamic sparsification of transformer
By combining dynamic sparsity based on Transformer with prototypes, the problem of insufficient interpretability of convolutional neural networks in image classification is solved, achieving more accurate image sub-block region localization and faster prototype fitting speed, thus improving the interpretability and accuracy of the model.
Patent Information
- Application Number
- CN202211612894.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-15
- Publication Date
- 2026-02-27
- Estimated Expiration
- 2042-12-15
AI Technical Summary
Existing prototype-based image classification methods mainly rely on convolutional neural networks, resulting in poor interpretability.
We employ a Transformer-based dynamic sparsification and prototype approach. By training the Vision Transformer for image classification and utilizing distillation techniques and dynamic sparsification capabilities, we learn multiple prototypes and perform image sub-block feature extraction and matching, thereby improving the interpretability of the model and the prototype fitting speed.
It improves the localization accuracy of image sub-block regions and the interpretability of the model, while accelerating the speed and accuracy of prototype fitting.
Smart Images
Figure CN115937634B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of image classification, and relates to a classification method and device based on dynamic sparsification of a Transformer and combining prototypes. BACKGROUND
[0002] An image classification task has always been an important task in computer vision tasks, and a prototype-based classification method is a method with an explainability advantage in image classification methods. The prototype-based classification method is derived from the human identification method of objects: the relationship between objects is established by "something" looking like "something", and then classification is performed; the classification method establishes image-level prototypes, and finds prototypes in different parts of the image to be classified for matching, so as to complete classification. Since the reasoning process of the model can be explained by the prototypes activated by the image, the prototype-based classification method has a certain artificial intelligence explainability, which makes this method have better application value, for example, in the medical field, doctors can feel the activated disease sign prototypes in the image to analyze the reasoning process of the artificial intelligence model, and then decide whether to adopt the diagnosis result given by the artificial intelligence.
[0003] Existing prototype-based classification methods are mainly based on convolutional neural networks, and have poor explainability. SUMMARY
[0004] The application provides a classification method and device based on dynamic sparsification of a Transformer and combining prototypes, which specifically adopts the following technical scheme:
[0005] A classification method based on dynamic sparsification of a Transformer and combining prototypes, comprising the following steps:
[0006] S1: performing image classification training on a Vision Transformer to obtain a first image sub-block feature extractor based on the Vision Transformer;
[0007] S2: using the first image feature extractor as a teacher model to perform image classification training on a Vision Transformer with dynamic sparsification capability by using a distillation method to obtain a second image sub-block feature extractor based on the Vision Transformer and having dynamic sparsification capability;
[0008] S3: performing feature extraction on a training image and learning to obtain a plurality of prototypes corresponding to different categories;
[0009] S4: performing prototype matching on an image to be classified to obtain a corresponding classification result.
[0010] Further, step S1 is specifically:
[0011] cutting a plurality of training images into a plurality of image sub-blocks of the same size;
[0012] inputting the cutting result into the Vision Transformer loaded with the initialization parameters to perform feature extraction of the image sub-blocks, to obtain a first vectorized feature representation sequence of each image;
[0013] modeling the first vectorized feature representation sequence by using a fully connected classifier to obtain a classification vectorized representation with a dimension of k, where k is the number of categories of the image to be predicted;
[0014] training the model to obtain a first image sub-block feature extractor based on the Vision Transformer.
[0015] Further, step S2 is specifically:
[0016] cutting a plurality of training images into a plurality of image sub-blocks of the same size;
[0017] adding a prediction module in a specified layer of the Vision Transformer, while maintaining a binary decision mask sequence globally in the model, to form a Vision Transformer with dynamic sparsification capability;
[0018] inputting the cutting result into the Vision Transformer with dynamic sparsification capability loaded with the initialization parameters to perform feature extraction of the image sub-blocks, to obtain a second vectorized feature representation sequence of each image;
[0019] modeling the second vectorized feature representation sequence by using a fully connected classifier to obtain a classification vectorized representation with a dimension of k, where k is the number of categories of the image to be predicted;
[0020] training the model in a distillation manner by taking the first image feature extractor as a teacher model to obtain a second image sub-block feature extractor based on the Vision Transformer and with dynamic sparsification capability.
[0021] Further, step S3 is specifically:
[0022] inputting the second vectorized feature representation sequence extracted in step S2 as input to construct a feature vector space, and reshaping the feature vectors by a fully connected layer to obtain the features of each sub-block of each image, and setting M prototypes for each image category, and fitting the prototypes to the feature vector space.
[0023] Further, step S4 is specifically:
[0024] cutting the image to be classified into a plurality of image sub-blocks of the same size;
[0025] inputting the cutting result into a second image sub-block feature extractor to obtain a third vectorized feature representation sequence;
[0026] matching the third vectorized feature representation sequence with the prototypes to obtain a classification result.
[0027] A classification device based on a Transformer and combining dynamic sparsification and prototypes, comprising:
[0028] a first image sub-block feature extractor obtained by training a Vision Transformer for image classification;
[0029] a second image sub-block feature extractor obtained by training a Vision Transformer with dynamic sparsification capability using a distillation method with the first image feature extractor as a teacher model;
[0030] a classification module comprising a plurality of prototypes corresponding to different categories, the plurality of prototypes corresponding to different categories being obtained by feature extraction and learning on training images, and the classification module performing prototype matching on an image to be classified to obtain a corresponding classification result.
[0031] Further, the first image sub-block feature extractor is obtained by the following method:
[0032] cutting a plurality of training images into a plurality of image sub-blocks of the same size;
[0033] inputting the cutting result into a Vision Transformer loaded with initial parameters to perform feature extraction on the image sub-blocks, and obtaining a first vectorized feature representation sequence of each image;
[0034] modeling the first vectorized feature representation sequence using a fully connected classifier to obtain a classification vectorized representation with a dimension of k, where k is the number of categories of the image to be predicted;
[0035] training the model to obtain a first image sub-block feature extractor based on the Vision Transformer.
[0036] Further, the second image sub-block feature extractor is obtained by the following method:
[0037] cutting a plurality of training images into a plurality of image sub-blocks of the same size;
[0038] A prediction module is added in a specified layer of the Vision Transformer, and a binary decision mask sequence is maintained globally in the model to form a Vision Transformer with dynamic sparsification capability;
[0039] The segmentation result is input into the Vision Transformer with dynamic sparsification capability loaded with the initialized parameters to perform feature extraction on the image molecular blocks, and a second vectorized feature representation sequence of each image is obtained.
[0040] A fully connected classifier is used to model the second vectorized feature representation sequence to obtain a classification vectorized representation with a dimension of k, wherein k is the number of categories of the image to be predicted.
[0041] The first image feature extractor is used as a teacher model to train the model in a distillation manner to obtain a second image sub-block feature extractor based on the Vision Transformer and having dynamic sparsification capability.
[0042] Further, the prototypes in the classification module are obtained by the following method:
[0043] The second vectorized feature representation sequence obtained by the foregoing extraction is input to construct a feature vector space, and the feature vectors are reshaped and mapped through a fully connected layer to obtain the features of each sub-block of each image.
[0044] Further, the specific method for obtaining the corresponding classification result by prototype matching of the image to be classified by the classification module is:
[0045] The image to be classified is segmented into multiple image sub-blocks of the same size;
[0046] The segmentation result is input into the second image sub-block feature extractor to obtain a third vectorized feature representation sequence.
[0047] The classification module matches the third vectorized feature representation sequence and the prototypes to obtain the classification result.
[0048] The Transformer-based dynamic sparsification combined with the prototype classification method and device provided by the application can better distinguish image sub-blocks and more accurately locate the image sub-block regions activated by the prototypes, thereby improving the model interpretability.
[0049] The advantages of this invention also lie in the classification method and apparatus based on Transformer dynamic sparsification combined with prototypes. The dynamic sparsification method is adopted on the basis of the Transformer model, which improves the speed and accuracy of prototype fitting by eliminating some low-value image sub-blocks. Attached Figure Description
[0050] Figure 1 This is a schematic diagram of the classification method based on Transformer dynamic sparsification combined with prototype of the present invention. Detailed Implementation
[0051] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.
[0052] like Figure 1 The diagram illustrates a classification method based on Transformer and dynamic sparsity combined with prototypes, comprising the following steps: S1: Training a Vision Transformer for image classification to obtain a first image sub-block feature extractor based on Vision Transformer. S2: Using the first image feature extractor as a teacher model, training a Vision Transformer with dynamic sparsity capability through distillation to obtain a second image sub-block feature extractor based on Vision Transformer and possessing dynamic sparsity capability. S3: Extracting features from the training images and learning multiple prototypes corresponding to different categories. S4: Performing prototype matching on the images to be classified to obtain the corresponding classification results. The following details the above steps.
[0053] For step S1: Image classification training of Vision Transformer to obtain the first image sub-block feature extractor based on Vision Transformer.
[0054] Step S1 is as follows:
[0055] Several training images are divided into H*W=N image sub-blocks of the same size.
[0056] The segmentation results are input into the Vision Transformer, which has been loaded with initialization parameters, to extract features from image blocks, resulting in the first vectorized feature representation sequence S = {V} for each image. 00 V 01 , ..., V 10 , ..., V HW}
[0057] A fully connected classifier is used to model the first vectorized feature representation sequence to obtain a k-dimensional classification vectorized representation, where k is the number of categories of the image to be predicted.
[0058] The model is trained to obtain the first image sub-block feature extractor based on Vision Transformer. Specifically, the cross-entropy loss function is used as the loss function for network training, and the loss function is defined as follows:
[0059]
[0060] x represents the k-dimensional feature vector obtained by the classifier, class represents the category label of the image, and j represents the j-th image category. The network is trained by minimizing the loss so that the network's prediction of image type is close to the true value. After training, the first image sub-block feature extractor based on Vision Transformer is obtained.
[0061] For step S2: The first image feature extractor is used as the teacher model and the image classification training of the Vision Transformer with dynamic sparsity capability is carried out by distillation to obtain the second image sub-block feature extractor based on Vision Transformer and with dynamic sparsity capability.
[0062] Step S2 is as follows:
[0063] Several training images are divided into H*W=N image sub-blocks of the same size.
[0064] By adding a prediction module to a specified layer of the Vision Transformer and maintaining a binary decision mask sequence globally within the model, a Vision Transformer with dynamic sparsity capability is constructed. Specifically, this involves adding a prediction module to a specified layer of the Vision Transformer and maintaining a binary decision mask sequence globally within the model. Where N represents the number of image sub-blocks. The input to the prediction module is the output of the previous Vision Transformer block. and the current decision mask Information is extracted from the output of the Vision Transformer using a fully connected network. Normalization is performed using the Softmax function, and the probability π of each image sub-block being deleted in this round is output. Based on π, the Gumbel-Softmax function is used to calculate the binary decision mask sequence D obtained from this layer. D is then used in conjunction with... Update the binary determination mask sequence by performing the fundamental product of the matrices. For binary determination mask sequences The specified image sub-blocks to be deleted are deleted in a way of cutting off the attention between sub-blocks. The specific method is to calculate whether there is an attention connection between sub-blocks, denoted as G. Wherein,
[0065]
[0066] The final result is represented as S' = {V' i, V' j,..., V' n}, wherein,
[0067]
[0068] Wherein, i and j respectively represent two image sub-blocks, P ij represents the attention score of i relative to j.
[0069] The segmentation result is input into the Vision Transformer with dynamic sparsification capability loaded with the initialized parameters to perform feature extraction of the image sub-blocks, obtaining the second vectorized feature representation sequence S' = {V' i, V' j,..., V' n} of each image. 00 01 10 HW
[0070] The second vectorized feature representation sequence is modeled by using a fully connected classifier to obtain a classification vectorized representation with a dimension of k, wherein k is the number of categories of the image to be predicted.
[0071] The first image feature extractor is used as a teacher model to train the model in a distillation manner to obtain a second image sub-block feature extractor based on the Vision Transformer and having dynamic sparsification capability. Specifically, the model is trained by the distillation method, and is constrained by a loss function. The loss function is divided into four parts:
[0072] The first part,
[0073]
[0074] Wherein, y represents the prediction result obtained by the fully connected classifier, and represents the ground truth, and this part of the loss function is used to constrain the prediction result of the model.
[0075] The second part,
[0076]
[0077] Wherein, t i i respectively represent the feature representation of the i-th sub-block output by the model and the teacher model, wherein, represents the value of the decision mask corresponding to the i-th sub-block in the s-th sparsification stage of the b-th sample.
[0078] the third part,
[0079]
[0080] wherein, y' represents the prediction result of the teacher model, and KL represents the KL divergence. The second and third parts of the loss function together constrain the model to minimize the difference between the prediction result and the feature representation of the model and the teacher model.
[0081] the last part,
[0082]
[0083] wherein, ρ = [ρ (1) ,..., ρ (s) ] represents the target retention rate after each sparsification stage, and this part of the loss function is used to constrain the final retention sub-block ratio to be close to the preset value.
[0084] The final loss function is,
[0085]
[0086] wherein, λ KL , λ distill and λ ratio are respective weights. The network is trained by minimizing the loss, so that the network prediction value of the image type is close to the true value. After training, a second image sub-block feature extractor based on Vision Transformer and having dynamic sparsification capability is obtained.
[0087] For step S3: feature extraction is performed on the training images, and a plurality of prototypes corresponding to different categories are learned.
[0088] Step S3 is specifically:
[0089] The second vectorized feature representation sequence S' = {V' 00 , V' 01 ,..., V' 10 ,..., V' HW} extracted in step S2 is taken as input to construct a feature vector space, and the feature vectors are reshaped and mapped through a fully connected layer to obtain the features of each sub-block of each image. For each image category, M prototypes are set, and the prototypes are fitted to the feature vector space. The number of prototypes corresponding to each image category is the same.
[0090] Specifically, first, training is performed from the sub-block features, which requires learning a meaningful latent feature space representation from the second vectorized feature representation sequence S' = {V'1, V'2,..., V'N} obtained by the feature extractor, which requires optimizing a fixed function as, 00 01 10 HW
[0091]
[0092] wherein CrsEnt is a cross-entropy loss function, mainly constraining the classification result,
[0093]
[0094] Clst represents a loss cost of clustering minimization, λ1 is an adjustable weight, and the default setting can be 1, and z represents the features of the image sub-block. Clst constrains the cost minimization of the prototype clustering, so that each training data has at least one sub-block latent representation close to the prototype of the correct class,
[0095]
[0096] Sep represents a loss cost representing the separation of different classes, and λ2 represents an adjustable weight, and the default setting can be 1. Sep constrains the cost minimization of the prototype separation, so that the latent space representation of each sub-block in the training data is far away from the prototype not belonging to its own class.
[0097] Then, the learning of the prototype fitting is performed, which requires optimizing a function as:
[0098]
[0099]
[0100] so that each prototype is fitted to the form closest to the latent representation of all sub-block features of the same class of images.
[0101] For step S4: performing prototype matching on the image to be classified to obtain the corresponding classification result.
[0102] Step S4 is specifically:
[0103] The image to be classified is divided into H*W=N image sub-blocks of the same size.
[0104] The division result is input into the second image sub-block feature extractor to obtain a third vectorized feature representation sequence.
[0105] The third vectorized feature representation sequence and the prototype are matched to obtain the classification result. In the present application, the L2 The similarity between the prototype and the feature is measured by the distance. The final classification is constrained using cross-entropy and sparse penalty, and the function is as follows:
[0106]
[0107] wherein CrsEnt represents a cross-entropy loss function for constraining the classification result, consistent with the full text. represents the (k, j) bit input in the matrix weight of the last h layer of the classifier.
[0108] The application also discloses a classification device based on a dynamic sparse combination of a prototype of a Transformer, comprising a first image sub-block feature extractor, a second image sub-block feature extractor, and a classification module.
[0109] The first image sub-block feature extractor is obtained by image classification training of a Vision Transformer. The second image sub-block feature extractor is obtained by image classification training of a Vision Transformer with dynamic sparse capability by using a distillation method with the first image feature extractor as a teacher model. The classification module comprises a plurality of prototypes corresponding to different categories, the plurality of prototypes corresponding to different categories are obtained by feature extraction and learning of training images, and the classification module performs prototype matching on the image to be classified to obtain a corresponding classification result.
[0110] Specifically, the first image sub-block feature extractor is obtained by the following method:
[0111] A plurality of training images are all cut into a plurality of image sub-blocks of the same size.
[0112] The cut result is input into the Vision Transformer loaded with the initialized parameters to perform feature extraction of the image sub-blocks, and a first vectorized feature representation sequence of each image is obtained.
[0113] A fully connected classifier is used to model the first vectorized feature representation sequence to obtain a classification vectorized representation with a dimension of k, wherein k is the number of categories of the image to be predicted.
[0114] The model is trained to obtain the first image sub-block feature extractor based on the Vision Transformer.
[0115] Specifically, the second image sub-block feature extractor is obtained by the following method:
[0116] A plurality of training images are all cut into a plurality of image sub-blocks of the same size.
[0117] A prediction module is added in a specified layer of the Vision Transformer, and a binary decision mask sequence is maintained globally in the model to form a Vision Transformer with dynamic sparsification capability.
[0118] The segmentation result is input into the Vision Transformer with dynamic sparsification capability loaded with the initialized parameters to perform feature extraction on the image molecular blocks, and a second vectorized feature representation sequence of each image is obtained.
[0119] A fully connected classifier is used to model the second vectorized feature representation sequence to obtain a classification vectorized representation with a dimension of k, where k is the number of categories of the image to be predicted.
[0120] The first image feature extractor is used as a teacher model to train the model in a distillation manner to obtain a second image sub-block feature extractor based on the Vision Transformer and having dynamic sparsification capability.
[0121] Specifically, the prototypes in the classification module are obtained by the following method:
[0122] The second vectorized feature representation sequence obtained by the foregoing extraction is input to construct a feature vector space, and the feature vectors are reshaped and mapped through a fully connected layer to obtain the features of each sub-block of each image. M prototypes are set for each image category, and the prototypes are fitted to the feature vector space.
[0123] Specifically, the specific method of obtaining the corresponding classification result by prototype matching of the image to be classified by the classification module is as follows:
[0124] The image to be classified is segmented into multiple image sub-blocks of the same size.
[0125] The segmentation result is input into the second image sub-block feature extractor to obtain a third vectorized feature representation sequence.
[0126] The classification module matches the third vectorized feature representation sequence and the prototypes to obtain the classification result.
[0127] It can be understood that the specific technical details in the classification device based on the dynamic sparsification of the Transformer combined with the prototypes are the same as the corresponding parts in the classification method based on the dynamic sparsification of the Transformer combined with the prototypes, and will not be repeated here.
[0128] The basic principles, main features and advantages of the present application are shown and described. Those skilled in the art should understand that the above embodiments do not limit the present application in any form, and any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present application.
Claims
1. A classification method based on Transformer dynamic sparsity combined with prototypes, characterized in that, Includes the following steps: S1: Train the Vision Transformer for image classification to obtain the first image sub-block feature extractor based on the Vision Transformer; S2: Using the first image feature extractor as the teacher model, the image classification training of the Vision Transformer with dynamic sparsity capability is carried out by distillation to obtain the second image sub-block feature extractor based on Vision Transformer and with dynamic sparsity capability. S3: Extract features from the training images and learn multiple prototypes corresponding to different categories; S4: Perform prototype matching on the image to be classified to obtain the corresponding classification result; Step S2 is as follows: Several training images are divided into multiple image sub-blocks of the same size; Add a prediction module to a specified layer of the Vision Transformer, and maintain a binary decision mask sequence globally in the model to form a Vision Transformer with dynamic sparsity capability. The segmentation results are input into a Vision Transformer with dynamic sparsification capability that has been loaded with initialization parameters to extract features from image blocks, resulting in a second vectorized feature representation sequence for each image. The second vectorized feature representation sequence is modeled using a fully connected classifier to obtain a k-dimensional classification vectorized representation, where k is the number of categories of the image to be predicted. The first image feature extractor is used as the teacher model and the model is trained by distillation to obtain the second image sub-block feature extractor based on Vision Transformer and with dynamic sparsity capability. Step S4 is as follows: The image to be classified is divided into multiple image sub-blocks of the same size; The segmentation result is input into the second image sub-block feature extractor to obtain the third vectorized feature representation sequence; The classification result is obtained by matching the third vectorized feature representation sequence with the prototype.
2. The classification method based on Transformer and dynamic sparsity combined with prototypes according to claim 1, characterized in that, Step S1 is as follows: Several training images are divided into multiple image sub-blocks of the same size; The segmentation results are input into the Vision Transformer with loaded initialization parameters to extract features from image blocks, resulting in the first vectorized feature representation sequence for each image. The first vectorized feature representation sequence is modeled using a fully connected classifier to obtain a k-dimensional classification vectorized representation, where k is the number of categories of the image to be predicted. The model is trained to obtain the first image sub-block feature extractor based on Vision Transformer.
3. The classification method based on Transformer and dynamic sparsity combined with prototypes according to claim 2, characterized in that, Step S3 is as follows: The second vectorized feature representation sequence extracted in step S2 is used as input to construct a feature vector space. The feature vectors are then shaped and mapped through a fully connected layer to obtain the features of each sub-block of each image. M prototypes are set for each image category, and the prototypes are fitted to the feature vector space.
4. A classification device based on Transformer-based dynamic sparsity combined with prototypes, characterized in that, Include: The first image sub-block feature extractor is obtained by training the Vision Transformer on image classification. The second image sub-block feature extractor is obtained by using the first image feature extractor as the teacher model and distilling it to train the Vision Transformer with dynamic sparsity capability for image classification. The classification module contains multiple prototypes corresponding to different categories. These prototypes are obtained by extracting features from the training images and learning them. The classification module performs prototype matching on the images to be classified to obtain the corresponding classification results. The second image sub-block feature extractor is obtained through the following method: Several training images are divided into multiple image sub-blocks of the same size; Add a prediction module to a specified layer of the Vision Transformer, and maintain a binary decision mask sequence globally in the model to form a Vision Transformer with dynamic sparsity capability. The segmentation results are input into a Vision Transformer with dynamic sparsification capability that has been loaded with initialization parameters to extract features from image blocks, resulting in a second vectorized feature representation sequence for each image. The second vectorized feature representation sequence is modeled using a fully connected classifier to obtain a k-dimensional classification vectorized representation, where k is the number of categories of the image to be predicted. The first image feature extractor is used as the teacher model and the model is trained by distillation to obtain the second image sub-block feature extractor based on Vision Transformer and with dynamic sparsity capability. The specific method for obtaining the corresponding classification result by performing prototype matching on the image to be classified through the classification module is as follows: The image to be classified is divided into multiple image sub-blocks of the same size; The segmentation result is input into the second image sub-block feature extractor to obtain the third vectorized feature representation sequence; The classification module matches the third vectorized feature representation sequence with the prototype to obtain the classification result.
5. The classification device based on Transformer-based dynamic sparsity combined with prototype as described in claim 4, characterized in that, The first image sub-block feature extractor is obtained through the following method: Several training images are divided into multiple image sub-blocks of the same size; The segmentation results are input into the Vision Transformer with loaded initialization parameters to extract features from image blocks, resulting in the first vectorized feature representation sequence for each image. The first vectorized feature representation sequence is modeled using a fully connected classifier to obtain a k-dimensional classification vectorized representation, where k is the number of categories of the image to be predicted. The model is trained to obtain the first image sub-block feature extractor based on Vision Transformer.
6. The classification device based on Transformer-based dynamic sparsity combined with prototype as described in claim 5, characterized in that, The prototype in the classification module is obtained through the following method: The second vectorized feature representation sequence extracted above is used as input to construct a feature vector space. The feature vectors are then shaped and mapped through a fully connected layer to obtain the features of each sub-block of each image. M prototypes are set for each image category, and the prototypes are fitted to the feature vector space.