Small sample image classification method using attention prompt Vision Transform

Abstract

The invention relates to a small sample image classification method by utilizing an attention prompt Vision Transform. The method comprises the following steps of S1, constructing an additional semantic tag for a data set by utilizing a deep neural interpretation method based on a Vision Transform architecture; s2, extracting an attention score logit of a top MSA module of the Vision Transform model, and converting the attention score logit into attention prediction; and S3, performing double-tag guidance on the model by using an attention tag and a semantic tag, so that the attention of the model is focused on a key entity in the image, and the representation capability of image features is improved to improve the performance of the model. According to the method, the extra attention label is constructed, the model is trained together with the semantic label, the noise from the noisy background in the features extracted by the model is reduced, the accuracy of small sample image classification is improved, and the method is of great significance to small sample image classification.

Description

Technical Field

[0001] The present invention relates to the technical field of computer image classification, and in particular to a small sample image classification method using an attention prompt Vision Transformer. Background Art

[0002] In the past decade, deep learning has made significant progress due to the availability of large amounts of labeled data. However, in many real-world scenarios, obtaining a large number of labeled samples is very challenging or costly. Therefore, few-shot learning (FSL) was introduced to address this problem, based on the human ability to learn new concepts and recognize categories from very few samples.

[0003] In recent years, many effective small sample learning methods have emerged, including small sample learning methods based on meta-learning, small sample learning methods based on transfer learning, and small sample learning methods based on data enhancement. Meta-learning, as the most popular small sample learning method, usually improves the generalization ability of the model by building a large number of small sample learning scenarios in the upstream dataset; transfer learning-based methods usually pre-train on a large upstream dataset, and then use the small sample dataset for fine-tuning in small sample scenarios to achieve the purpose of small sample image classification; since the scarcity of samples in small sample image classification is its main problem, data enhancement-based methods mainly generate new images through methods such as image generation to expand the dataset.

[0004] However, all of the above methods face a common problem, that is, when the foreground and background of the image are not clearly distinguished, the model is easily disturbed by the noisy background, and there will be a lot of noise in the extracted features, which will seriously affect the final classification performance. Moreover, since the samples in the new set are not involved in the training, this problem is particularly prominent. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a small sample image classification method using attention prompts for Vision Transformer by constructing an additional attention tag and cooperating with semantic tags to guide the Vision Transformer, so that the model's attention is focused on key entities, thereby significantly improving the representation ability of extracted features.

[0006] The technical problem to be solved by the present invention is achieved by the following technical solution. The present invention is a small sample image classification method using attention prompt Vision Transformer, the method comprising the following steps:

[0007] S1. Using deep neural interpretability methods based on the Vision Transformer architecture to build additional semantic labels for the dataset.

[0008] S2, extract the attention score logit of the top MSA module of the Vision Transformer model and convert it into attention prediction;

[0009] S3. Use attention labels and semantic labels to guide the model with dual labels, so that the model focuses on the key entities in the image, improves the representation ability of image features and improves the performance of the model.

[0010] The technical problem to be solved by the present invention can also be further achieved by the following technical solutions. For the small sample image classification method using the attention prompt Vision Transformer described above, the specific steps of step S1 are as follows:

[0011] S1.1. Initialize the pre-trained AG-CAM model. The AG-CAM model generates an attention mapping matrix of size 14*14*1 for the input image. The value of the attention mapping matrix reflects the relative importance between patches.

[0012] Given an image x of size 224*224*3, its attention mapping matrix is ​​expressed as:

[0013]

[0014] Among them A θ represents the AG-CAM model;

[0015] S1.2. AttnMatrix x Flatten and normalize it, convert it into a one-dimensional vector, and map all element values ​​to the interval [0, 1] to indicate the importance of each patch. The formula is as follows:

[0016]

[0017] Among them, Min(·) and Max(·) distributions represent the minimum and maximum operations of the vector, FA x Represents the flattened attention mapping matrix:

[0018]

[0019] Where Flatten(·) means flattening the matrix;

[0020] Due to NormAttn xAll the element values ​​of are in the interval [0, 1], which is converted into a probability distribution to represent the attention label:

[0021]

[0022] Q x (t) represents the attention label of image x.

[0023] The technical problem to be solved by the present invention can also be further achieved by the following technical solutions. For the small sample image classification method using the attention prompt Vision Transformer described above, the specific steps of step S2 are as follows:

[0024] S2.1. For Vision Transformer, given an image x, extract the top-level attention score logit:

[0025]

[0026] Among them, Q and K represent the Query and Key matrices of the MSA module, dk is the dimension of Key, h and H represent the current attention head and total number of heads of MSA respectively. The number of patches representing the slices of the model; logit x The attention score is a two-dimensional matrix, where the value of the element in the i-th row and j-th column represents the connection between the i-th patch and the j-th patch;

[0027] S2.2. Logit x Sum the second dimension to get the relative importance of each patch, that is, the distribution of the model's attention to image x:

[0028]

[0029] Among them, Sum(matrix,n) means summing the matrix matrix in the nth dimension, PredAttn x The shape of FA in formula (3) x Consistent, use the same processing method as formula (2) and formula (4) to process PredAttn x , we can get the predicted attention P x (t), P x (t) The value of each element represents the degree of attention the model pays to each patch during the feature extraction process.

[0030] The technical problem to be solved by the present invention can also be further achieved by the following technical solutions. For the small sample image classification method using the attention prompt Vision Transformer described above, the specific steps of step S3 are as follows:

[0031] S3.1. Attention label guidance, through the attention label Q x (t) Guide the model to predict the attention P x (t) Fitting to the attention label, the optimization objective is as follows:

[0032]

[0033] Where θ represents the parameters of the model, D base represents the basis set of few-shot learning, E[·] represents the error between two probability distributions;

[0034] KL divergence is used to measure the error between the two distributions, and the model parameters are updated using meta-training. The loss function is as follows:

[0035]

[0036] Among them, scaled is a hyperparameter used to scale the attention loss and classification loss to the same order of magnitude, N represents the number of classification categories, K represents the number of labeled samples for each category in the support set, M represents the number of unlabeled samples for each category in the query set, and D epi Represents all samples of an episode;

[0037] S3.2. Semantic label guidance

[0038] In the process of guiding with attention labels, we should also pay attention to the classification performance of the model to prevent overfitting due to only focusing on the attention labels. We introduce semantic classification loss and cooperate with attention loss to train the model together, and use cross entropy loss to measure classification loss:

[0039]

[0040] Where Q represents all samples in the query set, I(·) is an indicator function whose value is 1 if its argument is true, otherwise it is 0; x represents the true label of image x, n represents the category, Represents the probability that image x belongs to category n;

[0041] S3.3. Fusion loss function

[0042] The two loss functions are fused in the form of convex optimization, expressed as:

[0043] L total =αL ATTN +(1-α)L CLS (10)

[0044] The Vision Transformer is trained with the above loss function to focus on key entities and extract features with better representation capabilities.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] 1. The present invention constructs an additional attention label to cooperate with the semantic label to perform dual-label guidance on the VisionTransformer model, prompting the model to focus on the area where the key entities are located in the image, so as to enhance the representation ability of the features extracted by the model and thus improve the classification performance;

[0047] 2. In the present invention, attention labels and attention predictions exist in the form of probability distribution, which provides more delicate hints for the model. After one training on the base set, the model can have the ability to focus on key entities, and this ability can be well generalized to new sets to improve classification performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is an overall flow chart of the method of the present invention;

[0049] Figure 2 A process diagram for attention tag generation of the present invention;

[0050] Figure 3 A distribution change diagram of features in the feature space before and after attention prompting of the present invention;

[0051] Figure 4 A graph showing changes in attention of the model of the present invention to the images in the base set before and after the attention prompt;

[0052] Figure 5 This is a graph showing the attention changes of the model of the present invention on a newly focused image before and after the attention prompt. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] A small-sample image classification method using attention-cued Vision Transformer divides the dataset into two sub-datasets, a base set and a new set. The sample categories in the two sub-datasets have no intersection. When processing the N-way K-shot small-sample classification task, the samples in the new set are further subdivided into a support set and a query set. Specifically, the support set is composed of K labeled samples selected from each of the N categories in the new set, while the query set is composed of the remaining unlabeled samples in these N categories except the support set samples.

[0055] Reference Figure 1 , a small sample image classification method using attention cues Vision Transformer, including the following steps:

[0056] S1. Use a deep neural interpretability method based on the Vision Transformer architecture to build additional semantic labels for the dataset. The process is as follows: Figure 2 As shown;

[0057] S1.1. AG-CAM is an excellent deep neural network interpretable method based on the Vision Transformer architecture. It can efficiently and accurately locate the patches where the key entities in the image are located, and can give the relative importance of each patch.

[0058] S1.2. Initialize the AG-CAM model and extract the attention mapping matrix. For the input image x, its size is (224*224*3). After AG-CAM, it outputs a (14*14) attention mapping matrix. The process can be expressed as follows:

[0059]

[0060] Among them A θ Represents the AG-CAM model.

[0061] S1.3, AttnMatrix x The value of each element in represents the importance of each sliced ​​patch, that is, the attention that the model should pay. In order to better compare the relative importance of elements, the attention mapping matrix is ​​flattened and normalized, and the values ​​of its elements are mapped to the interval [0,1]. The formula is as follows:

[0062]

[0063] Among them, Min(·) and Max(·) distributions represent the minimum and maximum operations of the vector, FA x Represents the flattened attention mapping matrix:

[0064]

[0065] S1.4. In the present invention, attention labels exist in the form of probability distribution. x It has been mapped to the interval [0,1] and can be easily converted into a probability distribution to obtain the attention label in the following way:

[0066]

[0067] S2. Extract the attention score logit of the top-level MSA (multi-head self-attention mechanism) module of the Vision Transformer model and convert it into attention prediction;

[0068] S2.1. In Vision Transformer, the attention score logit of the top-level MAS module can reflect the model's attention to different areas of the image during feature extraction. For the input image x, the logit matrix of the top-level MSA module is extracted, which can be expressed as follows:

[0069]

[0070] Among them, Q and K represent the query and key matrices of the MSA module respectively, d k is the dimension of Key, h and H represent the current attention head and total number of heads of MSA respectively. The number of patches representing the slices of the model; logit x The attention score, whose value of the element in the i-th row and j-th column represents the connection between the i-th patch and the j-th patch.

[0071] S2.2. Transform the logit to convert it into an attention prediction in the same form as the attention label. Since the value of each element represents the connection between two patches, the second dimension of the logit matrix is ​​summed to convert it into a one-dimensional vector. The element value after conversion to one dimension represents the connection between the patch corresponding to the current index and all other patches. The larger the value, the more important the patch corresponding to the current index is. The converted attention distribution vector is as follows:

[0072]

[0073] Among them, Sum(matrix,n) means summing the matrix matrix in the nth dimension.

[0074] S3. Use attention labels and semantic labels to guide the model with dual labels, so that the model focuses on the key entities in the image, improves the representation ability of image features and improves the performance of the model.

[0075] S3.1. Guide the model through attention labels. In the above process, attention labels and attention predictions have been obtained. The process of attention label guidance is that the model makes the predicted attention P x The error between (t) and the true attention label is minimized, i.e.

[0076]

[0077] Where θ represents the parameters of the model, D base represents the basis set for few-shot learning, and E[·] represents the error between two probability distributions.

[0078] S3.2. In the present invention, in order to better measure the error, KL divergence is used to measure the error between the two distributions, and since the parameters of the model are updated by meta-training, the loss function is as follows:

[0079]

[0080] Among them, scaled is a hyperparameter used to scale the attention loss and classification loss to the same order of magnitude, N represents the number of classification categories, K represents the number of labeled samples for each category in the support set, M represents the number of unlabeled samples for each category in the query set, and D epi Represents all samples of an episode. Using this loss function to train the network allows the model to focus on the key entities in the image, but it may over-guide the model, that is, overfitting, so it is necessary to introduce classification loss to cooperate with the training of the model.

[0081] S3.3. Guided by semantic labels. In the above process, it is mentioned that in order to prevent the model from overfitting, classification loss needs to be combined with joint training. Semantic label guidance means extracting image features while extracting logit and performing classification.

[0082] In the present invention, a meta-classifier is used to classify features and classify samples in the meta-training query set, and the classification loss is as follows:

[0083]

[0084] Where Q represents all samples in the query set, I(·) is an indicator function whose value is 1 if its argument is true, otherwise it is 0; x represents the true label of image x, n represents the category, represents the probability that image x belongs to category n.

[0085] S3.4, fusion of two loss functions, joint training model. In the present invention, two loss functions are fused in the form of convex optimization to train the network, and the fused loss function is as follows:

[0086] L total =αL ATTN +(1-α)L CLS (20)

[0087] The network is trained multiple times by adjusting the value of α to find the optimal fusion factor; after training, the VisionTransformer model can focus on key entities, extract more representative features, and improve the performance of small sample image classification.

[0088] This application has conducted a large number of experiments on four benchmark datasets (miniImageNet, tieredImageNet, CIFAR-FS, FC100). In terms of experimental details, since the present invention is based on the Vision Transformer architecture, the backbone network selects the lightweight ViT-Small / 16. In terms of classifier selection, the DSN (Dense Score Net) meta-classifier proposed by the paper CPEA is selected as it uses both CLS token and Patch tokens for classification; in terms of model initialization, the model weights provided by FewTURE are used to initialize the model, and then the present invention is used to train on its basis. For each classification setting, 200 classification tasks are randomly selected from the base set for meta-training, the number of training rounds is 50, the learning rate of the model is set to 0.00001, and the Adam optimizer is used to optimize the network. In terms of hyperparameter settings, scaled and α are set to 10 and 0.3 respectively.

[0089] After the training is completed, 5-way 1-shot and 5-way 5-shot classification settings are constructed on the new sets of these four data sets to test the effectiveness of the present invention. 600 classification tasks are extracted from each setting, and the top-1 average accuracy and 95% confidence interval are given. The experimental results are shown in Tables 1 and 2.

[0090] Table 1: Comparison between this method and advanced experimental methods (miniImageNet and tieredImageNet)

[0091]

[0092] Table 2: Comparison between this method and advanced experimental methods (CIFAR-FS and FC100)

[0093]

[0094] The comparison results are presented in Tables 1 and 2 according to the year and the size of backbone parameters for intuitive comparison. The results of these methods are cited from their original works or related references. It can be seen from Tables 1 and 2 that the method proposed in the present invention has achieved outstanding results in various classification tasks, proving the effectiveness and advancement of the present invention.

[0095] In addition, two visualization experiments are conducted to further demonstrate the effectiveness of the present invention. Figure 3 As shown in the figure, a 5-way 1-shot classification task is randomly selected and the features before and after the attention prompt are extracted, and then mapped into a two-dimensional feature space through the TSNE technology; Figure 3 It is clearly shown that after the attention prompt, the extracted features are more clustered between the same category and more dispersed between different categories, that is, the intra-category distance is reduced and the inter-category distance is increased.

[0096] Depend on Figure 4 and Figure 5 As shown in Figure 2, some pictures are extracted from the base set and the new set respectively, and the model's attention area changes before and after the attention prompt are shown; it can be seen intuitively that after the attention prompt, Figure 4 In the example, the model shifts the focus of the base set images from the complex background to the key entities. In addition, for images with prominent features ( Figure 4 ), based on the model's ability to notice its key entities, after receiving attention prompts, the model can further focus on the key entities. Figure 5 Shows the changes in the model's attention to the pictures in the new set. Note that the categories of the pictures in the new set are not involved in the training of the model, but the figure shows that the model is still able to shift its attention to the key entities, proving the powerful generalization ability of the model after the attention prompt.

Claims

1. A small sample image classification method using attention-cued Vision Transformer, characterized by: The method comprises the following steps: S1. Using deep neural interpretability methods based on the Vision Transformer architecture to build additional semantic labels for the dataset. S2, extract the attention score logit of the top MSA module of the Vision Transformer model and convert it into attention prediction; S3. Use attention labels and semantic labels to guide the model with dual labels, so that the model focuses on the key entities in the image and improves the representation ability of image features.

2. The small sample image classification method using attention prompt Vision Transformer according to claim 1, characterized in that: The specific steps of step S1 are as follows: S1.

1. Initialize the pre-trained AG-CAM model. The AG-CAM model generates an attention mapping matrix of size 14*14*1 for the input image. The value of the attention mapping matrix reflects the relative importance between patches. Given an image x of size 224*224*3, its attention mapping matrix is ​​expressed as: Where Aθ represents the AG-CAM model; S1.

2. Flatten and normalize AttnMatrixx, convert it into a one-dimensional vector, and map all element values ​​to the interval [0,1] to indicate the importance of each patch. The formula is as follows: Among them, Min(·) and Max(·) distributions represent the minimum and maximum operations of the vector, and FAx represents the flattened attention mapping matrix: Where Flatten(·) means flattening the matrix; Due to NormAttn x All the element values ​​of are in the interval [0, 1], which is converted into a probability distribution to represent the attention label: Q x (t) represents the attention label of image x.

3. The small sample image classification method using attention prompt Vision Transformer according to claim 2 is characterized in that: The specific steps of step S2 are as follows: S2.

1. For Vision Transformer, given an image x, extract the top-level attention score logit: Among them, Q and K represent the query and key matrices of the MSA module respectively, d k is the dimension of Key, h and H represent the current attention head and total number of heads of MSA respectively. The number of patches representing the slices of the model; logit x The attention score is a two-dimensional matrix, where the value of the element in the i-th row and j-th column represents the connection between the i-th patch and the j-th patch; S2.

2. Logit x Sum the second dimension to get the relative importance of each patch, that is, the distribution of the model's attention to image x: Among them, Sum(matrix,n) means summing the matrix matrix in the nth dimension, PredAttn x The shape of FA in formula (3) x Consistent, use the same processing method as formula (2) and formula (4) to process PredAttn x , we can get the predicted attention P x (t), P x (t) The value of each element represents the degree of attention the model pays to each patch during the feature extraction process.

4. The small sample image classification method using attention prompting Vision Transformer according to claim 1, characterized in that: The specific steps of step S3 are as follows: S3.

1. Attention label guidance, through the attention label Q x (t) Guide the model to predict the attention P x (t) Fitting to the attention label, the optimization objective is as follows: Where θ represents the parameters of the model, D base represents the basis set of few-shot learning, E[·] represents the error between two probability distributions; KL divergence is used to measure the error between the two distributions, and the model parameters are updated using meta-training. The loss function is as follows: Among them, scaled is a hyperparameter used to scale the attention loss and classification loss to the same order of magnitude, N represents the number of classification categories, K represents the number of labeled samples for each category in the support set, M represents the number of unlabeled samples for each category in the query set, and D epi Represents all samples of an episode; S3.

2. Semantic label guidance In the process of guiding with attention labels, we should also pay attention to the classification performance of the model to prevent overfitting due to only focusing on the attention labels. We introduce semantic classification loss and cooperate with attention loss to train the model together, and use cross entropy loss to measure classification loss: Where Q represents all samples in the query set, I(·) is an indicator function whose value is 1 if its argument is true, otherwise it is 0; x represents the true label of image x, n represents the category, Represents the probability that image x belongs to category n; S3.

3. Fusion loss function The two loss functions are fused in the form of convex optimization, expressed as: L total =αL ATTN +(1-α)L CLS (10) The Vision Transformer is trained with the above loss function to focus on key entities and extract features with better representation capabilities.

Images