Small sample classification method based on decoupling visual prompt fine tuning

Abstract

The invention provides a small sample classification method based on decoupling visual prompt fine tuning, and aims to solve the problem that a model cannot focus on a key area related to classification when extracting image features in a scene with rare known samples. A pair of learnable decoupling visual prompts is introduced in the feature extraction stage, on one hand, semantic information related to classification in a support sample is learned and enhanced, and the information serves as a prompt to be transmitted to an unknown query sample; and on the other hand, interference of redundant features is extracted and eliminated. The self-adaptive dynamic adjustment of the image feature representation is realized, so that the classification performance is improved. The method is simple to use and good in effect, and can be widely applied to actual task scenes with rare data.

Description

Technical Field

[0001] The present invention relates to the technical field of small sample image classification in computer vision, and in particular to a small sample classification method based on decoupled visual cue fine-tuning. Background Art

[0002] With the rapid development of deep learning, models have strong representation capabilities through learning from massive data. However, in scenarios where data fields change and data is scarce, overfitting problems are still inevitable. In real application scenarios, such as medical and military fields, due to factors such as privacy, security, or high data annotation costs, it is impossible to access sufficient labeled training data. Therefore, in such scenarios, small sample learning has important research value and application needs.

[0003] Small sample learning aims to solve the problem of enabling models to learn quickly in scenarios with scarce data. At present, a commonly used paradigm for small sample learning tasks is: first, pre-training is performed on a larger data set so that the backbone network can initially acquire the ability to represent images; then, following the meta-learning training and testing strategy, random sampling is performed on the data set and a batch of "N-way K-shot" meta-classification tasks that meet the small sample setting are constructed: in an N-classification task, only K known samples are provided in each category. In each task, classification is performed by calculating the similarity relationship between unknown samples and known categories. The model is fine-tuned on a batch meta-training task and finally verified on a meta-test task without repeated categories. Since the input image contains not only the subject for classification, but also interference factors such as background, it hinders the performance of small sample classification. Summary of the invention

[0004] The purpose of the present invention is to solve the problem that the model cannot focus on the key areas related to classification when extracting image features, and to provide a small sample classification method based on decoupled visual cue fine-tuning. This method makes full use of the information of known support samples and introduces a pair of learnable decoupled visual cues in the feature extraction stage. On the one hand, it learns and enhances the semantic information related to classification in the support samples, and passes this information as a cue to the unknown query samples; on the other hand, it extracts and removes the interference of redundant features. It realizes the adaptive dynamic adjustment of image feature representation, thereby improving the classification performance.

[0005] The object of the present invention is achieved in that:

[0006] A small sample classification method based on decoupled visual cue fine-tuning, characterized by comprising the following steps:

[0007] Step 1: Dataset Construction

[0008] Collect image data and build different datasets for the pre-training stage and the small sample fine-tuning stage. The pre-training stage dataset consists of a large amount of unlabeled image data. In the small sample fine-tuning stage, a meta-learning training strategy is adopted to build a dataset based on tasks. Each task contains a support set S and a query set Q. The support set S contains N categories, and each category has K known category samples, which are recorded as "N-way K-shot". The query set Q contains unknown samples to be predicted.

[0009] Step 2: Backbone network pre-training

[0010] The backbone network adopts the Vision Transformer structure. The images in the pre-training dataset are first divided into image block sequences, and data diversity is improved through data enhancement. Some image blocks are randomly selected for masking, and the backbone network learns to predict the masked parts based on the unmasked areas through a self-supervised learning strategy. The parameters of the backbone network are optimized through multiple rounds of iterations by combining image-level and image-block-level self-distillation losses to ensure that the backbone network learns rich visual semantics.

[0011] Step 3: Decouple the design and use of visual cues

[0012] Based on the pre-trained backbone network, a pair of learnable decoupled visual cues is introduced: visual cues related to classification and visual cues that are not relevant to classification Where N, T, and D represent the number of support set categories, the token length of the visual cue, and the dimension of the token embedding representation, respectively; P contains N+1 Corresponding to N known support categories and 1 unknown query category, they are used to learn and enhance the semantic information corresponding to the classification and transfer the semantic information from the support to the query; p is used for samples of all categories to extract and remove the interference of redundant features; the specific steps of using this pair of visual cues to achieve adaptive dynamic adjustment of image feature representation are as follows:

[0013] 3-1: Each image in the task is divided and represented as a sequence of image blocks of length M Splice the class token C for each sequence 0 ∈R D At the same time, splicing corresponds to learnable visual cues related to classification The formula is as follows:

[0014]

[0015] [·] represents the splicing operation; h ) Represents the input feature representation and is sent to the backbone network. The calculation formula of the attention mechanism of each layer of the backbone network is as follows:

[0016]

[0017] Among them, the superscript l (0≤l≤11) represents the number of the backbone network layer, Attn represents the original self-attention mechanism, and the process is as follows:

[0018] Q=h 1 W ＝ ,K=h 1 W K ,V=h 1 W A

[0019]

[0020] W ＝ ,W K ,W A Represent different learnable weight matrices to obtain h 1 The projections Q, K, V, d in the three feature spaces are L represents the feature representation dimension of K, and Softmax represents the activation function;

[0021] 3-2: In order to weaken the correlation between redundant features and identification features, a new branch consisting of a value-value attention mechanism is introduced in the sixth layer of the backbone network. The value-value attention mechanism is expressed as:

[0022] V=h 1 W A

[0023]

[0024] d N represents the feature representation dimension of V; compared with the original self-attention mechanism, only Q, K are replaced by projection V; and the model parameters of the new branch composed of the value-value attention mechanism are shared with the original branch;

[0025] 3-3: In order to eliminate the interference of features irrelevant to classification without interfering with the original feature extraction process, starting from the sixth layer of the backbone network, a redundant feature extraction attention mechanism is used to replace the self-attention mechanism in the original branch; the key feature of the redundant feature extraction attention mechanism is that it only splices the learnable visual cues irrelevant to classification for the projection Q The formula used is as follows:

[0026]

[0027] The above formula can also be derived into the following form:

[0028]

[0029] Among them, the latter term is the same as the original self-attention output and can be directly used as the input of the attention mechanism of the next layer; the former term represents the redundant feature representation extracted from the image by the current layer, and the redundant feature representation is averaged in the task sample dimension and accumulated layer by layer in the subsequent layers; T represents the matrix transpose operation;

[0030] Step 4: Design and build a mechanism to support query delivery

[0031] Establish an instance query mechanism to realize the transfer of support set knowledge to query samples in the feature extraction stage: associate the category token C of each sample and the corresponding visual cue p related to classification in the form of key-value pairs, and calculate the query sample key C W,(＝) With each supporting sample key C ,(X) The similarity between them is used to select the most similar support sample, and the mean between the corresponding value of the most similar support sample and the corresponding value of the current query sample is calculated to update the visual hint related to the classification of the query sample. The formula is as follows:

[0032]

[0033] i, j represent the numbers of the support sample and query sample respectively, sim(·) represents the calculation of cosine similarity, and I represents the number of the support sample most relevant to the query sample;

[0034] Step 5: Computation of adaptive feature representation

[0035] The feature representation of the image is optimized using the visual cue representation and category tokens in the output layer, as follows:

[0036]

[0037] L represents the output layer number, λ1 and λ2 represent the weights of the influence of category token representation and visual cue representation on image adaptive feature representation, both of which are greater than 0;

[0038] Step 6: Query set sample category prediction

[0039] 6-1: Calculate the feature representation of each support sample and query sample feature representation The similarity matrix between They represent the feature representation of each image block of the support sample and the feature representation of each image block of the query sample. For each query sample, there are NK similarity matrices, and each element s in each similarity matrix is ij The calculation method is as follows:

[0040]

[0041] 6-2: Perform nonlinear transformation on the similarity matrix and sum it according to the supporting sample category to obtain the category prediction of the query sample. The formula is as follows:

[0042]

[0043] s aL,W Indicates category The similarity of each image patch between the support sample numbered k and the query sample, Indicates that the query sample belongs to the category The probability value of the query sample is selected as the predicted output category of the query sample.

[0044] Step 7: Model training in small sample fine-tuning phase

[0045] The cross entropy loss is used as the classification loss function, and the backpropagation gradient is used to update the visual cues and model parameters. The loss calculation is shown in the formula:

[0046]

[0047] Represents a query sample Predicted as class The probability value of Represents the predicted category of the query sample. When I(·)=1, otherwise I(·)=0, n ～ Indicates the number of query samples.

[0048] Compared with the prior art, the present invention has a simple method and can make full use of the information provided by the support set samples in the small sample task without the help of additional supplementary information. It only uses a pair of learnable decoupled visual cues to help the adaptive dynamic adjustment of the feature representation of the image. Thus, a more accurate prediction effect is obtained. The method is simple, the use effect is good, and it has good application prospects and commercial value. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Schematic diagram of the process of the present invention;

[0050] Figure 2 Dataset construction flow chart;

[0051] Figure 3 Backbone network pre-training flow chart;

[0052] Figure 4 Schematic diagram of the process of few-shot method based on learnable disentangled visual cues;

[0053] Figure 5Schematic diagram comparing the original attention mechanism and the redundant feature extraction attention mechanism;

[0054] Figure 6 A diagram of the mechanism that supports passing queries. DETAILED DESCRIPTION

[0055] In order to facilitate the understanding of the present invention, the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0056] Example

[0057] See also Figure 1 The present invention first collects and prepares data patterns suitable for different training stages; secondly, only image data is used to pre-train the parameters of the backbone network, so that it can initially acquire the ability to understand and represent images; then the present invention introduces the learnable decoupled visual cues, and fine-tunes them on the batch "N-way K-shot" small sample meta-task, so that the model can obtain small sample classification capabilities, and uses the test meta-task to test its capabilities. Specifically, it includes the following steps:

[0058] Step 1: Dataset Construction

[0059] See also Figure 2 , collect image data, and build different data sets for the pre-training stage and the small sample fine-tuning stage; the pre-training stage data set consists of a large amount of unlabeled image data; in the small sample fine-tuning stage, a meta-learning training strategy is adopted to build a data set based on tasks; each task contains a support set S and a query set Q, the support set S contains N categories, each category has K known category samples, which are recorded as "N-way K-shot"; the query set Q contains unknown samples to be predicted;

[0060] Step 2: Backbone network pre-training

[0061] See also Figure 3 The backbone network adopts the Vision Transformer structure; the images in the pre-training dataset are first divided into image block sequences, and data diversity is improved through data enhancement; some image blocks are randomly selected for masking, and the backbone network learns to predict the masked parts based on the unmasked areas through a self-supervised learning strategy; the parameters of the backbone network are optimized through multiple rounds of iterations by combining the self-distillation loss at the image level and the image block level to ensure that the backbone network learns rich visual semantics;

[0062] Step 3: Decouple the design and use of visual cues

[0063] See also Figure 4 Based on the pre-trained backbone network, a pair of learnable decoupled visual cues are introduced: visual cues related to classification and visual cues that are not relevant to classification Where N, T, and D represent the number of support set categories, the token length of the visual cue, and the dimension of the token embedding representation, respectively; P contains N+1 Corresponding to N known support categories and 1 unknown query category, they are used to learn and enhance the semantic information corresponding to the classification and transfer the semantic information from the support to the query; p is used for samples of all categories to extract and remove the interference of redundant features; the specific steps of using this pair of visual cues to achieve adaptive dynamic adjustment of image feature representation are as follows:

[0064] 3-1: Each image in the task is divided and represented as a sequence of image blocks of length M For each sequence, concatenate the class token C 0 ∈R D At the same time, splicing corresponds to learnable visual cues related to classification The formula is as follows:

[0065]

[0066] [·] represents the splicing operation; h ) Represents the input feature representation and is sent to the backbone network. The calculation formula of the attention mechanism of each layer of the backbone network is as follows:

[0067]

[0068] Among them, the superscript l (0≤l≤11) represents the number of the backbone network layer, Attn represents the original self-attention mechanism, and the process is as follows:

[0069] Q=h 1 W ＝ ,K=h 1 W K ,V=h 1 W A

[0070]

[0071] W ＝ ,W K ,W A Represent different learnable weight matrices to obtain h 1 The projections Q, K, W, d in the three feature spaces are L represents the feature representation dimension of K, and Softmax represents the activation function;

[0072] 3-2: In order to weaken the correlation between redundant features and identification features, a new branch consisting of a value-value attention mechanism is introduced in the sixth layer of the backbone network. The value-value attention mechanism is expressed as:

[0073] V=h 1 W A

[0074]

[0075] d N represents the feature representation dimension of V; compared with the original self-attention mechanism, only Q, K are replaced by projection V; and the model parameters of the new branch composed of the value-value attention mechanism are shared with the original branch;

[0076] 3-3: In order to eliminate the interference of features irrelevant to classification without interfering with the original feature extraction process, starting from the sixth layer of the backbone network, a redundant feature extraction attention mechanism is used to replace the self-attention mechanism in the original branch; see Figure 5 ((a) Original attention mechanism; (b) Redundant feature extraction attention mechanism), the key feature of the redundant feature extraction attention mechanism is to only splice the classification-independent learnable visual cues for the projection Q The formula used is as follows:

[0077]

[0078] The above formula can also be derived into the following form:

[0079]

[0080] Among them, the latter term is the same as the original self-attention output and can be directly used as the input of the attention mechanism of the next layer; the former term represents the redundant feature representation extracted from the image by the current layer, and the redundant feature representation is averaged in the task sample dimension and accumulated layer by layer in the subsequent layers; T represents the matrix transpose operation;

[0081] Step 4: Design and build a mechanism to support query delivery

[0082] See also Figure 6 , establish an instance query mechanism to achieve the transfer of support set knowledge to query samples in the feature extraction stage: associate the category token C of each sample and the corresponding visual cue p related to classification in the form of key-value pairs, and calculate the query sample key C W,(＝) With each supporting sample key C ,(X) The similarity between them is used to select the most similar support sample, and the mean between the corresponding value of the most similar support sample and the corresponding value of the current query sample is calculated to update the visual hint related to the classification of the query sample. The formula is as follows:

[0083]

[0084] i, j represent the numbers of the support sample and query sample respectively, sim(·) represents the calculation of cosine similarity, and I represents the number of the support sample most relevant to the query sample;

[0085] Step 5: Computation of adaptive feature representation

[0086] The feature representation of the image is optimized using the visual cue representation and category tokens in the output layer, as follows:

[0087]

[0088] L represents the output layer number, λ1 and λ2 represent the weights of the influence of category token representation and visual cue representation on image adaptive feature representation, both of which are greater than 0;

[0089] Step 6: Query set sample category prediction

[0090] 6-1: Calculate the feature representation of each support sample and query sample feature representation The similarity matrix between They represent the feature representation of each image block of the support sample and the feature representation of each image block of the query sample. For each query sample, there are NK similarity matrices, and each element s in each similarity matrix is ij The calculation method is as follows:

[0091]

[0092] 6-2: Perform nonlinear transformation on the similarity matrix and sum it according to the supporting sample category to obtain the category prediction of the query sample. The formula is as follows:

[0093]

[0094] s nk,ij Indicates category The similarity of each image patch between the support sample numbered k and the query sample, Indicates that the query sample belongs to the category The probability value of the query sample is selected as the predicted output category of the query sample.

[0095] Step 7: Model training in small sample fine-tuning phase

[0096] The cross entropy loss is used as the classification loss function, and the backpropagation gradient is used to update the visual cues and model parameters. The loss calculation is shown in the formula:

[0097]

[0098] Represents a query sample Predicted as class The probability value of Represents the predicted category of the query sample. When I(·)=1, otherwise I(·)=0, n q Indicates the number of query samples.

[0099] The implementation of all or part of the processes in the embodiments of the present invention is completed by scheduling related hardware through computer software programming. The computer software program can be stored as an entity in a computer-readable storage medium, and the medium can be any physical device that can carry computer software programs or codes, including but not limited to: USB flash drives, mobile hard disks, disks, optical disks, etc. The present invention can effectively solve image classification problems under the condition of rare known samples, can make full use of the feature information in known samples, learn content related to classification, exclude irrelevant content, and provide assistance for predicting unknown samples.

[0100] The above is only a further explanation of the present invention and is not intended to limit the present invention. Any equivalent implementation of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A small sample classification method based on decoupled visual cue fine-tuning, characterized in that The following steps are involved: Step 1: Dataset Construction Collect image data and build different datasets for the pre-training phase and small sample fine-tuning phase; The dataset in the pre-training stage is composed of a large amount of unlabeled image data. In the small sample fine-tuning stage, a meta-learning training strategy is adopted to construct a dataset based on tasks. Each task contains a support set S and a query set Q. The support set S contains N categories, and each category has K known category samples, which are recorded as "N-way K-shot". The query set Q contains unknown samples to be predicted. Step 2: Backbone network pre-training The backbone network adopts the Vision Transformer structure. The images in the pre-training dataset are first divided into image block sequences, and data diversity is improved through data enhancement. Some image blocks are randomly selected for masking, and the backbone network learns to predict the masked parts based on the unmasked areas through a self-supervised learning strategy. The parameters of the backbone network are optimized through multiple rounds of iterations by combining image-level and image-block-level self-distillation losses to ensure that the backbone network learns rich visual semantics. Step 3: Decouple the design and use of visual cues Based on the pre-trained backbone network, a pair of learnable decoupled visual cues is introduced: visual cues related to classification and visual cues that are not relevant to classification Among them, N, T p and D represent the number of support set categories, the token length of the visual cue, and the dimension of the token embedding representation, respectively; P aw Contains N+1 They correspond to N known support categories and 1 unknown query category, respectively, and are used to learn and enhance the semantic information corresponding to the classification, and transfer the semantic information from the support to the query; ag It is used for all categories of samples to extract and eliminate the interference of redundant features. The specific steps of using this pair of visual cues to achieve adaptive dynamic adjustment of image feature representation are as follows: 3-1: Each image in the task is divided and represented as a sequence of image blocks of length M For each sequence, concatenate the class token C 0 ∈R D At the same time, splicing corresponds to learnable visual cues related to classification The formula is as follows: [·] represents the splicing operation; h 0 Represents the input feature representation and is sent to the backbone network. The calculation formula of the attention mechanism of each layer of the backbone network is as follows: Among them, the superscript l (0≤l≤11) represents the number of the backbone network layer, Attn represents the original self-attention mechanism, and the process is as follows: Q=h l W Q ,K=h l W K ,V=h l W V W Q ,W K ,W V Represent different learnable weight matrices to obtain h l The projections Q, K, V, d in the three feature spaces are k represents the feature representation dimension of K, and Softmax represents the activation function; 3-2: In order to weaken the correlation between redundant features and identification features, a new branch consisting of a value-value attention mechanism is introduced in the sixth layer of the backbone network. The value-value attention mechanism is expressed as: V=h k W V d v represents the feature representation dimension of V; compared with the original self-attention mechanism, only Q, K are replaced by projection V; and the model parameters of the new branch composed of the value-value attention mechanism are shared with the original branch; 3-3: In order to eliminate the interference of features irrelevant to classification without interfering with the original feature extraction process, starting from the sixth layer of the backbone network, a redundant feature extraction attention mechanism is used to replace the self-attention mechanism in the original branch; the key feature of the redundant feature extraction attention mechanism is that it only splices the learnable visual cues irrelevant to classification for the projection Q The formula used is as follows: The above formula can also be derived into the following form: Among them, the latter term is the same as the original self-attention output and can be directly used as the input of the attention mechanism of the next layer; the former term represents the redundant feature representation extracted from the image by the current layer, and the redundant feature representation is averaged in the task sample dimension and accumulated layer by layer in the subsequent layers; Represents a matrix transpose operation; Step 4: Design and build a mechanism to support query delivery Establish an instance query mechanism to realize the transfer of support set knowledge to query samples in the feature extraction stage: associate the category token C of each sample with the corresponding visual cue p related to the classification aw , expressed in the form of key-value pairs, calculate the query sample key C j,(Q) With each supporting sample key C i,(S) The similarity between them is used to select the most similar support sample, and the mean between the corresponding value of the most similar support sample and the corresponding value of the current query sample is calculated to update the visual hint related to the classification of the query sample. The formula is as follows: i, j represent the numbers of the support sample and query sample respectively, sim(·) represents the calculation of cosine similarity, and I represents the number of the support sample most relevant to the query sample; Step 5: Computation of adaptive feature representation The feature representation of the image is optimized using the visual cue representation and category tokens in the output layer, as follows: L represents the output layer number, λ1 and λ2 represent the weights of the influence of category token representation and visual cue representation on image adaptive feature representation, both of which are greater than 0; Step 6: Query set sample category prediction 6-1: Calculate the feature representation of each support sample and query sample feature representation The similarity matrix between They represent the feature representation of each image block of the support sample and the feature representation of each image block of the query sample. For each query sample, there are NK similarity matrices, and each element s in each similarity matrix is ij The calculation method is as follows: 6-2: Perform nonlinear transformation on the similarity matrix and sum it according to the supporting sample category to obtain the category prediction of the query sample. The formula is as follows: s nk,ij represents the similarity of each image patch between the support sample numbered k in category n and the query sample, It represents the probability value that the query sample belongs to category n, and the category with the largest probability value is selected as the predicted output category of the query sample.

Images