A few-sample classification method based on decoupled visual cue fine-tuning

By introducing a fine-tuning method that decouples visual cues, the problem of the model being unable to focus on key regions in data-scarce scenarios is solved, enabling adaptive dynamic adjustment of image features and improving the accuracy and performance of small sample classification.

CN119180989BActive Publication Date: 2026-11-17EAST CHINA NORMAL UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411219489.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-02
Publication Date
2026-11-17
Estimated Expiration
2044-09-02

AI Technical Summary

Technical Problem

In scenarios where data is scarce, the model cannot focus on key regions related to classification when extracting image features, resulting in poor classification performance for small samples.

Method used

A fine-tuning method based on decoupled visual cues is adopted. By introducing a pair of learnable decoupled visual cues, the semantic information related to classification in the supporting samples is learned and enhanced, and the interference of redundant features is eliminated, so as to achieve adaptive dynamic adjustment of image feature representation.

Benefits of technology

It improves the accuracy and performance of small sample classification, and can make full use of support set sample information without relying on additional information, simplifying the method and improving prediction results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119180989B_ABST
    Figure CN119180989B_ABST
Patent Text Reader

Abstract

This invention provides a few-sample classification method based on decoupled visual cues and fine-tuning. Addressing the problem that models cannot focus on key classification-related regions when extracting image features in scenarios with scarce known samples, this invention fully utilizes information from known support samples. During the feature extraction stage, a pair of learnable decoupled visual cues are introduced. On one hand, these cues learn and enhance the classification-related semantic information in the support samples and pass this information as a cue to the unknown query sample. On the other hand, redundant features are extracted and removed. This achieves adaptive dynamic adjustment of image feature representation, thereby improving classification performance. This invention is simple to use and effective, and can be widely applied to real-world tasks with scarce data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of few-shot image classification technology in computer vision, and in particular to a few-shot classification method based on decoupled visual cue fine-tuning. Background Technology

[0002] With the rapid development of deep learning, models have acquired strong representational capabilities through learning from massive amounts of data. However, overfitting is still inevitable in scenarios involving small-sample problems where data is scarce and the data domain is changing. In real-world applications, such as in the medical and military fields, sufficient labeled training data is often unavailable due to privacy, security concerns, or the high cost of data annotation. Therefore, in such scenarios, few-sample learning has significant research value and application needs.

[0003] Few-shot learning aims to address the challenge of enabling models to learn rapidly in scenarios with scarce data. Currently, a common paradigm for few-shot learning tasks is as follows: First, pre-training is performed on a large-scale dataset to give the backbone network an initial ability to represent images. Then, following a meta-learning training and testing strategy, random sampling is performed on the dataset to construct batches of "N-way K-shot" meta-classification tasks that conform to the few-shot setting. In an N-classification task, only K known samples are provided for each class. In each task, classification is performed by calculating the similarity relationship between unknown samples and known classes. The model is fine-tuned on the batch meta-training task and finally validated on a meta-testing task with no duplicate classes. However, since input images often contain not only the subject used for classification but also interfering factors such as background, these factors can hinder few-shot classification performance. Summary of the Invention

[0004] The purpose of this invention is to address the problem that models cannot focus on key classification-related regions when extracting image features, and to provide a few-sample classification method based on decoupled visual cues for fine-tuning. This method fully utilizes information from known support samples, introducing a pair of learnable decoupled visual cues during the feature extraction stage. On one hand, it learns and enhances the classification-related semantic information in the support samples and passes this information as a cue to the unknown query sample; on the other hand, it extracts and removes redundant features. This achieves adaptive dynamic adjustment of image feature representation, thereby improving classification performance.

[0005] The objective of this invention is achieved as follows:

[0006] A few-sample classification method based on decoupled visual cue fine-tuning, characterized by the following steps:

[0007] Step 1: Building the dataset

[0008] Image data is collected to construct different datasets for the pre-training and few-shot fine-tuning stages. The pre-training dataset consists of a large amount of unlabeled image data. In the few-shot fine-tuning stage, a meta-learning training strategy is adopted to construct datasets on a task-by-task basis. Each task contains a support set S and a query set Q. The support set S contains N classes, and each class has K known class samples, which is denoted as "N-way K-shot". The query set Q contains unknown samples to be predicted.

[0009] Step 2: Backbone Network Pre-training

[0010] The backbone network adopts the Vision Transformer structure. The images in the pre-training dataset are first segmented into image patch sequences, and data augmentation is used to improve data diversity. Some image patches are randomly selected for masking. Through a self-supervised learning strategy, the backbone network learns to predict the masked parts based on the unmasked areas. The parameters of the backbone network are iteratively optimized in multiple rounds by combining image-level and image patch-level self-distillation losses to ensure that the backbone network learns rich visual semantics.

[0011] Step 3: Decoupling the design and application of visual cues

[0012] Based on a pre-trained backbone network, a pair of learnable decoupled visual cues are introduced: classification-related visual cues. and visual cues unrelated to classification Where N, T, and D represent the number of supported set categories, the token length of the visual cue, and the dimension of the token embedding representation, respectively; P contains N+1 tokens. Each category corresponds to N known support categories and one unknown query category. These are used to learn and enhance the semantic information corresponding to the classification, and to transfer this semantic information from the support categories to the query. The sample p is shared across all categories to extract and remove redundant features. The specific steps for adaptively and dynamically adjusting the image feature representation using this pair of visual cues are as follows:

[0013] 3-1: Each image in the task is divided and represented as a sequence of image blocks of length M. Concatenate category token C for each sequence 0 ∈R D At the same time, it splices together learnable visual cues related to classification. The formula is as follows:

[0014]

[0015] [·] represents a concatenation operation; h ) The representative input features are fed into the backbone network. The calculation formula for the attention mechanism of each layer of the backbone network is as follows:

[0016]

[0017] Wherein, the superscript l (0≤l≤11) represents the backbone network layer number, and Attn represents the original self-attention mechanism, the process of which is as follows:

[0018] Q = h 1 W = K = h 1 W K V = h 1 W A

[0019]

[0020] W = W K W A These represent different learnable weight matrices to obtain h. 1 Projections Q, K, V, d in the three feature spaces respectively L K represents the feature dimension, and Softmax represents the activation function.

[0021] 3-2: To weaken the correlation between redundant features and discriminative features, a new branch consisting of a value-value attention mechanism is introduced in the sixth layer of the backbone network. This value-value attention mechanism is represented as follows:

[0022] V = h 1 W A

[0023]

[0024] d N The feature representation dimension of V is represented; compared with the original self-attention mechanism, only the projected V replaces Q and K; and the model parameters of the new branch composed of the value-value attention mechanism are shared with the original branch.

[0025] 3-3: To eliminate interference from classification-irrelevant features during the original feature extraction process, starting from the sixth layer of the backbone network, a redundant feature extraction attention mechanism is used to replace the self-attention mechanism in the original branch. The key feature of the redundant feature extraction attention mechanism is that it only stitches together class-independent learnable visual cues for the projected Q. The formula used is as follows:

[0026]

[0027] The above formula can also be derived into the following form:

[0028]

[0029] The latter term is the same as the original self-attention output and can be directly used as the input of the next layer attention mechanism; the former term represents the redundant feature representation extracted from the image by the current layer, the redundant feature representation is averaged in the task sample dimension and accumulated layer by layer in subsequent layers; T represents the matrix transpose operation.

[0030] Step 4: Design and construction of a mechanism to support query passing

[0031] Establish an instance query mechanism to support the transfer of set knowledge to query samples during the feature extraction stage: associate each sample's category token C with its corresponding classification-related visual cue p, represented as a key-value pair, and calculate the query sample key C. W,(=) With each supporting sample key C ,(X) The similarity between the two is used to select the most similar supporting sample. The mean between the corresponding value of the most similar supporting sample and the corresponding value of the current query sample is calculated to update the classification-related visual cues for the query sample. The formula is expressed as follows:

[0032]

[0033] i and j represent the numbers of the support sample and the query sample, respectively; sim(·) represents the calculation of cosine similarity; and I represents the number of the support sample that is most relevant to the query sample.

[0034] Step 5: Calculation of Adaptive Feature Representation

[0035] The feature representation of the image is optimized using visual cue representations and category tokens from the output layer, as shown in the following formula:

[0036]

[0037] L represents the output layer number, and λ1 and λ2 represent the influence weights of the category token representation and visual cue representation on the image adaptive feature representation, respectively, both of which are greater than 0;

[0038] Step 6: Predict the category of the query set samples

[0039] 6-1: Calculate the feature representation for each support sample Representation of query sample features The similarity matrix between them Let s represent the feature representation of each image patch of the supporting sample and the feature representation of each image patch of the query sample, respectively. For each query sample, there are NK similarity matrices, and each element s in each similarity matrix... ij The calculation method is as follows:

[0040]

[0041] 6-2: Perform a non-linear transformation on the similarity matrix and sum it according to the supporting sample categories to obtain the category prediction of the query sample. The formula is expressed as follows:

[0042]

[0043] s aL,W Indicates category The similarity between each image patch of the support sample numbered k and the query sample. This indicates that the query sample belongs to a category. The probability value is used to select the category with the highest probability value as the predicted output category for the query sample.

[0044] Step 7: Model Training in the Small Sample Fine-tuning Stage

[0045] Cross-entropy loss is used as the classification loss function, and gradients are backpropagated to update visual cues and model parameters. The loss calculation is shown in the formula:

[0046]

[0047] Indicates query sample Predicted as category The probability value, Indicates the predicted category of the query sample, when When n is n, I(·) = 1, otherwise I(·) = 0. ~ This indicates the number of query samples.

[0048] Compared with existing technologies, this invention is simple in method and can fully utilize the information provided by the support set samples in small-sample tasks without relying on additional supplementary information. It only uses a pair of learnable, decoupled visual cues to help adaptively and dynamically adjust the feature representation of the image, thereby achieving more accurate prediction results. The method is simple, effective, and has good application prospects and commercial value. Attached Figure Description

[0049] Figure 1 Flowchart of this invention;

[0050] Figure 2 Dataset construction flowchart;

[0051] Figure 3 Backbone network pre-training flowchart;

[0052] Figure 4 A flowchart illustrating a few-sample method based on learnable decoupled visual cues;

[0053] Figure 5A comparative diagram of the original attention mechanism and the redundant feature extraction attention mechanism;

[0054] Figure 6 A diagram illustrating the mechanism for passing data to queries. Detailed Implementation

[0055] To facilitate understanding of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0056] Example

[0057] See Figure 1 This invention first collects and prepares data patterns suitable for different training stages; secondly, it pre-trains the backbone network parameters using only image data, enabling it to initially acquire the ability to understand and represent images; subsequently, it introduces the learnable decoupled visual cues proposed in this invention, fine-tuning them on a batch "N-way K-shot" few-shot meta-task to enable the model to acquire few-shot classification capabilities, and then tests its capabilities using a test meta-task. Specifically, it includes the following steps:

[0058] Step 1: Building the dataset

[0059] See Figure 2 Image data is collected to construct different datasets for the pre-training and few-shot fine-tuning stages. The pre-training dataset consists of a large amount of unlabeled image data. In the few-shot fine-tuning stage, a meta-learning training strategy is adopted to construct datasets on a task-by-task basis. Each task contains a support set S and a query set Q. The support set S contains N classes, and each class has K known class samples, which is denoted as "N-way K-shot". The query set Q contains unknown samples to be predicted.

[0060] Step 2: Backbone Network Pre-training

[0061] See Figure 3 The backbone network adopts the Vision Transformer structure. The images in the pre-training dataset are first segmented into image patch sequences, and data diversity is improved through data augmentation. Some image patches are randomly selected for masking. Through a self-supervised learning strategy, the backbone network learns to predict the masked parts based on the unmasked areas. The parameters of the backbone network are iteratively optimized in multiple rounds by combining image-level and image patch-level self-distillation losses to ensure that the backbone network learns rich visual semantics.

[0062] Step 3: Decoupling the design and application of visual cues

[0063] See Figure 4 Based on a pre-trained backbone network, a pair of learnable decoupled visual cues are introduced: classification-related visual cues. and visual cues unrelated to classification Where N, T, and D represent the number of supported set categories, the token length of the visual cue, and the dimension of the token embedding representation, respectively; P contains N+1 tokens. Each category corresponds to N known support categories and one unknown query category. These are used to learn and enhance the semantic information corresponding to the classification, and to transfer this semantic information from the support categories to the query. The sample p is shared across all categories to extract and remove redundant features. The specific steps for adaptively and dynamically adjusting the image feature representation using this pair of visual cues are as follows:

[0064] 3-1: Each image in the task is divided and represented as a sequence of image blocks of length M. Concatenate category token C for each sequence 0 ∈R D At the same time, it splices together learnable visual cues related to classification. The formula is as follows:

[0065]

[0066] [·] represents a concatenation operation; h ) The representative input features are fed into the backbone network. The calculation formula for the attention mechanism of each layer of the backbone network is as follows:

[0067]

[0068] Wherein, the superscript l (0≤l≤11) represents the backbone network layer number, and Attn represents the original self-attention mechanism, the process of which is as follows:

[0069] Q = h 1 W = K = h 1 W K V = h 1 W A

[0070]

[0071] W = W K W A These represent different learnable weight matrices to obtain h. 1 Projections Q, K, W, d in the three feature spaces respectively L K represents the feature dimension, and Softmax represents the activation function.

[0072] 3-2: To weaken the correlation between redundant features and discriminative features, a new branch consisting of a value-value attention mechanism is introduced in the sixth layer of the backbone network. This value-value attention mechanism is represented as follows:

[0073] V = h 1 W A

[0074]

[0075] d N The feature representation dimension of V is represented; compared with the original self-attention mechanism, only the projected V replaces Q and K; and the model parameters of the new branch composed of the value-value attention mechanism are shared with the original branch.

[0076] 3-3: To eliminate interference from features irrelevant to classification during the original feature extraction process, starting from the sixth layer of the backbone network, a redundant feature extraction attention mechanism replaces the self-attention mechanism in the original branches; see [reference needed]. Figure 5 (a) Original attention mechanism; (b) Redundant feature extraction attention mechanism. The key feature of the redundant feature extraction attention mechanism is that it only stitches together learnable visual cues that are independent of classification for the projected Q. The formula used is as follows:

[0077]

[0078] The above formula can also be derived into the following form:

[0079]

[0080] The latter term is the same as the original self-attention output and can be directly used as the input of the next layer attention mechanism; the former term represents the redundant feature representation extracted from the image by the current layer, the redundant feature representation is averaged in the task sample dimension and accumulated layer by layer in subsequent layers; T represents the matrix transpose operation.

[0081] Step 4: Design and construction of a mechanism to support query passing

[0082] See Figure 6 An instance query mechanism is established to support the transfer of set knowledge to query samples during the feature extraction stage: Each sample's category token C is associated with its corresponding classification-related visual cue p, represented as a key-value pair. The query sample key C is then calculated. W,(=) With each supporting sample key C ,(X) The similarity between the two is used to select the most similar supporting sample. The mean between the corresponding value of the most similar supporting sample and the corresponding value of the current query sample is calculated to update the classification-related visual cues for the query sample. The formula is expressed as follows:

[0083]

[0084] i and j represent the numbers of the support sample and the query sample, respectively; sim(·) represents the calculation of cosine similarity; and I represents the number of the support sample that is most relevant to the query sample.

[0085] Step 5: Calculation of Adaptive Feature Representation

[0086] The feature representation of the image is optimized using visual cue representations and category tokens from the output layer, as shown in the following formula:

[0087]

[0088] L represents the output layer number, and λ1 and λ2 represent the influence weights of the category token representation and visual cue representation on the image adaptive feature representation, respectively, both of which are greater than 0;

[0089] Step 6: Predict the category of the query set samples

[0090] 6-1: Calculate the feature representation for each support sample Representation of query sample features The similarity matrix between them Let s represent the feature representation of each image patch of the supporting sample and the feature representation of each image patch of the query sample, respectively. For each query sample, there are NK similarity matrices, and each element s in each similarity matrix... ij The calculation method is as follows:

[0091]

[0092] 6-2: Perform a non-linear transformation on the similarity matrix and sum it according to the supporting sample categories to obtain the category prediction of the query sample. The formula is expressed as follows:

[0093]

[0094] s nk,ij Indicates category The similarity between each image patch of the support sample numbered k and the query sample. This indicates that the query sample belongs to a category. The probability value is used to select the category with the highest probability value as the predicted output category for the query sample.

[0095] Step 7: Model Training in the Small Sample Fine-tuning Stage

[0096] Cross-entropy loss is used as the classification loss function, and gradients are backpropagated to update visual cues and model parameters. The loss calculation is shown in the formula:

[0097]

[0098] Indicates query sample Predicted as category The probability value, Indicates the predicted category of the query sample, when When n is n, I(·) = 1, otherwise I(·) = 0. q This indicates the number of query samples.

[0099] Implementing all or part of the processes in the embodiments of this invention is accomplished by scheduling relevant hardware through computer software programming. The computer software program can be stored as an entity in a computer-readable storage medium, which can be any physical device capable of carrying computer software programs or code, including but not limited to: USB flash drives, portable hard drives, disks, optical discs, etc. This invention can effectively solve image classification problems under conditions of scarce known samples. It can fully utilize the feature information in known samples, learn content related to classification, exclude irrelevant content, and provide assistance in predicting unknown samples.

[0100] The above is merely a further description of the present invention and is not intended to limit the present invention. All equivalent implementations of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A few-sample classification method based on decoupled visual cue fine-tuning, characterized in that, Includes the following steps: Step 1: Building the dataset Collect image data to construct different datasets for the pre-training phase and the few-sample fine-tuning phase; The pre-training stage dataset consists of a large amount of unlabeled image data; in the few-shot fine-tuning stage, a meta-learning training strategy is adopted to construct the dataset by task; each task contains a support set S and a query set Q. The support set S contains N classes, and each class has K known class samples, which is denoted as "N-way K-shot"; the query set Q contains unknown samples to be predicted. Step 2: Backbone Network Pre-training The backbone network adopts the Vision Transformer structure. The images in the pre-training dataset are first segmented into image patch sequences, and data augmentation is used to improve data diversity. Some image patches are randomly selected for masking. Through a self-supervised learning strategy, the backbone network learns to predict the masked parts based on the unmasked areas. The parameters of the backbone network are iteratively optimized in multiple rounds by combining image-level and image patch-level self-distillation losses to ensure that the backbone network learns rich visual semantics. Step 3: Decoupling the design and application of visual cues Based on a pre-trained backbone network, a pair of learnable decoupled visual cues are introduced: classification-related visual cues. and visual cues unrelated to classification Among them N, T p D and P represent the number of supported set categories, the token length of the visual cue, and the dimension of the token embedding representation, respectively; aw It contains N+1 Each category corresponds to N known support categories and 1 unknown query category. These are used to learn and enhance the semantic information corresponding to the categories, and to transfer this semantic information from the support categories to the query. ag Samples shared across all categories are used to extract and remove redundant features. The specific steps for adaptively and dynamically adjusting image feature representation using these visual cues are as follows: 3-1: Each image in the task is divided and represented as a sequence of image blocks of length M. Concatenate category token C for each sequence 0 ∈R D At the same time, it splices together learnable visual cues related to classification. The formula is as follows: [·] represents a splicing operation; h 0 represents the input feature representation, which is fed into the backbone network. The calculation formula of the attention mechanism of each layer of the backbone network is as follows: Wherein, the superscript l (0≤l≤11) represents the backbone network layer number, and Attn represents the original self-attention mechanism, the process of which is as follows: Q=h l W Q ,K=h l W K ,V=h l W V W Q W K W V These represent different learnable weight matrices to obtain h. l Projections Q, K, V, d in the three feature spaces respectively k K represents the feature dimension, and Softmax represents the activation function. 3-2: To weaken the correlation between redundant features and discriminative features, a new branch consisting of a value-value attention mechanism is introduced in the sixth layer of the backbone network. This value-value attention mechanism is represented as follows: V=h k W V d v The feature representation dimension of V is represented; compared with the original self-attention mechanism, only the projected V is used to replace Q and K; and the model parameters of the new branch composed of the value-value attention mechanism are shared with the original branch. 3-3: To eliminate interference from classification-irrelevant features during the original feature extraction process, starting from the sixth layer of the backbone network, a redundant feature extraction attention mechanism is used to replace the self-attention mechanism in the original branch. The key feature of the redundant feature extraction attention mechanism is that it only stitches together class-independent learnable visual cues for the projected Q. The formula used is as follows: The above formula can also be derived into the following form: The latter term is the same as the original self-attention output and can be directly used as the input of the next layer attention mechanism; the former term represents the redundant feature representation extracted from the image by the current layer. The redundant feature representation is averaged in the task sample dimension and accumulated layer by layer in subsequent layers. This represents the matrix transpose operation; Step 4: Design and construction of a mechanism to support query passing Establish an instance query mechanism to support the transfer of set knowledge to query samples during the feature extraction stage: associate each sample's category token C with the corresponding classification-related visual cue p. aw Represented in key-value pair form, the query sample key C is calculated. j,(Q) With each supporting sample key C i,(S) The similarity between the two is used to select the most similar supporting sample. The mean between the corresponding value of the most similar supporting sample and the corresponding value of the current query sample is calculated to update the classification-related visual cues for the query sample. The formula is expressed as follows: i and j represent the numbers of the support sample and the query sample, respectively; sim(·) represents the calculation of cosine similarity; and I represents the number of the support sample that is most relevant to the query sample. Step 5: Calculation of Adaptive Feature Representation The feature representation of the image is optimized using visual cue representations and category tokens from the output layer, as shown in the following formula: L represents the output layer number, and λ1 and λ2 represent the influence weights of the category token representation and visual cue representation on the image adaptive feature representation, respectively, both of which are greater than 0; Step 6: Predict the category of the query set samples 6-1: Calculate the feature representation for each support sample Representation of query sample features The similarity matrix between them Let s represent the feature representation of each image patch of the supporting sample and the feature representation of each image patch of the query sample, respectively. For each query sample, there are NK similarity matrices, and each element s in each similarity matrix... ij The calculation method is as follows: 6-2: Perform a non-linear transformation on the similarity matrix and sum it according to the supporting sample categories to obtain the category prediction of the query sample. The formula is expressed as follows: s nk,ij This represents the similarity between each image patch between the support sample numbered k in category n and the query sample. This represents the probability value that the query sample belongs to category n. The category with the highest probability value is selected as the predicted output category for the query sample.