A small sample image classification method based on global and local feature augmentation

Abstract

The application discloses a kind of small sample image classification methods based on global and local feature augmentation, comprising: image dataset is divided into training set, verification set and test set, test set is divided into support set and query set;All image dataset is respectively preprocessed;The image in the training set after pre-processing is respectively pre-trained, and the optimal feature extractor is obtained;The feature map of each image in the support set is extracted using the feature extractor, channel fusion is carried out, and the augmented set S1 of image samples in the support set is obtained;Local background smoothing is carried out, and the augmented set S2 of image samples in the support set is obtained;The feature map of each image in the query set is extracted using the feature extractor, and the distance between the image features in the support set and the augmented sets S1 and S2 is classified to obtain the prediction label.The application can be more accurate and efficient in general small sample image classification, fine-grained small sample image classification and multi-class image classification without introducing too many parameters.

Description

Technical Field

[0001] This invention relates to computer vision technology, and in particular to a few-sample image classification method based on global and local feature augmentation. Background Technology

[0002] Deep learning-based methods have achieved remarkable results in various image understanding tasks. However, these successes often require massive amounts of labeled training data. In image classification, training a reliable convolutional neural network often requires hundreds or even thousands of training data points per class. However, in many specific cases, data annotation is expensive, and only a limited number of labeled samples are accessible, which severely impacts the performance of deep learning models. In contrast, humans are fully capable of learning a new visual concept from one or a few examples and quickly transferring it to new data. To overcome this challenge, few-shot learning has emerged, aiming to enable reliable and rapid learning from limited data, and has attracted widespread attention from the community.

[0003] Currently, there are various advanced methods for few-sample image classification, which can be broadly classified into the following three categories:

[0004] (1) A few-sample image classification method based on metric learning;

[0005] (2) An optimized few-sample image classification method;

[0006] (3) Small sample image classification method based on fine-tuning.

[0007] Among them, few-shot classification methods based on metric learning aim to learn an embedding space of features in which data from different classes can be distinguished by a simple distance metric. Many methods have been proposed for this category. MatchingNet (matching networks) employs a novel nearest-neighbor method with embedded feature extractors and combines the advantages of parametric and non-parametric quantization methods for classification. ProtoNet (prototypical network) uses the mean of vectors in the same class as the prototype of the corresponding class and classifies test samples based on the distance to the prototypes of different classes. RelationNet (relation network) is the first to classify test samples by learning a deep metric function instead of the fixed metric function used in previous few-shot methods. Furthermore, many methods attempt to apply local feature representations to FSL instead of using global feature representations in the feature space. DN4 finds the class closest to the input image by comparing local descriptors between the image and the class, where each local descriptor corresponds to a local region of the image, and classifies based on the KNN algorithm by comparing the similarity between the test image and the local descriptors of each class. DC-IMP directly studies local activations and fuses the results to learn task-specific features. CrossTransformers find a coarse spatial correspondence between query and labeled images, and then calculates the distance between spatially corresponding features for final classification.

[0008] Optimized few-shot classification methods aim to learn good initializations, enabling the model to quickly adapt to new, unseen tasks using a series of training sets. As a representative example, Model-Agnostic Meta-Learning (MAML) follows a unimodal training paradigm and second-order gradients, learning to adapt quickly to new tasks with only a few gradient updates. It can be applied to any model, learning very good initialization parameters and converging rapidly with a small number of samples. Reptile (first-order meta-learning), like first-order MAML, uses only first-order gradient information to adjust and update parameters, repeatedly sampling a task for training, and continuously changing the initial parameters based on the weights trained on that task. ANIL (almost no inner loop) further explores the effectiveness of MAML by removing the inner loop. During training and testing, the inner loop only updates the parameters corresponding to the network head, aiming to obtain the inner loop update in the last layer, matching the performance of MAML with less computational cost. LEO (latent embedding optimization) learns a low-dimensional semantic embedding, decoupling optimization-based meta-learning techniques from the high-dimensional space of model parameters. This algorithm is more suitable for handling small sample problems. MetaOptNet (meta-learning with differentiable convex optimization) utilizes the implicit differential of the optimality condition of the convex problem of a convex linear classifier, namely the dual formula of the optimality problem of a linear support vector machine (SVM), and uses high-dimensional embeddings to improve the model's generalization ability.

[0009] Fine-tuning-based few-shot classification methods aim to use pre-training of the base class as a preprocessing step for FSL (Few-Shot Classification). Research has found that simple pre-training can help achieve few-shot learning. RFS (Reference-Free Shot Classification) is a groundbreaking example of a simple baseline model for few-shot classification. It pre-trains a feature extractor on a visible training set and then directly learns a classifier by fixing the feature extractor during the fine-tuning phase. This method outperforms all other few-shot methods of its time. Meta-transfer learning (MTL) combines the advantages of transfer learning and meta-learning algorithms. First, it pre-trains a feature extractor, giving the network good feature extraction and representation capabilities. Then, during the testing phase, it uses meta-fine-tuning to transfer the pre-trained network weights to the new task, mitigating problems such as overfitting caused by limited data. Furthermore, some work has found a correlation between self-supervised representation learning and fine-tuning-based few-shot classification methods. Both are divided into two phases. Self-supervised representation learning uses some pretext self-supervised tasks to extract image features in the first phase, and then applies these feature representations to different downstream tasks in the second phase. Therefore, BF3S (boosting few-shot visual learning with self-supervision) further improves the feature extractor's ability to generalize to new classes by employing methods such as image rotation angle prediction and image patch position prediction while training on few-shot tasks. SKD (self-supervised knowledge distillation) further introduces knowledge distillation, using distillation to constrain the embedding space, thereby further improving model performance. Summary of the Invention

[0010] Purpose of the invention: The purpose of this invention is to provide a few-sample image classification method based on global and local feature augmentation. This method follows the two-stage model in the transfer learning paradigm, pre-trains the base class using standard cross-entropy loss, and fine-tunes the new class using the trained embedding model.

[0011] Technical solution: The present invention provides a few-sample image classification method based on global and local feature augmentation, comprising the following steps:

[0012] The image dataset is divided into training, validation, and test sets, and the test set is further divided into support sets. and query set

[0013] The images obtained from the training set, validation set, and test set are preprocessed according to the required resolution.

[0014] Pre-training is performed on the image data in the preprocessed training set: First, small batches of image data are randomly extracted from the training set multiple times. Second, an enhanced copy of the image is created using a rotation transformation. Multiple feature extractors are trained using the enhanced image data. Then, an auxiliary loss based on rotation angle prediction is used to create the optimal output manifold to enhance the generalization ability of the feature extractor. An orthogonal regularization method is used to calculate the correlation between each channel in the feature map of the image data in the training set extracted by the feature extractor, constrain it with the identity matrix, and select the optimal feature extractor by evaluating the results of the feature extractor on the validation set.

[0015] The optimal feature extractor obtained through training is used to extract the support set. For each image's feature map, the importance of each channel is measured based on its channel weights. The k channels with the smallest weights are then fused with other image features to form a support set. Augmented set of image samples

[0016] The trained feature extractor is used to extract the support set. For each image feature map, clustering is used to select local descriptors of the foreground and background for each image feature. Local background smoothing is then performed using the centers of the background clusters. The smoothed image features are used as the support set. Augmented set of image samples

[0017] The query set is extracted using the trained feature extractor. The feature map of each image is then compared with the support set. He Zeng Guangji The distance between image features is used for classification to obtain predicted labels.

[0018] Furthermore, the image dataset can be a few-shot image classification dataset such as miniImageNet, tieredImageNet, or CIFAR-FS, or a fine-grained benchmark classification dataset such as CUB Birds.

[0019] Furthermore, the images in the obtained training, validation, and test sets are scaled to obtain a resolution of 84×84 for each image.

[0020] Furthermore, small batches of image data, denoted as B = {x, y}, are randomly sampled multiple times from the training set. The extracted feature maps are represented as follows: Feature map The formula for calculating the similarity between channels is:

[0021]

[0022] Among them, f i and f j It is a feature map The i-th and j-th channels in It is f j transpose, ||·|| F The norm of the matrix is ​​used to calculate the similarity matrix; thus, a C×C similarity matrix D is obtained.

[0023] The similarity matrix D is brought closer to the identity matrix I by the following formula:

[0024]

[0025] Among them, Γ os This is the loss function.

[0026] Furthermore, the trained feature extractor is used. Extract the support set under the corresponding test set Feature map of each image For each image feature, the importance of its channels is measured based on channel weights, and the k channels with the smallest weights are selected. Composition of channel feature-label pairs (f i Top ,y), where f i Top In the feature map The k channels with the smallest weights are selected from the feature maps of the image data, and then k channels are randomly selected from the feature maps of other image data. in In the feature map k randomly selected channels are merged to form an augmented set. The process is shown in the following formula:

[0027]

[0028] in, The λ represents the features after fusion, and λ∈[0.5,1.0] is the trade-off between the selected class label y′ and the original class label y, used to control the preservation of the augmented sample labels. Finally, the new samples after fusion are all used as the support set. Augmented sets corresponding to image samples Right now

[0029] Furthermore, the trained feature extractor is used. Extract the support set under the corresponding test set Feature map of each image And represent it as a set of local descriptors u i For the i-th local descriptor, a clustering algorithm is used to cluster the above set into two clusters. J i It is the i-th cluster formed by aggregation, and the cluster with the larger sum of the weights of the local descriptors within the two clusters is taken as the foreground cluster J. 1 Another one serves as the background cluster J 2 The center of each cluster is represented as Where the cluster center p of the i-th cluster i The calculation process is shown in the following formula:

[0030]

[0031] Among them, L i Indicates the i-th cluster; Let m represent the m-th response of the j-th local descriptor in the i-th cluster; finally, for local descriptors in the background cluster, the center of the cluster is used to replace them as the local background for smoothing, and the smoothing process is shown in the following equation:

[0032]

[0033] Among them, u i It is the foreground cluster J 1 In the local descriptor, p2 is the cluster center of the background cluster. It is the i-th local descriptor after background smoothing; the smoothed image features serve as the support set. Augmented sets corresponding to image samples Right now

[0034] Furthermore, the trained feature extractor is used. Extract the query set under the corresponding test set Feature map of each image x Then based on its relationship with the support set and what was obtained The image features are used to calculate the feature mean c for each category. . ,calculate The classification probability distribution is obtained by using softmax to calculate the distances between the class means and the values ​​of each class. The calculation process is shown in the following formula:

[0035]

[0036] Among them, c .′ is the feature mean of class k′, exp represents the exponential function, and d(·) is used to calculate the distance between feature vectors, commonly using Euclidean distance.

[0037] The present invention provides a few-sample image classification system based on global and local feature augmentation, comprising:

[0038] The image dataset partitioning module is used to divide the image dataset into training, validation, and test sets, and further partition the test set into support sets. and query set

[0039] The image data preprocessing module is used to scale all images in the image dataset to obtain images at the required resolution.

[0040] The pre-training module is used to train the feature extractor using preprocessed image data and select the optimal feature extractor by evaluating the results of the feature extractor on the validation set.

[0041] The testing module is used to extract the support set using a feature extractor. For each image feature map, the importance of each channel is measured based on its channel weights. Then, k channels with small weights (considered less important) are selected and fused with other image features to form a support set. Augmented set of image samples Simultaneously, for each image feature, clustering is used to select local descriptors of the foreground and background, and local background smoothing is performed using the center of the background cluster. The smoothed image features are then used as the support set. Augmented set of image samples Extracting query sets using a feature extractor The feature map of each image is then compared with the support set. He Zeng Guangji The distance between image features is used for classification to obtain predicted labels.

[0042] An apparatus of the present invention includes a memory and a processor, wherein:

[0043] Memory is used to store computer programs that can run on a processor;

[0044] A processor is configured to execute the steps of the aforementioned few-sample image classification method based on global and local feature augmentation when running the computer program.

[0045] The present invention provides a storage medium storing a computer program, which, when executed by at least one processor, implements the steps of the aforementioned few-sample image classification method based on global and local feature augmentation.

[0046] Beneficial Effects: Compared with existing technologies, this invention proposes a Semantic Orthogonalization Learning Framework (SOLF) to obtain orthogonal and diverse feature vectors. This is the first comprehensive exploration of how to learn and generate better data features using a purely semantically aware method. This invention also proposes a globally aware feature augmentation method and a locally aware feature augmentation method. The method follows the two-stage model in the transfer learning paradigm, pre-training the base class using standard cross-entropy loss and fine-tuning the new class using the trained embedding model. In the pre-training stage, this invention orthogonalsizes cross-channel semantic information to learn more diverse features. In the testing stage, this invention proposes globally and locally aware feature augmentation from both channel and spatial perspectives to further improve classification performance. Comparative experimental results on multiple benchmark datasets show that the proposed method achieves higher classification accuracy and efficiency in general few-shot classification tasks, fine-grained few-shot classification tasks, and multi-class classification tasks without introducing too many parameters. Attached Figure Description

[0047] Figure 1 This is a flowchart of the method of the present invention;

[0048] Figure 2 This is a flowchart of the pre-training and testing phases in the method of this invention;

[0049] Figure 3 This is a schematic diagram of the global perception feature enhancement module structure;

[0050] Figure 4 This is a schematic diagram of the local perception feature enhancement module structure;

[0051] Figure 5 These are the classification results under multiple class settings on miniImageNet. Detailed Implementation

[0052] To illustrate the purpose and advantages of the present invention in more detail, the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0053] Although there are many methods for few-shot classification, including metric-based few-shot learning methods, optimization-based few-shot learning methods, and pre-trained few-shot learning methods, these existing algorithms still have potential drawbacks:

[0054] (1) Feature extractors are prone to overfitting to visible base class data, sacrificing their ability to generalize to new classes;

[0055] (2) The simple data augmentation currently used, such as cropping and flipping, causes the new linear classifier to overfit to new, unseen classes due to the scarcity of training examples.

[0056] To address the first problem, this invention first introduces an additional self-supervised learning (SSL) task, namely, rotation prediction in the pre-training phase, thereby mitigating the overfitting problem on the base classes, which is considered a strong baseline. Furthermore, for well-trained convolutional neural networks (CNNs), different channels of feature maps typically respond to different semantics. However, the semantic relationships between different feature channels are usually not independent and identically distributed. While the relationships learned in the network may fit the visible base classes, they no longer fit the invisible new classes that are disjoint in class. Therefore, to further reduce the risk of overfitting, this invention proposes a Semantic Orthogonal Learning Framework (SOLF) in the pre-training phase, which learns more diverse and discriminative features by making the feature channels independent and orthogonal.

[0057] To address the second problem, this invention proposes a novel Global and Local Aware Feature Enhancement (GLFA) method from both channel and spatial perspectives. Benefiting from the creation of the Semantic Orthogonal Learning Framework (SOLF), the proposed GLFA replaces sample-independent (small-weight) feature channels within and between classes with channels randomly selected from other samples, resulting in enhanced features that contain more diverse and discriminative semantics. Furthermore, existing work has explored the detrimental effects of background information on few-shot classification tasks. Therefore, a novel Local Aware Background Smoothing method is proposed to suppress background perturbations in images for few-shot classification tasks, identifying local foreground and background regions separately in an unsupervised manner and neutrally smoothing background noise.

[0058] This invention follows the transfer learning paradigm to address the few-shot problem, considering two limitations of current few-shot methods: overfitting of the feature extractor to the base class and overfitting of the classifier to a small number of new classes. This invention proposes a Semantic Orthogonal Learning Framework (SOLF), which endows the learned feature channels with semantic diversity and orthogonality. Simultaneously, it proposes a global and local perceptual feature enhancement method from both channel and spatial perspectives. These methods effectively remove correlations between feature channels, learn more diverse and discriminative features, and enhance features at the feature level, effectively alleviating the overfitting problem during classifier training.

[0059] Therefore, the present invention proposes a few-shot image classification method based on global and local feature augmentation. The starting point is that current few-shot image classification methods still have potential drawbacks: (1) feature extractors easily overfit to visible base class data, thus sacrificing generalization ability for unseen new class images; (2) currently used simple data augmentation methods, such as cropping and flipping, cause new linear classifiers to overfit to unseen new class images due to the scarcity of training examples. Based on this, the contributions of the present invention are as follows: In the pre-training stage, a semantic orthogonal learning framework is proposed to obtain orthogonal and diverse feature vectors. This is the first comprehensive exploration of how to learn and generate better data features using a purely semantically aware method. In the testing stage, a global-and local-aware feature augmentation paradigm is proposed to improve sample diversity, thereby effectively alleviating the overfitting problem. Comparative experimental results on multiple benchmark datasets show that the algorithm proposed in this invention is competitive in general few-shot image classification tasks, fine-grained few-shot image classification tasks, and multi-class image classification tasks without introducing too many parameters.

[0060] like Figure 1 and Figure 2 As shown, a few-sample image classification method based on global and local feature augmentation according to the present invention includes the following steps:

[0061] S1. Divide the image dataset into a training set, a validation set, and a test set, and further divide the test set into a support set. and query set

[0062] This embodiment uses four types of image datasets for training, testing, and classification. Each type of image data is processed according to steps S1-S6.

[0063] Following the small sample image classification setup, the various image datasets used in the experiment were divided into training, validation, and test sets. A feature extractor was trained on the training set, evaluated on the validation set, and tested on the test set, which was further divided into a support set. and query set The performance of the final model was tested using a distance metric between the query set and the support set.

[0064] In this embodiment, the four types of image datasets required for the experiment are first divided into three small-sample image classification datasets: miniImageNet, tieredImageNet, and CIFAR-FS, and a fine-grained benchmark classification dataset, CUB Birds. Next, the datasets are further divided. For example, 64 classes out of 100 classes in the miniImageNet dataset are used as the training set, 16 classes as the validation set, and 20 classes as the test set; 351 classes out of 608 classes in the tieredImageNet dataset are used as the training set, 97 classes as the validation set, and 160 classes as the test set. The test set is further divided into a support set. and query set

[0065] S2. Scale the images in the divided image dataset so that each image has a resolution of 84×8.

[0066] S3. Pre-train the images in the training set after preprocessing in step S2: First, randomly sample small batches of image data from the training set multiple times, denoted as B = {x, y}. Second, use rotation transformation to create enhanced copies of the images. Train multiple feature extractors using the enhanced image data. Then, use the auxiliary loss of rotation angle prediction to create the optimal output manifold to enhance the generalization ability of the feature extractors. Use orthogonal regularization to calculate the correlation between channels in the feature map of the image data in the training set extracted by the feature extractors, constrain it with the identity matrix, and evaluate the results of the feature extractors on the validation set. Select the optimal feature extractor based on the best performance on the validation set.

[0067] In this embodiment, given an image X, a feature map can be extracted using a feature extractor. Therefore, it is further expressed as Simultaneously, a two-layer linear layer is used to learn the weights w for each channel. i (See Figure 3 China φ via g φ Get w i (where is a general term for the higher steps). Then, arbitrary channel pairs f can be easily computed. i and f j Cosine similarity D between i,j The calculation process is as follows:

[0068]

[0069] Among them, f i and f j It is a feature map The i-th and j-th channels in It is f j transpose, ||·|| F The norm of the matrix is ​​used to calculate the similarity matrix; thus, a C×C similarity matrix D is obtained.

[0070] The similarity matrix D is constrained to be close to the identity matrix I:

[0071]

[0072] Among them, Γ os This is the loss function.

[0073] As can be seen, the orthogonalization operation in this invention ensures that the similarity between channels and themselves (diagonal elements) tends to 1, while the similarity between channels and other channels (off-diagonal elements) tends to 0. This promotes channel diversity while ensuring channel independence. Furthermore, in the subsequent global sensing feature enhancement, channel interference will be greatly reduced due to the orthogonality between different channels.

[0074] S4. Use the optimal feature extractor obtained from step S3. Extract support set Feature map of each image For each image's feature map, the importance of each channel is measured based on its channel weights. The k least important channels are then selected and fused with other image features (global features) to form the support set. Augmented set of image samples

[0075] In this embodiment, the k channels with the smallest weights are... Composition of channel feature-label pairs (f i Top ,y), where f i Top In the feature map The k channels with the smallest weights are selected from the feature maps of the image data, and then k channels are randomly selected from the feature maps of other image data. in In the feature map k randomly selected channels are merged to form an augmented set. The process is shown in formula (3):

[0076]

[0077] Among them, f i and f j It is a feature map The i-th and j-th channels in the middle, The fusion is the corresponding feature, λ∈[0.5,1.0] is the trade-off between the selected class label y′ and the original class label y, used to control the preservation of augmented sample labels. This invention calculates f i Top and f i y′ The similarity between them is used as a measure of λ selection. If the similarity is greater than a threshold, it indicates that f i Top and f i y′ The semantic similarity between the channels is high. Therefore, a relatively small λ can be set to obtain a larger perturbation from the semantics of other channels, thus ensuring the enhanced class recognition of the samples and maximizing the simulation of diversity. Conversely, a smaller similarity should be assigned a larger λ between channel pairs to preserve the class label. The final fused new samples are all used as the support set. Augmented sets corresponding to image samples Right now

[0078] S5. Use the feature extractor trained in step S3. Extract support set Feature map of each image For each image feature, clustering is used to select local descriptors of the foreground and background. Then, local background smoothing (local features) is performed using the centers of the background clusters. The smoothed image features are used as the support set. Augmented set of image samples

[0079] In this embodiment, the feature map Represented as a set of local descriptors u i For the i-th local descriptor, the above set is clustered into two clusters using a clustering algorithm (k-means). J i It is the i-th cluster formed by aggregation, and the cluster with the larger sum of the weights of the local descriptors within the two clusters is taken as the foreground cluster J. 1 Another one serves as the background cluster J 2 The center of each cluster is represented as Where the cluster center P of the i-th cluster i The calculation process is shown in formula (4):

[0080]

[0081] Among them, L i This represents the number of local descriptors in the i-th cluster. Let represent the m-th response of the j-th local descriptor in the i-th cluster, and C be the dimension of the local descriptor in the channel direction. Finally, for local descriptors in the background cluster, the center of the cluster is used to replace them as the local background for smoothing. The smoothing process is shown in Equation (5):

[0082]

[0083] u i It is the foreground cluster J 1 In the local descriptor, P2 is the cluster center of the background cluster. It is the i-th local descriptor after background smoothing; the smoothed image features serve as the support set. Augmented sets corresponding to image samples Right now pass Figure 4 This provides a clearer understanding of the algorithm flow. Mean filtering is an effective image smoothing algorithm that simply replaces the center value of an image with the average value of all pixels in the image. Similarly, given a background cluster J... * and cluster center c * This invention will use J in the local descriptor dimension. * Each local descriptor in is set to c * .

[0084] S6. Use the feature extractor trained in step S3. Extract query set Feature map of each image x Then based on its relationship with the support set He Zeng Guangji The distance between image features is used for classification to obtain predicted labels;

[0085] In this embodiment, based on the feature map With support set The results obtained from steps S4 and S5 The image features are used to calculate the feature mean c for each category. . ,calculate The classification probability distribution is obtained by using softmax to calculate the distances between the class means and the values ​​of each class. The calculation process is shown in formula (6):

[0086]

[0087] Among them, c k′ is the feature mean of class k′, exp represents the exponential function, and d(·) is used to calculate the distance between feature vectors, commonly using Euclidean distance.

[0088] The present invention provides a few-sample image classification system based on global and local feature augmentation, comprising:

[0089] The image dataset partitioning module is used to divide the image dataset into training, validation, and test sets, and further partition the test set into support sets. and query set

[0090] The image data preprocessing module is used to scale all images in the image dataset to obtain images at the required resolution.

[0091] The pre-training module is used to train the feature extractor using preprocessed image data and select the optimal feature extractor by evaluating the results of the feature extractor on the validation set.

[0092] The testing module is used to extract the support set using a feature extractor. For each image feature map, the importance of each channel is measured based on its channel weights. Then, k channels with small weights (considered less important) are selected and fused with other image features to form a support set. Augmented set of image samples Simultaneously, for each image feature, clustering is used to select local descriptors of the foreground and background, and local background smoothing is performed using the center of the background cluster. The smoothed image features are then used as the support set. Augmented set of image samples Extracting query sets using a feature extractor The feature map of each image is then compared with the support set. He Zeng Guangji The distance between image features is used for classification to obtain predicted labels.

[0093] An apparatus of the present invention includes a memory and a processor, wherein:

[0094] Memory is used to store computer programs that can run on a processor;

[0095] The processor is configured to, when running the computer program, execute the steps of the few-sample image classification method based on global and local feature augmentation as described above, and achieve the technical effects described above.

[0096] The present invention provides a storage medium characterized in that the storage medium stores a computer program, which, when executed by at least one processor, implements the steps of the aforementioned few-sample image classification method based on global and local feature augmentation, and achieves the technical effects described above.

[0097] To evaluate the performance of the proposed method, it was compared with state-of-the-art few-shot learning models on four benchmark datasets. Since base-class-based pre-training methods have demonstrated superiority in many recent FSL studies, such as simpleshot, RFS-simple, and FEAT, most of the comparison methods in this invention, except for MetaOptNet, Boosting, ArL, and MABAS, employed base-class-based pre-training. Note that Boosting and ArL both used an additional SSL task during training. MABAS also used adversarial examples to generate samples during testing, which is closely related to the work of this invention. Furthermore, although CAN and TADAM do not use pre-training, they utilize global classification as an auxiliary training task by using global labels from base classes. S2M2 also applied SSL-assisted loss during the pre-training phase, which is closely related to the work of this invention.

[0098] Table 1 shows the results of the comparative experiments on the miniImageNet dataset, Table 2 shows the results on the tieredImageNet dataset, Table 3 shows the results on the CIFAR-FS dataset, and Table 4 shows the results on the fine-grained dataset CUB.

[0099] Table 1. Average accuracy of classification tasks on the miniImageNet dataset.

[0100]

[0101]

[0102] Table 2 shows the average accuracy of classification tasks on the Tier 2 ImageNet dataset.

[0103]

[0104] Table 3 shows the average accuracy of classification tasks on the CIFAR-FS dataset.

[0105]

[0106] Table 4 shows the average accuracy of classification tasks on the CUB dataset.

[0107]

[0108] As shown in Tables 1 to 4, the proposed strong baseline has achieved highly competitive results with the comparative methods, effectively demonstrating the potential of the transfer learning paradigm. Secondly, the proposed GLFA method can further and continuously improve performance on all datasets, outperforming the strong baseline. For example, in the 5-way 1-shot setting, GLFA outperforms the strong baseline by 1.69%, 2.33%, 0.59%, and 1.44% on the four datasets, respectively. This fully validates the effectiveness of the proposed SOLF framework and GLFA method. Finally, compared with state-of-the-art FSL methods, which do not rely on pre-trained methods such as DeepEMD, FEAT, and FRN, nor do they utilize SSL-assisted tasks such as ArL and Boosting, the proposed GLFA consistently outperforms these comparative FSL methods, achieving new state-of-the-art results on all datasets in both 5-way 1-shot and 5-way 5-shot settings.

[0109] In summary, based on the results and analysis, the following conclusions can be drawn: (1) the pre-training paradigm is indeed effective in the field of FSL; (2) the proposed global and local perceptual feature enhancement modules and semantic orthogonal learning framework (SOLF) are indeed effective because they can alleviate the overfitting problem in the meta-testing stage.

[0110] To investigate the impact of the core components of this invention—the Semantic Orthogonal Learning Framework (SOLF), Global Aware Feature Augmentation (GA Augmentation), and Local Aware Background Smoothing (LA Smoothing)—ablation experiments were conducted on miniImageNet, tieredImageNet, and CUB. The experimental results are shown in Table 5, where the results in the first row represent the strong baseline.

[0111] Table 4 Ablation Experiments of Core Components

[0112]

[0113] Experimental results show that all components can further improve classification accuracy on the three datasets compared to the strong baseline. Furthermore, both GA and SOLF alone show significant improvements over the baseline. Combining LA smoothing with GA enhancement also significantly improves performance. These results successfully demonstrate that the feature decorrelation (orthogonality) operation, SOLF, is beneficial for classification. The results indicate that genetic algorithm enhancement effectively alleviates the overfitting problem during the tuning phase, achieving better classification performance. In addition, background smoothing of the spatial region, i.e., LA smoothing, can also effectively reduce background noise interference, further improving classification performance.

[0114] To further verify the superiority of the proposed method, this invention compares it with other closely related pre-trained methods (such as RFS and MTL) on large-scale single tasks. Furthermore, ProtoNet and R2D2 were also reimplemented and pre-trained for comparison. Figure 5 The results show that the proposed method significantly outperforms other methods across all settings for 10-way-1-shot, 15-way-1-shot, and 20-way-1-shot tasks. This further demonstrates the good generalization ability of the proposed method, which performs well even on more challenging tasks. This is because the proposed SOLF method can decorrelate feature channels, and the proposed channel and spatial feature enhancements can diversify the enhanced features, both of which are beneficial for classification.

Claims

1. A few-sample image classification method based on global and local feature augmentation, characterized in that, Includes the following steps: The image dataset is divided into training, validation, and test sets, and the test set is further divided into support sets. and query set ; The images obtained from the training set, validation set, and test set are preprocessed according to the required resolution. Pre-training is performed on the image data in the preprocessed training set: First, small batches of image data are randomly extracted from the training set multiple times. Second, an enhanced copy of the image is created using a rotation transformation. Multiple feature extractors are trained using the enhanced image data. Then, an auxiliary loss based on rotation angle prediction is used to create the optimal output manifold to enhance the generalization ability of the feature extractor. The orthogonal regularization method of the semantic orthogonal learning framework is used to calculate the correlation between each channel in the feature map of the image data in the training set extracted by the feature extractor, constrain it with the identity matrix, and select the optimal feature extractor by evaluating the results of the feature extractor on the validation set. The optimal feature extractor obtained through training is used to extract the support set. For each image's feature map, the importance of each channel is measured based on its channel weights. The k channels with the smallest weights are then selected for global-aware channel fusion with other image features to form a support set. Augmented set of image samples ; The trained feature extractor is used to extract the support set. For each image feature map, clustering is used to select local descriptors of the foreground and background for each image feature. Locally perceptual background smoothing is then performed using the centers of the background clusters. The smoothed image features are used as the support set. Augmented set of image samples ; The query set is extracted using the trained feature extractor. The feature map of each image is then compared with the support set. He Zeng Guangji , The distance between image features is used for classification to obtain predicted labels.

2. The few-sample image classification method based on global and local feature augmentation according to claim 1, characterized in that, The image dataset can be a small-sample image classification dataset such as miniImageNet, tieredImageNet, or CIFAR-FS, or a fine-grained benchmark classification dataset such as CUB Birds.

3. The few-sample image classification method based on global and local feature augmentation according to claim 1, characterized in that, The images in the obtained training, validation, and test sets are all scaled to obtain an 84x resolution for each image.

84.

4. The few-sample image classification method based on global and local feature augmentation according to claim 1, characterized in that, Small batches of image data are randomly extracted multiple times from the training set, denoted as . The extracted feature map is represented as Feature map The formula for calculating the similarity between channels is: ； in, and It is a feature map The first in The and the first One channel, yes transpose, Used to calculate the norm of a matrix; thus obtaining a Similarity matrix ; Similarity matrix To the identity matrix The formula for calculating proximity is: ; in, This is the loss function.

5. The few-sample image classification method based on global and local feature augmentation according to claim 1, characterized in that, Using the trained feature extractor Extract the support set from the corresponding test set. Feature map of each image For each image feature, the importance of its channels is measured based on channel weights, and the k channels with the smallest weights are selected. Composition of channel features-label pairs ,in In the feature map The k channels with the smallest weights are selected from the feature maps of the image data, and then k channels are randomly selected from the feature maps of other image data. ,in In the feature map The k randomly selected channels are merged to form an augmented set. The process is shown in the following formula: ； in, These are the features corresponding to the fusion. The selected class tag and primitive class tags A compromise is made to control the preservation of the augmented sample labels, and the final fused samples are all used as the support set. Augmented sets corresponding to image samples ,Right now .

6. The few-sample image classification method based on global and local feature augmentation according to claim 1, characterized in that, Using the trained feature extractor Extract the support set from the corresponding test set. Feature map of each image And represent it as a set of local descriptors , For the first The local descriptors are used to cluster the above set into two clusters using a clustering algorithm. ,in It is the i-th cluster formed by aggregation, and the cluster with the larger sum of the weights of the local descriptors within the two clusters is taken as the foreground cluster. Another as a background cluster The center of each cluster is represented as , of which Cluster center of each cluster The calculation process is shown in the following formula: ； in, Indicates the first A cluster; Indicates the first In the cluster, the first The m-th response of each local descriptor; finally, for local descriptors in the background cluster, the center of the cluster is used to replace them as the local background for smoothing, as shown in the following equation: ； in, It is a foreground cluster Local descriptors in It is the cluster center of the background cluster. It is the first one after background smoothing Local descriptors; smoothed image features as support set Augmented sets corresponding to image samples ,Right now .

7. The few-sample image classification method based on global and local feature augmentation according to claim 1, characterized in that, Using the trained feature extractor Extract the query set under the corresponding test set. Each image Feature map Then, based on its relationship with the support set and what was obtained , The image features are used to calculate the feature mean for each category. ,calculate The classification probability distribution is obtained by using softmax to calculate the distances between the class means and the values ​​of each class. The calculation process is shown in the following formula: ； in, yes The feature mean of the category, where exp represents the exponential function. Used to calculate the distance between feature vectors, Euclidean distance is commonly used.

8. A few-sample image classification system based on global and local feature augmentation, characterized in that, include: The image dataset partitioning module is used to divide the image dataset into training, validation, and test sets, and further partition the test set into support sets. and query set ; The image data preprocessing module is used to preprocess the images in the obtained training set, validation set, and test set according to the required resolution. The pre-training module is used to pre-train the image data in the preprocessed training set. First, small batches of image data are randomly extracted from the training set multiple times. Second, an enhanced copy of the image is created using a rotation transformation. Multiple feature extractors are trained using the enhanced image data. Then, an auxiliary loss based on rotation angle prediction is used to create the optimal output manifold to enhance the generalization ability of the feature extractors. The orthogonal regularization method of the semantic orthogonal learning framework is used to calculate the correlation between each channel in the feature map of the image data in the training set extracted by the feature extractor, constrain it with the identity matrix, and select the optimal feature extractor by evaluating the results of the feature extractor on the validation set. The testing module is used to extract the support set using the optimal feature extractor obtained through training. For each image's feature map, the importance of each channel is measured based on its channel weights. The k channels with the smallest weights are then selected for global-aware channel fusion with other image features to form a support set. Augmented set of image samples ; The trained feature extractor is used to extract the support set. For each image feature map, clustering is used to select local descriptors of the foreground and background for each image feature. Locally perceptual background smoothing is then performed using the centers of the background clusters. The smoothed image features are used as the support set. Augmented set of image samples ; The query set is extracted using the trained feature extractor. The feature map of each image is then compared with the support set. He Zeng Guangji , The distance between image features is used for classification to obtain predicted labels.

9. A device for classifying few-sample images based on global and local feature augmentation, characterized in that, Includes memory and processor, wherein: Memory is used to store computer programs that can run on a processor; A processor, configured to, while running the computer program, perform the steps of a few-sample image classification method based on global and local feature augmentation as described in any one of claims 1-7.

10. A storage medium, characterized in that, The storage medium stores a computer program that, when executed by at least one processor, implements the steps of a few-sample image classification method based on global and local feature augmentation as described in any one of claims 1-7.

Images