A few-shot image classification method based on data augmentation and mutual attention

Abstract

This application relates to the field of image classification technology, and specifically to a few-shot image classification method based on data augmentation and mutual attention. For few-shot image classification tasks, the method transfers rich knowledge information from the base class to the new class, finds the nearest neighbor base class to the new class, and reconstructs the new class statistics using a weighted average method based on the distance between the statistics of the base class and the new class, thereby expanding the few-shot features of the new class. This parameter-free generation method simply and efficiently produces more diverse few-shot features. For the expanded few-shot data, a mutual attention mechanism is used to improve its matching performance with test samples, adaptively highlighting key information relevant to the classification task, generating feature representations more suitable for the new class task, while suppressing irrelevant and noisy information. Finally, a prototype network and support vector machine are jointly trained to find the optimal decision boundary for the few-shot classification task, effectively improving the accuracy of the few-shot classification task.

Description

Technical Field

[0001] The embodiments of this application relate to the field of image classification technology, and in particular to a few-sample image classification method based on data augmentation and mutual attention. Background Technology

[0002] With the rapid development of deep learning technology, model performance increasingly relies on large-scale labeled data. However, in real-world scenarios, acquiring large-scale data presents numerous challenges, such as difficulty in data acquisition (high sample collection costs in fields like medicine and remote sensing, where samples for certain categories may be extremely scarce), high labeling costs (labeling large-scale data requires significant manpower and expertise and is time-consuming and labor-intensive), insufficient data category diversity (target categories in many tasks change frequently or are uncertain, making it difficult to provide enough training samples for each category), and so on. Therefore, traditional deep learning methods often perform poorly when data is scarce, while few-shot learning, by simulating the human ability to learn from few samples, can address these problems to some extent.

[0003] Few-shot learning has shown broad application prospects in many fields, including but not limited to character generation, robotics, drug discovery, translation, and recommendation systems. It is particularly suitable for scenarios with high data collection costs and limited data volumes, such as image classification, image retrieval, object tracking, and gesture recognition. Few-shot learning technology provides a more flexible and efficient learning paradigm for the field of artificial intelligence, enabling machine learning systems to better adapt to diverse application needs. Essentially, few-shot learning learns to recognize unknown categories—categories that the model has not encountered during training—when data is scarce and annotations are minimal.

[0004] In practical few-shot learning tasks, training data is extremely scarce, with only a very small number of labeled samples for each category. This data scarcity leads to the following problems: First, data scarcity can cause overfitting and insufficient features. In few-shot learning scenarios, the lack of training data prevents the model from effectively learning robust feature representations from a limited number of samples, as the model is prone to overfitting to the training data due to the extremely small number of training samples. Second, in few-shot learning, the model's goal is to classify query samples using a small number of support samples. Therefore, modeling the relationship between support samples and query samples is crucial. Summary of the Invention

[0005] In view of this, embodiments of this application propose a few-shot image classification method based on data augmentation and mutual attention, which can effectively expand the few-shot image classification and sample the mutual attention mechanism to generate feature representations that are more suitable for the current new class tasks, thereby improving the accuracy of few-shot classification tasks and expanding the application scope of few-shot classification.

[0006] In a first aspect, embodiments of this application propose a few-shot image classification method based on data augmentation and mutual attention, comprising the following steps: obtaining a few-shot image set, which contains M categories, including A base classes and B new classes. Each few-shot image in each base class is labeled with a category label, and only some few-shot images in each new class are labeled with category labels; based on the few-shot images of all base classes, constructing several meta-training tasks conforming to the N-way-K-shot paradigm, performing meta-training on a pre-constructed classification model composed of a feature extraction module, a mutual attention module, and an output module, and using the feature extraction module to calculate the prototype of each base class; based on the few-shot images of all new classes, constructing several meta-testing tasks conforming to the N-way-K-shot paradigm, each meta-testing task containing N new classes, each new class... The system consists of K small sample images labeled with class tags, forming the support set, and the remaining small sample images forming the query set. Using a feature extraction module, feature vectors are extracted from each small sample image in the support set and query set for each new class. The prototype of each new class is calculated, and based on the prototypes of each new class and each base class, the nearest neighbor base class of each new class is determined. The distribution of the support set for each new class is reconstructed using a weighted average based on the prototypes of the nearest neighbor base classes, and random sampling is performed on the reconstructed distribution to obtain the expanded support set. Using a mutual attention module, a mutual attention mechanism is applied to each feature vector in the query set and the expanded support set to obtain query features of the query set and support features of the expanded support set. Based on each query feature, each support feature, and the output network, prototype loss and global classification loss are applied to the expanded support set to test the query set.

[0007] Optionally, the feature extraction module is used to calculate the prototypes of each base class, including:

[0008] The feature extraction module of the classification model completed by meta-training is used to extract features from each small sample image in each base class to obtain the feature vector of each small sample image in each base class.

[0009] Based on the feature vectors of each small sample image of each base class, the mean and variance of each base class are calculated. The mean and variance of each base class are expressed by the following formulas:

[0010] ;

[0011] ;

[0012] in, Indicates the first The total number of small sample images of each base class. Indicates the first The first base class Feature vectors of a small sample image Indicates the first The mean of each base class. Indicates the first The variance of each base class;

[0013] Based on the feature vectors of each small sample image of each base class and the mean of each base class, the covariance matrix of each base class is calculated. The covariance matrix of each base class is expressed by the formula:

[0014] ;

[0015] in, Indicates the first The covariance matrix of each base class.

[0016] Optionally, the feature extraction module is used to extract feature vectors from each small sample image in the support set and query set of each new class, and to calculate the prototype of each new class, including:

[0017] The feature extraction module of the classification model completed by meta-training is used to extract features from each small sample image in the support set and query set of each new class, and obtain the feature vector of each small sample image in the support set and query set of each new class.

[0018] The Tukey power-step method is used to transform the feature vectors of each sample image, resulting in the transformed feature vectors of each sample image. The transformed feature vectors of each sample image are expressed by the following formula:

[0019] ;

[0020] in, These are preset hyperparameters used to adjust the correction distribution. Recover the original feature vector in time. Indicates the first The first new class The transformed feature vector of a small sample image;

[0021] Based on the transformed feature vectors of each small sample image for each new class, the prototype of each new class is calculated, and the prototype of each new class is expressed by the formula:

[0022] ;

[0023] in, Indicates the first The total number of small sample images of each new class. Indicates the first The prototype of a new class.

[0024] Optionally, based on the prototypes of each new class and each base class, the nearest neighbor base class of each new class is determined, including:

[0025] Iterate through each new class, calculate the Euclidean distance between the prototype of the current new class and the mean of each base class, and select the prototype with the closest Euclidean distance. Each base class is selected as the nearest neighbor base class of the new class.

[0026] Let the first The set of nearest-neighbor base classes of each new class is , This can be expressed by the formula:

[0027] ;

[0028] ;

[0029] in, Indicates the first The set of Euclidean distances corresponding to each new class The elements in the array are arranged in ascending order. The Euclidean distance between the new class and the mean of each base class Indicates from Select before Operators for the top element.

[0030] Optionally, the distribution of the support set of the new class is reconstructed by weighted averaging based on the prototype of the nearest neighbor base class, and random sampling is performed in the reconstructed distribution to obtain the expanded support set, including:

[0031] Use sets Description of the The Euclidean distance between the prototype of the new class and the mean of each nearest neighbor base class. This can be expressed by the formula: , ;

[0032] right The softmax function is used for normalization to obtain the weights corresponding to each nearest neighbor base class. The weights corresponding to each nearest neighbor base class are expressed by the following formula:

[0033] ;

[0034] in, Indicates the first The first new class The weights corresponding to the nearest neighbor base classes;

[0035] Using the weighted average of the statistics of each nearest neighbor base class, the statistic of the nearest neighbor base class is calculated, and then... The distribution of each small sample image in the feature space of the support set of the new class is calibrated and reconstructed to obtain the reconstructed distribution. The distribution of the reconstructed support set of each new class is expressed by the formula:

[0036] ;

[0037] ;

[0038] ;

[0039] in, Indicates the first The distribution of the reconstructed support set of each new class. Indicates the first The first new class in the support set The calibration mean of a small sample of images. Show the first The first new class in the support set The calibration covariance matrix of a small sample image. These are preset hyperparameters used to reduce the distribution approximation error;

[0040] Random sampling is performed in each reconstructed distribution to generate m random samples with a multivariate normal distribution, resulting in the expanded support set; where the support set is expanded from the original... A small sample of images was expanded to A small sample of images.

[0041] Optionally, a mutual attention module is used to perform a mutual attention mechanism on each feature vector of the query set and the expanded support set to obtain each query feature of the query set and each support feature of the expanded support set, including:

[0042] Calculate the cosine similarity between each feature vector of the query set and each feature vector of the expanded support set at each pair of spatial locations to obtain the cross-correlation tensor.

[0043] A convolutional matching process is adopted, and the cross-correlation tensor is refined through 4D convolution and matching kernel. Based on the refined cross-correlation tensor, the query attention map and the support attention map are obtained.

[0044] The query features are obtained by multiplying each feature vector of the query set with the query attention map and performing a pooling operation. The support features are obtained by multiplying each feature vector of the expanded support set with the support attention map and performing a pooling operation.

[0045] Optionally, the cosine similarity between each feature vector of the query set and each feature vector of the expanded support set at each pair of spatial locations is calculated to obtain the cross-correlation tensor, which is achieved through the following formula:

[0046] ;

[0047] in, Represents the feature vector of the query set. This represents the feature vector of the expanded support set. This indicates the calculation of cosine similarity. Represents the cross-correlation tensor;

[0048] A convolutional matching process is employed, using 4D convolution and a matching kernel to refine the cross-correlation tensor. Based on the refined cross-correlation tensor, the query attention map and support attention map are obtained, implemented using the following formula:

[0049] ;

[0050] ;

[0051] in, This represents the refined cross-correlation tensor. and These represent the width and height of the refined cross-correlation tensor, respectively. These are preset weighted hyperparameters. This indicates a query attention graph. This indicates support for attention graphs;

[0052] The query features are obtained by multiplying each feature vector of the query set with the query attention map and then performing a pooling operation. Similarly, the support features are obtained by multiplying each feature vector of the expanded support set with the support attention map and then performing a pooling operation. This is achieved using the following formula:

[0053] ;

[0054] ;

[0055] in, Indicates query characteristics, This represents the support features of the expanded support set.

[0056] Optionally, based on each query feature, each support feature, and the output network, prototype loss and global classification loss are applied to the expanded support set, including:

[0057] Based on each query feature and the output network, a global classification loss is constructed. , This can be expressed by the formula:

[0058] ;

[0059] in, This indicates that small sample features in the query set are predicted as categories. The probability of;

[0060] Based on each query feature and each support feature, a prototype loss is constructed. , This can be expressed by the formula:

[0061] ;

[0062] ;

[0063] in, Indicates support for the centralized first The prototype of each category;

[0064] based on and Construct target loss This allows for testing of the query set. This can be expressed by the formula: , The preset balancing weights.

[0065] This application proposes a few-shot image classification method based on data augmentation and mutual attention, which has the following advantages compared with existing few-shot image classification methods.

[0066] First, this application transforms the parameterized generation method into a non-parametric distribution correction generation method. It uses the statistics of the base class of the auxiliary dataset to reconstruct the distribution of each sample in the support set, and then samples and expands the features of the support set according to the constructed non-parametric normal distribution. Compared with traditional data augmentation methods, the feature generation of this application avoids the high demand for computing resources of complex models such as traditional generative adversarial networks (GANs).

[0067] Second, the support set expansion process cannot dynamically capture information from new class query samples, thus the support samples may have poor adaptability to new class tasks. This application introduces a mutual attention module, which can adaptively highlight the importance of key samples or features in the support samples while suppressing irrelevant or noisy information. Through the dynamic interaction between support samples and query samples, the mutual attention mechanism can better uncover the correlation between support samples and query samples, generating feature representations that are more adapted to the current new class task.

[0068] Third, this application employs joint training of prototype loss and global loss, combining the two in a weighted manner to adjust the weights in the loss function, thereby achieving the best few-sample image classification effect, thus improving the accuracy of few-sample classification tasks and expanding the application scope of few-sample classification.

[0069] In a second aspect, embodiments of this application propose an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a few-sample image classification method based on data augmentation and mutual attention as described in the first aspect above.

[0070] Fourthly, embodiments of this application propose a computer-readable storage medium storing a computer program that, when executed by a processor, enables a few-sample image classification method based on data augmentation and mutual attention as described in the first aspect above.

[0071] It is understood that the beneficial effects of the second and third aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0072] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies of this application will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0073] Figure 1 This is a flowchart of a few-sample image classification method based on data augmentation and mutual attention provided in one embodiment of this application;

[0074] Figure 2 This is a schematic diagram of the structure of a classification model provided in one embodiment of this application;

[0075] Figure 3 This is a schematic diagram of the structure of the mutual attention module provided in one embodiment of this application;

[0076] Figure 4 This is a comparison chart of the classification results of the classification model on the mini-ImageNet dataset and the CUB dataset for small sample image classification in one embodiment of this application, showing the results of 5-way-1-shot and 5-way-5-shot respectively.

[0077] Figure 5 This is a comparison of the classification results of the classification model on the tiered-ImageNet dataset and the CIFAR-FS dataset for small sample image classification in one embodiment of this application.

[0078] Figure 6 This is a visualization diagram of t-SNE reconstructing a small sample normal distribution based on auxiliary information, provided in one embodiment of this application.

[0079] Figure 7 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0080] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. In the various embodiments of this application, many technical details are presented to enable the reader to better understand this application. However, even without these technical details and various variations and modifications based on the following embodiments, the technical solutions claimed in this application can be implemented. The division of the following embodiments is only for convenience of description and should not constitute any limitation on the specific implementation of this application. The various embodiments can be combined with and referenced by each other without contradiction.

[0081] In recent years, research teams at home and abroad have proposed some data augmentation-based few-sample image classification methods. By augmenting the data of few samples, the few-sample classification task can be transformed into a normal supervised task. The proposed data augmentation methods can be roughly divided into three categories.

[0082] The first category is small-sample image classification based on weak labels. After collecting a large amount of data, manually labeling this data is a time-consuming and resource-intensive task. How to automatically assign pseudo-labels to these unlabeled samples has become a key research issue. This method often uses semi-supervised or self-supervised learning strategies, training the model with a small amount of labeled data and a large amount of unlabeled data. Then, the model is used to predict unlabeled data, generating pseudo-labels. These pseudo-labels not only expand the training set but also improve the model through iteration. Inspired by transfer learning, Wang et al. exposed the upper layers of the network to a large amount of unlabeled data during the unsupervised meta-training stage, optimizing the low-density separator and decoupling it from specific categories, thus obtaining a more generalized representation. Wang et al. used a concurrent parametric response model to evaluate the fitness of the samples. Through this evaluation, pseudo-labeled data with high confidence can be gradually selected to enhance the generalization and fitting ability of the regression model, thereby improving small-sample classification performance. Li et al. studied the label noise propagation problem and improved the baseline method with a denoising network that learns the distribution of clean and noisy pseudo-labeled samples through a hybrid model. This helps to estimate the confidence values ​​of pseudo-labeled samples and select reliable samples with less noise from them for iterative optimization of the few-sample classifier.

[0083] The second category is few-shot image classification based on generative models. This method utilizes generative models, such as GANs or VAEs, to generate new image samples. AFHN explored using conditional generative adversarial networks (FANs) to generate richer recognition features. Duan et al. designed a simple yet highly effective WeditGAN, which transfers the model from the source domain to the few-shot target domain by relocating the latent space and learning a constant ∆w, thus completing few-shot image generation. Yu et al. proposed a few-shot feature enhancement based on a masked autoencoder (MFGN). The MFGN encoder acts as a feature extractor, taking a set of support samples of the same class as input and adding sample numbers. It extracts visual features from the input samples, and sends the visual features of the masked samples, the visual features of the unmasked samples, and the masked sample numbers to the decoder for reconstruction. After training, the encoder and decoder can be combined to generate more feature embeddings, enhancing the feature embeddings of the support set samples. Hong et al. presented a novel Delta generative adversarial network, which consists of a reconstruction subnetwork and a generation subnetwork. The reconstruction subnetwork captures the internal class transformation, i.e., delta, between pairs of the same class. The generative subnetwork generates deltas for specific samples of the input image and combines them with the input image to generate new images of the same category.

[0084] The third category is feature space-based enhancement methods. Common techniques in this approach include feature interpolation, feature fusion, and feature transformation. Laso is an early and representative work. Since each sample has multiple labels, more relevant samples can be generated in the hidden space by defining the intersection and difference of the feature spaces. PFEMed connects pre-trained general features with specific features to further enhance the semantic information of FSL features. Chen et al. proposed adding a set of reference images, which consists of many pairs of images of the same class. In the embedding space, the features of the reference pairs are added or subtracted. This not only enriches the diversity but also introduces reference features. Recently, Yang et al. studied the hypothesis that when the base class and the new class are semantically similar, their means and variances can be largely shared. The means and variances of the base class can correct the data distribution of the new class. In addition, Xu et al. proposed a framework for decomposing the variance in the dataset, where one represents the intra-class variance and the others represent the embedding of discriminative information. By iterative sampling, the intra-class variance can be added to the discriminative features. In this way, features are learned to the maximum extent while maintaining a large intra-class variance.

[0085] However, the above methods have certain problems.

[0086] First, current data augmentation methods, particularly those for unlabeled data, often employ semi-supervised learning strategies. These involve training an initial model using labeled data, then using this model to predict unlabeled data and generate pseudo-labels. The pseudo-labeled data is then used alongside the labeled data for further model training. However, generating pseudo-labels can lead to incorrect labels. These incorrect labels can be mixed into the training set, causing the model to learn erroneous information and impacting overall performance. Furthermore, GAN training is typically complex, requiring significant computational resources, and may encounter issues such as pattern collapse during sample generation.

[0087] Second, since few-shot learning studies use a small number of support samples to classify query samples, the relationship between support samples and query samples is crucial. Mutual attention mechanisms are used to align features between the support and query sets, reducing background noise and highlighting target regions to generate more discriminative representations. Current mutual attention mechanisms generate mutual attention maps by connecting various positions of query and support samples. However, since the positions and sizes of objects are not fixed, semantic inconsistencies often exist in the feature space. Directly connecting individual positions to generate mutual attention maps results in redundant connections and exacerbates semantic mismatch problems, leading to unreliable similarity confidence.

[0088] Third, in few-shot learning, the limited number of samples in each category means that the distribution of samples cannot fully cover the feature space of the categories. This directly leads to the model's inability to fully learn the feature diversity of each category during the training phase, making it difficult to accurately delineate the decision boundaries of each category during decision-making.

[0089] To address the aforementioned technical problems, one embodiment of this application proposes a few-shot image classification method based on data augmentation and mutual attention, applied to an electronic device. The electronic device can be a terminal or a server. In this embodiment and the following embodiments, the electronic device is described using a server as an example. The implementation details of the few-shot image classification method based on data augmentation and mutual attention proposed in this embodiment are described in detail below. The following implementation details are provided for ease of understanding and are not necessary for implementing this solution.

[0090] The core idea of ​​this paper is to reconstruct the statistics of the small sample dataset by using the base class statistics of similar auxiliary datasets, thereby expanding the features of the small sample dataset. Then, the attention mechanism is used to improve the matching performance of the small sample training samples and test samples. This can adaptively highlight the key information related to the classification task and generate feature representations that are more suitable for the current new class task. Finally, after adopting a dual optimization loss strategy, the model can provide sufficient decision boundaries between classes to improve the accuracy of small sample classification tasks.

[0091] The specific process of the few-sample image classification method based on data augmentation and mutual attention proposed in this embodiment can be as follows: Figure 1 As shown, it includes:

[0092] Step 101: Obtain a small sample image set. The small sample image set contains M categories, including A base classes and B new classes. Each small sample image in each base class is labeled with a category label, and only some small sample images in each new class are labeled with category labels.

[0093] In the specific implementation, the server first needs to obtain a small sample image set. In this embodiment, the small sample image set follows the typical Few-Shot classification setting. The small sample image set contains small sample images of M categories. Among the M categories, there are A base classes and B new classes. Each small sample image in each base class is labeled with a category label, and only some small sample images in each new class are labeled with a category label.

[0094] In one example, M is an integer greater than 2, and M = A + B.

[0095] It should be noted that in this embodiment, when processing few-shot image classification tasks, a cross-task mechanism of meta-learning is used for training and testing. Each task is called a meta-task, which includes two main parts: meta-training and meta-testing. In this embodiment, meta-training is implemented based on few-shot images of each base class in the few-shot image set, while meta-testing is implemented based on few-shot images of each new class in the few-shot image set.

[0096] Step 102: Based on the few sample images of all base classes, construct several meta-training tasks that conform to the N-way-K-shot paradigm, perform meta-training on the pre-constructed classification model consisting of a feature extraction module, a mutual attention module, and an output module, and use the feature extraction module to calculate the prototype of each base class.

[0097] In the specific implementation, after the server obtains the small sample image set, it needs to construct several meta-training tasks that conform to the N-way-K-shot paradigm (5-way-5-shot or 5-way-1-shot) based on the small sample images of all base classes. Meta-training is performed on the pre-constructed classification model consisting of a feature extraction module, a mutual attention module, and an output module. The feature extraction module of the classification model completed by meta-training is then used to calculate the prototype of each base class.

[0098] In one example, the specific components of a classification model can be as follows: Figure 2 As shown.

[0099] In one example, the prototype of the base class can be represented by a matrix of mean, variance, and covariance.

[0100] The server uses the feature extraction module of the classification model trained by meta-training to extract features from each sample image in each base class, thus obtaining the feature vector of each sample image in each base class.

[0101] Subsequently, based on the feature vectors of each small sample image of each base class, the mean and variance of each base class are calculated. The mean and variance of each base class are expressed by the following formula:

[0102] ;

[0103] ;

[0104] in, Indicates the first The total number of small sample images of each base class. Indicates the first The first base class Feature vectors of a small sample image Indicates the first The mean of each base class. Indicates the first The variance of each base class.

[0105] Next, based on the feature vectors of each small sample image of each base class and the mean of each base class, the covariance matrix of each base class is calculated. The covariance matrix of each base class is expressed by the formula:

[0106] ;

[0107] in, Indicates the first The covariance matrix of each base class.

[0108] Step 103: Based on the few sample images of all new classes, construct several meta-test tasks that conform to the N-way-K-shot paradigm. Each meta-test task contains N new classes, and each new class has K few sample images labeled with class labels. The few sample images labeled with class labels form the support set, and the remaining few sample images form the query set.

[0109] In one example, after the server completes all meta-training tasks, it obtains a meta-trained classification model. The next step is meta-testing to further improve performance. Based on few sample images of all new classes, the server constructs several meta-testing tasks conforming to the N-way-K-shot paradigm (5-way-5-shot or 5-way-1-shot). Each meta-testing task contains N new classes, and each new class has K few sample images labeled with class tags. The few sample images labeled with class tags form the support set, and the remaining few sample images form the query set.

[0110] Step 104: Using the feature extraction module, extract the feature vectors of each small sample image in the support set and query set of each new class, calculate the prototype of each new class, and determine the nearest neighbor base class of each new class based on the prototype of each new class and the prototype of each base class.

[0111] In the specific implementation, for each meta-test task, the server needs to use the feature extraction module of the classification model completed by meta-training to extract the feature vectors of each small sample image in the support set and query set of each new class, calculate the prototype of each new class, and determine the nearest neighbor base class of each new class based on the prototype of each new class and the prototype of each base class.

[0112] In one example, the server uses the feature extraction module of the meta-trained classification model to extract features from each sample image in the support set and query set of each new class, thereby obtaining the feature vectors of each sample image in the support set and query set of each new class.

[0113] Next, the server needs to transform the feature vectors of each sample image in the support set and query set of each new class into a form that better conforms to a normal distribution and is closer to a Gaussian distribution. This requires implementation based on Tukey's power-step method. The server performs data transformation on the feature vectors of each sample image based on Tukey's power-step method, obtaining the transformed feature vectors of each sample image. The transformed feature vectors of each sample image are expressed by the formula:

[0114] ;

[0115] in, These are preset hyperparameters used to adjust the correction distribution. Recover the original feature vector in time. Indicates the first The first new class The transformed feature vectors of a small sample image.

[0116] Following this, the server will calculate the prototype of each new class based on the transformed feature vectors of each small sample image for each new class. The prototype of each new class is expressed by the formula:

[0117] ;

[0118] in, Indicates the first The total number of small sample images of each new class. Indicates the first The prototype of a new class.

[0119] In one example, when determining the nearest neighbor base class for each new class, the server iterates through each new class, calculates the Euclidean distance between the prototype of the current new class and the mean of each base class, and selects the nearest neighbor base class. Let the nth base class be the nearest neighbor base class of the current new class. The set of nearest-neighbor base classes of each new class is , This can be expressed by the formula:

[0120] ;

[0121] ;

[0122] in, Indicates the first The set of Euclidean distances corresponding to each new class The elements in the array are arranged in ascending order. The Euclidean distance between the new class and the mean of each base class Indicates from Select before Operators for the top element.

[0123] Step 105: Reconstruct the distribution of the support set of the new class by weighted average based on the prototype of the nearest neighbor base class, and randomly sample from the reconstructed distribution to obtain the expanded support set.

[0124] In the specific implementation, after the server determines the nearest neighbor base class of each new class, it needs to reconstruct the distribution of the support set of the new class by weighted average based on the prototype of the nearest neighbor base class, and then perform random sampling in the reconstructed distribution to obtain the expanded support set.

[0125] In one example, for ease of description, the server uses a collection. Description of the The Euclidean distance between the prototype of the new class and the mean of each nearest neighbor base class. This can be expressed by the formula: , .

[0126] server The softmax function is used for normalization to obtain the weights corresponding to each nearest neighbor base class. The weights corresponding to each nearest neighbor base class are expressed by the following formula:

[0127] ;

[0128] in, Indicates the first The first new class The weights corresponding to the nearest neighbor base classes.

[0129] Once the weights are determined, reconstruction and calibration can proceed. The server uses a weighted average of the statistics of each nearest neighbor base class to perform the reconstructing and calibration. The distribution of each small sample image in the feature space of the support set of the new class is calibrated and reconstructed to obtain the reconstructed distribution. The distribution of the reconstructed support set of each new class is expressed by the formula:

[0130] ;

[0131] ;

[0132] ;

[0133] in, Indicates the first The distribution of the reconstructed support set of each new class. Indicates the first The first new class in the support set The calibration mean of a small sample of images. Show the first The first new class in the support set The calibration covariance matrix of a small sample image. These are preset hyperparameters used to reduce the distribution approximation error.

[0134] Finally, the server randomly samples from each reconstructed distribution to generate m random samples from a multivariate normal distribution, thus obtaining the expanded support set and achieving the goal of expanding the support set features. The support set is expanded from the original... A small sample image (feature) is expanded to A small sample of images (features).

[0135] Step 106: Using the mutual attention module, perform the mutual attention mechanism on each feature vector of the query set and the expanded support set to obtain each query feature of the query set and each support feature of the expanded support set.

[0136] In the specific implementation, after the support set is expanded, the server needs to use the mutual attention module of the classification model that has been meta-trained to perform a mutual attention mechanism on each feature vector of the query set and the expanded support set to obtain each query feature of the query set and each support feature of the expanded support set.

[0137] In one example, the specific structure of the mutual attention module can be as follows: Figure 3 As shown.

[0138] In one example, through a mutual attention mechanism, the server can adaptively highlight the importance of key samples or features in the supporting samples while suppressing irrelevant or noisy information. By dynamically interacting with the query samples, the mutual attention mechanism can better uncover the correlation between them, generating feature representations more suited to the current novel task.

[0139] In one example, the server calculates the cosine similarity between each feature vector of the query set and each feature vector of the expanded support set at each pair of spatial locations, resulting in a cross-correlation tensor. The cross-correlation tensor is a four-dimensional tensor.

[0140] The cosine similarity between each feature vector of the query set and each feature vector of the expanded support set at each pair of spatial locations is calculated to obtain the cross-correlation tensor, which can be achieved using the following formula:

[0141] ;

[0142] in, Represents the feature vector of the query set. This represents the feature vector of the expanded support set. This indicates the calculation of cosine similarity. Represents the cross-correlation tensor.

[0143] Next, the server will employ a convolutional matching process, refining the cross-correlation tensor through 4D convolutions and matching kernels. Batch normalization and ReLU activation functions need to be inserted between convolutions. Subsequently, based on the refined cross-correlation tensor, the server obtains the query attention map and the support attention map.

[0144] For samples with significant appearance variations in small-sample tasks, the cross-correlation tensor graph may contain unreliable similarity scores between query and support samples, i.e., match scores. To eliminate these unreliable matches, the cross-correlation tensor needs to be refined.

[0145] The cross-correlation tensor only represents the similarity between each position of the support sample and each position of the query sample in the feature space. Through the refined tensor, the server can generate an attention map, which shows the relationship between the query and the support.

[0146] A convolutional matching process is employed, using 4D convolution and a matching kernel to refine the cross-correlation tensor. Based on the refined cross-correlation tensor, the query attention map and support attention map are obtained, which can be achieved using the following formula:

[0147] ;

[0148] ;

[0149] in, This represents the refined cross-correlation tensor. and These represent the width and height of the refined cross-correlation tensor, respectively. These are preset weighted hyperparameters. This indicates a query attention graph. This indicates support for attention graphs.

[0150] Each location in the query sample is compared with all locations in the support image, resulting in a weighted matching distribution. The attention value represents the relevance of the query location to the best-matching region in the support image. The attention map helps the model extract features that aid in classification or matching by adjusting the relationship between the query and support samples during the learning process. This mutual attention mechanism enables the model to effectively classify with a small number of samples.

[0151] Finally, the server needs to multiply each feature vector of the query set with the query attention map and perform a pooling operation to obtain each query feature of the query set. Then, it needs to multiply each feature vector of the expanded support set with the support attention map and perform a pooling operation to obtain each support feature of the expanded support set.

[0152] The query features are obtained by multiplying the feature vectors of the query set with the query attention map and then performing a pooling operation. Similarly, the support features are obtained by multiplying the feature vectors of the expanded support set with the support attention map and then performing a pooling operation. This can be achieved using the following formula:

[0153] ;

[0154] ;

[0155] in, Indicates query characteristics, This represents the support features of the expanded support set.

[0156] It is important to note that the sum of the elements in the attention map is equal to 1, therefore and It can be considered as... and Weighted convex combination.

[0157] Step 107: Based on each query feature, each support feature, and the output network, prototype loss and global classification loss are applied to the expanded support set to test the query set.

[0158] In the specific implementation, in the meta-testing task, the server will finally test the query set by using prototype loss and global classification loss on the expanded support set based on each query feature, each support feature, and the output network.

[0159] In one example, the output network is an SVM. The server first constructs a global classification loss based on each query feature and the output network. , This can be expressed by the formula:

[0160] ;

[0161] in, This indicates that small sample features in the query set are predicted as categories. The probability of.

[0162] Next, the server needs to construct a prototype loss based on each query feature and each supporting feature. , This can be expressed by the formula:

[0163] ;

[0164] ;

[0165] in, Indicates support for the centralized first Prototypes for each category.

[0166] Finally, the server is based on and Construct target loss This allows for testing of the query set. This can be expressed by the formula: , The preset balancing weights.

[0167] In one example, in addition to meta-training and meta-testing, the server also needs to conduct experimental verification of the classification model completed by meta-testing, evaluate its performance, and select a classification model that can be deployed to real-world application scenarios.

[0168] This embodiment proposes a few-shot image classification method based on data augmentation and mutual attention, which has the following advantages compared with the few-shot image classification methods proposed so far.

[0169] First, this application transforms the parameterized generation method into a non-parametric distribution correction generation method. It uses the statistics of the base class of the auxiliary dataset to reconstruct the distribution of each sample in the support set, and then samples and expands the features of the support set according to the constructed non-parametric normal distribution. Compared with traditional data augmentation methods, the feature generation of this application avoids the high demand for computing resources of complex models such as traditional generative adversarial networks (GANs).

[0170] Second, the support set expansion process cannot dynamically capture information from new class query samples, thus the support samples may have poor adaptability to new class tasks. This application introduces a mutual attention module, which can adaptively highlight the importance of key samples or features in the support samples while suppressing irrelevant or noisy information. Through the dynamic interaction between support samples and query samples, the mutual attention mechanism can better uncover the correlation between support samples and query samples, generating feature representations that are more adapted to the current new class task.

[0171] Third, this application employs joint training of prototype loss and global loss, combining the two in a weighted manner to adjust the weights in the loss function, thereby achieving the best few-sample image classification effect, thus improving the accuracy of few-sample classification tasks and expanding the application scope of few-sample classification.

[0172] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0173] In one embodiment, in order to verify the performance of the classification model (hereinafter referred to as ours) obtained in this application, we conducted relevant simulation experiments.

[0174] The simulation experiment was conducted using PyCharm software and the Python language on an Intel(R) Xeon(R) Gold 5222 CPU @ 3.80GHz CPU, 125G of memory, and Ubuntu 20.04.2 operating system.

[0175] We evaluate the performance of our classification model on the mini-ImageNet, tiered-ImageNet, CUB, and CIFAR-FS datasets. The mini-ImageNet and tiered-ImageNet datasets are built upon the ImageNet dataset and contain a rich variety of categories covering various animals and objects. The CUB dataset is a more fine-grained dataset, including various bird species. Datasets with different granularities can have different feature reconstructions for their feature spaces. The CIFAR-FS dataset is built upon the CIFAR-100 dataset, and its categories cover a wide range of animals and objects. We aim to demonstrate the effectiveness and generality of our model on these four datasets.

[0176] Figure 4 This is a comparison chart showing the results of different models classifying 5-way-1-shot and 5-way-5-shot few-sample images on the mini-ImageNet and CUB datasets, respectively. Figure 5 This is a comparison chart showing the results of different models classifying 5-way-1-shot and 5-way-5-shot few-shot images on the Tiered-ImageNet and CIFAR-FS datasets, respectively. Figure 4 and Figure 5 As can be seen, the classification model proposed in this application has excellent performance.

[0177] besides, Figure 6 This is a visualization diagram of t-SNE based on auxiliary information to reconstruct a small sample normal distribution. Figure 6 In the diagram, the star represents support set features, the circle represents generated features, and the triangle represents query set features. Figure 6 As can be seen, the support set has been effectively expanded.

[0178] Another embodiment of this application proposes an electronic device, the specific structure of which is as follows: Figure 7As shown, it includes: at least one processor 201; and a memory 202 communicatively connected to the at least one processor 201; wherein the memory 202 stores instructions executable by the at least one processor 201, the instructions being executed by the at least one processor 201 to enable the at least one processor 201 to perform a few-sample image classification method based on data augmentation and mutual attention as described in the above method embodiments.

[0179] The memory and processor can be connected via a bus, which can include any number of interconnecting buses and bridges, connecting various circuits of one or more processors and the memory. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be described further herein. The bus interface is responsible for providing an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which further receives data and transmits it to the processor.

[0180] The processor manages the bus and general processing, and also provides various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory is used to store data used by the processor during operation.

[0181] Another embodiment of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, enables a few-sample image classification method based on data augmentation and mutual attention as described in the above method embodiments.

[0182] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (such as a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, portable hard drive, ROM (Read-Only Memory), RAM (Random Access Memory), magnetic disk, or optical disk.

[0183] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing this application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of this application.

Claims

1. A few-sample image classification method based on data augmentation and mutual attention, characterized in that, include: Obtain a small sample image set, which contains M categories. The M categories include A base classes and B new classes. Each small sample image in each base class is labeled with a category label, while only some small sample images in each new class are labeled with a category label. Based on few sample images of all base classes, several meta-training tasks conforming to the N-way-K-shot paradigm are constructed to perform meta-training on a pre-constructed classification model consisting of a feature extraction module, a mutual attention module, and an output module. The feature extraction module is then used to calculate the prototype of each base class. Based on the few sample images of all new classes, construct several meta-test tasks that conform to the N-way-K-shot paradigm. Each meta-test task contains N new classes, and each new class has K few sample images labeled with class labels. The few sample images labeled with class labels form the support set, and the remaining few sample images form the query set. Using the feature extraction module, feature vectors of each small sample image in the support set and query set of each new class are extracted, the prototype of each new class is calculated, and the nearest neighbor base class of each new class is determined based on the prototype of each new class and the prototype of each base class. The distribution of the support set of the new class is reconstructed by weighted average based on the prototype of the nearest neighbor base class, and random sampling is performed in the reconstructed distribution to obtain the expanded support set. Using the mutual attention module, a mutual attention mechanism is performed on each feature vector of the query set and the expanded support set to obtain each query feature of the query set and each support feature of the expanded support set. Based on each query feature, each support feature, and the output network, prototype loss and global classification loss are used on the expanded support set to test the query set.

2. The few-sample image classification method based on data augmentation and mutual attention according to claim 1, characterized in that, Using the feature extraction module, the prototypes of each base class are calculated, including: The feature extraction module of the classification model completed by meta-training is used to extract features from each small sample image in each base class to obtain the feature vector of each small sample image in each base class. Based on the feature vectors of each small sample image of each base class, the mean and variance of each base class are calculated. The mean and variance of each base class are expressed by the following formulas: ； ； in, Indicates the first The total number of small sample images of each base class. Indicates the first The first base class Feature vectors of a small sample image Indicates the first The mean of each base class. Indicates the first The variance of each base class; Based on the feature vectors of each small sample image of each base class and the mean of each base class, the covariance matrix of each base class is calculated. The covariance matrix of each base class is expressed by the formula: ； in, Indicates the first The covariance matrix of each base class.

3. The few-sample image classification method based on data augmentation and mutual attention according to claim 2, characterized in that, Using the feature extraction module, feature vectors are extracted from each small sample image in the support set and query set for each new class. The prototype of each new class is then calculated, including: The feature extraction module of the classification model completed by meta-training is used to extract features from each small sample image in the support set and query set of each new class, and obtain the feature vector of each small sample image in the support set and query set of each new class. The Tukey power-step method is used to transform the feature vectors of each sample image, resulting in the transformed feature vectors of each sample image. The transformed feature vectors of each sample image are expressed by the following formula: ； in, These are preset hyperparameters used to adjust the correction distribution. Recover the original feature vector in time. Indicates the first The first new class The transformed feature vectors of a small sample image; Based on the transformed feature vectors of each small sample image for each new class, the prototype of each new class is calculated, and the prototype of each new class is expressed by the formula: ； in, Indicates the first The total number of small sample images of each new class. Indicates the first The prototype of a new class.

4. The few-sample image classification method based on data augmentation and mutual attention according to claim 3, characterized in that, Based on the prototypes of each new class and each base class, determine the nearest neighbor base class of each new class, including: Iterate through each new class, calculate the Euclidean distance between the prototype of the current new class and the mean of each base class, and select the prototype with the closest Euclidean distance. Each base class is selected as the nearest neighbor base class of the new class. Let the first The set of nearest-neighbor base classes of each new class is , This can be expressed by the formula: ； ； in, Indicates the first The set of Euclidean distances corresponding to each new class The elements in the array are arranged in ascending order. The Euclidean distance between the new class and the mean of each base class Indicates from Select before Operators for the top element.

5. A few-sample image classification method based on data augmentation and mutual attention according to claim 4, characterized in that, The distribution of the support set of the new class is reconstructed by weighted averaging based on the prototype of the nearest neighbor base class, and random sampling is performed in the reconstructed distribution to obtain the expanded support set, including: Use sets Description of the The Euclidean distance between the prototype of the new class and the mean of each nearest neighbor base class. This can be expressed by the formula: , ; right The softmax function is used for normalization to obtain the weights corresponding to each nearest neighbor base class. The weights corresponding to each nearest neighbor base class are expressed by the following formula: ； in, Indicates the first The first new class The weights corresponding to the nearest neighbor base classes; Using the weighted average of the statistics of each nearest neighbor base class, the statistic of the nearest neighbor base class is calculated, and then... The distribution of each small sample image in the feature space of the support set of the new class is calibrated and reconstructed to obtain the reconstructed distribution. The distribution of the reconstructed support set of each new class is expressed by the formula: ； ； ； in, Indicates the first The distribution of the reconstructed support set of each new class. Indicates the first The first new class in the support set The calibration mean of a small sample of images. Show the first The first new class in the support set The calibration covariance matrix of a small sample image. These are preset hyperparameters used to reduce the distribution approximation error; Random sampling is performed in each reconstructed distribution to generate m random samples with a multivariate normal distribution, resulting in the expanded support set; where the support set is expanded from the original... A small sample of images was expanded to A small sample of images.

6. The few-sample image classification method based on data augmentation and mutual attention according to claim 1, characterized in that, Using a mutual attention module, a mutual attention mechanism is performed on the feature vectors of the query set and the expanded support set to obtain the query features of the query set and the support features of the expanded support set, including: Calculate the cosine similarity between each feature vector of the query set and each feature vector of the expanded support set at each pair of spatial locations to obtain the cross-correlation tensor. A convolutional matching process is adopted, and the cross-correlation tensor is refined through 4D convolution and matching kernel. Based on the refined cross-correlation tensor, the query attention map and the support attention map are obtained. The query features are obtained by multiplying each feature vector of the query set with the query attention map and performing a pooling operation. The support features are obtained by multiplying each feature vector of the expanded support set with the support attention map and performing a pooling operation.

7. A few-sample image classification method based on data augmentation and mutual attention according to claim 6, characterized in that, The cosine similarity between each feature vector of the query set and each feature vector of the expanded support set at each pair of spatial locations is calculated to obtain the cross-correlation tensor, which is achieved through the following formula: ； in, Represents the feature vector of the query set. This represents the feature vector of the expanded support set. This indicates the calculation of cosine similarity. Represents the cross-correlation tensor; A convolutional matching process is employed, using 4D convolution and a matching kernel to refine the cross-correlation tensor. Based on the refined cross-correlation tensor, the query attention map and support attention map are obtained, implemented using the following formula: ； ； in, This represents the refined cross-correlation tensor. and These represent the width and height of the refined cross-correlation tensor, respectively. These are preset weighted hyperparameters. This indicates a query attention graph. This indicates support for attention graphs; The query features are obtained by multiplying each feature vector of the query set with the query attention map and then performing a pooling operation. Similarly, the support features are obtained by multiplying each feature vector of the expanded support set with the support attention map and then performing a pooling operation. This is achieved using the following formula: ； ； in, Indicates query characteristics, This represents the support features of the expanded support set.

8. A few-sample image classification method based on data augmentation and mutual attention according to claim 7, characterized in that, Based on each query feature, each support feature, and the output network, prototype loss and global classification loss are applied to the expanded support set, including: Based on each query feature and the output network, a global classification loss is constructed. , This can be expressed by the formula: ； in, This indicates that small sample features in the query set are predicted as categories. The probability of; Based on each query feature and each support feature, a prototype loss is constructed. , This can be expressed by the formula: ； ； in, Indicates support for the centralized first The prototype of each category; based on and Construct target loss This allows for testing of the query set. This can be expressed by the formula: , The preset balancing weights.

9. An electronic device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform a few-sample image classification method based on data augmentation and mutual attention as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it can implement a few-sample image classification method based on data augmentation and mutual attention as described in any one of claims 1 to 8.

Images