Small sample image classification method based on data enhancement and mutual attention

Abstract

The invention relates to the technical field of image classification, in particular to a small sample image classification method based on data enhancement and mutual attention, aiming at a small sample image classification task, rich knowledge information on a base class is transferred to a new class, the nearest base class of the new class is searched, and the distance between the statistics of the base class and the new class is utilized to determine the classification result of the small sample image. And a weighted average method is adopted to reconstruct new-class statistics, so that new-class small sample features are expanded, and more diversified small sample features are simply and efficiently generated in a parameter-free generation mode. And for the expanded small sample data, through a mutual attention mechanism, the matching performance of the expanded small sample data and a test sample is improved, key information related to a classification task is adaptively highlighted, feature representation which is more adaptive to a new class task is generated, and irrelevant information and noise information are inhibited at the same time. And finally, an optimal decision boundary is searched for the small sample classification task by adopting joint training of a prototype network and a support vector machine, so that the accuracy of the small sample classification task is effectively improved.

Description

Technical Field

[0001] Embodiments of the present application relate to the technical field of image classification, and particularly to a few-shot image classification method based on data augmentation and mutual attention. Background Art

[0002] With the rapid development of deep learning technology, the performance of models increasingly depends on the support of large-scale labeled data. However, in actual scenarios, there are many difficulties in obtaining large-scale data. For example, data acquisition is difficult, the cost of sample collection is high in fields such as medicine and remote sensing, samples of certain categories may be extremely scarce, the annotation cost is high, annotating large-scale data requires a large amount of labor and professional knowledge and is time-consuming and laborious, the diversity of data categories is insufficient, the target categories in many tasks change frequently or are uncertain, and it is difficult to provide sufficient training samples for each category. Therefore, traditional deep learning methods often perform poorly in the case of scarce data, while few-shot learning can, to a certain extent, solve these problems by simulating the human's "few-shot learning" ability.

[0003] Few-shot learning shows broad application prospects in multiple fields, including but not limited to character generation, robotics, drug discovery, translation, recommendation systems, etc., and is particularly suitable for scenarios where the cost of data collection is high and the amount of data is limited, such as tasks like image classification, image retrieval, object tracking, gesture recognition, etc. Few-shot learning technology provides a more flexible and efficient learning paradigm for the field of artificial intelligence technology, enabling machine learning systems to better adapt to diverse application requirements. Few-shot learning essentially learns to recognize unknown categories in the case of scarce data and extremely few annotations, and this category is not seen by the model during training.

[0004] In actual few-shot learning tasks, the training data is extremely scarce, and each category usually has only a very small number of labeled samples. This data scarcity brings the following problems: First, data scarcity can lead to overfitting and insufficient features. In the few-shot learning scenario, the lack of training data makes it impossible for the model to effectively learn robust feature representations from limited samples. Because the number of training samples is extremely small, the model is prone to overfitting to the training data. Second, in few-shot learning, the goal of the model is to classify query samples using a small number of support samples. Therefore, modeling the relationship between support samples and query samples is crucial. Summary of the Invention

[0005] In view of this, embodiments of the present application propose a few-shot image classification method based on data augmentation and mutual attention, which can effectively expand few-shot samples, sample the mutual attention mechanism, generate feature representations that are more suitable for the current new class task, improve the accuracy of few-shot classification tasks, and expand the application scope of few-shot classification.

[0006] In a first aspect, an embodiment of the present application proposes a few-shot image classification method based on data augmentation and mutual attention, including the following steps: obtaining a few-shot image set, where there are a total of M categories in the few-shot image set, including A base classes and B new classes among the M categories, each small sample image in each base class is labeled with a class label, and only some small sample images in each new class are labeled with class labels; based on the small sample images of all base classes, constructing several meta-training tasks that conform to the N-way-K-shot paradigm, performing meta-training on a pre-constructed classification model composed of a feature extraction module, a mutual attention module, and an output module, and using the feature extraction module to calculate the prototypes of each base class; based on the small sample images of all new classes, constructing several meta-testing tasks that conform to the N-way-K-shot paradigm, each meta-testing task includes N new classes, and there are K small sample images labeled with class labels in each new class. The small sample images labeled with class labels form a support set, and the remaining small sample images form a query set; using the feature extraction module to extract the feature vectors of each small sample image in the support set and query set of each new class, calculate the prototypes of each new class, and determine the nearest neighbor base class of each new class based on the prototypes of each new class and the prototypes of each base class; performing weighted average reconstruction on the distribution of the support set of the new class based on the prototype of the nearest neighbor base class, and randomly sampling in the reconstructed distribution to obtain an augmented support set; using the mutual attention module to perform a mutual attention mechanism on the feature vectors of the query set and the augmented support set to obtain each query feature of the query set and each support feature of the augmented support set; based on each query feature, each support feature, and the output network, using a prototype loss and a global classification loss on the augmented support set to test the query set.

[0007] Optionally, using the feature extraction module to calculate the prototypes of each base class includes: Using the feature extraction module of the classification model completed by meta-training to extract the feature vectors of each small sample image in each base class, obtaining the feature vectors of each small sample image in each base class; Based on the feature vectors of each small sample image in each base class respectively, calculating the mean and variance of each base class, and the mean and variance of each base class are expressed by the formula: ; ; where represents the total number of small sample images in the th base class, represents the th small sample image in the th base class, represents the mean of the th base class, Denote the variance of the th base class; Based on the feature vectors of each small-sample image of each base class and the mean of each base class, calculate the covariance matrix of each base class. The covariance matrix of each base class is expressed by the formula: ; where, Denote the th covariance matrix of the base class.

[0008] Optionally, use the feature extraction module to extract the feature vectors of each small-sample image in the support set and query set of each new class, and calculate the prototype of each new class, including: Use the feature extraction module of the classification model completed by meta-training to perform feature extraction on each small-sample image in the support set and query set of each new class, and obtain the feature vectors of each small-sample image in the support set and query set of each new class; Based on Tukey's power ladder method, perform data transformation on the feature vectors of each small-sample image to obtain the transformed feature vectors of each small-sample image. The transformed feature vectors of each small-sample image are expressed by the formula: ; where, is a preset hyperparameter for adjusting the correction distribution, restore the original feature vector when Denote the th transformed feature vector of the th small-sample image of the th new class; Based on the transformed feature vectors of each small-sample image of each new class, calculate the prototype of each new class. The prototype of each new class is expressed by the formula: where, Denote the total number of small-sample images of the th new class, Denote the th prototype of the

[0009] Optionally, based on the prototypes of each new class and the prototypes of each base class, determine the nearest neighbor base class of each new class, including: Traverse each new class, calculate the Euclidean distance between the prototype of the current new class and the mean of each base class, and select the first base classes with the closest Euclidean distance as the nearest neighbor base classes of the current new class; Let the set of the nearest neighbor base classes of the th new class be , which is expressed by the formula: ; ; Among them, represents the set of Euclidean distances corresponding to the th new class, The elements in are the Euclidean distances between the th new class and the means of each base class, arranged in ascending order, represents the operator for selecting the top

[0010] Optionally, based on the prototypes of the nearest neighbor base classes, perform weighted average reconstruction on the distribution of the support set of the new class, and perform random sampling in the reconstructed distribution to obtain an augmented support set, including: Use the set to describe the Euclidean distances between the prototype of the th new class and the means of each nearest neighbor base class, which is expressed by the formula: , ; Perform normalization processing on using the softmax function to obtain the weights corresponding to each nearest neighbor base class. The weights corresponding to each nearest neighbor base class are expressed by the formula: ; Among them, represents the weight corresponding to the th nearest neighbor base class of the th new class; Use the weights corresponding to each nearest neighbor base class to weighted average the statistics of the nearest neighbor base classes, and calibrate and reconstruct the distribution of each small sample image in the feature space in the support set of the th new class to obtain the reconstructed distribution. The reconstructed distribution of the support set of the th new class is expressed by the formula: ; ; ; Among them, represents the reconstructed distribution of the support set of the th new class, represents the calibrated mean of the th small sample image in the support set of the th new class, represents the th small sample image in the support set of the The calibrated covariance matrix of a small number of sample images is a preset hyperparameter for reducing the distribution approximation error; Random sampling is performed in each reconstructed distribution to generate m random samples of multivariate normal distributions, obtaining an augmented support set; among them, the support set is expanded from the original number of small sample images to number of small sample images.

[0011] Optionally, using a mutual attention module, the mutual attention mechanism is performed on the feature vectors of the query set and the augmented support set to obtain the query features of the query set and the support features of the augmented support set, including: Calculate the cosine similarity between the feature vectors of the query set and the feature vectors of the augmented support set at each pair of spatial positions to obtain a cross-correlation tensor; Adopt a convolutional matching process to refine the cross-correlation tensor through 4D convolution and matching kernels, and based on the refined cross-correlation tensor, obtain a query attention map and a support attention map; Multiply the feature vectors of the query set by the query attention map and perform a pooling operation to obtain the query features of the query set, and multiply the feature vectors of the augmented support set by the support attention map and perform a pooling operation to obtain the support features of the augmented support set.

[0012] Optionally, calculate the cosine similarity between the feature vectors of the query set and the feature vectors of the augmented support set at each pair of spatial positions to obtain a cross-correlation tensor, which is implemented by the following formula: ; where represents the feature vector of the query set, represents the feature vector of the augmented support set, represents calculating the cosine similarity, represents the cross-correlation tensor; Adopt a convolutional matching process to refine the cross-correlation tensor through 4D convolution and matching kernels, and based on the refined cross-correlation tensor, obtain a query attention map and a support attention map, which is implemented by the following formula: ; ; where represents the refined cross-correlation tensor, and respectively represent the width and height of the refined cross-correlation tensor, is a preset weighted hyperparameter, represents the query attention map, represents the support attention map; Multiply each feature vector of the query set by the query attention map and perform a pooling operation to obtain each query feature of the query set. Multiply each feature vector of the augmented support set by the support attention map and perform a pooling operation to obtain each support feature of the augmented support set. This is achieved through the following formula: ; ; where, represents the query feature, represents the support feature of the augmented support set.

[0013] Optionally, based on each query feature, each support feature, and the output network, use the prototype loss and the global classification loss on the augmented support set, including: Construct the global classification loss based on each query feature and the output network , which is expressed by the formula: ; where, represents the probability that the few-shot feature in the query set is predicted as class ; Construct the prototype loss based on each query feature and each support feature , which is expressed by the formula: ; ; where, represents the prototype of the th class in the support set; Based on and construct the objective loss to test the query set, which is expressed by the formula: , is a preset balancing weight.

[0014] A few-shot image classification method based on data augmentation and mutual attention proposed in this application has the following advantages compared with the existing few-shot image classification methods.

[0015] First, this application transforms the parameterized generation method into a non-parameterized distribution correction generation method. It uses the statistics of the auxiliary dataset base class to reconstruct the distribution of each sample in the support set, and then samples and expands the features of the support set according to the constructed non-parameterized normal distribution. Compared with traditional data augmentation methods, the feature generation of this application avoids the high demand for computing resources by complex models such as traditional Generative Adversarial Networks (GANs).

[0016] Second, the process of support set expansion cannot dynamically capture the information of new class query samples. Therefore, the support samples may have poor adaptability to new class tasks. This application introduces a mutual attention module, which can adaptively highlight the importance of key samples or features in the support samples while suppressing irrelevant or noisy information. Through the dynamic interaction between the support samples and the query samples, the mutual attention mechanism can better mine the correlation between the support samples and the query samples, and generate a feature representation that is more suitable for the current new class task.

[0017] Third, this application uses joint training of prototype loss and global loss, combines the two in a weighted manner, and adjusts the weights in the loss function to achieve the best few-shot image classification effect, thereby improving the accuracy of the few-shot classification task and expanding the application scope of few-shot classification.

[0018] In a second aspect, an embodiment of this application provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is enabled to execute a few-shot image classification method based on data augmentation and mutual attention as described in the first aspect above.

[0019] In a fourth aspect, an embodiment of this application provides a computer-readable storage medium storing a computer program, which when executed by a processor, can implement a few-shot image classification method based on data augmentation and mutual attention as described in the first aspect above.

[0020] It can be understood that the beneficial effects of the above second aspect to the third aspect can refer to the relevant descriptions in the first aspect above, and will not be repeated here. Description of the Drawings

[0021] To more clearly illustrate the embodiments of this application or the technical solutions in the related art, the following will briefly introduce the drawings required for the description of the embodiments of this application or the related technology. Obviously, the drawings in the following description are only some embodiments of this application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0022] Figure 1 It is a flowchart of a few-shot image classification method based on data augmentation and mutual attention provided in an embodiment of the present application; Figure 2 It is a schematic structural diagram of a classification model provided in an embodiment of the present application; Figure 3 It is a schematic structural diagram of a mutual attention module provided in an embodiment of the present application; Figure 4 It is a comparison chart of the results of few-shot image classification of 5-way-1-shot and 5-way-5-shot by the classification model on the mini-ImageNet dataset and the CUB dataset respectively provided in an embodiment of the present application; Figure 5 It is a comparison chart of the results of few-shot image classification of 5-way-1-shot and 5-way-5-shot by the classification model on the tiered-ImageNet dataset and the CIFAR-FS dataset respectively provided in an embodiment of the present application; Figure 6 It is a t-SNE visualization schematic diagram of reconstructing few-shot normal distribution sampling based on auxiliary information provided in an embodiment of the present application; Figure 7 It is a schematic structural diagram of an electronic device provided in another embodiment of the present application. Detailed implementation manners

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below with reference to the accompanying drawings. In various embodiments of the present application, many technical details are proposed to help readers better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions required to be protected by the present application can still be implemented. The following division of various embodiments is only for convenience of description and should not constitute any limitation to the specific implementation manner of the present application. Various embodiments can be combined and cross-referenced with each other on the premise of no conflict.

[0024] In recent years, research teams at home and abroad have proposed some few-shot image classifications based on data augmentation. By performing data augmentation on few-shot samples, the few-shot classification task can be converted into a normal supervised task. The proposed data augmentation methods can be roughly divided into three categories.

[0025] The first category is few-shot image classification based on weak labels. After a large amount of data is collected, assigning artificial labels to this data is a time-consuming and resource-consuming task. How to automatically assign pseudo-labels to these unlabeled samples has become a key issue in current research. This method often uses semi-supervised learning or self-supervised learning strategies, using a small amount of labeled data and a large amount of unlabeled data to train a model, and then using this model to predict the unlabeled data to generate pseudo-labels. Pseudo-labels can not only expand the training set but also improve the model through iteration. Inspired by transfer learning, Wang et al. let the upper structure of the network access a large amount of unlabeled data in the unsupervised meta-training stage, optimized the low-density separator, and decoupled it from specific categories, thus obtaining a more general expression. Wang et al. adopted a concurrent parameter response model to evaluate the adaptability of samples. Through this evaluation, pseudo-labeled data with higher confidence can be gradually selected to enhance the generalization and fitting ability of the regression model, thereby improving the few-shot classification performance. Li et al. studied the problem of label noise propagation and improved the baseline method through a denoising network, which learns the distribution of clean and noisy pseudo-labeled samples through a mixture model. This helps to estimate the confidence values of pseudo-labeled samples and select reliable samples with less noise for iterative optimization of the few-shot classifier.

[0026] The second category is few-shot image classification based on generative models. This method uses generative models, such as GANs or VAEs, etc., to generate new image samples. AFHN explored the use of conditional generative adversarial networks to generate richer recognition features. Duan et al. designed a simple but highly effective WeditGAN, which migrates the model from the source domain to the few-shot target domain by repositioning the latent space and learning the constant ∆w to complete few-shot image generation. Yu et al. proposed few-shot feature enhancement based on the masked autoencoder MFGN. The MFGE encoder, as a feature extractor, can take a set of support samples of the same class as input and add sample numbers, extract the input samples as visual features, mask the visual features of some samples, send the visual features of the unmasked samples and the masked sample numbers to the decoder for reconstruction. After training, the encoder and decoder can be combined to generate more feature embeddings, enhancing the feature embeddings of the support set samples. Hong et al. proposed a novel Delta generative adversarial network, which consists of a reconstruction subnetwork and a generation subnetwork. The reconstruction subnetwork captures the intra-class transformation between pairs of the same class, i.e., delta. The generation subnetwork generates the delta of a specific sample for the input image and combines it with the input image to generate a new image of the same class.

[0027] The third category is the enhancement method based on the feature space. Common techniques of this method include feature interpolation, feature fusion, feature transformation, etc. Laso is an early and representative work. Since each sample has multiple labels, more relevant samples can be generated in the hidden space by defining the intersection and difference sets of the feature space. PFEMed connects the pre-trained general features with specific features to further enhance the semantic information of the FSL features. Chen et al. proposed adding a set of reference images, which consists of many image pairs of the same category. In the embedding space, the features of the reference pairs are added or subtracted. This not only enriches the diversity but also introduces reference features. Recently, Yang et al. studied the hypothesis that when the base classes and new classes are semantically similar, their means and variances can be largely shared. The means and variances of the base classes can correct the data distribution of the new classes. In addition, Xu et al. proposed a framework for decomposing the variance in the dataset, where one represents the within-class variance and the others represent the embeddings of discriminant information. By repeatedly sampling, the within-class variance can be added to the discriminant features. In this way, features are learned to the maximum extent while maintaining a large within-class variance.

[0028] However, there are certain problems with the above methods.

[0029] First, the data augmentation methods proposed so far, which are based on unlabeled data augmentation, often adopt a semi-supervised learning strategy. First, a preliminary model is trained using labeled data, and then the model is used to predict unlabeled data to generate pseudo-labels. Then, the pseudo-labeled data and the labeled data are used together for further training of the model. However, when generating pseudo-labels, incorrect labels may be produced. These incorrect labels will be mixed into the training set, causing the model to learn incorrect information and thus affecting the overall performance. In addition, the training of GAN is usually relatively complex, requiring a large amount of computing resources, and problems such as mode collapse may occur when generating samples.

[0030] Second, since few-shot learning studies classifying query samples using a small number of support samples, the relationship between the support samples and the query samples is crucial. The mutual attention mechanism is used to align the features of the support set and the query set, reducing background noise and highlighting the target area to generate a more discriminative representation. The mutual attention mechanism proposed so far generates a mutual attention map by connecting each position of the query sample and the support sample. However, since the position and size of the object are not fixed, there are often semantic inconsistencies in the feature space. Directly connecting each position to generate a mutual attention map has connection redundancy and will exacerbate the problem of semantic mismatch, resulting in unreliable similarity confidence.

[0031] Third, in few-shot learning, due to the limited number of samples in each category, the distribution of samples cannot fully cover the feature space of the category. This situation directly leads to the fact that the model cannot fully learn the feature diversity of each category during the training phase, and it is difficult to accurately divide the decision boundaries of each category during decision-making.

[0032] To solve the above technical problems, an embodiment of the present application proposes a few-shot image classification method based on data augmentation and mutual attention, which is applied to an electronic device. The electronic device can be a terminal or a server. In this embodiment and the following embodiments, the electronic device is taken as an example of a server for illustration. The implementation details of a few-shot image classification method based on data augmentation and mutual attention proposed in this embodiment are specifically described below. The following content is only implementation details provided for easy understanding and is not necessary for implementing this solution.

[0033] The core idea of this strength is to reconstruct the statistics of the small-sample dataset by using the statistics of the similar auxiliary dataset base class, expand the small-sample features, and then improve the matching performance of the small-sample training samples and test samples through the attention mechanism, which can adaptively highlight the key information related to the classification task and generate a feature representation more suitable for the current new class task. Finally, after adopting the dual optimization loss strategy, the model can provide sufficient decision boundaries between classes to improve the accuracy of the few-shot classification task.

[0034] The specific process of a few-shot image classification method based on data augmentation and mutual attention proposed in this embodiment can be as Figure 1 shown, including: Step 101, obtain a small-sample image set. The small-sample image set contains a total of M categories, where the M categories include A base classes and B new classes. Each small-sample image in each base class is labeled with a class label, and only some small-sample images in each new class are labeled with class labels.

[0035] In specific implementation, the server first needs to obtain a small-sample image set. The small-sample image set in this embodiment follows the typical Few-Shot classification setting. The small-sample image set contains a total of M categories of small-sample images, where the M categories include A base classes and B new classes. Each small-sample image in each base class is labeled with a class label, and only some small-sample images in each new class are labeled with class labels.

[0036] In one example, M is an integer greater than 2, and M = A + B.

[0037] It should be noted that when dealing with the small-sample image classification task in this embodiment, a meta-learning cross-task mechanism is adopted for training and testing. Each task is called a meta-task, and a meta-task includes two main parts: meta-training and meta-testing. In this embodiment, meta-training is implemented based on the small-sample images of each base class in the small-sample image set, and meta-testing is implemented based on the small-sample images of each new class in the small-sample image set.

[0038] Step 102: Based on the small-sample images of all base classes, construct several meta-training tasks that conform to the N-way-K-shot paradigm, perform meta-training on the pre-constructed classification model composed of a feature extraction module, a mutual attention module, and an output module, and use the feature extraction module to calculate the prototypes of each base class.

[0039] In a specific implementation, after the server obtains the small-sample image set, it is necessary to construct several meta-training tasks that conform to the N-way-K-shot paradigm (5-way-5-shot or 5-way-1-shot) based on the small-sample images of all base classes, perform meta-training on the pre-constructed classification model composed of a feature extraction module, a mutual attention module, and an output module, and use the feature extraction module of the classification model completed by meta-training to calculate the prototypes of each base class.

[0040] In one example, the specific composition of the classification model can be as Figure 2 shown.

[0041] In one example, the prototype of a base class can be represented by the mean, variance, and covariance matrix.

[0042] The server uses the feature extraction module of the classification model completed by meta-training to extract features from each small-sample image in each base class, and obtains the feature vectors of each small-sample image in each base class.

[0043] Subsequently, based on the feature vectors of each small-sample image in each base class, calculate the mean and variance of each base class. The mean and variance of each base class are expressed by the formula: ; ; where represents the total number of small-sample images of the th base class, represents the feature vector of the th small-sample image of the th base class, represents the mean of the th base class, represents the variance of the th base class.

[0044] Subsequently, based on the feature vectors of the small-sample images of each base class and the mean of each base class, the covariance matrix of each base class is calculated. The covariance matrix of each base class is expressed by the formula: ; where represents the covariance matrix of the th base class.

[0045] Step 103: Based on the small-sample images of all new classes, construct several meta-test tasks that conform to the N-way-K-shot paradigm. Each meta-test task contains N new classes, and each new class has K small-sample images labeled with class labels. The small-sample images labeled with class labels form the support set, and the remaining small-sample images form the query set.

[0046] In an example, after the server completes all meta-training tasks, a meta-trained classification model can be obtained. Next, meta-testing is required to further improve the performance. The server constructs several meta-test tasks that conform to the N-way-K-shot paradigm (5-way-5-shot or 5-way-1-shot) based on the small-sample images of all new classes. Each meta-test task contains N new classes, and each new class has K small-sample images labeled with class labels. The small-sample images labeled with class labels form the support set, and the remaining small-sample images form the query set.

[0047] Step 104: Use the feature extraction module to extract the feature vectors of the small-sample images in the support set and query set of each new class, calculate the prototype of each new class, and determine the nearest neighbor base class of each new class based on the prototype of each new class and the prototype of each base class.

[0048] In a specific implementation, for each meta-test task, the server needs to use the feature extraction module of the meta-trained classification model to extract the feature vectors of the small-sample images in the support set and query set of each new class, calculate the prototype of each new class, and determine the nearest neighbor base class of each new class based on the prototype of each new class and the prototype of each base class.

[0049] In an example, the server uses the feature extraction module of the meta-trained classification model to perform feature extraction on the small-sample images in the support set and query set of each new class, and obtains the feature vectors of the small-sample images in the support set and query set of each new class.

[0050] Next, the server needs to transform the feature vectors of the small-sample images in the support set and query set of each new class into a form that more conforms to the normal distribution and is closer to the Gaussian distribution. This requires implementation based on Tukey's power ladder. The server performs data transformation on the feature vectors of the small-sample images based on Tukey's power ladder method to obtain the transformed feature vectors of the small-sample images. The transformed feature vectors of the small-sample images are represented by the formula: ; Among them, is a preset hyperparameter for adjusting and correcting the distribution. When restoring the original feature vector, represents the th th transformed feature vector of the small-sample image of the

[0051] After that, the server will calculate the prototypes of each new class based on the transformed feature vectors of the small-sample images of each new class. The prototypes of each new class are represented by the formula: ; Among them, represents the total number of small-sample images of the th new class, represents the

[0052] In one example, when the server determines the nearest neighbor base classes of each new class, it traverses each new class, calculates the Euclidean distance between the prototype of the current new class and the mean of each base class, and selects the first base classes with the closest Euclidean distance as the nearest neighbor base classes of the current new class. Let the set of the nearest neighbor base classes of the th new class be , is represented by the formula: ; ; Among them, represents the set of Euclidean distances corresponding to the th new class, and the elements in are the Euclidean distances between the th new class and the means of each base class arranged in ascending order, represents the operator for selecting the first top elements from

[0053] Step 105: Based on the prototypes of the nearest neighbor base classes, perform weighted average reconstruction on the distribution of the support set of the new class, and perform random sampling in the reconstructed distribution to obtain an augmented support set.

[0054] In specific implementation, after the server determines the nearest neighbor base classes of each new class, it is necessary to perform weighted average reconstruction on the distribution of the support set of the new class based on the prototypes of the nearest neighbor base classes, and perform random sampling in the reconstructed distribution, so as to obtain an augmented support set.

[0055] In one example, for ease of description, the server uses the set to describe the Euclidean distance between the prototype of the th new class and the mean of each nearest neighbor base class, which is expressed by the formula: , .

[0056] The server performs normalization processing on using the softmax function to obtain the weights corresponding to each nearest neighbor base class. The weights corresponding to each nearest neighbor base class are expressed by the formula: ; where represents the weight corresponding to the th nearest neighbor base class of the th new class.

[0057] After determining the weights, reconstruction calibration can be performed. The server uses the weights corresponding to each nearest neighbor base class to weighted average the statistics of the nearest neighbor base class, and calibrates and reconstructs the distribution of each small sample image in the support set of the th new class in the feature space to obtain the reconstructed distribution. The reconstructed distribution of the support set of the th new class is expressed by the formula: ; ; ; where represents the reconstructed distribution of the support set of the th new class, represents the calibrated mean of the th small sample image in the support set of the th new class, represents the calibrated covariance matrix of the th small sample image in the support set of the th new class, and is a preset hyperparameter for reducing the distribution approximation error.

[0058] Finally, the server randomly samples in each reconstructed distribution to generate m random samples of multivariate normal distributions, and then the augmented support set can be obtained, achieving the purpose of augmenting the features of the support set. Among them, the support set is expanded from the original small-sample images (features) to small-sample images (features).

[0059] Step 106: Use the mutual attention module to perform the mutual attention mechanism on the feature vectors of the query set and the augmented support set, and obtain the query features of the query set and the support features of the augmented support set.

[0060] In a specific implementation, after the support set is augmented, the server needs to use the mutual attention module of the classification model completed by meta-training to perform the mutual attention mechanism on the feature vectors of the query set and the augmented support set, and obtain the query features of the query set and the support features of the augmented support set.

[0061] In an example, the specific structure of the mutual attention module can be as Figure 3 shown.

[0062] In an example, through the mutual attention mechanism, the server can adaptively highlight the importance of key samples or features in the support samples while suppressing irrelevant or noisy information. Through the dynamic interaction between the support samples and the query samples, the mutual attention mechanism can better mine the correlation between the support samples and the query samples and generate a feature representation that is more suitable for the current new-class task.

[0063] In an example, the server calculates the cosine similarity between the feature vectors of the query set and the feature vectors of the augmented support set at each pair of spatial positions to obtain a cross-correlation tensor. The cross-correlation tensor is a four-dimensional tensor.

[0064] Calculating the cosine similarity between the feature vectors of the query set and the feature vectors of the augmented support set at each pair of spatial positions to obtain a cross-correlation tensor can be achieved through the following formula: ; where represents the feature vector of the query set, represents the feature vector of the augmented support set, represents the calculation of the cosine similarity, represents the cross-correlation tensor.

[0065] Next, the server will adopt a convolutional matching process to refine the cross - correlation tensor through 4D convolution and a matching kernel. Batch normalization and ReLU activation functions need to be inserted between the convolutions. Subsequently, the server obtains a query attention map and a support attention map based on the refined cross - correlation tensor.

[0066] For samples with large appearance variations in few - shot tasks, the cross - correlation tensor map may contain unreliable similarities, i.e., matching scores, between query samples and support samples. To eliminate these unreliable matches, the cross - correlation tensor needs to be refined.

[0067] The cross - correlation tensor only represents the similarities between each position of the support samples and each position of the query samples in the feature space. Through the refined tensor, the server can generate attention maps, which show the relationship between the query and the support.

[0068] Adopting a convolutional matching process to refine the cross - correlation tensor through 4D convolution and a matching kernel, and obtaining a query attention map and a support attention map based on the refined cross - correlation tensor can be achieved through the following formulas: ; ; Among them, represents the refined cross - correlation tensor, and represent the width and height of the refined cross - correlation tensor respectively, is a preset weighted hyperparameter, represents the query attention map, represents the support attention map.

[0069] Each position of the query sample is compared with all positions of the support image to obtain a weighted matching distribution. The attention value represents the correlation between the query position and the most - matching region on the support image. The attention map will help the model extract features helpful for classification or matching by adjusting the relationship between the query sample and the support sample during the learning process. This mutual - attention mechanism enables the model to effectively classify with few samples.

[0070] Finally, the server needs to multiply each feature vector of the query set by the query attention map and perform a pooling operation to obtain each query feature of the query set, and multiply each feature vector of the augmented support set by the support attention map and perform a pooling operation to obtain each support feature of the augmented support set.

[0071] Multiply each feature vector of the query set by the query attention map and perform a pooling operation to obtain each query feature of the query set. Multiply each feature vector of the augmented support set by the support attention map and perform a pooling operation to obtain each support feature of the augmented support set. This can be achieved through the following formula: ; ; where, represents the query feature, represents the support feature of the augmented support set.

[0072] It should be noted that the sum of the elements of the attention map is equal to 1. Therefore, and can be regarded as a convex combination weighted by and .

[0073] Step 107: Based on each query feature, each support feature, and the output network, use the prototype loss and the global classification loss on the augmented support set to test the query set.

[0074] In a specific implementation, in the meta-test task, the server will finally use the prototype loss and the global classification loss on the augmented support set based on each query feature, each support feature, and the output network to test the query set.

[0075] In an example, the output network is an SVM. The server first constructs the global classification loss , which is expressed by the formula as: ; where, represents the probability that the few-shot feature in the query set is predicted as class .

[0076] Next, the server needs to construct the prototype loss , which is expressed by the formula as: ; ; where, represents the prototype of the th class in the support set.

[0077] Finally, the server constructs the objective loss and based on , so as to test the query set, It is expressed by the formula: , is a preset balance weight.

[0078] In one example, in addition to meta-training and meta-testing, the server also needs to conduct experimental verification on the classification model completed by meta-testing, evaluate the performance, so as to select a classification model that can be deployed to the actual application scenario.

[0079] In this embodiment, a few-shot image classification method based on data augmentation and mutual attention is proposed. Compared with the existing few-shot image classification methods, it has the following advantages.

[0080] First, the present application transforms the parameterized generation method into a non-parametric distribution correction generation method, uses the statistics of the auxiliary dataset base class to reconstruct the distribution of each sample in the support set, and then samples and expands the features of the support set according to the constructed non-parametric normal distribution. Compared with the traditional data augmentation method, the feature generation of the present application avoids the high demand for computing resources of complex models such as the traditional generative adversarial network GAN.

[0081] Second, the process of support set expansion cannot dynamically capture the information of the new class query samples. Therefore, the support samples may have poor adaptability to the new class tasks. The present application introduces a mutual attention module, which can adaptively highlight the importance of key samples or features in the support samples, while suppressing irrelevant or noisy information. Through the dynamic interaction between the support samples and the query samples, the mutual attention mechanism can better mine the correlation between the support samples and the query samples, and generate a feature representation that is more suitable for the current new class task.

[0082] Third, the present application adopts joint training of prototype loss and global loss, combines the two in a weighted manner, and adjusts the weights in the loss function to achieve the best few-shot image classification effect, thereby improving the accuracy of the few-shot classification task and expanding the application scope of few-shot classification.

[0083] The step division of the above various methods is only for clear description. When implemented, they can be combined into one step or some steps can be split into multiple steps. As long as the same logical relationship is included, it is within the protection scope of the present application; adding insignificant modifications or introducing insignificant designs to the algorithm or process, but not changing the core design of its algorithm and process are all within the protection scope of this application.

[0084] In one embodiment, in order to verify the performance of the classification model (hereinafter referred to as ours) finally obtained by the present application, we conducted relevant simulation experiments.

[0085] The simulation experiment was carried out using PyCharm software and Python language on a computer with an Intel(R) Xeon(R) Gold 5222 CPU @ 3.80GHz, 125G of memory, and the Ubuntu 20.04.2 operating system.

[0086] We evaluated the performance of the classification model on the mini-ImageNet dataset, tiered-ImageNet dataset, CUB dataset, and CIFAR-FS dataset. Both the mini-ImageNet dataset and the tiered-ImageNet dataset are constructed based on the ImageNet dataset, containing a rich variety of categories covering various animals and objects. The CUB dataset is a more fine-grained dataset including various birds. Datasets with different granularity levels can have different feature reconstructions for their feature spaces. The CIFAR-FS dataset is built on the basis of the CIFAR-100 dataset, and the categories of the CIFAR-FS dataset cover various animals and objects. We hope to demonstrate the effectiveness and generality of the model on the above four datasets.

[0087] Figure 4 It is a comparison chart of the results of few-shot image classification of 5-way-1-shot and 5-way-5-shot by different models on the mini-ImageNet dataset and the CUB dataset, Figure 5 It is a comparison chart of the results of few-shot image classification of 5-way-1-shot and 5-way-5-shot by different models on the tiered-ImageNet dataset and the CIFAR-FS dataset. As can be seen from Figure 4 and Figure 5 it can be seen that the classification model proposed in this application has good performance.

[0088] In addition, Figure 6 It is a t-SNE visualization schematic diagram of reconstructing few-shot normal distribution sampling based on auxiliary information, Figure 6 The stars in it represent the support set features, the circles represent the generated features, and the triangles represent the query set features. As can be seen from Figure 6 it can be seen that the support set has been effectively expanded.

[0089] Another embodiment of this application proposes an electronic device, and its specific structure is as shown in Figure 7As shown, it includes: at least one processor 201; and a memory 202 communicatively connected to the at least one processor 201; wherein, the memory 202 stores instructions executable by the at least one processor 201, and the instructions are executed by the at least one processor 201 to enable the at least one processor 201 to execute a few-shot image classification method based on data augmentation and mutual attention as described in the above method embodiments.

[0090] Among them, the memory and the processor can be connected by a bus. The bus can include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and the memory together. The bus can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and will not be further described herein. The bus interface is responsible for providing an interface between the bus and the transceiver. The transceiver can be an element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on the transmission medium. The data processed by the processor is transmitted over the wireless medium through the antenna. Further, the antenna also receives data and transmits the data to the processor.

[0091] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interface, voltage regulation, power management, and other control functions. The memory can be used to store data used by the processor when executing operations.

[0092] Another embodiment of the present application proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement a few-shot image classification method based on data augmentation and mutual attention as described in the above method embodiments.

[0093] That is, those skilled in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by instructing relevant hardware through a program. The program is stored in a storage medium, including several instructions to enable a device (such as a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the methods described in the various embodiments of the present application. The foregoing storage medium includes: USB flash drives, mobile hard disks, ROM (Read-Only Memory), RAM (Random Access Memory), magnetic disks, or optical discs and other various media that can store program codes.

[0094] Those of ordinary skill in the art can understand that the above embodiments are specific embodiments for implementing the present application. In actual applications, various changes can be made to them in form and details without departing from the spirit and scope of the present application.

Claims

1. A small sample image classification method based on data enhancement and mutual attention, characterized in that: include: Obtain a small sample image set, which contains M categories in total, including A base categories and B new categories. Each small sample image in each base category is marked with a category label, and only some small sample images in each new category are marked with a category label; Based on small sample images of all base classes, several meta-training tasks that conform to the N-way-K-shot paradigm are constructed to meta-train the pre-built classification model consisting of a feature extraction module, a mutual attention module, and an output module. The prototypes of each base class are calculated using the feature extraction module. Based on the small sample images of all new classes, several meta-test tasks that conform to the N-way-K-shot paradigm are constructed. Each meta-test task contains N new classes. In each new class, there are K small sample images annotated with category labels. The small sample images annotated with category labels constitute the support set, and the remaining small sample images constitute the query set. Using the feature extraction module, extract the feature vectors of each small sample image in the support set and query set of each new class, calculate the prototype of each new class, and determine the nearest neighbor base class of each new class based on the prototype of each new class and the prototype of each base class; Based on the prototype of the nearest neighbor base class, the distribution of the support set of the new class is reconstructed by weighted average, and random sampling is performed in the reconstructed distribution to obtain the expanded support set; Using the mutual attention module, the mutual attention mechanism is performed on the feature vectors of the query set and the expanded support set to obtain the query features of the query set and the support features of the expanded support set; The query set is tested using prototype loss and global classification loss on the augmented support set based on the query features, support features, and the output network.

2. The small sample image classification method based on data enhancement and mutual attention according to claim 1, characterized in that: Using the feature extraction module, the prototypes of each base class are calculated, including: Using the feature extraction module of the classification model completed by meta-training, feature extraction is performed on each small sample image in each base class to obtain a feature vector of each small sample image in each base class; Based on the feature vectors of each small sample image of each base class, the mean and variance of each base class are calculated. The mean and variance of each base class are expressed by the formula: ； ； in, Indicates The total number of small sample images of base classes, Indicates The base class The feature vector of a small sample image, Indicates The mean of the base classes, Indicates The variance of the base classes; Based on the feature vectors of each small sample image of each base class and the mean of each base class, the covariance matrix of each base class is calculated. The covariance matrix of each base class is expressed by the formula: ； in, Indicates The covariance matrix of the base classes.

3. The small sample image classification method based on data enhancement and mutual attention according to claim 2, characterized in that: Using the feature extraction module, extract the feature vectors of each small sample image in the support set and query set of each new class, and calculate the prototype of each new class, including: Using the feature extraction module of the classification model completed by meta-training, feature extraction is performed on each small sample image in the support set and query set of each new class to obtain feature vectors of each small sample image in the support set and query set of each new class; Based on Tukey's power ladder method, the feature vectors of each small sample image are transformed to obtain the transformed feature vectors of each small sample image. The transformed feature vectors of each small sample image are expressed by the formula: ； in, is the preset hyperparameter used to adjust the correction distribution, When the original eigenvector is restored, Indicates A new class The transformed feature vector of a small sample image; Based on the transformed feature vectors of each small sample image of each new class, the prototype of each new class is calculated. The prototype of each new class is expressed by the formula: ； in, Indicates The total number of small sample images of new classes, Indicates A prototype of a new class.

4. The small sample image classification method based on data enhancement and mutual attention according to claim 3, characterized in that: Based on the prototype of each new class and the prototype of each base class, the nearest neighbor base class of each new class is determined, including: Traverse each new class, calculate the Euclidean distance between the prototype of the current new class and the mean of each base class, and select the first one with the closest Euclidean distance. base classes as the nearest neighbor base classes of the current new class; Set up The set of nearest neighbor base classes of a new class is , It is expressed by the formula: ； ； in, Indicates The set of Euclidean distances corresponding to the new classes, The elements in are arranged in ascending order. The Euclidean distance between the new class and the mean of each base class, Indicates from Select the front operator for the top elements.

5. The small sample image classification method based on data enhancement and mutual attention according to claim 4, characterized in that: Based on the prototype of the nearest neighbor base class, the distribution of the support set of the new class is reconstructed by weighted average, and random sampling is performed in the reconstructed distribution to obtain the expanded support set, including: Using Collections Description The Euclidean distance between the prototype of the new class and the mean of each nearest neighbor base class, It is expressed by the formula: , ; right Use the softmax function for normalization to obtain the weights corresponding to each nearest neighbor base class. The weights corresponding to each nearest neighbor base class are expressed by the formula: ； in, Indicates A new class The weights corresponding to the nearest neighbor base classes; Use the weights corresponding to each nearest neighbor base class to weight the average statistics of the nearest neighbor base class, and The distribution of each small sample image in the support set of the new class in the feature space is calibrated and reconstructed to obtain the reconstructed distribution. The reconstructed distribution of the support set of a new class is expressed by the formula: ； ； ； in, Indicates The reconstructed distribution of the support set of the new class, Indicates The first The calibration mean of a small sample image, Shidi The first The calibration covariance matrix of the small sample images, is a preset hyperparameter used to reduce the distribution approximation error; Random sampling is performed in each reconstructed distribution to generate m random samples of multivariate normal distribution to obtain the expanded support set; the support set is expanded from the original A small sample image is expanded to A small sample image.

6. The small sample image classification method based on data enhancement and mutual attention according to claim 1, characterized in that: By using the mutual attention module, the mutual attention mechanism is performed on each feature vector of the query set and the expanded support set to obtain each query feature of the query set and each support feature of the expanded support set, including: Calculate the cosine similarity between each feature vector of the query set and each feature vector of the expanded support set at each pair of spatial positions to obtain the cross-correlation tensor; The convolution matching process is used to refine the cross-correlation tensor through 4D convolution and matching kernels. Based on the refined cross-correlation tensor, the query attention map and the support attention map are obtained. Each feature vector of the query set is multiplied by the query attention map and pooled to obtain each query feature of the query set. Each feature vector of the expanded support set is multiplied by the support attention map and pooled to obtain each support feature of the expanded support set.

7. The small sample image classification method based on data enhancement and mutual attention according to claim 6, characterized in that: The cosine similarity between each feature vector of the query set and each feature vector of the expanded support set at each pair of spatial positions is calculated to obtain the cross-correlation tensor, which is implemented by the following formula: ； in, A feature vector representing the query set, represents the eigenvector of the expanded support set, Indicates the calculation of cosine similarity, represents the cross-correlation tensor; The convolution matching process is adopted to refine the cross-correlation tensor through 4D convolution and matching kernel. Based on the refined cross-correlation tensor, the query attention map and the support attention map are obtained, which are implemented by the following formula: ； ； in, represents the refined cross-correlation tensor, and represent the width and height of the refined cross-correlation tensor, respectively. is the preset weighted hyperparameter, represents the query attention map, Indicates support for attention maps; Each feature vector of the query set is multiplied by the query attention map and pooled to obtain each query feature of the query set. Each feature vector of the expanded support set is multiplied by the support attention map and pooled to obtain each support feature of the expanded support set. This is achieved by the following formula: ； ； in, represents the query features, Represents the support features of the expanded support set.

8. The small sample image classification method based on data enhancement and mutual attention according to claim 7, characterized in that: Based on each query feature, each support feature and the output network, prototype loss and global classification loss are used on the expanded support set, including: Based on each query feature and output network, construct a global classification loss , It is expressed by the formula: ； in, Indicates that the minority sample features in the query set are predicted as categories probability; Based on each query feature and each supporting feature, construct a prototype loss , It is expressed by the formula: ； ； in, Express support for the concentration Prototypes of categories; based on and Constructing target loss , thereby testing the query set, It is expressed by the formula: , is the preset balance weight.

9. An electronic device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; In which, the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a small sample image classification method based on data enhancement and mutual attention as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it is possible to implement a small sample image classification method based on data enhancement and mutual attention as described in any one of claims 1 to 8.

Images