An in-class bias migration-based small sample image classification method

Abstract

The application provides a kind of training method of small sample image classification model based on intra-class bias migration, comprising: S1, obtaining image training set and support set, and with the sample in image training set as input, sample classification as output, the basic image classification model of image encoder and classifier is trained to convergence with image training set to obtain initialized image encoder;Wherein, the image training set includes multiple base classes, each base class has multiple samples with class labels;The support set includes multiple new classes different from the class of base class, each new class has a sample that meets the requirements of small sample task and has a class label;S2, the basic model composed of the initialized image encoder, pre-training language model, selector, multiple bias predictors and classifier is iteratively trained with the image training set and support set until convergence to obtain the small sample image classification model composed of image encoder and classifier.

Description

Technical Field

[0001] This invention relates to the field of computer vision, specifically to the field of image classification within computer vision, and more specifically, to a training method for a few-shot image classification model and a method for classifying few-shot images based on the trained model. Background Technology

[0002] Image classification is one of the most fundamental problems in computer vision, serving as the basis for research on other visual problems. It possesses significant academic value and broad application scenarios. In recent years, deep learning-based image classification methods have achieved remarkable results in fields such as public transportation, autonomous driving, and facial recognition, gradually becoming the mainstream approach for image classification research. However, most traditional methods in existing technologies train image classification models using large amounts of high-quality labeled data. These methods do not consider the difficulties in creating labeled data in real-world scenarios, or the inability to obtain sufficient labeled data. For example, creating labeled data for endangered animals, rare medical cases, and personalized user customization are all very difficult or extremely costly. In these real-world scenarios, deep learning models trained on large amounts of high-quality labeled data using traditional research methods fail to achieve the expected results. Therefore, this has prompted research on image classification to shift towards the field of few-shot learning. It should be noted that in this invention, the categories in a dataset with a large number of labeled samples are called base classes, categories with only a few labeled samples are called new classes, the dataset consisting of multiple new classes with a few labeled samples is called the support set, the unlabeled data for new classes used in testing, and the dataset consisting of multiple categories with a few unlabeled samples used for testing is called the query set.

[0003] Few-shot learning aims to build novel class models with generalization capabilities using a small amount of novel class data and a large amount of base class data. This addresses the image classification problem due to insufficient labeled data for novel classes, thereby expanding the application scenarios of deep learning algorithms and reducing production costs. In few-shot learning, existing research techniques for modeling categories with insufficient sample sizes can be categorized into three main types:

[0004] The first type of method constructs new samples by adding perturbations to the original samples. For example, in reference [1], a deformable network is learned by using the IDeMe-Net method. This network linearly combines the new class samples with the base class samples to obtain difficult samples that still retain discriminative information, thereby promoting the learning of the classification model. The MABAS method proposed in reference [2] uses the support set to learn the initial classifier and generates adversarial samples located at the boundary of the classifier by generating adversarial samples. This is done to update the encoder by increasing the classification margin of the adversarial samples, thereby obtaining a more compact feature space within the class. The core idea of ​​this type of method, represented by references [1] and [2], is to construct varied and difficult samples to expand the labeled data in small sample image categories. However, it does not have a fixed expectation for the specific performance of the perturbated samples in the image space, which leads to problems such as low quality and poor applicability of the expanded labeled data in the new class.

[0005] The second type of method directly utilizes the intra-class changes of the base class containing a large number of samples and labeled data for model learning. For example, the Δ-encoder proposed in reference [3], the DTN (Diversitytransfer network) proposed in reference [4], and the MVT (Meta variance transfer) proposed in reference [5] all learn to transfer the differences between similar sample pairs in the base class to the support set samples of the new class, hoping that the newly generated samples based on the differences between similar sample pairs in the base class will satisfy the same differences as the original samples. However, if the intra-class changes of the base class have poor applicability on the support set samples of the new class, it will directly lead to a huge difference between the samples generated based on the intra-class changes of the base class and the original samples, thus causing the deep learning model trained by this type of method to fail to achieve the expected results.

[0006] The third type of method is dedicated to estimating the complete data distribution of samples in the new class and then sampling a large number of samples from the estimated complete data distribution. For example, in order to estimate the complete data distribution of samples in the new class, the DC method (Distribution calibration) was proposed in reference [6]. This method assumes that the features of all classes follow a Gaussian distribution and uses statistical values ​​such as the mean and covariance matrix of similar base classes to estimate the mean and covariance matrix of the new class respectively. However, since a small number of samples in the new class can only describe limited class information, the new class information estimated by this method has problems such as incomplete estimated data distribution, low accuracy of estimated data, and large error in estimated new class information.

[0007] In summary, most deep learning models trained using existing research techniques for classifying small sample images suffer from problems such as poor information utilization efficiency, low classification accuracy for small sample images, and difficulty in applying them to real-world scenarios.

[0008] List of references:

[0009] [1]Chen Z, Fu Y, Wang YX, Ma L, Liu W, Hebert M.Image deformation meta-networks for one-shot learning[C] / / IEEE Conference on Computer Vision andPattern Recognition.2019:8680-8689.

[0010] [2]Kim J,Kim H,Kim G.Model-agnostic boundary-adversarial sampling fortest-time generalization in few-shot learning[C] / / European Conference onComputer Vision:volume 12346.2020:599-617.

[0011] [3]Schwartz E,Karlinsky L,Shtok J,Harary S,Marder M,Kumar A,Feris RS,Giryes R,Bronstein A M.Delta-encoder: an effective sample synthesis method for few-shot object recognition[C] / / Advances in Neural Information ProcessingSystems.2018:2850-2860.

[0012] [4]Chen M, Fang Y, Wang

[0013] [5]Park S, Han S, Baek J, Kim I, Song J, Lee H, Han J, Hwang S J. Metavariance transfer: Learning to augment from the others [C] / / InternationalConference on Machine Learning: volume 119.2020:7510-7520.

[0014] [6]Yang S, Liu L, Xu M. Free lunch for few-shot learning: Distributioncalibration[C] / / International Conference on Learning Representations.2021.

[0015] [7] Mangla P, Singh M, Sinha A, Kumari N, Balasubramanian VN, Krishnamurthy B. Charting the right manifold: Manifold mixup for few-shot learning[C] / / WACV.2020:2207-2216.

[0016] [8] Gidaris S, Bursuc A, Komodakis N, Pérez P, Cord M.Boosting few-shot visual learning with self-supervision[C] / / IEEE International Conference onComputer Vision.2019:8058-8067. Summary of the Invention

[0017] Therefore, the purpose of this invention is to overcome the shortcomings of the prior art and provide a training method for a few-shot image classification model based on intra-class bias transfer and a few-shot image classification method based on the trained model.

[0018] The objective of this invention is achieved through the following technical solution:

[0019] According to a first aspect of the present invention, a training method for a few-shot image classification model is provided. The method includes: S1, obtaining an image training set and a support set, and using samples in the image training set as input and sample classification as output, training a basic image classification model composed of an image encoder and a classifier using the image training set until convergence to obtain an initialized image encoder; wherein, the image training set includes multiple base classes, each base class having multiple samples with class labels; the support set includes multiple new classes with different classes from the base classes, each new class having samples that meet the requirements of the few-shot task and have class labels; S2, using the image training set and the support set to iteratively train the basic model composed of the initialized image encoder, a pre-trained language model, a selector, multiple bias predictors, and a classifier multiple times until convergence to obtain a few-shot image classification model composed of an image encoder and a classifier, wherein each iteration of training includes: S21, using the pre-trained language model to extract semantic features of each base class and each new class respectively, and using the image encoder to extract visual features of samples of each base class and each new class respectively; S22, using the selector based on the steps in S21... S23. Using the semantic features of each base class and each new class, and the visual features of the samples of each base class and each new class, calculate the similarity between each new class and each base class to determine all similar base classes corresponding to each new class; S24. Using a bias predictor, based on the visual features of the samples of each base class and each new class obtained in step S21, perform bias prediction on the new class and all similar base classes of the new class to their respective actual class centers, and calculate the class prediction centers of the new class and all similar base classes of the new class based on the prediction bias. Each new class corresponds to one bias predictor, and the bias prediction loss is calculated based on the prediction bias and actual bias of each class, and the class center prediction loss is calculated based on the class prediction center and actual class center of each class; S25. Using a classifier, based on the visual features of each new class and all samples of the new class obtained in step S21, obtain the classification prediction results of the samples corresponding to each new class and all similar base classes of the new class, and calculate the classification loss based on the classification prediction results and sample class labels; S26. Update the parameters of the basic model using the bias prediction loss, class center prediction loss, and classification loss.

[0020] In some embodiments of the present invention, the image encoder is a deep learning network.

[0021] In some embodiments of the present invention, the image encoder is one of Alexnet, Vggnet, Resnet, WRN, and WRN-28-10.

[0022] In some embodiments of the present invention, the selector includes a semantic branch network, a visual branch network, and a normalization layer. Step S22 includes: S221, using the semantic branch network in the selector to calculate the semantic similarity between each base class and each new class based on the semantic features of each base class and each new class; S222, using the visual branch network in the selector to calculate the visual similarity between each base class and each new class based on the visual features of the samples of each base class and each new class; S223, using the normalization layer in the selector to calculate the normalized similarity between each new class and each base class based on the semantic similarity obtained in step S221 and the visual similarity obtained in step S222, and determining all similar base classes corresponding to each new class based on a preset similarity threshold.

[0023] In some embodiments of the present invention, in step S221, the semantic similarity between each base class and each new class is calculated as follows:

[0024]

[0025] in, w represents the semantic similarity between the i-th new class and the j-th base class. i w represents the semantic features of the i-th new class. j f represents the semantic features of the j-th base class. s This represents the semantic branching network of the selector.

[0026] In some embodiments of the present invention, in step S222, the visual similarity between each base class and each new class is calculated as follows:

[0027]

[0028] in, This represents the visual similarity between the i-th new class and the j-th base class. Let represent the mean of the visual features of all samples in the i-th new class. f represents the mean visual feature of all samples in the j-th base class. v The visual branch network represents the selector.

[0029] In some embodiments of the present invention, in step S223, the normalized similarity between each new class and each base class is calculated as follows:

[0030]

[0031] Among them, S ij Let represent the normalized similarity between the i-th new class and the j-th base class, U represent the normalization coefficient, and β represent the learning parameters of the selector normalization layer. This represents the visual similarity between the i-th new class and the j-th base class. This represents the semantic similarity between the i-th new class and the j-th base class.

[0032] In some embodiments of the present invention, in step S23, the actual class center of each class in the current round of iterative training is determined as follows: based on the visual features of all samples corresponding to each class in the current round of iterative training, the mean of the visual features corresponding to each class in the current round of iterative training is calculated and used as the actual class center of each class, wherein each class refers to a new class or a base class.

[0033] In some embodiments of the present invention, in step S23, the deviation prediction loss is calculated as follows:

[0034]

[0035]

[0036] Among them, S ij c represents the normalized similarity between the i-th new class and the j-th base class. i This represents the actual class center of the i-th new class in the current iteration of training. This represents the visual feature of the k-th sample in the i-th new class. express Compared with the actual class center c of the new class in the current round of training iterations i The prediction deviation between them Represents the visual features of the k-th sample of the i-th new class. Compared with the actual class center c of the new class in the current round of training iterations i The actual deviation between them.

[0037] In some embodiments of the present invention, in step S23, the category prediction centers of the new class and all similar base classes of the new class are calculated respectively in the following manner:

[0038]

[0039] in, This represents the category prediction center for the i-th new class. Let K represent the visual features of the k-th sample in the i-th new class, where K represents the total number of samples in the i-th new class. express The prediction deviation between the actual class center of the new class in the current round of training iterations.

[0040] In some embodiments of the present invention, in step S23, the category center prediction loss is calculated as follows:

[0041]

[0042] in, c represents the class prediction center of the i-th new class in the current iteration of training. i This represents the actual class center of the i-th new class in the current iteration of training.

[0043] In some embodiments of the present invention, in step S24, the classification loss is calculated as follows:

[0044]

[0045] in, This represents the visual features of the k-th sample in the i-th new class. express The classification probability.

[0046] In some embodiments of the present invention, step S25 includes: updating the image encoder and classifier using classification loss; updating the selector using bias prediction loss; and updating the bias predictor using bias prediction loss, class center prediction loss, and classification loss.

[0047] According to a second aspect of the present invention, an image classification method is provided, the method comprising: T1, acquiring an image to be processed; T2, processing the image to be processed using a few-sample image classification model trained according to the method of the first aspect of the present invention to identify its image category.

[0048] Compared with the prior art, the advantages of the present invention are as follows:

[0049] 1. During the training process, this invention calculates the similarity between the new class and the base classes to select all base classes that are similar to the new class and meet the preset similarity threshold, thereby expanding the number of training samples and enriching the intra-class variations of the new class when the number of samples is insufficient.

[0050] 2. Based on the characteristic that the intra-class deviation distribution of the new class and its similar base classes are similar, this invention proposes a new deviation prediction loss, which enhances the effectiveness of augmented data.

[0051] 3. This invention uses meta-learning to train the deep learning model to be trained end-to-end, which improves the recognition effect of the trained model on image classification of small samples. Attached Figure Description

[0052] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0053] Figure 1 This is a flowchart illustrating a small-sample image classification model according to an embodiment of the present invention.

[0054] Figure 2 This is a schematic diagram of a training method for a few-sample image classification model according to an embodiment of the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the invention is further described in detail below through specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0056] As mentioned in the background section, existing methods for augmenting image categories with annotations in small sample sizes suffer from low quality and poor effectiveness of the augmented annotation data, as well as low accuracy of the trained deep learning models in classifying small sample images. To address these issues, this invention proposes a training scheme for a small sample image classification model using intra-class bias transfer. This scheme introduces bias learning and category center prediction tasks, calculating the intra-class bias of the new class and all its corresponding similar base classes. Category centers are then predicted based on the intra-class bias prediction, followed by classification prediction based on the predicted category centers. The model parameters are updated using bias prediction loss, category center prediction loss, and classification loss. Semantic features are introduced in the process of determining similar base classes for the new class, providing more complete category information compared to judgments based solely on visual features. By introducing bias learning and category center prediction tasks, the intra-class variation of small sample categories can be effectively increased, improving classification accuracy.

[0057] like Figure 1As shown, this invention provides a training method for a few-shot image classification model based on intra-class bias transfer. In summary, the method includes: S1, obtaining an image training set and a support set, and using samples from the image training set as input and sample classification as output, training a basic image classification model composed of an image encoder and a classifier using the image training set until convergence to obtain an initialized image encoder; S2, using the image training set and the support set to iteratively train the basic model composed of the initialized image encoder, a pre-trained language model, a selector, multiple bias predictors, and a classifier until convergence to obtain a few-shot image classification model composed of an image encoder and a classifier, and updating the parameters of the basic model using bias prediction loss, class center prediction loss, and classification loss. The initialization of the image encoder, the pre-trained language model, and the training of the selector and multiple bias predictors are several additional training tasks introduced in this invention. Their purposes are respectively to enhance the encoding capability of the image encoder, obtain the category semantic features of the new class and base classes to calculate the similarity between the new class and the base classes, select all base classes similar to the new class to expand the number of training samples, and calculate the bias prediction loss corresponding to each new class to enhance the effectiveness of the expanded training samples.

[0058] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0059] According to one embodiment of the present invention, such as Figure 2 As shown, the basic model of this invention consists of an image encoder, a pre-trained language model, a selector, multiple bias predictors, and a classifier. During multiple iterations of training the basic model, the same steps are performed in each iteration; therefore, the following description will illustrate the technical solution of this invention using the process of one iteration as an example.

[0060] According to one embodiment of the present invention, the image training set includes multiple base classes, each base class having multiple samples with category labels; the support set includes multiple new classes with categories different from the base classes, each new class having samples that meet the requirements of the few-shot task and have category labels; the image encoder is used to encode the sample images in the new class and the base classes into one-dimensional visual feature vectors; the selector is used to calculate the similarity between the new class and the base classes to select all base classes similar to the new class that meet a preset similarity threshold to expand the number of training samples for the few-shot image classification model; the bias predictor is used to predict the bias of the new class and all similar base classes of the new class to their respective actual category centers, and to calculate the category prediction centers of the new class and all similar base classes of the new class based on the predicted bias, wherein each new class corresponds to one bias predictor; the classifier is used to calculate the classification prediction results of the samples corresponding to the new class and all similar base classes of the new class.

[0061] According to one embodiment of the present invention, steps T1-T5 are executed during each iteration of training, and each step is described in detail below.

[0062] In step T1, a pre-trained language model is used to extract semantic features for each base class and each new class, and an image encoder is used to extract visual features for samples of each base class and each new class. It should be noted that the semantic features extracted in this invention are obtained by inputting the words of the category name into the pre-trained language model to obtain a fixed-length semantic vector. This fixed-length semantic vector is the semantic feature. For example, if the name of a category is "cat," then inputting "cat" into the pre-trained language model will output a fixed-length semantic vector, which is the semantic feature of the cat category.

[0063] In step T2, a selector calculates the similarity between each new class and each base class based on the semantic features of each base class and each new class obtained in step T1, as well as the visual features of samples from each base class and each new class, to determine all similar base classes corresponding to each new class. According to an embodiment of the present invention, the selector includes a semantic branch network, a visual branch network, and a normalization layer. It should be noted that the purpose of introducing semantic features in this invention is that a small sample of a new class cannot reflect the complete information of that new class; the introduced semantic features can supplement the visual features, thereby enabling a more accurate calculation of the similarity between the new class and the base classes. According to an embodiment of the present invention, the similarity between each new class and each base class is calculated as follows, including steps T21-T24, wherein each step is described as follows:

[0064] Step T21: Using the semantic branching network in the selector, based on the semantic features of each base class and each new class, calculate the semantic similarity between each base class and each new class in the following manner:

[0065]

[0066] in, w represents the semantic similarity between the i-th new class and the j-th base class. i w represents the semantic features of the i-th new class. j f represents the semantic features of the j-th base class. s The semantic branching network represents the selector;

[0067] Step T22: Using the visual branch network in the selector, based on the visual features of samples from each base class and each new class, calculate the visual similarity between each base class and each new class as follows:

[0068]

[0069] in, This represents the visual similarity between the i-th new class and the j-th base class. Let represent the mean of the visual features of all samples in the i-th new class. f represents the mean visual feature of all samples in the j-th base class. v Visual branching network representing the selector;

[0070] Step T23: Using the normalization layer in the selector, based on the semantic similarity obtained in step T21 and the visual similarity obtained in step T22, calculate the normalized similarity between each new class and each base class, and determine all similar base classes corresponding to each new class based on a preset similarity threshold. Preferably, the present invention calculates the normalized similarity between each new class and each base class in the following manner:

[0071]

[0072] Among them, S ij Let represent the normalized similarity between the i-th new class and the j-th base class, U represent the normalization coefficient, and β represent the learning parameters of the selector normalization layer. This represents the visual similarity between the i-th new class and the j-th base class. This represents the semantic similarity between the i-th new class and the j-th base class;

[0073] Step T24: After obtaining the normalized similarity between each new class and each base class through steps T21-T23, this invention determines all similar base classes corresponding to each new class based on a preset similarity threshold. Preferably, base classes with a similarity greater than or equal to the similarity threshold with the new class are taken as similar base classes of the new class. The preset similarity threshold can be set according to the actual scenario, and this invention does not impose specific limitations on it.

[0074] In step T3, a bias predictor (each bias predictor corresponds to a new class) is used to predict the bias from the new class and all similar base classes to their respective actual class centers based on the visual features of the samples of each base class and each new class obtained in step T1. The predicted class centers of the new class and all similar base classes are then calculated based on the predicted bias. The actual class center of each class (i.e., the new class and the base class) is determined by calculating the mean of the visual features corresponding to each class in the current iteration of training, based on the visual features of all samples corresponding to each class. This mean of the visual features is the actual class center of each class. According to an embodiment of the present invention, the predicted class centers of the new class and all similar base classes are calculated as follows:

[0075]

[0076] in, This represents the category prediction center for the i-th new class. Let K represent the visual features of the k-th sample in the i-th new class, where K represents the total number of samples in the i-th new class. express The prediction deviation between the actual class center of the new class in the current round of training iterations;

[0077] Furthermore, in this step, the deviation prediction loss is calculated based on the predicted deviation and actual deviation for each category, and the category center prediction loss is calculated based on the predicted category center and actual category center for each category. According to one embodiment of the present invention, the deviation prediction loss is calculated as follows:

[0078]

[0079]

[0080] Among them, S ij c represents the normalized similarity between the i-th new class and the j-th base class. i This represents the actual class center of the i-th new class in the current iteration of training. This represents the visual feature of the k-th sample in the i-th new class. express Compared with the actual class center c of the new class in the current round of training iterations i The prediction deviation between them Represents the visual features of the k-th sample of the i-th new class. Compared with the actual class center c of the new class in the current round of training iterations i The actual deviation between them;

[0081] According to one embodiment of the present invention, the category center prediction loss is calculated as follows:

[0082]

[0083] in, c represents the class prediction center of the i-th new class in the current iteration of training. i This represents the actual class center of the i-th new class in the current iteration of training.

[0084] In step T4, a classifier is used to obtain the classification probability of each new class and all samples of that new class based on the visual features obtained in step T1, as well as the classification probability of samples corresponding to each new class and all similar base classes of that new class. Furthermore, a classification prediction result is obtained. Thus, this invention can calculate the classification loss based on the sample classification prediction result and the sample category label. According to an embodiment of the invention, the classification probability of each new class and samples corresponding to all similar base classes of that new class is calculated as follows:

[0085]

[0086] in, This represents the class prediction center of the i-th new class in the current round of training iterations. Let τ represent the visual features of the k-th sample in the i-th new class, and let τ represent the hyperparameters. The summation formula is ∑ n In This refers to prediction centers of all categories.

[0087] According to one embodiment of the present invention, the classification loss is calculated as follows:

[0088]

[0089] in, This represents the visual features of the k-th sample in the i-th new class. express The classification probability.

[0090] In step T5, the parameters of the basic model are updated using bias prediction loss, class center prediction loss, and classification loss. According to one embodiment of the invention, the classification loss is used to update the image encoder and classifier; the bias prediction loss is used to update the selector; and the class center prediction loss, bias prediction loss, and classification loss are used to update the bias predictor. Preferably, when updating the bias predictor, the bias prediction loss and classification loss are adjusted by L... cls +a*L dt The calculation method updates the deviation predictor. According to one embodiment of the present invention, the value of parameter a is set to 0.5. It should be noted that the value of parameter a can be set according to the actual scenario, and the present invention does not impose a specific limitation on it.

[0091] According to an embodiment of the present invention, in the above training process, the present invention also introduces outer layer optimization and inner layer optimization to train the basic model so that the trained model has a better recognition effect. In the specific training process, the present invention trains the model by first using outer layer optimization and then using inner layer optimization. The outer layer optimization method refers to calculating the category prediction centers of all categories by fixing N bias predictors, and calculating the classification prediction result for each category. Then, the initial values ​​of the selector and bias predictor in the subsequent inner layer optimization are updated according to the classification loss and category center prediction loss corresponding to each category. The inner layer optimization method refers to updating the image encoder, bias predictor, and classifier in the basic model based on the selector and bias predictor initialized by the previous outer layer optimization by calculating the classification loss and category center prediction loss corresponding to each category.

[0092] To demonstrate the recognition performance of the few-shot image classification model trained using the training method proposed in this invention for image categories with insufficient sample size, a simulation experiment was conducted. In the simulation experiment, to avoid repeated training of the image encoder and save time, the inventors trained the image encoder on the miniImageNet dataset using the method described in reference [8], and trained the image encoder again on the CIFAR-FS dataset using the method described in reference [9] to obtain the image encoder used for initialization in this invention. Then, the few-shot image classification model trained by the method of this invention was used to classify query sets of few-shot image categories in 5-way 1-shot (5 image categories, each containing one sample) and 5-way 5-shot (5 image categories, each containing five samples). The experimental results of the classification method (i.e., the DevNet method in Table 1) trained by the method of this invention were compared with the experimental results of the Prototypical Network method, TA-Prototypical Network method, and MPA method. The comparison results are shown in Table 1. It should be noted that the image encoder is a deep learning network, which can be any one of AlexNet, VggNet, ResNet, WRN, and WRN-28-10. This invention does not impose any specific limitations on it.

[0093] As shown in Table 1, in the 5-way 1-shot group, the few-shot image classification model trained using the method of this invention achieved the highest recognition accuracy (60.75%) and the lowest error rate (±0.58%). Similarly, in the 5-way 1-shot group, the few-shot image classification model trained using the method of this invention also achieved the highest recognition accuracy (74.39%) and the lowest error rate (±0.50%). Therefore, the training method for the few-shot image classification model proposed in this invention can improve the recognition accuracy of the trained model and reduce the recognition error rate.

[0094] Table 1

[0095]

[0096] Furthermore, this invention also conducted ablation experiments on the image encoder in a few-sample image classification model trained using the training method of this invention, and compared the experimental results of this method (i.e., the DevNet method in Table 2) with those of the Baseline method and Only method. The experimental results of the method are compared to demonstrate its superior performance. The comparison results are shown in Table 2:

[0097] Table 2

[0098]

[0099] As shown in Table 2, the image encoder trained by this invention has the highest accuracy (72.31% and 87.24% respectively) in both the 5-way 1-shot and 5-way 1-shot groups, and the lowest error rate (±0.68% and ±0.46% respectively). This demonstrates that the image encoder trained by this invention possesses superior encoding capabilities and accurate feature extraction capabilities.

[0100] Compared with the prior art, the advantages of the present invention are as follows:

[0101] 1. During the training process, this invention calculates the similarity between the new class and the base classes to select all base classes that are similar to the new class and meet the preset similarity threshold, thereby expanding the number of training samples and enriching the intra-class variations of the new class when the number of samples is insufficient.

[0102] 2. Based on the characteristic that the intra-class deviation distribution of the new class and its similar base classes are similar, this invention proposes a new deviation prediction loss, which enhances the effectiveness of augmented data.

[0103] 3. This invention uses meta-learning to train the deep learning model to be trained end-to-end, which improves the recognition effect of the trained model on image classification of small samples.

[0104] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required function can be achieved.

[0105] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0106] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can be, for example, including but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0107] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A training method for a few-sample image classification model based on intra-class bias transfer, characterized in that, The method includes: S1. Obtain the image training set and support set, and use the samples in the image training set as input and the sample classification as output. Train the basic image classification model composed of the image encoder and classifier using the image training set until convergence to obtain the initialized image encoder; wherein, the image training set includes multiple base classes, each base class has multiple samples with class labels; the support set includes multiple new classes with different classes from the base classes, each new class has samples that meet the requirements of the few-sample task and have class labels; S2. Using the image training set and support set, iteratively train the basic model consisting of the initialized image encoder, pre-trained language model, selector, multiple bias predictors, and classifier until convergence. The converged image encoder and classifier form a few-sample image classification model, wherein each iteration of training includes: S21. Use a pre-trained language model to extract the semantic features of each base class and each new class respectively, and use an image encoder to extract the visual features of the samples of each base class and each new class respectively. S22. Using a selector, the similarity between each new class and each base class is calculated based on the semantic features of each base class and each new class obtained in step S21 and the visual features of the samples of each base class and each new class to determine all similar base classes corresponding to each new class. S23. Using a bias predictor, based on the visual features of the samples of each base class and each new class obtained in step S21, the bias prediction of the new class and all similar base classes of the new class to their respective actual class centers is performed. Based on the prediction bias, the class prediction centers of the new class and all similar base classes of the new class are calculated. Each new class corresponds to a bias predictor. The bias prediction loss is calculated based on the prediction bias and actual bias of each class. The class center prediction loss is calculated based on the class prediction center and actual class center of each class. S24. Using a classifier, based on the visual features of each new class and all samples of the new class obtained in step S21, we obtain the classification prediction results of each new class and all samples corresponding to the similar base classes of the new class, and calculate the classification loss based on the classification prediction results of the samples and the sample category labels. S25. Update the parameters of the basic model using bias prediction loss, class center prediction loss, and classification loss.

2. The method according to claim 1, characterized in that, The image encoder is a deep learning network.

3. The method according to claim 2, characterized in that, The image encoder is one of AlexNet, VggNet, ResNet, WRN, or WRN-28-10.

4. The method according to claim 1, characterized in that, The selector includes a semantic branch network, a visual branch network, and a normalization layer. Step S22 includes: S221. Using the semantic branching network in the selector, the semantic similarity between each base class and each new class is calculated based on the semantic features of each base class and each new class. S222. Using the visual branch network in the selector, calculate the visual similarity between each base class and each new class based on the visual features of the samples of each base class and each new class; S223. Using the normalization layer in the selector, based on the semantic similarity obtained in step S221 and the visual similarity obtained in step S222, calculate the normalized similarity between each new class and each base class, and determine all similar base classes corresponding to each new class based on the preset similarity threshold.

5. The method according to claim 4, characterized in that, In step S221, the semantic similarity between each base class and each new class is calculated as follows: in, Indicates the first A new category and the first Semantic similarity between base classes Indicates the first The semantic features of a new class Indicates the first Semantic features of each base class This represents the semantic branching network of the selector.

6. The method according to claim 4, characterized in that, In step S222, the visual similarity between each base class and each new class is calculated as follows: in, Indicates the first A new category and the first Visual similarity between base classes Indicates the first The mean of the visual features of all samples in the new class. Indicates the first The mean visual features of all samples in each base class. The visual branch network represents the selector.

7. The method according to claim 6, characterized in that, In step S223, the normalized similarity between each new class and each base class is calculated as follows: in, Indicates the first A new category and the first Normalized similarity between base classes Represents the normalization coefficient. This represents the learning parameters of the selector normalization layer. Indicates the first A new category and the first Visual similarity between base classes Indicates the first A new category and the first Semantic similarity between base classes.

8. The method according to claim 7, characterized in that, In step S23, the actual class center of each class in the current round of training iterations is determined as follows: Based on the visual features of all samples corresponding to each category in the current round of training iterations, the mean of the visual features corresponding to each category in the current round of training iterations is calculated and used as the actual category center of each category, where each category refers to a new class or a base class.

9. The method according to claim 8, characterized in that, In step S23, the deviation prediction loss is calculated as follows: in, Indicates the first A new category and the first Normalized similarity between base classes Indicates the th iteration in the current training round The actual category center of the new class, Indicates the first The first new class Visual features of each sample express Compared with the actual class center of the new class in the current round of training iterations The prediction deviation between them Indicates the first The first new class Visual features of each sample Compared with the actual class center of the new class in the current round of training iterations The actual deviation between them.

10. The method according to claim 8, characterized in that, In step S23, the category prediction centers of the new class and all similar base classes of the new class are calculated as follows: in, Indicates the first A new category prediction center, Indicates the first The first new class Visual features of each sample Indicates the first The total number of samples in each new class. express The prediction deviation between the actual class center of the new class in the current round of training iterations.

11. The method according to claim 10, characterized in that, In step S23, the category center prediction loss is calculated as follows: in, Indicates the th iteration in the current training round A new category prediction center, Indicates the th iteration in the current training round The actual category center of the new class.

12. The method according to claim 10, characterized in that, In step S24, the classification loss is calculated as follows: in, Indicates the first The first new class Visual features of each sample express The classification probability.

13. The method according to claim 12, characterized in that, Step S25 includes: The image encoder and classifier are updated using classification loss; The selector is updated using bias prediction loss; The bias predictor is updated using bias prediction loss, class center prediction loss, and classification loss.

14. An image classification method, characterized in that, The method includes: T1. Obtain the image to be processed; T2. The image to be processed is processed using a few-sample image classification model trained by any one of the methods described in claims 1-13 to identify its image category.

15. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the steps of the method according to any one of claims 1 to 13, 14.

16. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs that, when executed by one or more processors, cause the electronic device to perform the steps of the method as described in any one of claims 1 to 13, 14.

Images