Method and system for few-shot image classification based on generative adversarial network prototype correction

Abstract

The application discloses a few-shot image classification method and system based on generative adversarial network prototype correction, wherein the method comprises the following steps: dividing a data set into a training set, a validation set and a test set, training a feature embedding network and an adversarial network from the training set; extracting sample features by using the feature embedding network, correcting the sample features to obtain pre-corrected class prototypes, inputting noise and the pre-corrected class prototypes into the adversarial network to generate pseudo sample features, obtaining pseudo class prototypes; fusing the pseudo class prototypes and the pre-corrected class prototypes to obtain secondarily corrected class prototypes, performing similarity measurement on the training set samples to obtain classification loss of the samples, and using the classification loss to fine-tune the feature embedding network; verifying the performance of the feature embedding network by using the validation set to obtain an optimal feature embedding network; testing a few-shot task by using the optimal feature embedding network, correcting each class prototype in the test, and classifying query samples based on the corrected prototypes.

Description

TECHNICAL FIELD

[0001] The present document relates to the technical field of computers, and particularly relates to a few-shot image classification method and system based on generative adversarial network prototype correction. BACKGROUND

[0002] In recent years, deep learning technology has been widely applied in the field of computer vision, including visual image processing, target detection, image segmentation, etc. In these application scenarios, a deep learning model with excellent performance often needs a sufficient number of labeled image data for model training and updating. However, in many cases, due to factors such as difficulty in sample collection and high cost of manual labeling, it is impossible to obtain enough labeled images for model training. Therefore, effectively training a deep learning model using a small amount of sample data has become a popular research topic in recent years. This research work is also known as few-shot learning. However, training a deep learning model using a small amount of labeled image data is often challenging. To address this challenge, researchers have proposed a method based on metric learning, which aims to learn a metric space in which samples belonging to the same class are close in distance, and samples belonging to different classes are far in distance. In the method based on metric learning, the prototype network is a commonly used few-shot image classification algorithm, which learns the features of samples in each class and calculates the mean of these features to construct a class prototype. By comparing the Euclidean distance between the unknown sample and each class prototype, the classification of the unknown sample is realized. However, due to the lack of training samples in each class, the class prototype obtained by simply taking the mean may deviate from the true class center, resulting in poor generalization performance of the trained model. To address this problem, researchers often use data augmentation techniques to generate more training samples to improve the generalization ability of the model. Although this method can increase the number of training samples, it cannot ensure the discriminability of the generated samples, and in the augmentation process, noise may be introduced, which will adversely affect the training of the model. SUMMARY

[0003] The purpose of the present application is to provide a few-shot image classification method and system based on generative adversarial network prototype correction, which aims to solve the above-mentioned problems in the prior art.

[0004] The embodiment of the present application provides a few-shot image classification method based on generative adversarial network prototype correction, comprising:

[0005] dividing an image data set into a training set, a validation set and a test set, training a feature embedding network and a generative adversarial network according to the training set;

[0006] The sample features of the training set are extracted by using a feature embedding network, the sample features are corrected in advance to obtain an in-advance corrected class prototype, noise and the in-advance corrected class prototype are input into a generative adversarial network to generate pseudo sample features of each class, and a pseudo class prototype of each class is obtained based on the pseudo sample features of each class;

[0007] The pseudo class prototype and the in-advance corrected class prototype are fused to obtain a twice-corrected class prototype, similarity measurement is performed on query set samples in a training task sampled from the training set by using the twice-corrected class prototype, a classification loss of the training set samples is obtained, the feature embedding network is fine-tuned by using the classification loss, a plurality of few-shot verification tasks are generated based on a verification set, the performance of the fine-tuned feature embedding network is verified by using the few-shot verification tasks, and an optimal feature embedding network is obtained.

[0008] The optimal feature embedding network is used for few-shot task testing, each class prototype in the few-shot task testing is corrected, and classification of query samples in a test task is implemented based on the corrected class prototype.

[0009] An embodiment of the present application provides a few-shot image classification system based on generative adversarial network prototype correction, comprising:

[0010] A network module is configured to divide an image data set into a training set, a verification set and a test set, and obtain a feature embedding network and a generative adversarial network by training according to the training set;

[0011] A correction module is configured to extract sample features of the training set by using the feature embedding network, correct the sample features in advance to obtain an in-advance corrected class prototype, input noise and the in-advance corrected class prototype into the generative adversarial network, generate pseudo sample features of each class, and obtain a pseudo class prototype of each class based on the pseudo sample features of each class;

[0012] An adjustment module is configured to fuse the pseudo class prototype and the in-advance corrected class prototype to obtain a twice-corrected class prototype, perform similarity measurement on query set samples in a training task sampled from the training set by using the twice-corrected class prototype, obtain a classification loss of the training set samples, fine-tune the feature embedding network by using the classification loss, generate a plurality of few-shot verification tasks based on a verification set, verify the performance of the fine-tuned feature embedding network by using the few-shot verification tasks, and obtain an optimal feature embedding network.

[0013] A classification module is configured to use the optimal feature embedding network for few-shot task testing, correct each class prototype in the few-shot task testing, and implement classification of query samples in a test task based on the corrected class prototype.

[0014] The embodiment of the present application also provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer program implements the steps of the few-shot image classification method based on prototype correction of a generative adversarial network when executed by the processor.

[0015] The embodiment of the present application also provides a computer readable storage medium, wherein the computer readable storage medium stores an implementation program of information transmission, and the program implements the steps of the few-shot image classification method based on prototype correction of a generative adversarial network when executed by a processor.

[0016] The embodiment of the present application can have the following beneficial effects: the embodiment of the present application can solve the problems of insufficient training samples, deviation of initial class prototypes from real class centers and noise interference in generated pseudo sample features in few-shot learning, and can make the trained model have better generalization ability and improve the accuracy of the few-shot image classification task. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the one or more embodiments of the present application or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments described in the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0018] Figure 1 is a flow chart of the few-shot image classification method based on prototype correction of a generative adversarial network of the embodiment of the present application;

[0019] Figure 2 is a WGAN module training block diagram of the embodiment of the present application;

[0020] Figure 3 is a model training block diagram of the few-shot image classification method based on prototype correction of a generative adversarial network of the embodiment of the present application;

[0021] Figure 4 is a schematic diagram of the few-shot image classification system based on prototype correction of a generative adversarial network of the embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the person skilled in the art better understand the technical solutions in one or more embodiments of the present specification, the technical solutions in one or more embodiments of the present specification will be described clearly and completely in the following with reference to the drawings in one or more embodiments of the present specification. Obviously, the described embodiments are only a part of the embodiments of the present specification, not all. Based on one or more embodiments of the present specification, all other embodiments obtained by those skilled in the art without creative labor should belong to the protection scope of the present document.

[0023] Method embodiment

[0024] According to an embodiment of the present application, a few-shot image classification method based on generative adversarial network prototype correction is provided, Figure 1 is the flow chart of the few-shot image classification method based on generative adversarial network prototype correction of the embodiment of the present application, as Figure 1 shown, the few-shot image classification method based on generative adversarial network prototype correction according to the embodiment of the present application specifically includes:

[0025] Step S101: divide the image data set into training set, validation set and test set, and train the feature embedding network and the generative adversarial network according to the training set, specifically including:

[0026] Divide the image data set into training set, validation set and test set, train a feature embedding network according to the training set by small batch data sampling method and based on the training loss function shown in formula 1, and then train the Episodic training strategy, according to the adversarial training loss function shown in formula 2 and the discriminative regularization term shown in formula 3, to obtain the final training target of the Wasserstein generative adversarial network (Wasserstein Generative Adversarial Network, WGAN) shown in formula 4:

[0027]

[0028] Wherein, p(y=j|x i ) represents the probability that the sample x i is classified as class j, B represents the number of samples sampled in the training process, L F (φ) represents the loss calculated from a batch of training samples;

[0029]

[0030] Wherein, L GAN is the adversarial training loss, E is the Expected Value (expected value), λ is the hyperparameter, D supx and y represent the samples and their corresponding labels in the support set respectively, Z is the randomly sampled noise, represents the class prototype of the nth class, F is the feature embedding network, x n,k represents the kth sample of the nth class in the support set, K is the number of samples in the class, f n represents the feature of the sample in the class n, D is the discriminator, which aims to distinguish f n is the feature of the real sample, is the feature of the pseudo sample, α ~ U (0, 1), U (0, 1) means symmetric probability distribution;

[0031]

[0032] where L dst is the discriminative regularization term, F is the feature extraction network, represents the jth pseudo sample feature generated by the generator for the nth class, J represents the number of pseudo sample features generated by the generator for each class, F (x n,q ) represents the feature embedding of the query sample in the few-shot task, x n,q represents the qth sample of the nth class in the query set, M is the distance measurement function;

[0033]

[0034] where G is the generator in WGAN, D is the discriminator, L GAN is the adversarial training loss, L dst is the discriminative regularization term, L represents the final training target of WGAN.

[0035] Step S102: extract the sample features of the training set using the feature embedding network, correct the sample features in advance to obtain the pre-corrected class prototype, input the noise and the pre-corrected class prototype into the generative adversarial network, generate pseudo sample features for each class, and obtain pseudo class prototypes for each class based on the pseudo sample features for each class, which specifically includes:

[0036] The sample features of the training set are extracted by using a feature embedding network, the extracted sample features are corrected in advance, the pre-corrected class prototypes are obtained based on the corrected sample features, the noise and the pre-corrected class prototypes are input into the WGAN to generate pseudo sample features of each class, the cosine similarity between the pseudo sample features of each class and the pre-corrected class prototypes is calculated, the weight coefficients of the pseudo sample features of each class are obtained through a softmax function based on the cosine similarity, the weight coefficients are multiplied by the pseudo sample features of each class to obtain weighted pseudo sample features of each class, and the weighted pseudo sample features of each class are summed to obtain a pseudo class prototype of each class;

[0037] wherein the sample features of the training set are extracted by using the feature embedding network according to formula 5 and formula 6:

[0038] f n,k =F(x n,k ) formula 5;

[0039] f n,q =F(x n,q ) formula 6;

[0040] wherein F is the feature embedding network, x n,k and f n,k represent the kth sample of the nth class in the support set and its corresponding sample feature respectively, and x n,q and f n,q represent the qth sample of the nth class in the query set and its corresponding sample feature respectively.

[0041] The sample feature f n,k ∈R C×H×W is linearly transformed by using a 1x1 convolution layer with learnable parameters φ, δ and ε respectively to obtain a new feature map query vector key vector and value vector The dimension is converted to R C×H×W , wherein f n,k represents the kth sample feature of the nth class in the support set, C, H and W represent the number of channels, height and width of the sample feature respectively, and after transposing , matrix multiplication operation is performed with , and the attention map A ∈ R HW×HW is obtained by using a softmax function as shown in formula 8:

[0042]

[0043] A={a 1,1 ,…,a i,j ,…,a HW,HW} ∈ R HW×HWFormula 8;

[0044] Where exp represents the natural exponential function. Represents the transposed query vector The i-th position, Represents the key vector At the j-th position, a i,j This indicates the position of the i-th row and j-th column in attention map A. H and W represent the height and width of the sample feature, respectively, and R represents the dimension.

[0045] value vector Attention map A after transposition T Perform matrix multiplication, and then transform the resulting dimension into the same dimension R as the sample features. C×H×W Then, with sample features f n,k By adding element by element, we obtain the corrected sample feature f′ as shown in Formula 9. n,k :

[0046]

[0047] Where, f′ n,k For the corrected sample features, A T For the transposed attention map, f n,k This represents the feature of the k-th sample in the n-th class of the support set. Its value vector;

[0048] Based on the corrected sample features f′ n,k The pre-corrected category prototype is obtained according to Formula 10:

[0049]

[0050] Where, p n f′ represents the nth pre-corrected class prototype, K represents the number of samples of that class in the support set, and f′ n,k These are the corrected sample features.

[0051] Inputting the noise and pre-corrected class prototypes into WGAN generates pseudo-sample features for each class as shown in Equation 11:

[0052]

[0053] Where Z~N(0,1) represents the noise from random sampling, p n Let G represent the nth pre-corrected class prototype, and G denote the generator in WGAN. Let the j-th pseudo-sample feature generated by category n be represented;

[0054] The cosine similarity between each category pseudo sample feature and the pre-corrected category prototype is calculated, and based on the cosine similarity, the weight coefficient of each category pseudo sample feature is obtained through a softmax function as shown in formula 12:

[0055]

[0056] wherein, cos is the cosine similarity, represents the jth pseudo sample feature generated for the nth category, p n represents the nth pre-corrected category prototype, and N is the number of categories, represents the weight coefficient of the pseudo sample feature , p k represents the kth category prototype;

[0057] The weight coefficient is multiplied by the pseudo sample feature of each category to obtain the weighted pseudo sample feature of each category as shown in formula 13:

[0058]

[0059] wherein, represents the jth pseudo sample feature generated for the nth category, represents the normalized weight coefficient, represents the weight coefficient of the pseudo sample feature , and J represents the number of pseudo sample features generated by the generator for each category, is the weighted pseudo sample feature;

[0060] The weighted pseudo sample feature of each category is summed to obtain the pseudo category prototype of each category as shown in formula 14:

[0061]

[0062] wherein, J represents the number of pseudo sample features generated by the generator for the nth category, represents the jth weighted pseudo sample feature in the nth category, represents the pseudo category prototype of the nth category.

[0063] Step S103: The pseudo category prototype and the pre-corrected category prototype are fused to obtain a second-corrected category prototype. The similarity of the query set samples in the training task sampled from the training set is measured using the second-corrected category prototype, and the classification loss of the training set samples is obtained. The classification loss is used to fine-tune the feature embedding network, and a plurality of few-shot verification tasks are generated based on the verification set. The performance of the fine-tuned feature embedding network is verified using the few-shot verification task, and the feature embedding network with the best effect is obtained.

[0064] Step S104: performing a few-shot task test by using the optimal feature embedding network, correcting each class prototype in the few-shot task test, and implementing classification of a query sample in the test task based on the corrected class prototype.

[0065] The training block diagram of the few-shot image classification method of the embodiment of the present application is shown in the following figure, and the above technical solutions of the embodiment of the present application are described in detail. Figures 2-3 The few-shot image classification method based on prototype correction of the generative adversarial network of the embodiment of the present application specifically includes the following steps:

[0066] Step 1: preparing an image dataset of a current task, dividing the dataset into a training set, a validation set and a test set, randomly sampling N classes from the training set, and then randomly sampling K samples from each of the N classes to form a support set, and additionally, randomly sampling Q samples from each of the remaining samples of the N classes to form a query set, the support set and the query set form a few-shot learning task, which is usually referred to as an N-way, K-shot task;

[0067] Step 2: using the training set, pre-training a feature embedding network by using a mini-batch data sampling method, which is used as a feature extractor of subsequent task samples, and then training a Wasserstein generative adversarial network (Wasserstein Generative Adversarial Network, WGAN) by using an episodic training strategy, to obtain a trained WGAN;

[0068] Step 3: extracting sample image features in the few-shot learning task by using the pre-trained feature embedding network;

[0069] Step 4: using a self-attention module to correct the extracted sample features in advance to obtain more discriminative sample features, and obtaining a pre-corrected class prototype based on the sample features;

[0070] Step 5: inputting random noise and the pre-corrected class prototype into the WGAN to generate pseudo sample features for each class;

[0071] Step 6: calculating the cosine similarity between the pseudo sample features of each class and the pre-corrected class prototype, then obtaining a weight coefficient of the pseudo sample features by using a softmax function, multiplying the weight coefficient and the pseudo sample features, and thus obtaining weighted pseudo sample features;

[0072] Step 7: summing the weighted pseudo sample features in each class to obtain a pseudo class prototype of each class;

[0073] Step 8: fuse the pseudo-class prototype and the pre-corrected class prototype, and perform secondary correction on the pre-corrected prototype;

[0074] Step 9: using the corrected prototype, similarity measurement is performed on the query set sample, the classification loss of the query set sample is obtained, and the feature embedding network is fine-tuned;

[0075] Step 10: in the test stage, each class prototype in the support set in the few-shot learning task is corrected, and classification is realized on the query set sample based on the corrected prototype.

[0076] The training loss function of the feature embedding network in step 2 in the embodiment of the application is:

[0077]

[0078] Wherein, p(y=j|x i ) represents the probability that the sample x i is classified as the class j, B represents the number of samples in a batch during the training process, L F (φ) represents the loss calculated from a batch of training samples, and the loss can be used to update the feature embedding network model F.

[0079] WGAN is an improved GAN model by introducing the Wasserstein distance and optimizing the training process, aiming to solve the training instability and mode collapse problem in the traditional GAN, so as to improve the generation ability of the generator, and the training loss function of the WGAN in the embodiment of the application is divided into two parts, the first part is the adversarial training loss L GAN , and its expression is as follows:

[0080]

[0081] Wherein, D sup is the support set in the few-shot task, x and y respectively represent the sample in the support set and the corresponding label, represents the pseudo sample feature generated by the generator for the class n, Z is the randomly sampled noise, represents the class prototype of the n-th class, K is the number of samples in the class, f n represents the feature of the sample in the class n, D is the discriminator, which attempts to distinguish f n from the real sample feature, from the pseudo sample feature, α~U(0,1), the adversarial training loss L GAN The first two terms approximate the Wasserstein distance, and the third term is the penalty term of the gradient norm.

[0082] ​Furthermore, to ensure the discriminativeness of the generated pseudo-sample features, this embodiment of the invention adds a discriminative regularization term to explicitly encourage a high correlation between the generated pseudo-sample features and the features of query set samples from the same class. This term is used as the second part of the WGAN training loss function, and its expression is as follows:

[0083]

[0084] in, F(x) represents the j-th pseudo-sample feature generated by the generator for class n, J represents the number of pseudo-sample features generated by the generator for each class, and F(x) represents the number of pseudo-sample features generated by the generator for each class. n,q ) represents the feature embedding of the query sample in a few-shot task, and M is the distance metric function, which is Euclidean distance.

[0085] As can be seen from the above formula, the generated pseudo-sample features must have information of the same category as the query sample to ensure the discriminativeness of the pseudo-sample features. The final training objective of WGAN is as follows:

[0086]

[0087] In this embodiment of the invention, step 3 involves using a feature embedding network to extract sample features from a few-shot task, the expression of which is as follows:

[0088] f n,k =F(x) n,k ) Formula 5;

[0089] f n,q =F(x) n,q ) Formula 6;

[0090] Where F is the feature extraction network, x n,k and f n,k Let x represent the k-th sample of the n-th class in the support set and its corresponding sample features, respectively. n,q and f n,q These represent the q-th sample of the n-th category in the query set and its corresponding sample features, respectively.

[0091] In this embodiment of the invention, step 4 involves using a self-attention module to pre-correct the extracted sample features, and based on the pre-corrected sample features, obtaining the pre-corrected category prototype. The steps are as follows:

[0092] First, three 1×1 convolutional layers with learnable parameters φ, δ, and ε are used to process the sample features f. n,k ∈R C ×H×W Performing a linear transformation yields three new feature maps: the query vector. key vector Sum value vector Then their dimensions are converted to R C×H×W , where C, H and W represent the channel number, height and width of the sample feature respectively, and then is transposed and multiplied by , which calculates the similarity of each channel feature between and , establishes the global relationship between features, and finally obtains the attention map A∈R HW×HW through the softmax function, and the expression is:

[0093]

[0094]

[0095] wherein, represents the i-th position of the transposed query vector , represents the j-th position of , and a i,j represents the i-th row and j-th column position in the attention map A, is a scaling factor, and the value vector is multiplied by the transposed attention map A T , and then the dimensions of the result are converted to the same dimensions R C×H×W as the sample feature, finally, it is element-wise added with the sample feature f n,k , to obtain the modified sample feature f′ n,k , and the expression is:

[0096]

[0097] The modified sample feature f′ n,k is fused by category and its average is taken to obtain the pre-modified category prototype, and the expression is:

[0098]

[0099] wherein, p n represents the prototype of the n-th category, and K represents the number of samples of the category in the support set, and in the case of few samples, the pre-modified category prototype is more discriminative than the prototype obtained without the attention module, and can better represent the category to which it belongs;

[0100] In the embodiment of the application, step 5 generates a pseudo sample feature for each category, and the expression is:

[0101]

[0102] wherein Z~N(0, 1) represents a randomly sampled noise, p n represents a category prototype, G represents a generator in the WGAN, represents the jth pseudo sample feature generated for the category n.

[0103] The weight coefficient expression of the pseudo sample feature in step 6 in the embodiment of the application is as follows:

[0104]

[0105] wherein cos is the cosine similarity, N is the number of categories, represents the weight coefficient of the pseudo sample feature , and the weighted pseudo sample feature expression is as follows:

[0106]

[0107] wherein represents the normalized weight coefficient, J represents the number of pseudo sample features generated by the generator for each category, is the weighted pseudo sample feature, and according to the above formula, by assigning a smaller weight to the pseudo sample feature far from the category prototype, the problem that the noise sample is contained in the pseudo sample feature can be effectively solved.

[0108] The pseudo category prototype expression of each category in step 7 in the embodiment of the application is as follows:

[0109]

[0110] wherein J represents the number of pseudo sample features generated by the generator for the category n, represents the jth weighted pseudo sample feature in the category n, represents the pseudo category prototype of the nth category.

[0111] The secondary modification expression of the category prototype in step 8 in the embodiment of the application is as follows:

[0112]

[0113] wherein a and β are hyperparameters, used to control the proportion weight of the prototype modified in advance and the pseudo category prototype, represents the new prototype of the nth category after the secondary modification, and compared with the prototype p n modified in advance, the prototype modified after the secondary modification is more discriminative and can better represent the category to which it belongs.

[0114] The classification loss function and the prediction probability expression of the query set sample in step 9 in the embodiment of the application are as follows:

[0115]

[0116]

[0117] where M represents the metric function, here the Euclidean distance is selected, N is the number of categories, P(y=n|x n,q ) represents the probability of the query sample x n,q being classified as the nth category, D qur is the query set, and L(φ) represents the loss expression of the Euclidean distance between the query sample and the corresponding category prototype, which is used to update the parameters of the feature embedding network F.

[0118] The expression for classifying the query sample in the test set in step 10 in the embodiment of the application is:

[0119]

[0120] where f n,q represents the embedded feature of the query sample, represents the modified category prototype, M represents the metric function, here the Euclidean distance is selected, represents the predicted label of the query sample x n,q .

[0121] To sum up, in the embodiment of the application, firstly, a feature embedding network is pre-trained using the training set by adopting the Mini-batch data sampling method, which is used as a feature extractor for subsequent task samples, then the WGAN is trained by adopting the Episodic training training strategy, to obtain the trained WGAN, then the self-attention module is used to modify the extracted image sample features in advance, to obtain more discriminative sample features and corresponding category prototypes, on this basis, the generator generates pseudo sample features for the corresponding categories based on the category prototypes, then the cosine similarity between each category's pseudo sample features and the pre-modified category prototypes is calculated, and the weight coefficients of the pseudo sample features are obtained through the softmax function, then the weight coefficients are multiplied with the pseudo sample features to obtain weighted pseudo sample features, then the weighted pseudo sample features in each category are summed to obtain the pseudo category prototype of each category, the pseudo category prototype is fused with the corresponding pre-modified category prototype to obtain the twice-modified category prototype, finally, the metric method is adopted to complete the classification of the query sample in the few-shot task by using the modified category prototype.

[0122] The few-shot image classification method based on generative adversarial network prototype correction proposed in this invention uses a training algorithm based on generative adversarial network prototype correction, which can solve the problems of insufficient training samples, deviation of the initial class prototype from the true class center, and the presence of noisy samples in the generated pseudo-sample features in few-shot learning. This method can enable the trained model to have better generalization ability, which is beneficial to improving the accuracy of few-shot image classification tasks.

[0123] System Implementation Examples

[0124] According to embodiments of the present invention, a few-shot image classification system based on generative adversarial network prototype correction is provided. Figure 4 This is a schematic diagram of a few-shot image classification system based on generative adversarial network prototype correction according to an embodiment of the present invention, as shown below. Figure 4 As shown, the few-shot image classification system based on generative adversarial network prototype correction according to an embodiment of the present invention specifically includes:

[0125] Network module 40 is used to divide the image dataset into a training set, a validation set, and a test set, and to train a feature embedding network and a generative adversarial network based on the training set. Specifically, it is used for:

[0126] The image dataset is divided into a training set, a validation set, and a test set. A feature embedding network is trained on the training set using mini-batch data sampling and based on the training loss function shown in Equation 1. Then, using an episodic training strategy, and based on the adversarial training loss function shown in Equation 2 and the discriminative regularization term shown in Equation 3, the final training objective of the Wasserstein Generative Adversarial Network (WGAN) is obtained as shown in Equation 4.

[0127]

[0128] Where p(y=j|x) i ) represents sample x i The probability of being classified as category j, where B represents the number of samples sampled in the batch during training, and L... F (φ) represents the loss calculated from a single batch of training samples;

[0129]

[0130] Among them, L GAN To counteract the training loss, E is the Expected Value, λ is the hyperparameter, and D... sup Let x and y represent the support set in a few-shot task, where x and y represent the samples in the support set and their corresponding labels, respectively. represents the pseudo sample feature generated by the generator for the category n, Z is the randomly sampled noise, represents the category prototype of the nth category, F is the feature embedding network, x n,k represents the kth sample in the nth category in the support set, K is the number of samples in the category, f n represents the feature of the sample in the category n, D is the discriminator, and the purpose is to distinguish f n is the feature of the real sample, is the feature of the pseudo sample, α ~ U (0, 1), U (0, 1) means a symmetric probability distribution;

[0131]

[0132] wherein, L dst is a discriminative regularization term, F is a feature extraction network, represents the jth pseudo sample feature generated by the generator for the category n, J represents the number of pseudo sample features generated by the generator for each category, F (x n,q ) represents the feature embedding of the query sample in the few-shot task, x n,q represents the qth sample of the nth category in the query set, M is a distance measurement function;

[0133]

[0134] wherein, G is the generator in WGAN, D is the discriminator, L GAN is the adversarial training loss, L dst is a discriminative regularization term, L represents the final training target of WGAN;

[0135] The correction module 42 is configured to extract sample features of the training set by using a feature embedding network, correct the sample features in advance to obtain an in-advance corrected category prototype, input the noise and the in-advance corrected category prototype into a generative adversarial network, generate pseudo sample features of each category, and obtain a pseudo category prototype of each category based on the pseudo sample features of each category, and is specifically configured to:

[0136] The sample features of the training set are extracted by using a feature embedding network, the extracted sample features are corrected in advance, the pre-corrected class prototypes are obtained based on the corrected sample features, the noise and the pre-corrected class prototypes are input into the WGAN to generate pseudo sample features of each class, the cosine similarity between the pseudo sample features of each class and the pre-corrected class prototypes is calculated, the weight coefficients of the pseudo sample features of each class are obtained through a softmax function based on the cosine similarity, the weight coefficients are multiplied by the pseudo sample features of each class to obtain weighted pseudo sample features of each class, and the weighted pseudo sample features of each class are summed to obtain a pseudo class prototype of each class;

[0137] wherein the sample features of the training set are extracted by using the feature embedding network according to formula 5 and formula 6:

[0138] f n,k =F(x n,k ) formula 5;

[0139] f n,q =F(x n,q ) formula 6;

[0140] wherein F is the feature embedding network, x n,k and f n,k represent the kth sample of the nth class in the support set and its corresponding sample feature respectively, and x n,q and f n,q represent the qth sample of the nth class in the query set and its corresponding sample feature respectively.

[0141] The sample feature f n,k ∈R C×H×W is linearly transformed by using a 1x1 convolution layer with learnable parameters φ, δ and ε respectively to obtain a new feature map query vector key vector and value vector The dimension is converted to R C×H×W , wherein f n,k represents the kth sample feature of the nth class in the support set, C, H and W represent the number of channels, height and width of the sample feature respectively, and after transposing , matrix multiplication operation is performed with , and the attention map A ∈ R HW×HW is obtained by using a softmax function as shown in formula 8:

[0142]

[0143]

[0144] wherein exp represents a natural exponential function, denotes the transposed query vector the i-th position of denotes the key vector the j-th position of a i,j denotes the i-th row, j-th column position in the attention map A, is a scaling factor, and H and W represent the height and width of the sample feature respectively, and R represents the dimension;

[0145] The value vector is multiplied by the transposed attention map A T , the dimension of the multiplication result is converted to the same dimension R as the sample feature C×H×W , and then added element by element with the sample feature f n,k to obtain the corrected sample feature f′ as shown in equation 9 n,k :

[0146]

[0147] where f′ n,k is the corrected sample feature, A T is the transposed attention map, and f n,k denotes the k-th sample feature of the n-th class in the support set, is its value vector;

[0148] Based on the corrected sample feature f′ n,k , the pre-corrected class prototype is obtained according to equation 10:

[0149]

[0150] where p n represents the n-th pre-corrected class prototype, K represents the number of samples of the class in the support set, and f′ n,k is the corrected sample feature.

[0151] The noise and the pre-corrected class prototype are input into the WGAN to generate the pseudo sample feature of each class as shown in equation 11:

[0152]

[0153] where Z ~ N(0, 1) represents randomly sampled noise, p n represents the n-th pre-corrected class prototype, G represents the generator in the WGAN, denotes the j-th pseudo sample feature generated for class n;

[0154] The cosine similarity between each category pseudo sample feature and the pre-corrected category prototype is calculated, and based on the cosine similarity, the weight coefficient of each category pseudo sample feature is obtained through a softmax function as shown in formula 12:

[0155]

[0156] wherein, cos is the cosine similarity, represents the jth pseudo sample feature generated for the nth category, p n represents the nth pre-corrected category prototype, and N is the number of categories, represents the weight coefficient of the pseudo sample feature , p k represents the kth category prototype;

[0157] The weight coefficient is multiplied by the pseudo sample feature of each category to obtain the weighted pseudo sample feature of each category as shown in formula 13:

[0158]

[0159] wherein, represents the jth pseudo sample feature generated for the nth category, p represents the normalized weight coefficient, represents the weight coefficient of the pseudo sample feature , p is the weighted pseudo sample feature;

[0160] The weighted pseudo sample feature of each category is summed to obtain the pseudo category prototype of each category as shown in formula 14:

[0161]

[0162] wherein, J represents the number of pseudo sample features generated by the generator for the nth category, represents the jth weighted pseudo sample feature in the nth category, p represents the nth pseudo category prototype;

[0163] The adjustment module 44 is configured to fuse the pseudo category prototype and the pre-corrected category prototype to obtain a second-corrected category prototype. The similarity of the query set samples in the training task sampled from the training set is measured through the second-corrected category prototype, and the classification loss of the training set samples is obtained. The classification loss is used to fine-tune the feature embedding network, and a plurality of few-shot verification tasks are generated based on the verification set. The performance of the fine-tuned feature embedding network is verified by using the few-shot verification task, and the feature embedding network with the best effect is obtained.

[0164] The classification module 46 is configured to perform a few-shot task test by using the optimal feature embedding network, correct each class prototype in the few-shot task test, and classify the query sample in the test task based on the corrected class prototype.

[0165] The embodiment of the present application is a system embodiment corresponding to the above-mentioned method embodiment, and the specific operation of each module can be understood with reference to the description of the method embodiment, which will not be repeated here.

[0166] In addition to the above-mentioned module division, as a preferred embodiment, the embodiment of the present application can also adopt the following module division mode:

[0167] 1. A data preprocessing module for reading image data and labels, converting RGB images into feature tensor form;

[0168] 2. A pre-training network module for pre-training a feature embedding network and a WGAN, pre-training the feature embedding network by using a Mini-batch data sampling method and using the training set, using the pre-trained feature embedding network as a feature extractor for subsequent task samples, and training the WGAN by using an Episodic training strategy and using the trained WGAN as a pseudo sample feature generator for subsequent task categories;

[0169] 3. A self-attention module for correcting the extracted image sample features in advance to obtain more discriminative sample features and class prototypes, and generating more discriminative pseudo sample features based on the class prototypes;

[0170] 4. A pseudo sample feature generation module for generating pseudo sample features for each class;

[0171] 5. A weight generation module for assigning different weight coefficients to each generated pseudo sample feature to reduce the interference of noise sample features in the generated pseudo sample features;

[0172] 6. A corrected prototype module for secondary correction of the prototype, and the corrected prototype is more discriminative and more representative of the class to which it belongs than the original prototype in the few-shot scenario;

[0173] 7. A metric module for measuring the similarity between the query sample and the corrected prototype to classify the query sample in the few-shot task.

[0174] In summary, the embodiment of the present application has the following beneficial effects:

[0175] 1. The few-shot image classification method based on the prototype correction of the generative adversarial network provided in the embodiment of the present application can ensure the discriminability of the generated pseudo sample features, and in the embodiment of the present application, the self-attention is used to correct the original image features in advance, compared with the method of generating pseudo sample features directly based on the original image sample features, the method of using self-attention to correct the features in advance can obtain more discriminative prototypes and generate more discriminative pseudo sample features.

[0176] 2. The embodiment of the present application designs a generator discriminability regularization term to explicitly encourage the pseudo sample features generated by the generator to have high correlation with the query set sample features from the same class, thereby ensuring the discriminability of the generated pseudo sample features.

[0177] 3. The embodiment of the present application gives different weight coefficients to each generated pseudo sample feature by considering the similarity relationship between the generated pseudo sample features and the class prototypes, thereby reducing the interference of noise sample features in the generated pseudo sample features.

[0178] 4. The embodiment of the present application realizes secondary correction prototypes by fusing the pseudo class prototypes and the pre-corrected prototypes, and compared with the original prototypes in the few-shot situation, the corrected prototypes are more discriminative and can better represent the class to which they belong.

[0179] 5. The few-shot image classification method based on the prototype correction of the generative adversarial network provided in the embodiment of the present application can further solve the few-shot learning problem caused by insufficient training labeled samples on the basis of the existing technology, and the model obtained by training has better generalization ability, which is conducive to improving the accuracy of the few-shot image classification task.

[0180] Device embodiment one

[0181] The embodiment of the present application provides an electronic device, which comprises a memory, a processor and a computer program stored on the memory and executable on the processor, and the computer program implements the steps described in the method embodiment when executed by the processor.

[0182] Device embodiment two

[0183] The embodiment of the present application provides a computer readable storage medium, and the computer readable storage medium stores an implementation program of information transmission, and the program implements the steps described in the method embodiment when executed by a processor.

[0184] The computer readable storage medium described in the embodiment includes but is not limited to ROM, RAM, magnetic disk or optical disk, etc.

[0185] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions recorded in the above embodiments can be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A few-shot image classification method based on generative adversarial network prototype correction, characterized in that, include: The image dataset is divided into a training set, a validation set, and a test set. A feature embedding network and a generative adversarial network are trained based on the training set. The feature embedding network is used to extract sample features from the training set. The sample features are pre-corrected to obtain pre-corrected category prototypes. The noise and the pre-corrected category prototypes are input into the generative adversarial network to generate pseudo-sample features for each category. Based on the pseudo-sample features of each category, the pseudo-category prototype of each category is obtained. By fusing pseudo-class prototypes and pre-corrected class prototypes, a second-corrected class prototype is obtained. The second-corrected class prototype is then used to measure the similarity of the query set samples in the training task sampled from the training set, and the classification loss of the training set samples is obtained. The classification loss is then used to fine-tune the feature embedding network. Multiple few-shot validation tasks are generated based on the validation set. The performance of the fine-tuned feature embedding network is then verified using the few-shot validation tasks, and the feature embedding network with the best performance is obtained. The optimal feature embedding network is used for few-shot task testing. The prototype of each category in the few-shot task test is corrected, and the classification of query samples in the test task is realized based on the corrected category prototype. Specifically, the process involves using a feature embedding network to extract sample features from the training set, pre-correcting these features to obtain pre-corrected category prototypes, inputting noise and the pre-corrected category prototypes into a generative adversarial network to generate pseudo-sample features for each category, and obtaining pseudo-category prototypes for each category based on these pseudo-sample features. The training set sample features are extracted using a feature embedding network. The extracted sample features are then pre-corrected. Based on the corrected sample features, a pre-corrected category prototype is obtained. The noise and the pre-corrected category prototype are input into WGAN to generate pseudo-sample features for each category. The cosine similarity between the pseudo-sample features of each category and the pre-corrected category prototype is calculated. Based on the cosine similarity, the weight coefficient of the pseudo-sample features of each category is obtained through the softmax function. The weight coefficient is multiplied by the pseudo-sample features of each category to obtain the weighted pseudo-sample features of each category. Finally, the weighted pseudo-sample features of each category are summed to obtain the pseudo-category prototype of each category. Extracting sample features from the training set using feature embedding networks specifically includes: The feature embedding network is used to extract sample features from the training set according to Equations 5 and 6: Official 5; Official 6; in, For feature embedding networks, and They respectively represent the support for the centralization of the first The first category Each sample and its corresponding sample features and They represent the first in the query set. The first category Each sample and its corresponding sample features; The extracted sample features are pre-corrected, and the pre-corrected category prototypes are obtained based on the corrected sample features. Specifically, this includes: Using each with learnable parameters , and A 1×1 convolutional layer for sample features Perform a linear transformation to obtain a new feature map query vector. Key vector Sum value vector Transform the dimension ,in, Indicates support for the centralized first The first category Individual sample features, , and The number, height, and width of the channels represent the sample features, respectively. After transpose and Perform matrix multiplication and obtain the attention map as shown in Equation 8 using the softmax function. : Official 7; Official 8; in, This represents the natural exponential function. Represents the transposed query vector The One location, Represents the key vector The One location, Attention map The Middle Okay, number Column position, Scaling factor and These represent the height and width of the sample feature, respectively. Represents dimensions; value vector Attention map after transposition Perform matrix multiplication and transform the resulting dimension to be the same as the dimension of the sample features. Then, combined with sample features By adding element by element, we obtain the corrected sample features as shown in Formula 9. : Official 9; in, These are the corrected sample features. This is the transposed attention map. Indicates support for the centralized first The first category Individual sample features, Its value vector; Based on the corrected sample features The pre-corrected category prototype is obtained according to Formula 10: Formula 10; in, Representing the A pre-corrected category prototype, This indicates the number of samples supporting that category. These are the corrected sample features; Inputting the noise and pre-corrected class prototypes into WGAN generates pseudo-sample features for each class as shown in Equation 11: Official 11; in, Represents noise from random sampling. Representing the A pre-corrected category prototype, This refers to the generator in WGAN. Represented as category The generated first Features of pseudo-samples; Calculate the cosine similarity between the pseudo-sample features of each category and the pre-corrected category prototypes. Based on the cosine similarity, obtain the weight coefficients of the pseudo-sample features of each category as shown in Equation 12 using the softmax function. Official 12; in, For cosine similarity, Represented as category The generated first Features of pseudo-samples Representing the A pre-corrected category prototype, For the number of categories, Indicating pseudo-sample features The weighting coefficients, Indicates the first One category prototype; Multiplying the weight coefficients by the pseudo-sample features of each category yields the weighted pseudo-sample features of each category as shown in Formula 13: Official 13; in, Represented as category The generated first Features of pseudo-samples This represents the normalized weight coefficients. Indicating pseudo-sample features The weighting coefficients, This indicates the number of pseudo-sample features generated by the generator for each category. These are the weighted pseudo-sample features; Summing the weighted pseudo-sample features for each category yields the pseudo-class prototype for each category, as shown in Formula 14: Official 14; in, Indicates that the generator is a category The number of pseudo-sample features generated Indicate category The Middle Weighted pseudo-sample features Representing the The pseudo-class prototype of the class.

2. The method according to claim 1, characterized in that, The image dataset is divided into a training set, a validation set, and a test set. The feature embedding network and the generative adversarial network are trained based on the training set, specifically including: The image dataset is divided into training, validation, and test sets. A feature embedding network is trained on the training set using mini-batch data sampling and based on the training loss function shown in Equation 1. Then, an episodic training strategy is employed. Based on the adversarial training loss function shown in Equation 2 and the discriminative regularization term shown in Equation 3, the final training objective of the Wasserstein Generative Adversarial Network (WGAN) is obtained, as shown in Equation 4. Official 1; in, Indicates sample Classified as a category The probability, This indicates the number of samples sampled in a batch during training. This represents the loss calculated from a single batch of training samples; Official 2; in, To counteract training losses, For Expected Value, For hyperparameters, For the support set in few-shot tasks, and These represent the samples in the support set and their corresponding labels, respectively. Indicates that the generator is a category Generated pseudo-sample features, For randomly sampled noise, Indicates the first The class's category prototype, For feature embedding networks, Indicates support for the centralized first The first category One sample, The number of samples in this category. Indicate category Characteristics of the medium sample As a discriminator, its purpose is to distinguish... Features of real samples Features of pseudo-samples , , It refers to a symmetric probability distribution; Official 3; in, For discriminative regularization, For feature extraction networks, Indicates that the generator is a category The generated first Features of pseudo-samples This indicates the number of pseudo-sample features generated by the generator for each category. This represents the feature embedding of the query sample in a few-shot task. Indicates the first in the query set The first category One sample, It is a distance metric function; Official 4; in, For generators in WGAN, For discriminator, To counteract training losses, For discriminative regularization, This indicates the final training objective of WGAN.

3. A few-shot image classification system based on generative adversarial network prototype correction, characterized in that, include: The network module is used to divide the image dataset into a training set, a validation set, and a test set, and to train a feature embedding network and a generative adversarial network based on the training set. The correction module is used to extract sample features from the training set using a feature embedding network, pre-correct the sample features to obtain pre-corrected category prototypes, input the noise and pre-corrected category prototypes into a generative adversarial network to generate pseudo-sample features for each category, and obtain pseudo-category prototypes for each category based on the pseudo-sample features for each category. The adjustment module is used to fuse pseudo-class prototypes and pre-corrected class prototypes to obtain a second-corrected class prototype. The second-corrected class prototype is used to measure the similarity of the query set samples in the training task sampled from the training set to obtain the classification loss of the training set samples. The classification loss is used to fine-tune the feature embedding network. Multiple few-shot validation tasks are generated based on the validation set. The performance of the fine-tuned feature embedding network is verified by the few-shot validation tasks to obtain the feature embedding network with the best performance. The classification module is used to perform few-shot task testing using the optimal feature embedding network, correct each category prototype in the few-shot task test, and classify the query samples in the test task based on the corrected category prototype. The correction module is specifically used for: The system utilizes a feature embedding network to extract sample features from the training set. These extracted features are pre-corrected, and pre-corrected class prototypes are derived from these features. Noise and the pre-corrected class prototypes are input into a WGAN to generate pseudo-sample features for each class. The cosine similarity between the pseudo-sample features and the pre-corrected class prototypes for each class is calculated. Based on the cosine similarity, a softmax function is used to obtain the weight coefficients of the pseudo-sample features for each class. These weight coefficients are multiplied by the pseudo-sample features for each class to obtain weighted pseudo-sample features for each class. Finally, the weighted pseudo-sample features for each class are summed to obtain the pseudo-class prototype for each class. Specifically, this is used for: The feature embedding network is used to extract sample features from the training set according to Equations 5 and 6: Official 5; Official 6; in, For feature embedding networks, and They respectively represent the support for the centralization of the first The first category Each sample and its corresponding sample features and They represent the first in the query set. The first category Each sample and its corresponding sample features; Using each with learnable parameters , and A 1×1 convolutional layer for sample features Perform a linear transformation to obtain a new feature map query vector. Key vector Sum value vector Transform the dimension ,in, Indicates support for the centralized first The first category Individual sample features, , and The number, height, and width of the channels represent the sample features, respectively. After transpose and Perform matrix multiplication and obtain the attention map as shown in Equation 8 using the softmax function. : Official 7; Official 8; in, This represents the natural exponential function. Represents the transposed query vector The One location, Represents the key vector The One location, Attention map The Middle Okay, number Column position, Scaling factor and These represent the height and width of the sample feature, respectively. Represents dimensions; value vector Attention map after transposition Perform matrix multiplication and transform the resulting dimension to be the same as the dimension of the sample features. Then, combined with sample features By adding element by element, we obtain the corrected sample features as shown in Formula 9. : Official 9; in, These are the corrected sample features. This is the transposed attention map. Indicates support for the centralized first The first category Individual sample features, Its value vector; Based on the corrected sample features The pre-corrected category prototype is obtained according to Formula 10: Official 10; in, Representing the A pre-corrected category prototype, This indicates the number of samples supporting that category. These are the corrected sample features; Inputting the noise and pre-corrected class prototypes into WGAN generates pseudo-sample features for each class as shown in Equation 11: Official 11; in, Represents noise from random sampling. Representing the A pre-corrected category prototype, This refers to the generator in WGAN. Represented as category The generated first Features of pseudo-samples; Calculate the cosine similarity between the pseudo-sample features of each category and the pre-corrected category prototypes. Based on the cosine similarity, obtain the weight coefficients of the pseudo-sample features of each category as shown in Equation 12 using the softmax function. Official 12; in, For cosine similarity, Represented as category The generated first Features of pseudo-samples Representing the A pre-corrected category prototype, For the number of categories, Indicating pseudo-sample features The weighting coefficients, Indicates the first One category prototype; Multiplying the weight coefficients by the pseudo-sample features of each category yields the weighted pseudo-sample features of each category as shown in Formula 13: Official 13; in, Represented as category The generated first Features of pseudo-samples This represents the normalized weight coefficients. Indicating pseudo-sample features The weighting coefficients, This indicates the number of pseudo-sample features generated by the generator for each category. These are the weighted pseudo-sample features; Summing the weighted pseudo-sample features for each category yields the pseudo-class prototype for each category, as shown in Formula 14: Official 14; in, Indicates that the generator is a category The number of pseudo-sample features generated Indicate category The Middle Weighted pseudo-sample features Representing the The pseudo-class prototype of the class.

4. The system according to claim 3, characterized in that, The network module is specifically used for: The image dataset is divided into training, validation, and test sets. A feature embedding network is trained on the training set using mini-batch data sampling and based on the training loss function shown in Equation 1. Then, an episodic training strategy is employed. Based on the adversarial training loss function shown in Equation 2 and the discriminative regularization term shown in Equation 3, the final training objective of the Wasserstein Generative Adversarial Network (WGAN) is obtained, as shown in Equation 4. Official 1; in, Indicates sample Classified as a category The probability, This indicates the number of samples sampled in a batch during training. This represents the loss calculated from a single batch of training samples; Official 2; in, To counteract training losses, For Expected Value, For hyperparameters, For the support set in few-shot tasks, and These represent the samples in the support set and their corresponding labels, respectively. Indicates that the generator is a category Generated pseudo-sample features, For randomly sampled noise, Indicates the first The class's category prototype, For feature embedding networks, Indicates support for the centralized first The first category One sample, The number of samples in this category. Indicate category Characteristics of the medium sample As a discriminator, its purpose is to distinguish... Features of real samples Features of pseudo-samples , , It refers to a symmetric probability distribution; Official 3; in, For discriminative regularization, For feature extraction networks, Indicates that the generator is a category The generated first Features of pseudo-samples This indicates the number of pseudo-sample features generated by the generator for each category. This represents the feature embedding of the query sample in a few-shot task. Indicates the first in the query set The first category One sample, It is a distance metric function; Official 4; in, For generators in WGAN, For discriminator, To counteract training losses, For discriminative regularization, This indicates the final training objective of WGAN.

5. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the few-shot image classification method based on generative adversarial network prototype correction as described in any one of claims 1 to 2.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an implementation program for information transmission, which, when executed by a processor, implements the steps of the few-shot image classification method based on generative adversarial network prototype correction as described in any one of claims 1 to 2.

Images