A few-shot image classification method based on prototype generation

Abstract

The present invention discloses a few-shot image classification method based on prototype generation. First, the differential distribution among samples is mined from prior class sample pairs; then, a large number of samples with rich differences are generated for each class according to the differential distribution among samples; after that, class-adaptive weight coefficients are generated according to the support samples of each class; finally, the weighted average of all support samples of each class is performed according to the weight coefficient of each class to generate an ideal prototype representation. The present invention effectively solves the problem of inaccurate prototype construction in the few-shot classification task and achieves a good classification effect.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a small-sample image classification method. Background Art

[0002] Image processing technology plays an important role in the industrialization process of the new generation of artificial intelligence. As a basic technology in image processing, image classification lays a foundation for downstream tasks such as object detection and object segmentation. In practical applications, image classification often faces problems such as difficult data acquisition and difficult annotation. How to improve the performance of image classification algorithms under small-sample conditions has important research value. Traditional small-sample image classification algorithms focus on nearest-neighbor classification based on the prototype representation of each category in the feature space. For example, the literature "Snell J, Swersky K, Zemel R. Prototypical networks for few shot learning[C] / / Advances in Neural Information Processing Systems. 2017:4077-4087." maps the support samples of each category to the feature space, and then performs weighted average on all the support samples of this category with equal contribution degree as the prototype representation of this category; finally, classification is performed according to the Euclidean distance between the query sample and the prototypes of all categories. However, in small-sample tasks, the limited support samples of each category are difficult to accurately represent the distribution of this category; in addition, the assumption of equal contribution degree of all support samples results in poor prototype generation. Due to these two aspects of limitations, the prototypes constructed by traditional methods often deviate from the ideal category centroids, thus hindering the generalization ability of these prototype methods. Summary of the Invention

[0003] In order to overcome the deficiencies of the prior art, the present invention provides a small-sample image classification method based on prototype generation. First, the differential distribution between samples is mined from the prior category sample pairs; then, a large number of samples with rich differences are generated for each category according to the differential distribution between samples; after that, a category-adaptive weight coefficient is generated according to the support samples of each category; finally, the weighted average of all the support samples of this category is performed according to the weight coefficient of each category to generate an ideal prototype representation. The present invention effectively solves the problem of inaccurate prototype construction in small-sample classification tasks and achieves a good classification effect.

[0004] The technical solution adopted by the present invention to solve its technical problems includes the following steps:

[0005] Step 1: Small-sample image classification task includes a support set and a query set Q; wherein, the support set It contains N categories, each category has K labeled samples for providing supervision information; the query set Q is composed of some unlabeled samples of these N categories for performance testing; a reference sample set R is constructed for each few-shot image classification task, which is composed of sample pairs of L categories randomly sampled from the training set of the classification task and serves as auxiliary samples for calculating the inter-class difference distribution;

[0006] Step 2: Concatenate the sample pairs from the same category in the reference sample set R:

[0007]

[0008] Among them, represents the feature extractor, r i,1 and r i,2 respectively represent two samples of the i-th category in the reference sample set, means concatenating the two feature vectors to verify the feature dimension, represents the vector obtained after concatenating the sample pairs of the i-th category;

[0009] Step 3: Input into the difference distribution generator for inferring the difference distribution between samples; assume that the difference distribution between samples follows a Gaussian distribution The distribution parameters are:

[0010]

[0011] Among them, h η represents the difference distribution generator, L represents the total number of categories in the reference sample set, μ and σ 2 respectively represent the mean and variance of the difference distribution;

[0012] Step 4: Generate samples for each category in the few-shot image classification task;

[0013] For the n-th category, first randomly sample M sample difference descriptors Z n ={z n,i |i = 1, 2,..., M} based on the inter-class difference distribution; then, generate samples according to the difference descriptor z n,i and the average value s n of all support samples of this category:

[0014]

[0015] Among them, "·" represents dot product;

[0016] In this way, M samples are generated for each category;

[0017] Step 5: Generate weight coefficients for all samples of each category according to the supervision information provided by the support samples of each category;

[0018] For the small sample image classification task in the nth category, first input the average value s n of the support samples of this category into the encoder as prior information for inferring the class-adaptive weight distribution Then randomly sample the latent variable from this distribution, and finally use the decoder to map the latent variable to the weight coefficient W n ={w n,i |i = 1, 2, …, M + 1}; The whole process is described as:

[0019]

[0020] where, g φ and d θ represent the encoder and decoder in the weight generator respectively, and z n represents the latent variable sampled from the weight distribution ;

[0021] Step 6: Perform the final generation according to the samples generated for each category and the weight coefficients corresponding to each sample:

[0022]

[0023] where, is the ith sample generated for category n according to the difference distribution, w n,i represents the weight corresponding to the ith generated sample of the nth category, s n and w n,0 represent the average value of the original support samples in the nth category and their corresponding weights respectively, and p n represents the prototype generated by category n;

[0024] The prototypes P = {p |n = 1, 2, … N} of all categories in the small sample task are obtained by formula (5); n ;

[0025] Step 7: Perform small sample image classification according to the generated prototypes;

[0026] For each sample in the query set Q, calculate the cosine similarity between it and all category prototypes in this task; then, take the category corresponding to the prototype with the highest similarity score as the predicted category of this query sample; finally, calculate the cross-entropy loss function according to the predicted category and the true label:

[0027]

[0028] Among them, CE(·) represents the cross-entropy loss function, and y q represent the predicted label and the true label of the query sample respectively;

[0029] Step 8: End-to-end meta-learning training;

[0030] Perform Kullback–Leibler divergence regularization constraint on the distribution in the weight generator:

[0031]

[0032] where D KL represents the Kullback–Leibler divergence, represents the standard normal distribution, represents the weight distribution corresponding to class n, represents the divergence regularization loss;

[0033] Perform regularization constraint on the generated samples:

[0034]

[0035]

[0036]

[0037] where d intra represents the intra-class distance, and d inter represents the inter-class distance, represents the sample regularization loss;

[0038] Finally, the objective function is

[0039]

[0040] Perform end-to-end training using the stochastic gradient descent method according to the objective function.

[0041] Preferably, the feature extractor is composed of a ResNet-12 network, including four convolutional blocks, with the number of channels being 64, 128, 256, and 1024 respectively, and the output feature dimension being 1024.

[0042] Preferably, the differential distribution generator is composed of two fully connected layers, with an input dimension of 2048 and an output dimension of 2048. It is non-linearly activated through a LeakReLU activation function with a coefficient of 0.2 and is configured with a Dropout layer with a coefficient of 0.5 to prevent overfitting.

[0043] Preferably, both the encoder and the decoder in the weight generator are composed of a single fully-connected layer. Among them, the input dimension of the encoder is 1024, and the output dimension is 512. It is non-linearly activated by a LeakReLU activation function with a coefficient of 0.2 and configured with a Dropout layer with a coefficient of 0.5 to prevent overfitting. The input dimension of the decoder is 256, and the output dimension is 65. It is non-linearly activated by a LeakReLU activation function with a coefficient of 0.2 and configured with a Dropout layer with a coefficient of 0.5 to prevent overfitting.

[0044] The beneficial effects of the present invention are as follows:

[0045] The present invention models the differential distribution between samples, generates a large number of samples with intra-class differences, generates class-adaptive weight coefficients according to the supervision information of each category, and performs weighted averaging on all samples according to the weight coefficients to generate an ideal prototype. The present invention effectively solves the problem of inaccurate prototype construction in small-sample classification tasks and achieves a good classification effect. Detailed implementation manners

[0046] The present invention will be further described below in conjunction with embodiments.

[0047] A small-sample image classification method based on prototype generation includes the following steps:

[0048] Step 1: For a standard N-way K-shot small-sample image classification task It includes a support set and a query set Q. Among them, the support set includes N categories, and each category has K labeled samples for providing supervision information. The query set Q is composed of some unlabeled samples of these N categories for performance testing. A reference sample set R is constructed for each small-sample image classification task, which is composed of pairs of samples of L categories randomly sampled from the training set and used as auxiliary samples for calculating the inter-class differential distribution.

[0049] Step 2: Concatenate the sample pairs from the same category in the reference sample set R:

[0050]

[0051] Among them, represents the feature extractor, r i,1 and r i,2 respectively represent the two samples of the i-th category in the reference sample set, represents concatenating the two feature vectors to verify the feature dimension, represents the vector obtained after concatenating the sample pairs of the i-th category;

[0052] The feature extractor is composed of a ResNet-12 network, including four convolutional blocks with the number of channels being 64, 128, 256, and 1024 respectively, and the output feature dimension being 1024;

[0053] Step 3: Input into the difference distribution generator for inferring the difference distribution among samples; assume that the difference distribution among samples follows a Gaussian distribution The distribution parameters are:

[0054]

[0055] where h η represents the difference distribution generator, L represents the total number of categories in the reference sample set, and μ and σ 2 represent the mean and variance of the difference distribution respectively;

[0056] The difference distribution generator is composed of two fully connected layers, with an input dimension of 2048 and an output dimension of 2048. It is non-linearly activated through a LeakyReLU activation function with a coefficient of 0.2 and configured with a Dropout layer with a coefficient of 0.5 to prevent overfitting;

[0057] Step 4: Generate samples for each category in the few-shot image classification task according to the difference distribution;

[0058] For the nth category, first randomly sample M sample difference descriptors Z n ={z n,i |i = 1, 2,..., M} based on the inter-class difference distribution; then, generate samples according to the difference descriptor z n,i and the average value s n of all support samples of this category:

[0059]

[0060] where "·" represents dot product;

[0061] In this way, M samples are generated for each category;

[0062] Step 5: Generate weight coefficients for all samples of each category according to the supervision information provided by the support samples of each category;

[0063] For the nth category in the few-shot image classification task , first take the average value s n of the support samples of this category as prior information and input it into the encoder for inferring the class-adaptive weight distribution Then, randomly sample the latent variables from this distribution, and finally use the decoder to map the latent variables to the weight coefficients W n ={w n,i | i = 1, 2, …, M + 1}; The whole process is described as follows:

[0064]

[0065] where, g φ and d θ represent the encoder and decoder in the weight generator respectively, z n represents the latent variable sampled from the weight distribution ;

[0066] Both the encoder and decoder in the weight generator are composed of a single fully connected layer. Among them, the input dimension of the encoder is 1024, the output dimension is 512, and it is non-linearly activated by the LeakReLU activation function with a coefficient of 0.2 and configured with a Dropout layer with a coefficient of 0.5 to prevent overfitting; the input dimension of the decoder is 256, the output dimension is 65, and it is non-linearly activated by the LeakReLU activation function with a coefficient of 0.2 and configured with a Dropout layer with a coefficient of 0.5 to prevent overfitting;

[0067] Step 6: Perform the final generation according to the samples generated for each category and the weight coefficients corresponding to each sample:

[0068]

[0069] where, is the i-th sample generated for category n according to the difference distribution, w n,i represents the weight corresponding to the i-th generated sample in the n-th category, s n and w n,0 represent the average value of the original support samples in the n-th category and their corresponding weights respectively, p n represents the prototype generated for category n;

[0070] The prototypes of all categories in the few-shot task are obtained by formula (5) as P = {p n | n = 1, 2, … N};

[0071] Step 7: Perform few-shot image classification according to the generated prototypes;

[0072] For each sample in the query set Q, calculate the cosine similarity between it and all category prototypes in this task; then, take the category corresponding to the prototype with the highest similarity score as the predicted category of this query sample; finally, calculate the cross-entropy loss function according to the predicted category and the true label:

[0073]

[0074] Among them, CE(·) represents the cross-entropy loss function, and y q represent the predicted label and the true label of the query sample respectively;

[0075] Step 8: End-to-end meta-learning training;

[0076] To avoid model collapse, perform Kullback–Leibler divergence regularization constraint on the distribution in the weight generator to encourage the weight distribution of inference to be as close as possible to the standard normal distribution, that is:

[0077]

[0078] Among them, D KL represents the Kullback–Leibler divergence, represents the standard normal distribution, represents the weight distribution corresponding to class n, represents the divergence regularization loss;

[0079] To avoid inter-class confusion in the generated samples, perform regularization constraint on the generated samples to encourage the samples of the same class to be as close as possible and the samples of different classes to be as separated as possible. That is:

[0080]

[0081]

[0082]

[0083] Among them, d intra represents the intra-class distance, d inter represents the inter-class distance, represents the sample regularization loss;

[0084] Finally, the objective function is

[0085]

[0086] Perform end-to-end training using the stochastic gradient descent method according to the objective function. Specific embodiments:

[0088] 1. Data preprocessing

[0089] For the standard N-way K-shot few-shot image classification task contains a support set and a query set Q. Among them, the support set contains N categories, and each category has K labeled samples to provide supervision information; the query set is composed of some unlabeled samples of these N categories for performance testing. In addition, the present invention constructs a reference sample set R for each few-shot task. The reference sample set is composed of sample pairs of L categories randomly sampled from the training set and serves as auxiliary samples for calculating the inter-class difference distribution.

[0090] 2. Sample pair concatenation

[0091] Concatenate the sample pairs from the same category in the reference sample set:

[0092]

[0093] Among them, represents the feature extractor, r i,1 and r i,2 respectively represent the two samples of the i-th category in the reference sample set, represents the concatenation of two feature vectors to verify the feature dimension, represents the vector obtained after concatenating the sample pairs of the i-th category.

[0094] 3. Difference distribution modeling

[0095] Input into the difference distribution generator to infer the difference distribution between samples. The present invention assumes that the difference distribution between samples follows a Gaussian distribution The distribution parameters are:

[0096]

[0097] Among them, h η represents the difference distribution generator, L represents the total number of categories in the reference sample set, and μ and σ 2 respectively represent the mean and variance of the difference distribution.

[0098] 4. Generate samples

[0099] Generate samples for each category in the few-shot task according to the difference distribution. For the n-th category in the few-shot task the present invention first randomly samples M sample difference descriptors Z n ={z n,i |i = 1, 2,..., M} based on the inter-class difference distribution; then, sample generation is performed according to the difference descriptor z n,i and the average value s n of all support samples of this category:

[0100]

[0101] Among them, "·" represents the dot product. In this way, M samples are generated for each category.

[0102] 5. Generate weight coefficients

[0103] Generate weight coefficients for all samples of each category according to the supervision information provided by the support samples of each category. For the small sample task in the nth category, first take the average value s of the support samples of this category n as the prior information and input it into the encoder to infer the class-adaptive weight distribution Then randomly sample the latent variable from this distribution, and finally use the decoder to map the latent variable to the weight coefficient W n ={w n,i |i = 1, 2, …, M + 1}. The whole process can be described as:

[0104]

[0105] Among them, g φ and d θ respectively represent the encoder and decoder in the weight generator, and z n represents the latent variable sampled from the weight distribution .

[0106] 6. Generate prototypes

[0107] Perform the final generation according to the samples generated for each category and the weight coefficients corresponding to each sample:

[0108]

[0109] Among them, is the i-th sample generated for category n according to the difference distribution, and w n,i represents the weight corresponding to the i-th generated sample in the nth category, s n and w n,0 respectively represent the average value of the original support samples in the nth category and their corresponding weights, and p n represents the prototype generated by category n.

[0110] According to formula (5), the prototypes P = {p |n = 1, 2, … N} of all categories in the small sample task can be obtained. n

[0111] 7. Small sample classification

[0112] Perform few-shot image classification based on the generated prototypes. For each sample in the query set Q, calculate its cosine similarity with all class prototypes in the task. Then, take the class corresponding to the prototype with the highest similarity score as the predicted class of the query sample. Finally, calculate the cross-entropy loss function based on the predicted class and the true label:

[0113]

[0114] where CE(·) represents the cross-entropy loss function, and y q represent the predicted label and the true label of the query sample, respectively.

[0115] 8. End-to-end training

[0116] To avoid model collapse, the present invention performs Kullback–Leibler divergence regularization constraint on the distribution in the weight generator to encourage the weight distribution of inference to be as close as possible to the standard normal distribution, that is:

[0117]

[0118] where D KL represents the Kullback–Leibler divergence, represents the standard normal distribution, represents the weight distribution corresponding to class n, represents the divergence regularization loss.

[0119] In addition, to avoid class confusion in the generated samples, the present invention performs regularization constraint on the generated samples to encourage the samples of the same class to be as close as possible and the samples of different classes to be as separated as possible. That is:

[0120]

[0121]

[0122]

[0123] where N is the total number of classes in the few-shot task T, M is the number of samples generated for each class, p n represents the prototype generated for class n, represents the i-th generated sample corresponding to the n-th class, d intra represents the intra-class distance, d inter represents the inter-class distance, represents the sample regularization loss.

[0124] Finally, the objective function is:

[0125]

[0126] Among them represents the cross-entropy loss, represents the divergence regularization loss, represents the sample regularization loss. End-to-end training is performed using the stochastic gradient descent method according to the objective function.

[0127] In the few-shot image classification task, the present invention can obtain better classification accuracy. Taking the benchmark dataset miniImageNet for the few-shot classification task as an example, in the 5-way 1-shot and 5-way 5-shot tasks, the accuracy rates of the method of the present invention reach 62.49% and 77.12% respectively. Compared with the prototype network method under the same feature extractor, it is improved by 2.31% and 1.66% respectively.

Claims

1. A few-shot image classification method based on prototype generation, characterized in that It includes the following steps: Step 1: Few-shot image classification task includes a support set and a query set Q; among them, the support set includes N categories, and each category has K labeled samples for providing supervision information; the query set Q is composed of some unlabeled samples of these N categories for performance testing; construct a reference sample set R for each few-shot image classification task, which is composed of samples of L categories randomly sampled from the training set of the classification task as auxiliary samples for calculating the inter-class difference distribution; Step 2: Concatenate the sample pairs from the same category in the reference sample set R: Among them, represents the feature extractor, r i,1 and r i,2 respectively represent two samples of the i-th category in the reference sample set. represents the concatenation of two feature vectors to verify the feature dimension. represents the vector obtained after concatenating the sample pairs of the i-th category. Step 3: Input into the differential distribution generator for inferring the differential distribution among samples; assume that the differential distribution among samples follows a Gaussian distribution The distribution parameters are: Among them, h η represents the differential distribution generator, L represents the total number of categories in the reference sample set, and μ and σ 2 represent the mean and variance of the differential distribution respectively; Step 4: Generate samples for each category in the few-shot image classification task according to the differential distribution; For the n-th category, first randomly sample M inter-class difference descriptors Z based on the inter-class difference distribution n ={z n,i | i = 1, 2, …, M}; then, according to the difference descriptor z n,i and the mean value s of all support samples of this category n perform sample generation: where, "·" represents dot product; In this way, M samples are generated for each category; Step 5: Generate weight coefficients for all samples of each category according to the supervision information provided by the support samples of each category; For the n-th category in the small-sample image classification task First, the average value s of the support samples of this category n is input into the encoder as prior information for inferring the class-adaptive weight distribution Then, the latent variables are randomly sampled from this distribution, and finally the decoder is used to map the latent variables to the weight coefficients W n ={w n,i |i = 1, 2, …, M + 1}; The whole process is described as: Among them, g φ and d θ respectively represent the encoder and decoder in the weight generator, and z n represents the latent variable sampled from the weight distribution ; Step 6: Perform final generation according to the samples generated for each category and the weight coefficients corresponding to each sample; Among them, is the i-th sample generated for class n according to the differential distribution, w n,i represents the weight corresponding to the i-th generated sample of the n-th class, s n and w n,0 respectively represent the average value of the original support samples in the n-th class and their corresponding weights, p n represents the prototype generated for class n; Find the prototypes of all categories in the small-sample task by formula (5) where the prototype \(P = \{p\) n |n = 1, 2, … N\}; Step 7: Perform few-shot image classification according to the generated prototypes; For each sample in the query set Q, calculate its cosine similarity with all category prototypes in this task; then, take the category corresponding to the prototype with the highest similarity score as the predicted category of this query sample; finally, calculate the cross-entropy loss function according to the predicted category and the true label; Among them, CE(·) represents the cross-entropy loss function, and y q represent the predicted label and the true label of the query sample, respectively; Step 8: End-to-end meta-learning training; Perform Kullback–Leibler divergence regularization constraint on the distribution in the weight generator; where, D KL represents the Kullback–Leibler divergence, represents the standard normal distribution, represents the weight distribution corresponding to class n, represents the divergence regularization loss; Perform regularization constraint on the generated samples; where d intra represents the within-class distance, and d inter represents the between-class distance, represents the sample regularization loss; Finally, the objective function is Perform end-to-end training using the stochastic gradient descent method according to the objective function.

2. The small-sample image classification method based on prototype generation according to claim 1, wherein The feature extractor is composed of a ResNet-12 network, including four convolutional blocks with the number of channels being 64, 128, 256, and 1024 respectively, and the output feature dimension being 1024.

3. A small-sample image classification method based on prototype generation according to claim 1, characterized in that, The differential distribution generator consists of two fully connected layers, with an input dimension of 2048 and an output dimension of 2048. Nonlinear activation is performed through the LeakReLU activation function with a coefficient of 0.2, and a Dropout layer with a coefficient of 0.5 is configured to prevent overfitting.

4. A small-sample image classification method based on prototype generation according to claim 1, characterized in that, Both the encoder and decoder in the weight generator consist of one fully connected layer; among them, the encoder has an input dimension of 1024 and an output dimension of 512. Nonlinear activation is performed through the LeakReLU activation function with a coefficient of 0.2, and a Dropout layer with a coefficient of 0.5 is configured to prevent overfitting; the decoder has an input dimension of 256 and an output dimension of 65. Nonlinear activation is performed through the LeakReLU activation function with a coefficient of 0.2, and a Dropout layer with a coefficient of 0.5 is configured to prevent overfitting.

Images