A small sample image classification method based on class adaptive feature embedding

Abstract

The application discloses a small sample image classification method based on a class adaptive feature embedding, first, context information mining is carried out for each class in the small sample task; then, a class-specific implicit condition weight representation is constructed according to the context information of each class; then, weight parameters are generated for each class according to the implicit condition weight representation to establish a class adaptive embedding space; finally, classification is carried out in the class adaptive embedding space by using a Euclidean distance measurement mode. The method can well distinguish the current class from other classes, and good performance is achieved in small sample image classification.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a small sample image classification method based on a self-adaptive feature embedding. BACKGROUND

[0002] Image processing plays an important role in the industrialization of artificial intelligence. Image classification is a core technology in the field of image processing. In various application scenarios such as automatic driving, intelligent medical treatment and military remote sensing, image classification faces small sample problems such as difficulty in obtaining data and difficulty in labeling. Therefore, it is of important application value to study image classification technology for small samples. Small sample image classification aims to learn transferable knowledge from the training set and effectively classify unmarked samples in new categories under the guidance of a small number of marked samples. At present, the mainstream small sample image classification method focuses on learning a task-shared embedding space and using distance measurement in the embedding space for classification. Typically, the document "Snell J, Swersky K, Zemel R. Prototypical networks for few shot learning [C] / / Advances in Neural Information Processing Systems. 2017: 4077-4087." first learns an embedding space in the training set, then maps new categories in the test set to the embedding space, and finally uses the Euclidean distance measurement for classification. The document "Sung, Flood and Yang. Learning to compare: Relation network for few-shot learning [C] / / IEEE conference on computer vision and pattern recognition. 2018: 1199-1208." maps all categories in the small sample task to the trained embedding space and uses a deep network measurement for classification. The above methods all assume that all categories share a common embedding space. However, due to the great difference between different categories, it is difficult for the shared embedding space to describe the inherent properties of each category, so the generalization performance is poor. SUMMARY

[0003] In order to overcome the deficiencies of the prior art, the present application provides a small sample image classification method based on class adaptive feature embedding, first, the context information of each class in the small sample task is mined; then, the class-specific implicit condition weight representation is constructed according to the context information of each class; then, the weight parameters are generated for each class according to the implicit condition weight representation to establish a class-adaptive embedding space; finally, in the class-adaptive embedding space, the classification is carried out by using the Euclidean distance measurement method. The method can well distinguish the current class from other classes, and good performance is achieved in small sample image classification.

[0004] The technical solution adopted by the present application to solve its technical problems comprises the following steps:

[0005] Step 1: for small sample image classification task contains a support set and a query set Q; wherein the support set contains N classes, each class has K labeled samples, denoted as N-way K-shot; the query set is composed of part of unlabeled samples from the same N classes as the support set; the small sample image classification task requires using the support set information to construct a model, and then classifying the query set samples correctly;

[0006] Step 2: context information mining is performed on each class in the support set :

[0007]

[0008] Wherein, x n,k represents the deep feature of the kth support sample of the nth class obtained by the basic feature embedding module x m，l represents the deep feature of the lth support sample of the mth class obtained by the basic feature embedding module z n,j represents the context information obtained by feature dimension splicing of the kth support sample of the nth class and the lth support sample of the mth class in the task, represents feature splicing;

[0009] Step 3: input the context information z n,j of each class to the context encoder to construct a class-specific implicit condition weight representation:

[0010]

[0011] Wherein, d n,j represents the implicit condition weight representation corresponding to the jth context sample of the nth class in the task, φ represents the parametere Context encoder;

[0012] Step 4: Input the implicit conditional weight representation obtained in Step 3 into the weight generator to generate a class-adaptive embedding space:

[0013]

[0014] Where, θ n This represents the embedding space parameters generated for the nth class in this task, where P represents the total number of context samples corresponding to class n. The parameter is φ w Weight generator;

[0015] Step 5: Perform few-sample image classification based on the generated class adaptive embedding space:

[0016]

[0017] Where, x q x represents the test sample in query set Q. n This represents the average of all supporting samples for the nth category, i.e., the prototype of that category. This represents the class adaptive parameter θ generated in category n. n Zhang Cheng's embedding space contains the prototype x of this category. n Feature refinement is performed by multiplying the features by their parameters; similarly... This indicates feature refinement of the test samples; dist(·) represents the Euclidean distance, s(x q ,x n ) represents the query sample x within the embedding space constructed by category n. q Similarity score with category n;

[0018] The similarity score between the query sample and all categories is calculated sequentially, and the category with the highest similarity score is used as the predicted label for the query sample. Finally, based on the predicted labels Compared with the true label of the sample y q The cross-entropy loss between them is used as the objective function, i.e.

[0019]

[0020] Here, CE(·) represents the cross-entropy loss function. Based on the objective function, stochastic gradient descent is used for parameter optimization.

[0021] Furthermore, the basic feature embedding module is composed of ResNet-12 network, including four convolutional blocks, the number of channels is 64, 128, 256 and 640 respectively, and the output feature dimension is 640.

[0022] Further, the context encoder is composed of a full connection layer, the input dimension is 1280, the output dimension is 640, the non-linear activation is carried out through the LeakReLU activation function with a coefficient of 0.2, and the Dropout layer with a coefficient of 0.9 is configured to prevent overfitting.

[0023] Further, the weight generator is composed of a full connection layer, the input dimension is 640, the output dimension is 640, the non-linear activation is carried out through the LeakReLU activation function with a coefficient of 0.2, and the Dropout layer with a coefficient of 0.7 is configured to prevent overfitting; in addition, each dimension in the output 640-dimensional parameter vector is activated by the square, so that the generated weight parameters have nonlinear representation ability.

[0024] The beneficial effects of the application are as follows:

[0025] The application breaks the limitation brought by the shared embedding space, and builds a specific embedding space for each category, in which the sample from the current category can extract the inherent feature attribute of the category, so that the current category can be well distinguished from other categories; the method has good performance in small sample image classification. DETAILED DESCRIPTION

[0026] The application will be further described below in combination with embodiments.

[0027] A small sample image classification method based on class adaptive feature embedding, comprising the following steps:

[0028] Step 1: for small sample image classification task contains a support set and a query set Q. Wherein, the support set contains N categories, each category has K labeled samples, denoted as N-way K-shot; the query set is composed of some unlabeled samples from the same N categories. The small sample image classification task requires using the support set information to build a model, and then classifying the query set samples correctly.

[0029] Step 2: respectively mining context information for each category in the support set :

[0030]

[0031] Wherein, x n,k is the basic feature embedding module The deep feature of the kth support sample of the nth class, z n,j The context information obtained by concatenating the kth support sample of the nth class and the lth support sample of the mth class in the task in the feature dimension. Indicates feature concatenation.

[0032] The established basic feature embedding module is composed of a ResNet-12 network, including four convolutional blocks, with channel numbers of 64, 128, 256, and 640, and an output feature dimension of 640.

[0033] Step 3: input the context information z n,j of each class into the context encoder to construct a class-specific implicit condition weight representation:

[0034]

[0035] where d n,j represents the implicit condition weight representation corresponding to the jth context sample of the nth class in the task. represents the context encoder with parameter φ e .

[0036] The established context encoder is composed of a fully connected layer, with an input dimension of 1280 and an output dimension of 640, and is activated by a LeakReLU activation function with a coefficient of 0.2 for non-linear activation and a Dropout layer with a coefficient of 0.9 to prevent overfitting.

[0037] Step 4: input the implicit condition weight representation obtained in step 3 into the weight generator to generate a class-adaptive embedding space:

[0038]

[0039] where θ n represents the embedding space parameter generated by the nth class in the task. P represents the total number of context samples corresponding to class n. represents the weight generator with parameter φ w .

[0040] The established weight generator is composed of a fully connected layer, with an input dimension of 640 and an output dimension of 640, and is activated by a LeakReLU activation function with a coefficient of 0.2 for non-linear activation and a Dropout layer with a coefficient of 0.7 to prevent overfitting. In addition, each dimension in the output 640-dimensional parameter vector is activated by a quadratic function, so that the generated weight parameters have non-linear representation ability.

[0041] Step 5: Small sample image classification according to the generated class adaptive embedding space:

[0042]

[0043] where x q represents a test sample in the query set Q. x n represents the average of all support samples of the nth class, i.e. the prototype of the class. represents the class adaptive parameter θ n generated in the class n. n The prototype x q of the class is refined in the embedding space formed by Zhang Cheng. The feature refinement is in the form of point multiplication between the feature and the parameter. Similarly, represents the feature refinement of the query sample. dist(·) represents the Euclidean distance. s(x n ) represents the similarity score of the query sample x q with the class n in the embedding space constructed in the class n.

[0044] The similarity scores between the query sample and all classes are calculated in turn, and the class with the maximum similarity score is taken as the predicted label of the query sample Finally, the cross-entropy loss between the predicted label and the real label y q of the sample is taken as the objective function, i.e.

[0045]

[0046] where CE(·) represents the cross-entropy loss function. According to the objective function, the random gradient descent method is used for parameter optimization. Specific embodiments:

[0048] The application provides a small sample image classification method based on class adaptive feature embedding, and the specific process is as follows:

[0049] 1. Data preprocessing

[0050] For each small sample image classification task contains a support set and a query set Q. The support set contains N classes, and each class has K labeled samples, denoted as N-way K-shot. The query set is composed of some unlabeled samples from the same N classes. The small sample image classification task requires using the support set information to construct a model, and then classifying the query set samples correctly. In the training set, multiple small sample tasks are constructed for iterative training.

[0051] 2. Context information mining

[0052] In order to increase the discriminant ability between classes, the application mines the inter-class context information. The context information is mined for each class in the support set :

[0053]

[0054] where x n,k represents the deep feature of the kth support sample of the nth class obtained by the basic feature embedding module z n,j represents the context information obtained by feature dimension splicing between the kth support sample of the nth class and the lth support sample of the mth class in the task. represents feature splicing.

[0055] 3. Context encoder

[0056] According to the context information of each class, the specific implicit representation of each class is modeled by the encoder. Specifically, z n,j is input into the context encoder to construct the class-specific implicit conditional weight representation:

[0057]

[0058] where d n,j represents the implicit conditional weight representation corresponding to the jth context sample of the nth class in the task. represents the context encoder with parameter φ e .

[0059] 4. Weight generator

[0060] The implicit conditional weight representation obtained in step 3 is input into the weight generator to generate a class-adaptive embedding space:

[0061]

[0062] where θ n represents the embedding space parameter generated by the nth class in the task. P represents the total number of context samples corresponding to class n. represents the weight generator with parameter φ w .

[0063] 5. Small sample image classification

[0064] According to the generated class-adaptive embedding space, small sample image classification is performed:

[0065]

[0066] where x qx represents a test sample in the query set Q. n x represents the average of all support samples of the n-th class, i.e., the prototype of the class. x represents the class-adaptive parameter generated in the n-th class n x represents the prototype of the class in the embedding space formed by Zhang Cheng n Feature refinement is performed. The way of feature refinement is point multiplication operation between feature and parameter. Similarly, x represents feature refinement on the query sample. dist(·) represents the Euclidean distance. s(x q ,x n x represents the similarity score of the query sample x q and the n-th class in the embedding space constructed in the n-th class.

[0067] The similarity scores between the query sample and all classes are calculated in turn, and the class with the maximum similarity score is taken as the predicted label of the query sample Finally, the cross-entropy loss between the predicted label and the real label y q of the sample is taken as the objective function, i.e.

[0068]

[0069] where CE(·) represents the cross-entropy loss function. According to the objective function, the random gradient descent method is used to optimize the parameters.

[0070] In the test phase, for each sampled small sample task, an embedding space of the class-adaptive is constructed for each class in the task, and the similarity score between the current class and the query sample is calculated in each embedding space. After the calculation of all classes, the class with the highest similarity score is taken as the prediction result of the query sample.

[0071] The method of the present application can obtain better small sample image classification results. Taking the mini-ImageNet data set as an example, under the two settings of 5-way 1-shot and 5-way 5-shot, the accuracy of the method of the present application can reach 63.01% and 79.78% respectively. Compared with the benchmark method of the prototype network sharing embedding space, it is increased by 6.72% and 10.24% respectively.

Claims

1. A small sample image classification method based on class adaptive feature embedding, characterized in that, Comprising the following steps: Step 1: for a small sample image classification task a support set and a query set Q; wherein the support set contains N classes, each class has K labeled samples, denoted as N-way K-shot; the query set consists of part of unlabeled samples from the same N classes as the support set; the small sample image classification task requires using the support set information to build a model, and then correctly classifying the query set samples; Step 2: Contextual information mining for each category in the support set Step 2: Contextual information mining for each category in the support set wherein x n,k represents the deep feature of the kth support sample of the nth class obtained by the basic feature embedding module wherein x m，l represents the deep feature of the kth support sample of the nth class obtained by the basic feature embedding module wherein z n,j represents the context information obtained by performing feature dimension splicing on the kth support sample of the nth class and the lth support sample of the mth class in the task, and represents feature splicing. Step 3: Contextual information z n,j is input into the context encoder to construct the class-specific implicit conditioning weight representation: wherein d n,j represents the implicit condition weight representation corresponding to the jth context sample of the nth class in the task, represents the context encoder with parameters φ e . Step 4: input the implicit condition weight representation obtained in step 3 into the weight generator to generate a class-adaptive embedding space: where θ n represents the embedding space parameter generated by the nth class in this task, P represents the total number of context samples corresponding to class n, represents the weight generator with parameter φ w . Step 5: perform small sample image classification according to the generated class-adaptive embedding space: where x q represents the test sample in the query set Q, x n represents the average of all support samples of the nth class, i.e., the prototype of the class, represents the class-adaptive parameter θ n generated in the class n, which is used to perform feature refinement on the prototype x n of the class in the embedding space spanned by the features, and the feature refinement is performed by point multiplication between the feature and the parameter; similarly, represents the feature refinement on the test sample; dist(·) represents the Euclidean distance, s(x q ,x n ) represents the similarity score of the query sample x q to the class n in the embedding space constructed in the class n. The similarity scores between the query sample and all categories are calculated in sequence, and the category with the maximum similarity score is taken as the predicted label of the query sample Finally, the cross-entropy loss between the predicted label and the sample true label y q is taken as the objective function, that is, Wherein, CE(·) represents a cross-entropy loss function; according to the objective function, a stochastic gradient descent method is used for parameter optimization.

2. The method of claim 1, wherein the method is based on a class- adaptive feature embedding. The basic feature embedding module is composed of a ResNet-12 network, including four convolution blocks, the number of channels being 64, 128, 256, and 640 respectively, and the output feature dimension being 640.

3. The method of claim 1, wherein the method is based on a class- adaptive feature embedding. The context encoder is composed of one fully connected layer, the input dimension is 1280, the output dimension is 640, and the non-linear activation is performed through the LeakReLU activation function with a coefficient of 0.2 and the Dropout layer with a coefficient of 0.9 is configured to prevent overfitting.

4. The method of claim 1, wherein the method is based on a class- adaptive feature embedding. The weight generator is composed of one fully connected layer, the input dimension is 640, the output dimension is 640, and the non-linear activation is performed through the LeakReLU activation function with a coefficient of 0.2 and the Dropout layer with a coefficient of 0.7 is configured to prevent overfitting; in addition, each dimension in the output 640-dimensional parameter vector is activated by the square function, so that the generated weight parameters have non-linear representation ability.

Images