A small sample image classification method based on k-means clustering

Abstract

The present invention discloses a small sample image classification method based on k-means clustering. First, a task including a support set and a query set is sent to a feature extractor, and after continuous convolution block operations, deep image features are obtained. Then, a small sample classification module uses the deep image features of the query set and the support set to construct a similarity matrix to predict the support category to which the pictures in the query set belong. At the same time, a k-means clustering task clusters the image features output by the feature extractor to form a number of clusters, and assigns pseudo labels to the samples in each cluster. The cluster classification module divides the image features output by the feature extractor into k categories according to the pseudo labels. During the training process, multiple tasks share information and promote learning from each other. Compared with the existing small sample image classification model, the present invention significantly improves the accuracy of small sample image classification while maintaining the stability of the model.

Description

Technical Field

[0001] The present invention belongs to the field of small sample learning in computer vision, and mainly relates to a small sample image classification method based on k-means clustering. Background Art

[0002] In the past decade, deep learning has made breakthroughs in many fields, mainly due to good model structure, large amounts of training data and high-performance hardware support. Thanks to high-performance hardware support, people can now train deep neural networks with thousands of layers on millions of data. Training deep neural networks requires a large amount of labeled data, but sometimes labeling data requires experts in certain fields (such as labeling of medical images). In addition, labeling data is time-consuming and laborious, which is still far from human intelligence. The lack of a large amount of labeled data and the time-consuming and labor-intensive labeling of data have promoted the research of small sample learning, which exists in many practical applications, such as unmanned driving, medical testing, intelligent robots, AR technology, etc. In recent years, deep learning has achieved many impressive results due to its powerful ability to fit multi-layer nonlinear transformations, especially in image processing, speech recognition, natural language processing, etc. Although deep learning has a strong nonlinear fitting ability, it is based on a large amount of training labeled data. When facing a small data set, due to the serious lack of samples, overfitting often occurs.

[0003] Due to the lack of a large amount of data, the core problem of small sample learning is that the minimum empirical risk is unreliable, which often leads to overfitting problems. In order to alleviate this problem, prior knowledge must be used. In order to improve the ability of learning to use prior knowledge, there are currently three small sample learning methods, namely data enhancement methods, network model methods and algorithm optimization methods. The data enhancement method is to expand the data set through prior knowledge, which includes flipping, shearing, scaling, affine, rotation, etc. The small sample data set is expanded through data enhancement, and the obtained data and the original data constitute a new data set. In addition, the purpose of expanding the data set can be achieved by introducing unlabeled data sets and similar data sets, thereby enhancing the supervised classification effect. The network model method mainly includes metric learning models, learning models with external storage, and generative models. The metric learning model maps data from high-dimensional space to low-dimensional space by mapping, and uses Euclidean distance to measure the distance between samples. Similar samples are close to each other, otherwise they are far away from each other. The learning model with external storage uses external storage to store the trained information. Each test sample is represented by the weighted average of the content extracted from the memory, which can reduce the size of the hypothesis space and alleviate the overfitting problem. Algorithm optimization methods mainly include two categories: adjusting the existing parameters of the model and adjusting the parameters based on the meta-learner. Adjusting the existing parameters of the model includes fine-tuning with regularization terms and aggregating multi-task related parameters. Adjustment based on the meta-learner includes the initialization step based on the meta-learner and the output search step based on the meta-learner.

[0004] At present, metric learning has gradually become the mainstream small sample learning method, and has made a lot of achievements in small sample learning. Although metric learning has certain advantages in small sample learning, it still cannot alleviate the overfitting problem well. To this end, additional tasks are further introduced into the network structure. During the training process, multiple learning tasks share information and complement each other, which can improve each other's learning ability and further alleviate the overfitting problem in small sample learning. Summary of the invention

[0005] The present invention proposes a small sample image classification method based on k-means clustering. This method uses the query image features and the support class features to construct a similarity matrix, and judges whether the test image belongs to the support class according to the feature similarity. In addition, in order to further alleviate the overfitting problem in small sample learning, a k-means clustering task is introduced into the model, all image features are clustered into k clusters, and pseudo labels are assigned to the samples in the clusters. After that, the cluster classification module performs ordinary classification on the features according to the pseudo labels. The present invention further alleviates the overfitting problem in small sample learning, and greatly improves the accuracy of small sample image classification.

[0006] In order to achieve the above object, the present invention is implemented by the following technical solutions:

[0007] The present invention is a small sample image classification method based on k-means clustering, comprising the following steps:

[0008] Step 1: Given a task that includes a support set and a query set, resize all images in the task to make them uniform.

[0009] Step 2: Input all the images in the task into the feature extractor. After continuous convolution block operations, the feature map containing the query set and the support set will be output at the end of the feature extractor.

[0010] Step 3: The small sample classification module uses the feature graphs of the query image and the support class to determine the support class to which the query image should belong by constructing a similarity matrix between them;

[0011] Step 4: The clustering module uses k-means clustering to cluster the feature maps of the query image and the support class, clustering all feature maps into k clusters and assigning pseudo labels to the samples in all clusters;

[0012] Step 5: The cluster classification module uses the assigned pseudo labels to classify all feature maps from the feature extractor into k categories;

[0013] Step 6: Use the dataset to train the entire model. After all weight parameters are trained to the optimal state, test the model and input a set of images containing a new support set and query set. Modify the image size to unify it and execute steps 2 to 5 until the classification task is completed.

[0014] A further improvement of the present invention is that the image sizes in step 1 and step 6 are uniformly modified to 84*84*3.

[0015] A further improvement of the present invention is that the feature extractor in step 2 adopts a Conv-64F network, which includes 4 convolution blocks, the first two convolution blocks include a convolution layer, an activation function layer and a pooling layer, and the last two convolution blocks include a convolution layer and an activation function layer.

[0016] A further improvement of the present invention is that in step 3, the similarity matrix construction process is specifically as follows: the feature extractor is denoted as f Θ , where θ is the parameter of the feature extractor; given a picture, the feature map output by the feature extractor is After that, the dimension of the feature map P becomes That is, P = [p1, p2, ..., p m ], where p iRepresents the feature vector at the i-th position in the feature map, m = h × w represents the number of vectors; given a support class S containing N images τ (τ∈{1,2,...,C}), the N images are denoted as Z1,...,Z N , its feature map is expressed as Each query image is denoted as Q a (a∈{1,2,...,T}), the feature map can be recorded as Similarity matrix between each query image and supporting classes It can be expressed as

[0017]

[0018] Where L2 represents regularization. Each value in the similarity matrix represents a local similarity, so the similarity matrix contains m×mN local similarities. Then, the K largest values ​​are selected from each row in the similarity matrix, and the similarity values ​​of all rows are summed up, that is, the mK similarity values ​​are summed up to obtain the final similarity value ζ aτ , aτ The larger the value, the more likely the query image Q a With support class S τ The more similar.

[0019] The further improvement of the present invention is that the specific steps of step 4 are as follows:

[0020] Step 4.1: Assume that a task containing a support set and a query set contains r images, the feature extractor should output r feature maps, and the feature map sample set D = {x1, x2, ..., x r};

[0021] Step 4.2, k-means clustering divides the feature samples into k non-overlapping clusters {M l |l=1,2,....,k},where and

[0022] Step 4.3, record δ j ∈{1, 2, ..., k} represents the sample x j 's pseudo-labels.

[0023] A further improvement of the present invention is that the specific operation of step 4.2 is:

[0024] Step 4.21, randomly select k samples from D as the initial mean vector Step 4.22, set Calculate for each sample x j With each mean vector Distance: Where |||2 represents the Euclidean distance. Determine x based on the nearest mean vector j Cluster label: δ j = arg min i∈{1，2，...，k} d ji ; and the feature sample x j Assign to the corresponding cluster: Step 4.23, for each cluster, calculate the new mean vector: if The entire clustering process ends, and the final output is the cluster division And the pseudo labels δ1, δ2, ..., δ corresponding to each feature sample r Otherwise, continue to step 4.22 until The whole clustering process is completed.

[0025] A further improvement of the present invention is that the cluster classification module in step 5 includes two convolution blocks and a fully connected layer, and each convolution block includes a pooling layer.

[0026] A further improvement of the present invention is that the data sets used for training in step 6 include Mini ImageNet and Cub-200.

[0027] The beneficial effect of the present invention is that compared with the existing small sample image classification model, the method of the present invention alleviates the overfitting problem in small sample learning while maintaining the stability of the model, and significantly improves the accuracy of small sample image classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flow chart of the small sample image classification method based on k-means clustering proposed by the present invention. DETAILED DESCRIPTION

[0029] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more explicit definition of the protection scope of the present invention. However, these embodiments are illustrative and are only used to specifically describe the present invention, and should not be understood as limiting the present invention.

[0030] like Figure 1 As shown, the present invention is a small sample image classification method based on k-means clustering. The network model used in the present invention to perform classification tasks mainly includes four modules: a feature extractor module, a small sample classification module, a clustering module, and a cluster classification module. The specific classification method steps are as follows:

[0031] Step 1: Given a task that includes a support set (the support set contains multiple support classes) and a query set, unify the size of all images in the task to 84*84*3;

[0032] Step 2: All images in the task are input into the feature extractor. After continuous convolution block operations, the feature map containing the query set and the support set is output at the end of the feature extractor. The whole process obtains the multi-level information distribution of the image and enhances the high-frequency detail information.

[0033] The feature extractor uses the Conv-64F network, which contains 4 convolution blocks. The first two convolution blocks contain convolution layers, activation function layers, and pooling layers. In order to obtain rich feature information, the last two convolution blocks only contain convolution layers and activation function layers. This is mainly because in the small sample module, a similarity matrix needs to be constructed to evaluate the similarity between the features of the query image and the features of the support class. Generally, the larger the feature size, the more information it contains. In order to more accurately evaluate the similarity between two features, the last two convolution blocks no longer use pooling layers. The entire network structure and parameters are shown in Table 1:

[0034] Table 1 Conv-64F network structure and parameters

[0035]

[0036] Among them, Layer represents all network layers from the first convolution layer Conv1 that receives the original image input to the last layer; Size represents the convolution kernel or filter size and the number of channels of the current convolution layer or pooling layer; Stride represents the filter step size of the current network layer; Padding represents the Padding method used by the current network layer: same method or valid method.

[0037] When coming from a set of tasks that include a support set and a query set, the network first modifies the input size of all images to 84*84*3. The feature dimensions and sizes after the convolution calculation output of each layer of the feature extractor are shown in Table 2:

[0038] Table 2 Image size of each layer of Conv-64F network

[0039]

[0040] Among them, Layer represents the different network layers in the feature extractor, and Size represents the feature scale and number of channels of the input image after being processed by the corresponding network layer.

[0041] Step 3: The small sample classification module uses the feature graphs of the query image and the support class to determine the support class to which the query image should belong by constructing a similarity matrix between them;

[0042] The similarity matrix construction process is as follows: The feature extractor is denoted as f Θ , where θ is the parameter of the feature extractor. Given a picture, the feature map output by the feature extractor is After that, the dimension of the feature map P becomes That is, P = [p1, p2, ..., p m ], where p i represents the feature vector at the ith position in the feature map, and m = h × w represents the number of vectors. Given a support class Sτ(τ∈{1, 2, ..., C}) containing N images, the N images are denoted Z1, ..., Z N , its feature map can be expressed as Each query image is denoted as Q a (a∈{1,2,...,T}), the feature map can be recorded as Similarity matrix between each query image and supporting classes It can be expressed as

[0043]

[0044] Where L2 represents regularization, and each value in the similarity matrix represents a local similarity, so the similarity matrix contains m×mN local similarities. Then select the K largest values ​​from each row in the similarity matrix and sum the similarity values ​​of all rows, that is, sum the mK similarity values ​​to get the final similarity value ζ aτ ζ aτ The larger the value, the more likely the query image Q a With support class S τ The more similar.

[0045] Finally, the small sample classification loss function can be expressed as

[0046]

[0047] Where V a represents the true label of each query image, C represents the number of support classes contained in each task, and T represents the number of query images contained in the query set in each task.

[0048] Step 4: The clustering module uses k-means clustering to cluster the query image and the feature maps of the support class, clustering all feature maps into k clusters and assigning pseudo labels to the samples in all clusters;

[0049] Step 5: The entire cluster classification module is Ω, which contains two convolution blocks, a fully connected layer, and each convolution block contains a pooling layer. The two convolution blocks continue to operate on the features output by the feature extractor. This is mainly because the feature size output by the feature extractor is large and it is not convenient to directly flatten the input to the fully connected layer for classification. Therefore, two convolution blocks containing pooling layers are used to operate the image features. Finally, the cluster classification module divides all image features into k categories according to the pseudo labels generated by clustering. The loss function generated by this module can be expressed as

[0050]

[0051] Step 6: For the entire framework, the total loss can be expressed as L total =L few +L cluster . After building a deep neural network for performing small sample image classification tasks, start training the network weight parameters through back propagation. During the training process, multiple tasks share information and promote each other's learning. The training data sets include the MiniImageNet data set and the Cub-200 data set. Among them, the MiniImageNet data set contains 100 classes, each class has 600 images, and these 100 classes are usually divided into 64 training classes, 16 validation classes, and 20 test classes. The Cub-200 data set is a mainstream fine-grained data set, which contains 200 bird species. The 200 classes are divided into 130 training classes, 20 validation classes, and 50 test classes. After the training of each weight parameter reaches the optimal state, that is, when the total loss reaches the convergence condition, the model is tested, and a set of images containing a new support set and a query set are input. The network first uniformly modifies the image size to 84*84*3, and then performs the tasks described in steps 2 to 5 until the classification task is completed.

[0052] Few-shot image classification is different from conventional image classification. Formally speaking, the training set of few-shot learning contains many categories, and each category has multiple samples. During the training phase, the model randomly extracts C categories from the data set, and then extracts N samples from each category (a total of C*N data) as the support set of the model; then T samples are extracted from the remaining data of these C categories as the query set. The query set and the support set constitute a task. The goal of few-shot learning is to require the model to learn to accurately divide the samples in the query set into the corresponding support classes from C*N data. Such a task is called the C-way N-shot problem. Usually, C is 5 and N is 1 or 5. During the training process, the model randomly extracts thousands of tasks consisting of support sets and query sets to train the model until the model parameters are optimal, that is, when the total loss function reaches the convergence condition. In general, each training contains a different combination of categories. This training mechanism enables the model to learn the common parts of different tasks, such as how to extract important features. The model learned through this learning mechanism can also perform better classification when facing new tasks.

[0053] From the small sample image classification methods proposed in recent years, the accuracy is compared with the small sample image classification method proposed in this invention on the MiniImageNet and Cub-200 datasets, and the results are shown in Table 3 and Table 4 respectively:

[0054] Table 3 Experimental results of MiniImageNet test set

[0055]

[0056]

[0057] Table 4 Experimental results of Cub-200 test set

[0058]

[0059] Among them, Methods is the method involved in the comparison, Backbone indicates the type of feature extractor used by the method, and the two rightmost columns indicate the accuracy in the 5-way 1-shot and 5-way 5-shot cases respectively. Note that the accuracy is the average accuracy obtained by randomly selecting 1000 new task tests and repeating them 5 times. At the same time, 95% confidence is also given.

[0060] It can be seen from the data in the table that the present invention is superior to the experimental comparison object in accuracy, that is, precision, so the effectiveness of the present invention is proved.

[0061] The above is only a preferred embodiment of the present invention, and does not limit the present invention in any form. Although the present invention has been discussed above with a preferred embodiment, it is not used to limit the present invention. Any technician familiar with the profession can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A small sample image classification method based on k-means clustering, characterized by: The steps include: Step 1: Given a task that includes a support set and a query set, modify the sizes of all images in the task to be consistent; Step 2: All images in the task are input into the feature extractor. After continuous convolutional block operations, feature maps containing query set and support set are output at the end of the feature extractor. Step 3: The small sample classification module uses the feature graphs of the query image and the support class to determine the support class to which the query image should belong by constructing a similarity matrix between them; Step 4: The clustering module uses k-means clustering to cluster the feature maps of the query image and the support class, clustering all feature maps into k clusters and assigning pseudo labels to the samples in all clusters; Step 5: The cluster classification module uses the assigned pseudo labels to classify all feature maps from the feature extractor into k categories; Step 6: Use the data set to train the entire model. After the weight parameters are trained to the optimal state, test the model and input a set of images containing a new support set and query set. Modify the image size to unify it and execute steps 2 to 5 until the classification task is completed. In step 3, the similarity matrix construction process is as follows: let the feature extractor be f Θ , where θ is the parameter of the feature extractor; given a picture, the feature map output by the feature extractor is After that, the dimension of the feature map P becomes That is, P = [p1, p2, ..., p m ], where p i Represents the feature vector at the i-th position in the feature map, m = h × w represents the number of vectors; given a support class S containing N images τ (τ∈{1,2,...,C}), the N images are denoted as Z1,...,Z N , Its feature map is represented as Each query image is denoted as Q a (a∈{1,2,...,T}), the feature map is recorded as Similarity matrix between each query image and supporting classes It is expressed as: Where L2 represents regularization. Each value in the similarity matrix represents a local similarity, so the similarity matrix contains m×mN local similarities. Then, the K largest values ​​are selected from each row in the similarity matrix, and the similarity values ​​of all rows are summed up, that is, the mK similarity values ​​are summed up to obtain the final similarity value ζ aτ , aτ The larger the value, the more likely the query image Q a With support class S τ The more similar.

2. The small sample image classification method based on k-means clustering according to claim 1, characterized in that: The image sizes in steps 1 and 6 are uniformly modified to 84*84*3.

3. The small sample image classification method based on k-means clustering according to claim 1, characterized in that: The feature extractor in step 2 uses the Conv-64F network, which contains 4 convolution blocks. The first two convolution blocks include a convolution layer, an activation function layer, and a pooling layer, and the last two convolution blocks include a convolution layer and an activation function layer.

4. The small sample image classification method based on k-means clustering according to claim 1, characterized in that: Step 4 The specific steps are as follows: Step 4.1: Assume that a task including support set and query set contains r images in total. Then the feature extractor should output r feature maps. Let the feature map sample set D = {x1, x2, ..., x r }; Step 4.2, k-means clustering divides the feature samples into k non-overlapping clusters {M l |l=1,2,...,k},where and Step 4.3, record δ j ∈{1, 2, ..., k} represents the sample x j 's pseudo-labels.

5. The small sample image classification method based on k-means clustering according to claim 4, characterized in that: The specific operations of step 4.2 are as follows: Step 4.21, randomly select k samples from D as the initial mean vector Step 4.22, set Calculate for each sample x j With each mean vector Distance: Where || ||2 represents the Euclidean distance; x is determined based on the nearest mean vector. j Cluster markers: And the feature sample x j Assign to the corresponding cluster: Step 4.23, for each cluster, calculate the new mean vector: if Then the whole clustering process ends, and the final output cluster partition θ = {M1, M2, ...., M k } and the pseudo labels δ1, δ2, ..., δ corresponding to each feature sample r Otherwise, continue to step 4.22 until The whole clustering process is completed.

6. The small sample image classification method based on k-means clustering according to claim 1, characterized in that: The cluster classification module in step 5 contains two convolution blocks and a fully connected layer, and each convolution block contains a pooling layer.

7. The small sample image classification method based on k-means clustering according to claim 1, characterized in that: The datasets used for training in step 6 include MiniImageNet and Cub-200.

Images