A small sample image classification method based on improved prototype network

Abstract

The present invention relates to a small sample image classification method based on an improved prototype network, comprising: initializing an image data set of a current task, dividing the image data set of the current task after initialization into a training set and a test set; using the training set and combining with an external auxiliary data set to train a GNN to obtain a trained GNN; inputting the training set and the test set into the trained GNN respectively to respectively perform feature embedding, and obtaining the training set image features and the test set image features respectively through feature encoding; screening out valid sample feature class prototypes from the training set according to the training set image features; inputting the test set image features and the valid sample feature class prototypes into a CNN to obtain the similarities between the test set image samples and various types of prototypes respectively; and completing the classification of the test set image samples according to the similarities between the test set image samples and various types of prototypes. Compared with the prior art, the present invention can effectively improve the accuracy of small sample image classification.

Description

Technical Field

[0001] The present invention relates to the technical field of image classification, and in particular to a small sample image classification method based on an improved prototype network. Background Art

[0002] Prototype networks are widely used in image sample recognition and classification of small sample data sets. This algorithm is a simple embedding learning method (a general embedding model of a single embedding model) in small sample image classification methods. It can identify and learn the corresponding category of the image given a small number of samples in each category of image samples. Figure 1 As shown in the figure, the main idea is to map (feature embedding) image samples in each category into a space (CNN, Convolutional Neural Networks, convolutional neural network), learn a "good" mapping through the neural network, and then project each task sample into the same space;

[0003] Next, we extract their "mean" to represent the prototype of the class. The implementation principle is as follows:

[0004]

[0005] Then, the Euclidean distance is used as the distance metric of the metric module. During testing, the distance between the test set image sample and the prototype of each type of training set image sample is calculated, and the softmax function is used to process the calculated distance to determine the category label of the test data.

[0006] The above existing prototype network methods have the following shortcomings when classifying small sample images: First, when embedding sample features, due to the use of a relatively simple CNN network architecture (with fewer network layers), it is impossible to ensure that enough sample features relevant to the task can be obtained during feature extraction. At the same time, the CNN network architecture has poor encoding and expression capabilities for sample features, making it difficult to make the feature vectors of various sample prototypes in the feature embedding space have sufficiently obvious differences;

[0007] Second, the task relevance of the feature embedding network trained only with external auxiliary image sample sets is weak, which is not conducive to obtaining important sample features related to the task and is not conducive to subsequent accurate classification;

[0008] Third, using the "prototype" obtained by averaging to represent all samples of a known class will be a generalization. If the amount of data is sufficient, it may achieve better results. However, when the amount of existing data is small, there are usually interfering samples in the sample set in the actual scene. The prototype obtained by simply averaging will be affected by the interfering samples, and the subsequent classification of image samples will result in a large classification deviation;

[0009] Fourthly, when measuring, only the traditional fixed distance measurement is used to measure the similarity of two or more samples. This may work well for some simple and general image datasets, but for commonly used image datasets, since there are fewer common features between them, it is not easy to find the sample features related to the task, which makes some image samples easily ignored. If it is directly used to classify image samples, the image discrimination effect will appear to be general and the classification accuracy will be low. Summary of the invention

[0010] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a small sample image classification method based on an improved prototype network, by improving the existing prototype network and applying it to small sample image classification to improve the classification accuracy.

[0011] The purpose of the present invention can be achieved by the following technical solution: A small sample image classification method based on an improved prototype network comprises the following steps:

[0012] S1, initialize the image dataset of the current task, and divide the initialized image dataset of the current task into a training set and a test set;

[0013] S2. Use the training set and combine it with an external auxiliary data set to train the GNN (Graph Neural Network) to obtain a trained GNN.

[0014] S3, input the training set and the test set into the trained GNN respectively to perform feature embedding, and obtain the training set image features and the test set image features respectively through feature encoding;

[0015] S4, based on the image features of the training set, screen out valid sample feature class prototypes from the training set;

[0016] S5, input the test set image features and the valid sample feature class prototypes into CNN to obtain the similarity between the test set image samples and each type of prototype;

[0017] S6. According to the similarity between the image samples in the test set and various prototypes, the image samples in the test set are classified into categories.

[0018] Furthermore, the training set includes a plurality of image samples of known categories, and the test set includes a plurality of image samples of unknown categories.

[0019] Furthermore, the step S2 specifically includes the following steps:

[0020] S21. Select a corresponding external auxiliary image dataset according to the image dataset of the current task;

[0021] S22, initializing the external auxiliary image data set;

[0022] S23, randomly extract some samples from the training set and form mixed sub-task image sample pairs with the initialized external auxiliary image dataset;

[0023] S24. According to the subtask image sample pairs, complete the training of the GNN to obtain a trained GNN.

[0024] Furthermore, the step S23 specifically uses the training strategy of Episodic training to randomly extract some samples from the training set.

[0025] Furthermore, the step S24 specifically trains the embedding parameters according to the subtask image sample pairs, utilizing the updates of the edges and nodes of the GNN, through multiple learning and optimization, to obtain a trained GNN.

[0026] Furthermore, the step S4 specifically includes the following steps:

[0027] S41, calculating the weighted class prototype of the training set samples by using a weighted mean method according to the training set image features;

[0028] S42. Filter out valid sample feature class prototypes from the training set based on the weighted class prototypes.

[0029] Furthermore, the step S42 specifically uses the weighted class prototype as a task-related feature representation, and uses the task-related feature representation to screen the training set samples of the current task to distinguish between valid sample features and interfering sample features, and eliminates the interfering sample features to retain only valid sample features.

[0030] Furthermore, the step S42 specifically adopts a Boolean logic method to distinguish between valid sample features and interference sample features, wherein "1" represents a valid sample feature and "0" represents an interference sample feature.

[0031] Furthermore, the specific process of step S5 is: using CNN as a dynamic measurement network, inputting the test set image features and the valid sample feature class prototypes into CNN, and dynamically learning the measurement parameters in CNN. As the measurement parameters in CNN are continuously adjusted and updated, the network parameters and prototype weights are used to continuously calculate and update the similarity measurement between the test set image features and various prototypes in the training set. When calculating the measurement for the current task test set image samples, the image sample features are continuously used to correct the weight values ​​of the image class prototypes of each category to make the prototype more accurate, and finally the similarity between each test set image sample and each prototype is obtained.

[0032] Furthermore, the specific process of step S6 is: sorting the similarities between the test set image samples and each type of prototype according to the numerical value, wherein the class prototype corresponding to the maximum similarity is the determination category of the test set image samples.

[0033] Compared with the prior art, the present invention proposes a method for classifying small sample images based on an improved prototype network, and at the same time improves and perfects the embedding module and the measurement module, replaces the original feature embedding network, prototype calculation method and measurement method, and specifically adopts a GNN with more powerful sample feature embedding and encoding capabilities, which can obtain and save better image sample feature representation, help train and optimize to obtain better network embedding parameters, and further improve the accuracy of subsequent sample recognition and classification; secondly, combine part of the image sample set of the current task with the external auxiliary image data set, and jointly participate in the training of the feature embedding network; furthermore, after calculating the class prototype of the sample feature encoding of each class by the weighted mean method, it is used to represent the importance of each class, and Boolean logic is used to distinguish valid features from invalid features; in addition, CNN is used as a measurement module, and CNN is used to dynamically train and learn the relevant parameters of the measurement function, and the weights of various prototypes are continuously updated and adjusted, and the similarity measurement of the test set image samples is updated at the same time, so as to obtain the optimal similarity judgment result. This ensures that each image sample can participate in the classification process, effectively improves the learning ability of the prototype network for the small sample set of images, and improves the accuracy of classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A flow chart of the method for image classification using a traditional prototype network;

[0035] Figure 2 It is a schematic diagram of the method flow of the present invention;

[0036] Figure 3 It is a schematic diagram of the application process of the embodiment;

[0037] Figure 4This is a comparison chart of the image classification results of the traditional prototype network in the embodiment and the improved prototype network of the present invention. DETAILED DESCRIPTION

[0038] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0039] Example

[0040] like Figure 2 As shown, a small sample image classification method based on an improved prototype network includes the following steps:

[0041] S1, initialize the image data set of the current task, and divide the initialized image data set of the current task into a training set and a test set, wherein the training set includes a plurality of image samples of known categories, and the test set includes a plurality of image samples of unknown categories;

[0042] S2. Use the training set and external auxiliary data sets to train the GNN (Graph Neural Network) to obtain a trained GNN. Specifically:

[0043] S21. Select a corresponding external auxiliary image dataset according to the image dataset of the current task;

[0044] S22, initializing the external auxiliary image data set;

[0045] S23, randomly extract some samples from the training set (this embodiment uses the Episodic training strategy to randomly extract some samples from the training set), and form mixed sub-task image sample pairs with the initialized external auxiliary image dataset;

[0046] S24, completing the training of the GNN according to the subtask image sample pairs to obtain a trained GNN, specifically, using the update of the edges and nodes of the GNN, through multiple learning and optimization, to train the embedding parameters to obtain a trained GNN;

[0047] S3, input the training set and the test set into the trained GNN respectively to perform feature embedding, and obtain the training set image features and the test set image features respectively through feature encoding;

[0048] S4. According to the image features of the training set, select the effective sample feature class prototype from the training set, specifically:

[0049] S41, calculating the weighted class prototype of the training set samples by using a weighted mean method according to the training set image features;

[0050] S42. According to the weighted class prototype, filter out the valid sample feature class prototype from the training set:

[0051] The weighted class prototype is used as the task-related feature representation, and the task-related feature representation is used to screen the training set samples of the current task to distinguish between valid sample features and interference sample features (using Boolean logic, "1" represents valid sample features, and "0" represents interference sample features), and then the interference sample features are eliminated, and only the valid sample features are retained;

[0052] S5. Input the test set image features and the valid sample feature class prototypes into CNN to obtain the similarity between the test set image samples and each type of prototype. Specifically: use CNN as a dynamic measurement network, input the test set image features and the valid sample feature class prototypes into CNN, and dynamically learn the measurement parameters in CNN. As the measurement parameters in CNN are continuously adjusted and updated, the network parameters and prototype weights are used to continuously calculate and update the similarity measurement between the test set image features and each type of prototype in the training set. When calculating the measurement for the current task test set image samples, the image sample features are continuously used to correct the weight values ​​of the image class prototypes of each category to make the prototype more accurate, and finally obtain the similarity between each test set image sample and each type of prototype.

[0053] S6. According to the similarity between the image samples in the test set and each type of prototype, the image samples in the test set are classified into categories. Specifically, the similarities between the image samples in the test set and each type of prototype are sorted according to the numerical values. Among them, the class prototype corresponding to the maximum similarity is the determination category of the image samples in the test set.

[0054] This embodiment applies the above technical solution, such as Figure 3 As shown, it mainly includes the following contents:

[0055] I. According to the image dataset of the current task, select a sufficient amount of appropriate external auxiliary image datasets, initialize them, and divide the initialized current task dataset into a training set and a test set;

[0056] II. Adopt GNN as the network architecture (embedding module) for feature embedding, and then use the Episodic training strategy to randomly sample part of the training set samples and external auxiliary image datasets to form mixed sub-task image sample pairs, and use the GNN edge and node updates to learn and optimize the embedding parameters multiple times;

[0057] III. The image samples in the training set and the test set in the current task image small sample set are also combined into sample pairs, and feature embedding is performed through the trained GNN to encode them into sample feature coding pairs. At the same time, the weighted mean method is used instead of the direct mean method to calculate the weighted class prototype of each class in the feature coding pair of the training set samples as the task-related feature representation. The task-related feature representation is used to screen the training set samples of the current task to screen out more effective image sample features. Using Boolean logic, "1" represents effective image sample features and "0" represents interference sample features. At the same time, invalid (redundant) interference features are removed;

[0058] IV. Using CNN as a dynamic measurement network (function), the image feature encodings selected by the class prototypes of each category are mapped to the dynamic measurement network, and the measurement parameters in CNN are dynamically learned. As the learning of CNN parameters is constantly adjusted and updated, the network parameters and prototype weights are used to continuously calculate and update the similarity measurement between the image features of the test set and the class prototypes of each class of training set. When calculating the measurement for the image samples of the test set of the current task, the image sample features are continuously used to correct the weight values ​​of the image class prototypes of each category to make the prototypes more accurate, and finally the similarity between the image samples of each test set and the prototypes of each category is obtained;

[0059] V. Finally, compare the similarity measures of all known image class prototypes and each test set sample, sort them, and then determine the category of each image sample (unknown class sample) in the test set to complete the classification of the image sample.

[0060] In summary, the technical solution mainly improves and adjusts the embedding module and the metric module of the original prototype network to make the improved prototype network more representative and generalized for the learning and classification of small sample data sets of images. First, considering that the existing prototype network adopts a simple CNN network architecture with a small number of layers, the extracted image features are poorly represented, and the preservation and expression capabilities of image features are general, and it is easy to lose important image features (task-related features), and it is difficult to distinguish image samples of different categories well. In addition, the existing prototype network does not allow the image sample set in the current task to participate in the training of the feature embedding network without considering putting the image sample of the current task into the feature embedding network, which easily leads to the trained feature embedding network not being able to adapt well to the image data of the task, poor applicability, and not being able to capture the effective features related to the task in the task image data well; second, when the amount of data is sufficient, the use of "prototype" may achieve better results, but calculating the prototype is not an easy task. The prototype is affected by many factors. For example, in fact, in most real application fields, the existing "prototype" calculation method (simple mean) is limited by the amount of data, and when the amount of data is small, there will be inevitable boundary deviations. In addition, the available image samples will always contain some interfering data that is irrelevant to the task, which will seriously interfere with the calculation of the prototype, and these will reduce the accuracy of the prototype calculation; thirdly, the interpretability of using only a single distance metric to measure the similarity between samples is generally limited, and is only applicable to some simple ideal image data. However, in actual applications, most image data sets are relatively complex and have a small amount of data. The task relevance of different image sample sets is low. Judging whether image samples are similar and classifying them based solely on the distance is low in interpretability, and it is very likely to result in misclassification.

[0061] Based on the existing prototype network, this technical solution improves and optimizes its embedding module, prototype calculation method and measurement module. First, GNN is selected to replace CNN as the feature embedding network, taking advantage of the graph structure to help save more effective sample feature information, and using the loop update of edges and nodes to accelerate the learning and training of better embedding parameters to obtain better sample feature representation and improve the feature embedding ability of samples; second, the image data set of the current task is combined when training the feature embedding network to obtain important features related to the task; third, different weights are given to class prototypes of different categories as a means of judging the importance of samples of different categories, which helps to consider all picture samples and remove interfering picture sample features when classifying samples later, ensuring that all picture samples are classified and reducing misclassification; fourth, in view of the original simple fixed metric-Euclidean distance to calculate the similarity metric, CNN is used to continuously and dynamically train and learn the metric function to calculate the similarity metric of samples, further improving the efficiency and accuracy of sample metric, especially for complex or rare picture samples.

[0062] In order to verify the effectiveness of this technical solution, this embodiment uses the traditional prototype network and the improved prototype network to classify images. Figure 4 As shown, taking binary classification as an example, black and white represent the areas where the two types of samples are located, and samples of different colors appearing in the area represent that the samples are misclassified). The schematic diagram on the left represents the existing prototype network, and it can be seen that many sample points are classified into samples of the wrong category (white circle samples in the black area, black circle samples in the white area). In the schematic diagram on the right, the improved prototype network of the present invention, compared with the schematic diagram on the left, can be seen that only a few sample points are classified into samples of the wrong category (fewer white circle samples in the black area, and fewer black circle samples in the white area). In summary, the effect of the existing prototype network sample category division is general, and it is easy to produce a large classification error, while the improved prototype network of the present invention reduces the error, improves the classification effect, and makes the classification error within a controllable range.

Claims

1. A small sample image classification method based on an improved prototype network, characterized in that: The following steps are involved: S1, initialize the image dataset of the current task, and divide the initialized image dataset of the current task into a training set and a test set; S2. Use the training set and combine it with the external auxiliary data set to train the GNN to obtain a trained GNN; S3, input the training set and the test set into the trained GNN respectively to perform feature embedding, and obtain the training set image features and the test set image features respectively through feature encoding; S4, based on the image features of the training set, screen out valid sample feature class prototypes from the training set; S5, input the test set image features and the valid sample feature class prototypes into CNN to obtain the similarity between the test set image samples and each type of prototype; S6. According to the similarity between the image samples in the test set and various prototypes, the image samples in the test set are classified into categories.

2. The small sample image classification method based on improved prototype network according to claim 1, characterized in that: The training set includes a plurality of image samples of known categories, and the test set includes a plurality of image samples of unknown categories.

3. The small sample image classification method based on improved prototype network according to claim 1, characterized in that: The step S2 specifically includes the following steps: S21. Select a corresponding external auxiliary image dataset according to the image dataset of the current task; S22, initializing the external auxiliary image data set; S23, randomly extract some samples from the training set and form mixed sub-task image sample pairs with the initialized external auxiliary image dataset; S24. According to the subtask image sample pairs, complete the training of the GNN to obtain a trained GNN.

4. The small sample image classification method based on improved prototype network according to claim 3 is characterized in that: The step S23 specifically uses the training strategy of Episodic training to randomly extract some samples from the training set.

5. The small sample image classification method based on improved prototype network according to claim 3 is characterized in that: The step S24 specifically trains the embedding parameters based on the subtask image sample pairs, utilizing the updates of the edges and nodes of the GNN, through multiple learning and optimization, to obtain a trained GNN.

6. The small sample image classification method based on improved prototype network according to claim 2, characterized in that: The step S4 specifically comprises the following steps: S41, calculating the weighted class prototype of the training set samples by using a weighted mean method according to the training set image features; S42. Filter out valid sample feature class prototypes from the training set based on the weighted class prototypes.

7. The small sample image classification method based on improved prototype network according to claim 6, characterized in that: The step S42 specifically uses the weighted class prototype as a task-related feature representation, and uses the task-related feature representation to screen the training set samples of the current task to distinguish between valid sample features and interfering sample features, and removes the interfering sample features to retain only valid sample features.

8. The small sample image classification method based on improved prototype network according to claim 7, characterized in that: The step S42 specifically adopts a Boolean logic method to distinguish between valid sample features and interference sample features, wherein "1" represents a valid sample feature and "0" represents an interference sample feature.

9. The small sample image classification method based on improved prototype network according to claim 6, characterized in that: The specific process of step S5 is as follows: using CNN as a dynamic measurement network, inputting the test set image features and the valid sample feature class prototypes into CNN, and dynamically learning the measurement parameters in CNN. As the measurement parameters in CNN are continuously adjusted and updated, the network parameters and prototype weights are used to continuously calculate and update the similarity measurement between the test set image features and various prototypes in the training set. When calculating the measurement for the current task test set image samples, the image sample features are continuously used to correct the weight values ​​of the image class prototypes of each category to make the prototypes more accurate, and finally the similarity between each test set image sample and each prototype is obtained.

10. The small sample image classification method based on improved prototype network according to claim 9, characterized in that: The specific process of step S6 is: sorting the similarities between the test set image samples and each type of prototype according to the numerical value, wherein the class prototype corresponding to the maximum similarity is the determination category of the test set image samples.

Images