Small sample image classification method and device based on prototype neighbor network

Abstract

The invention discloses a small sample image classification method and device based on a prototype neighbor network. Firstly, data sets are divided into an auxiliary data set and a target data set, then the two data sets are divided into a support set and a query set, and feature extraction is performed on samples through a neural network to obtain high-dimensional features. And then calculating confidence and classification results of the prototype network and the K neighbor, and determining the category of the query sample through the confidence and the classification results. And then calculating the loss of the prototype network and the K-nearest neighbor, and combining the loss to optimize features and extract network parameters so as to improve the classification performance. And finally, carrying out image classification by utilizing the trained model. According to the method, the problems that a prototype obtained from a small amount of annotation data by a prototype network lacks representativeness and complex category distribution cannot be captured by a single-peak prototype hypothesis are solved, and a more optimized measurement space is learned through complementarity of the prototype network and the K-nearest neighbor algorithm; and the samples can be effectively classified only through a small number of support samples under complex data distribution.

Description

Technical Field

[0001] The present invention relates to a small sample image classification algorithm, and in particular to a small sample image classification method and device based on a prototype nearest neighbor network, belonging to the technical field of computer vision. Background Art

[0002] In the past decade, deep neural networks have made remarkable progress in various tasks such as image classification and speech recognition. However, the data-driven nature of deep learning is often limited by the shortage of labeled data, mainly due to the high cost of annotation and privacy issues. To address the reliance on a large amount of labeled data, few-shot classification (FSC) enables the classifier to adapt to unseen categories with only a small number of samples. In recent years, more and more FSC methods have been proposed, which can be roughly divided into three categories: (1) data-level methods, which directly generate new labeled data from original labeled data, unlabeled data, or labeled data from similar fields; (2) model-level methods, which use prior knowledge to reduce the size of the hypothesis space so that a reliable classifier can be learned with a small amount of labeled data; (3) algorithm-level methods, which efficiently search for the optimal hypothesis by providing appropriate initialization or guiding the search process.

[0003] Among the model-level methods, metric-based FSC methods focus on learning a low-dimensional metric space so that samples of the same category are closely clustered together, while samples of different categories are far away from each other. Therefore, a small hypothesis space can be constructed and a classifier can be learned using only a small amount of labeled data. Prototypical Networks (PN) is a typical example of this type. It forms a prototype by calculating the mean of the support samples (i.e., labeled data) of each category, and then classifies the query sample (i.e., test data) by identifying the distance between the query sample (i.e., test data) and the category prototype. Then, the metric space is learned by minimizing the distance between the query sample and its corresponding prototype while maximizing the distance with other prototypes. In such a metric space, PN can effectively learn a simple and reliable model using only a small number of samples. Due to its simplicity and effectiveness, PN has attracted increasing attention and spawned a variety of related algorithms. It should be noted that prototype-based methods usually assume that the distribution of categories is spherical and the categories are of similar size. However, many real-world categories are of different sizes, irregular shapes, or consist of different subcategories, which reduces the discriminative and expressive power of the prototype, and the reliance on the unimodal prototype assumption limits the adaptability of FSC and makes the model prone to underfitting problems.

[0004] Inspired by the K-nearest neighbor classification algorithm (KNN), the present invention introduces the nearest neighbor relationship to overcome the limitations of the prototype-based method. Specifically, the nearest neighbor relationship provides an additional assumption for PN: the category of a sample is usually consistent with the category of most of its neighbors. Therefore, when classifying samples, in addition to considering the distance from the prototype of each category, the relationship between the sample and its neighbors should also be considered comprehensively. The present invention calls this fused algorithm architecture Prototype-Neighbor Networks (PNN). PNN combines the complementary characteristics of PN and KNN, and can learn a more optimized metric space through a small number of labeled samples under complex data distribution, thereby effectively performing classification. Summary of the invention

[0005] In order to solve the deficiencies in the prior art, this application proposes a small sample image classification method based on a prototype nearest neighbor network. It is possible to learn a more optimized metric space by combining the complementarity of PN and KNN, so that the model can still effectively classify samples with only a small number of support samples under complex data distribution.

[0006] The present invention achieves the above technical objectives through the following technical means.

[0007] A small sample image classification method based on a prototype nearest neighbor network, comprising:

[0008] Step 1: divide the data set into an auxiliary data set and a target data set, further divide the auxiliary data set and the target data set into a support set and a query set, and extract features from all samples in the support set and the query set through a neural network to obtain a high-dimensional feature vector;

[0009] Step 2: Based on the high-dimensional feature vector, the confidence and classification results of PN and KNN of the query sample are calculated respectively, and the category to which the query sample belongs is determined by combining the confidence and classification results of PN and KNN;

[0010] Step 3, calculate the losses of PN and KNN, and then use the losses of PN and KNN to optimize the parameters of the feature extraction network (i.e., the neural network in step 1) to improve the classification performance;

[0011] Step 4: Finally, use the trained PNN model to perform image classification.

[0012] In the first aspect, the specific steps of the above step 1 are as follows:

[0013] Step 1.1, first you need to divide the available dataset into two main parts: target dataset D T and auxiliary dataset DA Among them, the target dataset is the data that the model relies on to solve the task in practical applications, while the auxiliary dataset is used to support the training and optimization of the model.

[0014] The category sets of these two data sets do not overlap, that is, their category label sets Y A and Y T Disjoint. The purpose of this is to ensure that the model's learning on the target dataset will not be disturbed by the categories in the auxiliary dataset, so that the model can focus on learning the category features required for the target task. In addition, this division method also allows the categories of the auxiliary dataset to be regarded as some "external" categories, which plays a role in providing more background knowledge for the target dataset, similar to providing additional prior knowledge for the model.

[0015] This division is done in the early stages of task design, and the way the dataset is divided directly affects the effectiveness of subsequent model training. By ensuring that the auxiliary dataset does not overlap with the target dataset categories, the model can learn a wider range of category representations, which helps improve its generalization ability on unknown categories.

[0016] In step 1.2, the dataset is further divided into a support set S and a query set Q. Each task starts from the target dataset D T and auxiliary dataset D A Random sampling in the form of a C-way K-shot task. Specifically:

[0017] Support set S: Each support set contains C categories, and each category has K samples. The support set is used to train the classifier and provide basic information for each category. Each category has a fixed number of samples that can represent the diversity of the category and are used to train the classifier to learn the characteristics of each category.

[0018] Query set Q: The query set is used to evaluate the classification performance of the model, and it contains samples from outside the support set. The query set has the same categories as the support set samples, but these samples are new and unseen samples used to test the performance of the model in real scenarios.

[0019] Each task is a C-way K-shot task, that is, the goal of the task is to learn how to distinguish C categories, and each category provides K samples. In this way, each task will be given a limited number of samples (K-shot) for the model to train and learn, and then classify the categories on this basis. In this way, the model can be effectively trained and inferred in the context of few samples.

[0020] In practice, this division is of great significance because it enables the model to continuously learn in one task and test in another task. In this way, the model can not only learn how to infer the correct category from a limited number of samples, but also continuously optimize and adjust its classification ability through training on multiple tasks.

[0021] Step 1.3: Send the image sample to the feature extraction network f θ Extract features in , where θ represents the parameters of the feature extraction network. Specifically, this process can be divided into the following sub-steps:

[0022] First, make sure that the image samples have been preprocessed. Preprocessing includes image size normalization, which is to adjust the original images of different sizes to the specific input size required by the feature extraction network, such as scaling images of various sizes to 64*64 pixels to fit the network input specification. At the same time, it also includes color space conversion, such as converting RGB images to grayscale images, or standardizing data so that the mean and standard deviation of image data conform to a specific distribution, thereby improving the stability and efficiency of network training.

[0023] Then, the preprocessed image samples are formally input into the feature extraction network f θ In the network, a series of convolutional layers, pooling layers, and nonlinear activation functions are used. In the convolutional layer, different convolutional kernels are used to perform convolution operations with the input image to extract local features of the image at different scales and directions. For example, a smaller convolutional kernel is used to capture detailed information of the image, such as edges and textures, while a larger convolutional kernel is used to focus on the overall structure of the image. The pooling layer is used to reduce the dimension of the feature map output by the convolutional layer. By retaining the main features and reducing the amount of data, it can not only reduce the computational complexity but also prevent overfitting to a certain extent. The pooling methods used include maximum pooling and average pooling. Maximum pooling selects the maximum value in the local area of ​​the feature map as the output, while average pooling calculates the average value of the local area. Nonlinear activation functions such as ReLU (Rectified Linear Unit) introduce nonlinear characteristics to the network, enabling the network to learn more complex functional relationships and enhance the network's expressive power. As image samples are passed layer by layer in the feature extraction network, the network will gradually extract features from low-level to high-level. Low-level features may be simple basic elements such as edges and corners, while high-level features may represent specific object categories, semantic information, etc. in the image. Finally, after being processed by the feature extraction network, a set of representative feature vectors are output, which will serve as an important input basis for subsequent tasks (such as classification, detection, segmentation, etc.) for further analysis and decision-making.

[0024] In general, the purpose of step 1 is to lay the foundation for the subsequent few-sample classification task through reasonable data set division and feature extraction process. By ensuring that the data set is divided reasonably and through effective feature extraction methods, the model can be effectively trained on the basis of a small number of samples, thereby achieving better classification performance in practical applications.

[0025] In the second aspect, the specific steps of the above step 2 are as follows:

[0026] Step 2.1, the prototype neighbor network PNN first calculates the mean vector of all supporting samples in each category k. This mean vector is called the prototype of the category, which is obtained by averaging the representation of all supporting samples of the category in the feature space. In this way, the prototype of category k represents the "center" or "typical features" of the samples of this category in the feature space.

[0027] Once the prototype of each category is calculated, the category prediction of the query sample q depends on the distance between the query sample and all category prototypes. Specifically, the PNN model calculates the distance between the query sample q and each category prototype, using the Euclidean distance or other appropriate metric. Then, these distances are weighted using an exponential function, so that prototypes with smaller distances can have a greater impact on the final prediction result.

[0028] Through the weighted distance value, the PNN model can predict which category the sample belongs to based on the similarity between the query sample and the prototype of each category. After weighting the distance, the prediction probability of all categories is adjusted to the range of 0 to 1 using the normalization operation to ensure that the sum of the prediction results of all categories is 1. In this way, the final category prediction is based on the similarity between the query sample and the prototype of each category, and the most likely category is determined by the size of the probability value.

[0029] Step 2.2, use the K-nearest neighbor (KNN) method to classify based on the distance between the query sample and all support samples. First, the PNN model calculates the distance between the query sample and each support sample, which is achieved by the output feature representation of the neural network, that is, calculating the distance in the feature space. These distances reflect the similarity between the query sample and each support sample in the feature space.

[0030] Then, the KNN algorithm will select the K support samples that are closest to the query sample. The choice of K value has an important impact on the classification effect. A smaller K value may make the model sensitive to noise, while a larger K value may make the model lose its sensitivity to local features. Therefore, choosing a suitable K value is the key to the effect of the KNN algorithm. The K value of the present invention is 5.

[0031] After selecting K nearest neighbor samples, the PNN model will determine the category of the query sample based on the categories of these neighbor samples. Specifically, the KNN algorithm will count the categories to which these K neighbor samples belong, and then select the category with the highest frequency as the predicted category of the query sample. The probability of this category can be calculated by the ratio of the number of samples belonging to this category in the neighbors to the total number of neighbors K, reflecting the credibility of the query sample belonging to a certain category.

[0032] Step 2.3 combines the prediction results of the two classification models, PN and KNN, to form the final classification decision of the prototype neighbor network (PNN). PN and KNN each have their own advantages. The PN method relies on the calculation of the prototype and the similarity measurement in the feature space, while KNN relies on direct neighbor sample information. When the prediction results of the two methods are consistent, PNN directly adopts the prediction results of KNN.

[0033] However, when the prediction results of PN and KNN are inconsistent, the PNN model combines the predictions of the two in a weighted manner. Specifically, the prediction probability of PN and the prediction probability of KNN will be weighted by a hyperparameter λ. The size of the hyperparameter λ determines the relative importance of the PN and KNN models in the final decision. A larger λ value can make the prediction results of PN dominate the decision, while a smaller λ value makes the prediction results of KNN more important.

[0034] Through this weighted decision-making mechanism, the PNN model can combine the advantages of the two models, not only improving the accuracy of classification, but also enhancing the robustness of the model in complex tasks. The introduction of the weighted mechanism enables the model to dynamically adjust its decision-making strategy in different situations and adapt to more changes and challenges.

[0035] Through the combination of the above three steps, the PNN model can effectively combine the advantages of the prototype network (PN) and the K nearest neighbor algorithm (KNN). When dealing with small sample classification tasks, the PNN model accurately calculates the distance between the category prototype and the query sample, combined with the neighbor sample information of KNN, and finally achieves efficient classification. The weighting mechanism enables the PNN model to flexibly deal with the inconsistency of the prediction results of PN and KNN, thereby providing more reliable and stable classification results.

[0036] In the third aspect, the specific steps of the above step 3 are as follows:

[0037] Step 3.1, the key technical point of the prototype network (called PN model) is to learn a low-dimensional metric space, the purpose of which is to make each sample as close as possible to the prototype of its category in this space, and away from the prototypes of other categories. This low-dimensional metric space can be regarded as an "embedding space" learned by the model, in which the separation between categories is clearer, which helps to improve classification accuracy.

[0038] To achieve this goal, the prototypical network is trained by optimizing a loss function. The core idea of ​​this loss function is to adjust the parameters of the prototypical network so that the representation of each sample is as close as possible to the prototype of the category to which it belongs. In turn, the prototypical network also requires that the sample be kept far away from the prototypes of other categories, which can effectively avoid confusion between categories. In order to minimize this classification error during training, the prototypical network calculates the category prediction probability of each sample, takes the logarithm of these probabilities, and then sums them. Through this process, the prototypical network can calculate its classification error on the support set S and the query set Q, and adjust the network parameters accordingly, so that the model gradually converges to an optimized metric space.

[0039] Specifically, the support set S provides samples and their true category labels, while the query set Q provides samples that need to be predicted. By continuously iteratively optimizing the loss function, the prototype network can find a better category boundary in the low-dimensional space, thereby improving the classification effect.

[0040] Step 3.2, the loss function calculation of the KNN algorithm (called the KNN model) is similar to that of the PN model, and both optimize the model by minimizing the classification error. In the KNN model, the classification error is calculated mainly through the distance between the query sample and the support sample. The class prediction probability of each support sample is determined by the distance from the query sample. The closer the neighbor sample is, the greater the impact on the final classification.

[0041] Similar to the PN model, the KNN model also calculates the category prediction probability of each sample, and then takes the logarithm of these probabilities and sums them up. The purpose of this is to encourage the model to reduce the possibility of misclassification and make category predictions more accurate. Ultimately, the KNN model minimizes the classification error on the support set S and query set Q by adjusting the network parameters.

[0042] The core advantage of the KNN model is that it makes predictions based on the categories of neighboring samples, so it relies more on the distribution of local data, while the PN model focuses more on the establishment of global category prototypes. When the two are used in combination, they can make up for their respective limitations and thus obtain stronger classification capabilities.

[0043] Step 3.3, - The training process of the entire PNN model enters the meta-learning stage. In this stage, the model randomly selects A A batch of C-way K-shot tasks are sampled from . Each task has its specific support set S and query set Q. The selection of these tasks is done through meta-learning, with the goal of allowing the model to generalize training on multiple tasks, thereby improving its performance on new tasks.

[0044] For each task, the samples in the support set S are used to calculate the category prototype vector of PN, and KNN will calculate a distance matrix based on these support samples. The calculated prototype and distance matrix will be used for the classification process of PN and KNN respectively. After that, PN and KNN will classify and predict the query set Q according to the methods in steps 2.3, 3.1 and 3.2, and calculate the corresponding classification loss.

[0045] Finally, in the meta-training phase, the classification prediction results of PN and KNN will be combined and used to calculate the total loss of the final PNN model. This total loss not only takes into account the classification errors of PN and KNN, but also combines the performance of the two models. Through this process, the PNN model can continuously optimize its performance on the support set and query set, thereby improving its generalization ability.

[0046] The calculation of the total loss involves weighting the losses of the PN model and the KNN model. The weight setting can be adjusted according to actual needs. After each training iteration, the PNN model updates the network parameters through back propagation so that the model can perform more accurately in the next task. The goal of the meta-training phase is to enable the PNN model to have the ability to perform efficient classification in different categories and tasks through a large number of task training.

[0047] The core purpose of step 3 is to train the PNN model through meta-learning so that it can quickly adapt to new tasks and accurately classify. In this process, PN and KNN improve model performance by optimizing their respective loss functions, and through task sampling and optimization in the meta-training phase, PNN can generalize on multiple tasks and improve the classification ability of the model. By continuously adjusting the parameters of the PN and KNN models, PNN can eventually learn an efficient classification strategy and achieve excellent performance in different tasks.

[0048] In the fourth aspect, the specific steps of the above step 4 are as follows:

[0049] Given the embedding network f learned through the meta-training phase θ and the target dataset D T, the present invention enters the actual testing phase. At this point, the model has learned an ability to quickly adapt to new tasks by training on multiple tasks. In this phase, the model is trained by extracting the target dataset D T A batch of C-way K-shot tasks are randomly sampled to verify its effect in practical applications.

[0050] For each new task, the samples in the support set S will be used to calculate the prototype vector and distance matrix of PN and KNN respectively. The samples in the support set provide prior information about the category for the query sample. The PN network represents the center of each category by calculating the prototype vector of the category, while KNN finds the nearest support sample based on the distance matrix. The results of PN and KNN are then combined to classify the samples in the query set Q.

[0051] The key to this process is the complementarity of PN and KNN in the metric space. PN provides a global representation for each category through the learned prototype vector, helping the model to determine the category center of a sample, while KNN determines the label of the query sample through samples in the local neighborhood. This combination of local and global information enables PNN to make accurate predictions in classification tasks using a small amount of labeled data.

[0052] Based on the above method, the present invention also proposes a computer-type device, including a memory and a processor, wherein the memory stores a program code corresponding to the above image classification method, and the processor can execute the above image classification method when executing the program code in the memory.

[0053] Beneficial effects of the present invention:

[0054] (1) Optimized metric space learning: Traditional metric learning methods usually rely on global distance metrics, while PNN establishes a more refined and optimized representation in the metric space by combining two different learning strategies, PN and KNN. PN effectively captures the global relationship between categories by learning prototype vectors, while KNN supplements local information through neighboring samples. This combination helps to build a more expressive metric space.

[0055] (2) Effective use of small amounts of labeled data: In many practical problems, obtaining a large amount of labeled data is a challenge. PNN, through its unique mechanism, can build a reliable FSC (Few-Shot Classification) model using only a small amount of labeled data. This means that PNN can handle small sample learning tasks more effectively in practical applications, especially in scenarios where data is scarce or expensive.

[0056] (3) Complementarity improves classification accuracy: PN and KNN optimize classification tasks from different perspectives. PN starts from the prototype of the category and helps the classification model learn the relationship between categories globally; while KNN provides more local information for the classification task by considering the nearest samples. The combination of the two not only improves the accuracy of the model, but also enables PNN to maintain good performance in a variety of tasks, especially when facing new tasks, it can quickly adapt and make efficient decisions.

[0057] (4) Lower computational complexity: Although PN and KNN seem to require more distance or prototype information to be calculated, PNN still has lower computational complexity compared to traditional deep learning methods, especially in resource-constrained environments when actually deployed. This low complexity makes PNN practically feasible in large-scale applications.

[0058] (5) Strong adaptability and generalization ability: With the help of meta-learning, PNN can learn an embedding space that is widely applicable to different tasks. This ability makes PNN have strong generalization ability. During the training process, the model is not limited to a specific task, but learns how to effectively classify in various scenarios through training on multiple tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 Flowchart of the small sample image classification method based on the prototype nearest neighbor network.

[0060] Figure 2 A detailed description of the prototype neighbor network during meta-training.

[0061] Figure 3 A detailed description of the prototype neighbor network during meta-testing. DETAILED DESCRIPTION

[0062] The present invention will be further described below in conjunction with the accompanying drawings.

[0063] like Figure 1 As shown, the present invention proposes a small sample image classification method based on a prototype nearest neighbor network, and its specific implementation steps are as follows:

[0064] Step S1 divides the data set into an auxiliary data set and a target data set, further divides the auxiliary data set and the target data set into a support set and a query set, and extracts features from the samples through a neural network to obtain a high-dimensional feature vector.

[0065] Step S11 divides the dataset into two mutually exclusive parts according to the standard small sample classification setting: target dataset D T and auxiliary dataset D AAmong them, the target data set D T Contains a small amount of labeled data, and the auxiliary dataset D A Contains a large amount of labeled data. It should be noted that D A and D T The category sets of are non-overlapping, that is, their category label sets Y A and Y T are disjoint ( represents the empty set). The goal of this setting is to utilize the A Prior knowledge, learn a classification model M for the target dataset D T Under the meta-learning paradigm, this learning process consists of two stages:

[0066] 1. Meta-training stage: aims to A Pre-train the model M to obtain prior knowledge from it.

[0067] 2. Meta-testing phase: quickly adapting the model M to the target dataset D T , using the prior knowledge obtained in the first stage to complete the classification task.

[0068] Step S12 uses an episodic training mechanism to randomly select a small number of samples in each task. For each task, a support set S = {(x 1 ,y 1 ),…,(x C*K ,y C*K )} and a query set Q = {(x C*K+1 ,y C*K+1 ),…,(x C*K+C*N ,y C*K+C*N )}, where x represents the sample, y corresponds to the sample label, and the superscripts of x and y represent the sequence number of the sample in the set. Each S contains C categories, and each category has K samples. Let S k represents a subset of S where all samples belong to category k. Note that S and Q have no intersection, but they share the same category label space. In the meta-training phase, S and Q are derived from D A In the meta-test phase, S and Q are sampled from D T where the labels of Q are invisible.

[0069] Step S13 sends the image sample to the feature extraction network f θ , extract the characteristics

[0070] , θ represents the parameters of the feature extraction network.

[0071] Step S2 calculates the confidence and classification results of PN and KNN respectively, and determines the category to which the query sample belongs by combining the confidence and classification results of PN and KNN;

[0072] Step S21 In the prototype network, for each category k, the mean vector of all supporting samples in the category is calculated as the prototype c k :

[0073]

[0074] x i Indicates S k The elements in .

[0075] Given a distance function d, the prototype network assigns the query sample x to the category closest to its prototype. The category probability distribution p of the query sample is obtained by calculating the distance between the query sample x and each prototype in the metric space. θ:PN (y=k|x):

[0076]

[0077] c k′ Represents the prototype of category k'.

[0078] Step S22 calculates the query sample q and each support sample s i The distance d(f θ (q),f θ (s i )) and select the K supporting samples closest to q. Then, the query sample q is classified according to the majority class of these K nearest neighbors. Let KNN(q; y=j) represent the number of samples belonging to class j among the K nearest neighbors of q. In the standard KNN algorithm, the probability that q belongs to class j is p(y=j|q):

[0079]

[0080] When the number of samples from multiple categories in the K nearest neighbors of q is the same, classification conflicts may occur. Let K(q) represent the set of K nearest neighbors of q, K j (q) represents the subset of the K nearest neighbors of q that belong to category j. In order to solve this problem, the present invention considers the distances of these nearest neighbors and modifies the formula in the metric space as follows:

[0081]

[0082] s i 、s j They represent the sets K(q) and K j (q) element, p θ:KNN(y=j|q) represents the probability that the label y of a sample is j given a query sample q based on the parameter θ under the KNN model.

[0083] Step S23 combines the two classification models of PN and KNN, and PNN classifies the query sample q∈Q according to the following formula:

[0084]

[0085] Among them, y θ:PN (q) and y θ:KNN (q) represents the predicted category label of q by PN and KNN respectively. θ:PN (y|q) and p θ:KNN (y|q) represents the predicted probability of q by PN and KNN respectively. θ:PN (y|q) and p θ:KNN The value range of (y|q) may be different, so it needs to be normalized before using the above formula. The hyperparameter λ∈[0,1] is used to adjust the weights of PN and KNN.

[0086] Step S3 calculates the PN loss and the KNN loss, and then uses the PN and KNN losses to optimize the parameters of the feature extraction network to improve the classification performance;

[0087] Step S31 The key to PN is to learn a low-dimensional metric space so that each sample is closer to the prototype of its category and away from the prototypes of other categories. The loss function of PN is defined as:

[0088]

[0089] p θ:PN (y i |x i ) means that based on the parameter θ and under the PN model, given the input sample x i When , the output label y i probability.

[0090] Step S32 The loss function of KNN is defined as:

[0091]

[0092] p θ:KNN (y i |x i ) means that based on the parameter θ and under the KNN model, given the input sample x i When , the output label y i probability.

[0093] Step S33 In the meta-training stage, the present invention randomly selects D AA large number of C-way K-shot tasks are sampled in . For each task, the prototype vector of PN and the distance matrix of NN are calculated using the support set S. Subsequently, PN and KNN classify the query set Q and calculate the corresponding losses using the formulas in steps 2.3, 3.1, and 3.2. Finally, the prediction results and total loss of the query set are output by the combined module and used to iteratively update f θ , the total loss is:

[0094] L PNN (S,Q;θ)=L PN (S,Q;θ)+L KNN (S,Q;θ)

[0095] By learning through the gradient descent algorithm, the goal of the PNN model is to minimize the total loss:

[0096]

[0097] Step S4 gives the embedding network f learned through the meta-training phase θ and the target dataset D T The present invention is from D T A batch of C-way K-shot tasks are randomly sampled from . For each new task, the prototype vector and distance matrix of PN and KNN are calculated using the support set S, respectively. Subsequently, the results of PN and KNN are combined to classify the query set Q.

[0098] Figure 2 The specific details of the prototype nearest neighbor network in the meta-training stage are shown. First, the auxiliary data set is divided into a support set and a query set, and the features of the samples are extracted through the embedding network to obtain a high-dimensional feature vector. Next, the confidence, classification results and loss of PN and KNN are calculated respectively, and the category of the query sample is determined by combining the confidence and classification results of PN and KNN. Then, the loss of PN and KNN is used to optimize the parameters of the feature extraction network to improve the classification performance.

[0099] Figure 3 The specific details of the prototype nearest neighbor network in the meta-testing phase are shown. First, the target dataset is divided into a support set and a query set. The image samples are extracted into high-dimensional vectors through the embedding network learned in the meta-training phase. Next, the confidence and classification results of PN and KNN are calculated respectively, and finally classification is performed by combining the confidence and classification results of PN and KNN.

[0100] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. All equivalent methods or changes that do not deviate from the technical creation of the present invention should be included in the scope of protection of the present invention.

Claims

1. A small sample image classification method based on prototype nearest neighbor network, characterized in that: These include: Step 1: divide the data set into an auxiliary data set and a target data set, and then divide the auxiliary data set and the target data set into a support set and a query set, extract features from the image samples through a neural network, and obtain a high-dimensional feature vector; Step 2: Based on the high-dimensional feature vector, the confidence and classification results of the prototype network PN and KNN are calculated respectively, and the category of the query sample is determined by combining the confidence and classification results of PN and KNN; the final classification decision of the prototype neighbor network (PNN model) is formed by combining the prediction results of PN and KNN; Step 3, calculate the losses of PN and KNN, and then use the losses of PN and KNN to optimize the parameters of the feature extraction network to improve the classification performance; in this process, the total loss of the PNN model is obtained; Step 4: Finally, use the trained PNN model to perform image classification.

2. The small sample image classification method based on prototype nearest neighbor network according to claim 1 is characterized in that: The specific process of step 1 includes: Step 1.1: Divide the available dataset into two parts: target dataset D T and auxiliary dataset D A ; Among them, the target data set D T It is the data that the model relies on to solve the task in practical application, and the auxiliary data set D A It is used to support model training and optimization; Among them, the target dataset D T and auxiliary dataset D A The category sets of are non-overlapping, that is, their category label sets Y A and Y T do not intersect; Step 1.2: further divide the dataset into support set S and query set Q; each task starts from the target dataset D T and auxiliary dataset D A Random sampling in the ,forms a C-way K-shot task; Step 1.3: Send the image sample to the feature extraction network f θ Features are extracted in , where θ represents the parameters of the feature extraction network.

3. The small sample image classification method based on prototype nearest neighbor network according to claim 2 is characterized in that: The step 1.2 is specifically as follows: The support set S: Each support set contains C categories, each category has K samples, and the support set is used to train the classifier, providing basic information of each category. Each category has a fixed number of samples, which can represent the diversity of the category and are used to train the classifier to learn the characteristics of each category; The query set Q: The query set is used to evaluate the classification performance of the model. It contains samples outside the support set. The query set has the same category as the support set samples, but these samples are new and unseen samples and are used to test the performance of the model in actual scenarios. Each of the tasks is a C-way K-shot task, that is, the goal of the task is to learn how to distinguish C categories, and each category provides K samples. In this way, each task will be given a limited number of samples (K-shot) for the model to train and learn, and then perform category classification on this basis.

4. The small sample image classification method based on prototype nearest neighbor network according to claim 2 is characterized in that: The step 1.3 is as follows: First, complete the image sample preprocessing, including image size normalization, color space conversion, and data standardization; Then, the preprocessed image samples are input into the feature extraction network f θ In; the network θ Includes convolutional layers, pooling layers, and nonlinear activation functions; In the convolution layer, different convolution kernels are used to perform convolution operations with the input image to extract local features of the image at different scales and directions. Smaller convolution kernels are used to capture the detailed information of the image, while larger convolution kernels are used to capture the overall structure of the image. The pooling layer is used to reduce the dimension of the feature map output by the convolution layer. By retaining the main features and reducing the amount of data, it can not only reduce the computational complexity but also prevent overfitting to a certain extent. The pooling methods include maximum pooling and average pooling. Maximum pooling is used to select the maximum value in the local area of ​​the feature map as the output, and average pooling is used to calculate the average value of the local area. Nonlinear activation function introduces nonlinear characteristics to the network, enabling the network to learn more complex functional relationships and enhance the network's expressiveness. As image samples are passed layer by layer in the feature extraction network, the network gradually extracts features from low-level to high-level. The low-level features include simple edges, corners and other basic elements, while the high-level features include specific object categories and semantic information in the image. Finally, after being processed by the feature extraction network, a set of representative feature vectors are output. These feature vectors will serve as important input basis for subsequent tasks for further analysis and decision-making.

5. The small sample image classification method based on prototype nearest neighbor network according to claim 1, characterized in that: The step 2 is specifically as follows: Step 2.1, the prototype nearest neighbor network first calculates the mean vector of all supporting samples in each category k. This mean vector is called the prototype of the category. It is obtained by averaging the representation of all supporting samples of the category in the feature space. In this way, the prototype of category k represents the "center" or "typical features" of the samples of the category in the feature space. Once the prototype of each category is calculated, the category prediction of the query sample q depends on the distance between the query sample and all category prototypes. Specifically, the model uses the Euclidean distance to calculate the distance between the query sample q and each category prototype. Then, these distances are weighted using an exponential function. Prototypes with smaller distances have a greater impact on the final prediction result. Using the weighted distance value, the model can predict which category the sample belongs to based on the similarity between the query sample and the prototype of each category; After weighting the distance, the prediction probabilities of all categories are adjusted to the range of 0 to 1 using the normalization operation to ensure that the sum of the prediction results of all categories is 1. In this way, the final category prediction is based on the similarity between the query sample and the prototype of each category, and the most likely category is determined by the size of the probability value. Step 2.2, using the K-nearest neighbor (KNN) method to classify based on the distance between the query sample and all support samples; first, the distance between the query sample and each support sample is calculated, which is achieved through the output feature representation of the deep network, that is, the distance in the feature space is calculated. These distances reflect the similarity between the query sample and each support sample in the feature space; Next, the KNN algorithm is used to select the K support samples closest to the query sample. After the K nearest neighbor samples are selected, the model determines the category of the query sample based on the categories of these neighbor samples. Specifically, the KNN algorithm counts the categories to which these K neighbor samples belong, and then selects the category with the highest frequency as the predicted category of the query sample. The probability of this category can be calculated by the ratio of the number of samples belonging to this category in the neighbors to the total number of neighbors K, which reflects the credibility of the query sample belonging to a certain category. Step 2.3, combine the prediction results of the two classification models PN and KNN to form the final classification decision of the prototype neighbor network (PNN); when the prediction results of PN and KNN are consistent, the prediction results of KNN are directly adopted; when the prediction results of PN and KNN are inconsistent, the predictions of the two are combined in a weighted manner. Specifically, the prediction probability of PN and the prediction probability of KNN are weighted and selected by a hyperparameter λ.

6. The small sample image classification method based on prototype nearest neighbor network according to claim 5, characterized in that: The step 3 is as follows: Step 3.1, the prototype network PN is trained by optimizing a loss function; this loss function is to adjust the parameters of the network to make the representation of each sample as close as possible to the prototype of the category to which it belongs. In turn, the prototype network also requires the sample to keep a distance from the prototypes of other categories to avoid confusion between categories; in order to minimize this classification error during training, the prototype network calculates the category prediction probability of each sample, takes the logarithm of these probabilities, and then performs a sum operation. Through this process, the prototype network can calculate its classification error on the support set S and the query set Q, and adjust the network parameters accordingly, so that the model gradually converges to an optimized metric space; Step 3.2, the calculation of the loss function of KNN is similar to that of PN, both of which optimize the model by minimizing the classification error; in KNN, the classification error is calculated by the distance between the query sample and the support sample. The category prediction probability of each support sample is determined by the distance to the query sample. The closer the neighbor sample is, the greater the impact on the final classification; Similar to the processing method of PN, KNN also calculates the category prediction probability of each sample, and then takes the logarithm of these probabilities and sums them. Finally, KNN minimizes the classification error on the support set S and the query set Q by adjusting the network parameters; Step 3.3: The training process of the entire model enters the meta-learning stage. In this stage, the model randomly extracts A A batch of C-way K-shot tasks are sampled in , each task has its specific support set S and query set Q. The selection of these tasks is carried out through meta-learning, with the aim of allowing the model to be generalized on multiple tasks and improve its performance on new tasks.

7. The small sample image classification method based on prototype nearest neighbor network according to claim 6 is characterized in that: In step 3.1, the support set S provides samples and labels of their true categories, while the query set Q provides samples that need to be predicted; by continuously iteratively optimizing the loss function, the prototype network can find a better category boundary in the low-dimensional space to improve the classification effect.

8. The small sample image classification method based on prototype nearest neighbor network according to claim 6 is characterized in that: In step 3.3, for each task, the samples in the support set S are used to calculate the category prototype vector of PN, and KNN calculates a distance matrix based on these support samples; the calculated prototype vector and distance matrix are used in the classification process of PN and KNN respectively, after which PN and KNN will classify and predict the query set Q according to the methods in steps 2.3, 3.1 and 3.2, and calculate the corresponding classification loss; In the meta-training stage, the final total loss is calculated by combining the classification prediction results of PN and KNN. This total loss not only takes into account the classification errors of PN and KNN, but also combines the performance of the two models. Through this process, the model can continuously optimize its performance on the support set and query set and improve its generalization ability. The calculation of the total loss involves weighting the losses of PN and KNN, and the weight setting can be adjusted according to actual needs; After each training iteration, the network parameters are updated through back propagation so that the model can perform more accurately in the next task. The goal of the meta-training stage is to enable the model to have the ability to efficiently classify in different categories and tasks through a large number of task training.

9. The small sample image classification method based on prototype nearest neighbor network according to claim 6, characterized in that: The step 4 is specifically as follows: The model is constructed by extracting the target dataset D T A batch of C-way K-shot tasks are randomly sampled to verify its effect in practical applications; For each new task, the samples in the support set S will be used to calculate the prototype vector and distance matrix of PN and KNN respectively. The samples in the support set S provide prior information of the category for the query sample. The PN network represents the center of each category by calculating the prototype vector of the category, while KNN finds the nearest support sample based on the distance matrix. The results of PN and KNN are then combined to classify the samples in the query set Q.

10. A computer device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores program codes corresponding to the methods of claims 1 to 9, and the processor can execute the methods of claims 1 to 9 when executing the program codes in the memory.

Images