Small-sample classification method based on visual reconstruction

Abstract

The invention aims to overcome the defects that a user can only see a training result, cannot see a relationship between a query set and a support set and cannot screen samples in the support set, and provides a visual method which is visual and vivid and can observe the influence of the support set on a classification result in a classification process. In order to solve the technical problem, the technical scheme of the invention is as follows: the small-sample classification method based on visual reconstruction is used for improving the quality of a support set and improving a classification result through an interaction method under the condition of few sample sets, and comprises the following steps: S1, reconstructing samples of a query set; s2, visual generation of a similar matrix of a support set sample; s3, supporting a recommendation algorithm of a set sample; and S4, performing visual generation of intra-class and inter-class aggregation degrees of the replacement process.

Description

Technical Field

[0001] The present invention relates to the field of image classification, and in particular to a reconstruction-based few-sample classification visualization method. Background Art

[0002] In real-world scenarios, some categories have only a small amount of data or a small amount of labeled data, and labeling unlabeled data will consume a lot of time and manpower. In contrast, humans can learn quickly with only a small amount of data. For example, a five or six-year-old child has never seen a penguin, but if he is shown an image of a penguin, when he enters the zoo and sees a real penguin, he will immediately recognize that it is the "penguin" he has seen in the image. This is the gap between machine learning and human learning. Inspired by the view of human learning, the concept of few-shot classification was proposed. In order to support few-shot classification, machine learning methods are often used. However, these methods have a problem that users can only see the training results, but cannot see the relationship between the query set and the support set, and cannot filter the samples in the support set. If some samples in the support set are of poor quality, it will also have a negative impact on the classification results. Therefore, by visualizing the relationship between the query set and the support set, people can better understand the classification role of each support set sample, which helps to filter the support set samples and improve the classification results.

[0003] How to visualize the impact of samples in the support set on the classification results is the most core element. Its main purpose is to link the support set and the query set, layout all sample information in a two-dimensional space, and retain the impact of the support set samples on the classification, so that users can analyze the data set in a more intuitive and effective way. In the current methods for few-sample classification, there is still a lack of a fully mature method available, but many machine learning methods have emerged. One is based on model fine-tuning. This method usually pre-trains the model on large-scale data, and fine-tunes the parameters of the fully connected layer or the top layers of the neural network model on the target small sample data set to obtain the fine-tuned model. If the distribution of the target data set and the source data set is similar, the model fine-tuning method can be used. The second is based on data enhancement. This method refers to data expansion or feature enhancement of the original small sample data set with the help of auxiliary data or auxiliary information. Data expansion is to add new data to the original data set. The third is based on transfer learning. This method uses old knowledge to learn new knowledge. The main goal is to quickly transfer the knowledge that has been learned to a new field.

[0004] Finally, although these technologies have achieved good results, there are still some shortcomings, such as overfitting caused by model fine-tuning. In terms of visualization, visualization for few samples is still missing, and there is also a lack of methods to link the query set and the support set so that users can continue to mine. Summary of the invention

[0005] The purpose of the present invention is to overcome the above-mentioned problem that users can only see the training results but cannot see the relationship between the query set and the support set and cannot filter the samples in the support set, and to provide an intuitive and visual classification method that can observe the impact of the support set on the classification results during the classification process.

[0006] In order to solve the above technical problems, the technical solution of the present invention is: a few-sample classification method based on visual reconstruction, which is used to improve the quality of the support set and the classification result through an interactive method when the sample set is small, including the following steps: S1. Reconstruction of query set samples: For a given support set and query set, use the Few-Shot Reconstruction Networks (FRN) reconstruction algorithm to reconstruct each sample in the query set through the support set and classify it through the new samples, because images from the same category should be easier to reconstruct because their feature maps contain similar embeddings, while images from different categories will be more difficult and produce larger reconstruction errors. Then use T-SNE to reduce the dimension and visualize it through a scatter plot. The core idea of ​​the T-SNE algorithm is to generate a representation of a low-dimensional space by maintaining the local similarities between data points in a high-dimensional space. For samples in the support set, the scatter information includes the classification of the scatter points around the point to improve the correlation between the support set and the query set during image classification; S2. Visual generation of similarity matrix of support set samples: including similarity matrices between multiple classes, and through the similarity matrix display, users can filter out support samples with poor classification effects for subsequent replacement operations; S3, recommendation algorithm for support set samples: After filtering out low-quality support samples in S2, the recommendation algorithm is used to sort the results in descending order, and high-quality samples are selected based on the recommendation score and scatter plot layout information. The low-quality samples filtered out in S2 are replaced with the high-quality samples, and steps S1 to S3 are repeated.

[0007] S4. Visual generation of intra-class and inter-class aggregation degree during the replacement process: By visualizing the aggregation degree within the class during the continuous replacement of support samples, the quality of the update operation is reflected, and the user can judge the correctness of the replacement operation based on this; Preferably, in step S1, the query set is reconstructed from the support set by the FRN algorithm through the visualization method of reconstruction-based few-sample classification defined in the present invention, so that the query set is closer to the target set, and a scatter plot is generated.

[0008] Preferably, in step S2, the similarity between samples is calculated by CKA, wherein CKA calculation is an existing commonly used method for calculating the similarity between two features. In the first case, if there is one or several samples in the class whose intra-class similarity scores are low, it means that the sample has poor feature similarity with other samples in the class, which often means that the sample may have features different from other samples in the class due to shooting angle or background. In this case, the quality of the sample is poor, otherwise the quality is better. In the second case, if there is one or several samples in class A and most of the samples in class B with high similarity scores, it means that the sample has good feature similarity with the samples in class B, which often means that if the sample is used as a support set, there is a greater probability that the query sample will be classified into class B during classification. In this case, the quality of the sample is poor, otherwise the quality is better.

[0009] Preferably, in step S3, the scatter point position information before and after reconstruction is obtained by the recommendation algorithm defined in the present invention, taking into account the two-dimensional area enclosed by the class and the number of positive samples under the area. If the enclosed area is larger, the number of positive samples under the area will also be more, otherwise it will be less. Therefore, the ratio of the area before reconstruction to the number of positive samples under the area is taken as the first factor; the ratio of the area after reconstruction to the number of positive samples under the area is taken as the second factor; in addition, considering the quality of the replacement element itself, it is reflected by the F1 score and the D score (F1-F2). If the F1 score and the D score are both higher, it means that the sample is more willing to believe that it is of this class and its characteristics are more obvious. Therefore, the F1 score is taken as the third factor; the D score (F1-F2) is taken as the fourth factor. The recommendation algorithm includes four indicators, and the recommendation scores of different samples are output according to different situations. The higher the score, the better the classification effect after the sample is replaced. Select samples from high to low, and observe the position information of the sample in the scatter plot. If there are already points in the support set sample around the point, it is not recommended, because most of the feature information of the point is already available, and a sample needs to be found to make up for the feature information that is not available in this class. According to this principle, select samples with high recommendation scores and no duplication of feature information of the point.

[0010] Preferably, in step S4, the average distance from each point in the same class to the class center point is used as the ordinate and the number of iterations is used as the abscissa to draw curves before and after reconstruction, showing the quality of classification results during the iterative replacement of samples.

[0011] The present invention has the following characteristics and beneficial effects: Although the existing few-sample classification algorithms have achieved certain results, these methods still lack the connection between the support set and the query set and the role of the modified samples in the support set in the classification process. The present invention reconstructs new samples from the query set through the support set through a reconstruction method, and classifies through the new samples, and displays the reconstructed results through a visualization method, so that in the classification process, the influence of the support set samples on the query set is displayed, solving the above-mentioned existing problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a flow chart of an embodiment of the present invention.

[0013] Figure 2 This is a comparison chart of the CUB dataset using the FRN algorithm in an embodiment of the present invention.

[0014] Figure 3 This is a diagram showing the effect of the CKA algorithm similarity according to an embodiment of the present invention.

[0015] Figure 4 This is a process indicator diagram for improving support samples according to an embodiment of the present invention. DETAILED DESCRIPTION

[0016] The invention is described in detail below with reference to the accompanying drawings and specific examples.

[0017] The technical solution of the present invention is: a visualization method for few-sample classification based on reconstruction, such as Figure 1 The following steps are shown: Reconstruction of query set samples: CUB (Caltech-UCSD Birds) dataset is a popular image dataset for bird recognition and classification. Ten classes are randomly selected, and 5 of them are randomly selected as the support set, and the rest are used as the query set. For a given support set and query set, the Few-Shot Reconstruction Networks (FRN) reconstruction algorithm is used to reconstruct each sample in the query set through the support set, and classify it through the new sample, and then reduce the dimension through T-SNE and visualize it through scatter plots.

[0018] Here are the specific steps: In the FRN reconstruction phase, it takes graph data as input, and we obtain the feature vector set S of the supporting samples through the feature extractor c , and query sample Q, we need to find a matrix W such that W×S c ≈Q. Find the optimal It is equivalent to solving the linear least squares problem, and the formula is as follows: Where ||·|| is the Frobenius norm, and in ridge regression, σ is the parameter used to weight the penalty term. Then the optimal and The formula can be expressed as: Where S c T YesS c The transpose of , I is a unit vector. By calculating The Euclidean distance between each Q is used to classify. Finally, the T-SNE algorithm is used to reduce the dimension of each point to a two-dimensional space. The result is as follows Figure 2 shown.

[0019] Figure 2 The left side in the middle is the scattered point information after reconstruction and the right side is the scattered point information before reconstruction.

[0020] For samples in the support set, the scatter point information includes the classification of the scatter points around the point to improve the correlation between the support set and the query set in the image classification process; Visual generation of similarity matrix of support samples: Make similarity matrix for samples in support set, including similarity matrix between multiple classes, and screen out support samples with poor classification effect through similarity matrix display, which will be used for subsequent replacement operation. The following are the specific steps: In the image screening stage, first, users can Figure 2 Observe whether there are significant supporting samples that do not belong to this class, but are classified into this class. The user views the samples. If this happens, it is mostly because the angle of the photo and the background color of the photo have classified the photo into this class. In this case, the quality of this supporting sample is not very good and needs to be replaced. Second, CKA can be used to select two close classes in the scatter plot for similarity analysis, and filter out supporting samples with low similarity within the class and high similarity between classes. Figure 3 The horizontal and vertical axes of are each picture. The first 5 columns are of the same class, and the last 5 columns are of the same class. The numbers in each rectangle represent the similarity. For the convenience of observation, each similarity value is magnified 100 times. Figure 3 As shown in the third column of the fifth row in the figure, this point represents the similarity between the third and fifth pictures in the first class, which is only 0.1. The two pictures on the right do not look like birds of the same class, proving that the similarity between the two samples is very low. However, in the eighth column of the same row, the similarity value is 11.4. In this case, the quality of the sample is not very good. Then you can choose to replace the sample. Now delete the support sample and use it in the subsequent replacement process until you replace the sample.

[0021] Recommendation algorithm for support set samples: After filtering out low-quality support samples in S2, use the recommendation algorithm to sort the results in descending order, and select high-quality samples based on the recommendation score and scatter plot layout information. Use the high-quality samples to replace the low-quality samples filtered out in S2, and repeat steps S1 to S3; the following are the specific steps: In the replacement phase operation, each possible replacement sample is calculated through the recommendation algorithm of the present invention, and the calculation process includes four parameters. The ratio V1 of the area enclosed by the support samples before reconstruction and the number of positive samples in the area, the ratio RecV1 of the area enclosed by the support samples after reconstruction and the number of positive samples in the area, as well as the F1 score in the classification result and the difference D between the F1 score and the F2 score. Among them, V1 and RecV1 represent the classification of the support sample set only when the enclosed area is smaller and the number of positive samples covered by the area is more, and vice versa. The higher the F1 and D, the more confident it is that the sample belongs to the class, and the more obvious the characteristics of the sample. Then the formula of the recommendation algorithm can be expressed as: Among them, α, β, γ, and δ are four hyperparameters, Rs i is the area enclosed by the support samples of the i-th class after reconstruction. For the first term, the denominator means the number of samples belonging to the i-th class in the area enclosed by the support samples of the i-th class. The denominator is S i is the area enclosed by the i-th class support samples. For the second term, in the reconstructed scatter plot, the denominator is the number of samples belonging to the i-th class in the area enclosed by the i-th class support samples. The denominator is Rs i is the area enclosed by the i-th class of support samples. After the user sets different hyperparameters, a recommended value will be obtained for each sample in this class. Samples are selected from high to low, and the position information of the sample in the scatter plot is observed. If there are already points in the support set samples around this point, it is not recommended, because most of the feature information of this point is already available, and a sample needs to be found to make up for the feature information that is not available in this class. According to this principle, select a sample with a high recommendation score and no duplication of feature information of this point. After replacement, repeat steps S1 to S3.

[0022] Visual generation of the degree of intra-class aggregation during the replacement process: By visualizing the degree of intra-class aggregation during the continuous replacement of support samples, the quality of the update operation can be reflected, thereby judging the correctness of the replacement operation; Whether the quality and classification effect are improved after replacement can be determined by Figure 4 Come and observe, Figure 4The horizontal axis is the number of iterations, and the vertical axis is the average distance from each point in the class to the class center. The solid line represents the unreconstructed sample, and the dotted line represents the reconstructed sample. The smaller the distance, the more compact the class is, and the better the classification effect is, and vice versa. Figure 4 It is found that the fourth time reaches the lowest point, when the distance between reconstruction and non-reconstruction is the smallest, so the quality of the support sample set is the best, and the accuracy of the classification result is relatively the highest. In fact, the accuracy of this type of sample reaches 86.23%, which is the highest score compared to other groups. Through the above explanation, by combining FRN and visualization methods to replace samples of poor quality, the classification effect of the overall model can be improved compared to the case of using only FRN algorithm.

[0023] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments including components are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. A few-sample classification method based on visual reconstruction, characterized in that: The following steps are involved: S1. Reconstruction of query set samples: For a given support set and query set, use the FRN reconstruction algorithm to reconstruct each sample in the query set through the support set, and classify it through the new sample; Dimensionality reduction through T-SNE and visualization through scatter plots; S2. Visual generation of similarity matrix of support set samples: including similarity matrices between multiple classes, and through the similarity matrix display, users can filter out support samples with poor classification effects for subsequent replacement operations; S3, recommendation algorithm for support set samples: after filtering out low-quality support samples in S2, the recommendation algorithm is used to sort the results in descending order, and high-quality samples are selected through recommendation scores and scatter plot layout information; the low-quality samples filtered out in S2 are replaced with the high-quality samples, and steps S1 to S3 are repeated; S4. Visual generation of intra-class and inter-class aggregation levels during the replacement process: The quality of the update operation is reflected by visualizing the intra-class and inter-class aggregation levels during the continuous replacement of support set samples. Users can use this to judge the correctness of the replacement operation.

2. The method for few-sample classification based on visual reconstruction according to claim 1, characterized in that: The query set is reconstructed from the support set through the FRN algorithm, making the query set closer to the target set and generating a scatter plot; The recommendation algorithm in step S3 includes: According to the scatter plot position information before and after reconstruction; The ratio of the area before reconstruction to the number of positive samples under the area is used as the first factor; the ratio of the area after reconstruction to the number of positive samples under the area is used as the second factor; the score F1 of the classification result is used as the third factor; the scores F1, F2 of the classification result, score D = (F1-F2), D is used as the fourth factor; Calculate the recommendation score based on the above four factors, and arrange the samples in descending order according to the recommendation score; then observe the position of the samples in the scatter plot from high to low, and choose whether to select the point for replacement.

3. The method for few-sample classification based on visual reconstruction according to claim 2, characterized in that: In step S3, the ratio V1 of the area enclosed by the support samples before reconstruction and the number of positive samples in the area, the ratio RecV1 of the area enclosed by the support samples after reconstruction and the number of positive samples in the area, and the F1 score and the difference D between the F1 score and the F2 score in the classification result are calculated; Among them, V1 and RecV1 are only when the enclosed area is smaller and the number of positive samples covered by the area is more, which means that the classification of the support sample set is better, otherwise it is worse; the higher the F1 and D are, the more confident that the sample belongs to the class and the more obvious the characteristics of the sample are; The formula of the recommendation algorithm is: Among them, α, β, γ, and δ are four hyperparameters, Rs i is the area enclosed by the support samples after the i-th type reconstruction; For the first term, the denominator means the number of samples belonging to the i-th class in the area enclosed by the i-th class support samples; the denominator is S i is the area enclosed by the i-th type of support samples; For the second item, in the reconstructed scatter plot, the denominator is the number of samples belonging to the i-th class in the area enclosed by the i-th class support samples; the denominator is Rs i is the area enclosed by the i-th type of support samples; A recommended value for each sample in this category is obtained through the hyperparameters set by the user, and the sample is replaced by the user-selected sample in S2; after the replacement, a scatter plot is regenerated according to the FRN algorithm.

4. The method for few-sample classification based on visual reconstruction according to claim 1, characterized in that: In step S1, in the FRN reconstruction phase, it takes the graph data as input and obtains the feature vector set S of the supporting samples through the feature extractor. c , and query sample Q, find the matrix W such that W×S c ≈Q; find the optimal It is equivalent to solving the linear least squares problem, and the formula is as follows: Where ||·|| is the Frobenius norm. In ridge regression, σ is the parameter used to weight the penalty term. The optimal and The formula can be expressed as: Where S c T YesS c The transpose of , I is a unit vector; by calculating The Euclidean distance between each Q is used for classification; finally, the T-SNE algorithm is used to reduce the dimension of each point to a two-dimensional space.

Images