A few-shot classification method based on visual reconstruction

Abstract

The present application aims at overcoming the problem that users can only see the training result, and cannot see the relationship between the query set and the support set and cannot screen the samples in the support set, and provides a visual method which can visually and figuratively observe the influence of the support set on the classification result in the classification process. In order to solve the above technical problem, the technical scheme of the present application is as follows: a few-sample classification method based on visual reconstruction, which is used to improve the quality of the support set and the classification result through an interactive method in the case of few samples, and comprises the following steps: S1, reconstruction of the query set samples; S2, visual generation of the similarity matrix of the support set samples; S3, recommendation algorithm of the support set samples; S4, visual generation of the intra-class and inter-class aggregation degree of the replacement process.

Description

Technical Field

[0001] The present invention relates to the field of image classification, and in particular to a reconstruction-based few-sample classification visualization method. Background Art

[0002] In real-world scenarios, certain categories have only a small amount of data or labeled data, and labeling unlabeled data consumes significant time and effort. In contrast, humans can quickly learn with minimal data. For example, a five- or six-year-old child may have never seen a penguin, but if shown an image of one, upon seeing a real penguin at a zoo, they will immediately recognize it as the image they've seen before. This illustrates the gap between machine learning and human learning. Inspired by insights from human learning, the concept of few-shot classification was proposed. Machine learning methods are often used to support few-shot classification. However, these methods suffer from the problem that users only see the training results and lack visibility into the relationship between the query set and the support set, nor can they filter the samples in the support set. Poor quality samples in the support set can negatively impact classification results. Therefore, visualizing the relationship between the query set and the support set can help people better understand the classification role of each support set sample, aiding in filtering support set samples and improving classification results.

[0003] Visualizing the impact of samples in the support set on classification results is a crucial element. Its primary goal is to connect the support set and query set, lay out all sample information in a two-dimensional space, and preserve the impact of support set samples on classification, allowing users to analyze datasets in a more intuitive and effective manner. While currently there is still a lack of fully mature methods for few-shot classification, many machine learning approaches have emerged. One approach is based on model fine-tuning. This approach typically pre-trains a model on large-scale data and then fine-tunes the parameters of the fully connected layers or the top few layers of the neural network model on a target small-shot dataset to obtain a fine-tuned model. If the target dataset and the source dataset have similar distributions, model fine-tuning can be used. The second approach is based on data augmentation. This involves augmenting the existing small-shot dataset with auxiliary data or information, either by adding data or enhancing features. Data augmentation involves adding new data to the original dataset. The third approach is based on transfer learning, which leverages existing knowledge to learn new knowledge. Its primary goal is to quickly transfer existing knowledge to a new domain.

[0004] While these techniques have achieved promising results, they still have some drawbacks, such as overfitting caused by model fine-tuning. In terms of visualization, there is still a lack of visualization for few samples, as well as a method to link the query set with the support set, allowing users to continue mining. Summary of the Invention

[0005] The purpose of the present invention is to overcome the above-mentioned problem that users can only see the training results but cannot see the relationship between the query set and the support set, and cannot filter the samples in the support set. It provides an intuitive and vivid classification method that can observe the impact of the support set on the classification results during the classification process.

[0006] In order to solve the above technical problems, the technical solution of the present invention is: a few-sample classification method based on visual reconstruction, which is used to improve the quality of the support set and the classification results through an interactive method when the sample set is small, including the following steps:

[0007] S1. Reconstruction of query set samples: For a given support set and query set, the Few-Shot Reconstruction Networks (FRN) reconstruction algorithm is used to reconstruct each sample in the query set through the support set and classify it through the new samples, because images from the same category should be easier to reconstruct because their feature maps contain similar embeddings, while images from different categories will be more difficult and produce larger reconstruction errors. Then the dimensionality is reduced through T-SNE and visualized through scatter plots. The core idea of ​​the T-SNE algorithm is to generate a representation of the low-dimensional space by maintaining the local similarity between data points in the high-dimensional space. For samples in the support set, the scatter information includes the classification of the scatter points around the point to improve the correlation between the support set and the query set in the image classification process;

[0008] S2. Visual generation of similarity matrices for support set samples: This includes similarity matrices between multiple classes. Through the similarity matrix display, users can filter out support samples with poor classification results for subsequent replacement operations.

[0009] S3, Recommendation Algorithm for Support Set Samples: After filtering out low-quality support samples in S2, the recommendation algorithm sorts the results in descending order and selects high-quality samples based on a comprehensive consideration of the recommendation score and scatter plot layout information. These high-quality samples replace the low-quality samples filtered out in S2, and steps S1 to S3 are repeated.

[0010] S4. Visualization of the degree of intra-class and inter-class aggregation during the replacement process: By visualizing the degree of intra-class aggregation during the continuous replacement of support samples, the quality of the update operation is reflected, allowing users to judge the correctness of the replacement operation.

[0011] Preferably, in step S1, the query set is reconstructed from the support set by the FRN algorithm through the visualization method of reconstruction-based few-shot classification defined in the present invention, so that the query set is closer to the target set, and a scatter plot is generated.

[0012] Preferably, in step S2, the similarity between samples is calculated by CKA, wherein CKA calculation is an existing commonly used method for calculating the similarity between two features. In the first case, if there is one or several samples in the class whose intra-class similarity scores are on the low side, it means that the feature similarity between the sample and other samples in the class is poor, which often means that the sample may have features different from other samples in the class due to shooting angle or background. In this case, the quality of the sample is poor, otherwise the quality is better. In the second case, if there is one or several samples in class A and most of the samples in class B with high similarity scores, it means that the feature similarity between the sample and the sample in class B is good, which often means that if the sample is used as a support set, there is a greater probability that the query sample will be classified into class B during classification. In this case, the quality of the sample is poor, otherwise the quality is better.

[0013] Preferably, in step S3, the scatter point position information before and after reconstruction is obtained by the recommendation algorithm defined in the present invention, taking into account the two-dimensional area enclosed by the class and the number of positive samples under the area. If the enclosed area is larger, the number of positive samples under the area will also be more, and vice versa. Therefore, the ratio of the area before reconstruction to the number of positive samples under the area is used as the first factor; the ratio of the area after reconstruction to the number of positive samples under the area is used as the second factor; in addition, the quality of the replacement element itself is taken into account, which is reflected by the F1 score and the D score (F1-F2). If both the F1 score and the D score are higher, it means that we are more willing to believe that the sample belongs to this class and its characteristics are more obvious. Therefore, the F1 score is used as the third factor; the D score (F1-F2) is used as the fourth factor. The recommendation algorithm includes four indicators. According to different situations, the recommendation scores of different samples are output. The higher the score, the better the classification effect after the sample is replaced. Select samples from high to low and observe their position in the scatter plot. If a point in the support set already exists near that point, it is not recommended because most of the feature information for that point is already available. Finding a sample to fill in the missing feature information for that class is necessary. Based on this principle, select samples with high recommendation scores and no duplicate feature information for that point.

[0014] Preferably, in step S4, the average distance from each point in the same class to the class center point is used as the vertical coordinate and the number of iterations is used as the horizontal coordinate to draw curves before and after reconstruction, showing the quality of the classification results during the iterative replacement of samples.

[0015] The present invention has the following features and beneficial effects: While existing few-shot classification algorithms have achieved some success, these methods still lack the ability to link the support set with the query set and the role of modified samples in the support set in the classification process. The present invention uses a reconstruction method to reconstruct new samples from the query set using the support set, and then classifies them using these new samples. The reconstructed results are then displayed using a visualization method, allowing the influence of support set samples on the query set to be displayed during the classification process, thus solving the aforementioned problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Flowchart of an embodiment of the present invention.

[0017] Figure 2 This is a comparison chart of the CUB dataset using the FRN algorithm in an embodiment of the present invention.

[0018] Figure 3 This is a diagram showing the effect of the CKA algorithm similarity in an embodiment of the present invention.

[0019] Figure 4 This is a process indicator diagram for improving support samples according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The invention is described in detail below with reference to the accompanying drawings and specific examples.

[0021] The technical solution of the present invention is: a visualization method for few-sample classification based on reconstruction, such as Figure 1 The following steps are shown:

[0022] Reconstructing query set samples: The CUB (Caltech-UCSD Birds) dataset is a popular image dataset for bird identification and classification. We randomly select 10 classes and 5 of them as the support set, with the remaining as the query set. Given a support set and query set, we use the Few-Shot Reconstruction Networks (FRN) reconstruction algorithm to reconstruct each query set sample using the support set. We then classify the new samples using the new samples, perform dimensionality reduction using T-SNE, and visualize the results using scatter plots.

[0023] Here are the specific steps:

[0024] In the FRN reconstruction phase, it takes graph data as input, and we obtain the feature vector set S of the support sample through the feature extractor c , and query sample Q, we need to find a matrix W such that W×S c ≈Q. Find the optimal It is equivalent to solving the linear least squares problem, and the formula is as follows:

[0025]

[0026] Where ‖·‖ is the Frobenius norm, and in ridge regression, σ is the parameter used to weight the penalty term. Then, the optimal and The formula can be expressed as:

[0027]

[0028] Among them S c T It's S c The transpose of , I is a unit vector. By calculating Finally, the T-SNE algorithm is used to reduce the dimension of each point to a two-dimensional space. The result is as follows Figure 2 shown.

[0029] Figure 2 The left side is the scattered information after reconstruction and the right side is the scattered information before reconstruction.

[0030] For samples in the support set, the scatter point information includes the classification of the scattered points around the point to improve the correlation between the support set and the query set in the image classification process;

[0031] Visualization of the similarity matrix of support samples: Generate a similarity matrix for the samples in the support set, including similarity matrices between multiple classes. This matrix is ​​then used to screen out support samples with poor classification results for subsequent replacement operations. The specific steps are as follows:

[0032] In the image screening stage, first, users can Figure 2 Observe whether there are significant supporting samples that do not belong to this class, but are classified into this class. The user checks the samples. If this happens, it is mostly because the angle of the photo and the background color have caused the photo to be classified into this class. In this case, the quality of this supporting sample is not good and needs to be replaced. Second, CKA can be used to select two close classes in the scatter plot for similarity analysis, screening supporting samples with low similarity within the class and high similarity between classes. Figure 3 The horizontal and vertical coordinates are for each picture. The first 5 columns are of the same class, and the last 5 columns are of the same class. The numbers in each rectangle represent the similarity. For the convenience of observation, each similarity value is magnified 100 times. Figure 3As shown in the third column of the fifth row, the similarity between the third and fifth images in the first class is only 0.1. The two images on the right do not resemble the same bird, demonstrating a low similarity between the two samples. However, the eighth column of the same row shows a similarity value of 11.4. Therefore, the quality of this sample is not good. Therefore, we can perform a replacement operation on this sample, deleting the supporting sample and using it in subsequent replacement processes until it is replaced.

[0033] Recommendation algorithm for support set samples: After filtering out low-quality support samples in S2, the recommendation algorithm is used to sort the results in descending order, and high-quality samples are selected based on a comprehensive consideration of the recommendation score and scatter plot layout information. The high-quality samples are used to replace the low-quality samples filtered out in S2, and steps S1 to S3 are repeated. The specific steps are as follows:

[0034] During the replacement phase, each possible replacement sample is calculated through the recommendation algorithm of the present invention, and the calculation process includes four parameters. The ratio V1 of the area enclosed by the support samples before reconstruction and the number of positive samples in the area, the ratio RecV1 of the area enclosed by the support samples after reconstruction and the number of positive samples in the area, as well as the F1 score in the classification result and the difference D between the F1 score and the F2 score. Among them, V1 and RecV1 are only when the enclosed area is smaller and the number of positive samples covered by the area is more, which means that the classification of the support sample set is better, and vice versa. The higher F1 and D are, the more confident it is that the sample belongs to this category, and the more obvious the characteristics of the sample are. Then the formula of the recommendation algorithm can be expressed as:

[0035]

[0036] Among them, α, β, γ, and δ are four hyperparameters, Rs i is the area enclosed by the support samples of the i-th class after reconstruction. For the first term, the numerator means the number of samples belonging to the i-th class in the area enclosed by the support samples of the i-th class. The denominator is S i is the area enclosed by the support samples of the i-th category. For the second term, in the reconstructed scatter plot, the numerator is the number of samples belonging to the i-th category in the area enclosed by the support samples of the i-th category. The denominator is Rs iis the area enclosed by the support samples of class i. By setting different hyperparameters, the user will obtain a recommendation score for each sample in that class. Samples are selected from highest to lowest and their position in the scatter plot is observed. If a point in the support set already exists near that point, it is not recommended because most of the feature information for that point is already available. A sample is needed to fill in the missing feature information for that class. Based on this principle, a sample with a high recommendation score and non-duplicate feature information for that point is selected. After replacement, steps S1 to S3 are repeated.

[0037] Visualization of the degree of intra-class aggregation during the replacement process: By visualizing the degree of intra-class aggregation during the continuous replacement of support samples, the quality of the update operation can be reflected, thereby judging the correctness of the replacement operation;

[0038] Whether the quality and classification effect are improved after replacement can be determined by Figure 4 Come and observe, Figure 4 The horizontal axis is the number of iterations, and the vertical axis is the average distance from each point in the class to the class center. The solid line represents the unreconstructed sample, and the dotted line represents the reconstructed sample. The smaller the distance, the more compact the class, and the better the classification effect, and vice versa. Figure 4 We found that the fourth pass reached a relatively low point, with the distance between the reconstructed and unreconstructed samples at its minimum. Therefore, the quality of the supporting samples for this class is the best, and the classification accuracy is also relatively high. Indeed, the accuracy for this class of samples reached 86.23%, the highest score compared to the other groups. The above demonstrates that combining FRN and visualization to replace poor-quality samples can improve the overall classification performance of the model compared to using only the FRN algorithm.

[0039] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. It will be apparent to those skilled in the art that various changes, modifications, substitutions, and variations of these embodiments, including components, without departing from the principles and spirit of the present invention are still within the scope of protection of the present invention.

Claims

1. A few-shot classification method based on visual reconstruction, characterized in that: The following steps are involved: S1. Reconstruction of query set samples: For a given support set and query set, use the FRN reconstruction algorithm to reconstruct each sample in the query set through the support set, and classify it based on the new sample; Dimensionality reduction was performed using T-SNE and visualization was performed using scatter plots; S2. Visual generation of similarity matrices for support set samples: This includes similarity matrices between multiple classes. Through the similarity matrix display, users can filter out support samples with poor classification results for subsequent replacement operations. S3, recommendation algorithm for support set samples: After filtering out low-quality support samples in S2, the recommendation algorithm is used to sort the results in descending order, and high-quality samples are selected based on the recommendation score and scatter plot layout information; the high-quality samples are used to replace the low-quality samples filtered out in S2, and steps S1 to S3 are repeated; The query set is reconstructed from the support set using the FRN algorithm, making the query set closer to the target set and generating a scatter plot. The recommendation algorithm in step S3 includes: based on the scatter plot position information before and after reconstruction; The ratio V1 of the area enclosed by the support samples before reconstruction and the number of positive samples in the area, the ratio RecV1 of the area enclosed by the support samples after reconstruction and the number of positive samples in the area, as well as the F1 score and the difference D between the F1 score and the F2 score in the classification result; Among them, V1 and RecV1 are only when the enclosed area is smaller and the number of positive samples covered by the area is more, which means that the classification of the support sample set is better, and vice versa. The higher the F1 and D, the more confident it is that the sample belongs to the class and the more obvious the characteristics of the sample are. Formula for the recommendation algorithm: Among them, α, β, γ, and δ are four hyperparameters, Rs i is the area enclosed by the support samples after reconstruction of the i-th type; For the first term, the numerator means the number of samples belonging to the i-th category in the area enclosed by the i-th category support sample; the denominator is S i is the area enclosed by the support samples of the i-th category; For the second item, in the reconstructed scatter plot, the numerator is the number of samples belonging to the i-th class in the area enclosed by the i-th class support sample; the denominator is Rs i is the area enclosed by the support samples of the i-th category; The user-defined hyperparameters will generate a recommended value for each sample in this category, and the samples will be sorted in descending order based on the recommended value. The position of the sample in the scatter plot will then be observed from high to low, and the user will be asked to select the point for replacement. The user selects samples in S2 to replace them; after replacement, a scatter plot will be generated again according to the FRN algorithm; S4. Visual generation of intra-class and inter-class aggregation during the replacement process: By visualizing the intra-class and inter-class aggregation during the continuous replacement of support set samples, the quality of the update operation is reflected, and users can use this to judge the correctness of the replacement operation.

2. The method for few-sample classification based on visual reconstruction according to claim 1, characterized in that: In step S1, in the FRN reconstruction phase, it takes the graph data as input and obtains the feature vector set S of the support sample through the feature extractor. c , and query sample Q, find the matrix W so that W×S c ≈Q; find the optimal It is equivalent to solving the linear least squares problem, and the formula is as follows: Where ‖·‖ is the Frobenius norm, and in ridge regression, σ is the parameter used to weight the penalty term; the optimal and The formula can be expressed as: Among them S c T It's S c The transpose of , I is a unit vector; by calculating The Euclidean distance between each Q is used for classification; finally, the dimensionality of each point is reduced to two-dimensional space through the T-SNE algorithm.

Images