Method for selecting samples to be marked in classification task and method for classifying data sets
Through feature extraction and log-probability ratio active selection technology, the problem of low utilization efficiency of labeled data in small sample learning is solved, and high accuracy classification is achieved in cold start settings, which significantly improves classification performance.
Patent Information
- Application Number
- CN202410708025.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-06-05
- Filing Date
- 2024-06-03
- Publication Date
- 2024-12-06
AI Technical Summary
In small sample learning, especially in direct push small sample classification (TFSC), it is difficult for the prior art to effectively utilize limited labeling data to improve classification accuracy, especially in cold start settings, the initial labeling stage based on unstable models , more challenging.
By extracting sample features using feature extractors, statistically infer their probability distribution in the category, and using logarithmic probability ratio (lpr) to actively select the samples to be marked, and combined with preprocessing steps such as normalization and graph smoothing, improve statistics Inference to improve classification accuracy.
It realizes high accuracy classification in small sample learning environment, especially in cold start settings, effectively utilizes limited labeled data, which significantly improves classification accuracy and is better than traditional TFSC methods.
Smart Images

Figure CN119089328A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of machine learning, and in particular to small sample learning and classification. Background Art
[0002] Over the past few years, the few-shot learning problem has attracted increasing attention. Few-shot learning is a machine learning technique for problems with few-shot examples. In contrast to traditional machine learning, which requires a large amount of data to train a model, in few-shot learning, only a few examples are used to train the model. In this case, the goal of few-shot learning is to train a model with only a few examples and use it to classify new, unseen examples.
[0003] Different branches of few-shot learning are being explored, with major branches including meta-learning, where the goal is to acquire task-level meta-knowledge that models leverage to quickly adapt to new tasks with very few labeled examples. Another major branch is metric learning, where models leverage distance metrics to estimate the similarity between two inputs. Knowing this allows generalization to new categories where there are few labeled instances. These approaches require pre-training the model on more general tasks, and progress in this area has made available a myriad of efficiently trained feature extractors with data augmentation techniques, ensembles, and various other techniques. Classification in the inductive setting has relied primarily on simple methods, while a more diverse range of methods have been used in the transductive setting, yielding more promising results.
[0004] Few-shot classification is a specific type of few-shot learning where the goal is to classify a set of new examples based on a small number of labeled examples. In particular, few-shot classification involves the problem of learning to classify a small set of data examples without a given training set but with a very limited labeling budget. This problem is likely to arise in situations where data samples are available but their labeling is expensive. These situations are often encountered in real-world industrial applications. In this context, the key challenge of few-shot classification is to effectively utilize the limited labeled data to improve classification accuracy.
[0005] When the total number of data samples is large, self-supervised learning (SSL) has recently emerged as a promising solution, showing the ability to achieve high accuracy with a very limited number of labeled samples, but requiring a large number of total samples.
[0006] When the number of data samples decreases, a specific type of few-shot learning using transductive reasoning, namely transductive few-shot classification (TFSC), has been proposed in the art, which recognizes new categories with only a few examples and includes many solutions that can similarly achieve high accuracy. In TFSC, the model is trained on a small set of labeled data and then applied to a large set of unlabeled data for prediction. The basic principle is to use a large general dataset to train effective feature extractors, hoping that they will be able to generate meaningful such features for the considered few-shot task. The goal of TFSC is to improve the accuracy of few-shot learning by exploiting the correlation between labeled and unlabeled data. In addition, TFSC has applications in many fields, including computer vision, natural language processing, and robotics, where it is often used for tasks such as object recognition, text classification, and object manipulation.
[0007] It should be noted that in the context of the TFSC benchmark, labeled samples for few-shot tasks are usually assumed to be drawn uniformly at random. Choosing which samples to label in classification tasks is not a new problem. In fact, many solutions have been proposed in the active learning community to deal with this problem. However, in the field, solutions proposed in active learning are still not suitable for the few-shot case, as most of the used techniques with strong theoretical foundations usually rely heavily on the model used. These techniques usually seek to reduce the variance of the learner, or make choices based on the uncertainty of the model. These techniques become challenging in the cold-start setting, where we start with zero labeled examples. Having too few samples and / or labeling budgets can make this even more challenging, as one cannot afford an initial phase of irrelevant labeling based on unstable models.
[0008] The present invention aims to improve these disadvantages. Summary of the invention
[0009] In this regard, according to one aspect of the present invention, a method for selecting samples to be labeled in a classification task in a data set is provided, wherein the data set, such as an image data set, includes N samples belonging to K categories, the method comprising the following steps:
[0010] S10: extract features of N samples using a feature extractor;
[0011] S20: Statistically inferring the probability distribution of the extracted features in the K categories to obtain the density of each sample relative to the distribution of the K categories; and
[0012] S30: Using density to actively select samples to label.
[0013] The method proposed above provides a way to select samples to be labeled in a self-supervised manner. It can achieve very high accuracy compared to the classic TFSC method.
[0014] In one embodiment, the method according to the first aspect also includes a step of preprocessing the extracted features of N samples before the step of statistically inferring the probability distribution, for example by two-step normalization or graphic smoothing, so as to improve the accuracy and efficiency of the machine learning model by ensuring that the input data is suitable for the algorithm used.
[0015] In another embodiment, the step of selecting samples to be labeled using density includes:
[0016] S31: Calculate the log probability ratio (lpr) of each sample, where the log probability ratio is the ratio of the log density of the distribution of different label features;
[0017] S32: In the classification task, select a sample with the lowest logarithmic probability ratio in each category as the sample to be marked;
[0018] S33: using the selected sample with the lowest log probability ratio and the selected sample with the highest log probability ratio to improve the statistical inference; and
[0019] S34: Repeat steps S31 to S33 until a sufficient number of samples to be labeled are selected.
[0020] With this two-layer approach, i.e., inferring the distribution of samples across categories and selecting first the most likely labeled samples in each category and then the least likely labeled samples, it achieves better accuracy among all proposed baselines.
[0021] According to another aspect of the present invention, there is provided a method for classifying a data set including N samples belonging to K categories, such as an image data set, comprising the following steps:
[0022] S10: extract features of N samples using a feature extractor;
[0023] S20: Statistically inferring the probability distribution of the extracted features in the K categories to obtain the density of each sample relative to the distribution of the K categories; and
[0024] S30: Assign categories to samples using density, where the highest density can determine the category of the sample.
[0025] Utilizing this approach, datasets with limited labeling budget can be classified in a cold-start setting in a more efficient and accurate manner.
[0026] Similarly, in one embodiment, the method according to the second aspect also includes a step of preprocessing the extracted features of N samples before the step of statistically inferring the probability distribution, for example by two-step normalization or graphic smoothing, so as to improve the accuracy and efficiency of the machine learning model by ensuring that the input data is suitable for the algorithm used.
[0027] In another embodiment, the step of selecting samples to be labeled using density includes:
[0028] S31: Calculate the log probability ratio (lpr) of each sample, where the log probability ratio is the ratio of the log density of the distribution of different label features;
[0029] S32: In the classification task, select a sample with the lowest logarithmic probability ratio in each category as the sample to be marked;
[0030] S33: using the selected sample with the lowest log probability ratio and the selected sample with the highest log probability ratio to improve the statistical inference; and
[0031] S34: Repeat steps S31 to S33 until a sufficient number of samples to be labeled are selected.
[0032] In another embodiment, the step of assigning categories to samples using density includes:
[0033] - Use density to select samples to label;
[0034] - labeling the selected samples; and
[0035] -Classify the labeled samples.
[0036] According to yet another aspect of the present invention, there is provided a computer program product comprising instructions to be executed by a processor, the instructions being suitable for performing any one of the methods described above.
[0037] In this regard, the present invention aims to classify small, initially unlabeled datasets given a very limited labeling budget and leads to new approaches in data-scarce learning settings. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Other features and advantages of the present invention will appear from the following description with reference to the accompanying drawings, in which:
[0039] Figure 1 A flow chart of an exemplary classification method according to the present invention is shown. DETAILED DESCRIPTION
[0040] Figure 1An exemplary classification method according to the present invention is shown. The method is based on the classic setting in small sample classification, where it is assumed that a basic dataset is given that may include many categories and labeled samples in each category. This dataset can be used to train a general feature extractor fθ. Next, given a small dataset consisting of N samples belonging to K new categories These categories are disjoint from those of the base dataset, and samples can be arbitrarily distributed between the categories. The method according to the invention can predict the category of each of the N samples after revealing the labels of at most l≤N samples. The performance of all samples (including labeled samples) is weighted accuracy measured.
[0041] For commodities, in the present invention, the problem to be solved is called a K-way, l-label, N-sample task. The samples in the present invention are equivalent to the query set in the standard TFSC setting, and the labels are equivalent to the total number of samples. The main difference is that the labeled samples in the present invention are selected rather than randomly drawn, and the possible result is that each category has a different number of labels, including some of them may be zero in the worst case.
[0042] The proposed method according to the present invention relies on multiple steps. The first step may include training a feature extractor to convert the original image into a high-dimensional feature vector that is more easily distinguishable. For example, multiple preprocessing steps may be performed on the obtained feature vector, including normalization and image smoothing. Next, it is proposed to use the expectation maximization (EM) method to infer the probability distribution of all samples. Finally, a sequential active learning process is proposed, first picking the samples with the highest confidence for each class, and then picking the samples with the lowest confidence. In the following paragraphs, a comprehensive explanation of each of these steps is provided.
[0043] Feature Extraction S10
[0044] In this step, creating a feature extractor that provides meaningful representations for samples outside the domain of the categories it was originally trained on is an important problem in machine learning. Especially in the field of few-shot learning, this is a very common step, where well-designed methods can significantly improve the quality of features. For example, the classic ResNet12 architecture originally proposed in "Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. arXiv preprint arXiv:1904.04232, 2019" is able to achieve an average accuracy of 57% on the 5-step one-shot problem generated by the mini-ImageNet dataset, while the same architecture with some adjustments can achieve over 70% accuracy as described in "Yassir Bendou, Yuqing Hu, Raphael Lafargue, Giulia Lioi, Bastien Pasdeloup, Stéphane Pateux, and Vincent Gripon. Easy: Ensemble augmented-shot y-shaped learning: State-of-the-art few-shot classification with simple ingredients, 2022".
[0045] Since training an effective feature extractor is already a classic step in the literature on small sample learning in this field, the present invention can simply use the existing model.
[0046] Pretreatment S10a
[0047] Next, optionally, a two-step normalization can be performed, where the eigenvectors obtained in the previous step are first centered (the mean of the resulting eigenvectors is 0) and then projected onto the unit sphere (the norm of each eigenvector becomes 1). It is represented as the resulting eigenvector x and the eigenvector matrix X.
[0048] In addition, graph smoothing can be used to exploit the information contained in unlabeled samples. It is recommended to adopt the method described in "Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simplifying graph convolutional networks. In International conference on machine learning, pages 6861–6871. PMLR, 2019". First, create a graph whose vertices are samples and whose edges depend on the cosine similarity between the relevant feature vectors. That is, for each vertex, its m most similar neighbors can be retained, thus obtaining a binary graph whose adjacency matrix is represented by A. Then we calculate where β and κ are hyperparameters. The feature vector X is transformed into Z = MX. Note that this operation has the effect of mixing each coordinate of the feature vector with the same coordinate in a similar vector, resulting in a smoothing operation with the benefit of reducing the impact of outliers.
[0049] Statistical Inference S20
[0050] Then, it is proposed to represent zi as the i-th preprocessed feature vector corresponding to the i-th sample in N. A simple model is given where for each class k, the distribution of the corresponding z / y=k follows a multivariate Gaussian distribution with mean μk and variance Σk. If this is treated as an isotropic distribution, Σk becomes σkI, where σk∈R+ and I is the identity matrix.
[0051] Under this assumption, given good estimates of the mean and variance, a probabilistically optimal classifier can be obtained. On the other hand, estimates can be obtained by assigning examples to clusters. Algorithms that rely on expectation maximization (EM) allow finding local optimal solutions for data partitioning. There are two reasons for focusing on the soft K-means algorithm; the first reason is that the soft K-means method does not explicitly or implicitly exploit any prior knowledge about the distribution of marginal labels, which makes it more robust to sample class imbalance. The second reason is related to the unsupervised nature of the algorithm, since local optimal clustering with zero labels can be calculated, although the quality is not optimal. Next, an estimate of p(z / y=k) is obtained.
[0052] Actively choose S30
[0053] After the statistical inference step, the goal is to quantify the potential performance improvement that could result from labeling a data point. Classic solutions in this area rely on heuristics, including:
[0054] Margin: For any zi, the margin criterion is the difference between the highest probability (i.e. the probability of the assigned cluster) and the second best probability. The sample with the highest margin is considered the most confident.
[0055] k-medoid: A medoid is the cluster element that is closest to its centroid. The sample with the lowest distance to its cluster centroid is the sample with the highest confidence.
[0056] Instead of using these classical methods, in the present invention, a new criterion based on log density ratio is proposed, which can be called log probability soft K-means sampling (LSS). That is, the sample with the lowest log probability ratio (lpr) of belonging to its cluster k but not to other clusters is considered to be the most confident sample.
[0057] The density logarithm of z / y=k is as follows:
[0058]
[0059] The log probability ratio lpr is defined as the ratio of the log density of the distribution of different label features:
[0060]
[0061] for and The logarithmic probability ratio of sample z of cluster k is as follows:
[0062]
[0063] The mean and standard deviation of the categorical feature distribution are estimated using clustering as follows:
[0064]
[0065] Using the logarithm of the density rather than the density itself makes the calculation more robust to the estimate. If the standard deviation remains unchanged, the ratio becomes:
[0066]
[0067] A log-odds ratio close to 0 indicates high confidence that the sample belongs to the cluster, while a ratio close to 1 indicates high uncertainty.
[0068] Once the criteria are selected, the value of each sample can be calculated and the samples to be labeled can be selected based on the value. In the present embodiment, as an example, the original sequential pipeline is used for active selection. The pipeline consists of several rounds. The first round includes selecting K samples for labeling, so as to produce the highest confidence according to the selected criteria (e.g., lpr is close to 0), that is, one label for each cluster. The remaining round labels are selected to produce the samples with the lowest possible confidence (e.g., lpr is close to 1). The present invention proposes such a two-layer strategy that the first label strongly represents the category with a lower risk of outliers, and then uses these additional priors to improve statistical inference and select the most uncertain labels. In fact, the samples with the strongest confidence are likely to be predicted correctly, so there is no need to manually label them.
[0069] For the first round, multiple random initializations are used for soft K-means when no labels are available. Each initialization generates different clusters π1,π2,....πn. The cluster that minimizes the sum of the distances from the sample to its nearest centroid is then selected.
[0070] For subsequent rounds, soft K-means is performed, with the centroids initialized to the mean of the labels corresponding to each class. In other words, the centroid of the i-th cluster is initialized to the mean of all labels from class i, or arbitrarily to zero if there are no labels for class i. This allows for higher quality clusters that can be used in another round to select the next K labels, or to classify samples in the final round.
[0071] Therefore, through the small sample configuration in the present invention, the feature extractor can be used to obtain a "better" data distribution, where the diversity sampling method and the uncertainty-based sampling method can be mixed to obtain an effective labeling pipeline.
[0072] The labeled samples in the dataset can then be assigned to different categories.
[0073] For example, when using the method according to the present invention to solve a multi-class classification problem in an image classification dataset, the example selection process in the first round usually focuses on identifying high and low confidence images. High confidence examples are those where the object of interest is clearly visible and easy to distinguish. In contrast, low confidence examples tend to be images with unusual zoom levels, atypical angles on the object, or where the object is more in the background. This distinction helps to better understand and analyze the upcoming classification challenge.
[0074] In some exemplary experiments conducted, two different feature extractors are employed to evaluate the proposed Active Few Shot Classification (AFSC) method in comparison with the conventional Transductive Few Shot Classification (TFSC) method. The first feature extractor, ResNet-18, is used to compare the performance with the state-of-the-art methods in the classical TFSC setting. The second feature extractor, Ensembled Augmented Shots Y-shaped ResNet-12, is utilized due to its ability to achieve top performance on the classical TFSC problem, making it suitable for the remaining experiments.
[0075] To evaluate the effectiveness of using AFSC, the performance of the TFSC method was compared with that of the AFSC method. The results show that the proposed active method outperforms random label selection, proving the value of using AFSC. In addition, the soft K-means statistical inference method was used under the same conditions to further validate the research results.
[0076] In summary, the experiments show that the AFSC method according to the present invention provides significant advantages in accuracy compared to the traditional TFSC method, demonstrating its potential in real-world applications.
[0077] Furthermore, the present invention may be implemented in various fields, such as computer vision, natural language processing, and robotics.As an example, an exemplary embodiment of detecting and marking defects in industrial products is provided.
[0078] In this exemplary case, industrial machines sometimes have defects in their products. Therefore, it is necessary to detect these defects and label the products through the streaming data of the products. Each machine can be used for different products. Classically trained machine learning models have poor adaptability to different products. However, by using active few-shot classification according to the present invention, few new examples need to be annotated by humans, and the performance of the classification model on new products is significantly improved.
[0079] In the context of the present invention, a process can be started using a sample of automatically collected product images. The first step is to extract features from the collected images, which are then used in an iterative process of clustering and subsequently selecting images to be annotated. At the beginning of the first round of this iterative process, the images are clustered into as many clusters as product categories (for example: no defects / defect type 1 / defect type 2, etc.). The next step of this round includes calculating the confidence score of each image belonging to its designated cluster using the above criteria. For each cluster, the image with the highest confidence is selected for labeling or annotation. At the end of this first round, the images are clustered again, but this time using an initialization that depends on the labeled images. This initialization allows better and more accurate defect clustering. And subsequent rounds focus on selecting low confidence score images for annotation because they allow better disambiguation of clusters. Each round produces better clustering than the previous round, and the process stops when the clusters satisfactorily represent the product categories.
[0080] In addition, it is known to those skilled in the art that the above exemplary embodiments according to the present invention can be implemented in a variety of ways, such as program instructions executed by a processor, as a software module, microcode, a computer program product on a computer readable medium, a logic circuit, an application specific integrated circuit, firmware, etc. The embodiments of the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment comprising hardware and software elements. In a preferred embodiment, the present invention is implemented in software, including but not limited to firmware, resident software, microcode, etc.
[0081] In addition, embodiments of the present invention may take the form of a computer program product accessible from a computer-available or computer-readable medium that provides program code used by or in conjunction with a computer, a processing device, or any instruction execution system. For the purposes of this specification, a computer-available or computer-readable medium may be any device capable of containing, storing, transmitting, or transmitting a program used by or in conjunction with an instruction execution system, a device, or a device. The medium may be an electronic, magnetic, optical, or semiconductor system (or device or device). Examples of computer-readable media include, but are not limited to, semiconductor or solid-state memory, magnetic tape, removable computer disk, RAM, read-only memory (ROM), hard disk, optical disk, etc. Examples of current optical disks include compact disk-read only memory (CD-ROM), compact disk-read / write (CD-R / W), and DVD.
[0082] This paper proposes a new approach for Active Few-Shot Classification (AFSC). This is considered as a new paradigm opponent to Transductive Few-Shot Classification (TFSC). A simple approach relying on two main steps is proposed: a statistical inference method and an active sampling step, where the samples to be labeled can be selected based on log-probability soft K-means sampling. Furthermore, in this paper, benchmarks from the field of TFSC are adopted and it is demonstrated that the proposed framework can achieve new levels in accuracy compared to TFSC.
[0083] The embodiments described above are examples of the present invention. Various modifications may be made to them without departing from the scope of the present invention as defined in the appended claims.
Claims
1. A method for selecting samples to be labeled in a classification task in a dataset, wherein: The dataset consists of N samples belonging to K categories, including the following steps: S10: extract features of N samples using a feature extractor; S20: Statistically inferring the probability distribution of the extracted features in the K categories to obtain the density of each sample relative to the distribution of the K categories; and S30: Use density to select samples to label.
2. The method according to claim 1, wherein: It also includes the following steps: S11: Preprocess the extracted features of the N samples.
3. The method according to claim 2, wherein: The extracted features are preprocessed by two-step normalization or graph smoothing.
4. The method according to any one of claims 1 to 3, wherein: The step of selecting samples to be labeled using the density is based on active selection.
5. The method according to claim 4, wherein: The steps of selecting samples to be labeled using the density include: S31: Calculate the log probability ratio (lpr) of each sample, where the log probability ratio is the ratio of the log density of the distribution of different label features; S32: Select a sample with the lowest log probability ratio in each category as the sample to be marked in the classification task; S33: using the selected sample with the lowest log probability ratio and the selected sample with the highest log probability ratio to improve the statistical inference; and S34: Repeat steps S31 to S33 until a sufficient number of samples to be labeled are selected.
6. The method according to claim 5, wherein: The log probability ratio (lpr) is defined by the following formula: in, zi is the i-th extracted feature vector corresponding to the i-th sample among N samples.
7. The method according to claim 1, wherein: The step of statistically inferring the probability distribution of the extracted features among K categories is based on the soft K-means algorithm.
8. The method according to claim 1, wherein: The dataset is an image dataset.
9. A method for classifying a data set comprising N samples belonging to K categories, comprising the following steps: S10: extract features of N samples using a feature extractor; S20: Statistically inferring the probability distribution of the extracted features in the K categories to obtain the density of each sample relative to the distribution of the K categories; and S30: Assign categories to samples using density.
10. The method according to claim 9, wherein: It also includes the following steps: S11: Preprocess the extracted features of the N samples.
11. The method according to claim 10, wherein: The extracted features are preprocessed by two-step normalization or graph smoothing.
12. The method according to any one of claims 9 to 11, wherein: The steps to assigning categories to samples using density include: - Use density to select samples to label; - labeling the selected samples; and -Classify the labeled samples.
13. The method according to claim 12, wherein: The steps of selecting samples to be labeled using the density include: S31: Calculate the log probability ratio (lpr) of each sample, where the log probability ratio is the ratio of the log density of the distribution of different label features; S32: Select a sample with the lowest log probability ratio in each category as the sample to be marked in the classification task; S33: using the selected sample with the lowest log probability ratio and the selected sample with the highest log probability ratio to improve the statistical inference; and S34: Repeat steps S31 to S33 until a sufficient number of samples to be labeled are selected.
14. The method according to claim 13, wherein: The log probability ratio (lpr) is defined by the following formula: in, zi is the i-th extracted feature vector corresponding to the i-th sample among N samples.
15. The method according to claim 9, wherein: The highest density determines the class of the sample.
16. A computer program product comprising instructions to be executed by a processor, the instructions being adapted to perform the method of any one of claims 1 to 15 when executed by the processor.
