Inference device, inference method, and inference program
By employing centroid-based clustering and weighted voting with adjusted correction coefficients, the method addresses catastrophic forgetting and biased voting in CNNs, maintaining high classification accuracy and reducing memory usage.
Patent Information
- Application Number
- JP2021115106
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-07-12
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-07-12
AI Technical Summary
Existing convolutional neural network (CNN) models suffer from catastrophic forgetting when adapting to new tasks, leading to decreased accuracy in old tasks, and methods like CBCL face classification accuracy issues due to biased voting from uneven training data distribution.
The proposed solution involves a feature extraction process using a pre-trained model, followed by centroid-based clustering and weighted voting, where the correction coefficients are adjusted based on the number of centroids and sample distribution to ensure fair voting and improved classification accuracy.
This approach prevents deterioration in classification accuracy by ensuring fair voting and reducing unnecessary centroids, enhancing memory efficiency and maintaining high accuracy even with biased data distributions.
Smart Images

Figure 0007586005000001 
Figure 0007586005000002 
Figure 0007586005000003
Abstract
Description
[Technical field]
[0001] The present invention relates to inference techniques. [Background technology]
[0002] Humans can learn new knowledge through long-term experience and can retain old knowledge without forgetting it. On the other hand, the knowledge of a Convolutional Neural Network (CNN) depends on the dataset used for training, and in order to adapt to changes in the data distribution, the parameters of the CNN must be retrained for the entire dataset. As CNN learns new tasks, its estimation accuracy for old tasks decreases. Thus, when CNN performs continuous training, it is inevitable that it will suffer from catastrophic forgetting, in which the learning results of old tasks are forgotten while learning a new task.
[0003] As a method to avoid fatal forgetting, class incremental learning has been proposed, which involves additionally learning new classes for a trained model, enabling classification of new classes in addition to classification of existing classes.
[0004] Patent Document 1 proposes a method called Centroid-Based Concept Learning (CBCL) as a cognitive inspiration model for incremental learning. In CBCL, features extracted from data samples are clustered and stored in the form of centroids (centers of gravity), and labels are predicted by weighted voting based on nearby centroids. [Prior art documents] [Patent documents]
[0005] [Patent Document 1] A. Ayub and AR Wagner, "Cognitively-Inspired Model for Incremental Learning Using a Few Examples," 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2020, pp. 897-906. Summary of the Invention [Problem to be solved by the invention]
[0006] In CBCL, when performing weighted voting for candidate classes in the label prediction process, the inverse of the number of training data samples is given as a correction coefficient to correct the score of the candidate class. However, if there is a bias in the number of training data samples for each class, voting will not be fair, and it is expected that classification accuracy will deteriorate.
[0007] The present invention has been made in consideration of these circumstances, and its purpose is to provide an inference technique that can prevent a deterioration in classification accuracy even when there is a bias in the number of training data samples for each class. [Means for solving the problem]
[0008] In order to solve the above problem, an inference device of one embodiment of the present invention includes a feature extraction unit that extracts a feature vector of an input sample using a trained model, a nearby centroid selection unit that selects a plurality of centroids of each class that are in the vicinity of the feature vector of the input sample, and a weighted voting unit that calculates a prediction score of a candidate class to which the selected centroid belongs by applying a weight according to the distance between the feature vector of the input sample and the selected centroid, calculates a corrected prediction score by correcting the prediction score of the candidate class by a correction coefficient according to the number of centroids of the candidate class, and predicts the class for which the corrected prediction score is maximum as the class of the input sample.
[0009] Another aspect of the present invention is an inference method, which includes a feature extraction step of extracting a feature vector of an input sample using a trained model, a nearby centroid selection step of selecting a plurality of centroids of each class in the vicinity of the feature vector of the input sample, and a weighted voting step of calculating a prediction score of a candidate class to which the selected centroid belongs by weighting the feature vector of the input sample according to the distance between the feature vector of the input sample and the selected centroid, calculating a corrected prediction score by correcting the prediction score of the candidate class by a correction coefficient according to the number of centroids of the candidate class, and predicting the class with the maximum corrected prediction score as the class of the input sample.
[0010] Any combination of the above components, and any transformation of the present invention into a method, device, system, recording medium, computer program, etc., are also effective as aspects of the present invention. Effect of the Invention
[0011] According to the present invention, even if there is a bias in the number of training data samples for each class, it is possible to prevent a deterioration in classification accuracy. [Brief description of the drawings]
[0012] [Figure 1] FIG. 4 is a diagram illustrating a process flow of incremental learning according to the first embodiment. [Diagram 2] FIG. 1 is a configuration diagram of a machine learning device according to a first embodiment. [Diagram 3] 3(a) and 3(b) are flowcharts illustrating the machine learning procedure performed by the machine learning device of FIG. [Figure 4] FIG. 11 is a configuration diagram of an inference device according to a second embodiment. [Diagram 5] 5 is a flowchart illustrating an inference procedure performed by the inference device of FIG. 4. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0013] (First embodiment) In CBCL, a pre-trained model is used as a feature extractor without updating it. CBCL does not assume classification of the pre-trained classes (called "base classes") used to generate the pre-trained model, but assumes continuous learning of new classes (called "new classes") that are unrelated to the pre-training.
[0014] On the other hand, many incremental learning methods generally perform learning to enable classification of additional new classes while retaining the classification ability of the base classes. In order to not only enable classification of new classes, but also to prevent the classification accuracy of the base classes from deteriorating drastically, algorithms have been devised that take into account the prevention of catastrophic forgetting.
[0015] CBCL is not an algorithm that is based on base class clustering, but in the first embodiment, it is assumed that CBCL will be extended to clustering of both base classes and new classes, as in general incremental learning.
[0016] FIG. 1 is a diagram for explaining the process flow when CBCL is extended to incremental learning that combines base class clustering and new class clustering.
[0017] A pre-trained model generated using the base class training dataset 300 is pre-selected as a feature extractor (S10). The pre-trained model is not updated in the subsequent processing and is fixed.
[0018] A series of processes for applying the CBCL algorithm to the base classes is called the "initial session." In the initial session, the training data set 300 of the base classes is input, and centroids are generated for each class for the base classes (S20). The data samples input in the initial session are preferably all or a part of the training data used in pre-learning, or data of the same class and with similar characteristics as the training data used in pre-learning. The number of classes of the base classes is influenced by the number of classes that can be classified in pre-learning.
[0019] A series of processes for applying the CBCL algorithm to a new class is called an "addition session." After the initial session, in each addition session, training data sets 310, 320, and 330 of the new class are input, and centroids are generated for each new class (S21, S22, S23). The number of new classes is not limited. It is assumed that the addition sessions are continuously repeated on a session-by-session basis, and the addition sessions are as follows: an addition session S1 (S21) for a new class C1, an addition session S2 (S22) for a new class C2, an addition session S3 (S23) for a new class C3, and so on. In each addition session, a centroid is generated for each new class. At this time, each session inherits the centroid for each class from the previous session. Note that the new class handled in the subsequent sessions is mainly a class that has not been previously handled, but there may be cases where a previously handled class is handled again.
[0020] 2 is a configuration diagram of a machine learning device 100 according to a first embodiment. The machine learning device 100 includes a feature extraction unit 10, a trained model storage unit 20, a centroid generation unit 30, a centroid storage unit 50, and a label prediction unit 60. The centroid generation unit 30 includes a clustering unit 40, and the label prediction unit 60 includes a neighborhood centroid selection unit 70 and a weighted voting unit 80.
[0021] The machine learning device 100 receives training data as input, learns clustering using the CBCL method, generates centroids for each class, and performs label prediction for unknown data samples based on the centroids. If a series of learning processes for a certain training data set is considered as a session, CBCL assumes that learning is continuously repeated on a session-by-session basis. The number of classes in the training data set in each session is not limited. The processing content of a session to which the CBCL algorithm is applied will be described with reference to FIG. 1, but the processing content is similar in the initial session and each additional session.
[0022] In each session, when a training data set to be learned is input to the machine learning device 100, the feature extraction unit 10 reads a pre-selected pre-trained model from the trained model storage unit 20, and extracts feature vectors of input samples of the training data set using the pre-trained model as a feature extraction means. An example of a sample is an image, but the sample is not necessarily limited to an image.
[0023] In the learning phase, the feature extraction unit 10 supplies the extracted feature vectors of the input samples to the centroid generation unit 30. The centroid generation unit 30 clusters the input samples based on the feature vectors of the input samples and the labels of the classes to which the input samples belong, to generate centroids for each class, and stores the generated centroids for each class in the centroid storage unit 50.
[0024] The centroid generation unit 30 includes a clustering unit 40, and generates centroids for each class by repeating a machine learning technique called Agg-Var clustering, for example. The number of centroids for each class is one or more, and is not constant across all classes.
[0025] The clustering unit 40 calculates the distance between the feature vector of the input sample and each centroid of the class to which the input sample belongs, and selects the nearest centroid among the multiple centroids of the class to which the input sample belongs, which is the closest to the feature vector of the input sample. The clustering unit 40 determines whether to integrate the feature vector of the input sample with the nearest centroid or to newly add the feature vector of the input sample as the centroid of the class to which the input sample belongs by comparing the distance dmin between the input sample and the nearest centroid with the threshold D. Specifically, when the distance dmin between the input sample and the nearest centroid is less than the threshold D (dmin < D), the clustering unit 40 integrates the feature vector of the input sample with the nearest centroid, and when the distance dmin is greater than or equal to the threshold D (dmin ≥ D), the clustering unit 40 newly adds the feature vector of the input sample as the centroid of the class to which the input sample belongs.
[0026] When Agg-Var clustering is applied to all of the training data sets for each session, the centroids of all classes in that session are determined. Since the centroids of the previous session are carried over, the centroids are updated in the subsequent session while including the centroids of the classes learned up to the previous session. When dealing with existing classes in the subsequent session as described above, the Agg-Var clustering process is performed with the centroids already generated for those existing classes as the initial state.
[0027] In the inference phase, the feature extraction unit 10 supplies the feature vector extracted from the unknown sample to the label prediction unit 60. The label prediction unit 60 predicts the class label of each sample based on the determined centroids stored in the centroid storage unit 50.
[0028] In the label prediction unit 60, the nearest centroid selection unit 70 selects the n nearest centroids for the feature vector of the unknown sample considering the distances from all centroids of each class.
[0029] The weighted voting unit 80 performs weighted voting by calculating a predicted score Pred(y) for each candidate class y to which each of the n centroids selected by the neighborhood centroid selection unit 70 belongs, with weighting according to the distance between the feature vector of the unknown sample and the n neighborhood centroids. For example, the reciprocal of the distance is used as the weight according to the distance. In other words, the shorter the distance, the larger the weight given to the vote.
[0030] The weighted voting unit 80 predicts the label assigned to the candidate class having the maximum prediction score Pred(y) as the class label of the unknown sample.
[0031] In addition, the CBCL processing such as Agg-Var clustering and calculation of prediction scores can be performed using the method described in Patent Document 1.
[0032] In conventional CBCL, the threshold D, which is a parameter of Agg-Var clustering, is a fixed value that is not controlled and is used once per learning. Here, we consider the problems that arise when processing each session as described in Fig. 1 and Fig. 2 with the threshold D fixed according to conventional CBCL.
[0033] Training datasets are usually selected so that the number of samples per class and the feature distribution are not significantly biased for each set. However, when comparing the base class dataset and the new class dataset, the number of samples per class and the feature distribution are likely to be different.
[0034] In the CBCL algorithm, since it is desirable to use a model with excellent feature extraction capabilities as the trained model, the trained model should be based on big data, and it is assumed that the training data for the base class is, for example, several hundred samples or more per class. On the other hand, the training data for the new class may be big data, but it is preferable that it is small data, that is, the number of samples is small. In particular, as described in Patent Document 1, it is ideal to use a few-shot format, that is, training data with, for example, only a few samples per class. Learning using big data has practical issues such as the effort required to prepare many samples in advance and the need for long learning times, and incremental learning using small data is preferable in order to gradually learn in a few times like humans.
[0035] Therefore, in FIG. 1, if the threshold D, which is a parameter of Agg-Var clustering, is common in both the initial session and the additional session, it is expected that the number of centroids will be biased between the base class and the new class. In general, if the base class is big data, the number of centroids will be large, and if the new class is small data, the number of centroids will be small. In that case, in the label prediction process, voting based on the centroid will be dominant for the base class, and voting will not be performed fairly between the base class and the new class, which is expected to deteriorate the classification accuracy. In addition, although many centroids of the base class will be stored, they will contain many unnecessary centroids compared to the centroids of the new class, which is expected to reduce the memory usage efficiency.
[0036] Therefore, in the first embodiment, the threshold D, which is a parameter of Agg-Var clustering, is set to an appropriate value for each of the initial session in which the training data set of the base class is clustered and the additional session in which the training data set of the new class is clustered. The threshold Df of the initial session and the threshold Di of the additional session are set to be larger than the threshold Di, and are controlled so that the centroids of the base class are as few as possible. In other words, the threshold Di is set to be smaller than the threshold Df, and are controlled so that the centroids of the new class are as many as possible. This improves the fairness of voting based on centroids and improves the classification accuracy compared to the case where the threshold D is common to the base class and the new class. Furthermore, by reducing the centroids of the base class, unnecessary centroids do not need to be stored, and the efficiency of memory usage can be improved.
[0037] It is desirable that the threshold Df for the initial session and the threshold Di for the additional session are set so that the distribution of distances between centroids in a certain class is similar. For example, the thresholds Df and Di are set based on the probability distribution of the feature data, based on the original training data or feature data extracted using a pre-trained model. If we assume that the probability distribution follows a normal distribution, the larger the average squared variance σ for each class of the training data for the base class, the greater the relationship of Df>>Di, that is, the greater the difference between the values of the thresholds Df and Di. This makes it possible to reduce the number of centroids for each class in the base class as much as possible.
[0038] 3(a) and 3(b) are flowcharts illustrating the machine learning procedure performed by the machine learning device 100 of FIG.
[0039] Figure 3(a) shows the flow of the clustering process in the learning stage. The threshold Df of the initial session is set as the threshold D for clustering (S30). The clustering process of the initial session is executed (S32). The threshold Di of the additional session is set for the threshold D (S34). The clustering process of the additional session is executed (S36). When clustering the next additional session (Y in S38), it returns to step S36, and if not (N in S38), it ends.
[0040] Figure 3(b) shows the detailed procedures of the clustering process for the initial session in step S32 and the clustering process for the additional session in Figure 3(a). The clustering processes for the initial session and the additional session are the same. In the clustering process of the initial session, samples of the training dataset of the base class are input, and in the clustering process of the additional session, samples of the training dataset of the new class are input.
[0041] The feature extraction unit 10 extracts the feature vector of the input sample using the learned model (S40).
[0042] The clustering unit 40 selects the nearest centroid closest to the feature vector of the input sample from among the multiple centroids of the class to which the input sample belongs (S42). The clustering unit 40 compares the distance dmin between the feature vector of the input sample and the nearest centroid with the threshold D (S44).
[0043] When dmin < D (Y in S46), the clustering unit 40 integrates the feature vector of the input sample into the nearest centroid (S48). When dmin ≥ D (N in S46), the feature vector of the input sample is newly added as the centroid of the class to which it belongs (S50).
[0044] (Second Embodiment) 4 is a configuration diagram of an inference device 200 according to the second embodiment. The inference device 200 includes a feature extraction unit 10, a trained model storage unit 20, a centroid generation unit 30, a centroid storage unit 50, and a label prediction unit 60. The centroid generation unit 30 includes a clustering unit 40, and the label prediction unit 60 includes a neighboring centroid selection unit 70 and a weighted voting unit 80.
[0045] Machine learning device 100 of the first embodiment and inference device 200 of the second embodiment differ in the configurations and operations of clustering unit 40 and weighted voting unit 80, but have the other configurations and operations in common.
[0046] In the clustering unit 40 of the machine learning device 100 of the first embodiment, the threshold value D, which is a parameter used in clustering, is different between the initial session and the additional session, but in the inference device 200 of the second embodiment, the parameters used in clustering may be common to all additional sessions, without performing base class clustering as in conventional CBCL. Alternatively, when the inference device 200 of the second embodiment also performs clustering of the initial session as in the first embodiment, the threshold value D, which is a parameter used in clustering, may also be different between the initial session and the additional session in the clustering unit 40 of the inference device 200.
[0047] The configuration and operation of weighted voting section 80 in label prediction section 60 of inference device 200 of the second embodiment will now be described.
[0048] The weighted voting unit 80 calculates a prediction score Pred(y) for each candidate class y to which each of the n centroids selected by the nearby centroid selection unit 70 belongs by weighting the candidate class according to the distance between the feature vector of the unknown sample and the centroid, and further multiplies the prediction score Pred(y) by a correction coefficient A to calculate a corrected prediction score Pred(y)', and predicts the label assigned to the class for which the corrected prediction score Pred(y)' is maximum as the label of the unknown sample.
[0049] Here, in conventional CBCL, this correction factor A is the number of samples in the training data of the class, N y That is, A=1 / N y and Pred(y)'=(1 / N y )Pred(y).
[0050] This correction coefficient A=1 / N y is based on the premise that a class with a large number of samples is likely to have a large number of centroids, and therefore, if the decision is based solely on the maximum value of Pred(y), the class with the large number of samples may be dominant, which is unfair.
[0051] Here, the training dataset is usually selected so that there is no significant bias in the number of samples per class and the feature distribution for each set. However, in an ideal application, it is desirable for sessions to be repeated semi-permanently, so the number of samples per class and the feature distribution are not necessarily uniform in all sessions. In addition, in conventional CBCL, it is assumed that new classes are continuously learned without assuming that the base class is classified, but there are many applications in which CBCL is extended to classify both base classes and new classes. In such cases, the base classes are often big data (e.g., several hundred samples per class or more) and the new classes are small data, ideally in a few-shot format (e.g., about a few samples per class).
[0052] If the number of samples and feature distributions are uniform across all sessions and classes, the correction coefficient A is the same as in conventional CBCL, A=1 / N y However, if this is not the case, the correction coefficient A should be set to 1 / N. yIn this case, the predicted score Pred(y)' of a candidate class with a larger number of samples will be smaller, making it less likely that the candidate class will be selected as a label, and the predicted score Pred(y)' of a candidate class with a smaller number of samples will be larger, making it more likely that the candidate class will be selected as a label. Therefore, it is expected that voting will not be fair depending on the number of samples for each class, and classification accuracy will deteriorate.
[0053] Therefore, in the second embodiment, the number of centroids for each class is reflected in the correction coefficient A by which Pred(y) is multiplied. * y For example, the correction coefficient A is 1 / N, which is the reciprocal of the centroid number. * y By setting the weighting factor to 1, the influence of each centroid on class label prediction, i.e., the energy, is made equal across classes. This improves the fairness of centroid-based voting and prevents deterioration of classification accuracy even when the number of samples or feature distributions in each class are not uniform.
[0054] In this way, in the second embodiment, the weighted voting unit 80 applies a correction coefficient A=1 / N * y The corrected prediction score Pred(y)' is calculated by multiplying the corrected prediction score Pred(y)' by 0, and the class for which the corrected prediction score Pred(y)' is maximum is predicted as the class label of the unknown sample.
[0055] FIG. 5 is a flowchart illustrating an inference procedure by inference device 200 of FIG.
[0056] The feature extraction unit 10 extracts a feature vector of an unknown sample using the trained model (S70).
[0057] The neighborhood centroid selection unit 70 selects n neighborhood centroids for the feature vector of the unknown sample (S72).
[0058] The weighted voting unit 80 assigns weights according to the distance between the feature vector of the unknown sample and the n-neighborhood centroid, and calculates the predicted score of the candidate class to which the n-neighborhood centroid belongs (S74). The weighted voting unit 80 corrects the predicted score of the candidate class with a correction coefficient according to the number of centroids of the candidate class (S76). The weighted voting unit 80 predicts that the class with the maximum corrected predicted score is the class label (S78).
[0059] The various processes of the machine learning device 100 and the inference device 200 described above can be realized not only as devices using hardware such as a CPU and a memory, but also as firmware stored in a ROM (read-only memory) or a flash memory, or as software for a computer, etc. The firmware program or software program can be provided by recording it on a recording medium readable by a computer, etc., or can be transmitted and received with a server via a wired or wireless network, or can be transmitted and received as data broadcasting of terrestrial or satellite digital broadcasting.
[0060] As described above, according to the machine learning device 100 of the first embodiment, when CBCL is extended to incremental learning that combines an initial session for clustering the base class and an additional session for clustering the new class, by differentiating the parameters used in clustering between the base class and the new class, it is possible to improve the classification accuracy of the base class and the new class and reduce the number of unnecessary centroids to make memory use more efficient.
[0061] According to the inference device 200 of the second embodiment, by reflecting the centroid number in the correction coefficient when voting for the candidate class in the label prediction process of CBCL, it is possible to prevent the classification accuracy from deteriorating even if there is a bias in the number of samples or feature distribution of the training data for each class. In particular, when CBCL is extended to perform clustering of both the base class and the new class, there is often a large bias in the number of samples and feature distribution between the base class and the new class, but fairness can be ensured by correcting the vote for the candidate class based on the centroid number.
[0062] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each treatment process, and that such modifications are also within the scope of the present invention. [Explanation of symbols]
[0063] 10 feature extraction unit, 20 trained model storage unit, 30 centroid generation unit, 40 clustering unit, 50 centroid storage unit, 60 label prediction unit, 70 nearby centroid selection unit, 80 weighted voting unit, 100 machine learning device, 200 inference unit.
Claims
1. a feature extraction unit that extracts a feature vector of an input sample using the trained model; a neighborhood centroid selection unit for selecting a plurality of centroids for each class that are in the neighborhood of the feature vector of the input sample; a weighted voting unit that calculates a prediction score of a candidate class to which the selected centroid belongs by weighting the feature vector of the input sample according to a distance between the feature vector of the input sample and the selected centroid, calculates a corrected prediction score by correcting the prediction score of the candidate class by a correction coefficient according to the number of centroids of the candidate class, and predicts the class with the maximum corrected prediction score as the class of the input sample, The inference device, wherein the correction coefficient is the inverse of the centroid number of the candidate class.
2. a clustering unit that generates a centroid for each class by clustering samples of a training data set of a base class used in generating the trained model, and then clusters samples of a training data set of a new class to be newly added to generate a centroid for each class; 2. The inference device according to claim 1, wherein the weighted voting unit corrects the predicted score of the candidate class by a correction coefficient corresponding to the number of centroids of the base class when the candidate class is a base class, and corrects the predicted score of the candidate class by a correction coefficient corresponding to the number of centroids of the new class when the candidate class is a new class.
3. a feature extraction step of extracting a feature vector of an input sample using the trained model; a neighborhood centroid selection step of selecting a plurality of centroids for each class that are in the neighborhood of the feature vector of the input sample; A weight is assigned according to the distance between the feature vector of the input sample and the selected centroid to calculate a prediction score of a candidate class to which the selected centroid belongs, and a corrected prediction score is calculated by correcting the prediction score of the candidate class with a correction coefficient according to the number of centroids of the candidate class, and the class with the maximum corrected prediction score is designated as the class of the input sample. and a weighted voting step for predicting the The inference method, wherein the correction factor is the inverse of the centroid number of the candidate class.
4. a feature extraction step of extracting a feature vector of an input sample using the trained model; a neighborhood centroid selection step of selecting a plurality of centroids for each class that are in the neighborhood of the feature vector of the input sample; a weighted voting step of calculating a prediction score of a candidate class to which the selected centroid belongs by weighting the feature vector of the input sample according to the distance between the feature vector of the input sample and the selected centroid, calculating a corrected prediction score by correcting the prediction score of the candidate class by a correction coefficient according to the number of centroids of the candidate class, and predicting the class with the maximum corrected prediction score as the class of the input sample; An inference program, wherein the correction coefficient is the inverse of the centroid number of the candidate class.
Citation Information
Patent Citations
Abnormal behavior sample generation method and system
CN113095446A
Sound processing device and sound processing method
JP2012181280A
Information processing device, information processing method, and information processing program
JP2020184235A
Method and apparatus for determining training of unknown data related to neural networks
KR1020190004429A
Anomaly Detection Method
KR1020200056183A
