Machine learning device, machine learning method, and machine learning program

Abstract

To provide a machine learning technique that makes it possible to improve the accuracy of class classification.SOLUTION: A new class image generation unit 61 processes an image of a basic class and generates an image of a new class. An image feature amount output unit 62 is learned in advance by the image of the basic class, and outputs image feature amounts by using the image of the basic class or the image of the new class as an input. A language classification weight output unit 65 is learned in advance by the image of the basic class and a sentence explaining the image of the basic class, and outputs a language classification weight by using the sentence explaining the image of the basic class as an input. An image classification weight output unit 64 calculates an average value of the image feature amounts for each class by using the image feature amounts as an input, and outputs the result as an image classification weight for each class. An optimization unit 66 optimizes the image classification weight by using the image classification weight and the language classification weight as inputs, and outputs a reconstruction classification weight.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to machine learning techniques. [Background technology]

[0002] While humans can learn new knowledge and retain old knowledge through long-term experience, the knowledge of a Convolutional Neural Network (CNN) depends on the dataset it was trained on, and adapting to changes in the data distribution requires retraining the CNN parameters on the entire dataset.

[0003] A more efficient and practical method is incremental learning, or continual learning, which learns new tasks while reusing previously acquired knowledge. In particular, incremental learning in classification tasks is a method that allows a system to learn and classify new classes (new classes) from a state in which it can classify base classes (classes learned in the past).

[0004] On the other hand, in deep learning, there is a phenomenon called catastrophic forgetting, in which previously acquired knowledge is largely lost, resulting in a significant decline in task performance. This phenomenon is particularly problematic in continuous learning. In continuous learning for classification tasks, the biggest challenge is to suppress catastrophic forgetting and acquire classification performance for new classes while maintaining classification performance for basic classes.

[0005] On the other hand, since new tasks often only have a small amount of sample data available, few-shot learning has been proposed as a method for efficient learning with a small amount of training data. Normally, several thousand or more samples are required for learning, but in few-shot learning, learning is performed with a small number of samples (for example, a few samples).

[0006] In addition, Class Incremental Learning (CIL) has been proposed, which allows classification of new classes (new classes) by additionally learning base classes to a trained model. In CIL, tasks are continuously added to a trained model for class classification, and classification performance for new classes and past classes is required for new tasks. Note that the training data for new tasks is usually big data.

[0007] A method called Few-Shot Class Incremental Learning (FSCIL) has been proposed that combines continuous learning, which learns new classes without fatal forgetting of the learning results of base classes, with few-shot learning, which learns new classes that are fewer in number than the base classes (Non-Patent Document 1). Continuous few-shot learning can learn base classes from a large data set, and new classes from a small amount of sample data. FSCIL is an incremental learning scenario for class classification that is similar to CIL, but it is significantly different in that the learning data for new tasks is small (small data).

[0008] Continually Evolved Classifiers (CEC) has been proposed as a continuous few-shot learning method (Non-Patent Document 1). CEC constructs a pseudo continuous learning task by rotating the original image into a base class image, which is a pseudo new class image, and learns a graph attention network (GAT). [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] Zhang, C., Song, N., Lin, G., Zheng, Y., Pan, P., & Xu, Y. (2021). Few-shot incremental learning with continually evolved classifiers. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 12455-12464). [Non-Patent Document 2] Nishida, K., Nishida, K., & Nishioka, S. (2022). Improving Few-Shot Image Classification Using Machine-and User-Generated Natural Language Descriptions. arXiv preprint arXiv:2207.03133. [Non-Patent Document 3] Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020, November). A simple framework for contrastive learning of visual representations. In International conference on machine learning (pp. 1597-1607). PMLR. Summary of the Invention [Problem to be solved by the invention]

[0010] In the method described in Non-Patent Document 1, since the feature representation for classifying images of basic classes has already been trained, there is a possibility that the training of the graph model may be insufficient if only rotated trained images are used. Therefore, there is a problem that sufficient classification accuracy may not be obtained.

[0011] The present invention has been made in consideration of these circumstances, and its purpose is to provide a machine learning technique that can improve classification accuracy by optimizing feature representation using linguistic information including visual concepts of basic classes. [Means for solving the problem]

[0012] In order to solve the above problem, a machine learning device according to one embodiment of the present invention is a machine learning device that continuously learns a smaller number of new class images compared to base class images, and includes: a new class image generation unit that processes the base class images to generate new class images; an image feature output unit that has been pre-trained with the base class images, the image feature output unit taking the base class images or the new class images as input and outputting image features; a language classification weight output unit that has been pre-trained with the base class images and sentences describing the base class images, the language classification weight output unit taking the sentences describing the base class images as input and outputting language classification weights; an image classification weight output unit that takes the image features as input, calculates an average value of the image features for each class, and outputs this as an image classification weight for each class; an optimization unit that takes the image classification weights and the language classification weights as inputs, optimizes the image classification weights, and outputs a reconstructed classification weight; and a classification unit that uses the reconstructed classification weights as class classification weights, and outputs a class classification from the image features output by the image feature output unit and the class classification weights.

[0013] Another aspect of the present invention is a machine learning method, which is a machine learning method for continuously learning a small number of new class images compared to a base class image, and includes a new class image generating step of processing the base class image to generate a new class image, an image feature output step of outputting an image feature using an image feature output module pre-trained with the base class image and an image of the new class, a language classification weight output step of outputting a language classification weight using an image of the base class and a sentence explaining the image of the base class, an image classification weight output step of inputting the image feature, calculating an average value of the image feature for each class, and outputting it as an image classification weight for each class, an optimization step of inputting the image classification weight and the language classification weight, optimizing the image classification weight, and outputting a reconstructed classification weight, and a classification step of using the reconstructed classification weight as a weight for class classification and outputting a class classification from the image feature outputted by the image feature output step and the class classification weight.

[0014] Any combination of the above components, and any transformation of the present invention into a method, device, system, recording medium, computer program, etc., are also effective as aspects of the present invention. Effect of the Invention

[0015] According to the present invention, it is possible to provide a machine learning technique that can improve classification accuracy by optimizing feature representation using linguistic information including visual concepts of basic classes. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram for explaining a conventional CEC technique. [Diagram 2] 2(a) and 2(b) are diagrams showing the classification accuracy of the basic classes used in training the GAT versus the image rotation angle in the conventional CEC. [Diagram 3] FIG. 1 is a diagram showing an example of classification weights input to a GAT in conventional pseudo continuous learning. [Figure 4] 11 is a diagram showing an example of class classification weights and language classification weights input to a GAT in pseudo continuous learning according to the present embodiment. FIG. [Diagram 5] 5(a) and 5(b) are diagrams comparing the output of the GAT in pseudo continuous learning between the conventional technique and this embodiment. [Figure 6] FIG. 1 is a functional block diagram for explaining a configuration of a pseudo continuous learning module of a conventional machine learning device that uses CEC. [Figure 7] FIG. 2 is a functional block diagram for explaining a configuration of a pseudo continuous learning module of a machine learning device according to the present embodiment that uses a CEC. [Figure 8] FIG. 11 is a functional block diagram for explaining another configuration of the pseudo continuous learning module of the machine learning device according to the present embodiment that uses CEC. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] CEC is a method to improve FSCIL's issues of forgetting base classes and overfitting to new classes by separating the feature extractor and classifier and propagating contextual information between classifiers using a graph model.

[0018] Fig. 1 is a diagram for explaining a conventional CEC method. As shown in Fig. 1, CEC is composed of stages 1 to 3. A conventional machine learning device 100 includes a pre-training module 30 used in stage 1, a pseudo continuous learning module 40 used in stage 2, and a new class learning module 50 used in stage 3.

[0019] Stage 1 is a pre-learning stage. In stage 1, a large amount of base class data set (hereinafter, base data set) 10 is used in the pre-training module 30 to pre-learn the weights of the backbone CNN 32 of the pre-training module 30 by standard supervised learning. The base data set 10 includes N data samples. An example of the data sample is image data, but is not limited to this. For example, in the case of the CIFAR100 data set, the base data set 10 includes 60 classes x 500 images of image data. The base data set 10 may include data sets of multiple different classes. The backbone CNN 32 is a convolutional neural network pre-trained with the base data set 10. The backbone CNN 32 has weights of the feature extractor R and a base class classification weight W0, which is a weight vector of the base class classifier. The base class classification weight W0 indicates the average feature amount of the data samples of the base data set 10. By fixing the parameters of the feature extractor R of the pre-trained backbone CNN 32 in the subsequent stages, forgetting of the base class is suppressed.

[0020] Stage 2 is a pseudo continuous learning stage. In stage 2, the weights of GAT 44 are trained in the pseudo continuous learning module 40 to generate a classifier adapted to all classes by propagating the contextual information of each class. The GAT 44 is trained in an episodic manner by constructing a pseudo continuous learning task from a dataset of rotated images generated by rotating the images of the base dataset 10. Hereinafter, the dataset generated based on the base dataset 10 in the pseudo continuous learning stage is referred to as the pseudo dataset 15.

[0021] In stage 2, base class classification weights are learned based on feature vectors generated by inputting a pseudo dataset 15, which is another dataset of base classes, to a feature extractor R of a backbone CNN 32 pre-trained in stage 1. By inputting the base class classification weights W0 learned in stage 1 and the base class classification weights learned in stage 2 to a pseudo continuous learning module GAT 44, these base class classification weights are adapted and reconstructed in GAT 44, and a reconstructed classification weight 45 of W'0 is output. Hereinafter, the reconstructed classification weights output from GAT are referred to as reconstructed classification weights.

[0022] The episode format will now be described. Each episode is composed of a support set and a query set. In the pseudo continuous learning stage, both the support set and the query set are composed of the base dataset 10 and the pseudo dataset 15. In stage 2, in each episode, based on the support samples in the given support set, the query samples in both the base dataset 10 and the pseudo dataset 15 included in the query set are classified, and the parameters of GAT44 are updated so as to minimize the classification loss.

[0023] Here, the reason why the rotated images of the base class are used in the pseudo-continuous learning task is that the backbone CNN32 has already learned feature representations for successfully classifying the images of the base class in stage 1, and therefore GAT44 would not be able to learn well if the images of the base class were used as is. The parameters of GAT44 after learning are fixed in the subsequent stages.

[0024] Stage 3 is a classifier learning and adaptation stage. In stage 3, a new class learning module 50 learns a classifier using a small number of new class datasets (hereinafter, new datasets) 20 given per session, and all classifiers learned in the current session and previous sessions are input to a GAT 53 of the new class learning module 50, where all classifiers are adapted. The GAT 53 of the new class learning module 50 is a GAT learned in the pseudo-continuous learning stage. Inference of a query is performed by the classifier adapted by the GAT 53. The new dataset 20 includes k data samples, which is smaller than the base dataset 10. The new dataset 20 may include datasets of multiple different classes.

[0025] In stage 3, new class classification weights are learned for each session based on the feature vector generated by inputting the new dataset 20 to the feature extractor R of the backbone CNN 32 pre-trained in stage 1. By inputting the basic class classification weight W0 learned in stage 1 and all new class classification weights {W1, ..., Wi} learned in each session up to the i-th session in stage 3 to the GAT 53 of the new class learning module 50, the classification weights of all classes input to the GAT 53 are adapted and reconstructed, and the reconstructed classification weights 54 of {W'0, W'1, ..., W'i} are output from the GAT 53.

[0026] In the machine learning device 200 of this embodiment, attention is focused on the pseudo continuous learning in stage 2, and improvements are made to the pseudo continuous learning module 40. The rest of the configuration is the same as that of the conventional machine learning device 100.

[0027] Figures 2(a) and 2(b) show the classification accuracy versus rotation angle of the base class images used in GAT training in the conventional CEC. Figure 2(a) shows the average classification accuracy versus rotation angle, and Figure 2(b) shows the rate of decline in average classification accuracy from the initial session to the final session versus rotation angle. Figure 2(a) confirms that classification accuracy is high when the rotation angles are 90°, 180°, and 270°. Figure 2(b) also shows that the rate of decline in average classification accuracy from the initial session to the final session is small when the rotation angles are 90°, 180°, and 270°, and that forgetting of the base class is suppressed. From this, it is considered that in pseudo-continuous learning, it is desirable to use images that are visually far removed from the base class images in terms of improving classification accuracy.

[0028] Figure 3 shows an example of classification weights input to the GAT in conventional pseudo-continuous learning. Here, the classification weights of each class in the feature space are visualized in a two-dimensional space. Classification weights are also called "prototypes."

[0029] GAT is input with 5 prototypes of randomly selected base classes (B1-B5) and 5 prototypes of new classes (N1-N5) artificially generated by rotating the images of the base classes. The new classes N1, N2, N3, N4, and N5 are the rotated versions of the base classes B1, B2, B3, B4, and B5, respectively.

[0030] As shown in Figure 3, the prototype of the base class averaged image features and the prototype of the new class averaged features of the rotated image of that base class are visually similar and close in the feature space, so the GAT may not be able to learn well. Furthermore, the GAT parameters are optimized only by the cross-entropy loss, so the adjustment of the prototype may be limited.

[0031] 4 is a diagram showing an example of prototypes and language classification weights input to the GAT in the pseudo continuous learning of this embodiment. Here, the prototypes and language classification weights of each class in the feature space are visualized in a two-dimensional space.

[0032] GAT is input with 5 randomly selected prototypes of base classes (B1-B5), 5 prototypes of new classes (N1-N5) artificially generated by rotating the images of the base classes, and the language classification weights (T1-T5) of the 5 base classes (B1-B5). The new classes N1, N2, N3, N4, and N5 are the rotated versions of the base classes B1, B2, B3, B4, and B5, respectively. The language classification weights T1, T2, T3, T4, and T5 are the language classification weights of the base classes B1, B2, B3, B4, and B5, respectively.

[0033] Here, as an example, the language classification weights use feature representations including visual concepts of basic classes generated by a trained Text Encoder model that has been fully trained with big data of image and language pairs, as described in Non-Patent Document 2. Specifically, the language classification weights of the basic classes are generated from sentences that explain the images of the basic classes using a Text Encoder. Note that the language classification weights are referred to as text representations or text features in Non-Patent Document 2.

[0034] In addition to the conventional cross-entropy loss, the contrastive loss described in non-patent document 3 is used for learning the parameters of the GAT. Specifically, the parameters of the GAT are pseudo-continuously learned so that the prototype of the base class approaches the language classification weights including the visual concept of the base class, and the prototype of the new class moves away from the language classification weights of the base class from which it is rotated.

[0035] 5(a) and 5(b) are diagrams comparing the output of the GAT in the pseudo continuous learning between the conventional technique and this embodiment. The output of the GAT is a prototype after being adjusted by the pseudo continuous learning.

[0036] As shown in Figure 5(a), the amount of movement of the prototype in the conventional technology is small, which limits the separation accuracy.

[0037] As shown in Fig. 5(b), in this embodiment, the prototype of the base class after adjustment approaches the language classification weight of the base class, and the prototype of the new class after adjustment moves away from the language classification weight of the base class from which it was rotated. In this way, the GAT can be effectively trained by optimization using the language classification weight, and the amount of movement of the prototype increases, improving the classification accuracy of each class.

[0038] 6 is a functional block diagram for explaining the configuration of a pseudo continuous learning module 40 of a conventional machine learning device 100 that uses CEC. The pseudo continuous learning module 40 includes a new class image generation unit 61, an image feature output unit 62, an image classification weight output unit 64, an optimization unit 66, and a classification unit 67.

[0039] The new class image generation unit 61 generates a pseudo dataset 15 of new class images by rotating the images of the basic classes of the basic dataset 10 used in the pre-training module 30 , and supplies it to the image feature output unit 62 .

[0040] The image feature output unit 62 receives the pseudo dataset 15 of the new class image as an input, extracts a feature vector of the pseudo dataset 15 of the new class image, and supplies the extracted image feature to the image classification weight output unit 64. The image feature output unit 62 corresponds to the feature extractor R of the backbone CNN 32 that has pre-trained the basic class classification weights in stage 1.

[0041] The image classification weight output unit 64 calculates image classification weights for the pseudo data set 15 of the new class image by averaging the feature vectors of the pseudo data set 15 of the new class image for each class, and supplies the weights to the optimization unit 66.

[0042] The optimization unit 66 corresponds to the GAT 44, and receives as input the basic class classification weights W0 of the backbone CNN 32 in which the classification weights of the basic classes have been pre-trained in stage 1, and the basic class classification weights of the pseudo dataset 15 supplied from the image classification weight output unit 64, and outputs the reconstructed classification weights by meta-learning the dependency between the basic dataset 10 and the pseudo dataset 15 and adapting all the input class classification weights. In the pseudo continuous learning module 40, meta-learning of the GAT, which is a meta module, is performed in an episode format. The parameters of the optimization unit 66 are optimized and updated for each episode using a query set consisting of the basic dataset 10 and the pseudo dataset 15. The optimization method of the optimization unit 66 uses a method of minimizing the cross-entropy loss described in Non-Patent Document 1. The optimization unit 66 supplies the reconstructed classification weights obtained in this manner to the classification unit 67.

[0043] The classification unit 67 uses the reconstructed classification weights as classification weights, and outputs a class classification from the image feature values ​​output by the image feature value output unit 62 and the classification weights.

[0044] 7 is a functional block diagram for explaining the configuration of the pseudo continuous learning module 40 of the machine learning device 200 of this embodiment that uses CEC. The pseudo continuous learning module 40 includes a new class image generation unit 61, an image feature output unit 62, an image classification weight output unit 64, a language classification weight output unit 65, an optimization unit 66, and a classification unit 67. Here, configurations and operations common to the functional blocks of the pseudo continuous learning module 40 of the conventional machine learning device 100 in FIG. 6 are omitted as appropriate, and different configurations and operations will be explained.

[0045] The language classification weight output unit 65 is pre-trained using basic class images and sentences (called "captions") that explain the basic class images, and uses the captions of the basic class images as input to generate language classification weights, which are language features of the basic class images, and supplies them to the optimization unit 66.

[0046] The optimization unit 66 receives the image classification weights and the language classification weights as input, optimizes the image classification weights, calculates reconstructed classification weights, and supplies them to the classification unit 67. Specifically, the optimization unit 66 calculates the reconstructed classification weights by minimizing the constructive loss so as to bring the image classification weights of the base class closer to the language classification weights and move the image classification weights of the new class away from the language classification weights.

[0047] In the pseudo continuous learning module 40 of the machine learning device 200 of this embodiment, by minimizing the constructive loss based on the language classification weights, the distance between classes is increased more than in the conventional technology, thereby improving classification accuracy.

[0048] 8 is a functional block diagram for explaining another configuration of the pseudo continuous learning module 40 of the machine learning device 200 of this embodiment that uses CEC. The pseudo continuous learning module 40 includes a new class image generation unit 61, an image feature output unit 62, a language feature output unit 63, an image classification weight output unit 64, a language classification weight output unit 65, an optimization unit 66, and a classification unit 67. Here, the functional block differs from that of the pseudo continuous learning module 40 of the machine learning device 200 of FIG. 7 in that a language feature output unit 63 is further provided.

[0049] When there are multiple captions for one image of the basic class, the language feature output unit 63 extracts language features from each of the multiple captions and supplies them to the language classification weight output unit 65. The language classification weight output unit 65 calculates a language classification weight by averaging the language features and supplies it to the optimization unit 66. The other configurations and operations are the same as those in Fig. 7.

[0050] The various processes of the machine learning device 200 described above can be realized not only as a device using hardware such as a CPU and a memory, but also as firmware stored in a ROM (read-only memory) or flash memory, or software for a computer, etc. The firmware program or software program can be provided by recording it on a recording medium readable by a computer, etc., or can be transmitted and received with a server via a wired or wireless network, or can be transmitted and received as a data broadcast of terrestrial or satellite digital broadcasting.

[0051] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each treatment process, and that such modifications are also within the scope of the present invention.

[0052] As a method for generating an image of a new class from an image of a base class, an image of the new class may be generated by processing the image of the base class in a manner other than rotating the image of the base class. For example, an image of the new class may be generated by dividing the image of the base class into a plurality of regions and exchanging the divided regions with each other. [Explanation of symbols]

[0053] 10 base dataset, 15 pseudo dataset, 20 new dataset, 30 pre-training module, 32 backbone CNN, 40 pseudo continuous learning module, 44, 53 GAT, 45, 54 reconstructed classification weights, 50 new class learning module, 61 new class image generation unit, 62 image feature output unit, 63 language feature output unit, 64 image classification weight output unit, 65 language classification weight output unit, 66 optimization unit, 67 classification unit, 100, 200 machine learning device.

Claims

1. A machine learning device that continuously learns a small number of images of a new class compared to images of a base class, a new class image generating unit that processes the image of the basic class to generate an image of a new class; an image feature output unit that has been pre-trained using an image of the base class, and outputs image features using an image of the base class or an image of the new class as input; a language classification weight output unit that is pre-trained using the image of the base class and a sentence describing the image of the base class, and outputs a language classification weight using the sentence describing the image of the base class as an input; an image classification weight output unit that receives the image feature amount, calculates an average value of the image feature amount for each class, and outputs the average value as an image classification weight for each class; an optimization unit that receives the image classification weights and the language classification weights as input, optimizes the image classification weights, and outputs reconstructed classification weights; a classification unit that uses the reconstructed classification weights as weights for class classification and outputs a class classification from the image features output by the image feature output unit and the weights for class classification.

2. The optimization unit is The machine learning device according to claim 1 , wherein the image classification weights of the base class are trained to be closer to the language classification weights, and the image classification weights of the new class are trained to be farther from the language classification weights.

3. A machine learning method for continuously learning a small number of images of a new class compared to images of a base class, comprising: a new class image generation step of processing the basic class image to generate a new class image; an image feature output step of outputting image features by using an image feature output module pre-trained by an image of the base class and inputting an image of the base class or an image of the new class; a language classification weight output step of outputting a language classification weight using a language classification weight output module that has been pre-trained using the image of the base class and a sentence describing the image of the base class, with the sentence describing the image of the base class as an input; an image classification weight output step of inputting the image feature amount, calculating an average value of the image feature amount for each class, and outputting the average value as an image classification weight for each class; an optimization step of optimizing the image classification weights and the language classification weights as inputs, optimizing the image classification weights, and outputting reconstructed classification weights; A machine learning method comprising: a classification step of using the reconstructed classification weights as weights for class classification, and outputting a class classification from the image features output by the image feature output step and the weights for class classification.

4. A machine learning program that continuously learns a small number of new class images compared to a base class image, a new class image generation step of processing the basic class image to generate a new class image; an image feature output step of outputting image features by using an image feature output module pre-trained by an image of the base class and inputting an image of the base class or an image of the new class; a language classification weight output step of outputting a language classification weight using a language classification weight output module that has been pre-trained using the image of the base class and a sentence describing the image of the base class, with the sentence describing the image of the base class as an input; an image classification weight output step of inputting the image feature amount, calculating an average value of the image feature amount for each class, and outputting the average value as an image classification weight for each class; an optimization step of optimizing the image classification weights and the language classification weights as inputs, optimizing the image classification weights, and outputting reconstructed classification weights; A machine learning program characterized by causing a computer to execute a classification step of using the reconstructed classification weights as class classification weights and outputting a class classification from the image features output by the image feature output step and the class classification weights.

Images