Machine learning device, machine learning method, and machine learning program

Abstract

A new class image generation unit (61) processes an image of a base class and generates an image of a new class. An image feature quantity output unit (62) has been pre-trained by the image of the base class and uses the image of the base class or the image of the new class as input to output an image feature quantity. A language classification weight output unit (65) has been pre-trained by the image of the base class and a sentence describing the image of the base class, and uses the sentence describing the image of the base class as input to output a language classification weight. An image classification weight output unit (64) uses the image feature quantity as input to calculate the average value of the image feature quantity for each class and outputs the average value as an image classification weight for each class. An optimization unit (66) uses the image classification weight and the language classification weight as input to optimize the image classification weight and outputs a reconstruction classification weight.

Description

Machine learning device, machine learning method, and machine learning program

[0001] The present invention relates to machine learning techniques.

[0002] While humans can learn new knowledge and retain old knowledge through long-term experience, the knowledge of a convolutional neural network (CNN) is dependent on the dataset it was trained on, and adapting to changes in the data distribution requires retraining the CNN parameters for the entire dataset.

[0003] A more efficient and practical approach is incremental learning, which reuses previously acquired knowledge while learning new tasks. In particular, incremental learning for classification tasks is a method that allows a system to learn and classify new classes (new classes) based on the ability to classify base classes (classes learned previously).

[0004] On the other hand, in deep learning, there is a phenomenon called catastrophic forgetting, in which previously acquired knowledge is significantly lost, resulting in a significant decline in task performance. This phenomenon is particularly problematic in continuous learning. In continuous learning for classification tasks, the biggest challenge is to suppress catastrophic forgetting and acquire classification performance for new classes while maintaining classification performance for basic classes.

[0005] On the other hand, since new tasks often require only a small amount of sample data, few-shot learning has been proposed as a method for efficiently learning with a small amount of training data. While learning normally requires several thousand or more samples, few-shot learning uses only a small number of samples (e.g., a few samples).

[0006] In addition, Class Incremental Learning (CIL) has been proposed, which allows for the classification of new classes (new classes) by additionally training base classes on a trained model. In CIL, tasks are continuously added to a trained model for classification, and classification performance for new and past classes is required for new tasks. Note that the training data for new tasks is usually big data.

[0007] A method called Few-Shot Class Incremental Learning (FSCIL) has been proposed (Non-Patent Document 1), which combines continuous learning, which learns new classes without fatal forgetting from the learning results of base classes, with few-shot learning, which learns new classes that are fewer in number than the base classes. Continuous few-shot learning can learn base classes from a large data set and new classes from a small number of sample data. FSCIL is an incremental learning scenario for classification similar to CIL, but differs significantly in that the training data for new tasks is small (small data).

[0008] Continually Evolved Classifiers (CEC) has been proposed as a continuous few-shot learning method (Non-Patent Document 1). CEC constructs a pseudo continuous learning task by rotating the original image into a base class image, which is then used as a pseudo new class image, and trains a graph attention network (GAT).

[0009] Zhang, C., Song, N., Lin, G., Zheng, Y., Pan, P., & Xu, Y. (2021). Few-shot incremental learning with continually evolved classifiers. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 12455-12464). Nishida, K., Nishida, K., & Nishioka, S. (2022). Improving Few-Shot Image Classification Using Machine-and User-Generated Natural Language Descriptions. arXiv preprint arXiv:2207.03133.Chen, T., Kornblith, S., Norouzi, M., & Hinton, G. (2020, November). A simple framework for contrastive learning of visual representations. In International conference on machine learning (pp. 1597-1607). PMLR.

[0010] In the method described in Non-Patent Document 1, the feature representation for classifying images into basic classes has already been trained, so simply using trained images that have only been rotated can result in insufficient training of the graph model, which can lead to the problem of not being able to achieve sufficient classification accuracy.

[0011] This embodiment has been made in consideration of the above circumstances, and its purpose is to provide a machine learning technology that can improve classification accuracy by optimizing feature representation using linguistic information that includes the visual concepts of basic classes.

[0012] In order to solve the above problem, a machine learning device according to one aspect of the present embodiment is a machine learning device that continuously learns a smaller number of new class images than base class images, and includes: a new class image generation unit that processes the base class images to generate new class images; an image feature output unit that has been pre-trained with the base class images, and that takes the base class images or the new class images as input and outputs image features; a language classification weight output unit that has been pre-trained with the base class images and sentences describing the base class images, and that takes the sentences describing the base class images as input and outputs language classification weights; an image classification weight output unit that takes the image features as input, calculates an average value of the image features for each class, and outputs this as an image classification weight for each class; an optimization unit that takes the image classification weights and the language classification weights as input, optimizes the image classification weights, and outputs reconstructed classification weights; and a classification unit that uses the reconstructed classification weights as classification weights, and outputs a class classification from the image features output by the image feature output unit and the class classification weights.

[0013] Another aspect of the present embodiment is a machine learning method for continuously learning a smaller number of new class images than base class images, the method including: a new class image generation step of processing the base class images to generate new class images; an image feature output step of using an image feature output module pre-trained with the base class images to output image features from the base class images or the new class images; a language classification weight output step of using a language classification weight output module pre-trained with the base class images and a sentence describing the base class images to output language classification weights from the sentence describing the base class images; an image classification weight output step of receiving the image features as input, calculating an average value of the image features for each class, and outputting the average value as an image classification weight for each class; an optimization step of receiving the image classification weights and the language classification weights as input, optimizing the image classification weights, and outputting reconstructed classification weights; and a classification step of using the reconstructed classification weights as classification weights, and outputting a class classification from the image features output by the image feature output step and the class classification weights.

[0014] Any combination of the above components, and conversion of the expression of this embodiment into a method, device, system, recording medium, computer program, etc. are also valid aspects of this embodiment.

[0015] According to this embodiment, it is possible to provide a machine learning technique that can improve classification accuracy by optimizing feature representation using linguistic information that includes the visual concept of a basic class.

[0016] 2(a) and 2(b) are diagrams illustrating the classification accuracy versus the rotation angle of an image of a basic class used in training a GAT in a conventional CEC. FIG. 2(a) is a diagram illustrating an example of class classification weights input to a GAT in conventional pseudo continuous learning. FIG. 2(b) is a diagram illustrating an example of class classification weights and language classification weights input to a GAT in pseudo continuous learning of the present embodiment. FIG. 5(a) and FIG. 5(b) are diagrams comparing the output of a GAT in pseudo continuous learning between the prior art and the present embodiment. FIG. 5(a) is a functional block diagram illustrating the configuration of a pseudo continuous learning module of a conventional machine learning device that uses a CEC. FIG. 5(b) is a functional block diagram illustrating the configuration of a pseudo continuous learning module of a machine learning device of the present embodiment that uses a CEC. FIG. 5(b) is a functional block diagram illustrating another configuration of the pseudo continuous learning module of a machine learning device of the present embodiment that uses a CEC.

[0017] CEC is a method to improve the problems of FSCIL, namely forgetting base classes and overfitting to new classes, by separating the feature extractor and classifier and propagating contextual information between classifiers using a graph model.

[0018] Fig. 1 is a diagram illustrating a conventional CEC technique. As shown in Fig. 1, CEC is composed of stages 1 to 3. A conventional machine learning device 100 includes a pre-training module 30 used in stage 1, a pseudo continuous learning module 40 used in stage 2, and a new class learning module 50 used in stage 3.

[0019] Stage 1 is the pre-training stage. In Stage 1, a large base class dataset (hereinafter referred to as the base dataset) 10 is used in the pre-training module 30 to pre-train the weights of the backbone CNN 32 of the pre-training module 30 using standard supervised learning. The base dataset 10 includes N data samples. An example of a data sample is image data, but this is not limited to this. For example, in the case of the CIFAR100 dataset, the base dataset 10 includes 60 classes x 500 images of image data. The base dataset 10 may also include datasets of multiple different classes. The backbone CNN 32 is a convolutional neural network pre-trained on the base dataset 10. The backbone CNN 32 has weights for the feature extractor R and a base class classification weight W0, which is a weight vector for the base class classifier. The base class classification weight W0 indicates the average feature value of the data samples in the base dataset 10. By fixing the parameters of the feature extractor R of the pre-trained backbone CNN 32 in subsequent stages, forgetting of the base classes is suppressed.

[0020] Stage 2 is the pseudo continuous training stage. In Stage 2, the pseudo continuous training module 40 trains the weights of the GAT 44 to propagate contextual information for each class and generate a classifier that is adapted to all classes. The GAT 44 is trained episodically by constructing a pseudo continuous training task from a dataset of rotated images generated by rotating images in the base dataset 10. Hereinafter, the dataset generated based on the base dataset 10 in the pseudo continuous training stage is referred to as the pseudo dataset 15.

[0021] In stage 2, base class classification weights are learned based on feature vectors generated by inputting a pseudo dataset 15, which is another dataset of base classes, into the feature extractor R of the backbone CNN 32 pre-trained in stage 1. The base class classification weights W0 learned in stage 1 and the base class classification weights learned in stage 2 are input into the pseudo continuous learning module GAT 44, where these base class classification weights are adapted and reconstructed, thereby outputting reconstructed classification weights 45 of W'0. Hereinafter, the reconstructed classification weights output from the GAT will be referred to as reconstructed classification weights.

[0022] The episode format will now be described. Each episode consists of a support set and a query set. In the pseudo continuous learning stage, the support set and the query set are both composed of a base dataset 10 and a pseudo dataset 15. In Stage 2, in each episode, query samples in both the base dataset 10 and the pseudo dataset 15 included in the query set are classified based on the support samples in the given support set, and the parameters of the GAT 44 are updated to minimize the classification loss.

[0023] The reason why rotated images of the base class are used in the pseudo-continuous training task is that the backbone CNN 32 has already learned feature representations for successfully classifying images of the base class in Stage 1, and therefore, if the images of the base class are used as they are, the GAT 44 will not be successfully trained. The parameters of the GAT 44 after training are fixed in the subsequent stages.

[0024] Stage 3 is a classifier learning and adaptation stage. In Stage 3, a new class learning module 50 learns classifiers using a small number of new class datasets (hereinafter referred to as new datasets) 20 given per session, and all classifiers learned in the current session and previous sessions are input into a GAT 53 of the new class learning module 50, where all classifiers are adapted. The GAT 53 of the new class learning module 50 is a GAT learned in the pseudo-continuous learning stage. Query inference is performed using the classifiers adapted by the GAT 53. The new dataset 20 includes k data samples, which is smaller than the base dataset 10. The new dataset 20 may include datasets of multiple different classes.

[0025] In Stage 3, new class classification weights are learned for each session based on the feature vectors generated by inputting the new dataset 20 to the feature extractor R of the backbone CNN 32 pre-trained in Stage 1. By inputting the base class classification weight W0 learned in Stage 1 and all new class classification weights {W1, ..., W1} learned in each session up to the i-th session in Stage 3 into the GAT 53 of the new class learning module 50, the classification weights of all classes input to the GAT 53 are adapted and reconstructed, and the GAT 53 outputs reconstructed classification weights 54 of {W'0, W'1, ..., W'i}.

[0026] In the machine learning device 200 of this embodiment, attention is focused on the pseudo continuous learning in stage 2, and improvements are made to the pseudo continuous learning module 40. The rest of the configuration is the same as that of the conventional machine learning device 100.

[0027] 2(a) and 2(b) show the classification accuracy versus rotation angle of the base class images used in GAT training in conventional CEC. FIG. 2(a) shows the average classification accuracy versus rotation angle, and FIG. 2(b) shows the rate of decline in average classification accuracy from the initial session to the final session versus rotation angle. From FIG. 2(a), it can be seen that classification accuracy is high when the rotation angles are 90°, 180°, and 270°. Furthermore, from FIG. 2(b), it can be seen that the rate of decline in average classification accuracy from the initial session to the final session is small when the rotation angles are 90°, 180°, and 270°, suppressing forgetting of the base class. From this, it is considered desirable to use images that are visually far removed from the base class images in pseudo-continuous learning from the perspective of improving classification accuracy.

[0028] Figure 3 shows an example of classification weights input to the GAT in conventional pseudo-continuous learning. The classification weights for each class in the feature space are visualized in two-dimensional space. The classification weights are also called "prototypes."

[0029] The GAT is input with five prototype classes (B1 to B5) of randomly selected base classes and five prototype classes (N1 to N5) of pseudo-new classes generated by rotating the images of the base classes. The new classes N1, N2, N3, N4, and N5 are rotated versions of the base classes B1, B2, B3, B4, and B5, respectively.

[0030] As shown in Figure 3, there is no significant visual difference between a prototype obtained by averaging the image features of a base class and a prototype obtained by averaging the features of a rotated image of that base class, and they are located close to each other in the feature space, which may result in insufficient training of the GAT. Furthermore, since the training of the GAT parameters is optimized solely by the cross-entropy loss, the adjustment of the prototype may be limited.

[0031] 4 is a diagram showing an example of prototypes and language classification weights input to the GAT in the pseudo-continuous learning of this embodiment. Here, the prototypes and language classification weights of each class in the feature space are visualized in a two-dimensional space.

[0032] The GAT is input with five randomly selected prototype classes (B1 to B5) of base classes, five prototype classes (N1 to N5) of new classes pseudo-generated by rotating the images of the base classes, and the language classification weights (T1 to T5) of the five base classes (B1 to B5). The new classes N1, N2, N3, N4, and N5 are rotated versions of the base classes B1, B2, B3, B4, and B5, respectively. The language classification weights T1, T2, T3, T4, and T5 are the language classification weights of the base classes B1, B2, B3, B4, and B5, respectively.

[0033] Here, as an example, the language classification weights use feature representations including the visual concepts of the base classes generated by a trained text encoder model that has been fully trained using big data that pairs images and language, as described in Non-Patent Document 2. Specifically, the language classification weights of the base classes are generated from sentences that explain the images of the base classes using a text encoder. Note that the language classification weights are referred to as text representations or text features in Non-Patent Document 2.

[0034] In addition to the conventional cross-entropy loss, the contrastive loss described in Non-Patent Document 3 is used for training the GAT parameters. Specifically, the GAT parameters are trained by pseudo-continuous training so that the prototype of the base class approaches the language classification weights containing the visual concept of the base class, and the prototype of the new class moves away from the language classification weights of the base class from which it was rotated.

[0035] 5(a) and 5(b) are diagrams comparing the output of the GAT in the pseudo continuous learning between the prior art and this embodiment. The output of the GAT is a prototype after adjustment by the pseudo continuous learning.

[0036] As shown in FIG. 5(a), the conventional technology has a small amount of prototype movement, which limits the separation accuracy.

[0037] As shown in Figure 5(b), in this embodiment, the adjusted prototype of the base class approaches the language classification weight of the base class, and the adjusted prototype of the new class moves away from the language classification weight of the base class from which it was rotated. In this way, the GAT can be effectively trained by optimization using the language classification weight, which increases the amount of movement of the prototype and improves the classification accuracy of each class.

[0038] 6 is a functional block diagram illustrating the configuration of a pseudo continuous learning module 40 of a conventional machine learning device 100 that uses CEC. The pseudo continuous learning module 40 includes a new class image generation unit 61, an image feature output unit 62, an image classification weight output unit 64, an optimization unit 66, and a classification unit 67.

[0039] The new class image generation unit 61 generates a pseudo dataset 15 of new class images by rotating the images of the basic class of the basic dataset 10 used in the pre-training module 30 and supplies it to the image feature output unit 62.

[0040] The image feature output unit 62 receives the pseudo data set 15 of the new class image as input, extracts a feature vector of the pseudo data set 15 of the new class image, and supplies the extracted image feature to the image classification weight output unit 64. The image feature output unit 62 corresponds to the feature extractor R of the backbone CNN 32 that pre-trained the basic class classification weights in stage 1.

[0041] The image classification weight output unit 64 calculates the image classification weight of the pseudo data set 15 of the new class image by averaging the feature vectors of the pseudo data set 15 of the new class image for each class, and supplies it to the optimization unit 66.

[0042] The optimizer 66 corresponds to the GAT 44 and receives as input the base class classification weights W0 of the backbone CNN 32, which has pre-trained the base class classification weights in Stage 1, and the base class classification weights of the pseudo dataset 15 supplied from the image classification weight output unit 64. The optimizer 66 performs meta-learning of the dependencies between the base dataset 10 and the pseudo dataset 15, and adapts all input class classification weights to output reconstructed classification weights. The pseudo continuous learning module 40 performs meta-learning of the GAT, which is a meta-module, in an episodic format. The parameters of the optimizer 66 are optimized and updated for each episode using a query set consisting of the base dataset 10 and the pseudo dataset 15. The optimization method used by the optimizer 66 is the method of minimizing cross-entropy loss described in Non-Patent Document 1. The optimizer 66 supplies the reconstructed classification weights obtained in this manner to the classifier 67.

[0043] The classification unit 67 uses the reconstructed classification weights as classification weights, and outputs a classification from the image feature values ​​output by the image feature value output unit 62 and the classification weights.

[0044] 7 is a functional block diagram illustrating the configuration of the pseudo continuous learning module 40 of the machine learning device 200 of this embodiment that uses CEC. The pseudo continuous learning module 40 includes a new class image generation unit 61, an image feature output unit 62, an image classification weight output unit 64, a language classification weight output unit 65, an optimization unit 66, and a classification unit 67. Here, configurations and operations that are common to the functional blocks of the pseudo continuous learning module 40 of the conventional machine learning device 100 in FIG. 6 will be omitted as appropriate, and only different configurations and operations will be described.

[0045] The language classification weight output unit 65 is pre-trained using basic class images and sentences (called "captions") that explain the basic class images, and uses the captions of the basic class images as input to generate language classification weights, which are language features of the basic class images, and supplies them to the optimization unit 66.

[0046] The optimization unit 66 receives the image classification weights and the language classification weights as input, optimizes the image classification weights, calculates reconstructed classification weights, and supplies them to the classification unit 67. Specifically, the optimization unit 66 calculates the reconstructed classification weights by minimizing the constructive loss so that the image classification weights of the basic classes approach the language classification weights and the image classification weights of the new classes move away from the language classification weights.

[0047] In the pseudo continuous learning module 40 of the machine learning device 200 of this embodiment, by minimizing the constructive loss based on the language classification weight, the distance between classes is increased compared to conventional techniques, thereby improving classification accuracy.

[0048] 8 is a functional block diagram illustrating another configuration of the pseudo continuous learning module 40 of the machine learning device 200 of this embodiment that uses CEC. The pseudo continuous learning module 40 includes a new class image generation unit 61, an image feature output unit 62, a language feature output unit 63, an image classification weight output unit 64, a language classification weight output unit 65, an optimization unit 66, and a classification unit 67. This differs from the functional block of the pseudo continuous learning module 40 of the machine learning device 200 of FIG. 7 in that a language feature output unit 63 is further provided.

[0049] When there are multiple captions for one image of the basic class, the language feature output unit 63 extracts language features from each of the multiple captions and supplies them to the language classification weight output unit 65. The language classification weight output unit 65 calculates a language classification weight by averaging the language features and supplies it to the optimization unit 66. The rest of the configuration and operation are the same as those in Fig. 7.

[0050] The various processes of the machine learning device 200 described above can be realized not only as a device using hardware such as a CPU and memory, but also as firmware stored in a ROM (read-only memory) or flash memory, or software for a computer, etc. The firmware program or software program can be provided by being recorded on a computer-readable recording medium, or can be transmitted and received with a server via a wired or wireless network, or can be transmitted and received as data broadcasting via terrestrial or satellite digital broadcasting.

[0051] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of the components and treatment processes, and that such modifications are also within the scope of the present invention.

[0052] As a method for generating an image of a new class from an image of a base class, an image of the new class may be generated by processing the image of the base class in a way other than rotating the image of the base class. For example, an image of the new class may be generated by dividing the image of the base class into a plurality of regions and swapping the divided regions with each other.

[0053] The present invention can be used in machine learning techniques.

[0054] 10 Base dataset, 15 Pseudo dataset, 20 New dataset, 30 Pre-training module, 32 Backbone CNN, 40 Pseudo continuous learning module, 44, 53 GAT, 45, 54 Reconstructed classification weights, 50 New class learning module, 61 New class image generation unit, 62 Image feature output unit, 63 Language feature output unit, 64 Image classification weight output unit, 65 Language classification weight output unit, 66 Optimization unit, 67 Classification unit, 100, 200 Machine learning device.

Claims

1. A machine learning device that continuously learns a smaller number of new class images compared to base class images, comprising: a new class image generation unit that processes the base class images to generate new class images; an image feature output unit that has been pre-trained with the base class images, the image feature output unit taking the base class images or the new class images as input and outputting image features; a language classification weight output unit that has been pre-trained with the base class images and sentences describing the base class images, the language classification weight output unit taking the sentences describing the base class images as input and outputting language classification weights; an image classification weight output unit that takes the image features as input, calculates an average value of the image features for each class, and outputs it as an image classification weight for each class; an optimization unit that takes the image classification weights and the language classification weights as input, optimizes the image classification weights, and outputs a reconstructed classification weight; and a classification unit that uses the reconstructed classification weights as class classification weights, and outputs a class classification from the image features output by the image feature output unit and the class classification weights.

2. The machine learning device according to claim 1, characterized in that the optimization unit learns to bring the image classification weights of the base class closer to the language classification weights and to move the image classification weights of the new class away from the language classification weights.

3. A machine learning method for continuously learning a smaller number of new class images compared to base class images, comprising: a new class image generating step of processing the base class images to generate new class images; an image feature output step of outputting image features using an image feature output module pre-trained with the base class images and receiving an image of the base class or an image of the new class; a language classification weight output step of outputting language classification weights using a language classification weight output module pre-trained with the base class images and a sentence describing the base class image; an image classification weight output step of receiving the image features as an input, calculating an average value of the image features for each class, and outputting the average value as an image classification weight for each class; an optimization step of receiving the image classification weights and the language classification weights as input, optimizing the image classification weights, and outputting a reconstructed classification weight; and a classification step of using the reconstructed classification weights as class classification weights, and outputting a class classification from the image features output by the image feature output step and the class classification weights.

4. A machine learning program for continuously learning a smaller number of new class images compared to base class images, comprising: a new class image generating step of processing the base class images to generate new class images; an image feature output step of outputting image features using an image feature output module pre-trained with the base class images and taking the base class images or the new class images as input; a language classification weight output step of outputting language classification weights using a language classification weight output module pre-trained with the base class images and a sentence describing the base class images as input; an image classification weight output step of taking the image features as input, calculating an average value of the image features for each class, and outputting it as an image classification weight for each class; an optimization step of taking the image classification weights and the language classification weights as input, optimizing the image classification weights, and outputting a reconstructed classification weight; and a classification step of using the reconstructed classification weights as class classification weights, and outputting a class classification from the image features output by the image feature output step and the class classification weights.

Images