MACHINE LEARNING DEVICE, MACHINE LEARNING PROCEDURES AND MACHINE LEARNING PROGRAM

Description

TECHNICAL FIELD

[0001] The present invention relates to machine learning technologies.BACKGROUND ART

[0002] Human beings can learn new knowledge through experiences over a prolonged period of time and can maintain old knowledge without forgetting it. Meanwhile, the knowledge of a convolutional neutral network (CNN) depends on the dataset used in learning. To adapt to a change in data distribution, it is necessary to re-train CNN parameters in response to the entirety of the dataset. In CNN, the precision estimation for old tasks will be decreased as new tasks are learned. Thus, catastrophic forgetting cannot be avoided in CNN. Namely, the result of learning old tasks is forgotten as new tasks are being learned in successive learning.

[0003] Incremental learning or continual learning is proposed as a scheme to avoid catastrophic forgetting. Continual learning is a learning method that improves a current trained model to learn new tasks and new data as they occur, instead of training the model from scratch.

[0004] On the other hand, since new tasks often have only a few pieces of sample data available, few-shot learning has been proposed as a method for efficient learning with a small amount of training data. In few-shot learning, new tasks are learned using another small amount of parameters without relearning parameters that have been learned once.

[0005] A method called incremental few-shot learning (IFSL) has been proposed, which combines continual learning, where a novel class is learned without catastrophic forgetting of the result of learning the base class, and few-shot learning, where a novel class with fewer examples as compared to the base class is learned (Non-Patent Literature 1). In incremental few-shot learning, base classes can be learned from a large dataset and novel classes can be learned from a small number of sample data pieces.

[0006] [Non-Patent Literature 1] Cheraghian, A., Rahman, S., Fang, P., Roy, S. K., Petersson, L., & Harandi, M. (2021). Semantic-aware Knowledge Distillation for Few-Shot Class-Incremental Learning. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 2534-2543)SUMMARY OF INVENTION

[0007] As an incremental few-shot learning method, there is Semantic-aware Knowledge Distillation (SaKD) described in Non-Patent Literature 1. In incremental few-shot learning, SaKD uses semantic (meaning) information of each class as ground truth (correct answer data) for image classification tasks. In general, an image dataset to which semantic information has been added can be used at the time of pre-learning of basic classes. However, semantic information may not be added to images used at the time of learning of novel classes. In order to learn a novel class, SaKD needs semantic information corresponding to an image of the novel class as the correct answer data, and there is a problem that images without semantic information cannot be learned.

[0008] The present invention addresses the issue described above, and a general purpose thereof is to provide a machine learning technology that allows for learning of novel classes without semantic information.

[0009] In order to solve the aforementioned problems, a machine learning device, method and program for performing incremental learning of an image classification task, as set forth in the appended claims are provided. [Fig. 1] Fig. 1 is a configuration diagram of a conventional machine learning device. [Fig. 2A] Fig. 2A is a diagram explaining the configuration and operation of a machine learning device according to the present embodiment at the time of learning a base class. [Fig. 2B] Fig. 2B is a diagram explaining the configuration and operation of a machine learning device according to the present embodiment at the time of learning a pseudo few-shot class. [Fig. 2C] Fig. 2C is a diagram explaining the configuration and operation of a machine learning device according to the present embodiment at the time of learning a novel class. [Fig. 3] Fig. 3 is a flowchart explaining an incremental few-shot learning procedure performed by the machine learning device according to the present embodiment. DESCRIPTION OF EMBODIMENTS

[0010] Fig. 1 is a configuration diagram of a conventional machine learning device 100. The machine learning device 100 includes a semantic vector generation unit 110, a feature extraction unit 120, a mapping unit 130, and an optimization unit 140.

[0011] In SaKD, it is assumed that semantic information for an input image is given as correct answer data when learning the base class and when learning a new class. The semantic information is, for example, in the case of an image of a cat, text information such as black or male added to the image of the cat.

[0012] At the time of the learning of a base class, the image of the base class and semantic information thereof are input to the machine learning device 100.

[0013] The semantic vector generation unit 110 converts semantic information 1 of the image of the base class into a semantic vector s, and provides the semantic vector s to the optimization unit 140 as correct answer data.

[0014] The feature extraction unit 120 extracts a feature vector g from an image x of the base class and provides the feature vector g to the mapping unit 130.

[0015] The mapping unit 130 infers a semantic vector y from the feature vector g of the image x of the base class and provides the semantic vector y to the optimization unit 140.

[0016] The optimization unit 140 calculates the distance in a semantic space between the inferred semantic vector y of the base class and the correct answer semantic vector s as a loss, and optimizes the parameters of the feature extraction unit 120 and the parameters of the mapping unit 130 such that the loss is minimized.

[0017] In the same manner, at the time of the learning of a novel class, an image of the novel class and semantic information thereof are input to the machine learning device 100.

[0018] The semantic vector generation unit 110 converts semantic information 1 of the image of the novel class into a semantic vector s, and provides the semantic vector s to the optimization unit 140 as correct answer data.

[0019] The feature extraction unit 120 extracts a feature vector g from an image x of the novel class and provides the feature vector g to the mapping unit 130.

[0020] The mapping unit 130 infers a semantic vector y from the feature vector g of the image x of the novel class and provides the semantic vector y to the optimization unit 140.

[0021] The optimization unit 140 calculates the distance in a semantic space between the inferred semantic vector y of the novel class and the correct answer semantic vector s as a loss, and optimizes the parameters of the feature extraction unit 120 and the parameters of the mapping unit 130 such that the loss is minimized.

[0022] Fig. 2A to Fig. 2C are configuration diagrams of a machine learning device 200 according to an embodiment of the present invention. The machine learning device 200 includes a semantic vector generation unit 210, a feature extraction unit 220, a mapping unit 230, and a semantic prediction unit 250.

[0023] Images are used as data input to the machine learning device 200 in the figures. However, in embodiments not falling within the scope of the claims, the input data may be arbitrary data not limited to images.

[0024] Fig. 2A is a diagram explaining the configuration and operation of a machine learning device 200 at the time of learning a base class.

[0025] At the time of the learning of a base class, the image of the base class and semantic information thereof are input to the machine learning device 200. The operation at the time of the learning of the base class is the same as that at the time of the learning of a base class in the conventional machine learning device 100.

[0026] The semantic vector generation unit 210 converts semantic information 1 of the image of the base class into a semantic vector s, and provides the semantic vector s to the optimization unit 240 as correct answer data.

[0027] The feature extraction unit 220 extracts a feature vector g from an image x of the base class and provides the feature vector g to the mapping unit 230.

[0028] The mapping unit 230 infers a semantic vector y from the feature vector g of the base class and provides the semantic vector y to the optimization unit 240.

[0029] The optimization unit 240 calculates the distance in a semantic space between the estimated semantic vector y of the base class and the correct answer semantic vector s as a loss, and optimizes the parameters of the feature extraction unit 220 and the parameters of the mapping unit 230 such that the loss is minimized.

[0030] Fig. 2B is a diagram explaining the configuration and operation of the machine learning device 200 at the time of learning a pseudo few-shot class. In Fig. 2B, the parameters of the feature extraction unit 220 that has learned the base class shown in Fig. 2A are fixed for use.

[0031] An image of a pseudo few-shot class is generated from the base class. For example, five images of the base class are randomly selected, and the pseudo few-shot class is meta-learned by sequentially inputting the images into the machine learning device 200 in an episodic format as images of the pseudo few-shot class.

[0032] At the time of the meta-learning of the pseudo few-shot class, the images of the pseudo few-shot class and semantic information thereof are input to the machine learning device 200.

[0033] The semantic vector generation unit 210 converts semantic information 1 of the images of the pseudo few-shot class into a semantic vector s, and provides the semantic vector s to the optimization unit 240 as correct answer data.

[0034] The feature extraction unit 220 extracts a feature vector g from an image x of the pseudo few-shot class and provides the feature vector g to the semantic prediction unit 250.

[0035] The semantic prediction unit 250 is a module similar to the mapping unit 230, and the parameters of the mapping unit 230 that has learned the base class are used for the initial parameters of the semantic prediction unit 250.

[0036] The semantic prediction unit 250 infers a semantic vector y from the feature vector g of the pseudo few-shot class and provides the semantic vector y to the optimization unit 240.

[0037] The optimization unit 240 calculates the distance in a semantic space between the estimated semantic vector y of the pseudo few-shot class and the correct answer semantic vector s as a loss, and optimizes the parameters of the semantic prediction unit 250 such that the loss is minimized. Since the parameters are fixed in the feature extraction unit 220 so as not to forget the knowledge of the base class, the parameters are not optimized here. Thereby, the semantic prediction unit 250 is trained in advance in a meta-learning process using the pseudo few-shot class.

[0038] For the loss function during meta-learning, the cosine distance of the semantic estimated vector y output from the semantic prediction unit 250 and the semantic correct answer vector s output from the semantic vector generation unit 210 are used, and the learning is proceeded such that this cosine distance is minimized, that is, the semantic estimated vector y approaches the semantic correct answer vector s.

[0039] Fig. 2C is a diagram explaining the configuration and operation of the machine learning device 200 at the time of learning a novel class. In Fig. 2C, the parameters of the feature extraction unit 220 that has learned the base class shown in Fig. 2A are fixed for use.

[0040] An image of the novel class may not have semantic information added to the image. A learning method used for an image of the novel class for which semantic information is not added will be explained.

[0041] At the time of the learning of a novel class, an image of the novel class is input to the machine learning device 200, and the semantic prediction unit 250 in Fig. 2B trained in advance in a meta-learning process is used for predicting semantic information from the image of the novel class.

[0042] The feature extraction unit 220 extracts a feature vector g from an image x of the novel class and provides the feature vector g to the mapping unit 230 and the semantic prediction unit 250.

[0043] The semantic prediction unit 250 predicts the semantic vector s from the feature vector g extracted from the image x of the novel class, and provides the semantic vector s to the optimization unit 240 as correct answer data.

[0044] The mapping unit 230 infers a semantic vector y from the feature vector g of the novel class and provides the semantic vector y to the optimization unit 240.

[0045] The optimization unit 240 calculates the distance in a semantic space between the estimated semantic vector y of the novel class and the correct answer semantic vector s predicted by the semantic prediction unit 250 as a loss, and optimizes the parameters of the mapping unit 230 such that the loss is minimized. Since the parameters are fixed in the feature extraction unit 220 so as not to forget the knowledge of the base class, the parameters are not optimized here. As a result, the mapping unit 230 is fine-tuned using the novel class.

[0046] When semantic information is added to the image of the novel class, it is only necessary for the semantic vector generation unit 210 to generate the correct answer semantic vector from the semantic information of the image of the novel class using the configuration explained in Fig. 2A and perform the same learning as that for the base class. This configuration makes it possible to learn and infer a novel class regardless of the presence or absence of semantic information corresponding to the novel class.

[0047] Fig. 3 is a flowchart explaining an incremental few-shot learning procedure performed by the machine learning device 200 according to the present embodiment.

[0048] An image of a novel class is input to the machine learning device 200 (S10). The feature extraction unit 220 extracts a feature vector from the image of the novel class (S20).

[0049] The mapping unit 230 generates an estimated semantic vector from the feature vector of the image of the novel class (S30).

[0050] When semantic information is added to the image of the novel class (Y at S40), the semantic vector generation unit 210 generates a correct answer semantic vector from the semantic information of the image of the novel class (S50).

[0051] When semantic information is not added to the image of the novel class (N at S40), the semantic prediction unit 250 predicts a correct answer semantic vector from the feature vector of the image of the novel class (S60).

[0052] The optimization unit 240 optimizes the parameters of the mapping unit 230 such that the distance between the estimated semantic vector and the correct answer semantic vector is minimized (S70).

[0053] The various processes of a machine learning device 200 explained above can be realized as a device using hardware such as a CPU and memory. Alternatively, the processes can be implemented by firmware stored in a read-only memory (ROM), a flash memory, etc., or by software on a computer, etc. The firmware program or the software program may be made available on, for example, a computer readable recording medium. Alternatively, the programs may be transmitted to and / or received from a server via a wired or wireless network. Still alternatively, the programs may be transmitted and / or received in the form of data transmission over terrestrial or satellite digital broadcast systems.

[0054] As described above, the machine learning device 200 according to the present embodiment generates a pseudo few-shot class from a base class and trains, in advance in a meta-learning process, a semantic prediction unit that predicts semantic information from an input image of the pseudo few-shot class. When a small number of novel classes are learned, the semantic prediction information generated by the semantic prediction unit trained in a meta-learning process is used as correct answer data so as to continuously learn the novel classes. This makes it possible to learn and infer novel classes without semantic information.INDUSTRIAL APPLICABILITY

[0055] The present invention is applicable to machine learning technology.REFERENCE SIGNS LIST

[0056] 100 machine learning device, 110 semantic vector generation unit, 120 feature extraction unit, 130 mapping unit, 140 optimization unit, 200 machine learning device, 210 semantic vector generation unit, 220 feature extraction unit, 230 mapping unit, 240 optimization unit, 250 semantic prediction unit

Claims

1. A machine learning device (200) for performing incremental few-shot learning of an image classification task, the device comprising: a feature extraction unit (220) that is configured to extract a feature vector from an input image; a semantic vector generation unit (210) that is configured to generate a semantic vector from semantic information added to the input image; a semantic prediction unit (250) that is configured to generate a second semantic vector from the feature vector of the input image; a mapping unit (230) that has learned a base class and that is configured to generate a third semantic vector from the feature vector of the input image; and an optimization unit (240) that is configured to optimize parameters of the mapping unit (230) using the second semantic vector generated by the semantic prediction unit (250) as a correct answer semantic vector such that a distance between the third semantic vector generated by the mapping unit (230) and the correct answer semantic vector is minimized when learning the novel class using as input image an image of the novel class that does not have semantic information added to the image, wherein the semantic prediction unit (250) has been trained in advance in a meta-learning process such that a distance between a first semantic vector generated by the semantic vector generation unit (210) from semantic information added to an input image of a pseudo few-shot class and a second semantic vector generated by the semantic prediction unit (250) from a feature vector extracted by the feature unit (220) from the input image of the pseudo few-shot class is minimized, wherein the input image of the pseudo few-shot class is an image selected from images of the base class.

2. The machine learning device (200) according to Claim 1, wherein the optimization unit (240) is configured to optimize the parameters of the mapping unit (230) such that a distance between an estimated semantic vector generated by the mapping unit (230) and a first correct answer semantic vector generated by the semantic vector generation unit (210) is minimized when semantic information is added to the input image of the novel class.

3. A machine learning method for performing incremental few-shot learning of an image classification task, the method comprising: extracting a feature vector from an input image; generating a first semantic vector from semantic information added to the input image; generating a second semantic vector from the feature vector of the input image by using a semantic prediction module; generating a third semantic vector from the feature vector of the input image by using a mapping module that has learned a base class; and optimizing parameters of the mapping module using the second semantic vector generated by the semantic prediction module as a correct answer semantic vector such that a distance between the third semantic vector generated by the mapping module and the correct answer semantic vector is minimized when learning the novel class using as input image an image of the novel class that does not have semantic information added to the image, wherein the semantic prediction module has been trained in advance in a meta-learning process such that a distance between a first semantic vector generated from semantic information added to an input image of a pseudo few-shot class and a second semantic vector generated by the semantic prediction module from a feature vector extracted from the input image of the pseudo few-shot class is minimized, wherein the input image of the pseudo few-shot class is an image selected from images of the base class.

4. A machine learning program for performing incremental few-shot learning of an image classification task, the program comprising modules including: a feature extraction module that, when the program is executed by a computer, causes the computer to extract a feature vector from an input image; a semantic vector generation module that, when the program is executed by the computer, causes the computer to generate a first semantic vector from semantic information added to the input image; a semantic prediction module that, when the program is executed by the computer, causes the computer to generate a second semantic vector from the feature vector of the input image; a mapping module that has learned a base class and that, when the program is executed by the computer, causes the computer to generate a third semantic vector from the feature vector of the input image; and an optimization module that, when the program is executed by the computer, causes the computer to optimize parameters of the mapping module using the second semantic vector generated by the semantic prediction module as a correct answer semantic vector such that a distance between the third semantic vector generated by the mapping module and the correct answer semantic vector is minimized when learning the novel class using as input image an image of the novel class that does not have semantic information added to the image, wherein the semantic prediction module has been trained in advance in a meta-learning process such that a distance between a first semantic vector generated by the semantic vector generation unit (210) from semantic information added to an input image of a pseudo few-shot class and a second semantic vector generated by the semantic prediction unit (250) from a feature vector extracted by the feature unit (220) from the input image of the pseudo few-shot class is minimized, wherein the input image of the pseudo few-shot class is an image selected from images of the base class.