Machine learning device, machine learning method, and machine learning program

Abstract

To provide a machine learning technology capable of learning a new class that has no semantic information.SOLUTION: A feature extraction unit 220 extracts feature vectors from input data. A semantic prediction unit 250 is a pre-meta-trained module that generates semantic vectors from the feature vectors of the input data. A mapping unit 230 is a module that generates semantic vectors from the feature vectors of the input data and has learned base classes. When semantic information is not assigned to the input data of a new class when learning the new class, an optimization unit 240 uses the semantic vector generated by the semantic prediction unit as a ground truth semantic vector, and optimizes parameters of the mapping unit so that a distance between the semantic vector generated by the mapping unit and the ground truth semantic vector is minimized.SELECTED DRAWING: Figure 2C

Description

[Technical Field]

[0001] The present invention relates to machine learning techniques. [Background technology]

[0002] Humans can learn new knowledge through long-term experience and retain previous knowledge. However, the knowledge of a convolutional neural network (CNN) depends on the dataset used for training, and adapting to changes in the data distribution requires retraining the CNN parameters for the entire dataset. As a CNN learns new tasks, its estimation accuracy for previous tasks decreases. Thus, continuous training of a CNN inevitably leads to catastrophic forgetting, in which the learning results of previous tasks are forgotten while learning a new task.

[0003] Incremental learning or continual learning has been proposed as a method to avoid fatal forgetting. Incremental learning is a learning method in which, when a new task or new data arises, the model is improved and learned, rather than learning from scratch.

[0004] On the other hand, since new tasks often require only a small amount of sample data, few-shot learning has been proposed as a method for efficiently learning with a small amount of training data. In few-shot learning, new tasks are learned using a small amount of different parameters, without re-learning previously learned parameters.

[0005] A method called incremental few-shot learning (IFSL) has been proposed, which combines continuous learning, which learns new classes without fatal forgetting from the learning results of base classes, with few-shot learning, which learns new classes that are fewer in number than the base classes (Non-Patent Document 1). In incremental few-shot learning, base classes can be learned from a large dataset, and new classes can be learned from a small amount of sample data. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Cheraghian, A., Rahman, S., Fang, P., Roy, SK, Petersson, L., & Harandi, M. (2021). Semantic-aware Knowledge Distillation for Few-Shot Class-Incremental Learning. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 2534-2543). Summary of the Invention [Problem to be solved by the invention]

[0007] An example of a continuous few-shot learning method is SaKD (Semantic-aware Knowledge Distillation), described in Non-Patent Document 1. In continuous few-shot learning, SaKD uses semantic information for each class as ground truth (correct answer data) for image classification tasks. Generally, when pre-training a base class, an image dataset with semantic information can be used, but images used in training a new class may not have semantic information. In SaKD, semantic information corresponding to images of the new class is required as correct answer data to train a new class, and there is a problem in that training cannot be performed on images without semantic information.

[0008] The present invention has been made in view of the above circumstances, and its purpose is to provide a machine learning technique that can learn new classes that do not have semantic information. [Means for solving the problem]

[0009] In order to solve the above problem, a machine learning device of one embodiment of the present invention includes: a feature extraction unit that extracts feature vectors from input data; a semantic vector generation unit that generates semantic vectors from semantic information assigned to the input data; a semantic prediction unit that has been meta-learned in advance and generates semantic vectors from the feature vectors of the input data; a mapping unit that has learned a base class and generates semantic vectors from the feature vectors of the input data; and an optimization unit that, when semantic information is not assigned to the input data of a new class when learning the new class, sets the semantic vector generated by the semantic prediction unit as a correct semantic vector and optimizes parameters of the mapping unit so that the distance between the semantic vector generated by the mapping unit and the correct semantic vector is minimized.

[0010] Another aspect of the present invention is a machine learning method, including: a feature extraction step of extracting a feature vector from input data; a semantic vector generation step of generating a semantic vector from semantic information assigned to the input data; a semantic prediction step of generating a semantic vector from the feature vector of the input data using a semantic prediction module that has been meta-learned in advance; a mapping step of generating a semantic vector from the feature vector of the input data using a mapping module that has trained a base class; and an optimization step of, when semantic information is not assigned to input data of a new class during training of the new class, determining the semantic vector generated by the semantic prediction step as a correct semantic vector and optimizing parameters of the mapping module so as to minimize a distance between the semantic vector generated by the mapping step and the correct semantic vector.

[0011] Any combination of the above components, and any transformation of the present invention into a method, device, system, recording medium, computer program, etc., are also valid aspects of the present invention. [Effects of the Invention]

[0012] According to the present invention, it is possible to provide a machine learning technique that can learn new classes that do not have semantic information. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a configuration diagram of a conventional machine learning device. [Figure 2A] FIG. 2 is a diagram illustrating the configuration and operation of the machine learning device according to the present embodiment when learning a base class. [Figure 2B] FIG. 10 is a diagram illustrating the configuration and operation of the machine learning device according to the present embodiment when learning a pseudo few-shot class. [Figure 2C]FIG. 2 is a diagram illustrating the configuration and operation of the machine learning device according to the present embodiment when learning a new class. [Figure 3] 10 is a flowchart illustrating a continuous few-shot learning procedure performed by the machine learning device according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] 1 is a configuration diagram of a conventional machine learning device 100. The machine learning device 100 includes a semantic vector generation unit 110, a feature extraction unit 120, a mapping unit 130, and an optimization unit 140.

[0015] SaKD assumes that semantic information for input images is provided as correct answer data both when training the base class and when training new classes. For example, in the case of a cat image, semantic information is text information such as black and male that is attached to the cat image.

[0016] When learning the base class, images of the base class and their semantic information are input to the machine learning device 100.

[0017] The semantic vector generation unit 110 converts the semantic information l of the base class image into a semantic vector s, and provides the semantic vector s to the optimization unit 140 as correct answer data.

[0018] The feature extraction unit 120 extracts a feature vector g from the base class image x and provides it to the mapping unit 130 .

[0019] The mapping unit 130 infers a semantic vector y from the feature vector g of the base class image x and provides the semantic vector y to the optimization unit 140 .

[0020] The optimization unit 140 calculates the distance in the semantic space between the inferred semantic vector y of the base class and the correct semantic vector s as a loss, and optimizes the parameters of the feature extraction unit 120 and the mapping unit 130 so as to minimize the loss.

[0021] Similarly, when learning a new class, images of the new class and their semantic information are input to the machine learning device 100.

[0022] The semantic vector generation unit 110 converts the semantic information l of the image of the new class into a semantic vector s, and provides the semantic vector s to the optimization unit 140 as correct answer data.

[0023] The feature extraction unit 120 extracts a feature vector g from the image x of the new class and provides it to the mapping unit 130 .

[0024] The mapping unit 130 infers a semantic vector y from the feature vector g of the image x of the new class and provides the semantic vector y to the optimization unit 140 .

[0025] The optimization unit 140 calculates the distance in the semantic space between the inferred semantic vector y of the new class and the correct semantic vector s as a loss, and optimizes the parameters of the feature extraction unit 120 and the mapping unit 130 so as to minimize the loss.

[0026] 2A to 2C are configuration diagrams of a machine learning device 200 according to an embodiment of the present invention. The machine learning device 200 includes a semantic vector generation unit 210, a feature extraction unit 220, a mapping unit 230, an optimization unit 240, and a semantic prediction unit 250.

[0027] Here, an image is taken as an example of data input to the machine learning device 200, but the input data is not limited to an image and may be any data.

[0028] FIG. 2A is a diagram illustrating the configuration and operation of a machine learning device 200 when learning a base class.

[0029] During base class training, images of the base class and their semantic information are input to the machine learning device 200. The operation during base class training is similar to base class training of the conventional machine learning device 100.

[0030] The semantic vector generation unit 210 converts the semantic information l of the base class image into a semantic vector s, and provides the semantic vector s to the optimization unit 240 as correct answer data.

[0031] The feature extraction unit 220 extracts a feature vector g from the base class image x and provides it to the mapping unit 230 .

[0032] The mapping unit 230 infers a semantic vector y from the base class feature vector g and provides the semantic vector y to the optimization unit 240 .

[0033] The optimization unit 240 calculates the distance in the semantic space between the estimated semantic vector y of the base class and the correct semantic vector s as a loss, and optimizes the parameters of the feature extraction unit 220 and the mapping unit 230 so as to minimize the loss.

[0034] 2B is a diagram illustrating the configuration and operation of the machine learning device 200 during learning of the pseudo few-shot class. In FIG. 2B, the base class in FIG. 2A is used with the parameters of the trained feature extraction unit 220 fixed.

[0035] Images of the pseudo few-shot class are generated from the base class. For example, five images of the base class are randomly selected and sequentially input to the machine learning device 200 in an episodic format as images of the pseudo few-shot class, and the pseudo few-shot class is meta-learned.

[0036] During meta-learning of the pseudo few-shot class, images of the pseudo few-shot class and their semantic information are input to the machine learning device 200.

[0037] The semantic vector generation unit 210 converts the semantic information l of the image of the pseudo few shot class into a semantic vector s, and provides the semantic vector s to the optimization unit 240 as correct answer data.

[0038] The feature extraction unit 220 extracts a feature vector g from the image x of the pseudo few-shot class and provides it to the semantic prediction unit 250 .

[0039] The semantic prediction unit 250 is a module similar to the mapping unit 230, and the initial parameters of the semantic prediction unit 250 use the parameters of the mapping unit 230 that has already learned the base class.

[0040] The semantic prediction unit 250 infers a semantic vector y from the feature vector g of the pseudo few-shot class and provides the semantic vector y to the optimization unit 240 .

[0041] The optimization unit 240 calculates the distance in semantic space between the estimated semantic vector y of the pseudo few-shot class and the ground truth semantic vector s as a loss, and optimizes the parameters of the semantic prediction unit 250 to minimize the loss. The feature extraction unit 220 fixes the parameters so as not to forget the knowledge of the base class, so the parameters are not optimized here. This allows the semantic prediction unit 250 to meta-learn in advance using the pseudo few-shot class.

[0042] The loss function during meta-learning uses the cosine distance between the semantic estimation vector y output from the semantic prediction unit 250 and the semantic correct answer vector s output from the semantic vector generation unit 210, and learning is carried out to minimize this cosine distance, i.e., to bring the semantic estimation vector y closer to the semantic correct answer vector s.

[0043] 2C is a diagram illustrating the configuration and operation of machine learning device 200 when learning a new class. In FIG. 2C, the base class in FIG. 2A is used with the parameters of feature extraction unit 220 that has already learned fixed.

[0044] Images of a new class may not have semantic information attached to them. Here, we explain the learning method for images of a new class that do not have semantic information attached to them.

[0045] When learning a new class, images of the new class are input to the machine learning device 200, and the pre-meta-trained semantic prediction unit 250 of FIG. 2B is used to predict semantic information from the images of the new class.

[0046] The feature extraction unit 220 extracts a feature vector g from the image x of the new class and provides it to the mapping unit 230 and the semantic prediction unit 250 .

[0047] The semantic prediction unit 250 predicts a semantic vector s from a feature vector g extracted from an image x of a new class, and provides the semantic vector s to the optimization unit 240 as correct answer data.

[0048] The mapping unit 230 infers a semantic vector y from the feature vector g of the new class and provides the semantic vector y to the optimization unit 240 .

[0049] The optimization unit 240 calculates the distance in semantic space between the estimated semantic vector y of the new class and the correct semantic vector s predicted by the semantic prediction unit 250 as a loss, and optimizes the parameters of the mapping unit 230 to minimize the loss. Since the feature extraction unit 220 fixes the parameters so as not to forget the knowledge of the base class, the parameters are not optimized here. This allows the mapping unit 230 to be fine-tuned using the new class.

[0050] If semantic information is attached to the images of the new class, the semantic vector generation unit 210 can generate a correct semantic vector from the semantic information of the images of the new class using the configuration described in Fig. 2A, and then perform learning in the same way as for the base class. This configuration enables learning and inference for the new class regardless of whether or not semantic information corresponding to the new class is attached.

[0051] FIG. 3 is a flowchart illustrating a procedure for continuous few-shot learning by the machine learning device 200 according to this embodiment.

[0052] An image of a new class is input to the machine learning device 200 (S10). The feature extraction unit 220 extracts a feature vector from the image of the new class (S20).

[0053] The mapping unit 230 generates an estimated semantic vector from the feature vector of the image of the new class (S30).

[0054] If semantic information is assigned to the image of the new class (Y in S40), the semantic vector generation unit 210 generates a correct semantic vector from the semantic information of the image of the new class (S50).

[0055] If semantic information is not assigned to the image of the new class (N in S40), the semantic prediction unit 250 predicts a correct semantic vector from the feature vector of the image of the new class (S60).

[0056] The optimization unit 240 optimizes the parameters of the mapping unit 230 so that the distance between the estimated semantic vector and the correct semantic vector is minimized (S70).

[0057] The various processes of the machine learning device 200 described above can naturally be realized as a device using hardware such as a CPU and memory, but can also be realized by firmware stored in a ROM (read-only memory) or flash memory, or software for a computer, etc. The firmware program or software program can be provided by being recorded on a computer-readable recording medium, or can be transmitted and received with a server via a wired or wireless network, or can be transmitted and received as data broadcasting via terrestrial or satellite digital broadcasting.

[0058] As described above, according to the machine learning device 200 of this embodiment, pseudo few-shot classes are generated from base classes, and a semantic prediction unit that predicts semantic information from input images of the pseudo few-shot classes is meta-learned in advance. When learning a small number of new classes, the new classes are continuously learned using the semantic prediction information generated by the meta-learned semantic prediction unit as ground truth data. This enables learning and inference of new classes that do not have semantic information.

[0059] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of the respective components and treatment processes, and that such modifications are also within the scope of the present invention. [Explanation of symbols]

[0060] 100 machine learning device, 110 semantic vector generation unit, 120 feature extraction unit, 130 mapping unit, 140 optimization unit, 200 machine learning device, 210 semantic vector generation unit, 220 feature extraction unit, 230 mapping unit, 240 optimization unit, 250 semantic prediction unit.

Claims

1. a feature extraction unit that extracts feature vectors from input data; a semantic vector generation unit that generates a semantic vector from semantic information assigned to the input data; a pre-meta-trained semantic predictor that generates a semantic vector from the feature vector of the input data; a base-class trained mapping unit that generates a semantic vector from the feature vector of the input data; an optimization unit that, when semantic information is not assigned to input data of a new class during learning of the new class, sets the semantic vector generated by the semantic prediction unit as a correct semantic vector, and optimizes parameters of the mapping unit so that a distance between the semantic vector generated by the mapping unit and the correct semantic vector is minimized.

2. 2. The machine learning device according to claim 1, wherein, when semantic information is added to the input data of the new class, the optimization unit sets the semantic vector generated by the semantic vector generation unit as a correct semantic vector, and optimizes parameters of the mapping unit so that a distance between the semantic vector generated by the mapping unit and the correct semantic vector is minimized.

3. the semantic vector generation unit generates a semantic vector from semantic information assigned to input data of a pseudo few-shot class selected from the base class; the semantic prediction unit generates a semantic vector from a feature vector of the input data of the pseudo few shot class; 3. The machine learning device according to claim 1, wherein the optimization unit optimizes parameters of the semantic prediction unit so as to minimize a distance between the semantic vector generated by the semantic vector generation unit and the correct semantic vector, using the semantic vector generated by the semantic prediction unit as a correct semantic vector.

4. a feature extraction step of extracting a feature vector from the input data; a semantic vector generation step of generating a semantic vector from semantic information assigned to the input data; a semantic prediction step of generating a semantic vector from the feature vector of the input data using a pre-meta-trained semantic prediction module; a mapping step of generating a semantic vector from the feature vector of the input data using a mapping module that has learned base classes; and optimizing parameters of the mapping module so that the distance between the semantic vector generated by the mapping step and the correct semantic vector is minimized when semantic information is not assigned to input data of the new class during learning of the new class.

5. a feature extraction step of extracting a feature vector from the input data; a semantic vector generation step of generating a semantic vector from semantic information assigned to the input data; a semantic prediction step of generating a semantic vector from the feature vector of the input data using a pre-meta-trained semantic prediction module; a mapping step of generating a semantic vector from the feature vector of the input data using a mapping module that has learned base classes; and an optimization step of optimizing parameters of the mapping module so as to minimize the distance between the semantic vector generated by the mapping step and the correct semantic vector, when semantic information is not assigned to input data of the new class during learning of the new class.

Images