Machine learning device, machine learning method, and machine learning program

Abstract

To provide a machine learning device, method, and program for performing learning for a new class not having semantic information.SOLUTION: In a machine learning device 200, a feature extraction unit 220 extracts feature vectors from input data. A semantic prediction unit 250 is a pre-meta-trained module that generates semantic vectors from feature vectors of the input data. A mapping unit 230 is a module that has learned base classes and generates semantic vectors from the feature vectors of the input data. When semantic information is not assigned to the input data of a new class during learning of the new class, an optimization unit 240 uses the semantic vector generated by the semantic prediction unit as a correct semantic vector, and optimizes parameters of the mapping unit so that a distance between the semantic vector generated by the mapping unit and the correct semantic vector is minimized.SELECTED DRAWING: Figure 2C

Description

[Technical Field]

[0001] The present invention relates to machine learning techniques. [Background technology]

[0002] Humans can learn new knowledge and forget old knowledge through long-term experience. On the other hand, convolutional neural networks (Convolutional Neural Networks) The knowledge of a neural network (CNN) depends on the dataset used for training. To adapt to the change in distribution, the CNN parameters must be retrained for the entire dataset. As CNNs learn new tasks, they lose their ability to perform old tasks. In this way, continuous learning in CNN leads to a decrease in the accuracy of learning new tasks. Catastrophic forgetting, in which learning results from previous tasks are forgotten during learning, is avoided. I can't get it.

[0003] Incremental learning (or continual learning) is a method to avoid fatal forgetting. Continuous learning is the process of learning when new tasks or new data arise. Instead of training a model from scratch, we train it by improving the current trained model. It is a learning method.

[0004] On the other hand, new tasks often have only a small amount of sample data available, so Few-shot learning is a method for efficiently learning with small training data. In few-shot learning, the parameters learned once are not retrained, but are instead retrained. It learns new tasks using a small number of parameters.

[0005] Learning new classes without catastrophic forgetting of the results of learning base classes Continuous learning and few-shot learning, which learns new classes that are few in number compared to the base classes. This is called Incremental Few Shot Learning (IFSL), which combines In continuous few-shot learning, the basic class is It learns from a large dataset for the classes and only learns from a small number of sample data for new classes. You can learn from. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Cheraghian, A., Rahman, S., Fang, P., Roy, SK, Petersson, L., & Harandi, M. (2021). Semantic-aware Knowledge Distillation for Few-Shot Class-Incremental Learning. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 2534-2543). Summary of the Invention [Problem to be solved by the invention]

[0007] As a continuous few-shot learning method, SaKD (Semantic-aware Knocking Discrete Knocking) is described in Non-Patent Document 1. SaKD is a continuous few-shot learning method that uses a The semantic information of the image is used as ground truth (correct data) for image classification. Generally, when pre-training the base classes, semantic information is added to the Image datasets are available, but the images used for learning new classes have semantic In SaKD, in order to learn a new class, Semantic information corresponding to images in a new class is required as ground truth data. There was a problem in that it was not possible to learn from images that did not have contextual information.

[0008] The present invention has been made in view of the above circumstances, and its object is to To provide a machine learning technology that can learn new classes that do not have . [Means for solving the problem]

[0009] In order to solve the above problem, a machine learning device according to one aspect of the present invention calculates features from input data. A feature extraction unit extracts vectors and semantic information from the input data. a semantic vector generation unit for generating a semantic vector; A pre-meta-learned semantic vector is generated from the feature vector. a tick prediction unit that generates a semantic vector from the feature vector of the input data; The mapping unit that has learned the base class and the new class are used when learning the new class. If semantic information is not attached to the input data, the semantic prediction The semantic vector generated by the unit is used as a correct semantic vector, The semantic vector generated by the mapping unit and the correct semantic vector Optimization of the parameters of the mapping unit so that the distance between the vectors is minimized. Includes parts.

[0010] Another aspect of the present invention is a machine learning method for generating a feature vector from input data. a feature extraction step for extracting semantic information from the input data; A semantic vector generation step generates a semantic vector, and a meta-learning step generates a semantic vector. Using a trained semantic prediction module, a feature vector of the input data is extracted. The semantic prediction step generates semantic vectors from the dataset, and the base class learning step generates semantic vectors from the dataset. A trained mapping module is used to convert the feature vectors of the input data into semantics. A mapping step for generating a tick vector, and a step for generating a new class when learning a new class. If semantic information is not attached to the input data, the semantic prediction The semantic vector generated by step 1 is called the correct semantic vector. Then, the semantic vector generated by the mapping step and the correct answer The parameters of the mapping module are adjusted so that the distance between the semantic vectors is minimized. and optimizing the data.

[0011] Any combination of the above components, and the expression of the present invention may be used as a method, an apparatus, a system, a recording medium, Conversions between the body, computer program, etc. are also valid aspects of the present invention. be. [Effects of the Invention]

[0012] According to the present invention, it is possible to learn new classes that do not have semantic information. We can provide machine learning technology that can [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a configuration diagram of a conventional machine learning device. [Figure 2A] FIG. 2 is a diagram illustrating the configuration and operation of the machine learning device according to the present embodiment when learning a base class. [Figure 2B] FIG. 10 is a diagram illustrating the configuration and operation of the machine learning device according to the present embodiment when learning a pseudo few-shot class. [Figure 2C] FIG. 2 is a diagram illustrating the configuration and operation of the machine learning device according to the present embodiment when learning a new class. [Figure 3] 10 is a flowchart illustrating a continuous few-shot learning procedure performed by the machine learning device according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] FIG. 1 is a diagram showing the configuration of a conventional machine learning device 100. The machine learning device 100 uses semantics a metric vector generator 110, a feature extractor 120, a mapping unit 130, and an optimizer Includes 140.

[0015] In SaKD, the sensitivity of the input image is calculated both when learning the base class and when learning the new class. It is assumed that semantic information is given as the correct answer data. For example, in the case of a cat image, the cat's color information is assigned to the image, such as black, male, etc. This is text information.

[0016] During base class training, the base class images and their semantic information are machine-learned. It is input to the device 100.

[0017] The semantic vector generation unit 110 generates the semantic information l of the base class image. is converted into a semantic vector s, and the semantic vector s is used as the correct data. The result is given to the optimization unit 140.

[0018] The feature extraction unit 120 extracts a feature vector g from the base class image x and performs mapping section 130.

[0019] The mapping unit 130 converts the feature vector g of the base class image x into a semantic vector The vector y is inferred and the semantic vector y is provided to the optimizer 140.

[0020] The optimizer 140 compares the inferred semantic vector y of the base class with the correct semantic vector y. The distance between the semantic vectors s in the semantic space is calculated as the loss, and the loss is minimized. The parameters of the feature extraction unit 120 and the mapping unit 130 are optimized so that

[0021] Similarly, when learning a new class, images of the new class and their semantic information are machine-generated. It is input to the learning device 100.

[0022] The semantic vector generation unit 110 generates the semantic information l of the image of the new class as Convert it into a semantic vector s, and use the semantic vector s as the correct data. The result is provided to the formulating unit 140.

[0023] The feature extraction unit 120 extracts a feature vector g from the image x of the new class, and the mapping unit Give it to 130.

[0024] The mapping unit 130 converts the feature vector g of the image x of the new class into a semantic vector The semantic vector y is inferred and provided to the optimization unit 140.

[0025] The optimization unit 140 compares the inferred semantic vector y of the new class with the correct semantic vector y. The distance of the semantic vector s in the semantic space is calculated as the loss, and the loss is minimized. The parameters of the feature extraction unit 120 and the mapping unit 130 are optimized so that

[0026] 2A to 2C are diagrams illustrating the configuration of a machine learning device 200 according to an embodiment of the present invention. The machine learning device 200 includes a semantic vector generation unit 210, a feature extraction unit 220, a mapper The processing unit 200 includes a coding unit 230, an optimization unit 240, and a semantic prediction unit 250.

[0027] Here, an image is taken as an example of data input to the machine learning device 200. The input data is not limited to an image, and may be any data.

[0028] FIG. 2A is a diagram illustrating the configuration and operation of a machine learning device 200 when learning a base class. do.

[0029] During base class training, the base class images and their semantic information are machine-learned. The operation of the base class learning device 200 is the same as that of the conventional machine learning device 100. This is similar to learning base classes.

[0030] The semantic vector generation unit 210 generates the semantic information l of the base class image. is converted into a semantic vector s, and the semantic vector s is used as the correct data. The result is given to the optimization unit 240.

[0031] The feature extraction unit 220 extracts a feature vector g from the base class image x and performs mapping section 230.

[0032] The mapping unit 230 converts the base class feature vector g into the semantic vector y and provides the semantic vector y to the optimization unit 240.

[0033] The optimizer 240 compares the estimated semantic vector y of the base class with the correct semantic vector y. The distance between the semantic vectors s in the semantic space is calculated as the loss, and the loss is minimized. The parameters of the feature extraction unit 220 and the mapping unit 230 are optimized so that

[0034] FIG. 2B illustrates the configuration and operation of the machine learning device 200 during learning of the pseudo few-shot class. In FIG. 2B, the base class in FIG. 2A is used as the parameter of the feature extraction unit 220 that has already learned the feature. The meter is fixed in place.

[0035] Generate pseudo-few-shot class images from the base class. For example, randomly select 5 Select images from the base class one by one and classify them into episode-shaped images as pseudo-few-shot class images. The formula is input sequentially to the machine learning device 200 to perform meta-learning of the pseudo few-shot class.

[0036] During meta-learning of pseudo-few-shot classes, images of pseudo-few-shot classes and their sequences are Contact information is input to the machine learning device 200.

[0037] The semantic vector generation unit 210 generates the semantic vectors of the images of the pseudo few shot classes. The check information l is converted into a semantic vector s, and the semantic vector s is used as the correct answer data. The data is provided to the optimization unit 240.

[0038] The feature extraction unit 220 extracts a feature vector g from the image x of the pseudo few shot class, The data is provided to the semantic prediction unit 250.

[0039] The semantic prediction unit 250 is a module similar to the mapping unit 230. The initial parameters of the dynamic prediction unit 250 are the same as those of the mapping unit 2 Uses 30 parameters.

[0040] The semantic prediction unit 250 predicts the semantic data from the feature vector g of the pseudo few-shot class. The semantic vector y is inferred and the semantic vector y is provided to the optimizer 240.

[0041] The optimization unit 240 optimizes the estimated semantic vector y of the pseudo few-shot class and The distance of the correct semantic vector s in the semantic space is calculated as the loss. Optimize the parameters of the semantic predictor 250 to minimize the loss. Since the unit 220 fixes the parameters so as not to forget the knowledge of the base class, In this case, the parameters are not optimized. This allows the semantic predictor 250 to predict pseudo-minority words. It is meta-trained in advance using shot classes.

[0042] The loss function during meta-learning is the semantic prediction output from the semantic prediction unit 250. The estimated vector y and the semantic vector output from the semantic vector generation unit 210 The cosine distance of the correct vector s is used, and the cosine distance is minimized. The semantic estimation vector y is trained to approach the semantic correct vector s. Go ahead.

[0043] FIG. 2C is a diagram illustrating the configuration and operation of the machine learning device 200 when learning a new class. In FIG. 2C, the parameters of the feature extractor 220 that has already learned the base class in FIG. 2A are fixed. It is used as such.

[0044] Images in a new class may not have semantic information. We will explain the learning method for new classes of images that do not have semantic information attached.

[0045] When learning a new class, an image of the new class is input to the machine learning device 200, and the image shown in FIG. 2B The pre-meta-trained semantic predictor 250 predicts semantic information from images of new classes. It is used to predict block information.

[0046] The feature extraction unit 220 extracts a feature vector g from the image x of the new class, and the mapping unit 230 and the semantic prediction unit 250.

[0047] The semantic prediction unit 250 calculates a new class of image x from a feature vector g extracted from the image x. The semantic vector s is predicted from the data, and the semantic vector s is used as the correct data. The result is given to the optimization unit 240.

[0048] The mapping unit 230 converts the feature vector g of the new class into a semantic vector y. The semantic vector y is inferred and provided to the optimizer 240 .

[0049] The optimizer 240 optimizes the estimated semantic vector y of the new class and the semantic vector The semantic vector s predicted by the semantic prediction unit 250 is compared with the correct semantic vector s. The spatial distance is calculated as a loss, and the parameters of the mapping unit 230 are set so that the loss is minimized. The feature extraction unit 220 optimizes the parameters so as not to forget the knowledge of the base class. Since the data is fixed, we do not optimize the parameters here. The unit 230 is fine-tuned using the new classes.

[0050] In addition, if semantic information is attached to the images of the new class, as explained in Figure 2A, Using the above configuration, the semantic vector generation unit 210 generates the semantic vector of the image of the new class. The correct semantic vector is generated from the semantic information and trained in the same way as the base class. This configuration allows the new class to be used regardless of whether or not there is semantic information corresponding to the new class. ,It becomes possible to learn and infer new classes.

[0051] FIG. 3 illustrates a procedure for continuous few-shot learning by the machine learning device 200 of this embodiment. 1 is a flowchart showing the process of

[0052] An image of a new class is input to the machine learning device 200 (S10). The feature extraction unit 220 ,A feature vector is extracted from the image of the new class (S20).

[0053] The mapping unit 230 derives an estimated semantic vector from the feature vector of the image of the new class. A torque is generated (S30).

[0054] If semantic information is attached to the image of the new class (Y in S40), the semantic The semantic vector generation unit 210 generates a correct semantic vector from the semantic information of the image of the new class. A tick vector is generated (S50).

[0055] If the image in the new class does not have semantic information (N in S40), The semantic prediction unit 250 derives the correct semantic vector from the feature vector of the image of the new class. The torque is predicted (S60).

[0056] The optimization unit 240 calculates the distance between the estimated semantic vector and the correct semantic vector. The parameters of the mapping unit 230 are optimized so that is minimized (S70).

[0057] The various processes of the machine learning device 200 described above are performed using hardware such as a CPU and memory. It can be realized as a device using ROM (Read Only Memory) firmware stored in the device, flash memory, etc., and software such as computers This can also be achieved by firmware programs, software The program can be provided by recording it on a computer-readable recording medium, or by transmitting it via wired Alternatively, you can send and receive data to and from the server via a wireless network, or via terrestrial or satellite data. It is also possible to transmit and receive it as digital broadcast data.

[0058] As described above, according to the machine learning device 200 of this embodiment, the suspected classes are generated from the base classes. Generate pseudo-few-shot classes and extract semantic information from the input images of the pseudo-few-shot classes. The semantic prediction unit that predicts information is meta-learned in advance. When performing the task, the semantic prediction information generated by the meta-learned semantic prediction unit is used as the correct answer. This allows us to continuously learn new classes using the data without semantic information. This enables learning and inference for new and novel classes.

[0059] The present invention has been described above based on the embodiments. The embodiments are merely examples, and the respective structures thereof are not intended to be limiting. The fact that various variations are possible in the combination of components and each treatment process, and that such variations It will be understood by those skilled in the art that such modifications are also within the scope of the present invention. [Explanation of symbols]

[0060] 100 machine learning device, 110 semantic vector generation unit, 120 feature extraction Output section, 130 Mapping section, 140 Optimization section, 200 Machine learning device, 21 0 Semantic vector generation unit, 220 Feature extraction unit, 230 Mapping unit, 240 Optimization part, 250 Semantic prediction part.

Claims

1. a feature extraction unit that extracts feature vectors from input data; generating a semantic vector from the feature vector of the input data; a trained semantic predictor; generating a semantic vector from the feature vector of the input data; A mapping part that has learned the class, The semantic vector generated by the semantic prediction unit is used as a correct semantic vector. The semantic vector generated by the mapping unit is used as a semantic vector. The parameters of the mapping unit are calculated so that the distance between the correct semantic vector and the correct semantic vector is minimized. and an optimization unit that optimizes the meter.

2. a feature extraction step of extracting a feature vector from the input data; Using a pre-meta-learned semantic prediction module, a semantic prediction step for generating a semantic vector from the feature vector; The features of the input data are calculated using a mapping module that has learned the base class. a mapping step to generate a semantic vector from the vector; The semantic vector generated by the semantic prediction step is called a correct answer. The semantic vector generated by the mapping step is used as a semantic vector. The mapping is performed so that the distance between the semantic vector and the correct semantic vector is minimized. and optimizing the parameters of the input module. How to learn.

3. a feature extraction step of extracting a feature vector from the input data; Using a pre-meta-learned semantic prediction module, a semantic prediction step for generating a semantic vector from the feature vector; The features of the input data are calculated using a mapping module that has learned the base class. a mapping step to generate a semantic vector from the vector; The semantic vector generated by the semantic prediction step is called a correct answer. The semantic vector generated by the mapping step is used as a semantic vector. The mapping is performed so that the distance between the semantic vector and the correct semantic vector is minimized. and optimizing the parameters of the programming module. A machine learning program characterized by:

Images