Machine learning device, machine learning method, and machine learning program
By generating language features containing visual concepts, selecting base-class images as reproduced data, combined with pre-training neural networks, the problem of degradation of classification accuracy caused by randomly selected images in the SaB method is solved, and more efficient incremental learning is achieved.
Patent Information
- Application Number
- JP2023122295
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-07-27
- Publication Date
- 2025-02-06
AI Technical Summary
In the existing SaB method, when randomly selecting the base class image as the reproduced data, the effective characteristics of the base class cannot be fully reflected, resulting in a decrease in classification accuracy.
By entering text describing the base class image, a language feature containing visual concepts is generated, and an image close to the language feature is selected as the reproduced data, combining the pre-trained neural network for classification and update weights.
Improve the classification accuracy of base-class images and realize effective incremental learning.
Smart Images
Figure 2025018521000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to machine learning techniques. [Background technology]
[0002] While humans can learn new knowledge and retain old knowledge through long-term experience, the knowledge of a Convolutional Neural Network (CNN) depends on the dataset it was trained on, and adapting to changes in the data distribution requires retraining the CNN parameters on the entire dataset.
[0003] A more efficient and practical method is incremental learning, or continual learning, which learns new tasks while reusing previously acquired knowledge. In particular, incremental learning in classification tasks is a method that allows a system to learn and classify new classes (new classes) from a state in which it can classify base classes (classes learned in the past).
[0004] On the other hand, in deep learning, there is a phenomenon called catastrophic forgetting, in which previously acquired knowledge is largely lost, resulting in a significant decline in task performance. This phenomenon is particularly problematic in continuous learning. In continuous learning for classification tasks, the biggest challenge is to suppress catastrophic forgetting and acquire classification performance for new classes while maintaining classification performance for basic classes.
[0005] On the other hand, since new tasks often only have a small amount of sample data available, few-shot learning has been proposed as a method for efficient learning with a small amount of training data. Normally, several thousand or more samples are required for learning, but in few-shot learning, learning is performed with a small number of samples (for example, a few samples).
[0006] In addition, Class Incremental Learning (CIL) has been proposed, which allows classification of new classes (new classes) by additionally learning base classes to a trained model. In CIL, tasks are continuously added to a trained model for class classification, and classification performance for new classes and past classes is required for new tasks. Note that the training data for new tasks is usually big data.
[0007] A method called Few-Shot Class Incremental Learning (FSCIL) has been proposed that combines continuous learning, which learns new classes without fatal forgetting of the learning results of base classes, with few-shot learning, which learns new classes that are fewer in number than the base classes (Non-Patent Document 1). Continuous few-shot learning can learn base classes from a large data set, and new classes from a small amount of sample data. FSCIL is an incremental learning scenario for class classification that is similar to CIL, but it is significantly different in that the learning data for new tasks is small (small data).
[0008] SaB (Split-and-Bridge) has been proposed as one of the continuous learning methods in class classification learning (see, for example, Non-Patent Document 2). SaB realizes high adaptability to new classes and suppresses forgetting of past knowledge while suppressing the increase in the size of the network. SaB consists of a split phase in which the network is divided into partitions for past knowledge and new knowledge in an incremental task and learning is performed, and a bridge phase in which the network is then recombined and learning is performed. In the split phase, the lower layer of the network is shared by the past knowledge and new knowledge, and the upper layer of the network is divided and assigned to each of the past knowledge and new knowledge, so that the past knowledge and new knowledge in the local space are acquired separately (learning is performed simultaneously). In the bridge phase, the divided networks are combined to learn integrated knowledge of the past knowledge (basic class) and new knowledge (new class). [Prior art documents] [Non-patent literature]
[0009] [Non-Patent Document 1] Zhang, C., Song, N., Lin, G., Zheng, Y., Pan, P., & Xu, Y. (2021). Few-shot incremental learning with continually evolved classifiers. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 12455-12464). [Non-Patent Document 2] Jong-Yeong Kim, Dong-Wan Choi. (2021). “Split-and-Bridge: Adaptable Class Incremental Learning within a Single Neural Network.” In Proceedings of the AAAI Conference on Artificial Intelligence (pp. 8137-8145). [Non-Patent Document 3] Nishida, K., Nishida, K., & Nishioka, S. (2022). Improving Few-Shot Image Classification Using Machine-and User-Generated Natural Language Descriptions. arXiv preprint arXiv:2207.03133. Summary of the Invention [Problem to be solved by the invention]
[0010] In SaB, images of new classes and some images of base classes are used as rehearsal data for learning incremental tasks. In SaB, some images of base classes are used during incremental learning, but the images are selected randomly. When images are selected randomly, there is an issue that effective features representing base classes cannot be sufficiently reflected in incremental learning.
[0011] The present invention has been made in consideration of these circumstances, and its purpose is to provide a machine learning technique that can improve classification accuracy by selecting images for base classes as rehearsal data based on linguistic features including the visual concept of the base class. [Means for solving the problem]
[0012] In order to solve the above problem, a machine learning device according to an embodiment of the present invention includes a language feature output unit that receives as input a text describing an image of a base class and outputs a language feature, an image feature output unit that receives as input an image of the base class and outputs an image feature, and a base class image selection unit that receives as input the language feature, the image feature, and an image of the base class and selects an image of the base class corresponding to the image feature that has a distance from the language feature that is equal to or less than a predetermined threshold. The device further includes a pre-trained neural network. The neural network includes a lower neural network section that receives as input a base class image and a new class image selected by the base class image selection section and outputs a value, and an upper neural network section that is disposed on the output side of the lower neural network section, the upper neural network section including a base class classification output section that receives as input an output value of the lower neural network section based on the base class image and the new class image and outputs a base class classification that is a class classification based on the base class image and the new class image, and a new class classification output section that receives as input an output value of the lower neural network section based on the new class image and outputs a new class classification that is a class classification based on the new class image. The device further includes a loss calculation section that calculates a loss of the base class classification and a loss of the new class classification based on the base class classification and the new class classification, and an update section that updates the weights of the lower neural network section, the weights of the base class classification output section, and the weights of the new class classification output section based on a loss obtained by adding together the loss of the base class classification and the loss of the new class classification.
[0013] Another aspect of the present invention is also a machine learning device. The device includes a language feature output unit that receives text describing an image of a base class as an input and outputs a language feature, an image feature output unit that receives an image of a base class as an input and outputs an image feature, and a base class image selection unit that receives the language feature, the image feature, and an image of the base class as an input and selects an image of the base class corresponding to the image feature having a distance from the language feature equal to or less than a predetermined threshold. The device further includes a pre-trained neural network. The neural network includes a neural network lower layer unit that receives an image of the base class and an image of a new class selected by the base class image selection unit as input and outputs a value, and an upper layer unit of the neural network arranged on the output side of the lower layer unit of the neural network, the upper layer unit receiving an output value of the lower layer unit of the neural network based on the image of the base class and the image of the new class, and outputting a base class classification or a new class classification that is a class classification based on the image of the base class and the image of the new class. The device further includes a loss calculation unit that calculates a loss of the base class classification and a loss of the new class classification based on the base class classification and the new class classification, and an update unit that updates the weights of the lower layer section of the neural network and the weights of the upper layer section of the neural network based on a loss obtained by adding together the loss of the base class classification and the loss of the new class classification.
[0014] Yet another aspect of the present invention is a machine learning method, which includes a language feature output step of receiving as input a text describing an image of a base class and outputting a language feature, an image feature output step of receiving as input an image of a base class and outputting an image feature, a base class image selection step of receiving as input the language feature, the image feature, and an image of the base class and selecting an image of the base class corresponding to the image feature having a distance from the language feature equal to or less than a predetermined threshold value, a step of receiving as input, in a lower layer part of a pre-trained neural network, an image of the base class selected by the base class image selection step and an image of a new class, and outputting a value, and a step of receiving, in an upper layer part of the neural network arranged on the output side of the lower layer part of the neural network, an image of the base class selected by the base class image selection step and an image of a new class, and outputting a value, The method includes a step of receiving an output value of the lower layer section of the neural network based on the image of the new class as an input, and outputting a base class classification which is a class classification based on the image of the base class and the image of the new class, and a new class classification output unit receiving an output value of the lower layer section of the neural network based on the image of the new class as an input, and outputting a new class classification which is a class classification based on the image of the new class; a loss calculation step of calculating a loss of the base class classification and a loss of the new class classification based on the base class classification and the new class classification; and an update step of updating weights of the lower layer section of the neural network, weights of the base class classification output unit, and weights of the new class classification output unit based on a loss obtained by adding together the loss of the base class classification and the loss of the new class classification.
[0015] Any combination of the above components, and any transformation of the present invention into a method, device, system, recording medium, computer program, etc., are also effective as aspects of the present invention. Effect of the Invention
[0016] According to the present invention, a machine learning technique can be provided that can improve classification accuracy by selecting images for a base class as rehearsal data based on linguistic features including the visual concept of the base class. [Brief description of the drawings]
[0017] [Figure 1] FIG. 2 is a diagram illustrating a configuration of a pre-training module. [Diagram 2] FIG. 13 is a diagram illustrating the configuration of a NN used in the split phase of SaB. [Diagram 3] FIG. 1 is a functional block diagram for explaining the configuration of a conventional machine learning device used in the split phase of SaB. [Figure 4] FIG. 1 is a functional block diagram for explaining the configuration of a conventional machine learning device used in the bridge phase of SaB. [Diagram 5] 11A and 11B are diagrams illustrating a method for selecting a base class image in the present embodiment. [Figure 6] FIG. 2 is a functional configuration diagram of the machine learning device of the present embodiment in the split phase. [Figure 7] FIG. 2 is a functional configuration diagram of the machine learning device according to the present embodiment in a bridge phase. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0018] First, an overview of SaB, a conventional technique, will be given. In SaB, class classification is performed using a general neural network (hereinafter sometimes referred to as "NN") model.
[0019] First, in the basic task of incremental learning, the classification of basic classes is pre-trained using big data in a NN. Figure 1 is a diagram illustrating the configuration of a pre-training module 30. The pre-training module 30 includes a NN 32 and a basic class classification weight Θt of the NN 32.
[0020] The basic class data set 10 includes N samples. An example of a sample is an image, but is not limited to this. The NN32 is a neural network that pre-trains the basic class data set 10. The weight of the NN32 is Θt.
[0021] In the incremental task of incremental learning in SaB, learning is performed in the split phase based on learned weights, and the learned weights are further trained in the bridge phase.
[0022] The purpose of the split phase is to learn past knowledge (base class) in a local space that classifies only past classes of past tasks for the current incremental task, and new knowledge (new class) in a local space that classifies only new classes of the current incremental task. For this reason, in the split phase, the upper layer of the NN32 is divided into two partitions: one part that uses weight θo to learn the base class, and one part that uses weight θn to learn the new class. In the lower layer of the NN32, weight θs is shared between the base class and the new class. At this time, the base class uses <θs, θo> to calculate a loss, and the new class uses <θs, θn> to calculate a loss, and learning is based on the loss obtained by adding up the respective losses.
[0023] FIG. 2 is a diagram illustrating the configuration of the NN32 used in the split phase of SaB. As shown in FIG. 2, in SaB, the NN lower layer part 110 consisting of one or more layers on the input side and the NN upper layer part 120 consisting of one or more layers on the output side of the NN lower layer part 110 are set. The total weight of the NN32 is Θt, of which the weight θs is used in the NN lower layer part 110, and the two divided basic class classification weights θo and new class classification weights θn are used in the NN upper layer part 120. The NN upper layer part 120 includes a basic class classification output part 121 that uses the basic class classification weight θo and a new class classification output part 122 that uses the new class classification weight θn. Here, before the split phase, a preprocessing is performed to make the weights to be cut in the split phase sparse. Each node of the basic class classification output part 121 and each node of the new class classification output part 122 are not connected, and there is no propagation between these nodes. As a method for setting the NN lower layer unit 110 with weight θs, the basic class classification output unit 121 with weight θo, and the new class classification output unit 122 with weight θn based on the pre-trained NN 32 with weight Θt, for example, the method described in Non-Patent Document 2 is used.
[0024] FIG. 3 is a functional block diagram for explaining the configuration of a conventional machine learning device 100 used in the split phase of SaB. The machine learning device 100 in FIG. 3 represents a state before learning of an incremental task. The dataset 1 includes rehearsal data 15 of a base class and a dataset 20 of a new class. The rehearsal data 15 of the base class is a dataset that is a part of the dataset 10 of the base class, and includes n samples (N>n). The dataset 20 of the new class includes k samples. An example of a sample is an image, but is not limited to this.
[0025] The conventional machine learning device 100 includes a first trained NN 32s that has been trained in advance on a base class, a first loss calculation unit 130s, and a first update unit 140s. The first trained NN 32s includes an NN lower layer unit 110s and an NN upper layer unit 120s.
[0026] The NN lower layer unit 110s receives the basic class data and the new class data as input, and outputs a value using the weight θs for both the basic class data and the new class data.
[0027] As described above, in SaB, the NN upper layer unit 120s includes a basic class classification output unit 121 that uses a weight θo and a new class classification output unit 122 that uses a weight θn. The basic class classification output unit 121 receives an output value of the NN lower layer unit 110s based on the data of the basic class and the new class, and outputs a class classification based on the data of the basic class and the new class (hereinafter referred to as basic class classification) using the weight θo. The new class classification output unit 122 receives an output value of the NN lower layer unit 110s based on the data of the new class, and outputs a class classification based on the data of the new class (hereinafter referred to as new class classification) using the weight θn.
[0028] The first loss calculation unit 130s inputs the base class classification and the new class classification from the NN upper layer unit 120s, calculates a knowledge distillation loss Lkd based on the base class classification, and calculates a cross-entropy loss Llce based on the new class classification.
[0029] The first update unit 140s inputs the knowledge distillation loss Lkd and the cross-entropy loss Llce from the first loss calculation unit 130s, and updates the weights θs, θo, and θn based on the loss obtained by adding the knowledge distillation loss Lkd and the cross-entropy loss Llce. In this update of the weights θs, θo, and θn, the weights θs, θo, and θn of the NN lower layer unit 110s are updated so as to reduce the sum of the knowledge distillation loss Lkd and the cross-entropy loss Llce. As a calculation method of the loss of classification in the first loss calculation unit 130s and an update method in the first update unit 140s, for example, a method described in Non-Patent Document 2 is used.
[0030] The series of processes in the split phase described above is repeatedly executed according to one or more epoch numbers defined as hyperparameters.
[0031] In the bridge phase, the purpose is to learn integrated knowledge for classifying all past classes and new classes of the current incremental task, and the integrated knowledge is learned for the weights θs, θo, and θn updated in the split phase. In the bridge phase, nodes that were not connected between the basic class classification output unit 121 and the new class classification output unit 122 in Figure 2 are connected, and learning is performed in a normal fully connected NN state.
[0032] 4 is a functional block diagram for explaining the configuration of a conventional machine learning device 100 used in the bridge phase of SaB. Here, points that overlap with the explanation of the configuration of a conventional machine learning device 100 used in the split phase of SaB will be omitted as appropriate, and only the differences will be explained.
[0033] The conventional machine learning device 100 includes a second trained NN 32b trained in the split phase, a second loss calculation unit 130b, and a second update unit 140b. In the bridge phase, the second trained NN 32b uses the weights of the classifier trained in the first trained NN 32s, i.e., the weights θs, θo, and θn updated by the first update unit 140s in the split phase, as initial values. The second trained NN 32b includes an NN lower layer unit 110b that uses the weight θs updated in the split phase, and an NN upper layer unit 120b that uses a weight θp obtained by integrating the weights θo and θn updated in the split phase.
[0034] The second trained NN32b receives the base class data and the new class data as input, and outputs a class classification based on the base class data and the new class data using weights θs and θp (hereinafter, referred to as integrated class classification). The data input to the second trained NN32b is the same data as that used in the split phase. The second trained NN32b has the same number of layers and nodes as the first trained NN32s, and corresponds to a configuration in which all nodes of adjacent layers in the base class classification output unit 121 and the new class classification output unit 122 of the first trained NN32s are connected. The NN lower layer unit 110b of the second trained NN32b has the same number of layers and nodes as the NN lower layer unit 110s of the first trained NN32s. The NN upper layer unit 120b of the second trained NN 32b has the same number of layers and nodes as the NN upper layer unit 120s of the first trained NN 32s, and corresponds to a configuration in which all nodes of adjacent layers in the basic class classification output unit 121 and the new class classification output unit 122 of the first trained NN 32s are connected. Therefore, the NN upper layer unit 120b of the second trained NN 32b corresponds to a configuration in which the basic class classification output unit 121 and the new class classification output unit 122 of the NN upper layer unit 120s of the first trained NN 32s are integrated.
[0035] The second loss calculation unit 130b inputs the integrated class classification from the second trained NN 32b, calculates the knowledge distillation loss Lkd and the cross entropy loss Lce based on the integrated class classification, and calculates the sum of the knowledge distillation loss Lkd and the cross entropy loss Lce as the loss of the class classification. The sum of the knowledge distillation loss Lkd and the cross entropy loss Lce in the bridge phase is an example of the loss of the class classification.
[0036] The second update unit 140b updates the weights θs and θp of the second trained NN 32b based on the classification loss. For example, the second update unit 140b inputs the classification loss from the second loss calculation unit 130b and updates the weights θs and θp based on the classification loss. In this update of the weights θs and θp, the weights θs, θp of the second trained NN 32b are updated so as to reduce the classification loss.
[0037] The series of processes in the bridge phase described above is repeatedly executed according to one or more epoch numbers defined as hyperparameters.
[0038] Conventional SaB assumes CIL and uses big data, i.e., a large number of samples, for new classes in incremental tasks.
[0039] Next, an embodiment of the present invention will be described. In the conventional technology, some images of the base class are used as rehearsal data during incremental learning, and the images are selected randomly. In this embodiment, linguistic features including the visual concept of the base class are generated, and images having features in the vicinity of the linguistic features are selected as images for the base class (rehearsal data).
[0040] FIG. 5 is a diagram for explaining a method for selecting a base class image in this embodiment.
[0041] The Image Encoder 300 and Text Encoder 310 in Fig. 5 use a trained model that has been fully trained using big data that pairs images with text describing the images, as described in Non-Patent Document 3 as an example. A plurality of images of a basic class are passed through the Image Encoder 300 to obtain image features for each image. Furthermore, text describing the images of the basic class is passed through the Text Encoder 310 to obtain language features. Here, the formats of the acquired image features and language features are aligned so that they can be projected into the same feature space 320.
[0042] The text describing the base class image describes the visual concept of the base class image, for example, "The bird is gray with white and has a short beak."
[0043] In the feature space 320, images having image features 340 in the vicinity of linguistic features 330 including the visual concept of the base class are selected and used as images (rehearsal data) of the base class during incremental learning.
[0044] According to this embodiment, an image having a feature quantity that represents the visual concept of a base class can be used during incremental learning, enabling effective learning of the base class.
[0045] 6 is a functional configuration diagram of the machine learning device 200 of this embodiment in the split phase. The machine learning device 200 of this embodiment includes a NN lower layer unit 110s, a base class classification output unit 121, a new class classification output unit 122, a first loss calculation unit 130s, a first update unit 140s, a language feature output unit 210, an image feature output unit 220, and a base class image selection unit 230.
[0046] The NN lower layer unit 110s, the base class classification output unit 121, the new class classification output unit 122, the first loss calculation unit 130s, and the first update unit 140s of the machine learning device 200 of this embodiment in Figure 6 correspond to the NN lower layer unit 110s, the base class classification output unit 121, the new class classification output unit 122, the first loss calculation unit 130s, and the first update unit 140s of the conventional machine learning device 100 in Figure 3, respectively.
[0047] The machine learning device 200 of this embodiment differs from the conventional machine learning device 100 in that, in addition to the configuration of the conventional machine learning device 100, it further includes a language feature output unit 210, an image feature output unit 220, and a base class image selection unit 230.
[0048] The language feature output unit 210 receives an input of text describing a base class image, extracts language features from the text describing the base class image using a trained model trained from the image and the text describing the image, and supplies the language features to the base class image selection unit 230. The language feature output unit 210 corresponds to the Text Encoder 310 in FIG. 5, for example.
[0049] The image feature output unit 220 receives an input of all base class images, extracts image features of each image of the base class using a trained model trained from images and text describing the images, and supplies the image features of each image to the base class image selection unit 230. The image feature output unit 220 corresponds to the Image Encoder 300 in FIG. 5, for example.
[0050] The base class image selection unit 230 receives input of language features, image features of each image, and all base class images, selects base class images having image features that are within a predetermined threshold distance from the language features as rehearsal data, and supplies the selected base class images to the NN lower layer unit 110s.
[0051] The NN lower layer unit 110s receives an input of a new class image and an input of a base class image selected as rehearsal data, and outputs values for the base class image and the new class image using a weight θs. The NN lower layer unit 110s supplies an output based on the base class image and an output based on the new class image to the base class classification output unit 121, and supplies an output based on the new class image to the new class classification output unit 122.
[0052] The base class classification output unit 121 receives an output based on the base class image and an output based on the new class image from the NN lower layer unit 110s, outputs the base class classification using the weight θo, and supplies it to the first loss calculation unit 130s.
[0053] The new class classification output unit 122 receives an output based on the new class image from the NN lower layer unit 110s as input, outputs a new class classification using the weight θn, and supplies the new class classification to the first loss calculation unit 130s.
[0054] The first loss calculation unit 130s receives an input of a base class classification and a new class classification, calculates a knowledge distillation loss based on the base class classification, calculates a cross-entropy loss based on the new class classification, and supplies the results to the first update unit 140s.
[0055] The first update unit 140s updates the weights θs, θo, and θn so as to reduce the sum of the knowledge distillation loss and the cross-entropy loss.
[0056] 7 is a functional configuration diagram of the machine learning device 200 of this embodiment in the bridge phase. The machine learning device 200 of this embodiment includes a NN lower layer unit 110b, a class classification output unit 120b, a second loss calculation unit 130b, a second update unit 140b, a language feature output unit 210, an image feature output unit 220, and a base class image selection unit 230.
[0057] The NN lower layer unit 110b, the class classification output unit 120b, the second loss calculation unit 130b, and the second update unit 140b of the machine learning device 200 of this embodiment in Figure 7 correspond to the NN lower layer unit 110b, the NN upper layer unit 120b, the second loss calculation unit 130b, and the second update unit 140b of the conventional machine learning device 100 in Figure 4, respectively.
[0058] The operations of the language feature output unit 210, the image feature output unit 220, and the base class image selection unit 230 are the same as those in the split phase of FIG. 6, and therefore will not be described.
[0059] The NN lower layer unit 110b receives an input of a new class image and an input of a base class image selected as rehearsal data, and outputs values for the base class image and the new class image using weights θs. The NN lower layer unit 110b supplies an output based on the base class image and an output based on the new class image to the classification output unit 120b.
[0060] The class classification output unit 120b receives as input the output based on the base class image and the output based on the new class image from the NN lower layer unit 110b, outputs the base class classification and the new class classification using a weight θp obtained by integrating the weights θo and θn updated in the split phase, and supplies them to the second loss calculation unit 130b.
[0061] The second loss calculation unit 130b calculates a knowledge distillation loss and a cross-entropy loss based on the integrated classification that integrates the outputs of the basic class classification and the new class classification, and supplies the calculated loss to the second update unit 140b.
[0062] The second update unit 140b updates the weights θs and θp so as to reduce the sum of the knowledge distillation loss and the cross-entropy loss.
[0063] According to the machine learning device 200 of this embodiment, effective class incremental learning is possible by generating linguistic features including the visual concept of a base class and selecting images having features in the vicinity of the linguistic features as images (rehearsal data) for the base class.
[0064] The various processes of the machine learning device 200 described above can be realized not only as a device using hardware such as a CPU and a memory, but also as firmware stored in a ROM (read-only memory) or flash memory, or software for a computer, etc. The firmware program or software program can be provided by recording it on a recording medium readable by a computer, etc., or can be transmitted and received with a server via a wired or wireless network, or can be transmitted and received as a data broadcast of terrestrial or satellite digital broadcasting.
[0065] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each treatment process, and that such modifications are also within the scope of the present invention. [Explanation of symbols]
[0066] 1 Dataset, 10 Base class dataset, 15 Base class rehearsal data, 20 New class dataset, 30 Pre-training module, 32 NN, 110 NN lower layer, 120 NN upper layer, 121 Base class classification output unit, 122 New class classification output unit, 130 Loss calculation unit, 140 Update unit, 100, 200 Machine learning device, 210 Language feature output unit, 220 Image feature output unit, 230 Base class image selection unit.
Claims
1. a language feature output unit that receives a text describing an image of a basic class as input and outputs language features; an image feature output unit that receives an image of a base class as an input and outputs an image feature; a base class image selection unit that receives the language feature, the image feature, and an image of the base class, and selects an image of the base class corresponding to the image feature that has a distance from the language feature equal to or less than a predetermined threshold; A pre-trained neural network, a neural network lower layer unit which receives the base class image selected by the base class image selection unit and the new class image and outputs a value; the neural network upper part is disposed on the output side of the neural network lower part, the neural network upper part including: a basic class classification output unit that receives an output value of the neural network lower part based on the base class image and the new class image, and outputs a basic class classification that is a class classification based on the base class image and the new class image; and a new class classification output unit that receives an output value of the neural network lower part based on the new class image, and outputs a new class classification that is a class classification based on the new class image; the neural network comprising: a loss calculation unit that calculates a loss of the base class classification and a loss of the new class classification based on the base class classification and the new class classification; an update unit that updates the weights of the lower layer unit of the neural network, the weights of the base class classification output unit, and the weights of the new class classification output unit based on a loss obtained by adding up the loss of the base class classification and the loss of the new class classification; A machine learning device comprising:
2. The machine learning device according to claim 1 , wherein the language feature output unit and the image feature output unit have completed pre-learning using as input an image of the base class and a text describing the image of the base class, respectively.
3. a language feature output unit that receives a text describing an image of a basic class as input and outputs language features; an image feature output unit that receives an image of a base class as an input and outputs an image feature; a base class image selection unit that receives the language feature, the image feature, and an image of the base class as input, and selects an image of the base class corresponding to the image feature that has a distance from the language feature equal to or smaller than a predetermined threshold; A pre-trained neural network, a neural network lower layer unit which receives the base class image selected by the base class image selection unit and the new class image and outputs a value; an upper layer part of the neural network arranged on the output side of the lower layer part of the neural network, the upper layer part of the neural network receiving an output value of the lower layer part of the neural network based on the image of the base class and the image of the new class, and outputting a basic class classification or a new class classification which is a class classification based on the image of the base class and the image of the new class; the neural network comprising: a loss calculation unit that calculates a loss of the base class classification and a loss of the new class classification based on the base class classification and the new class classification; an update unit that updates the weights of the lower layer of the neural network and the weights of the upper layer of the neural network based on a loss obtained by adding together the loss of the basic class classification and the loss of the new class classification; A machine learning device comprising:
4. a language feature output step of inputting a text describing an image of a basic class and outputting a language feature; an image feature output step of inputting an image of a base class and outputting an image feature; a base class image selection step of inputting the language feature, the image feature, and an image of the base class, and selecting an image of the base class corresponding to the image feature having a distance from the language feature equal to or less than a predetermined threshold; a step of inputting the base class image selected by the base class image selection step and the new class image in a lower layer of the pre-trained neural network and outputting a value; In an upper part of the neural network arranged on the output side of the lower part of the neural network, a basic class classification output unit receives an output value of the lower part of the neural network based on the image of the basic class and the image of the new class, and outputs a basic class classification which is a class classification based on the image of the basic class and the image of the new class, and a new class classification output unit receives an output value of the lower part of the neural network based on the image of the new class, and outputs a new class classification which is a class classification based on the image of the new class; a loss calculation step of calculating a loss of the base class classification and a loss of the new class classification based on the base class classification and the new class classification; an updating step of updating weights of the lower layer section of the neural network, weights of the base class classification output section, and weights of the new class classification output section based on a loss obtained by adding up the loss of the base class classification and the loss of the new class classification; A machine learning method comprising:
5. a language feature output step of inputting a text describing an image of a basic class and outputting a language feature; an image feature output step of inputting an image of a base class and outputting an image feature; a base class image selection step of inputting the language feature, the image feature, and an image of the base class, and selecting an image of the base class corresponding to the image feature having a distance from the language feature equal to or less than a predetermined threshold; a step of inputting the base class image selected by the base class image selection step and the new class image in a lower layer of the pre-trained neural network and outputting a value; In an upper part of the neural network arranged on the output side of the lower part of the neural network, a basic class classification output unit receives an output value of the lower part of the neural network based on the image of the basic class and the image of the new class, and outputs a basic class classification which is a class classification based on the image of the basic class and the image of the new class, and a new class classification output unit receives an output value of the lower part of the neural network based on the image of the new class, and outputs a new class classification which is a class classification based on the image of the new class; a loss calculation step of calculating a loss of the base class classification and a loss of the new class classification based on the base class classification and the new class classification; an updating step of updating weights of the lower layer section of the neural network, weights of the base class classification output section, and weights of the new class classification output section based on a loss obtained by adding up the loss of the base class classification and the loss of the new class classification; A machine learning program characterized by causing a computer to execute the above.