Machine learning device, machine learning method, and machine learning program

Abstract

A language feature output unit (210) inputs a text describing base class image and outputs language feature. An image feature output unit (220) inputs a base class image and outputs an image feature. A base class image selection unit (230) inputs an language feature, the image feature, and the base class image and selects a base class image corresponding to the image feature having a distance from the language feature less than or equal to a predetermined threshold value. A neural network lower-layer unit (110s) inputs the base class image selected by the base class image selection unit (230) and a novel class image and outputs a value based on the base class image and a value based on the novel class image. A base class classification output unit (121) outputs a base class classification based on the base class image and the novel class image. A novel class classification output unit (122) outputs a novel class classification based on a novel class image.

Description

Machine learning device, machine learning method, and machine learning program

[0001] The present invention relates to machine learning techniques.

[0002] While humans can learn new knowledge and retain old knowledge through long-term experience, the knowledge of a convolutional neural network (CNN) is dependent on the dataset it was trained on, and adapting to changes in the data distribution requires retraining the CNN parameters for the entire dataset.

[0003] A more efficient and practical approach is incremental learning, which reuses previously acquired knowledge while learning new tasks. In particular, incremental learning for classification tasks is a method that allows a system to learn and classify new classes (new classes) based on the ability to classify base classes (classes learned previously).

[0004] On the other hand, in deep learning, there is a phenomenon called catastrophic forgetting, in which previously acquired knowledge is significantly lost, resulting in a significant decline in task performance. This phenomenon is particularly problematic in continuous learning. In continuous learning for classification tasks, the biggest challenge is to suppress catastrophic forgetting and acquire classification performance for new classes while maintaining classification performance for basic classes.

[0005] On the other hand, since new tasks often require only a small amount of sample data, few-shot learning has been proposed as a method for efficiently learning with a small amount of training data. While learning normally requires several thousand or more samples, few-shot learning uses only a small number of samples (e.g., a few samples).

[0006] In addition, Class Incremental Learning (CIL) has been proposed, which allows for the classification of new classes (new classes) by additionally training base classes on a trained model. In CIL, tasks are continuously added to a trained model for classification, and classification performance for new and past classes is required for new tasks. Note that the training data for new tasks is usually big data.

[0007] A method called Few-Shot Class Incremental Learning (FSCIL) has been proposed (Non-Patent Document 1), which combines continuous learning, which learns new classes without fatal forgetting from the learning results of base classes, with few-shot learning, which learns new classes that are fewer in number than the base classes. Continuous few-shot learning can learn base classes from a large data set and new classes from a small number of sample data. FSCIL is an incremental learning scenario for classification similar to CIL, but differs significantly in that the training data for new tasks is small (small data).

[0008] Split-and-Bridge (SaB) has been proposed as one of the continuous learning methods in classification learning (see, for example, Non-Patent Document 2). SaB achieves high adaptability to new classes and suppresses forgetting of past knowledge while suppressing network size growth. SaB consists of a split phase in which the network is divided into partitions for past knowledge and new knowledge for incremental tasks and training is performed, and a bridge phase in which the network is then recombined and training is performed. In the split phase, the lower layer of the network is shared between past knowledge and new knowledge, and the upper layer of the network is divided and assigned to each of the past knowledge and new knowledge, thereby acquiring past knowledge and new knowledge separately in the local space (learning is performed simultaneously). In the bridge phase, the divided networks are combined to learn integrated knowledge of past knowledge (base class) and new knowledge (new class).

[0009] Zhang, C., Song, N., Lin, G., Zheng, Y., Pan, P., & Xu, Y. (2021). Few-shot incremental learning with continually evolved classifiers. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 12455-12464). Jong-Yeong Kim, Dong-Wan Choi. (2021). “Split-and-Bridge: Adaptable Class Incremental Learning within a Single Neural Network.” In Proceedings of the AAAI Conference on Artificial Intelligence (pp. 8137-8145). Nishida, K., Nishida, K., & Nishioka, S. (2022). Improving Few-Shot Image Classification Using Machine- and User-Generated Natural Language Descriptions. arXiv preprint arXiv:2207.03133.

[0010] In SaB, images from the new class and some of the images from the base class are used as rehearsal data for incremental task learning. SaB uses some of the images from the base class during incremental learning, but these images are selected randomly. When images are selected randomly, there is an issue that effective features representing the base class cannot be fully reflected in incremental learning.

[0011] This embodiment has been made in consideration of these circumstances, and its purpose is to provide a machine learning technology that can improve classification accuracy by selecting images for base classes as rehearsal data based on linguistic features including the visual concepts of the base classes.

[0012] To solve the above problem, a machine learning device according to one aspect of the present embodiment includes a language feature output unit that receives as input text describing an image of a base class and outputs language features, an image feature output unit that receives as input an image of the base class and outputs image features, and a base class image selection unit that receives as input the language features, the image features, and an image of the base class and selects an image of the base class corresponding to the image features that are at a distance from the language features that is equal to or less than a predetermined threshold. The device further includes a pre-trained neural network. The neural network includes a neural network lower layer section that receives as input a base class image and a new class image selected by the base class image selection section and outputs a value, and a neural network upper layer section that is arranged on the output side of the neural network lower layer section, the neural network upper layer section including: a base class classification output section that receives as input an output value of the neural network lower layer section based on the base class image and the new class image and outputs a base class classification that is a class classification based on the base class image and the new class image, and a new class classification output section that receives as input an output value of the neural network lower layer section based on the new class image and outputs a new class classification that is a class classification based on the new class image. The device further includes a loss calculation section that calculates a loss of the base class classification and a loss of the new class classification based on the base class classification and the new class classification, and an update section that updates the weights of the neural network lower layer section, the weights of the base class classification output section, and the weights of the new class classification output section based on a loss obtained by adding together the loss of the base class classification and the loss of the new class classification.

[0013] Another aspect of the present embodiment is also a machine learning device. This device includes: a language feature output unit that receives text describing a base class image as input and outputs language features; an image feature output unit that receives a base class image as input and outputs image features; and a base class image selection unit that receives the language features, the image features, and the base class image as input and selects the base class image corresponding to the image features that are a distance from the language features that is equal to or less than a predetermined threshold. This device further includes a pre-trained neural network. This neural network includes: a neural network lower layer unit that receives as input the base class image and a new class image selected by the base class image selection unit and outputs a value; and an neural network upper layer unit that is disposed on the output side of the neural network lower layer unit and receives as input output values ​​of the neural network lower layer unit based on the base class image and the new class image, and outputs a base class classification or a new class classification that is a class classification based on the base class image and the new class image. The device further includes a loss calculation unit that calculates a loss of the base class classification and a loss of the new class classification based on the base class classification and the new class classification, and an update unit that updates the weights of the lower layer section of the neural network and the weights of the upper layer section of the neural network based on a loss obtained by adding together the loss of the base class classification and the loss of the new class classification.

[0014] Yet another aspect of this embodiment is a machine learning method, which includes a language feature output step of receiving as input a text describing an image of a base class and outputting a language feature, an image feature output step of receiving as input an image of a base class and outputting an image feature, a base class image selection step of receiving as input the language feature, the image feature, and an image of the base class and selecting an image of the base class corresponding to the image feature having a distance from the language feature equal to or less than a predetermined threshold, a step of receiving as input, in a lower layer part of a pre-trained neural network, an image of the base class selected in the base class image selection step and an image of a new class, and outputting a value, and an upper layer part of the neural network arranged on the output side of the lower layer part of the neural network, a base class classification output unit of the image of the base class and The method includes a step of receiving an output value of the neural network lower layer section based on the image of the new class as input, and outputting a base class classification which is a class classification based on the image of the base class and the image of the new class, and a new class classification output section receiving an output value of the neural network lower layer section based on the image of the new class as input, and outputting a new class classification which is a class classification based on the image of the new class; a loss calculation step of calculating a loss of the base class classification and a loss of the new class classification based on the base class classification and the new class classification; and an update step of updating the weights of the neural network lower layer section, the weights of the base class classification output section, and the weights of the new class classification output section based on a loss obtained by adding together the loss of the base class classification and the loss of the new class classification.

[0015] Any combination of the above components, and conversion of the expression of this embodiment into a method, device, system, recording medium, computer program, etc. are also valid aspects of this embodiment.

[0016] According to this embodiment, a machine learning technique can be provided that can improve classification accuracy by selecting images for base classes as rehearsal data based on linguistic features including the visual concept of the base class.

[0017] FIG. 1 is a diagram illustrating the configuration of a pre-training module. FIG. 2 is a diagram illustrating an example of the configuration of a NN used in the split phase of SaB. FIG. 3 is a functional block diagram illustrating the configuration of a conventional machine learning device used in the split phase of SaB. FIG. 4 is a functional block diagram illustrating the configuration of a conventional machine learning device used in the bridge phase of SaB. FIG. 5 is a diagram illustrating a method for selecting base class images in this embodiment. FIG. 6 is a functional configuration diagram of the machine learning device of this embodiment in the split phase. FIG. 7 is a functional configuration diagram of the machine learning device of this embodiment in the bridge phase.

[0018] First, an overview of SaB, a conventional technique, will be given. In SaB, class classification is performed using a general neural network (hereinafter sometimes referred to as "NN") model.

[0019] First, in the basic task of incremental learning, a neural network (NN) is pre-trained to classify basic classes using big data. Figure 1 illustrates the configuration of a pre-training module 30. The pre-training module 30 includes a neural network (NN) 32 and basic class classification weights Θt of the NN 32.

[0020] The base class data set 10 includes N samples. An example of a sample is an image, but is not limited to this. The NN 32 is a neural network that pre-trains the base class data set 10. The weight of the NN 32 is Θt.

[0021] In the incremental task of incremental learning in SaB, learning is performed in the split phase based on learned weights, and the learned weights are further trained in the bridge phase.

[0022] The purpose of the split phase is to learn past knowledge (base classes) in a local space that classifies only past classes of past tasks for the current incremental task, and new knowledge (new classes) in a local space that classifies only new classes of the current incremental task. Therefore, in the split phase, the upper part of the NN 32 is divided into two partitions: one that uses weights θo to learn the base classes, and one that uses weights θn to learn the new classes. In the lower part of the NN 32, weights θs are shared between the base classes and the new classes. At this time, the base classes use <θs, θo> to calculate losses, and the new classes use <θs, θn> to calculate losses, and the learning is based on the sum of the respective losses.

[0023] FIG. 2 illustrates the configuration of the neural network (NN) 32 used in the split phase of SaB. As shown in FIG. 2, SaB configures the NN 32 with a lower NN layer 110 consisting of one or more layers on the input side and an upper NN layer 120 consisting of one or more layers on the output side of the lower NN layer 110. The overall weight of the NN 32 is Θt, of which the lower NN layer 110 uses a weight θs, and the upper NN layer 120 uses two divided base class classification weights θo and new class classification weights θn. The upper NN layer 120 includes a base class classification output unit 121 that uses the base class classification weight θo and a new class classification output unit 122 that uses the new class classification weight θn. Here, prior to the split phase, preprocessing is performed to sparse the weights to be cut in the split phase. The nodes of the basic class classification output unit 121 are not connected to the nodes of the new class classification output unit 122, and there is no propagation between these nodes. As a method for setting the NN lower layer unit 110 with weight θs, the basic class classification output unit 121 with weight θo, and the new class classification output unit 122 with weight θn based on the pre-trained NN 32 with weight Θt, for example, the method described in Non-Patent Document 2 is used.

[0024] FIG. 3 is a functional block diagram illustrating the configuration of a conventional machine learning device 100 used in the split phase of SaB. The machine learning device 100 in FIG. 3 represents the state before learning of the incremental task. Dataset 1 includes rehearsal data 15 for the base class and a dataset 20 for the new class. The rehearsal data 15 for the base class is a partial dataset of the dataset 10 for the base class and includes n samples (N>n). The dataset 20 for the new class includes k samples. An example of a sample is, but is not limited to, an image.

[0025] The conventional machine learning device 100 includes a first trained neural network (NN) 32s that has pre-trained a base class, a first loss calculation unit 130s, and a first update unit 140s. The first trained NN 32s includes a NN lower layer unit 110s and a NN upper layer unit 120s.

[0026] The NN lower layer unit 110s receives the basic class data and the new class data as input, and outputs a value using the weight θs for both the basic class data and the new class data.

[0027] As described above, in SaB, the NN upper layer unit 120s includes a base class classification output unit 121 that uses a weight θo and a new class classification output unit 122 that uses a weight θn. The base class classification output unit 121 receives an output value from the NN lower layer unit 110s based on data of the base class and the new class, and outputs a class classification based on the data of the base class and the new class (hereinafter referred to as the base class classification) using the weight θo. The new class classification output unit 122 receives an output value from the NN lower layer unit 110s based on data of the new class, and outputs a class classification based on data of the new class (hereinafter referred to as the new class classification) using the weight θn.

[0028] The first loss calculation unit 130s inputs the base class classification and the new class classification from the NN upper layer unit 120s, calculates a knowledge distillation loss Lkd based on the base class classification, and calculates a cross-entropy loss Llce based on the new class classification.

[0029] The first update unit 140s inputs the knowledge distillation loss Lkd and the cross-entropy loss Llce from the first loss calculation unit 130s and updates the weights θs, θo, and θn based on the loss obtained by adding together the knowledge distillation loss Lkd and the cross-entropy loss Llce. In updating the weights θs, θo, and θn, the weights θs, θo, and θn of the NN lower layer unit 110s are updated so as to reduce the sum of the knowledge distillation loss Lkd and the cross-entropy loss Llce. As a calculation method for the loss of classification in the first loss calculation unit 130s and an update method in the first update unit 140s, for example, the method described in Non-Patent Document 2 is used.

[0030] The series of processes in the split phase are repeatedly executed according to one or more epoch numbers defined as hyperparameters.

[0031] In the bridge phase, the purpose is to learn integrated knowledge for classifying all past classes and new classes of the current incremental task, and integrated knowledge is learned for the weights θs, θo, and θn updated in the split phase. In the bridge phase, nodes that were not connected between the basic class classification output unit 121 and the new class classification output unit 122 in Figure 2 are connected, and learning is performed in a fully connected normal NN state.

[0032] 4 is a functional block diagram for explaining the configuration of a conventional machine learning device 100 used in the bridge phase of SaB. Here, overlapping points with the explanation of the configuration of a conventional machine learning device 100 used in the split phase of SaB will be omitted as appropriate, and only the differences will be explained.

[0033] The conventional machine learning device 100 includes a second trained NN 32b that has been trained in the split phase, a second loss calculation unit 130b, and a second update unit 140b. In the bridge phase, the second trained NN 32b uses, as initial values, the weights of the classifier that have been trained in the first trained NN 32s, i.e., the weights θs, θo, and θn updated by the first update unit 140s in the split phase. The second trained NN 32b includes a NN lower layer unit 110b that uses the weight θs updated in the split phase, and a NN upper layer unit 120b that uses a weight θp that combines the weights θo and θn updated in the split phase.

[0034] The second trained NN 32b receives base class data and new class data as input and outputs a class classification based on the base class data and new class data using weights θs and θp (hereinafter referred to as integrated class classification). The data input to the second trained NN 32b is the same data as that used in the split phase. The second trained NN 32b has the same number of layers and nodes as the first trained NN 32s, and corresponds to a configuration in which all nodes in adjacent layers in the base class classification output unit 121 and new class classification output unit 122 of the first trained NN 32s are connected. The NN lower layer unit 110b of the second trained NN 32b has the same number of layers and nodes as the NN lower layer unit 110s of the first trained NN 32s. The NN upper layer unit 120b of the second trained NN 32b has the same number of layers and nodes as the NN upper layer unit 120s of the first trained NN 32s, and corresponds to a configuration in which all of the nodes in adjacent layers in the basic class classification output unit 121 and the new class classification output unit 122 of the first trained NN 32s are connected together. Therefore, the NN upper layer unit 120b of the second trained NN 32b corresponds to a configuration in which the basic class classification output unit 121 and the new class classification output unit 122 of the NN upper layer unit 120s of the first trained NN 32s are integrated.

[0035] The second loss calculation unit 130b inputs the integrated class classification from the second trained NN 32b, calculates the knowledge distillation loss Lkd and the cross-entropy loss Lce based on the integrated class classification, and calculates the sum of the knowledge distillation loss Lkd and the cross-entropy loss Lce as the class classification loss. The sum of the knowledge distillation loss Lkd and the cross-entropy loss Lce in the bridge phase is an example of a class classification loss.

[0036] The second update unit 140b updates the weights θs and θp of the second trained NN 32b based on the classification loss. For example, the second update unit 140b inputs the classification loss from the second loss calculation unit 130b and updates the weights θs and θp based on the classification loss. In this update of the weights θs and θp, the weights θs, θp of the second trained NN 32b are updated so as to reduce the classification loss.

[0037] The series of processes in the bridge phase are repeatedly executed according to one or more epoch numbers defined as hyperparameters.

[0038] Conventional SaB assumes CIL and uses big data, i.e., a large number of samples, for new classes in incremental tasks.

[0039] Next, an embodiment of the present invention will be described. In conventional techniques, some images of the base class are used as rehearsal data during incremental learning, and these images are selected randomly. In this embodiment, linguistic features containing the visual concepts of the base class are generated, and images with features close to the linguistic features are selected as images for the base class (rehearsal data).

[0040] FIG. 5 is a diagram for explaining a method for selecting a basic class image in this embodiment.

[0041] The Image Encoder 300 and Text Encoder 310 in Fig. 5 use a trained model that has been fully trained using big data that pairs images with text that describes the images, as described in Non-Patent Document 3, for example. Multiple images of a basic class are passed through the Image Encoder 300 to obtain image features for each image. Furthermore, text that describes the images of the basic class is passed through the Text Encoder 310 to obtain language features. Here, the formats of the obtained image features and language features are matched so that they can be projected into the same feature space 320.

[0042] The text describing the base class image describes the visual concept of the base class image, for example, "This bird is gray with white and has a short beak."

[0043] In the feature space 320, images having image features 340 near linguistic features 330 containing the visual concept of the base class are selected and used as images (rehearsal data) of the base class during incremental learning.

[0044] According to this embodiment, an image having a feature amount that represents the visual concept of a base class can be used during incremental learning, thereby enabling effective learning of the base class.

[0045] 6 is a functional configuration diagram of the machine learning device 200 of this embodiment in the split phase. The machine learning device 200 of this embodiment includes a NN lower layer unit 110s, a base class classification output unit 121, a new class classification output unit 122, a first loss calculation unit 130s, a first update unit 140s, a language feature output unit 210, an image feature output unit 220, and a base class image selection unit 230.

[0046] The NN lower layer unit 110s, the base class classification output unit 121, the new class classification output unit 122, the first loss calculation unit 130s, and the first update unit 140s of the machine learning device 200 of this embodiment in Figure 6 correspond to the NN lower layer unit 110s, the base class classification output unit 121, the new class classification output unit 122, the first loss calculation unit 130s, and the first update unit 140s of the conventional machine learning device 100 in Figure 3, respectively.

[0047] The machine learning device 200 of this embodiment differs from the conventional machine learning device 100 in that, in addition to the configuration of the conventional machine learning device 100, it further includes a language feature output unit 210, an image feature output unit 220, and a base class image selection unit 230.

[0048] The language feature output unit 210 receives an input of text describing a base class image, extracts language features from the text describing the base class image using a trained model trained from the image and the text describing the image, and supplies the language features to the base class image selection unit 230. The language feature output unit 210 corresponds to the Text Encoder 310 in FIG. 5, for example.

[0049] The image feature output unit 220 receives input of all base class images, extracts image features of each base class image using a trained model trained from images and text describing the images, and supplies the image features of each image to the base class image selection unit 230. The image feature output unit 220 corresponds to the image encoder 300 in FIG. 5, for example.

[0050] The base class image selection unit 230 receives input of language features, image features of each image, and all base class images, selects base class images having image features that are within a predetermined threshold distance from the language features as rehearsal data, and supplies the selected base class images to the NN lower layer unit 110s.

[0051] The NN lower layer unit 110s receives an input of a new class image and an input of a base class image selected as rehearsal data, and outputs values ​​for the base class image and the new class image using a weight θs. The NN lower layer unit 110s supplies an output based on the base class image and an output based on the new class image to a base class classification output unit 121, and supplies an output based on the new class image to a new class classification output unit 122.

[0052] The base class classification output unit 121 receives the output based on the base class image and the output based on the new class image from the NN lower layer unit 110s, outputs the base class classification using the weight θo, and supplies it to the first loss calculation unit 130s.

[0053] The new class classification output unit 122 receives an output based on the new class image from the NN lower layer unit 110s as input, outputs a new class classification using the weight θn, and supplies it to the first loss calculation unit 130s.

[0054] The first loss calculation unit 130s receives input of the base class classification and the new class classification, calculates a knowledge distillation loss based on the base class classification, calculates a cross-entropy loss based on the new class classification, and supplies the results to the first update unit 140s.

[0055] The first update unit 140s updates the weights θs, θo, and θn so as to reduce the sum of the knowledge distillation loss and the cross-entropy loss.

[0056] 7 is a functional configuration diagram of the machine learning device 200 of this embodiment in the bridge phase. The machine learning device 200 of this embodiment includes a NN lower layer unit 110b, a class classification output unit 120b, a second loss calculation unit 130b, a second update unit 140b, a language feature output unit 210, an image feature output unit 220, and a base class image selection unit 230.

[0057] The NN lower layer unit 110b, the class classification output unit 120b, the second loss calculation unit 130b, and the second update unit 140b of the machine learning device 200 of this embodiment in Figure 7 correspond to the NN lower layer unit 110b, the NN upper layer unit 120b, the second loss calculation unit 130b, and the second update unit 140b of the conventional machine learning device 100 in Figure 4, respectively.

[0058] The operations of the language feature output unit 210, the image feature output unit 220, and the base class image selection unit 230 are the same as those in the split phase of FIG. 6, and therefore will not be described here.

[0059] The NN lower layer unit 110b receives an input of a new class image and an input of a base class image selected as rehearsal data, and outputs values ​​for the base class image and the new class image using a weight θs. The NN lower layer unit 110b supplies an output based on the base class image and an output based on the new class image to the classification output unit 120b.

[0060] The class classification output unit 120b receives as input the output based on the base class image and the output based on the new class image from the NN lower layer unit 110b, and outputs the base class classification and the new class classification using a weight θp obtained by integrating the weights θo and θn updated in the split phase, and supplies them to the second loss calculation unit 130b.

[0061] The second loss calculation unit 130b calculates the knowledge distillation loss and the cross-entropy loss based on the integrated classification obtained by integrating the outputs of the basic classification and the new classification, and supplies the results to the second update unit 140b.

[0062] The second update unit 140b updates the weights θs and θp so as to reduce the sum of the knowledge distillation loss and the cross-entropy loss.

[0063] According to the machine learning device 200 of this embodiment, effective class incremental learning is possible by generating linguistic features that include the visual concept of a base class and selecting images that have features close to the linguistic features as images (rehearsal data) for the base class.

[0064] The various processes of the machine learning device 200 described above can be realized not only as a device using hardware such as a CPU and memory, but also as firmware stored in a ROM (read-only memory) or flash memory, or software for a computer, etc. The firmware program or software program can be provided by being recorded on a computer-readable recording medium, or can be transmitted and received with a server via a wired or wireless network, or can be transmitted and received as data broadcasting via terrestrial or satellite digital broadcasting.

[0065] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of the components and treatment processes, and that such modifications are also within the scope of the present invention.

[0066] The present invention can be used in machine learning techniques.

[0067] 1 Dataset, 10 Base class dataset, 15 Base class rehearsal data, 20 New class dataset, 30 Pre-training module, 32 NN, 110 NN lower layer section, 120 NN upper layer section, 121 Base class classification output section, 122 New class classification output section, 130 Loss calculation section, 140 Update section, 100, 200 Machine learning device, 210 Language feature output section, 220 Image feature output section, 230 Base class image selection section.

Claims

1. The neural network upper layer section includes: a language feature output section which receives as input a text describing a base class image and outputs a language feature; an image feature output section which receives as input a base class image and outputs an image feature; a base class image selection section which receives as input the language feature, the image feature, and the base class image, and selects the base class image corresponding to the image feature having a distance from the language feature equal to or less than a predetermined threshold; a pre-trained neural network, the neural network upper layer section including: a base class image selected by the base class image selection section and a new class image, and outputs a value; an upper layer section of the neural network disposed on the output side of the lower layer section of the neural network, the base class classification output section which receives as input an output value of the lower layer section of the neural network based on the base class image and the new class image, and outputs a base class classification which is a class classification based on the base class image and the new class image; and a new class classification output section which receives as input an output value of the lower layer section of the neural network based on the new class image, and outputs a new class classification which is a class classification based on the new class image; a loss calculation unit that calculates a loss of the base class classification and a loss of the new class classification based on the base class classification and the new class classification; and an update unit that updates weights of a lower layer section of the neural network, weights of the base class classification output section, and weights of the new class classification output section based on a loss obtained by adding together the loss of the base class classification and the loss of the new class classification.

2. The machine learning device described in claim 1, characterized in that the language feature output unit and the image feature output unit have each completed pre-learning using images of the base class and text describing the images of the base class as input.

3. The neural network includes: a language feature output unit that receives as input a text describing a base class image and outputs a language feature; an image feature output unit that receives as input a base class image and outputs an image feature; a base class image selection unit that receives as input the language feature, the image feature, and the base class image, and selects the base class image corresponding to the image feature that has a distance from the language feature that is equal to or less than a predetermined threshold; a pre-trained neural network, the neural network having a lower layer unit that receives as input a base class image and a new class image selected by the base class image selection unit, and outputs a value; and an upper layer unit of the neural network that is disposed on the output side of the lower layer unit of the neural network, the upper layer unit receiving as input an output value of the lower layer unit based on the base class image and the new class image, and outputting a base class classification or a new class classification that is a class classification based on the base class image and the new class image; and a loss calculation unit that calculates a loss of the base class classification and a loss of the new class classification based on the base class classification and the new class classification. and an update unit that updates weights in a lower layer of the neural network and weights in an upper layer of the neural network based on a loss obtained by adding together a loss of the basic class classification and a loss of the new class classification.

4. The machine learning device described in claim 3, characterized in that the language feature output unit and the image feature output unit have completed pre-learning using images of the base class and text describing the images of the base class as input, respectively.

5. A language feature output step of receiving a text describing a base class image as input and outputting a language feature; an image feature output step of receiving a base class image as input and outputting an image feature; a base class image selection step of receiving the language feature, the image feature and the base class image as input and selecting the base class image corresponding to the image feature having a distance from the language feature equal to or less than a predetermined threshold; a step of receiving, in a lower layer part of the neural network of a pre-trained neural network, an image of the base class selected in the base class image selection step and an image of a new class as input and outputting a value; a step of receiving, in an upper layer part of the neural network arranged on the output side of the lower layer part of the neural network, an output value of the lower layer part of the neural network based on the image of the base class and the image of the new class, outputting a base class classification which is a class classification based on the image of the base class and the image of the new class, and a new class classification output unit of an upper layer part of the neural network arranged on the output side of the lower layer part of the neural network, an output value of the lower layer part of the neural network based on the image of the new class, and outputting a new class classification which is a class classification based on the image of the new class; a loss calculation step of calculating a loss of the base class classification and a loss of the new class classification based on the base class classification and the new class classification; and an update step of updating weights of the lower layer section of the neural network, weights of the base class classification output section, and weights of the new class classification output section based on a loss obtained by adding together the loss of the base class classification and the loss of the new class classification.

6. A language feature output step of receiving as input a text describing a base class image and outputting a language feature; an image feature output step of receiving as input an image of a base class and outputting an image feature; a base class image selection step of receiving as input the language feature, the image feature and an image of the base class and selecting an image of the base class corresponding to the image feature having a distance from the language feature equal to or less than a predetermined threshold; a step of receiving as input, in a lower layer part of the neural network of a pre-trained neural network, an image of the base class selected in the base class image selection step and an image of a new class and outputting a value; a step of receiving, in an upper layer part of the neural network arranged on the output side of the lower layer part of the neural network, an output value of the lower layer part of the neural network based on the image of the base class and the image of the new class, and outputting a base class classification which is a class classification based on the image of the base class and the image of the new class, and a new class classification output unit of an upper layer part of the neural network arranged on the output side of the lower layer part of the neural network, an output value of the lower layer part of the neural network based on the image of the new class, and outputting a new class classification which is a class classification based on the image of the new class; A machine learning program that causes a computer to execute the following steps: a loss calculation step of calculating a loss of the base class classification and a loss of the new class classification based on the base class classification and the new class classification; and an update step of updating weights of the lower layer section of the neural network, weights of the base class classification output section, and weights of the new class classification output section based on a loss obtained by adding together the loss of the base class classification and the loss of the new class classification.

Images