Image classification apparatus, image classification method, and image classification program

Abstract

To provide an image classification technique configured to improve generalization performance and classification accuracy of an image of an additional class for additional learning using a small number of images.SOLUTION: A sentence feature output unit 30 receives a processed sentence generated by adding words to image label information and outputs sentence features. A sentence feature similarity calculation unit 40 holds sentence weight vectors of basic classes and sentence weight vectors of additional classes, the sentence weight vectors being sentence features of all basic classes and additional classes output, before continual learning, by a sentence feature output unit 30, and receives the sentence features to calculate sentence similarity. An image feature similarity calculation unit 20 holds image weight vectors of the basic classes updated in learning of the basis classes, and image weight vectors of the additional classes, the image weight vectors of the additional classes being sentence features of all additional classes output, before continual learning, by the sentence feature output unit 30, and receives the image features to calculate image similarity.SELECTED DRAWING: Figure 21

Description

[Technical Field]

[0001] The present invention relates to image classification techniques. [Background technology]

[0002] Humans can learn new knowledge through long-term experience and retain previous knowledge. However, the knowledge of deep neural networks (DNNs) using convolutional neural networks (CNNs) and other such networks is dependent on the dataset used for training. To adapt to changes in the data distribution, the DNN parameters must be retrained for the entire dataset. As DNNs learn new tasks, their estimation accuracy for previous tasks declines. Thus, continuous training of DNNs inevitably leads to catastrophic forgetting, in which the learning results of previous tasks are forgotten while learning a new task.

[0003] Incremental learning or continual learning has been proposed as a method to avoid catastrophic forgetting. Incremental learning is a learning method in which, when a new task or new data arises, the model is improved and learned, rather than learning from scratch.

[0004] Furthermore, humans can learn new knowledge from a small number of images. On the other hand, AI that uses deep learning, such as convolutional neural networks, relies on big data (a large number of images) for training. It is known that when AI that uses deep learning is trained with a small number of images, it falls into overfitting, which has good local performance but poor generalization performance.

[0005] Few-shot learning has been proposed as a method to avoid overfitting. Few-shot learning is a learning method that uses big data to learn basic knowledge for basic tasks, and then uses the basic knowledge to learn new knowledge from a small number of images for a new task.

[0006] Few shot class incremental learning is a method for solving the problems of both continuous learning and few shot learning, and is a technology that uses averaged feature vectors as weight vectors (Patent Document 1).

[0007] There is also a technique for matching feature vectors of text with feature vectors of images (Non-Patent Document 1), and a technique for generating captions for images (Non-Patent Document 2). [Prior art documents] [Non-patent literature]

[0008] [Patent Document 1] Patent application No. 2023-039351 [Non-Patent Document 1] Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PMLR, 2021. [Non-patent document 2] Mokady, Ron, Amir Hertz, and Amit H. Bermano. "Clipcap: Clip prefix for image captioning." arXiv preprint arXiv:2111.09734 (2021). Summary of the Invention [Problem to be solved by the invention]

[0009] In the prior art, there was a problem that the classification accuracy of images in additional classes was not sufficiently high when additional learning was performed using a small number of images.

[0010] The present invention has been made in light of these circumstances, and its purpose is to provide an image classification technology that can improve the generalization performance and classification accuracy of images in additional classes in additional learning with a small number of images by utilizing feature vectors of sentences. [Means for solving the problem]

[0011] In order to solve the above problem, an image classification device according to one aspect of the present invention is an image classification device that performs continuous learning, and includes an image information acquisition unit that receives an image having multiple labels as an input and acquires image label information, a sentence processing unit that receives the image label information as an input and adds words to the image label information to generate a processed sentence, a sentence feature output unit that receives the processed sentence as an input and outputs sentence features, and a processing unit that uses all of the sentence features of basic classes and additional classes output by the sentence feature output unit before the continuous learning as sentence weight vectors, stores the sentence weight vectors of the basic classes and the sentence weight vectors of the additional classes, and receives the sentence features as input and calculates sentence similarity. an image feature output unit that takes the image as an input and outputs image features; an image feature similarity calculation unit that uses the sentence features of all additional classes output by the sentence feature output unit before continued learning as image weight vectors of the additional classes, holds the image weight vectors of the base classes updated in learning the base classes and the image weight vectors of the additional classes, takes the image features as an input, and calculates image similarities; and a learning unit that uses the sentence similarities and the image similarities as inputs, performs loss calculations, and updates the image weight vectors of the base classes and the image weight vectors of the additional classes in the image feature similarity calculation unit.

[0012] Another aspect of the present invention is an image classification method that performs continuous learning, and includes an image information acquisition step that receives an image having multiple labels as an input and acquires image label information, a sentence processing step that receives the image label information as an input and adds words to the image label information to generate a processed sentence, a sentence feature output step that receives the processed sentence as an input and outputs sentence features, and a sentence feature output step that uses all of the sentence features of the base classes and additional classes output before the continuous learning as sentence weight vectors, holds the sentence weight vectors of the base classes and the sentence weight vectors of the additional classes, and receives the sentence features as an input and calculates sentence similarity. an image feature output step of taking the image as an input and outputting image features; an image feature similarity calculation step of using all of the sentence features of additional classes output by the sentence feature output step before continued learning as image weight vectors of the additional classes, holding the image weight vectors of the base classes updated in learning the base classes and the image weight vectors of the additional classes, taking the image features as an input and calculating image similarities; and a learning step of performing loss calculations using the sentence similarities and the image similarities as inputs and updating the image weight vectors of the base classes and the image weight vectors of the additional classes.

[0013] Any combination of the above components, and any transformation of the present invention into a method, device, system, recording medium, computer program, etc., are also valid aspects of the present invention. [Effects of the Invention]

[0014] According to the present invention, it is possible to provide an image classification technique that can improve the generalization performance and classification accuracy of images in an additional class through additional learning using a small number of images. [Brief explanation of the drawings]

[0015] [Figure 1] 10 is a flowchart illustrating a learning procedure for a basic class and an additional class. [Figure 2] 1 is a configuration diagram of an image classification device according to a first embodiment. [Figure 3] FIG. 1 is a diagram illustrating the configuration of an image classification device during learning of a base class. [Figure 4] 10 is a flowchart illustrating a learning procedure for a basic class. [Figure 5] FIG. 10 is a diagram illustrating the configuration of an image classification device when learning an additional class. [Figure 6] 10 is a flowchart illustrating a learning procedure for an additional class. [Figure 7] FIG. 1 is a diagram illustrating the configuration of an image classification device during inference. [Figure 8] 10 is a flowchart illustrating an inference procedure. [Figure 9] FIG. 10 is a diagram illustrating a configuration of an image classification device according to a first modification of the first embodiment. [Figure 10] 10 is a flowchart illustrating a procedure for learning a basic class in the first modification of the first embodiment. [Figure 11] FIG. 10 is a diagram illustrating the configuration of an image classification device according to a second modification of the first embodiment. [Figure 12] 10 is a flowchart illustrating a procedure for learning a basic class in the second modification of the first embodiment. [Figure 13] FIG. 1 is a diagram illustrating all combinations of M sentences and N images. [Figure 14] FIG. 10 is a diagram illustrating the configuration of an image classification device according to a third modification of the first embodiment. [Figure 15] 13 is a flowchart illustrating a learning procedure for a basic class in Modification 3 of Embodiment 1. [Figure 16] FIG. 10 is a diagram illustrating the configuration of an image classification device according to a second embodiment. [Figure 17] 11 is a flowchart showing a learning procedure for a basic class according to the third embodiment. [Figure 18] 18(a) to 18(f) are diagrams showing examples of a composite image generated by the image synthesis unit by synthesizing images of class i and class k. [Figure 19] FIG. 10 is a diagram illustrating a configuration of an image classification device according to a first modification of the second embodiment. [Figure 20] 10 is a flowchart illustrating a procedure for learning a basic class in the first modification of the second embodiment. [Figure 21] FIG. 10 is a diagram illustrating the configuration of an image classification device according to a third embodiment. [Figure 22] 11 is a flowchart illustrating a learning procedure for a basic class according to the third embodiment. [Figure 23] FIG. 10 is a diagram illustrating the configuration of an image classification device according to a first modification of the third embodiment. [Figure 24] 13 is a flowchart illustrating a procedure for learning a basic class in the first modification of the third embodiment. [Figure 25] FIG. 10 is a diagram illustrating the configuration of an image classification device according to a second modification of the third embodiment. [Figure 26] 13 is a flowchart illustrating a procedure for learning a basic class in Modification 2 of Embodiment 3. DETAILED DESCRIPTION OF THE INVENTION

[0016] First, we describe the basic training dataset and the additional training dataset.

[0017] The basic training dataset is a supervised dataset that contains a large number of basic classes (e.g., about 100 to 1000 classes), and each basic class is composed of a large number of training images (e.g., 500 to 3000 images). The basic training dataset is assumed to have a sufficient amount of data for learning a general classification task alone. Here, the number of basic classes is assumed to be 60, and the number of training images for each basic class is assumed to be 500. Each of the 60 basic classes is assumed to be assigned a class number and a label. For example, the relationship between class numbers and labels is as follows: Class Number: Label 1: Dog 2: Cat 3: Bird

[0018] In contrast, an additional training dataset is a supervised dataset that includes a small number of additional classes (e.g., 1 to 10 classes), with each additional class consisting of a small number of training images (e.g., 1 to 10 images). Although a small number of images is used here, a large number of images is also acceptable as long as the class is small. Here, the number of additional classes is set to 5, and the number of training images for each additional class is set to 5. Similar to the base classes, the additional classes are assigned class numbers and labels.

[0019] The class names of the base class and additional classes are expressed as text.

[0020] In class addition learning, the classes that are learned first are called base classes (known classes), and the classes that are added later (unknown classes) are called additional classes.

[0021] Figure 1 is a flowchart explaining the learning procedure for base classes and additional classes. During base class learning, the base classes are learned using a basic training data set (S10). During additional class learning, additional classes are additionally learned in each additional session using an additional training data set to the classes learned up to that point (S20).

[0022] Inference can be performed after training the base class or after completing an additional session. In the classification (inference) stage, the base class and all additional classes that have been trained can be simultaneously classified.

[0023] This embodiment is assumed to be supervised learning, in which a sentence with a label (class name) and an image are input during learning, and an image is input during classification to infer the class.

[0024] When the number of images in the additional class is small, it is difficult to improve the accuracy of the additional class, which is known as overfitting.

[0025] The purpose of the image classification device 100 of this embodiment is to improve the classification accuracy of the additional class when the number of images in the additional class is small.

[0026] (Embodiment 1) 2 is a configuration diagram of an image classification device 100 according to Embodiment 1. The image classification device 100 includes an image classification prediction model 110, a sentence classification prediction model 120, a learning unit 50, and a classification determination unit 60.

[0027] The image classification prediction model 110 includes an image feature output unit 10 and an image feature similarity calculation unit 20. The image classification prediction model 110 outputs an image cosine similarity as a prediction of the class classification of an input image.

[0028] The image feature output unit 10 is configured with a deep neural network and calculates an image feature vector (image feature) of the input image. The image feature output unit 10 outputs the image feature vector to the image feature similarity calculation unit 20.

[0029] The weight vectors of the deep neural network of the image feature output unit 10 are fixed after being fine-tuned using the data set of the base class.

[0030] Here, ResNet-18 is used as the image feature output unit 10. The image feature output unit 10 is not limited to ResNet-18, and may be a deep neural network such as ViT (Vision Transformer), ResNet-50, ResNet-101, or ResNet-50x4.

[0031] The image feature similarity calculation unit 20 is configured with a fully connected layer, and the fully connected layer has weight vectors for multiple classes. That is, a weight vector exists for each class.

[0032] The image feature similarity calculation unit 20 calculates image cosine similarity, which is the feature similarity between the input image feature vector and the weight vector of each class.

[0033] The image feature similarity calculation unit 20 outputs the image cosine similarity to the classification determination unit 60 and the learning unit 50 .

[0034] The weight vectors of the fully connected layer of the image feature similarity calculation unit 20 include both the base class and the additional class. The weight vectors of each class are fixed after learning.

[0035] The text classification prediction model 120 includes a text feature output unit 30 and a text feature similarity calculation unit 40. The text classification prediction model 120 outputs a text cosine similarity as a prediction of the class classification of the input text.

[0036] The text feature output unit 30 is configured with a deep neural network and calculates a text feature vector (text feature) of an input text. An example of the input text is a class name.

[0037] The sentence feature output unit 30 outputs the sentence feature vector of the class name to the sentence feature similarity calculation unit 40. The sentence feature similarity calculation unit 40 holds the sentence feature vector of the class name input from the sentence feature output unit 30 as a weight vector (sentence weight) of the class.

[0038] Furthermore, the sentence feature output unit 30 outputs the sentence feature vector of the class name to the image feature similarity calculation unit 20. The image feature similarity calculation unit 20 holds the sentence feature vector of the class name input from the sentence feature output unit 30 as the initial value of the weight vector (image weight) of the class.

[0039] In Non-Patent Document 1, the image feature output unit (ImageEncoder) and the text feature output unit (TextEncoder) are trained so that they can be matched in feature space. That is, the text feature output unit 30 is trained so that the text feature vector of a class name and the image feature vector of an image belonging to that class can be matched in feature space.

[0040] In Non-Patent Document 1, training is performed on an extremely large dataset consisting of 400 million images and their description text. Therefore, it is known that the text feature output unit can output appropriate features for a variety of texts, and can obtain text features similar to those obtained when only the class name is used, even for texts containing class names. The text feature output unit in Non-Patent Document 1 corresponds to the text feature output unit 30 in this embodiment.

[0041] The image feature output unit and the text feature output unit in Non-Patent Document 1 are said to be good at classifying conceptual classes, but not good at classifying detailed classes.

[0042] The image feature output unit in Non-Patent Document 1 is generally a very large-scale neural network, so it is preferable to use it in a cloud or a data center, and it is not preferable to use it on a terminal (edge).

[0043] Therefore, in this embodiment, the accuracy of class classification is improved by utilizing the conceptual classification ability of the text feature output unit, which is smaller in scale than the image feature output unit in Non-Patent Document 1 and therefore is expected to be used at the edge. Conceptual classification ability means the ability to perform classification with high generalization performance.

[0044] The text feature similarity calculation unit 40 is configured with a fully connected layer, and the fully connected layer has weight vectors for multiple classes.

[0045] The text feature similarity calculation unit 40 calculates text cosine similarity, which is feature similarity, from the input text feature vector and weight vectors of multiple classes.

[0046] The text feature similarity calculation unit 40 outputs the text cosine similarity to the learning unit 50.

[0047] The learning unit 50 learns (optimizes) the image feature output unit 10 and the image feature similarity calculation unit 20. The detailed operation of the learning unit will be described later.

[0048] The classification determination unit 60 outputs the class having the maximum similarity from the input image cosine similarities as the classification result.

[0049] [Basic class learning] Learn the basic classes before learning more.

[0050] Fig. 3 is a diagram illustrating the configuration of image classification device 100 during base class training. Fig. 4 is a flowchart illustrating the base class training procedure. Hereinafter, the flow of base class training by image classification device 100 according to embodiment 1 will be described with reference to Figs. 3 and 4.

[0051] First, it is assumed that the image feature output unit 10 has been trained on an image dataset with a sufficient amount of data, such as ImageNet, before the basic class dataset. Here, it is assumed that the image feature output unit 10 has been trained in advance, but it does not have to be trained.

[0052] The class names of all basic classes are input to the text feature output unit 30 to obtain text feature vectors of all basic classes (S100).

[0053] The text feature vectors of all basic classes are set as text weight vectors of the text feature similarity calculation unit 40 (S101). A weight vector exists for each class.

[0054] The text feature vectors of all basic classes are set as image weight vectors of the image feature similarity calculation unit 20 (S102).

[0055] The image feature output unit 10 and the image feature similarity calculation unit 20 are fine-tuned using a basic training data set including basic classes.

[0056] The number of epochs and batch size for the base class are assumed to be set in advance. Here, the number of epochs for the base class is 200 and the batch size is 128, but this is not limited to these. The number of batches (iterations) is a natural number obtained by dividing the total number of images of all classes in the base class by the batch size and rounding up the decimal points.

[0057] Data for the batch size is acquired from the basic training dataset (S109). A set of images and class names for the batch size is obtained. Hereinafter, processing will be performed in batch size units.

[0058] The text feature output unit 30 calculates a text feature vector from the class name and outputs it to the text feature similarity calculation unit 40 (S111).

[0059] The text feature similarity calculation unit 40 calculates the text cosine similarity and outputs it to the learning unit 50 (S112).

[0060] An image is input to the image feature output unit 10 .

[0061] The image feature output unit 10 calculates an image feature vector of the image and outputs it to the image feature similarity calculation unit 20 (S114).

[0062] The image feature amount similarity calculation unit 20 calculates the image cosine similarity and outputs it to the learning unit 50 (S115).

[0063] The learning unit 50 uses the text cosine similarity and the image cosine similarity to optimize the weight vectors of the image feature output unit 10 and the image feature similarity calculation unit 20 (S116).

[0064] For one training image, a sentence and an image are input to the learning unit 50. The learning unit 50 learns about the combination of the sentence and the image.

[0065] The learning unit 50 calculates the cross-entropy loss between the image cosine similarity and the correct label (class name) of the input image.

[0066] Furthermore, the learning unit 50 calculates the distillation loss so that the distribution of the image cosine similarity and the distribution of the text cosine similarity become closer. Specifically, the distribution of the text cosine similarity is used as a training distribution, and the image feature output unit 10 and the image feature similarity calculation unit 20 are trained so that the distribution of the image cosine similarity becomes closer to the training distribution.

[0067] Here, we use L2 loss as the distillation loss, but we can also use temperature softmax or KL divergence. Also, before calculating the distillation loss, we normalize the image cosine similarity and the text cosine similarity.

[0068] The learning unit 50 weights the cross-entropy loss Lce and the distillation loss Lkd, and calculates the overall loss L by adding the weighted cross-entropy loss Lce and the distillation loss Lkd. L=(1-α)*Lce+α*Lkd

[0069] α is a hyperparameter that takes a value between 0 and 1. The larger α is, the more the distribution of image cosine similarity is optimized to be closer to the distribution of text cosine similarity.

[0070] Generally, α depends on the number of images in the additional class. The larger the number of images in the additional class, the smaller α is used to emphasize the cross-entropy loss. The smaller the number of images in the additional class, the larger α is used to emphasize the distillation loss. In other words, when the number of images in the additional class is small, learning is performed to approach the teacher distribution.

[0071] The learning unit 50 optimizes the weight vectors of the image feature output unit 10 and the image feature similarity calculation unit 20 by backpropagation using an optimization method such as stochastic gradient descent (SGD) or Adam so as to minimize the overall loss.

[0072] [Additional class study] Fig. 5 is a diagram illustrating the configuration of image classification device 100 when training an additional class. Fig. 6 is a flowchart illustrating the procedure for training an additional class. Hereinafter, the flow of training an additional class by image classification device 100 according to embodiment 1 will be described with reference to Figs. 5 and 6.

[0073] From here on, we will explain the operation of additional learning, which allows the additional classes to be classified after being given additional classes. Additional learning can be performed repeatedly.

[0074] The procedure for additional learning is the same as that for the basic class in the learning procedure of Fig. 4, except for the operation of the learning unit 50. The operation of the learning unit 50 during additional learning differs from the operation during learning of the basic class in that the image feature output unit 10 is not trained.

[0075] The class names of all the additional classes are input to the text feature output unit 30 to obtain text feature vectors of all the additional classes (S800).

[0076] The text feature vectors of all the additional classes are set as text weight vectors (hereinafter sometimes simply referred to as "weight vectors") of the text feature amount similarity calculation unit 40 (S801).

[0077] The text feature vectors of all the additional classes are set as image weight vectors (hereinafter, sometimes simply referred to as "weight vectors") of the image feature amount similarity calculation unit 20 (S802).

[0078] Before the continuous learning, the processes of steps S800, S801, and S802 are performed as pre-learning. That is, the sentence feature similarity calculation unit 40 sets the sentence feature vectors of all additional classes output by the sentence feature output unit 30 before the continuous learning as sentence weight vectors of the additional classes, holds the sentence weight vectors of the base classes set in the learning of the base classes and the sentence weight vectors of the additional classes, and calculates the sentence similarity using the sentence features as input.

[0079] In addition, the image feature similarity calculation unit 20 uses the sentence feature vectors of all additional classes output by the sentence feature output unit 30 before continued learning as image weight vectors of the additional classes, holds the image weight vectors of the base classes updated in learning the base classes and the image weight vectors of the additional classes, and calculates image similarities using the image features as inputs.

[0080] In this way, the sentence feature vector is used as the initial value of the image weight vector of the image feature similarity calculation unit 20, and the image weight vector of the image feature similarity calculation unit 20 will be updated in subsequent continuous learning, thereby improving the performance of continuous learning for the additional class.

[0081] Here, the number of epochs for the additional class is set to 20, and the batch size is set to 64, but other values ​​may be used without being limited to these.

[0082] Data for the batch size is obtained from the additional training dataset (S809). A set of image data and class names for the batch size is obtained. Hereafter, processing will be performed in batch size units.

[0083] The text feature output unit 30 calculates a text feature vector from the class name and outputs it to the text feature similarity calculation unit 40 (S111).

[0084] The text feature similarity calculation unit 40 calculates the text cosine similarity and outputs it to the learning unit 50 (S112).

[0085] An image is input to the image feature output unit 10 .

[0086] The image feature output unit 10 calculates an image feature vector of the image and outputs it to the image feature similarity calculation unit 20 (S114).

[0087] The image feature amount similarity calculation unit 20 calculates the image cosine similarity and outputs it to the learning unit 50 (S115).

[0088] The learning unit 50 optimizes the weight vector of the image feature amount similarity calculation unit 20 using the text cosine similarity and the image cosine similarity (S816).

[0089] [Inference behavior] Fig. 7 is a diagram illustrating the configuration of the image classification device 100 during inference. Fig. 8 is a flowchart illustrating the inference procedure. Hereinafter, the flow of inference by the image classification device 100 of this embodiment will be described with reference to Figs. 7 and 8.

[0090] The image feature output unit 10 calculates an image feature vector of the input image and outputs it to the image feature similarity calculation unit 20 (S200).

[0091] The image feature amount similarity calculation unit 20 calculates image cosine similarity, which is the feature similarity between the input image feature vector and the weight vectors of multiple classes, and outputs it to the classification determination unit 60 (S201).

[0092] The classification determination unit 60 selects the class having the maximum feature similarity from the input image cosine similarities (S202).

[0093] As described above, the image classification device 100 of this embodiment can prevent forgetting of the base class by fixing the weight vectors of the image feature output unit 10 trained with the base class. This is because, when there are only a few images of the additional class, there is a high possibility of overfitting occurring if the weight vectors of the image feature output unit 10 are trained. Therefore, when there are only a few images of the additional class, to prevent forgetting, only the image feature similarity calculation unit 20 is trained. Note that, although the weight vectors of the image feature output unit 10 are fixed during training of the additional class here, even during training of the additional class, it is also possible to fix the weight vectors of layers far from the output layer of the image feature output unit 10 and train the weight vectors of layers closer to the output layer. For example, if the image feature output unit 10 is Resnet-18, among the convolutional layers CONV-1 (output feature map size: 112x112), CONV-2 (output feature map size: 56x56), CONV-3 (output feature map size: 28x28), CONV-4 (output feature map size: 14x14), and CONV-5 (output feature map size: 7x7), only CONV-5 may be the learning target, or only CONV-4 and CONV-5 may be the learning targets.

[0094] The sentence feature output unit 30 is trained in advance with an extremely large amount of data so that the sentence feature vector of a class name and the image feature vector of an image belonging to that class can be matched in feature space. Because the sentence feature output unit 30 is trained with an extremely large amount of data, it has high generalization performance. However, it is generally the image feature output unit 10 that is trained with an extremely large data set together with the sentence feature output unit 30 that can perform matching in feature space.

[0095] In this embodiment, in learning the basic classes, sentences with class names are used, and the tendency of the distribution of cosine similarity of the sentence feature output unit 30 with high generalization performance is used as a teacher to distill into the image feature output unit 10 and the image feature similarity calculation unit 20, while training images can be used to learn so that specific class classification can be performed by cross-entropy loss. This makes it possible to match the output of the image feature output unit 10 and the output of the sentence feature output unit 30 in feature space, thereby improving the generalization performance and classification accuracy of the image feature output unit 10 and the image feature similarity calculation unit 20.

[0096] Similarly, in learning additional classes, the generalization performance and classification accuracy of the image feature similarity calculation unit 20 can be improved by using sentences of class names and training images.

[0097] Furthermore, when learning additional classes with a small number of images, the generalization performance and classification accuracy of the image feature similarity calculation unit 20 can be improved by fixing the weight vector of the image feature output unit 10 and using the knowledge of the basic class as is.

[0098] (First Modification of First Embodiment) FIG. 9 is a diagram illustrating the configuration of an image classification device 100 according to a first modification of the first embodiment. FIG. 10 is a flowchart illustrating the procedure for learning the base class of the first modification of the first embodiment. Differences from the first embodiment will be explained using FIGS. 9 and 10. The difference from the first embodiment is that a sentence processing unit 70 is added in the first modification of the first embodiment. The learning method for the base class and the learning method for the additional class are the same as those in the first embodiment.

[0099] The sentence processing unit 70 generates one or more processed sentences from one input class name. The processed sentences include the class name.

[0100] For example, if the class name is "dog", sentences including the class name such as "dog", "this is a dog", "this image is a dog", "this is a photo of a dog", "that is a dog", etc. are generated. The sentences generated by the sentence processing unit 70 are called processed sentences. The processed sentences may include words related to images such as image, photograph, picture, etc.

[0101] The text processing unit 70 outputs the text including the class name to the text classification prediction model 120.

[0102] [Basic Class Learning] Steps S100, S101, and S102 are the same as those in the first embodiment shown in FIG.

[0103] The class name is input to the text processing unit 70 .

[0104] The text processing unit 70 generates a processed text from the class name and outputs it to the text feature output unit 30 (S110).

[0105] The text feature output unit 30 calculates a text feature vector and outputs it to the text feature similarity calculation unit 40 (S111).

[0106] The text feature similarity calculation unit 40 calculates the text cosine similarity and outputs it to the learning unit 50 (S112).

[0107] An image is input to the image feature output unit 10 .

[0108] The image feature output unit 10 calculates an image feature vector and outputs it to the image feature similarity calculation unit 20 (S114).

[0109] The image feature amount similarity calculation unit 20 calculates the image cosine similarity and outputs it to the learning unit 50 (S115).

[0110] The learning unit 50 uses the text cosine similarity and the image cosine similarity to optimize the weight vectors of the image feature output unit 10 and the image feature similarity calculation unit 20 (S116).

[0111] [Additional class study] The operation of learning is the same as that of the basic class except that the learning unit 50 does not optimize the weight vector of the image feature output unit 10 but optimizes the weight vector of the image feature similarity calculation unit 20 .

[0112] As described above, in learning the basic classes, processed sentences generated by the sentence processing unit 70 are used to learn specific class classification by cross-entropy loss using training images while distilling the knowledge of the sentence feature output unit 30, which has high generalization performance, into the image feature output unit 10 and the image feature similarity calculation unit 20. This makes it possible to match the output of the image feature output unit 10 and the output of the sentence feature output unit 30 in feature space, thereby improving the generalization performance and classification accuracy of the image feature output unit 10 and the image feature similarity calculation unit 20.

[0113] Similarly, in learning the additional classes, the generalization performance and classification accuracy of the image feature similarity calculation unit 20 can be improved by using the processed sentences generated by the sentence processing unit 70 and the training images.

[0114] In the first modification of the first embodiment, the sentence feature output unit 30 may use not only the processed sentence but also the sentence itself (class name) as input. In this case, it is not essential that the sentence processing unit 70 generates the processed sentence when learning the basic class.

[0115] (Modification 2 of Embodiment 1) FIG. 11 is a diagram illustrating the configuration of an image classification device 100 according to a second modification of the first embodiment. FIG. 12 is a flowchart illustrating the learning procedure for the base class in the second modification of the first embodiment. Differences from the first modification of the first embodiment will be explained using FIGS. 11 and 12. The difference from the first modification of the first embodiment is that an image processing unit 80 is further added. The configuration when learning the base class and the configuration when learning the additional class are also the same.

[0116] The image processing unit 80 processes the input image to generate one or more processed images.

[0117] For example, the input image is subjected to random image transformations such as horizontal flip, vertical flip, affine transformation (rotation), brightness conversion, resizing, and clipping. The image generated by the image processing unit 80 in this manner is defined as a processed image.

[0118] The image processing unit 80 outputs the processed image to the image classification prediction model 110.

[0119] [Basic Class Learning] Steps S100, S101, and S102 are the same as those in the first embodiment shown in FIG.

[0120] The class name is input to the text processing unit 70 .

[0121] The text processing unit 70 generates a processed text from the class name and outputs it to the text feature output unit 30 (S110).

[0122] The text feature output unit 30 calculates a text feature vector and outputs it to the text feature similarity calculation unit 40 (S111).

[0123] The text feature similarity calculation unit 40 calculates the text cosine similarity and outputs it to the learning unit 50 (S112).

[0124] An image is input to the image processing unit 80 .

[0125] The image processing unit 80 generates a processed image from the input image and outputs it to the image feature output unit 10 (S113).

[0126] The image feature output unit 10 calculates an image feature vector and outputs it to the image feature similarity calculation unit 20 (S114).

[0127] The image feature amount similarity calculation unit 20 calculates the image cosine similarity and outputs it to the learning unit 50 (S115).

[0128] The learning unit 50 uses the text cosine similarity and the image cosine similarity to optimize the weight vectors of the image feature output unit 10 and the image feature similarity calculation unit 20 (S116).

[0129] The learning unit 50 receives input of one or more sentence cosine similarities and one or more image cosine similarities for one training image. The learning unit 50 learns about all combinations of one or more sentences or processed sentences and one or more images or processed images. FIG. 13 is a diagram illustrating all combinations of M sentences and processed sentences and N images and processed images. In this example, M and N are natural numbers greater than or equal to 3. The first row of FIG. 13 shows the combination of the 0th sentence or processed sentence and the 0th image or processed image, and the second row shows the combination of the 0th sentence or processed sentence and the 1st image or processed image. Note that the learning unit 50 learning about all combinations of sentences or processed sentences and images or processed images is not limited to Variation 2 of Embodiment 1, and may be applied to other embodiments or other variations.

[0130] [Additional class study] The operation of learning is the same as that of the basic class except that the learning unit 50 does not optimize the weight vector of the image feature output unit 10 but optimizes the weight vector of the image feature similarity calculation unit 20 .

[0131] As described above, in learning the base class, the knowledge of the sentence feature output unit 30 with high generalization performance is distilled into the image feature output unit 10 and the image feature similarity calculation unit 20 using sentences processed by the sentence processing unit 70, while the image feature output unit 10 and the image feature similarity calculation unit 20 can be trained to perform specific class classification using cross-entropy loss using training images processed by the image processing unit 80. This enables the output of the image feature output unit 10 and the output of the sentence feature output unit 30 to match in feature space, thereby improving the generalization performance and classification accuracy of the image feature output unit 10 and the image feature similarity calculation unit 20.

[0132] Similarly, in learning additional classes, the generalization performance and classification accuracy of the image feature similarity calculation unit 20 can be improved by using sentences processed by the sentence processing unit 70 and training images processed by the image processing unit 80.

[0133] In the second modification of the first embodiment, the sentence feature output unit 30 may use not only the processed sentence but also the sentence itself (class name) as input. The image feature output unit 10 may use not only the processed image but also the image itself as input. In this case, sentence processing by the sentence processing unit 70 and image processing by the image processing unit 80 during learning of the basic class are not essential.

[0134] (Third Modification of First Embodiment) FIG. 14 is a diagram illustrating the configuration of an image classification device 100 according to a third modification of the first embodiment. FIG. 15 is a flowchart illustrating the procedure for learning the base class of the third modification of the first embodiment. Differences from the first modification of the first embodiment will be explained using FIGS. 14 and 15. The difference from the first modification of the first embodiment is that a non-target object obscuring unit 90 is added in the third modification of the first embodiment. The configuration for learning the base class and the configuration for learning the additional class are also the same.

[0135] The non-target object obscuring unit 90 performs processing to obscure objects (called "non-target objects") included in the input image that are unrelated to the class of the input image, and generates a non-target object obscured image.

[0136] For example, the non-target object obscuring unit 90 masks the range of the non-target object by painting it with a specific color. In addition to simply painting it with a specific color, the non-target object range may be blurred using a smoothing filter or a median filter, or may be mosaicked or noise may be added.

[0137] In this way, the non-target object obfuscation unit 90 reduces information other than the target object of the training class by completely or partially removing information other than the target object of the training class from the image by masking or blurring the range of the non-target object.

[0138] If the range of objects in the training class is specified, masking or blurring is performed on the area outside that range. If the range of objects in the training class is not specified, masking or blurring may be performed on the peripheral area of ​​the image.

[0139] [Basic Class Learning] Steps S100, S101, S102, and S109 are the same as those in the first embodiment shown in FIG.

[0140] The class name is input to the text processing unit 70 .

[0141] The text processing unit 70 generates a processed text from the class name and outputs it to the text feature output unit 30 (S110).

[0142] The text feature output unit 30 calculates a text feature vector and outputs it to the text feature similarity calculation unit 40 (S111).

[0143] The text feature similarity calculation unit 40 calculates the text cosine similarity and outputs it to the learning unit 50 (S112).

[0144] The image is input to a non-object obfuscation unit 90 .

[0145] The non-target object obscuring unit 90 generates an image in which the non-target object in the image is obscured, and outputs the image to the image feature output unit 10 (S113').

[0146] The image feature output unit 10 calculates an image feature vector and outputs it to the image feature similarity calculation unit 20 (S114).

[0147] The image feature amount similarity calculation unit 20 calculates the image cosine similarity and outputs it to the learning unit 50 (S115).

[0148] The learning unit 50 uses the text cosine similarity and the image cosine similarity to optimize the weight vectors of the image feature output unit 10 and the image feature similarity calculation unit 20 (S116).

[0149] [Additional class study] The operation of learning is the same as that of the basic class except that the learning unit 50 does not optimize the weight vector of the image feature output unit 10 but optimizes the weight vector of the image feature similarity calculation unit 20 .

[0150] As described above, in learning the base class, the image feature output unit 10 and the image feature similarity calculation unit 20 can be trained to perform specific class classification using cross-entropy loss using high-quality training images in which information other than objects has been reduced by the non-object obfuscation unit 90, while the knowledge of the sentence feature output unit 30 with high generalization performance is distilled into the image feature output unit 10 and the image feature similarity calculation unit 20 using sentences processed by the sentence processing unit 70. This enables the output of the image feature output unit 10 and the output of the sentence feature output unit 30 to match in feature space, thereby improving the generalization performance and classification accuracy of the image feature output unit 10 and the image feature similarity calculation unit 20.

[0151] Similarly, in learning additional classes, the generalization performance and classification accuracy of the image feature similarity calculation unit 20 can be improved by using sentences processed by the sentence processing unit 70 and high-quality training images in which information other than objects has been reduced by the non-object obfuscation unit 90.

[0152] In the third modification of the first embodiment, the text feature output unit 30 may use not only the processed text but also the text itself (class name) as input. The image feature output unit 10 may use not only the image in which the non-target object is obscured but also the image itself as input. In this case, it is not essential to generate the processed text by the text processing unit 70 or to reduce the image information by the non-target object obscuring unit 90 when learning the basic class.

[0153] (Embodiment 2) Fig. 16 is a diagram illustrating the configuration of an image classification device 100 according to the second embodiment. Fig. 17 is a flowchart illustrating the learning procedure of a base class according to the third embodiment. Differences from the first embodiment will be explained using Figs. 16 and 17. The difference from the first embodiment is that a sentence synthesis unit 75 and an image synthesis unit 85 are added in the second embodiment.

[0154] The sentence synthesis unit 75 receives two or more sentences relating to different classes as input, and outputs one or more synthesized sentences to the sentence feature output unit 30. For example, if the class name of class a is "dog" and the class name of class c is "cat," the sentence synthesis unit 75 generates the synthesized sentence "dog and cat." Similar to the sentence processing unit 70, the sentence synthesis unit 75 may include words related to images, such as image, photograph, and picture, in the synthesized sentence. For example, "This is a photo of a dog and a cat," or "This photo shows a dog and a cat."

[0155] Image synthesis unit 85 receives two or more images each relating to a different class as input, and outputs one synthesized image to image feature output unit 10. For example, when an image of a dog in class a and an image of a cat in class c are input, a synthesized image with the dog on the left and the cat on the right is generated.

[0156] [Basic Class Learning] Steps S100, S101, S102, and S109 are the same as those in the first embodiment shown in FIG.

[0157] The class names of two classes (class a and class c) are input to the sentence synthesis unit 75 .

[0158] Class a and class c are the classes obtained in step S109. Class c is a class different from class a and is selected according to a predetermined rule. For example, the order of classes to be combined is predetermined, and class c is the class whose class number is two numbers larger than class a. Here, α is fixed at 2, but it may be a random number. CN indicates the number of basic classes. c=(a+α)%CN

[0159] The sentence synthesis unit 75 generates a sentence including the class names of class a and class c, and outputs the generated sentence to the sentence feature output unit 30 (S210).

[0160] The text feature output unit 30 calculates a text feature vector and outputs it to the text feature similarity calculation unit 40 (S111).

[0161] The text feature similarity calculation unit 40 calculates the text cosine similarity and outputs it to the learning unit 50 (S112).

[0162] Images of two classes (class a and class c) are input to the image synthesis unit 85.

[0163] The image synthesis unit 85 randomly selects an image of class a and an image of class c, generates a synthesized image synthesized by the randomly selected synthesis method, and outputs the generated synthesized image to the image feature output unit 10 (S213). The generated synthesized image may be one or more.

[0164] It is assumed that the synthesis method is determined in advance in the image synthesis unit 85 and the text synthesis unit 75. The detailed operation of the image synthesis unit 85 will be described later.

[0165] The image feature output unit 10 calculates an image feature vector and outputs it to the image feature similarity calculation unit 20 (S114).

[0166] The image feature amount similarity calculation unit 20 calculates the image cosine similarity and outputs it to the learning unit (S115).

[0167] The learning unit 50 uses the text cosine similarity and the image cosine similarity to optimize the weight vectors of the image feature output unit 10 and the image feature similarity calculation unit 20 (S116).

[0168] [Additional class study] The operation of learning is the same as that of the basic class except that the learning unit 50 does not optimize the weight vector of the image feature output unit 10 but optimizes the weight vector of the image feature similarity calculation unit 20 .

[0169] 18(a) to 18(f) are diagrams showing examples of composite images generated by the image synthesis unit 85 by synthesizing images of class a and class c.

[0170] The image synthesis unit 85 synthesizes the images of class a and class c based on the following synthesis method: The image synthesis unit 85 may synthesize the images of class a and class c based on one of the following synthesis methods, or may synthesize the images of class a and class c based on a plurality of the following synthesis methods.

[0171] Clipping synthesis involves cutting out and synthesizing a portion of the training images for class a and class c. Figure 18(a) shows an image for class a, Figure 18(b) shows an image for class c, and Figures 18(c) to (f) show examples of clipping-synthesized images.

[0172] 18(c) shows an example in which the left half region of class a and the right half region of class c are clipped and then combined. In this case, the left half region contains an object of class a, and the right half region contains an object of class c.

[0173] 18(d) shows an example in which the upper half area of ​​class a and the lower half area of ​​class c are clipped and combined. In this case, the upper half area contains objects of class a, and the lower half area contains objects of class c.

[0174] 18(e) shows an example in which the central region of class a and the outer region of class c are clipped and then combined. In this case, the central region contains an object of class a, and the outer region contains an object of class c.

[0175] 18(f) shows that the upper left and lower right regions of class a and the upper left and lower right regions of class c are clipped and combined. In this case, the upper left and lower right regions contain objects of class a, and the upper left and lower right regions contain objects of class c.

[0176] 18(c) to 18(f) show an example in which class a and class c are clipped equally, but this is not limiting. For example, in FIG. 18(c), the left 1 / 4 region of class a and the right 3 / 4 region of class c may be clipped and then combined.

[0177] Alternatively, object detection technology can be used to detect regions containing objects of a class and then combine them. When combining, scaling or padding can be applied to the size of the region into which the detected region is combined.

[0178] Moreover, instead of clipping and combining, images reduced by scaling may be combined.

[0179] Although an example of combining two images is shown here, three or more images may be combined.

[0180] As described above, in learning the basic classes, the knowledge of the sentence feature output unit 30 with high generalization performance obtained by the sentence synthesis unit 75 is distilled into the image feature output unit 10 and the image feature similarity calculation unit 20, while the image feature output unit 10 and the image feature similarity calculation unit 20 can be trained to perform specific class classification using cross-entropy loss using training images containing objects of multiple classes obtained by the image synthesis unit 85. This enables the output of the image feature output unit 10 and the output of the sentence feature output unit 30 to match in feature space, thereby improving the generalization performance and classification accuracy of the image feature output unit 10 and the image feature similarity calculation unit 20.

[0181] Similarly, in learning additional classes, the generalization performance and classification accuracy of the image feature similarity calculation unit 20 can be improved by using sentences containing multiple class names obtained by the sentence synthesis unit 75 and training images containing multiple class objects obtained by the image synthesis unit 85.

[0182] Furthermore, it is expected that learning can be done efficiently by learning multiple classes simultaneously.

[0183] The number of composite sentences may be one or more. When there are multiple composite sentences, the feature similarity may be calculated by averaging the multiple sentence feature vectors, for example.

[0184] (Modification 1 of Embodiment 2) FIG. 19 is a diagram illustrating the configuration of an image classification device 100 according to a first modification of the second embodiment. FIG. 20 is a flowchart illustrating a learning procedure for a base class according to the first modification of the second embodiment. Differences from the second embodiment will be explained using FIGS. 19 and 20. The difference from the second embodiment is that in the first modification of the second embodiment, image synthesis information data is supplied from the image synthesis unit 85 to the sentence synthesis unit 75.

[0185] [Basic class learning] Steps S100, S101, S102, and S109 are the same as those in the first embodiment shown in FIG.

[0186] Images of two classes (class a and class c) are input to the image synthesis unit 85.

[0187] The image synthesis unit 85 generates a synthetic image by synthesizing images of two classes (class a and class c) using a synthesis method that randomly selects the images, and outputs the synthetic image to the image feature output unit 10 (S310).

[0188] The image synthesis unit 85 outputs image synthesis information indicating the selected synthesis method to the text synthesis unit 75 (S311).

[0189] For example, if class a is "dog" and class c is "cat" and the synthesis method is to place class a on the left and class c on the right, the image synthesis information data "class a on the left and class c on the right" is output.

[0190] The image feature output unit 10 calculates an image feature vector and outputs it to the image feature similarity calculation unit 20 (S114).

[0191] The image feature amount similarity calculation unit 20 calculates the image cosine similarity and outputs it to the learning unit (S115).

[0192] The text synthesis unit 75 generates a text including the class names of the two classes (class a and class c) according to the input image synthesis information, and outputs the generated text to the text feature output unit 30 (S312).

[0193] For example, if class a is "dog" and class c is "cat" and the image synthesis information is "class a on the left, class c on the right", the sentence "dog on the left, cat on the right" is generated.

[0194] The text feature output unit 30 calculates a text feature vector and outputs it to the text feature similarity calculation unit 40 (S111).

[0195] The text feature similarity calculation unit 40 calculates the text cosine similarity and outputs it to the learning unit 50 (S112).

[0196] The learning unit 50 uses the text cosine similarity and the image cosine similarity to optimize the weight vectors of the image feature output unit 10 and the image feature similarity calculation unit 20 (S116).

[0197] [Additional class study] The operation of learning is the same as that of the basic class except that the learning unit 50 does not optimize the weight vector of the image feature output unit 10 but optimizes the weight vector of the image feature similarity calculation unit 20 .

[0198] As described above, in learning the basic classes, the sentences containing class names of multiple classes obtained by the sentence synthesis unit 75 are highly consistent with the training images containing objects of multiple classes obtained by the image synthesis unit 85. Therefore, by effectively distilling the knowledge of the sentence feature output unit 30, which has high generalization performance, the generalization performance and classification accuracy of the image feature output unit 10 and the image feature similarity calculation unit 20 can be improved.

[0199] Similarly, in learning the additional class, the generalization performance and classification accuracy of the image feature similarity calculation unit 20 can be improved.

[0200] (Embodiment 3) Fig. 21 is a diagram illustrating the configuration of an image classification device 100 according to Embodiment 3. Fig. 22 is a flowchart illustrating the procedure for learning a base class according to Embodiment 3. Differences from Modification 1 of Embodiment 1 will be explained using Figs. 21 and 22. The difference from Modification 1 of Embodiment 1 is that an image information acquisition unit 95 is added in Embodiment 3.

[0201] The image information acquisition unit 95 acquires image label information from an input image having multiple labels, and outputs the image label information to the text processing unit 70. The text processing unit 70 generates a processed text from the image label information.

[0202] For example, when an image of a cat with a grassy background is input, the image information acquisition unit 95 acquires the information "Object: Cat, Background: Grassland" attached to the image as image label information. The sentence processing unit 70 generates processed sentences such as "The background is a grassy field and there is a cat," "There is a cat, and the background is a grassy field," "There is a cat in a grassy field," and "This is an image of a cat and a grassy field" from the image label information "Object: Cat, Background: Grassland."

[0203] [Basic class learning] Steps S100, S101, S102, and S109 are the same as those in the first modification of the first embodiment shown in FIG.

[0204] The image information acquisition unit 95 acquires image label information from an image having multiple labels, and outputs it to the text processing unit 70 (S400).

[0205] The text processing unit 70 generates a processed text from the image label information of the image and outputs it to the text feature output unit 30 (S410).

[0206] The text feature output unit 30 calculates a text feature vector and outputs it to the text feature similarity calculation unit 40 (S111).

[0207] The text feature similarity calculation unit 40 calculates the text cosine similarity and outputs it to the learning unit 50 (S112).

[0208] An image is input to the image feature output unit 10 .

[0209] The image feature output unit 10 calculates an image feature vector of the image and outputs it to the image feature similarity calculation unit 20 (S114).

[0210] The image feature amount similarity calculation unit 20 calculates the image cosine similarity and outputs it to the learning unit 50 (S115).

[0211] The learning unit 50 uses the sentence cosine similarity and the image cosine similarity to optimize the weight vectors of the image feature output unit 10 and the image feature similarity calculation unit 20 (S116). Note that one processed sentence may be generated from one piece of image label information, or multiple processed sentences may be generated.

[0212] [Additional class study] The operation of learning is the same as that of the basic class except that the learning unit 50 does not optimize the weight vector of the image feature output unit 10 but optimizes the weight vector of the image feature similarity calculation unit 20 .

[0213] As described above, in learning the basic class, for images with multiple labels, by using sentences that appropriately describe the images obtained by the sentence processing unit 70, the knowledge of the sentence feature output unit 30, which has high generalization performance, can be effectively distilled, and the generalization performance and classification accuracy of the image feature output unit 10 and the image feature similarity calculation unit 20 can be improved.

[0214] Similarly, in learning the additional class, the generalization performance and classification accuracy of the image feature similarity calculation unit 20 can be improved.

[0215] (Modification 1 of Embodiment 3) FIG. 23 is a diagram illustrating the configuration of an image classification device 100 according to Modification 1 of Embodiment 3. FIG. 24 is a flowchart illustrating the learning procedure for the basic classes of Modification 1 of Embodiment 3. The differences from Modification 2 of Embodiment 1 will be explained using FIGS. 23 and 24. The differences from Modification 2 of Embodiment 1 are that Modification 1 of Embodiment 3 adds an image information acquisition unit 95, and image processing information data is supplied from the image processing unit 80 to the text processing unit 70.

[0216] The image information acquisition unit 95 acquires image label information from an input image having multiple labels and outputs it to the text processing unit 70. The image processing unit 80 outputs the image processing information to the text processing unit 70. The text processing unit 70 generates a processed text from the image label information and the image processing information. Note that the number of processed texts generated may be one or multiple.

[0217] For example, when an image of a cat with a grassy background is input, the image information acquisition unit 95 acquires "Object: Cat, Background: Grassland" as image label information. The image processing unit 80 generates a processed image by rotating, flipping, etc. the input image, and outputs image processing information such as "Rotate 90 degrees" and "Flip" to the text processing unit 70. The text processing unit 70 generates a sentence such as "The image of a cat with a grassy background was rotated 90 degrees" from the image label information "Object: Cat, Background: Grassland" and the image processing information "Rotate 90 degrees." The generated sentence indicates the processing details of the image, thereby increasing the consistency between the processed sentence and the processed image.

[0218] [Basic class learning] Steps S100, S101, S102, and S109 are the same as those in the second modification of the first embodiment shown in FIG.

[0219] The image information acquisition unit 95 acquires image label information from an image having multiple labels, and outputs it to the text processing unit 70 (S400).

[0220] The image processing unit 80 outputs the image processing information of the image to the text processing unit 70 (S408).

[0221] The text processing unit 70 generates text from the image label information and image processing information of the image, and outputs the generated text to the text feature output unit 30 (S410').

[0222] The text feature output unit 30 calculates a text feature vector and outputs it to the text feature similarity calculation unit 40 (S111).

[0223] The text feature similarity calculation unit 40 calculates the text cosine similarity and outputs it to the learning unit 50 (S112).

[0224] The image processing unit 80 processes the image, generates a processed image, and outputs it to the image feature output unit 10 (S113). The number of processed images may be one or more. Furthermore, the number of sentences generated for one processed image may be one or more.

[0225] The image feature output unit 10 calculates an image feature vector and outputs it to the image feature similarity calculation unit 20 (S114).

[0226] The image feature amount similarity calculation unit 20 calculates the image cosine similarity and outputs it to the learning unit 50 (S115).

[0227] The learning unit 50 uses the text cosine similarity and the image cosine similarity to optimize the weight vectors of the image feature output unit 10 and the image feature similarity calculation unit 20 (S116).

[0228] [Additional class study] The operation of learning is the same as that of the basic class except that the learning unit 50 does not optimize the weight vector of the image feature output unit 10 but optimizes the weight vector of the image feature similarity calculation unit 20 .

[0229] (Modification 2 of Embodiment 3) FIG. 25 is a diagram illustrating the configuration of an image classification device 100 according to Modification 2 of Embodiment 3. FIG. 26 is a flowchart illustrating the learning procedure for the base classes of Modification 2 of Embodiment 3. The differences from Modification 3 of Embodiment 1 will be explained using FIGS. 23 and 24. The differences from Modification 3 of Embodiment 1 are that Modification 2 of Embodiment 3 adds an image information acquisition unit 95, and obfuscation information data is supplied from the non-target object obfuscation unit 90 to the sentence processing unit 70.

[0230] The image information acquisition unit 95 acquires image label information from an input image having multiple labels and outputs it to the sentence processing unit 70. The non-target object obfuscation unit 90 outputs obfuscation information indicating how the non-target objects in the input image were obscured to the sentence processing unit 70. The sentence processing unit 70 generates multiple sentences from the image label information and the obfuscation information.

[0231] For example, when an image of a cat with a grassy background is input, the image information acquisition unit 95 acquires "Object: Cat, Background: Grassland" as image label information. The non-target object obscuration unit 90 applies obscuration processing such as mosaic to the background of the input image to generate a non-target object obscured image, and outputs obscuration information such as "mosaic applied," "color faded," or "noise added" to the text processing unit 70. The text processing unit 70 generates a sentence such as "There is a cat in the grassy background that is mosaicked" from the image label information "Object: Cat, Background: Grassland" and the obscuration information "mosaic applied." The generated sentence represents the obscuration of the non-target object in the image, which increases the consistency between the processed sentence and the processed image and also eliminates unnecessary information, allowing for more effective learning.

[0232] [Basic class learning] Steps S100, S101, S102, and S109 are the same as those in the third modification of the first embodiment shown in FIG.

[0233] The image information acquisition unit 95 acquires image label information from an image having multiple labels, and outputs it to the text processing unit 70 (S400).

[0234] The non-target object obfuscation unit 90 outputs the image obfuscation information to the text processing unit 70 (S409).

[0235] The text processing unit 70 generates text from the image label information and obfuscation information of the image, and outputs the generated text to the text feature output unit 30 (S410'').

[0236] The text feature output unit 30 calculates a text feature vector and outputs it to the text feature similarity calculation unit 40 (S111).

[0237] The text feature similarity calculation unit 40 calculates the text cosine similarity and outputs it to the learning unit 50 (S112).

[0238] The non-target object obscuring unit 90 generates an image in which the non-target object in the image is obscured, and outputs the image to the image feature output unit 10 (S113').

[0239] The image feature output unit 10 calculates an image feature vector and outputs it to the image feature similarity calculation unit 20 (S114).

[0240] The image feature amount similarity calculation unit 20 calculates the image cosine similarity and outputs it to the learning unit 50 (S115).

[0241] The learning unit 50 uses the text cosine similarity and the image cosine similarity to optimize the weight vectors of the image feature output unit 10 and the image feature similarity calculation unit 20 (S116).

[0242] [Additional class study] The operation of learning is the same as that of the basic class except that the learning unit 50 does not optimize the weight vector of the image feature output unit 10 but optimizes the weight vector of the image feature similarity calculation unit 20 .

[0243] In the above embodiment and its modified examples, for convenience of explanation, the case of one sentence or one image has been described, but there may be multiple sentences and multiple images. Specifically, there may be multiple images for one sentence, multiple sentences for one image, or multiple sentences and multiple images. In other words, the output of the sentence processing unit 70, sentence synthesis unit 75, image processing unit 80, image synthesis unit 85, and non-target object obfuscation unit 90 may be one, or two or more.

[0244] The various processes of the image classification device 100 described above can be realized not only as a device using hardware such as a CPU and memory, but also as firmware stored in a ROM (read-only memory) or flash memory, or software for a computer, etc. The firmware program or software program can be provided by being recorded on a computer-readable recording medium, or can be transmitted and received with a server via a wired or wireless network, or can be transmitted and received as data broadcasting via terrestrial or satellite digital broadcasting.

[0245] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of the components and treatment processes, and that such modifications are also within the scope of the present invention. [Explanation of symbols]

[0246] 10 image feature output unit, 20 image feature similarity calculation unit, 30 text feature output unit, 40 text feature similarity calculation unit, 50 learning unit, 60 classification determination unit, 70 text processing unit, 75 text synthesis unit, 80 image processing unit, 85 image synthesis unit, 90 non-target object obfuscation unit, 95 image information acquisition unit, 100 image classification device, 110 image classification prediction model, 120 text classification prediction model.

Claims

1. An image classification device that performs continuous learning, an image information acquisition unit that receives an image having multiple labels as input and acquires image label information; a text processing unit that receives the image label information as an input and adds words to the image label information to generate a processed text; a sentence feature output unit that receives the processed sentence as an input and outputs sentence features; a sentence feature similarity calculation unit that sets the sentence features of all the base classes and additional classes output by the sentence feature output unit before the continuous learning as sentence weight vectors, holds the sentence weight vectors of the base classes and the sentence weight vectors of the additional classes, inputs the sentence features, and calculates sentence similarities; an image feature output unit that receives the image as an input and outputs an image feature; an image feature similarity calculation unit that sets all of the sentence features of the additional classes output by the sentence feature output unit before the continuous learning as image weight vectors of the additional classes, holds the image weight vectors of the base classes updated in the learning of the base classes and the image weight vectors of the additional classes, and uses the image features as inputs to calculate image similarities; An image classification device comprising a learning unit that performs loss calculation using the text similarity and the image similarity as input, and updates the image weight vector of the base class and the image weight vector of the additional class of the image feature similarity calculation unit.

2. an image processing unit that receives the image, processes the image, outputs a processed image, and outputs image processing information indicating how the image has been processed; the image feature output unit receives the processed image as an input and outputs an image feature; The image classification device according to claim 1, characterized in that the sentence processing unit receives the image label information and the image processing information as input, and generates a processed sentence by adding words related to the image processing information to the image label information.

3. an image processing unit that receives the image as an input, performs processing to obscure objects included in the image that are unrelated to the class of the image, to generate a non-object obscured image, and further generates obscuration information indicating what kind of obscuration processing has been performed on the image, the image feature output unit receives the non-object obscured image as an input and outputs an image feature; The image classification device according to claim 1, wherein the sentence processing unit receives the image label information and the obfuscation information as input, and generates processed sentences by adding words related to the obfuscation information to the image label information.

4. 1. An image classification method that performs continuous learning, comprising: an image information acquisition step in which an image having multiple labels is input and image label information is acquired; a sentence processing step in which the image label information is input and words are added to the image label information to generate a processed sentence; a sentence feature output step of inputting the processed sentence and outputting sentence features; a sentence feature similarity calculation step of setting the sentence features of all the basic classes and additional classes output in the sentence feature output step before the continuous learning as sentence weight vectors, retaining the sentence weight vectors of the basic classes and the sentence weight vectors of the additional classes, inputting the sentence features, and calculating sentence similarities; an image feature output step of receiving the image as an input and outputting an image feature; an image feature similarity calculation step of setting all of the sentence features of the additional classes output in the sentence feature output step before the continuous learning as image weight vectors of the additional classes, holding the image weight vectors of the base classes updated in the learning of the base classes and the image weight vectors of the additional classes, inputting the image features, and calculating image similarities; An image classification method comprising a learning step of performing a loss calculation using the text similarity and the image similarity as inputs and updating the image weight vector of the base class and the image weight vector of the additional class.

5. An image classification program that performs continuous learning, an image information acquisition step in which an image having multiple labels is input and image label information is acquired; a sentence processing step in which the image label information is input and words are added to the image label information to generate a processed sentence; a sentence feature output step of inputting the processed sentence and outputting sentence features; a sentence feature similarity calculation step of setting the sentence features of all the basic classes and additional classes output in the sentence feature output step before the continuous learning as sentence weight vectors, retaining the sentence weight vectors of the basic classes and the sentence weight vectors of the additional classes, inputting the sentence features, and calculating sentence similarities; an image feature output step of receiving the image as an input and outputting an image feature; an image feature similarity calculation step of setting all of the sentence features of the additional classes output in the sentence feature output step before the continuous learning as image weight vectors of the additional classes, holding the image weight vectors of the base classes updated in the learning of the base classes and the image weight vectors of the additional classes, inputting the image features, and calculating image similarities; a learning step of calculating a loss using the sentence similarity and the image similarity as input, and updating the image weight vector of the base class and the image weight vector of the additional class.

Images