Image classification device, image classification method and image classification program

Abstract

The image feature output unit (10) has completed pre-learning based on the text and the image, takes the image as input, and outputs the image feature quantity. The image prototype generation unit (20) calculates the image feature quantity for each category and outputs the image prototype for each category. The text feature output unit (50) has completed pre-learning based on the text and the image, takes the text describing the category as input, and outputs the text feature quantity. The similarity calculation unit (30) saves the image prototype of the basic category as the weight of the basic category and the text feature quantity of the additional category as the weight of the additional category, and calculates the similarity by taking the image feature quantity output from the image feature output unit (10) as input. The classification unit (40) takes the similarity as input and determines the classification of the image.

Description

Technical Field

[0001] This invention relates to image classification technology. Background Technology

[0002] Humans can learn new knowledge through long-term experience and retain it without forgetting old knowledge. On the other hand, the knowledge of deep neural networks (DNNs) using convolutional neural networks (CNNs) depends on the dataset used for learning. To adapt to changes in data distribution, the parameters of the DNN need to be relearned across the entire dataset. In DNNs, as new tasks are learned, the inference accuracy for old tasks decreases. Thus, in DNNs, if continuous learning occurs, catastrophic forgetting—the forgetting of learning results from old tasks while learning new ones—is unavoidable.

[0003] As a method to avoid catastrophic forgetting, incremental learning (or continuous learning) has been proposed. Continuous learning is a learning method that, when new tasks or new data are generated, instead of learning from the initial learning model, improves the currently learned model.

[0004] Furthermore, humans can learn new knowledge from a small number of images. On the other hand, artificial intelligence that utilizes deep learning methods such as convolutional neural networks relies on large amounts of data (a large number of images) used in the learning process. It is known that if artificial intelligence using deep learning is learned from a small number of images, it will fall into overfitting, resulting in good local performance but poor generalization performance.

[0005] As a method to avoid overfitting, few-shot learning has been proposed. Few-shot learning is a learning method that uses large datasets to learn basic knowledge in a basic task, and then uses that basic knowledge to learn new knowledge from a small number of images in a new task.

[0006] As a method to address the challenges of both continuous learning and few-shot learning, there is few-shot class incremental learning, which uses the averaged feature vector as the weight vector (Patent Document 1). Additionally, there is a technique for matching the feature vectors of text with the feature vectors of images (Non-Patent Document 1).

[0007] Existing technical documents Patent documents Patent Document 1: Japanese Patent Application No. 2023-039351; Non-patent literature 1: Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International Conference on machine learning.PMLR, 2021. Summary of the Invention

[0008] In prior art, there is a problem that the classification accuracy of images with additional categories is not high enough for incremental learning based on a small number of images.

[0009] This embodiment was made in view of the following situation, and its purpose is to provide an image classification technique that can improve the classification accuracy of images with additional categories based on incremental learning of a small number of images.

[0010] To address the aforementioned issues, an image classification apparatus according to one embodiment includes: an image feature output unit that has completed pre-learning based on text and images, takes an image as input, and outputs image feature quantities; an image prototype generation unit that calculates the image feature quantities for each category and outputs image prototypes for each category; a text feature output unit that has completed pre-learning based on text and images, takes a text describing the category as input, and outputs text feature quantities; a similarity calculation unit that stores the image prototypes of the basic category as weights of the basic category and the text feature quantities of the additional category as weights of the additional category, takes the image feature quantities output from the image feature output unit as input, and calculates similarity; and a classification unit that takes the similarity as input and determines the classification of the image.

[0011] Another embodiment of this method is an image classification method. This method includes: an image feature output step, where pre-learning based on text and images has been completed, the image is taken as input, and image features are output; an image prototype generation step, where the image features are calculated for each category, and image prototypes for each category are output; a text feature output step, where pre-learning based on text and images has been completed, a text describing the category is taken as input, and text feature values ​​are output; a similarity calculation step, where the image prototypes of the basic category are saved as weights for the basic category, and the text feature values ​​of the supplementary category are saved as weights for the supplementary category, and the image features output from the image feature output step are taken as input to calculate similarity; and a classification step, where the similarity is taken as input to determine the image classification.

[0012] Furthermore, any combination of the above-mentioned constituent elements, or the manifestation of this embodiment in a way that is converted between methods, apparatus, systems, recording media, computer programs, etc., is also effective as a way of this embodiment.

[0013] According to this embodiment, an image classification technique can be provided that can improve the classification accuracy of images with additional categories based on incremental learning of a small number of images. Attached Figure Description

[0014] Figure 1 This is a structural diagram of the image classification device in an embodiment.

[0015] Figure 2 This is a flowchart illustrating the incremental learning processing steps performed by the image classification apparatus of the embodiment.

[0016] Figure 3 This is a diagram illustrating an example of calculating an additional category prototype.

[0017] Figure 4 This is a diagram illustrating an example of the operation of the weight generation unit in an embodiment.

[0018] Figure 5 This is a flowchart illustrating the classification process performed by an image classification device.

[0019] Figure 6 This is a diagram illustrating the image classification device of Modified Example 1.

[0020] Figure 7 This is a flowchart illustrating the incremental learning process performed by the image classification device of Modified Example 1.

[0021] Figure 8 This is a structural diagram of the image classification device in Modified Example 2.

[0022] Figure 9 This is a flowchart illustrating the incremental learning process performed by the image classification device of Modified Example 2.

[0023] Figure 10 This is a diagram illustrating the image classification device of Modified Example 3.

[0024] Figure 11 This is a flowchart illustrating the incremental learning process performed by the image classification device of Modified Example 3.

[0025] Figure 12 This is a diagram illustrating the operation of the weight generation unit in variation example 3.

[0026] Figure 13 This is a diagram illustrating the image classification device of Modified Example 4.

[0027] Figure 14This is a flowchart illustrating the incremental learning process performed by the image classification device of Modified Example 4.

[0028] Figure 15 This is a diagram illustrating an example of a superordinate category.

[0029] Figure 16 This is a diagram illustrating the calculation of the prototype of the additional category performed by the additional category prototype generation unit of Variation Example 4.

[0030] Figure 17 This is another example illustrating a higher category.

[0031] Figure 18 This is a structural diagram of the image classification device in Modified Example 5.

[0032] Figure 19 This is a flowchart illustrating the incremental learning process performed by the image classification device of Modified Example 5.

[0033] Figure 20 This is a diagram illustrating the operation of the weight generation unit in variation 5.

[0034] Figure 21 This is a structural diagram of the image classification device in Modified Example 6.

[0035] Figure 22 This is a structural diagram of the image classification device in Modified Example 7. Detailed Implementation

[0036] First, the basic training dataset and the incremental training dataset will be explained.

[0037] The basic training dataset contains multiple basic categories (e.g., around 100 to 1000 categories), and each basic category is a supervised dataset consisting of multiple training images (e.g., 500 to 3000 images). The basic training dataset is sufficient to learn a general classification task on its own. Here, the number of basic categories is set to 60, and the number of training images for each basic category is set to 500.

[0038] In contrast, the incremental training dataset contains a small number of additional classes (e.g., around 1 to 10 classes), each of which is a supervised dataset consisting of a small number of training images (e.g., around 1 to 10). Here, we set it to a small number of images, but a small number of classes can also consist of multiple images. Here, we set the number of additional classes to 5 and the number of training images for each additional class to 5.

[0039] The category names of basic and additional categories are represented by text (sentences).

[0040] In incremental category learning, the initially learned categories are called base categories (known categories), and the categories learned incrementally (unknown categories) are called append categories. Base category learning uses a base dataset to learn the base categories. Append category learning uses an incremental dataset to add append categories to the categories learned so far.

[0041] During the classification (reasoning) phase, it is possible to classify both the basic category and all additional categories simultaneously.

[0042] In incremental learning, there are two main approaches: one that relearns the image feature extraction unit and another that does not. Maintaining the accuracy of the basic categories is relatively easy when the image feature extraction unit is not relearned, but improving the accuracy of additional categories presents challenges. Furthermore, it is difficult to improve the accuracy of additional categories when the number of images for additional categories is small.

[0043] The purpose of the image classification apparatus 100 according to the embodiments of the present invention is to improve the classification accuracy of additional categories without requiring the image feature extraction unit to relearn when the number of images to be added to a small number of categories is small.

[0044] (Example) Figure 1 This is a structural diagram of the image classification apparatus 100 according to an embodiment. The image classification apparatus 100 includes an image feature output unit 10, an image prototype generation unit 20, a similarity calculation unit 30, a classification unit 40, a sentence feature output unit 50, a sentence prototype generation unit 60, an additional category prototype generation unit 70, and a weight generation unit 80.

[0045] The image feature output unit 10 is a pre-learned neural network model obtained by learning from the text and the image. It takes the image as input and outputs image features.

[0046] The image feature output unit 10 is composed of a deep neural network, which calculates the image feature vector (image feature quantity) of the input image. The image feature output unit 10 outputs the image feature vector to the image prototype generation unit 20 and the weight generation unit 80.

[0047] Here, it is assumed that the weight parameters of the image feature output unit 10 are learned using a sufficiently large image dataset from ImageNet before the basic category dataset. Furthermore, the weight parameters of the image feature output unit 10 are fixed after fine-tuning using the basic category dataset.

[0048] Here, ResNet-18 is used as the image feature output unit 10. The dimension of the feature vector of ResNet-18 is 512. The image feature output unit 10 is not limited to ResNet-18, and can also be a deep neural network such as ViT-B / 32 (image feature dimension is 512), ViT-L / 14 (image feature dimension is 768), RN50 (image feature dimension is 1024), RN101 (image feature dimension is 512), RN50x4 (image feature dimension is 640), etc., which are extended models of ResNet.

[0049] The image prototype generation unit 20 calculates image feature quantities for each category (e.g., calculates the mean, median, representative value, etc.) and outputs the image prototype for each category.

[0050] The similarity calculation unit 30 consists of fully connected layers, each with weight vectors for multiple categories. That is, the weight vectors exist for each category.

[0051] The similarity calculation unit 30 calculates the cosine similarity, which is the similarity between the input image feature quantity and the weight vectors of multiple categories. The similarity calculation unit 30 outputs the cosine similarity of the multiple categories to the classification unit 40.

[0052] The weight vector of a fully connected layer includes both base class and appended class weights. Here, each class's weight vector is a representative vector of the image features of that class, also known as the prototype. The prototype is set as the average of the image features.

[0053] Assume that the weight vectors of the basic categories in the similarity calculation unit 30 are pre-calculated and fixed using the dataset of the basic categories.

[0054] The image feature output unit 10 takes all training images of the basic categories contained in the basic training dataset as input and outputs the image feature quantities of all training images of the basic categories. The image prototype generation unit 20 takes the image feature quantities of all training images of the basic categories as input, averages the image feature quantities by category, and outputs the weight vector (image prototype) of each basic category.

[0055] The similarity calculation unit 30 receives the weight vectors of the basic categories from the image prototype generation unit 20. On the other hand, it receives the weight vectors of the additional categories from the weight generation unit 80. The specific incremental method will be described later.

[0056] The similarity calculation unit 30 saves the image prototypes of the basic categories output from the image prototype generation unit 20 as the weights of the basic categories, saves the prototypes of the additional categories output from the weight generation unit 80 as the weights of the additional categories, and calculates the similarity by taking the image features output from the image feature output unit 10 as input.

[0057] The classification unit 40 selects the category with the highest similarity from the similarity calculated by the similarity calculation unit 30. The similarity is, for example, cosine similarity, which represents the similarity between each category. The classification unit 40 selects the category with the highest cosine similarity.

[0058] The sentence feature output unit 50 is a pre-learned neural network model obtained by learning from sentences and images. It takes sentences describing the category as input and outputs sentence features.

[0059] The sentence feature output unit 50 is composed of a deep neural network, which calculates sentence feature vectors (sentence feature quantities) based on sentences associated with the input category name.

[0060] The sentence feature output unit 50 outputs the sentence feature vector to the sentence prototype generation unit 60.

[0061] The sentence prototype generation unit 60 calculates sentence feature quantities for each category (e.g., calculates the average, median, representative value, etc.) and outputs sentence prototypes for each category.

[0062] Here, when there is only one sentence related to the category, the sentence features directly become the sentence prototype, so the structure of the sentence prototype generation unit 60 can be omitted. It should be noted that in this case, the sentence features output from the sentence feature output unit 50 are directly input into the appended category prototype generation unit 70 as the document prototype.

[0063] The neural network model of the sentence feature output unit 50 utilizes a text encoder, or transformer model, learned in the common feature space of image and text as shown in Non-Patent Document 1. Therefore, the sentence features output by the sentence feature output unit 50 can match the image features.

[0064] In Non-Patent Document 1, an image encoder and a text encoder are trained to match image feature vectors and sentence feature vectors of a certain category in a feature space.

[0065] Text encoders and image encoders are not good at detailed category classification, but they are good at conceptual classification.

[0066] Image encoders are typically very large-scale neural networks trained on massive datasets. On the other hand, text encoders do not necessarily use neural networks as large as image encoders.

[0067] Here, the conceptual classification capabilities of a small-scale text encoder can be leveraged to learn from a small amount of data to improve category classification accuracy.

[0068] Furthermore, here, we assume that the dimension of the feature vector output by the image feature output unit 10 is the same as the dimension of the feature vector output by the sentence feature output unit 50, which is 512. By making the dimensions consistent, direct matching is possible without dimensionality reduction, thus improving matching accuracy.

[0069] The prototype calculated based on the feature vector output by the image feature output unit 10 is called the "image prototype", and the prototype calculated based on the feature vector output by the sentence feature output unit 50 is called the "sentence prototype".

[0070] The additional category prototype generation unit 70 generates additional category prototypes using the image prototypes and text prototypes of the basic categories.

[0071] More specifically, the supplementary category prototype generation unit 70 calculates the difference between the sentence prototypes of base categories located near the sentence prototypes of the supplementary category and the image prototypes of base categories that are the same as the sentence prototypes of base categories located near the sentence prototypes of the supplementary category. Furthermore, the supplementary category prototype generation unit 70 generates supplementary prototypes by adding this difference to the sentence feature quantity of the supplementary category. Here, it is sufficient to have at least one neighbor. Therefore, the difference can be the difference with only one neighbor, or it can be the average of the differences with multiple neighbors. Furthermore, "neighbor" can also be defined as within a specified distance.

[0072] The additional category prototype generation unit 70 outputs the additional category prototype to the weight generation unit 80.

[0073] The weight generation unit 80 takes the image feature quantity of the additional category and the prototype of the additional category as input, calculates the image feature quantity of the additional category and the prototype of the additional category, and generates the weight vector of the additional category for the similarity calculation unit 30.

[0074] Figure 2 This is a flowchart illustrating the processing steps of incremental learning performed by the image classification device 100.

[0075] A sentence describing a basic category is input into the sentence feature output unit 50. If no suitable sentence describing the basic category exists, the category name of the basic category can be used as the sentence. The sentence feature output unit 50 calculates the sentence feature quantity of the basic category based on the input sentence, and the sentence prototype generation unit 60 generates a sentence prototype of the basic category based on the sentence feature quantity of the basic category, and outputs the sentence prototype of the basic category to the additional category prototype generation unit 70.

[0076] The image feature output unit 10 takes the images of the basic categories as input and outputs image features. The image prototype generation unit 20 takes the image features as input and outputs the image prototypes (weight vectors) of the basic categories to the additional category prototype generation unit 70.

[0077] The additional category prototype generation unit 70 stores the text prototypes and image prototypes (weight vectors) of the basic categories.

[0078] Next, the processing steps for incremental learning, which enable classification of additional categories after they have been provided, will be described. Incremental learning does not require large processing such as optimization and can be performed repeatedly.

[0079] To simplify the explanation, we will add the additional categories one by one, but you can also add multiple categories at once.

[0080] The system provides text describing the additional category j and K images as data for the additional category. Here, K is any integer greater than or equal to 1. If no suitable text describing the additional category exists, the category name of the additional category can also be used as the text. The text is input to the text feature output unit 50, and the K images are input to the image feature output unit 10.

[0081] When K images are input, the image feature output unit 10 calculates the K image feature quantities FVn_Img(j,k) for category j and provides them to the weight generation unit 80. Here, k=0,1,2,...,K-1.

[0082] When the input sentence describes the addition of a category, the sentence feature output unit 50 calculates the sentence prototype PVn_Com(j) of the addition category j and provides it to the addition category prototype generation unit 70.

[0083] The additional category prototype generation unit 70 selects the sentence prototypes of M basic categories (e.g., M=3) located near the sentence prototype of the additional category j (S20). Here, M is any integer greater than or equal to 1.

[0084] The additional category prototype generation unit 70 calculates the movement vector MVb(i) from the text prototype PVb_Com(i) to the image prototype PVb_Img(i) for the selected M basic categories i, as shown in the following formula. Here, i=1,2,...,M.

[0085] MVb(i)=PVb_Img(i)-PVb_Com(i) The additional category prototype generation unit 70 calculates the average movement vector MVb_ave of the basic categories of the M basic categories as follows:

[0086] MVb_ave = ΣMVb(i) / M The additional category prototype generation unit 70 adds the average movement vector of the basic category to the sentence prototype of the additional category j, calculates the additional category prototype PPVn_Img(j) of the additional category j as follows, and provides it to the weight generation unit 80.

[0087] PPVn_Img(j)=PVn_Com(j)+MVb_ave Figure 3 This is a diagram illustrating an example of calculating the additional category prototype.

[0088] First, select the basic categories B, C, and E (black circles) whose text prototypes are located near the text prototype of the additional category j. Next, calculate the movement vectors MVb(B), MVb(C), and MVb(E) from the text prototypes of the basic categories B, C, and E to the image prototype. Average MVb(B), MVb(C), and MVb(E) to calculate the average movement vector MVb_ave. The additional category prototype of the additional category j is calculated by adding the average movement vector to the text prototype of the additional category j.

[0089] Figure 4 This is a diagram illustrating an example of the operation of the weight generation unit 80.

[0090] The weight generation unit 80 averages the K image features of the additional category j input from the image feature output unit 10 and the additional category prototype of the additional category j input from the additional category prototype generation unit 70, calculates the image prototype PVn_Img(j) of the additional category j as shown in the following formula, and outputs the image prototype of the additional category j to the similarity calculation unit 30.

[0091] PVn_Img(j)=(ΣFVn_Img(j,k)+PPVn_Img(j)) / (K+1) The similarity calculation unit 30 adds the image prototype of the appended category j as the weight vector of the appended category j to the fully connected layer.

[0092] Therefore, the image classification device 100 is able to classify additional category j in addition to the basic category.

[0093] As another example, if it is known that the accuracy of the sentence prototype of the additional category j output by the sentence feature output unit 50 is high, then the weight of the additional category prototype (α in the following formula) can be increased to calculate the image prototype after weighted average of the additional category j, as shown in the following formula. For example, let α be 1.2, etc., and set α to be greater than 1.

[0094] PVn_Img(j)=ΣFVn_Img(j,k) / K+α×PPVn_Img(j) In addition, the average of the moving vectors of neighboring classes is used here, but statistics other than the average, such as the median, maximum, and minimum values ​​of the moving vectors of neighboring classes, can also be used.

[0095] Next, the processing steps of the classification (inference) performed by the image classification device 100 after classifying additional categories will be explained.

[0096] Figure 5 This is a flowchart illustrating the classification process performed by the image classification device 100.

[0097] The input image is input to the image feature output unit 10. The image feature output unit 10 calculates the image features of the input image and provides them to the similarity calculation unit 30.

[0098] The similarity calculation unit 30 calculates the similarity between the input image feature quantity and the weight vector of all classes, and provides the similarity of all classes to the classification unit 40.

[0099] The classification unit 40 selects the category with the highest similarity from all categories. This determines the category of the input image.

[0100] As described above, the image classification apparatus 100 of the embodiment calculates additional category prototypes using the literal prototypes of the additional categories and the average movement vectors calculated based on the literal prototypes of the basic categories and the image prototypes of the basic categories. Furthermore, the image prototypes of the additional categories are averaged with K image features of the additional categories to calculate the image prototypes of the additional categories as the weight vector of the similarity calculation unit 30. Therefore, even if the image data contains only a small number of additional categories, high-precision image prototypes of the additional categories can be obtained by using high-precision image prototypes of the basic categories calculated from multiple images. In addition, by performing a weighted average of the image prototypes of the additional categories based on the precision of the literal prototypes of the additional categories output by the literal prototype generation unit 60, the image classification apparatus 100 can classify additional categories with high precision in addition to the basic categories.

[0101] Furthermore, in the above embodiments, as a sentence prototype located near the sentence prototype of the additional category j, the additional category prototype generation unit 70 selects a basic category that has been learned with a sufficient amount of data, but it may also select an additional category that was input before the additional category j as a sentence prototype located nearby.

[0102] (Variation Example 1) Here, a simpler structure of the image classification apparatus 100 of the embodiment will be described. Specifically, a variation in which only category names (labels) are added to the categories without image data will be described.

[0103] Figure 6 This is a diagram illustrating the image classification apparatus 100 of Modified Example 1. The difference from the embodiment is that there is no weight generation unit 80, and no processing flow associated with the weight generation unit 80. Therefore, even without image data for additional categories, it is possible to learn additional categories by using the category names of the additional categories as text data.

[0104] When learning is performed in a way that the feature space of the sentence feature output unit 50 and the feature space of the image feature output unit 10 are close feature spaces, additional categories can be learned even without image data with additional categories by using sentence prototypes generated based on the category names of the additional categories.

[0105] For example, a feature space that is close to the text feature output unit 50 and the image feature output unit 10 is a feature space with the same dimension. When the text features output by the text feature output unit 50 and the image features output by the image feature output unit 10 in a certain category are mapped to the same feature space, the distance between the text features and the image features becomes closer.

[0106] Even for additional categories that only have category names and no images, the image classification device 100 can classify additional category j in addition to the basic category by utilizing the additional category prototype.

[0107] In Variation 1, there is no weight generation unit 80. The additional category prototype generation unit 70 outputs the additional category prototype of the additional category j as the weight vector of the additional category j to the similarity calculation unit 30. As described above, since the textual features and image features have the same dimension, the distance becomes closer when the textual features and image features in a certain category are mapped to the same feature space. Therefore, the relationship between the textual prototype and image prototype of the additional category j generated based on the textual features and image features respectively is also the same. Therefore, the additional category prototype generation unit 70 can generate additional category prototypes using the textual prototype of the additional category j and the image prototype of the basic category.

[0108] Figure 7 This is a flowchart illustrating the incremental learning processing steps performed by the image classification device 100 in Modified Example 1. The difference lies in omitting... Figure 2 Steps S16 and S28 are modified by replacing step S30 with step S32. Other steps are similar to... Figure 2 The processing steps are the same, so the explanation of the common processing steps is omitted, and only the differences are explained.

[0109] In step S32, the similarity calculation unit 30 adds the prototype of the additional category j as the weight vector of the additional category j to the fully connected layer.

[0110] (Variation Example 2) Here, a simpler structure of variation 1 will be described. Figure 8 This is a structural diagram of the image classification apparatus 100 of Modified Example 2. The difference from Modified Example 1 is that no category prototype generation unit 70 is added. Therefore, the sentence prototypes output from the sentence prototype generation unit 60 (and the sentence features output from the sentence feature output unit 50 when there is only one sentence related to the category (e.g., only the category name)) are used as the weight vector of the similarity calculation unit 30.

[0111] Modification 2 can be applied to the case where learning is performed in a way that the feature space of the sentence feature output unit 50 and the feature space of the image feature output unit 10 are close feature spaces. The sentence prototype output from the sentence prototype generation unit 60 can be directly used as the weight vector of the similarity calculation unit 30, thus simplifying the structure and reducing the processing load of the image classification device 100.

[0112] Furthermore, even with image data without additional categories, incremental learning can be performed simply by adding the category names. Consequently, since it's unnecessary to consider background or other irrelevant elements within the image, the accuracy of the weight vector can be improved.

[0113] Figure 9 This is a flowchart illustrating the incremental learning processing steps performed by the image classification device 100 in Modified Example 2. (Details omitted) Figure 2 In the incremental learning processing steps of the embodiment, steps S10, S12, S14, S16, S20, S22, S24, S26, and S28 are replaced by step S19 and step S30 is replaced by step S34.

[0114] When a sentence with an additional category is input, the sentence feature output unit 50 outputs the sentence feature values ​​to the sentence prototype generation unit 60. The sentence prototype generation unit 60 calculates the sentence prototype PVn_Com(j) of the additional category j based on the sentence feature values ​​and provides it to the similarity calculation unit 30.

[0115] The similarity calculation unit 30 adds the sentence prototype of the appended category j as the weight vector of the appended category j to the fully connected layer.

[0116] (Variation Example 3) Here, a simplified structure of the embodiment will be described. Figure 10 This is a diagram illustrating the image classification device 100 of Modified Example 3. (And...) Figure 1 The difference in the embodiment is that no additional category prototype generation unit 70 is added.

[0117] The weight generation unit 80 averages the sentence prototype output from the sentence prototype generation unit 60 (in the case where there is only one sentence related to the category (e.g., only the category name) and the image feature output from the image feature output unit 10) to generate the weight vector of the similarity calculation unit 30.

[0118] Learning is performed in a manner where the feature space of the sentence feature output unit 50 and the feature space of the image feature output unit 10 are similar feature spaces. In the case of image data with additional categories, by utilizing the sentence prototype of category j, which uses sentence output as high-precision but conceptual information, and the image features of additional category j, which uses image output as specific information, the weight generation unit 80 can generate an appropriate weight vector for the additional categories of the similarity calculation unit 30.

[0119] Figure 11 This is a flowchart illustrating the incremental learning processing steps performed by the image classification device 100 in Modified Example 3. The difference lies in omitting... Figure 2 In the incremental learning processing steps of the embodiment, steps S10, S12, S14, S20, S22, S24, and S26 are replaced by step S29. Other steps are the same as... Figure 2 The processing steps are the same, so the explanation of the common processing steps is omitted, and only the differences are explained.

[0120] In step S29, the weight generation unit 80 averages the K image features of the additional category j input from the image feature output unit 10 and the sentence prototype input from the sentence prototype generation unit 60 (the sentence features output from the sentence feature output unit 50 when there is only one sentence related to the category), calculates the image prototype PVn_Img(j) of the additional category j, and outputs the image prototype of the additional category j to the similarity calculation unit 30.

[0121] Figure 12 This is a diagram illustrating the operation of the weight generation unit 80 in variation example 3.

[0122] The weight generation unit 80 averages the K image features of the additional category j input from the image feature output unit 10, namely FVn_Img(j,k), and the text prototype of the additional category j input from the text prototype generation unit 60, namely PVn_Com(j), and calculates the image prototype PVn_Img(j) of the additional category j as follows.

[0123] PVn_Img(j)=(ΣFVn_Img(j,k)+PVn_Com(j)) / (K+1) (Variation Example 4) Here, a variation of the additional category prototype generation unit 70 will be described. For the dataset used in the learning of basic categories, a hyperclassification is defined relative to a given category.

[0124] Figure 13 This diagram illustrates the image classification apparatus 100 of Modification 4. The difference from the embodiment is that higher-level category information is provided to the additional category prototype generation unit 70. In the embodiment, the sentence prototypes of the base categories located near the sentence prototypes of the additional category are used for calculating the additional category prototypes. In Modification 4, a higher-level category is predefined, and the sentence prototypes of the base categories belonging to the same higher-level category as the additional category are used for calculating the additional category prototypes.

[0125] If we compare images of different categories that share the same superordinate category, they will have similar features. Therefore, when there is a large deviation between the image prototype and the text prototype, calculating additional category prototypes by using image prototypes of different categories that share the same superordinate category can improve accuracy compared to using the correlation of the text prototype.

[0126] Alternatively, Modification 1 can be applied to Modification 4, omitting the weight generation part 80.

[0127] Figure 14 This is a flowchart illustrating the incremental learning processing steps performed by the image classification device 100 in Modified Example 4. The difference lies in... Figure 2 In the embodiment, step S20 of the incremental learning processing steps is replaced by step S21, except that... Figure 2 The processing steps are the same, so the explanation of the common processing steps is omitted, and only the differences are explained.

[0128] In step S21, the additional category prototype generation unit 70 selects sentence prototypes from the M basic categories belonging to the superordinate category of the additional category j.

[0129] Figure 15 This is a diagram illustrating an example of a superclass. A superclass is a concept that is superior to a class. Figure 15This is an example from the CIFAR100 dataset. For instance, if the superordinate category is set to aquatic mammals, and the appended category is dolphin, then the base categories that share the same superordinate category are beaver, otter, seal, and whale. For example, aquatic mammals share features similar as images, such as having tail fins.

[0130] Figure 16 This diagram illustrates the calculation of the prototype for the additional category performed by the additional category prototype generation unit 70.

[0131] The basic categories that are the same as the superordinate category of the additional category j are designated as category A', category B', category B', category C', and category D'. The movement vectors from the text prototype to the image prototype are designated as MVA', MVB', MVC', and MVD' in the order of the above basic categories. The additional category prototype generation unit 70 adds the text prototype of the additional category j to the average of MVA', MVB', MVC', and MVD', i.e., the average movement vector, to calculate the prototype of the additional category j.

[0132] For example, if the added category j is beaver, and the aquatic mammals in the superordinate category include dolphins, otters, seals, and whales, which are included in the basic category, then the prototypes of dolphins, otters, seals, and whales become A', B', C', and D', respectively.

[0133] As another example related to the superordinate category, it can also be used as follows: Figure 17 As shown, the general classification hierarchy of organisms is defined by order, family, genus, species, etc., to define the superordinate category.

[0134] When using categories with the same superordinate category, it can be expected that the closer the prototype is to the superordinate category, the more subordinate the superordinate category.

[0135] In addition, the average of the movement vectors of categories with the same superordinate category is used here, but the median, maximum, minimum, etc. of the movement vectors can also be used, for example.

[0136] (Variation Example 5) Figure 18 This is a structural diagram of the image classification device 100 of Modified Example 5. The difference between the structure of Modified Example 5 and the embodiment is that the additional category prototype generation unit 70 is replaced by the basic category sentence prototype selection unit 72, and the operation of the weight generation unit 80 is different.

[0137] The basic category sentence prototype selection unit 72 selects the basic category sentence prototype located near the additional category sentence prototype, and provides the basic category sentence prototype and the additional category sentence prototype to the weight generation unit 80.

[0138] The weight generation unit 80 averages the K image features of the additional category input from the image feature output unit 10, the L sentence prototypes of the basic category input from the basic category sentence prototype selection unit 72, and the sentence prototype of one additional category input from the basic category sentence prototype selection unit 72, calculates the image prototype of the additional category, and sets it as the weight vector of the additional category of the similarity calculation unit 30. Here, L is any integer greater than or equal to 1.

[0139] Figure 19 This is a flowchart illustrating the incremental learning processing steps performed by the image classification apparatus 100 in Modified Example 5. The difference in Modified Example 5 is that... Figure 2 Steps S10, S12, and S14 of the incremental learning processing steps in the embodiment are replaced with... Figure 19 Steps S11 and S15, Figure 2 Steps S20, S22, S24, S26, and S28 are replaced with Figure 19 Steps S23 and S27. Others are the same as... Figure 2 The processing steps are the same, so the explanation of the common processing steps is omitted, and only the differences are explained.

[0140] In step S11, the sentence prototype generation unit 60 outputs the sentence prototypes of the basic categories to the basic category sentence prototype selection unit 72.

[0141] In step S15, the category prototype generation unit 70 adds a category prototype generation unit to store the sentence prototypes of the basic categories.

[0142] In step S23, the basic category sentence prototype selection unit 72 selects the sentence prototype of the additional category j and the sentence prototypes of the L basic categories located near the sentence prototype of the additional category j.

[0143] In step S27, the weight generation unit 80 averages the K image features of the additional category j input from the image feature output unit 10, the L basic category sentence prototypes input from the basic category sentence prototype selection unit 72, and the sentence prototype of the additional category, calculates the image prototype PVn_Img(j) of the additional category j, and outputs it as the weight vector of the additional category j to the similarity calculation unit 30. Figure 20 The diagram shows an example of the operation of the weight generation unit 80 in Modification 5. As an example, symbols 200a, 200b, 200c, 200d, and 200e represent five image features of the additional category j; symbols 210a, 210b, and 210c represent sentence prototypes of three basic categories; and symbol 220 represents a sentence prototype of an additional category. Symbol 230 represents the image prototype of the additional category j calculated by averaging the five image features of the additional category j, the sentence prototypes of the three basic categories, and the sentence prototype of the additional category.

[0144] It is known that image prototypes calculated using a small number of image features of a certain class have insufficient ability to adapt to unknown image data (generalization performance). Therefore, by utilizing the literal prototypes of additional class j and the literal prototypes of nearby basic classes, the generalization performance can be improved.

[0145] (Variation Example 6) Figure 21 This is a structural diagram of the image classification device 100 of Modified Example 6. The difference in the structure of Modified Example 6 is that the basic category text prototype selection unit 72 of Modified Example 5 is replaced by the basic category image prototype selection unit 74, and the operation of the weight generation unit 80 is different.

[0146] The basic category image prototype selection unit 74 selects L basic category image prototypes located near the additional category text prototypes, and provides the selected basic category image prototypes and additional category text prototypes to the weight generation unit 80.

[0147] The weight generation unit 80 averages the K image features of the additional category input from the image feature output unit 10, the L image prototypes of the basic category input from the basic category image prototype selection unit 74, and the sentence prototype of the additional category input from the basic category image prototype selection unit 74, calculates the image prototype of the additional category, and sets it as the weight vector of the additional category of the similarity calculation unit 30.

[0148] In variation example 6, Figure 19 Step S23 is replaced as follows. The basic category image prototype selection unit 74 selects the sentence prototype of the additional category j and the image prototypes of the L basic categories located near the sentence prototype of the additional category j.

[0149] In variation example 6, Figure 19 Step S27 is replaced as follows. The weight generation unit 80 averages the K image features of the additional category j input from the image feature output unit 10, the L image prototypes of the basic categories input from the basic category image prototype selection unit 74, and the sentence prototype of the additional category, and calculates the image prototype PVn_Img(j) of the additional category j.

[0150] It is known that image prototypes calculated using a small number of image features have insufficient generalization performance. Therefore, by utilizing image prototypes of basic classes near class j, the generalization performance can be improved.

[0151] (Variation Example 7) Figure 22This is a structural diagram of the image classification device 100 of Modified Example 7. The difference in the structure of Modified Example 7 is that the basic category text prototype selection unit 72 of Modified Example 5 and the basic category image prototype selection unit 74 of Modified Example 6 are replaced by the basic category prototype selection unit 76, and the operation of the weight generation unit 80 is different.

[0152] The basic category prototype selection unit 76 selects the basic category's sentence prototype and the basic category's image prototype located near the additional category's sentence prototype, and provides the basic category's sentence prototype, the basic category's image prototype, and the additional category's sentence prototype to the weight generation unit 80.

[0153] The weight generation unit 80 averages the K image features of the additional category input from the image feature output unit 10, the L sentence prototypes of the basic categories selected by the basic category prototype selection unit 76, the M image prototypes of the basic categories, and the sentence prototype of the additional category input from the basic category prototype selection unit 76, calculates the image prototype of the additional category, and sets it as the weight vector of the additional category of the similarity calculation unit 30.

[0154] In variation example 7, Figure 19 Step S23 is replaced as follows. The basic category prototype selection unit 76 selects the sentence prototype of the additional category j, the sentence prototypes of L basic categories located near the sentence prototype of the additional category j, and the image prototypes of M basic categories located near the sentence prototype of the additional category j.

[0155] In variation example 7, Figure 19 Step S27 is replaced as follows. The weight generation unit 80 averages the K image features of the additional category input from the image feature output unit 10, the sentence prototype of one additional category input from the basic category prototype selection unit 76, the sentence prototypes of L basic categories selected by the basic category prototype selection unit 76, and the image prototypes of M basic categories selected by the basic category prototype selection unit 76 to calculate the image prototype PVn_Img(j) of the additional category j.

[0156] It is known that the generalization performance of image prototypes calculated using image features from a small number of images is insufficient. Therefore, by utilizing image prototypes of the base classes and the base class's sentence prototypes near the sentence prototype of class j, the generalization performance can be improved.

[0157] The various processing steps of the image classification device 100 described above can be implemented using hardware such as a CPU and memory, or using software such as firmware stored in ROM (Read-Only Memory) and flash memory, or computer software. This firmware or software program can be provided by recording on a readable recording medium such as a computer, or it can be transmitted and received via wired or wireless networks and servers, or it can be transmitted and received as data broadcast via terrestrial waves or satellite digital broadcasting.

[0158] The present invention has been described above based on embodiments. Those skilled in the art should understand that the embodiments are illustrative, and various modifications can exist in the combination of these constituent elements and processes, and such modifications are also within the scope of the present invention.

[0159] Industrial applicability This invention can be applied to image classification technology.

[0160] Explanation of reference numerals in the attached figures: 10: Image feature output unit; 20: Image prototype generation unit; 30: Similarity calculation unit; 40: Classification unit; 50: Sentence feature output unit; 60: Sentence prototype generation unit; 70: Additional category prototype generation unit; 72: Basic category sentence prototype selection unit; 74: Basic category image prototype selection unit; 76: Basic category prototype selection unit; 80: Weight generation unit; 100: Image classification device.

Claims

1. An image classification device, characterized in that, include: The image feature output unit, which has completed pre-learning based on text and images, takes the image as input and outputs image features; The image prototype generation unit calculates the image feature quantities for each category and outputs the image prototypes for each category. The sentence feature output unit has completed pre-learning based on sentences and images. It takes sentences describing the category as input and outputs sentence features. The similarity calculation unit stores the image prototypes of the basic category as weights for the basic category and the textual features of the supplementary category as weights for the supplementary category. It then uses the image features output from the image feature output unit as input to calculate the similarity. The classification department takes the similarity as input and determines the classification of the image.

2. The image classification device according to claim 1, characterized in that, It also includes a weight generation unit, which uses the image feature quantities of the additional category output from the image feature quantity output unit and the sentence feature quantities of the additional category output from the sentence feature quantity output unit to generate the weights of the additional category of the similarity calculation unit.

3. The image classification device according to claim 1, characterized in that, It also includes an additional category prototype generation unit, which uses the difference between the sentence prototype of the base category located near the sentence feature value of the additional category and the image prototype corresponding to the sentence prototype of the base category located near the sentence feature value of the additional category, and adds the difference to the sentence feature value of the additional category to generate an additional category prototype. The similarity calculation unit saves the image prototypes of the basic categories output by the image prototype generation unit as the weights of the basic categories and the image prototypes of the additional categories as the weights of the additional categories. It also calculates the similarity by taking the image features output by the image feature output unit as input.

4. The image classification device according to claim 3, characterized in that, It also includes a weight generation unit, which takes the image feature quantity of the additional category and the prototype of the additional category as input, and calculates the image feature quantity of the additional category and the prototype of the additional category to generate the weight of the additional category of the similarity calculation unit.

5. The image classification device according to claim 4, characterized in that, The additional category prototype generation unit uses the difference between the sentence prototype of the basic category whose sentence feature quantity belongs to the same superordinate category as the additional category and the image prototype corresponding to the sentence prototype of the basic category whose sentence feature quantity belongs to the same superordinate category as the additional category, and adds the difference to the sentence feature quantity of the additional category to generate the additional category prototype.

6. An image classification method, characterized in that, include: The image feature output step has completed pre-learning based on text and images, taking the image as input and outputting image features; The image prototype generation step calculates the image feature quantity for each category and outputs the image prototype for each category. The sentence feature output step has completed pre-learning based on sentences and images. It takes sentences describing the category as input and outputs sentence features. The similarity calculation step involves saving the image prototype of the basic category as the weight of the basic category and the text feature quantity of the supplementary category as the weight of the supplementary category. The image feature quantity output from the image feature quantity output step is used as input to calculate the similarity. as well as The classification step takes the similarity as input and determines the classification of the image.

7. An image classification program, characterized in that, Have the computer perform the following steps: The image feature output step has completed pre-learning based on text and images, taking the image as input and outputting image features; The image prototype generation step calculates the image feature quantity for each category and outputs the image prototype for each category. The sentence feature output step has completed pre-learning based on sentences and images. It takes sentences describing the category as input and outputs sentence features. The similarity calculation step involves saving the image prototype of the basic category as the weight of the basic category and the text feature quantity of the supplementary category as the weight of the supplementary category. The image feature quantity output from the image feature quantity output step is used as input to calculate the similarity. as well as The classification step takes the similarity as input and determines the classification of the image.

Images