Image classification device, image classification method, and image classification program
The image classification device enhances accuracy and efficiency by applying multimodal prompts to a transformer-based system, addressing the limitations of existing image classification techniques.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- JVC KENWOOD CORP
- Filing Date
- 2024-10-01
- Publication Date
- 2026-04-13
AI Technical Summary
Existing image classification techniques using pre-trained transformers for image recognition do not achieve sufficiently high accuracy.
An image classification device and method that utilizes a multimodal base model of language and images by generating and applying multimodal prompts to a transformer-based system, including language and image feature extraction units, to classify images efficiently and accurately.
The proposed method enables high-efficiency and high-accuracy image classification by reducing the discrepancy between language and image feature vectors, thereby improving classification accuracy and reducing power consumption and resource requirements.
Smart Images
Figure 2026063698000001_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to image classification technology.
Background Art
[0002] An image-based transformer trained using large-scale learning data is used as a base model for image recognition (Non-Patent Document 1). Techniques for efficiently reusing a pre-trained transformer using a prompt are known (Non-Patent Document 2). Also, for language-based transformers, techniques for reusing a pre-trained transformer for various tasks using a prompt are known (Non-Patent Document 3). There is a technique for using a multi-modal base model of language and image for image recognition (Non-Patent Document 4). There is a technique for inputting a prompt from different layers of a transformer (Non-Patent Document 5).
Prior Art Documents
Non-Patent Documents
[0003]
Non-Patent Document 1
Non-Patent Document 2
Non-Patent Document 3
[0004] These prior art techniques had the problem that the image classification accuracy was not sufficiently high.
[0005] This invention has been made in view of these circumstances, and its purpose is to provide an image classification technology that can classify images efficiently and accurately by applying multimodal prompts to a multimodal base model of language and images. [Means for solving the problem]
[0006] To solve the above problems, an image classification device in one aspect of the present invention includes: a language embedding data generation unit that generates class name embedding data from a class name list containing a plurality of class names; an image embedding data generation unit that divides an image into a plurality of segmented images, generates embedding data for each segmented region, and generates image embedding data from the embedding data for each segmented region; a language feature extraction unit that includes a plurality of transformer basic blocks that extract language feature vectors of class names from the class name embedding data; an image feature extraction unit that includes a plurality of transformer basic blocks that extract image feature vectors from the image embedding data; and a classification unit that classifies an image into one of a plurality of class names using the language feature vectors of class names and the image feature vectors. The text prompt of the class name list includes a plurality of text prompt vectors. Each of the plurality of text prompt vectors is input to a different transformer basic block of the language feature extraction unit, and each of the plurality of text prompt vectors is input to a different transformer basic block of the image feature extraction unit.
[0007] Another aspect of the present invention is an image classification method. This method includes: a language embedding data generation step of generating class name embedding data from a class name list containing a plurality of class names; an image embedding data generation step of dividing an image into a plurality of segmented images, generating embedding data for each segmented region, and generating image embedding data from the embedding data for each segmented region; a language feature extraction step of extracting language feature vectors of class names from the class name embedding data using a plurality of transformer base blocks; an image feature extraction step of extracting image feature vectors from the image embedding data using a plurality of transformer base blocks; and a classification step of classifying an image to one of a plurality of class names using the language feature vectors of class names and the image feature vectors. The text prompt of the class name list contains a plurality of text prompt vectors. Each of the plurality of text prompt vectors is input to a different transformer base block in the language feature extraction step, and each of the plurality of text prompt vectors is input to a different transformer base block in the image feature extraction step.
[0008] Furthermore, any combination of the above components, as well as conversions of the expression of the present invention between methods, apparatus, systems, recording media, computer programs, etc., are also valid embodiments of the present invention. [Effects of the Invention]
[0009] According to the present invention, it is possible to provide an image classification technology that can classify images with high efficiency and high accuracy. [Brief explanation of the drawing]
[0010] [Figure 1] This is a diagram showing the configuration of the image classification device of Embodiment 1. [Figure 2] This is a flowchart showing the image classification procedure using the image classification device of Embodiment 1. [Figure 3] This is a diagram showing the configuration of the language embedding data generation unit of Embodiment 1. [Figure 4] It is a flowchart showing the procedure for generating text embedding data by the language embedding data generation unit of Embodiment 1. [Figure 5] It is a diagram showing the data structure of the class name list embedding data sequence of Embodiment 1. [Figure 6] It is a diagram showing the data structure of the class name embedding data of class i in Embodiment 1. [Figure 7] It is a configuration diagram of the image embedding data generation unit of Embodiment 1. [Figure 8] It is a flowchart showing the procedure for generating image embedding data by the image embedding data generation unit of Embodiment 1. [Figure 9] It is a diagram showing the data structure of the image embedding data sequence of Embodiment 1. [Figure 10] It is a configuration diagram of the image classification device according to Embodiment 2. [Figure 11] It is a configuration diagram of the image embedding data generation unit of Embodiment 2. [Figure 12] It is a flowchart showing the procedure for generating image embedding data by the image embedding data generation unit of Embodiment 2. [Figure 13] It is a diagram showing the data structure of the image embedding data sequence of Embodiment 2. [Figure 14] It is a configuration diagram of the image classification device according to Embodiment 3. [Figure 15] It is a configuration diagram of the image embedding data generation unit of Embodiment 3. [Figure 16] It is a flowchart showing the procedure for generating image embedding data by the image embedding data generation unit of Embodiment 3. [Figure 17] It is a diagram showing the data structure of the image embedding data sequence of Embodiment 3. [Figure 18] It is a configuration diagram of the image classification device of the modification example. [Figure 19] It is a configuration diagram of the language embedding data generation unit of the modification example. [Figure 20]It is a flowchart showing the generation procedure of text embedding data by the language embedding data generation unit of the modification example. [Figure 21] It is a configuration diagram of the image classification device according to Embodiment 4. [Figure 22] It is a configuration diagram of the language embedding data generation unit according to Embodiment 4. [Figure 23] It is a configuration diagram of the image embedding data generation unit according to Embodiment 4. [Figure 24] It is a diagram showing the data structure of the class name embedding data of class i according to Embodiment 4. [Figure 25] It is a diagram showing the data structure of the image embedding data sequence according to Embodiment 4. [Figure 26] It is a configuration diagram of the image classification device according to Embodiment 5. [Figure 27] It is a configuration diagram of the image prompt generation unit according to Embodiment 5. [Figure 28] It is a configuration diagram of the image classification device according to Embodiment 6. [Figure 29] It is a configuration diagram of the language feature extraction unit according to Embodiment 6. [Figure 30] It is a configuration diagram of the image feature extraction unit according to Embodiment 6. [Figure 31] It is a diagram for explaining a configuration in which the text prompt of the class name list is input to the language feature extraction unit and the image feature extraction unit in Embodiment 6. [Figure 32] It is a configuration diagram of the image classification device according to Embodiment 7. [Figure 33] It is a diagram for explaining a configuration in which the text prompt of the class name list is input to the language feature extraction unit and the image feature extraction unit, and the image prompt is input to the language feature extraction unit and the image feature extraction unit in Embodiment 7. [Figure 34] It is a diagram for explaining another configuration in which the text prompt of the class name list is input to the language feature extraction unit and the image feature extraction unit, and the image prompt is input to the language feature extraction unit and the image feature extraction unit in Embodiment 7.
Modes for Carrying Out the Invention
[0011] (Prerequisite technology) The underlying technologies for embodiments of the present invention will now be described. Non-patent document 1 discloses a Vision Transformer (ViT) as a base model for image recognition models. It is known that using ViT as the feature extraction unit in an image classification device can achieve higher accuracy than using a Convolutional Neural Network (CNN) as the feature extraction unit.
[0012] As shown in Figure 1 of Non-Patent Document 1, ViT divides the input image into multiple fixed-size regions (9 regions in Figure 1), linearly embeds each divided region, adds positional embedding to generate embedding data, and inputs the sequence of embedding data to the transformer's encoder. The sequence of embedding data is converted into feature vectors by the transformer's encoder and classified by the classifier.
[0013] However, training the transformer's encoder requires training on a large dataset, which presents the challenge of requiring enormous power consumption. Therefore, as shown in Figure 2 of Non-Patent Document 2, a technique has been disclosed to efficiently reuse a trained transformer for a variety of tasks by adding prompts, which are task-related embedded data, to the sequence of embedded data that serves as the input to the transformer's encoder. The prompts can be adjusted to adapt the trained transformer to unknown tasks.
[0014] Originally, transformers are fundamental models for processing natural language, and regarding language-based transformers, as shown in Figure 2 of Non-Patent Document 3, a technique has been disclosed for reusing trained transformers for various tasks using prompts.
[0015] Furthermore, as shown in Figure 1 of Non-Patent Document 4, a multimodal base model of language and images is disclosed. The technology in Non-Patent Document 4 enables high-precision image classification by matching language feature vectors obtained from a language system transformer with image feature vectors obtained from ViT.
[0016] As shown in Figure 2 of Non-Patent Literature 5, a technique is disclosed in which accuracy is improved in incremental learning by inputting a common prompt for all tasks and prompts for each task from different layers of the transformer's encoder. Non-Patent Literature 5 is a unimodal technique and does not consider the use of multimodal features.
[0017] In embodiments of the present invention, a technique is proposed for highly efficient and accurate image classification by applying multimodal prompts to a multimodal base model of language and images.
[0018] (Embodiment 1) The image classification device 100 of Embodiment 1 generates multimodal or crossmodal text prompts.
[0019] Figure 1 is a diagram showing the configuration of the image classification device 100 of Embodiment 1. The image classification device 100 of Embodiment 1 includes a language embedding data generation unit 10, a language feature extraction unit 30, a text prompt storage unit 50, an image embedding data generation unit 20, an image feature extraction unit 40, and a classification unit 70.
[0020] Figure 2 is a flowchart showing the image classification procedure by the image classification device 100 of Embodiment 1. The configuration and operation of the image classification device 100 of Embodiment 1 will be explained with reference to Figures 1 and 2.
[0021] The class name list is input to the language embedding data generation unit 10. The class name list contains multiple class names, and multiple class names are listed together. The class names are text information. Here, we assume there are 100 class names (N=100). N represents the number of class names.
[0022] The language embedding data generation unit 10 reads a text prompt from the text prompt storage unit 50, generates a class name list embedding data sequence, and generates class name embedding data for each class name from the class name list embedding data sequence (S100). The language embedding data generation unit 10 outputs the class name embedding data for each class name to the language feature extraction unit 30.
[0023] The detailed configuration and operation of the language embedding data generation unit 10, as well as the data structure of the class name list embedding data sequence and class name embedding data, will be described later.
[0024] Class name embedding data is input to the language feature extraction unit 30. The language feature extraction unit 30 is a trained language transformer.
[0025] The language feature extraction unit 30 calculates the language feature vector of the input class name embedding data (S101). The language feature extraction unit 30 outputs the language feature vector to the classification unit 70.
[0026] An image is input to the image embedding data generation unit 20. The image embedding data generation unit 20 generates an image embedding data sequence (S110). The image embedding data generation unit 20 outputs the image embedding data sequence to the image feature extraction unit 40.
[0027] The detailed configuration and operation of the image embedding data generation unit 20, and the data structure of the image embedding data sequence will be described later.
[0028] The image feature extraction unit 40 receives an image embedding data sequence as input. The image feature extraction unit 40 is a pre-trained vision transformer for images.
[0029] The image feature extraction unit 40 calculates an image feature vector (S111). The image feature extraction unit 40 outputs the image feature vector to the classification unit 70.
[0030] The image feature extraction unit 40, which is an image transformer, is multimodally trained with the language feature extraction unit 30, which is a language transformer.
[0031] The classification unit 70 receives language feature vectors of class names from the language feature extraction unit 30 and image feature vectors from the image feature extraction unit 40.
[0032] The classification unit 70 stores language feature vectors of all class names input from the language feature extraction unit 30 and image feature vectors of all classes input from the image feature extraction unit 40.
[0033] The classification unit 70 calculates the cosine similarity between the linguistic feature vector of the class name and the image feature vector for all classes, and selects the class name with the highest cosine similarity (S120). Here, cosine similarity is used as the classification metric to select the class with the highest cosine similarity, but the class with the minimum Euclidean distance may also be selected using Euclidean distance.
[0034] Here, the class names included in the class name list do not have to be the class names learned during multimodal training of the language feature extraction unit 30 and the image feature extraction unit 40. Of course, the class names included in the class name list may also be the class names learned during multimodal training of the language feature extraction unit 30 and the image feature extraction unit 40.
[0035] Figure 3 is a diagram showing the configuration of the language embedding data generation unit 10 of Embodiment 1. Figure 4 is a flowchart showing the procedure for generating text embedding data by the language embedding data generation unit 10 of Embodiment 1. The configuration and operation of the language embedding data generation unit 10 of Embodiment 1 will be explained with reference to Figures 3 and 4.
[0036] The language embedding data generation unit 10 includes a text embedding unit 12 and a text prompt addition unit 14.
[0037] The class name list is input to the language embedding data generation unit 10. The text embedding unit 12 repeats the following process for N times the number of class names included in the class name list.
[0038] The text embedding unit 12 generates text embedding data by embedding class names into Dl-dimensional vectors using a pre-trained embedding layer (S200). The text embedding unit 12 outputs the text embedding data to the text prompt addition unit 14. The text embedding data is a Dl-dimensional vector. Here, Dl = 768. Dl is not limited to 768.
[0039] The text prompt addition unit 14 reads a text prompt corresponding to the class name list from the text prompt storage unit 50 and adds the text prompt corresponding to the class name list before the text embedding data (S210).
[0040] Here, the text prompt for the class name list is generated using the language feature extraction unit 30, as well as the image feature extraction unit 40 which has been multimodally trained with the language feature extraction unit 30. At this time, the text prompt is pre-trained multimodally to optimize the loss such that the distance between the language feature vectors of all class names included in the class name list is large, and the loss such that the distance between the image prototypes of all class names included in the class name list is large. Note that the language feature extraction unit 30 and the image feature extraction unit 40 are only used for inference and not for training.
[0041] In this way, by training the text prompt for the class name list using the language feature extraction unit 30, as well as the image feature extraction unit 40 which has been multimodally trained with the language feature extraction unit 30, the discrepancy between the language feature vector and the image feature vector for class names included in the class name list can be reduced compared to when the multimodal training of the language feature extraction unit 30 and the image feature extraction unit 40 is completed. In other words, by reducing the discrepancy between the language feature vector and the image feature vector, the classification accuracy can be improved.
[0042] Furthermore, if the class name included in the text prompt of the class name list is an unknown class that was not included in the training data when the language feature extraction unit 30 and the image feature extraction unit 40 were multimodally trained, the effect of improving classification accuracy will be particularly significant.
[0043] Furthermore, the cost required to learn the text prompts in the class name list is far less than the cost required to learn the image feature extraction unit 40 and the language feature extraction unit 30. Therefore, power consumption can be reduced compared to performing multimodal learning on the language feature extraction unit 30 and the image feature extraction unit 40 for each class name included in the class name list, and the size of the circuits, software, and memory involved in learning can be reduced, making edge learning easier.
[0044] Here, the model was pre-trained to optimize the loss such that the distance between the linguistic feature vectors of all class names in the class name list is large, and the loss such that the image prototypes of all class names in the class name list are large. However, it is also possible to pre-train the model to optimize only the loss such that the distance between the linguistic feature vectors of all class names in the class name list is large. Alternatively, cross-modal training may be performed beforehand to optimize only the loss such that the image prototypes of all class names in the class name list are large.
[0045] The text prompt for the class name list is a (Lpl × Dl) dimension matrix. Here, Lpl is set to 4, but any integer value greater than or equal to 1 will suffice.
[0046] Figure 5 shows the data structure of the class name list embedding data sequence in Embodiment 1. When all class names included in the class name list are processed, a class name list embedding data sequence is generated in which the text prompt for the class name list and the text embedding data for each class name are arranged in order. The class name list embedding data sequence is a ((Lpl+N)×Dl) dimension matrix.
[0047] Figure 6 shows the data structure of the class name embedding data for class i in Embodiment 1. When the class name list embedding data sequence is generated, a ((Lpl+1)×Dl)-dimensional matrix, consisting of the text prompt for the class name list and the text embedding data for the class name, is sequentially input to the language feature extraction unit 30 for each class name from class name 1 to class name N. The ((Lpl+1)×Dl)-dimensional matrix, consisting of the text prompt for the class name list and the text embedding data for the class name, is used as the class name embedding data.
[0048] Here, the text embedding unit 12 generates a class name list embedding data sequence. Alternatively, the text embedding unit 12 may generate class name embedding data for all classes included in the class name list and output them sequentially, without generating a class name list embedding data sequence.
[0049] Figure 7 is a diagram showing the configuration of the image embedding data generation unit 20 of Embodiment 1. Figure 8 is a flowchart showing the procedure for generating image embedding data by the image embedding data generation unit 20 of Embodiment 1. The configuration and operation of the image embedding data generation unit 20 of Embodiment 1 will be explained with reference to Figures 7 and 8.
[0050] The image embedding data generation unit 20 includes an image splitting unit 22, a split image embedding unit 24, and a position embedding data addition unit 26.
[0051] The image is input to the image embedding data generation unit 20. The input image, as an example, consists of RGB components with a size of 256(W)x256(H).
[0052] The image splitting unit 22 divides the input image into 32x32 divided images (S300). Here, the input image is divided into Li = 64 × 3 = 192 divided images. The image splitting unit 22 outputs each divided image to the divided image embedding unit 24. The following process is repeated for the number of divided images Li.
[0053] The segmented images are input to the segmented image embedding unit 24 from the image segmentation unit 22. The segmented image embedding unit 24 uses a trained embedding layer to embed each segmented image into a Di-dimensional vector, generating segmented image embedding data (S310). Here, Di = 768. Di is not limited to 768.
[0054] The position embedding data addition unit 26 adds the position embedding data for each segmented image before the segmented image embedding data (S311). The position embedding data is assumed to be pre-trained Di-dimensional embedding data.
[0055] Here, the positional embedding data is information indicating the position of the segmented images, and is assumed to be learned during multimodal training of the language feature extraction unit 30 and the image feature extraction unit 40. Other positional embedding data may also be used, as long as it was learned when training the image transformer.
[0056] As a result, an image embedding data sequence is generated, consisting of position embedding data for each segmented image and segmented image embedding data.
[0057] Figure 9 shows the data structure of the image embedding data sequence in Embodiment 1. The image embedding data sequence consists of position embedding data for each segmented image 1 to Li and segmented image embedding data, and is a matrix of size (2 × Li) × Di.
[0058] Once an image embedding data sequence is generated, the image embedding data sequence is sequentially input to the image feature extraction unit 40.
[0059] (Embodiment 2) Figure 10 is a diagram showing the configuration of an image classification device 100 according to Embodiment 2. The image classification device 100 of Embodiment 2 includes a language embedding data generation unit 10, a language feature extraction unit 30, a text prompt storage unit 50, an image embedding data generation unit 20, an image feature extraction unit 40, and a classification unit 70.
[0060] In the image classification device 100 of Embodiment 1, the text prompt is input from the text prompt storage unit 50 to the language embedding data generation unit 10. In the image classification device 100 of Embodiment 2, the text prompt is input from the text prompt storage unit 50 to both the language embedding data generation unit 10 and the image embedding data generation unit 20. However, the other configurations and operations are the same as those of the image classification device 100 of Embodiment 1. Only the differences from Embodiment 1 will be explained here.
[0061] Figure 11 is a diagram showing the configuration of the image embedding data generation unit 20 of Embodiment 2. Figure 12 is a flowchart showing the procedure for generating image embedding data by the image embedding data generation unit 20 of Embodiment 2. The configuration and operation of the image embedding data generation unit 20 of Embodiment 2 will be explained with reference to Figures 11 and 12.
[0062] The image embedding data generation unit 20 includes an image splitting unit 22, a split image embedding unit 24, a position embedding data addition unit 26, and a text prompt addition unit 28.
[0063] The configuration and operation (steps S300, S310, S311) of the image division unit 22, the divided image embedding unit 24, and the position embedding data addition unit 26 of Embodiment 2 are the same as those (steps S300, S310, S311) of the image division unit 22, the divided image embedding unit 24, and the position embedding data addition unit 26 of Embodiment 1.
[0064] The segmented image embedding data, to which position embedding data has been added, is input to the text prompt adding unit 28 from the position embedding data adding unit 26. The text prompt adding unit 28 performs the following processing on the input segmented image embedding data.
[0065] The text prompt addition unit 28 reads the text prompt from the text prompt storage unit 50 and adds the text prompt before the segmented image embedding data to which all position embedding data has been added (S330).
[0066] If the dimensions of the text prompt and the image embedding data are different, the text prompt is transformed from a (Lpl × Dl)-dimensional matrix to a (Lpl × Di)-dimensional matrix.
[0067] Based on the above, an image embedding data sequence is generated, consisting of a text prompt, position embedding data for each segmented image, and segmented image embedding data.
[0068] Figure 13 shows the data structure of the image embedding data sequence in Embodiment 2. The image embedding data sequence consists of text prompts 1 to Lpl, position embedding data for each segmented image 1 to Li, and segmented image embedding data, and is a matrix of size (Lpl + 2 × Li) × Di.
[0069] As described above, since the image feature extraction unit 40 and the language feature extraction unit 30 are pre-trained in a multimodal manner, there is a correlation between the image feature vectors and the language feature vectors. Furthermore, since the text prompt for the class name list is generated using the language feature extraction unit 30 and the image feature extraction unit 40, the text prompt for the class name list is correlated with the image feature vectors output by the image feature extraction unit 40.
[0070] Therefore, by adding a text prompt to the image embedding data sequence, the correlation between the image feature vector output by the image feature extraction unit 40 and the language feature vector output by the language feature extraction unit 30 can be reflected, and the behavior of the image feature extraction unit 40 can be constrained so that the image feature vector and the language feature vector do not diverge, thereby improving classification accuracy.
[0071] (Embodiment 3) Figure 14 is a diagram showing the configuration of an image classification device 100 according to Embodiment 3. The image classification device 100 of Embodiment 3 includes a language embedding data generation unit 10, a language feature extraction unit 30, a text prompt storage unit 50, an image embedding data generation unit 20, an image feature extraction unit 40, an image prompt storage unit 60, and a classification unit 70.
[0072] The image classification device 100 of Embodiment 3 differs from the image classification device 100 of Embodiment 1 in that an image prompt storage unit 60 is added, and image prompts are input from the image prompt storage unit 60 to the image embedding data generation unit 20. However, the other configurations and operations are the same as those of the image classification device 100 of Embodiment 1. Here, only the differences from Embodiment 1 will be explained.
[0073] Figure 15 is a diagram showing the configuration of the image embedding data generation unit 20 of Embodiment 3. Figure 16 is a flowchart showing the procedure for generating image embedding data by the image embedding data generation unit 20 of Embodiment 3. The configuration and operation of the image embedding data generation unit 20 of Embodiment 3 will be explained with reference to Figures 15 and 16.
[0074] The image embedding data generation unit 20 includes an image splitting unit 22, a split image embedding unit 24, a position embedding data addition unit 26, and an image prompt addition unit 27.
[0075] The configuration and operation (steps S300, S310, S311) of the image division unit 22, the divided image embedding unit 24, and the position embedding data addition unit 26 of Embodiment 3 are the same as those (steps S300, S310, S311) of the image division unit 22, the divided image embedding unit 24, and the position embedding data addition unit 26 of Embodiment 1.
[0076] The segmented image embedding data, to which position embedding data has been added, is input to the image prompt adding unit 27 from the position embedding data adding unit 26. The image prompt adding unit 27 performs the following processing on the input segmented image embedding data.
[0077] The image prompt addition unit 27 reads the image prompt from the image prompt storage unit 60 and adds the image prompt before the segmented image embedding data to which all position embedding data has been added (S320).
[0078] Assume the image prompt is a (Lpi × Di)-dimensional matrix. Here, Lpi is set to 4, but any integer value greater than or equal to 1 will suffice.
[0079] Here, the image prompt is generated using the image feature extraction unit 40, as well as the multimodal-learned language feature extraction unit 30. At this time, the image prompt is pre-trained using multimodal learning to optimize the loss such that the distance between the language feature vectors of all class names included in the class name list is large, and the loss such that the distance between the image prototypes of all class names included in the class name list is large. Note that the language feature extraction unit 30 and the image feature extraction unit 40 are only used for inference and not for training.
[0080] In this way, by training the image prompt using the image feature extraction unit 40, as well as the language feature extraction unit 30 which has been multimodally trained with the image feature extraction unit 40, the discrepancy between the language feature vector and the image feature vector for class names included in the class name list can be reduced compared to when the multimodal training of the language feature extraction unit 30 and the image feature extraction unit 40 is completed. In other words, by reducing the discrepancy between the language feature vector and the image feature vector, the classification accuracy can be improved.
[0081] Furthermore, if the class name included in the text prompt of the class name list is an unknown class that was not included in the training data when the language feature extraction unit 30 and the image feature extraction unit 40 were trained in multimodal learning, the effect of improving classification accuracy will be particularly significant.
[0082] Furthermore, the cost required for learning image prompts is far less than the cost of learning the image feature extraction unit 40 and the language feature extraction unit 30. Therefore, power consumption can be reduced compared to performing multimodal learning on the language feature extraction unit 30 and the image feature extraction unit 40 again for the class names included in the class name list, and the size of the circuits, software, and memory involved in learning can be reduced, making edge learning easier.
[0083] Here, the model was pre-trained to optimize the loss such that the distance between the linguistic feature vectors of all class names in the class name list is large, and the loss such that the image prototypes of all class names in the class name list are large. However, it is also possible to pre-train the model to optimize only the loss such that the distance between the linguistic feature vectors of all class names in the class name list is large. Alternatively, cross-modal training may be performed beforehand to optimize only the loss such that the image prototypes of all class names in the class name list are large.
[0084] Based on the above, an image embedding data sequence is generated, consisting of an image prompt, position embedding data for each segmented image, and segmented image embedding data.
[0085] Figure 17 shows the data structure of the image embedding data sequence in Embodiment 3. The image embedding data sequence consists of image prompts 1 to Lpi, positional embedding data for each segmented image 1 to Li, and segmented image embedding data, and is a matrix of size (Lpi + 2 × Li) × Di.
[0086] (modified version) Figure 18 is a diagram showing the configuration of a modified image classification device 100. The modified image classification device 100 includes a language embedding data generation unit 10, a language feature extraction unit 30, an image embedding data generation unit 20, an image feature extraction unit 40, an image prompt storage unit 60, and a classification unit 70.
[0087] In the image classification device 100 of Embodiment 2, text prompts are input from the text prompt storage unit 50 to the language embedding data generation unit 10 and the image embedding data generation unit 20. However, in the modified image classification device 100, the text prompt storage unit 50 is replaced by an image prompt storage unit 60, and image prompts are input from the image prompt storage unit 60 to the language embedding data generation unit 10 and the image embedding data generation unit 20. The only difference is that the other configurations and operations are the same as those of the image classification device 100 of Embodiment 2. Only the differences from Embodiment 2 will be explained here.
[0088] Figure 19 is a diagram showing the configuration of the modified language embedding data generation unit 10. Figure 20 is a flowchart showing the procedure for generating text embedding data by the modified language embedding data generation unit 10. The configuration and operation of the modified language embedding data generation unit 10 will be explained with reference to Figures 19 and 20.
[0089] The language embedding data generation unit 10 includes a text embedding unit 12 and an image prompt addition unit 16.
[0090] The class name list is input to the language embedding data generation unit 10. The text embedding unit 12 repeats the following process for N times the number of class names included in the class name list.
[0091] The text embedding unit 12 generates text embedding data by embedding class names into Dl-dimensional vectors using a pre-trained embedding layer (S200). The text embedding unit 12 outputs the text embedding data to the image prompt addition unit 16.
[0092] The image prompt addition unit 16 reads the image prompt from the image prompt storage unit 60 and adds the image prompt before the text embedding data (S220).
[0093] As a result, class name embedding data is generated, consisting of image prompts and text embedding data for each class name. Note that when pre-training the image prompts in this modified example using multimodal training, they should be trained with images of a high level of abstraction, such as graphic symbols.
[0094] (Embodiment 4) Figure 21 is a diagram showing the configuration of the image classification device 100 of Embodiment 4. The image classification device 100 of Embodiment 4 includes a language embedding data generation unit 10, a language feature extraction unit 30, a text prompt storage unit 50, an image embedding data generation unit 20, an image feature extraction unit 40, an image prompt storage unit 60, and a classification unit 70.
[0095] The image classification device 100 of Embodiment 4 is configured by combining the text prompt storage unit 50 of the image classification device 100 of Embodiment 2 with the image prompt storage unit 60 of the modified image classification device 100.
[0096] Figure 22 is a diagram showing the configuration of the language embedding data generation unit 10 of Embodiment 4. The language embedding data generation unit 10 includes a text embedding unit 12, a text prompt addition unit 14, and an image prompt addition unit 16.
[0097] The class name list is input to the language embedding data generation unit 10. The text embedding unit 12 embeds the class names included in the class name list into a Dl-dimensional vector, generates text embedding data, and outputs it to the text prompt addition unit 14.
[0098] The text prompt addition unit 14 reads a text prompt corresponding to the class name list from the text prompt storage unit 50, adds the text prompt corresponding to the class name list before the text embedding data, and outputs it to the image prompt addition unit 16.
[0099] The image prompt addition unit 16 reads the image prompt from the image prompt storage unit 60 and adds the image prompt before the text embedding data.
[0100] As a result, class name embedding data is generated, consisting of a text prompt for the class name list, an image prompt, and text embedding data for each class name. Figure 24 shows the data structure of the class name embedding data for class i in Embodiment 4.
[0101] Figure 23 is a diagram showing the configuration of the image embedding data generation unit 20 of Embodiment 4. The image embedding data generation unit 20 includes an image splitting unit 22, a split image embedding unit 24, a position embedding data addition unit 26, an image prompt addition unit 27, and a text prompt addition unit 28.
[0102] The image is input to the image embedding data generation unit 20. The image splitting unit 22 splits the input image into split images and outputs them to the split image embedding unit 24.
[0103] The segmented image embedding unit 24 embeds each segmented image into a Di-dimensional vector, generates segmented image embedding data, and outputs it to the position embedding data addition unit 26.
[0104] The position embedding data addition unit 26 adds the position embedding data for each segmented image before the segmented image embedding data and outputs it to the image prompt addition unit 27.
[0105] The image prompt addition unit 27 reads the image prompt from the image prompt storage unit 60, adds the image prompt before the segmented image embedding data to which all position embedding data has been added, and outputs it to the text prompt addition unit 28.
[0106] The text prompt addition unit 28 reads a text prompt corresponding to the class name list from the text prompt storage unit 50 and adds the text prompt corresponding to the class name list before the segmented image embedding data to which all position embedding data has been added.
[0107] As a result, an image embedding data sequence is generated, consisting of a text prompt for the class name list, an image prompt, position embedding data for each segmented image, and segmented image embedding data. Figure 25 shows the data structure of the image embedding data sequence in Embodiment 4.
[0108] Here, text prompts and image prompts are generated using the image feature extraction unit 40, as well as the image feature extraction unit 40 and the multimodally learned language feature extraction unit 30.
[0109] The text prompt considers all classes included in the class name list, while the image prompt considers only one class. Therefore, the image prompt and the text prompt have the highest correlation in that order.
[0110] Therefore, prompts are hint information, and to maintain consistency in the transformation's processing, hint information with a higher level of abstraction (less correlation) is placed in the first half of the image embedding data sequence, while hint information with a higher level of specificity (more correlation) is placed in the second half of the image embedding data sequence. In other words, the image embedding data sequence is in the order of text prompt, image prompt, and segmented image embedding data with all position embedding data attached. Similarly, class name embedding data is in the order of text prompt, image prompt, and text embedding data of the class name.
[0111] As described above, by adding text prompts and image prompts in this order before the image embedding data sequence and class name embedding data, the language feature extraction unit 30 and the image feature extraction unit 40 can process information sequentially from conceptual to concrete, thereby improving classification accuracy.
[0112] (Embodiment 5) Figure 26 is a diagram showing the configuration of the image classification device 100 of Embodiment 5. The image classification device 100 of Embodiment 5 includes a language embedding data generation unit 10, a language feature extraction unit 30, a text prompt storage unit 50, an image embedding data generation unit 20, an image feature extraction unit 40, an image prompt generation unit 80, and a classification unit 70.
[0113] The image classification device 100 of Embodiment 5 differs from the image classification device 100 of Embodiment 3 in that the image prompt storage unit 60 is replaced by an image prompt generation unit 80, and the image is input to the image prompt generation unit 80. However, the other configurations and operations are the same as the image classification device 100 of Embodiment 1. Here, only the differences from Embodiment 3 will be explained.
[0114] Figure 27 is a diagram showing the configuration of the image prompt generation unit 80 in Embodiment 5. The image prompt generation unit 80 includes an image splitting unit 82 and an embedding unit 84.
[0115] An image is input to the image prompt generation unit 80. The image splitting unit 82 splits the image into Lpi divided images and inputs the Lpi divided images to the embedding unit 84. The embedding unit 84 generates Lpi image prompts.
[0116] The image prompt generation unit 80 outputs the generated image prompt to the image embedding data generation unit 20. The image embedding data generation unit 20 generates image embedding data with the image prompt added. The configuration and operation of the image embedding data generation unit 20 in Embodiment 5 are the same as those of the image embedding data generation unit 20 in Embodiment 3.
[0117] The image prompt generation unit 80 is assumed to have been pre-trained using the pre-trained language feature extraction unit 30 and image feature extraction unit 40, with training data associated with the class name list. The images included in the training data are different from the images to be inferred. For example, the training data associated with the class name list when the language feature extraction unit 30 and image feature extraction unit 40 were multimodally trained may be used, or training data associated with a different class name list may be used. By training the image prompt generation unit 80 under conditions different from those when the language feature extraction unit 30 and image feature extraction unit 40 were multimodally trained, it is expected that the image prompt generation unit 80 will improve its classification accuracy.
[0118] As described above, in an image classification device 100 that classifies images based on language feature vectors output by the language feature extraction unit 30 and image feature vectors output by the image feature extraction unit 40, the accuracy of classifying unlearned class names can be improved by adding an image prompt generated from the input image before the segmented image embedding data to which all position embedding data has been added.
[0119] Furthermore, in the image classification device 100 of Embodiment 5, since the image prompt is generated from the input image, there is no need to generate the image prompt in advance.
[0120] (Embodiment 6) Figure 28 is a diagram showing the configuration of the image classification device 100 of Embodiment 6. The image classification device 100 of Embodiment 6 includes a language embedding data generation unit 10, a language feature extraction unit 30, a text prompt storage unit 50, an image embedding data generation unit 20, an image feature extraction unit 40, and a classification unit 70.
[0121] In the image classification device 100 of Embodiment 2, the text prompt is input from the text prompt storage unit 50 to the language embedding data generation unit 10 and the image embedding data generation unit 20. However, in the image classification device 100 of Embodiment 6, the text prompt is input from the text prompt storage unit 50 to the language feature extraction unit 30 and the image feature extraction unit 40. The only difference is that the configuration and operation are the same as the image classification device 100 of Embodiment 2. Here, only the differences from Embodiment 2 will be explained. Note that, in this embodiment, the text prompt is input to both the language feature extraction unit 30 and the image feature extraction unit 40, but the text prompt may be input only to the language feature extraction unit 30.
[0122] Figure 29 is a diagram showing the configuration of the language feature extraction unit 30 of Embodiment 6. The language feature extraction unit 30 includes M connected transformer basic blocks (reference numerals 32-1, 32-2, ..., 32-M). Here, M = 12, but any integer greater than or equal to 2 is acceptable. The configuration of the transformer basic blocks is known and corresponds to the transformer encoder shown in Figure 1 of Non-Patent Literature 1.
[0123] Figure 30 is a diagram showing the configuration of the image feature extraction unit 40 of Embodiment 6. The image feature extraction unit 40 includes N connected transformer basic blocks (reference numerals 42-1, 42-2, ..., 42-N). Here, N=12, but any integer greater than or equal to 2 is acceptable.
[0124] Figure 31 illustrates the configuration in Embodiment 6 in which a text prompt of a class name list is input from the text prompt storage unit 50 to the language feature extraction unit 30 and the image feature extraction unit 40.
[0125] The text prompt memory unit 50 stores text prompts for a class name list, which are composed of text prompt vectors 1, 2, ..., Lpl. The text prompt vectors 1, 2, ..., Lpl are input to the transformer basic blocks 1, 2, ..., Lpl of the language feature extraction unit 30 and the transformer basic blocks 1, 2, ..., Lpl of the image feature extraction unit 40, respectively.
[0126] The input to Transformer Basic Block 1 is text prompt vector 1 followed by the class name embedding data. The input to Transformer Basic Block 2 is text prompt vector 2 followed by the output data of Transformer Basic Block 1. Similarly, the input to Transformer Basic Block i (i=2, ..., Lpl) is text prompt vector i followed by the output data of Transformer Basic Block i-1.
[0127] Here, M=N=12, and the text prompt vector i (i=1, ..., Lpl) is input to the transformer base block i of the language feature extraction unit 30 and the transformer base block i of the image feature extraction unit 40. Here, Lpl is set to 4, but it is sufficient that Lpl≦M and Lpl≦N.
[0128] By inputting text prompt vectors sequentially from the transformer basic blocks closest to the input side of the language feature extraction unit 30 and the image feature extraction unit 40, it is expected that the subsequent transformer basic blocks will process only the valid text prompt vectors.
[0129] Furthermore, since the text prompt for the class name list is generated using the language feature extraction unit 30, as well as the multimodally trained image feature extraction unit 40, the image feature extraction unit 40 can interpret the text prompt for the class name list. Therefore, by inputting the text prompt for the class name list into the transformer basic block of the image feature extraction unit 40, the transformer basic block of the image feature extraction unit 40 can execute processing according to the class name list, thereby improving classification accuracy.
[0130] (Embodiment 7) Figure 32 is a diagram showing the configuration of the image classification device 100 of Embodiment 7. The image classification device 100 of Embodiment 7 includes a language embedding data generation unit 10, a language feature extraction unit 30, a text prompt storage unit 50, an image embedding data generation unit 20, an image feature extraction unit 40, an image prompt storage unit 60, and a classification unit 70.
[0131] In the image classification device 100 of Embodiment 4, text prompts are input from the text prompt storage unit 50 to the language embedding data generation unit 10 and the image embedding data generation unit 20, and image prompts are input from the image prompt storage unit 60 to the language embedding data generation unit 10 and the image embedding data generation unit 20. However, in the image classification device 100 of Embodiment 7, text prompts are input from the text prompt storage unit 50 to the language feature extraction unit 30 and the image feature extraction unit 40, and image prompts are input from the image prompt storage unit 60 to the language feature extraction unit 30 and the image feature extraction unit 40. The only difference is that the configuration and operation are the same as the image classification device 100 of Embodiment 4. Here, only the differences from Embodiment 4 will be explained. Note that here, text prompts are input to the language feature extraction unit 30 and the image feature extraction unit 40, and image prompts are input to the language feature extraction unit 30 and the image feature extraction unit 40, but text prompts may be input only to the language feature extraction unit 30, and image prompts may be input only to the image feature extraction unit 40.
[0132] The configuration of the language feature extraction unit 30 and the image feature extraction unit 40 in Embodiment 7 is the same as that of the language feature extraction unit 30 and the image feature extraction unit 40 in Embodiment 6, with the language feature extraction unit 30 including M transformer basic blocks and the image feature extraction unit 40 including N transformer basic blocks.
[0133] Figure 33 illustrates the configuration in Embodiment 7 in which text prompts of a class name list are input from the text prompt storage unit 50 to the language feature extraction unit 30 and the image feature extraction unit 40, and image prompts are input from the image prompt storage unit 60 to the language feature extraction unit 30 and the image feature extraction unit 40.
[0134] The text prompt memory unit 50 stores text prompts for a class name list, which are composed of text prompt vectors 1, 2, ..., Lpl. The text prompt vectors 1, 2, ..., Lpl are input to the transformer basic blocks 1, 2, ..., Lpl of the language feature extraction unit 30 and the transformer basic blocks 1, 2, ..., Lpl of the image feature extraction unit 40, respectively.
[0135] The image prompt storage unit 60 stores image prompts consisting of image prompt vectors 1, 2, ..., Lpi. The image prompt vectors 1, 2, ..., Lpi are input to the transformer basic blocks 1, 2, ..., Lpi of the language feature extraction unit 30 and the transformer basic blocks 1, 2, ..., Lpi of the image feature extraction unit 40, respectively.
[0136] In Embodiment 6, text prompt vectors 1, 2, ..., Lpl constituting the text prompt were input to the transformer basic blocks 1, 2, ..., Lpl of the language feature extraction unit 30 and the transformer basic blocks 1, 2, ..., Lpl of the image feature extraction unit 40, respectively. In Embodiment 7, however, image prompt vectors 1, 2, ..., Lpi constituting the image prompt are also input to the transformer basic blocks 1, 2, ..., Lpi of the language feature extraction unit 30 and the transformer basic blocks 1, 2, ..., Lpi of the image feature extraction unit 40.
[0137] In Figure 33, for simplicity, we illustrate the relationship by assuming Lpl = Lpi = 2. While we will explain using Lpl = Lpi = 2 here, other combinations are also acceptable.
[0138] Here, the image prompt is generated using the image feature extraction unit 40, as well as the multimodally trained language feature extraction unit 30. Therefore, the language feature extraction unit 30 can interpret the image prompt. By inputting the image prompt into the transformer basic block of the language feature extraction unit 30, the transformer basic block of the language feature extraction unit 30 can perform processing according to the image, thereby improving classification accuracy.
[0139] Figure 34 illustrates another configuration in Embodiment 7, in which a text prompt of a class name list is input from the text prompt storage unit 50 to the language feature extraction unit 30 and the image feature extraction unit 40, and an image prompt is input from the image prompt storage unit 60 to the language feature extraction unit 30 and the image feature extraction unit 40.
[0140] Multiple text prompt vectors constituting the text prompts in the class name list are input to multiple transformer basic blocks of the language feature extraction unit 30 and multiple transformer basic blocks of the image feature extraction unit 40, and multiple image prompt vectors constituting the image prompts are input to multiple transformer basic blocks of the language feature extraction unit 30 and multiple transformer basic blocks of the image feature extraction unit 40. However, the transformer basic blocks of the language feature extraction unit 30 and image feature extraction unit 40 into which the text prompt vectors are input do not overlap with the transformer basic blocks of the language feature extraction unit 30 and image feature extraction unit 40 into which the image prompt vectors are input.
[0141] In other words, in the language feature extraction unit 30, the transformer base block that receives the text prompt vector is different from the transformer base block that receives the image prompt vector. Similarly, in the image feature extraction unit 40, the transformer base block that receives the text prompt vector is different from the transformer base block that receives the image prompt vector.
[0142] Figure 34 explains the case where Lpl = Lpi = 2 for simplicity, but other combinations are also acceptable.
[0143] In the example shown in Figure 34, the text prompt vectors 1 and 2 that constitute the text prompt are input to the transformer basic blocks 1 and 2 (code 32-1 and 32-2) of the language feature extraction unit 30 and the transformer basic blocks 1 and 2 (code 42-1 and 42-2) of the image feature extraction unit 40, respectively.
[0144] The image prompt vectors 1 and 2 that constitute the image prompt are input to the transformer basic blocks 3 and 4 (code 32-3, 32-4) of the language feature extraction unit 30 and the transformer basic blocks 3 and 4 (code 42-3, 42-4) of the image feature extraction unit 40, respectively.
[0145] In this way, by ensuring that the transformer base blocks that are affected by the class name list text prompt and image prompt do not overlap, the processing load of each transformer base block can be leveled out.
[0146] The various processes of the image classification device 100 described above can, of course, be implemented using hardware such as a CPU and memory, but can also be implemented using firmware stored in ROM (read-only memory) or flash memory, or software on a computer. The firmware program and software program can be recorded on a recording medium readable by a computer and provided, transmitted and received with a server via a wired or wireless network, or transmitted and received as data broadcasting on terrestrial or satellite digital broadcasting.
[0147] The present invention has been described above based on embodiments. The embodiments are illustrative, and it will be understood by those skilled in the art that various modifications are possible in combinations of their components and processing processes, and that such modifications also fall within the scope of the present invention. [Explanation of symbols]
[0148] 10 Language embedding data generation unit, 12 Text embedding unit, 14 Text prompt addition unit, 16 Image prompt addition unit, 20 Image embedding data generation unit, 22 Image splitting unit, 24 Split image embedding unit, 26 Position embedding data addition unit, 27 Image prompt addition unit, 28 Text prompt addition unit, 30 Language feature extraction unit, 40 Image feature extraction unit, 50 Text prompt storage unit, 60 Image prompt storage unit, 70 Classification unit, 80 Image prompt generation unit, 82 Image splitting unit, 84 Embedding unit, 100 Image classification device.
Claims
1. A language embedding data generation unit generates class name embedding data from a class name list containing multiple class names, An image embedding data generation unit divides an image into multiple segmented images, generates embedding data for each segmented region, and generates image embedding data from the embedding data for each segmented region. A language feature extraction unit, which includes multiple transformer base blocks, extracts language feature vectors of class names from the aforementioned class name embedding data. An image feature extraction unit, which includes multiple transformer basic blocks, extracts image feature vectors from the aforementioned image embedding data, It includes a classification unit that uses the language feature vector of the class name and the image feature vector to classify an image into one of several class names, The text prompt in the aforementioned class name list includes multiple text prompt vectors, Each of the multiple text prompt vectors is input to a different transformer base block of the language feature extraction unit, An image classification device characterized in that each of the plurality of text prompt vectors is input to a different transformer basic block of the image feature extraction unit.
2. The image prompt for the aforementioned image includes multiple image prompt vectors, Each of the aforementioned multiple image prompt vectors is input to a different transformer base block of the language feature extraction unit, The image classification apparatus according to claim 1, characterized in that each of the plurality of image prompt vectors is input to a different transformer basic block of the image feature extraction unit.
3. In the language feature extraction unit, the text prompt and the image prompt are input to different transformer base blocks. The image classification apparatus according to claim 2, characterized in that the text prompt and the image prompt are input to different transformer base blocks in the image feature extraction unit.
4. The image classification device according to any one of claims 1 to 3, characterized in that the text prompt is pre-trained using the image feature extraction unit so that the distance between image prototypes of all class names included in the class name list is large.
5. The image classification apparatus according to claim 2 or 3, characterized in that the image prompt is pre-trained using the language feature extraction unit such that the distances between the language feature vectors of all class names included in the class name list are large.
6. A language embedding data generation step that generates class name embedding data from a class name list containing multiple class names, Image embedding data generation step: Divide an image into multiple segmented images, generate embedding data for each segmented region, and generate image embedding data from the embedding data for each segmented region. A language feature extraction step that extracts language feature vectors of class names from the class name embedding data using multiple basic transformer blocks, An image feature extraction step includes extracting image feature vectors from the image embedding data using multiple basic transformer blocks, The process includes a classification step of classifying an image into one of several class names using the language feature vector of the class name and the image feature vector, The text prompt in the aforementioned class name list includes multiple text prompt vectors, Each of the multiple text prompt vectors is input to a different transformer base block in the language feature extraction step, An image classification method characterized in that each of the plurality of text prompt vectors is input to a different transformer base block in the image feature extraction step.
7. A language embedding data generation step that generates class name embedding data from a class name list containing multiple class names, Image embedding data generation step: Divide an image into multiple segmented images, generate embedding data for each segmented region, and generate image embedding data from the embedding data for each segmented region. A language feature extraction step that extracts language feature vectors of class names from the class name embedding data using multiple basic transformer blocks, An image feature extraction step includes extracting image feature vectors from the image embedding data using multiple basic transformer blocks, The computer is instructed to perform a classification step in which it classifies an image into one of several class names using the language feature vector of the class name and the image feature vector. The text prompt in the aforementioned class name list includes multiple text prompt vectors, Each of the multiple text prompt vectors is input to a different transformer base block in the language feature extraction step, An image classification program characterized in that each of the multiple text prompt vectors is input to a different transformer base block in the image feature extraction step.