Image classification device, image classification method, and image classification program
The image classification technology improves accuracy by using an associative knowledge graph to expand relevant labels and construct an image knowledge graph, addressing low classification accuracy due to missing background associations.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-09-06
- Publication Date
- 2026-03-18
AI Technical Summary
Image classification accuracy is low when there are few background objects in the image and necessary associations are missing.
An image classification device and method that utilizes an associative knowledge graph to extract and expand relevant associative labels, calculate embedding representation vectors, and construct an image knowledge graph to improve classification accuracy by incorporating high-relevance information.
Enhances image classification accuracy by supplementing information highly relevant to the subject using associative labels, even when background objects are scarce.
Smart Images
Figure 2026049277000001_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to image classification technology.
Background Art
[0002] Humans can learn the characteristics of objects and the relationships that occur between them, and classify images. In order to apply this biological idea and establish the potential relevance between objects, image knowledge graphs have been proposed (Non-Patent Document 1, Non-Patent Document 2). By utilizing the image knowledge graph, it is possible to improve the classification accuracy of images by considering the relationships between objects.
Prior Art Documents
Non-Patent Documents
[0003]
Non-Patent Document 1
Non-Patent Document 2
Summary of the Invention
Problems to be Solved by the Invention
[0004] However, in image classification using an image knowledge graph, there is a problem that the classification accuracy of the main subject of an image is low when there are few background objects in the image and necessary associations are missing.
[0005] This invention has been made in view of these circumstances, and its purpose is to provide an image classification technology that improves the classification accuracy of an image subject using information highly relevant to the subject. [Means for solving the problem]
[0006] To solve the above problems, an image classification device in one aspect of the present invention includes: an associative label extraction unit that extracts a plurality of associative labels associated with the true label of the subject of an image from an associative knowledge graph; an embedding representation vector calculation unit that calculates the embedding representation vectors of the plurality of associative labels and calculates the embedding representation vectors of a plurality of peripheral labels obtained from the image for the true label of the subject and averages them; a similarity calculation unit that calculates the cosine similarity between the embedding representation vector of each associative label and the averaged embedding representation vector of the peripheral labels; an associative label selection unit that selects the embedding representation vector of an associative label that has a high similarity to the averaged embedding representation vector of the peripheral labels; a peripheral label expansion unit that uses the selected associative labels as peripheral labels to generate a cell graph in which the subject is the center node and the peripheral labels are the side nodes, and constructs an image knowledge graph from the cell graph of the image; and a classification unit that classifies the subject of an image using the image knowledge graph.
[0007] Another aspect of the present invention is an image classification method. This method comprises: an associative label extraction step of extracting a plurality of associative labels associated with the true label of the subject of an image from an associative knowledge graph; an embedding representation vector calculation step of obtaining embedding representation vectors of a plurality of peripheral labels obtained from the image and averaging them with respect to the true label of the subject; a similarity calculation step of calculating the cosine similarity between the embedding representation vector of each associative label and the averaged embedding representation vector of the peripheral labels; an associative label selection step of selecting the embedding representation vector of an associative label that has a high similarity to the averaged embedding representation vector of the peripheral labels; a peripheral label expansion step of generating a cell graph with the subject as the center node and the peripheral labels as side nodes using the selected associative labels as peripheral labels, and constructing an image knowledge graph from the cell graph of the image; and a classification step of classifying the subject of an image using the image knowledge graph.
[0008] Furthermore, any combination of the above components, as well as conversions of the expression of the present invention between methods, apparatus, systems, recording media, computer programs, etc., are also valid embodiments of the present invention. [Effects of the Invention]
[0009] According to the present invention, it is possible to provide an image classification technology that improves the classification accuracy of an image subject using information highly relevant to the subject. [Brief explanation of the drawing]
[0010] [Figure 1] This is a diagram showing the configuration of an image classification device according to Embodiment 1. [Figure 2] Figures 2(a) and 2(b) illustrate the extension of the cell graph. [Figure 3] This is a flowchart showing the image classification procedure using the image classification device according to Embodiment 1. [Figure 4] This figure illustrates the cell graph expanded by the peripheral label expansion unit of Embodiment 2. [Figure 5]This is a flowchart showing the image classification procedure using the image classification device according to Embodiment 2. [Modes for carrying out the invention]
[0011] Referring to Non-Patent Document 1, we will explain the underlying technology of image knowledge graphs.
[0012] A biological perspective is crucial in image processing. Not only does the visual system derive over 80% of the information humans receive, but it also possesses high-speed, high-precision, and efficient image processing capabilities. In current image classification tasks, convolutional neural networks (CNNs) often focus on pixel processing, neglecting semantic relationships and the mechanisms of the human brain.
[0013] With the advancement of image analysis and processing technologies, the information contained in images is becoming increasingly complex. Humans can learn the features of objects and the relationships between them to classify images. The main concerns are how to effectively utilize the semantic relationships of categories and how to apply knowledge of biological vision to image classification. In this approach, we propose the concept of an Image Knowledge Graph (IKG) that incorporates semantic and scene associations to fully consider the relationships between objects (external and internal).
[0014] Image knowledge graphs utilize knowledge graph inference models similar to biological visual information processing models. The following four steps are taken to establish potential relationships between objects. (1) Image category extraction (2) Establishment of scene (logical) associations and semantic associations (3) Construction of an image knowledge graph (4) Refinement of image classification
[0015] (1) Image category extraction First, extract the association (related) categories. Extract the related categories from the predictions of the CNN. Each category is a node in the image knowledge graph.
[0016] For example, extract the related categories for two categories, namely computer and TV, from a large number of images of computers and a large number of images of TVs. The background information of the computer images includes a keyboard, a mouse, a computer desk, a web page, etc. The background information of the TV images includes a remote control, a sofa, a living room, children, etc. By comparing such background information, the computer and the TV can be identified (see Figure 2 of Non-Patent Document 1).
[0017] Taking into account such background categories, the objects in the image can be associated.
[0018] (2) Establish scene association and semantics association Next, divide the relationships between categories into two types of relevance according to the situation. One is semantics (meaningful) association, and the other is scene (logical) association.
[0019] Semantics association is an association based on semantic similarity. For example, a tiger and a cat have a high similarity because both belong to the same Felidae family, while a cat and a table have little similarity. Therefore, a tiger and a cat are associated by semantics association.
[0020] Scene association evaluates the similarity regarding the environment where the object exists. For example, a cat and a table, a dog and a frisbee are relevant in the scene of the image. In this way, the peripheral objects related to the main object in the scene are associated (see Figure 3 of Non-Patent Document 1).
[0021] An image knowledge graph is constructed by integrating semantic associations and scene associations (see Figure 4 in Non-Patent Document 1). By using the image knowledge graph, background information can be utilized to improve recognition accuracy when identifying objects in an image.
[0022] (3) Construction of an image knowledge graph Next, the CNN prediction results are combined with node and edge information to construct a global image knowledge graph (see Figure 5 in Non-Patent Document 1).
[0023] There are four steps to constructing a graph. (i) Obtain predicted labels from the image and split them into true labels and marginal labels. (ii) Construct a cell graph with true labels as center nodes and peripheral labels as side nodes. (iii) All related cell graphs are combined into subgraphs. Each subgraph is a main category. (iv) Combine all subgraphs to construct a global graph. The global graph contains knowledge information (weights) about objects and their relationships.
[0024] (4) Refinement of image classification Finally, we use an image knowledge graph as additional information for image classification. We refine image classification using two methods: one is to directly utilize external knowledge (such as WordNet semantic web), and the other is to utilize an image knowledge graph.
[0025] In the refinement process, each predicted label has weights obtained from the image knowledge graph and the word corpus. Next, a complete connectivity calculation is performed for all top-10 labels. Finally, the label with the highest average correlation to each predicted label is designated as top-1. Predicted labels for a single image are obtained from a CNN model. The confidence value for each label is the label weight, which is combined with association weights obtained from the knowledge graph or word corpus (see Figure 6 in Non-Patent Literature 1).
[0026] In experimental examples where image objects were classified using the aforementioned prerequisite techniques, a problem arose where, if there were few background objects in the image and necessary associations were missing, the correct label of the object could not be predicted, resulting in a high failure rate for refining the image classification (see Figure 13 in Non-Patent Literature 1).
[0027] Next, embodiments of the present invention will be described. The image classification technology of the embodiments can improve classification accuracy by supplementing relevant information when there are few background objects in an image.
[0028] (Embodiment 1) Figure 1 is a diagram showing the configuration of an image classification device 100 according to Embodiment 1. The image classification device 100 includes an associative label extraction unit 10, an embedded representation vector calculation unit 20, a similarity calculation unit 30, an associative label selection unit 40, a peripheral label expansion unit 50, a classification unit 60, an associative knowledge graph storage unit 70, and an image knowledge graph storage unit 80.
[0029] The associative knowledge graph memory unit 70 stores associative knowledge graphs between words. SWOW (Small World of Words), an example of an associative knowledge graph, is a knowledge graph obtained from crowdsourced word associations and is a dataset of English word associations. The associative knowledge graph is generated, for example, when a cue word (e.g., "camp") is presented, by associating it with words spontaneously generated by humans (e.g., "hiking", "nature", etc.) (see Figure 1 in Non-Patent Literature 2).
[0030] Conventionally, when background information for an image is scarce and there are few surrounding labels for the subject (true label), necessary associations are missing, resulting in low classification accuracy. However, the image classification device 100 uses an associative knowledge graph to expand information highly related to the subject (true label) through association.
[0031] The associative label extraction unit 10 extracts multiple associative labels associated with the subject (true label) from the associative knowledge graph.
[0032] The embedding representation vector calculation unit 20 calculates an embedding representation vector for all extracted associative labels. The embedding representation vector calculation unit 20 also calculates an embedding representation vector for all surrounding labels obtained from the image for the subject (true label) and averages them.
[0033] The similarity calculation unit 30 calculates the cosine similarity between the embedding representation vector of each associative label and the averaged embedding representation vector of the surrounding labels.
[0034] The associative label selection unit 40 selects the top few embedding representation vectors of associative labels that have a high similarity to the averaged embedding representation vectors of surrounding labels.
[0035] The peripheral label extension unit 50 uses the top few selected associative labels as peripheral labels to generate a cell graph in which the main object is the center node and the peripheral labels are the side nodes. The peripheral label extension unit 50 constructs an image knowledge graph from the cell graph of the image and stores it in the image knowledge graph storage unit 80.
[0036] The classification unit 60 uses the image knowledge graph stored in the image knowledge graph storage unit 80 to classify and label the subject of the image.
[0037] Figures 2(a) and 2(b) illustrate the extension of the cell graph.
[0038] Figure 2(a) is a cell graph generated by obtaining surrounding labels from the background information of an image. Surrounding labels are associated with the main subject (true label), but the number of surrounding labels is small when the background information of the image is sparse.
[0039] Figure 2(b) shows the cell graph expanded by the peripheral label expansion unit 50. By selecting associative labels with a high similarity to the peripheral labels and using them as peripheral labels, the peripheral labels are expanded, resulting in an increase in the number of peripheral labels for the subject (true label).
[0040] Figure 3 is a flowchart showing the image classification procedure by the image classification device 100 according to Embodiment 1.
[0041] The associative label extraction unit 10 refers to the associative knowledge graph stored in the associative knowledge graph storage unit 70 and extracts multiple associative labels associated with the true label of the subject of the image (S10).
[0042] The embedding representation vector calculation unit 20 calculates the embedding representation vector for each extracted associative label (S20). The embedding representation vector calculation unit 20 obtains multiple surrounding labels from the image for the true label of the main subject of the image, calculates the embedding representation vector for each surrounding label, and averages them (S30).
[0043] The similarity calculation unit 30 calculates the cosine similarity between the embedding representation vector of each associative label and the averaged embedding representation vector of the surrounding labels (S40).
[0044] The associative label selection unit 40 selects multiple associative label embedding vectors in order of their similarity to the averaged surrounding label embedding vectors (S50).
[0045] The peripheral label setting unit 50 uses the selected associative labels as peripheral labels to generate a cell graph with the main object as the center node and the peripheral labels as side nodes (S60). The peripheral label setting unit 50 constructs an image knowledge graph from the image cell graph and stores it in the image knowledge graph storage unit 80 (S70).
[0046] The classification unit 60 classifies the subject of the image by referring to the image knowledge graph stored in the image knowledge graph storage unit 80 (S80).
[0047] (Embodiment 2) The image classification device 100 according to Embodiment 2 includes the same configuration as the image classification device 100 according to Embodiment 1, but the operation of the peripheral label extension unit 50 differs from that of Embodiment 1. Only the differences will be explained here.
[0048] The peripheral label extension unit 50 integrates the selected associative labels into the image knowledge graph as associative associations. In this process, the knowledge information (weights) regarding the associations between objects uses the similarity values calculated by the similarity calculation unit 30. Furthermore, since the reliability of associative information is lower compared to scene associations (factual information contained in the image), its weight is relatively reduced.
[0049] Specifically, the peripheral label extension unit 50 uses the top few most similar associative labels selected by the associative label selection unit 40 as peripheral labels by assigning weights based on their similarity, generating a cell graph in which peripheral labels are connected to the subject, and constructing an image knowledge graph from the cell graph.
[0050] Figure 4 illustrates the cell graph expanded by the peripheral label expansion unit 50 of Embodiment 2. In the cell graph, the weights of the edges connecting the subject and the associated label are set based on similarity.
[0051] Here, in the cell graph, the edge weights of associative labels are made smaller than the edge weights of peripheral labels. This is because peripheral labels are derived from factual information contained in the image and are therefore highly reliable, while associative labels are derived from associative information in the associative knowledge graph and are therefore less reliable.
[0052] Furthermore, the greater the similarity between associative labels and their surrounding labels, the greater the edge weight of the associative label. This is because higher similarity to surrounding labels indicates higher reliability.
[0053] Figure 5 is a flowchart showing the image classification procedure by the image classification device 100 according to Embodiment 2. Steps S10, S20, S30, S40, S50, S70, and S80 of the image classification procedure by the image classification device 100 according to Embodiment 2 are the same as steps S10, S20, S30, S40, S50, S70, and S80 of the image classification procedure by the image classification device 100 according to Embodiment 1 shown in Figure 3. The only difference is that step S60 of the image classification device 100 according to Embodiment 1 is replaced by step S62 in the image classification procedure by the image classification device 100 according to Embodiment 2. Here, the explanation of the identical steps is omitted, and only the differing step S62 will be explained.
[0054] The surrounding label setting unit 50 assigns weights to the selected associative labels based on their similarity and uses them as surrounding labels to generate a cell graph with the main label as the center node and the surrounding labels as side nodes (S62). Here, it is preferable that the surrounding label setting unit 50 sets a larger edge weight for the associative label as the degree of similarity with the surrounding label is higher in the cell graph, and sets the edge weight of the associative label to be smaller than the edge weight of the surrounding label.
[0055] According to the image classification device 100 of Embodiments 1 and 2, the classification accuracy of the subject can be improved by expanding information highly related to the subject of the image through association. According to the image classification device 100 of Embodiment 2, the associative information can be weighted based on the reliability of the associative information and associated with the subject of the image.
[0056] The various processes of the image classification device 100 described above can, of course, be implemented using hardware such as a CPU and memory, but can also be implemented using firmware stored in ROM (read-only memory) or flash memory, or software on a computer. The firmware program and software program can be recorded on a recording medium readable by a computer and provided, transmitted and received with a server via a wired or wireless network, or transmitted and received as data broadcasting on terrestrial or satellite digital broadcasting.
[0057] The present invention has been described above based on embodiments. The embodiments are illustrative, and it will be understood by those skilled in the art that various modifications are possible in combinations of their components and processing processes, and that such modifications also fall within the scope of the present invention. [Explanation of Symbols]
[0058] 10 Associative label extraction unit, 20 Embedding representation vector calculation unit, 30 Similarity calculation unit, 40 Associative label selection unit, 50 Surrounding label expansion unit, 60 Classification unit, 70 Associative knowledge graph storage unit, 80 Image knowledge graph storage unit, 100 Image classification device.
Claims
1. An associative label extraction unit extracts multiple associative labels from an associative knowledge graph that are associated with the true label of the main subject of the image, An embedding representation vector calculation unit calculates the embedding representation vectors of the multiple associative labels, calculates the embedding representation vectors of the multiple surrounding labels obtained from the image and averages them with respect to the true label of the subject, A similarity calculation unit that calculates the cosine similarity between the embedding representation vector of each associative label and the averaged embedding representation vector of the surrounding labels, An associative label selection unit that selects the embedding representation vector of an associative label that has a high similarity to the embedding representation vector of the averaged peripheral labels, A peripheral label extension unit generates a cell graph using selected associative labels as peripheral labels, with the main subject as the center node and peripheral labels as side nodes, and constructs an image knowledge graph from the cell graph of the image. An image classification device characterized by comprising a classification unit that classifies the subject of an image using an image knowledge graph.
2. The image classification apparatus according to claim 1, characterized in that the peripheral label extension unit uses the highly similar associative labels selected by the associative label selection unit as peripheral labels by assigning weights based on similarity, thereby generating a cell graph in which peripheral labels are connected to the subject.
3. The image classification apparatus according to claim 2, characterized in that the peripheral label extension unit sets the edge weight of the associative label to be smaller than the edge weight of the peripheral label in the cell graph.
4. An associative label extraction step extracts multiple associative labels from an associative knowledge graph that are associated with the true label of the subject of the image, An embedding representation vector calculation step involves obtaining the embedding representation vectors of the multiple associative labels, and averaging the embedding representation vectors of the multiple surrounding labels obtained from the image with respect to the true label of the subject, A similarity calculation step that calculates the cosine similarity between the embedding representation vector of each associative label and the averaged embedding representation vector of the surrounding labels, An associative label selection step that selects an associative label embedding representation vector that has a high similarity to the averaged peripheral label embedding representation vector, A peripheral label extension step is performed to generate a cell graph using the selected associative labels as peripheral labels, with the subject as the center node and the peripheral labels as side nodes, and to construct an image knowledge graph from the cell graph of the image. An image classification method characterized by comprising a classification step of classifying the subject of an image using an image knowledge graph.
5. An associative label extraction step extracts multiple associative labels from an associative knowledge graph that are associated with the true label of the subject of the image, An embedding representation vector calculation step involves obtaining the embedding representation vectors of the multiple associative labels, and averaging the embedding representation vectors of the multiple surrounding labels obtained from the image with respect to the true label of the subject, A similarity calculation step that calculates the cosine similarity between the embedding representation vector of each associative label and the averaged embedding representation vector of the surrounding labels, An associative label selection step that selects an associative label embedding representation vector that has a high similarity to the averaged peripheral label embedding representation vector, A peripheral label extension step is performed to generate a cell graph using the selected associative labels as peripheral labels, with the subject as the center node and the peripheral labels as side nodes, and to construct an image knowledge graph from the cell graph of the image. An image classification program characterized by having a computer perform a classification step of classifying the subject of an image using an image knowledge graph.