Image processing device, image processing method and program

Abstract

To detect areas corresponding to objects from images with higher accuracy.SOLUTION: An image processing apparatus comprises an image reception unit, a query reception unit, a word heat map calculation unit, a weight calculation unit, a query heat map calculation unit, a detection unit, and an output control unit. The image reception unit receives input of images. The query reception unit receives input of a query containing one or more words. The word heat map calculation unit calculates, for each word, a word heat map indicating the degree of association between each partial area included in the image and the word. The weight calculation unit calculates the weight of each word. The query heat map calculation unit calculates a query heat map indicating the association between each partial area included in the image and the query based on the word heat map and the weights. The detection unit detects an image area corresponding to an object related to the query from the image based on the query heat map. The output control unit outputs the image area.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] There are known techniques for detecting image regions of objects that match text (query) entered as search criteria from an input image. For example, a technique called weakly supervised Visual Grounding (VG) uses a learning model that does not require teaching of rectangles corresponding to the image region of an object, and when an image and query are presented, an image region (rectangle) corresponding to the query is detected. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Li et al. “Align before Fuse: Vision and Language Representation Learning with Momentum Distillation”, NeurIPS 2021. [Non-patent document 2] Selvaraju et.al., “Grad-cam: Visual explanations from deep networks via gradient-based localization”, ICCV, pp. 618-626 2017. [Non-patent document 3] Ren et.al., "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 6, pp. 1137-1149, 1 June 2017. Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to provide an image processing device, an image processing method, and a program that can detect an area corresponding to an object from an image with higher accuracy. [Means for solving the problem]

[0005] An image processing device according to an embodiment includes an image receiving unit, a query receiving unit, a word heat map calculation unit, a weight calculation unit, a query heat map calculation unit, a detection unit, and an output control unit. The image receiving unit receives an input of an image. The query receiving unit receives an input of a query including one or more words. The word heat map calculation unit calculates, for each word, a word heat map indicating the degree of association between each partial region included in the image and the word. The weight calculation unit calculates a weight for each word. The query heat map calculation unit calculates, based on the word heat map and the weights, a query heat map indicating the degree of association between each partial region included in the image and the query. The detection unit detects, from the image, an image region corresponding to an object related to the query based on the query heat map. The output control unit outputs the image regions. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a block diagram of an image processing apparatus according to an embodiment. [Figure 2] FIG. 2 is a diagram showing an outline of processing by an image processing device. [Figure 3] FIG. 10 is a diagram showing an example of a process for learning a model. [Figure 4] FIG. 10 is a diagram showing an example of a word heatmap. [Figure 5] FIG. 10 is a diagram showing an example of the result of dependency structure analysis. [Figure 6] 4 is a flowchart of image processing according to an embodiment. [Figure 7] FIG. 10 is a diagram showing an example of an output by an output control unit. [Figure 8]FIG. 1 is a hardware configuration diagram of an image processing apparatus according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0007] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of an image processing apparatus according to the present invention will be described in detail below with reference to the accompanying drawings.

[0008] Weakly supervised VG can be implemented, for example, by the following procedure. A pre-trained text-image matching model (such as a weakly supervised VG model) is used to calculate a word heatmap for each word in the query. The word heatmap is information that indicates the degree of association (relevance) between a subregion (e.g., one or more pixels) in an image and the word. Add up the word heatmaps for each word to calculate a heatmap for the entire query (query heatmap). Detect candidate image regions corresponding to objects from the image. From the detected candidates, identify candidates that match the peaks in the query heat map, and output the identified candidates as image regions corresponding to the object.

[0009] In the above procedure, all words in the query are treated equally. In other words, no distinction is made between words corresponding to the target object (main object) that should be searched for in the query and words corresponding to objects other than the target object (sub-objects). This can lead to erroneous detection of sub-objects.

[0010] In the following embodiment, by analyzing the dependency relationships between words, it is possible to distinguish between words corresponding to main objects and words corresponding to sub-objects. Different weights are then assigned to the distinguished words. Furthermore, the query heat map is redefined as a weighted sum of word heat maps. For example, taking into account the relationships between words, the weights of the word heat map for words corresponding to the main object are emphasized (increased). This makes it easier to detect the main object. In other words, it becomes possible to more accurately detect areas corresponding to objects from images using weakly supervised VG or the like.

[0011] Fig. 1 is a block diagram showing an example of the configuration of an image processing device 100 according to this embodiment. As shown in Fig. 1, the image processing device 100 includes an image receiving unit 101, a query receiving unit 102, a word heat map calculation unit 103, a weight calculation unit 104, a query heat map calculation unit 105, a detection unit 106, an output control unit 107, a storage unit 121, and a display 122.

[0012] The image processing device 100 uses the above-mentioned units to detect and output an image area (rectangle) that matches the input query from the input image. Fig. 2 is a diagram showing an overview of the processing by the image processing device 100.

[0013] An image 201 and a query 202 are input to the image processing device 100. Although colors are not shown in Fig. 2, the image 201 is assumed to be an image in which, for example, a woman wearing a blue jacket is positioned on the right side and a person wearing clothes other than blue is positioned on the left side. The image processing device 100 detects an image region 212 that matches the query 202, "woman in blue jacket," and displays it together with the image 211 on, for example, the display 122.

[0014] Each of the above units will be described in detail below with reference to Fig. 1. The storage unit 121 stores various information used in the image processing device 100. For example, the storage unit 121 stores an input image, an input query, a model used to calculate a heat map, and the like.

[0015] The storage unit 121 can be configured from any commonly used storage medium such as a flash memory, a memory card, a RAM (Random Access Memory), an HDD (Hard Disk Drive), and an optical disk.

[0016] Here, an example of a model that can be used in this embodiment will be described. Fig. 3 is a diagram showing an example of a process for learning a model. Process 301 shows an example of a process for learning a VG model using supervised VG. Process 302 shows an example of a process for learning a VG model using weakly supervised VG.

[0017] In both processes, a VG model is trained using images containing a dog and a query representing a "jumping dog." In supervised VG, a rectangle 311 is specified in the image, but in supervised VG, no rectangle instruction is required. In weakly supervised VG, since no rectangle instruction is required, a direct mapping from the query to an image region cannot be learned. Instead, the target rectangle is indirectly estimated using a heat map.

[0018] 1, the display 122 is a display device for displaying various types of information used in the image processing device 100. The display 122 is realized by, for example, a liquid crystal display, a touch panel, or the like.

[0019] The image receiving unit 101 receives an input of an image. Any method for inputting an image may be used, but examples of such methods include receiving an image from an external device via a network, and reading out a specified image from images stored in a storage medium such as the storage unit 121.

[0020] The query receiving unit 102 receives an input of a query including one or more words. The query may be input in any manner, including, for example, receiving the query from an external device via a network, or receiving the text of the query input by a user using an input device such as a keyboard.

[0021] The word heat map calculation unit 103 calculates a word heat map for each word included in the query, for example, using a VG model that has been trained in advance and stored in the storage unit 121. The words included in the query can be extracted using a technique such as morphological analysis. The word heat map may be calculated by any method, but for example, GradCAM (Gradient-weighted Class Activation Mapping, Non-Patent Document 2) can be used.

[0022] Fig. 4 is a diagram showing an example of a calculated word heatmap. Fig. 4 shows an example of a word heatmap calculated for some words (woman, working, computer, desk) included in the query, for example, "the woman is working on her computer at the desk." Note that Fig. 4 shows an example of a heatmap in which the degree of association between a partial region and a word is represented by the density of black dots. The degree of association may also be represented by different colors, etc.

[0023] Images 401-404 are example images on which word heat maps calculated for "woman," "working," "computer," and "desk," respectively, are superimposed. Images 401-404 include word heat maps with peaks 411-414. A peak is a region that includes an area with the highest degree of association with a word. For example, in image 401, peak 411 exists in the area near a person's face, which is an area associated with "woman."

[0024] 1, the weight calculation unit 104 calculates the weight of each word included in the query. First, the weight calculation unit 104 extracts, from the query, one or more words WA (first words) corresponding to the main object and one or more words WB (second words) corresponding to the sub-object, for example, by using one of the following two extraction methods:

[0025] (Extraction method M1) The weight calculation unit 104 searches the query for specific words that are predetermined as words that connect the main object and the sub-object. The specific words include, for example, prepositions such as "in," "holding," "carry," "with," "on," "by," and "near." If the query includes multiple specific words, the weight calculation unit 104 searches for any one of the multiple specific words (for example, the specific word that appears first).

[0026] The weight calculation unit 104 extracts one or more words that appear before the specific word in the query and one or more words that appear after the specific word as the words WA and WB, respectively.

[0027] (Extraction method M2) The weight calculation unit 104 obtains a tree structure representing the dependency structure between multiple words included in the query based on the dependency structure analysis. The weight calculation unit 104 extracts the word corresponding to the root of the tree structure and words related to this word as the word WA, and extracts words other than the word WA as the word WB.

[0028] For example, the weight calculation unit 104 detects words (words WA) related to the main object from the query using dependency analysis of spaCy, an open-source natural language processing library. The words related to the main object may include not only the words corresponding to the root of the tree structure, but also, for example, words that modify the words.

[0029] Dependency relationships in a sentence refer to the dependency relationships between words and phrases. Dependencies refer to modifying / modified relationships and dependency relationships. Analyzing the dependency relationships in a sentence is called dependency structure analysis. Dependency structures are often expressed as tree structures. This tree structure has words and phrases as nodes. Phrases and words are expressed as parent-child relationships of modifying / modified.

[0030] Figure 5 shows an example of the results of dependency structure analysis. Figure 5 shows an example of the results of dependency structure analysis for the query "suitcase with cat on it." In this example, "suitcase" is the word corresponding to the root, and is extracted as the word WA. If the query contains a word that modifies "suitcase," such as "white suitcase," this word is also extracted as the word WA. Words other than the word WA are extracted as responsible WBs.

[0031] Returning to FIG. 1, the weight calculation unit 104 calculates different weights between the word WA and the word WB. For example, the weight calculation unit 104 calculates the weight of each word so that the weight of the word WA calculated by the extraction method M1 or the extraction method M2 is greater than the weight of the word WB. For example, the weight calculation unit 104 calculates a weight m(w)=α (α>1) for all the words WA, and a weight m(w)=1 for the word WB. In this way, the weight calculation unit 104 emphasizes the weight of the word WA corresponding to the main object.

[0032] The query heat map calculation unit 105 calculates a query heat map based on the word heat map and the weights. The query heat map corresponds to information indicating the degree of association between each partial region included in the image and the query. For example, the query heat map calculation unit 105 calculates the query heat map by a weighted sum of the word heat maps using the weights.

[0033] The following formula (1) shows an example of calculating a query heatmap Heatmap(q). Note that q represents the query. Heatmap(w) represents the word heatmap for word w. The sum (Σ) means to find the sum over all words w. Heatmap(q)=Σ(m(w)×Heatmap(w))...(1)

[0034] As described above, the weight value of the word corresponding to the main object is calculated to be large. Therefore, the query heatmap is calculated to emphasize the partial area related to the main object. As a result, the main object becomes easier to detect.

[0035] The detection unit 106 detects an image region corresponding to an object related to the query from the input image based on the query heat map. For example, the detection unit 106 first detects one or more candidate image regions corresponding to one or more objects from the image. As a method for detecting the candidate image regions, for example, a method using Region Proposal Networks (Non-Patent Document 3) can be applied.

[0036] The detection unit 106 detects, from among one or more candidate image regions, a candidate that includes a partial region (e.g., a pixel) that the query heat map indicates has a high degree of relevance to the query as an image region corresponding to an object related to the query. For example, the detection unit 106 detects, from among the candidate image regions, a candidate that best matches a peak in the query heat map.

[0037] The method for detecting an image region is not limited to the above, and any method using a query heat map may be used. For example, the detection unit 106 may detect an image region using a model that is trained to input a query heat map and output an image region.

[0038] The output control unit 107 controls the output of various information used in the image processing device 100. For example, the output control unit 107 outputs (displays) the image area detected by the detection unit 106 to the display 122.

[0039] The image area to be output may be only the image area corresponding to the main object, but it may also be possible to output image areas corresponding to sub-objects.

[0040] In this case, the weight calculation unit 104 calculates the weights for the words WA and WB using a plurality of patterns in which the weights are different from each other. An example using two patterns is shown below. Pattern P1: Weight m(w) of word WA = α (α>1), Weight m(w) of word WB = 1 Pattern P2: Weight m(w) of word WB = α (α>1), Weight m(w) of word WA = 1

[0041] The query heat map calculation unit 105 calculates multiple query heat maps corresponding to each of the multiple patterns. The detection unit 106 detects multiple image regions corresponding to each of the multiple query heat maps. The output control unit 107 outputs the multiple image regions detected in this manner, corresponding to each of the multiple patterns. The output control unit 107 may output the multiple image regions in different modes. For example, the output control unit 107 displays a rectangle surrounding an image region corresponding to a main object and a rectangle surrounding an image region corresponding to a sub-object in different colors. The output mode is not limited to color, and may be any other mode, such as line thickness, explanatory text, or a blinking method of the rectangle.

[0042] Each of the above units (image receiving unit 101, query receiving unit 102, word heat map calculation unit 103, weight calculation unit 104, query heat map calculation unit 105, detection unit 106, and output control unit 107) is realized, for example, by one or more processors. For example, each of the above units may be realized by having a processor such as a CPU (Central Processing Unit) execute a program, i.e., by software. Each of the above units may be realized by a processor such as a dedicated IC (Integrated Circuit), i.e., by hardware. Each of the above units may be realized by using a combination of software and hardware. When multiple processors are used, each processor may realize one of the units, or two or more of the units.

[0043] Next, a description will be given of image processing by the image processing device 100 according to this embodiment. Fig. 6 is a flowchart showing an example of image processing according to this embodiment.

[0044] The image receiving unit 101 receives an input of an image, and the query receiving unit 102 receives an input of a query (step S101). The word heat map calculation unit 103 calculates a word heat map for each word included in the query using the input image (step S102).

[0045] The weight calculation unit 104 calculates a weight for each word included in the query (step S103). The query heat map calculation unit 105 calculates a query heat map by a weighted sum using the word heat maps and the corresponding weights (step S104).

[0046] The detection unit 106 detects an image region corresponding to a peak in the query heat map from the input image (step S105). The output control unit 107 outputs the detected image region to, for example, the display 122 (step S106), and ends the image processing.

[0047] Fig. 7 is a diagram showing an example of output by the output control unit 107. Fig. 7 shows an example of output when an image 701 and a query 702 "suitcase with cat on it" are input. Note that "suitcase" is a word corresponding to a target object (main object) to be searched for by the query 702, and "cat" is a word corresponding to an object (sub-object) other than the target object.

[0048] The left side of the arrow shows an example of output using a query heat map calculated without using the method of this embodiment (without using word weights). In this case, the peak of the query heat map is near the cat's head, so a rectangle 712 surrounding the cat is output as the detected image area. Note that rectangle 711 corresponds to the image area corresponding to the main object ("suitcase") that should originally be output.

[0049] The right arrow shows an example of output using a query heat map calculated using word weights according to the method of this embodiment. According to this embodiment, the weight of the word "suitcase" corresponding to the main object is calculated to be large, and as a result, the peak of the query heat map is located near the corner of the suitcase. This allows the rectangle 721 indicating the image area corresponding to the main object ("suitcase") to be correctly detected.

[0050] In this way, the image processing device of this embodiment analyzes word dependencies, calculates the weight of each word, and calculates a query heat map as a weighted sum of word heat maps. By using the query heat map calculated in this way, it is possible to detect areas corresponding to objects in an image with higher accuracy.

[0051] Next, the hardware configuration of the image processing device according to the embodiment will be described with reference to Fig. 8. Fig. 8 is an explanatory diagram showing an example of the hardware configuration of the image processing device according to the embodiment.

[0052] The image processing device according to the embodiment includes a control device such as a CPU 51, a storage device such as a ROM (Read Only Memory) 52 and a RAM 53, a communication I / F 54 that connects to a network and communicates, and a bus 61 that connects each part.

[0053] The image processing device may be physically configured as one piece of hardware (such as a personal computer or a server), or may be physically configured as a combination of two or more pieces of hardware. The image processing device may also be constructed as a server in a cloud environment.

[0054] The programs executed by the image processing apparatus according to the embodiment are provided in advance in the ROM 52 or the like.

[0055] The program executed by the image processing device according to the embodiment may be configured to be provided as a computer program product by being recorded in an installable or executable file format on a computer-readable recording medium such as a CD-ROM (Compact Disk Read Only Memory), a flexible disk (FD), a CD-R (Compact Disk Recordable), or a DVD (Digital Versatile Disk).

[0056] Furthermore, the program executed by the image processing device according to the embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Also, the program executed by the image processing device according to the embodiment may be provided or distributed via a network such as the Internet.

[0057] The program executed by the image processing device according to the embodiment can cause a computer to function as each unit of the image processing device described above. In this computer, the CPU 51 can read the program from a computer-readable storage medium onto a main storage device and execute the program.

[0058] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0059] 100 Image processing device 101 Image Reception Department 102 Query Reception Unit 103 Word heatmap calculation section 104 Weight calculation unit 105 Query heatmap calculation part 106 Detector 107 Output control section 121 Storage section 122 Display

Claims

1. an image receiving unit that receives an input of an image; a query receiving unit that receives an input of a query including a plurality of words; a word heat map calculation unit that calculates, for each of the words, a word heat map that indicates the degree of association between each partial region included in the image and the word; a weight calculation unit that calculates a weight for each of the words; a query heat map calculation unit that calculates a query heat map indicating a degree of association between each partial region included in the image and the query based on the word heat map and the weights; a detector that detects, from the image, an image region corresponding to an object related to the query based on the query heatmap; an output control unit that outputs the image area, The weight calculation unit detecting a specific word in the query that is a predetermined preposition, and calculating different weights between one or more first words that appear before the specific word in the query and one or more second words that appear after the specific word in the query; or obtaining a tree structure representing a dependency structure between a plurality of words included in the query based on a dependency structure analysis, and calculating different weights between a first word related to a word corresponding to a root of the tree structure and a second word other than the first word; Image processing device.

2. the weight calculation unit calculates the weight of the first word to be greater than the weight of the second word; The image processing device according to claim 1 .

3. the query heat map calculation unit calculates the query heat map by a weighted sum of the word heat maps using the weights; The image processing device according to claim 1 .

4. the detection unit detects, from the image, one or more candidate image regions corresponding to one or more objects, and detects, from among the one or more candidate image regions, a candidate that includes the partial region that is indicated by the query heat map to have a high degree of relevance to the query, as an image region corresponding to the object related to the query; The image processing device according to claim 1 .

5. the detection unit detects the image regions using a model trained to input the query heatmap and output the image regions corresponding to objects related to the query; The image processing device according to claim 1 .

6. the weight calculation unit calculates the weights using a plurality of patterns in which the weights of the words are different from one another; the query heat map calculation unit calculates a plurality of the query heat maps corresponding to the plurality of patterns, respectively; the detection unit detects a plurality of the image regions corresponding to the plurality of the query heatmaps, the output control unit outputs a plurality of the image regions. The image processing device according to claim 1 .

7. An image processing method executed by an image processing device, comprising: an image receiving step of receiving an input of an image; a query receiving step of receiving an input of a query including a plurality of words; a word heat map calculation step of calculating, for each of the words, a word heat map indicating a degree of association between each partial region included in the image and the word; a weight calculation step of calculating a weight for each of the words; a query heat map calculation step of calculating a query heat map indicating a degree of association between each partial region included in the image and the query based on the word heat map and the weights; detecting, based on the query heatmap, image regions from the image corresponding to objects related to the query; an output control step of outputting the image area, The weight calculation step detecting a specific word in the query that is a predetermined preposition, and calculating different weights between one or more first words that appear before the specific word in the query and one or more second words that appear after the specific word in the query; or obtaining a tree structure representing a dependency structure between a plurality of words included in the query based on a dependency structure analysis, and calculating different weights between a first word related to a word corresponding to a root of the tree structure and a second word other than the first word; Image processing methods.

8. On the computer, an image receiving step of receiving an input of an image; a query receiving step of receiving an input of a query including a plurality of words; a word heat map calculation step of calculating, for each of the words, a word heat map indicating a degree of association between each partial region included in the image and the word; a weight calculation step of calculating a weight for each of the words; a query heat map calculation step of calculating a query heat map indicating a degree of association between each partial region included in the image and the query based on the word heat map and the weights; detecting, based on the query heatmap, image regions from the image corresponding to objects related to the query; an output control step of outputting the image area; The weight calculation step detecting a specific word in the query that is a predetermined preposition, and calculating different weights between one or more first words that appear before the specific word in the query and one or more second words that appear after the specific word in the query; or obtaining a tree structure representing a dependency structure between a plurality of words included in the query based on a dependency structure analysis, and calculating different weights between a first word related to a word corresponding to a root of the tree structure and a second word other than the first word; program.

Images