Image retrieval device, image retrieval method, and storage medium

Abstract

To provide an image retrieval device, an image retrieval method, and a storage medium capable of performing image retrieval with high accuracy.SOLUTION: An image retrieval apparatus 1X is provided with first acquisition means 30X, second acquisition means 33X, integration means 342X, and retrieval means 36X. The first acquisition means 30X acquires input information related to retrieval. The second acquisition means 33X acquires object area information relating to an area of an object included in an image of an image database to be searched. The integration means 342X calculates an image feature value obtained by integrating a local feature value extracted from the image and the object area information as the feature value of the image. The retrieval means 36X retrieves an image related to the input information from the image database based on the similarity between the image feature amount and the feature amount of the input information.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present disclosure relates to the technical fields of an image search device, an image search method, and a storage medium for searching for images. [Background technology]

[0002] Systems that perform image searches based on information entered by a user are known. For example, Patent Document 1 discloses an image search system in which a server device acquires images taken by multiple cameras and registers them in an image database, and when a user enters text such as a keyword using a terminal device, the system searches for or extracts images corresponding to the entered text. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7366204 Summary of the Invention [Problem to be solved by the invention]

[0004] In an image search system that converts an image into features and searches for images based on the features, if the area of ​​an object in the image is small, information about the object may be lost when the image is converted into features, resulting in low search accuracy for the object.

[0005] In view of the above-mentioned problems, one object of the present disclosure is to provide an image search device, an image search method, and a storage medium that are capable of performing image searches with high accuracy. [Means for solving the problem]

[0006] One aspect of the image search device is a first acquisition means for acquiring input information related to a search; a second acquisition means for acquiring object region information relating to a region of an object included in an image in the image database that is the target of the search; an integration means for integrating the local feature extracted from the image and the object region information to calculate an image feature as a feature of the image; a search means for searching the image database for an image related to the input information based on the similarity between the image feature amount and the feature amount of the input information; The image search device has the following features.

[0007] One aspect of the image search method includes: The computer Get search input information, Obtaining object region information relating to the region of an object included in an image of the image database to be searched; calculating an image feature amount obtained by integrating the local feature amount extracted from the image and the object region information as a feature amount of the image; searching the image database for images related to the input information based on the similarity between the image feature amount and the feature amount of the input information; This is an image search method.

[0008] One aspect of the storage medium is Get search input information, Obtaining object region information relating to the region of an object included in an image of the image database to be searched; calculating an image feature amount obtained by integrating the local feature amount extracted from the image and the object region information as a feature amount of the image; The storage medium stores a program that causes a computer to execute a process of searching the image database for images related to the input information based on the similarity between the image features and the features of the input information. [Effects of the Invention]

[0009] As an example of an effect of the present disclosure, it becomes possible to perform image searches with high accuracy. [Brief explanation of the drawings]

[0010] [Figure 1] 1 shows a schematic configuration of an image search system. [Figure 2] 1 shows the hardware configuration of an image search device. [Figure 3] 2 is an example of a functional block of an image search device. [Figure 4] FIG. 10 is a diagram illustrating a schematic flow of a process for generating image features. [Figure 5] FIG. 10 is a diagram showing an outline of correction of an original query in a second mode. [Figure 6] FIG. 10 is a diagram illustrating an overview of a second selection example of an original query to be corrected. [Figure 7] 1 is an example of a flowchart illustrating an outline of a process executed by an image search device. [Figure 8] 3 is an example of a functional block of an image feature extraction unit. [Figure 9] 2 is an example of a functional block of an image search device. [Figure 10] FIG. 1 is a block diagram of an image search device. [Figure 11] 10 is a flowchart illustrating an example of a processing procedure of an image search device. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of an image search device, an image search method, and a storage medium will be described with reference to the drawings.

[0012] First Embodiment (1) System Configuration Figure 1 shows the schematic configuration of an image retrieval system 100. The image retrieval system 100 searches for images related to search input information, which is information entered for the search, and displays the image search results. The image retrieval system 100 mainly includes an image retrieval device 1, a storage device 2, a display device 3, and an input device 4.

[0013] The image retrieval device 1 searches for images in an image database (DB) 21 stored in the storage device 2 based on search input information specified by an input signal supplied from the input device 4, and displays information indicating the search results on the display device 3. In this case, the image retrieval device 1 searches for images based on the similarity between features extracted from each image in the image DB 21 and features extracted from the search input information, and displays information indicating the inspection results on the display device 3. Hereinafter, the features of each image used to calculate the above-mentioned similarity to generate search results will be referred to as "image features," and the features of the search input information will be referred to as "search input features." Note that "features" are quantitative representations of features and are expressed in a predetermined tensor format.

[0014] The storage device 2 is a memory that stores various information necessary for the processing of the image search device 1, and includes an image DB 21.

[0015] The image DB 21 is a database of images that are the target (i.e., the search range) of image search by the image search device 1. Hereinafter, each image registered in the image DB 21 will also be referred to as a "candidate image." A candidate image includes the regions of one or more objects.

[0016] The storage device 2 may be an external storage device such as a hard disk connected to or built into the image search device 1, or may be a portable storage medium such as a flash memory. The storage device 2 may also be a server device that performs data communication with the image search device 1. The storage device 2 may also be composed of multiple devices.

[0017] The display device 3 displays information under the control of the image search device 1. Examples of the display device 3 include a display, a projector, etc. When the display device 3 receives a display signal supplied from the image search device 1, it displays information based on the received display signal.

[0018] The input device 4 is an interface that accepts user input, which is external input based on operations by a user performing an image search using the image search system 100, and includes, for example, a touch panel, buttons, a keyboard, a voice input device, etc. The input device 4 supplies an input signal generated based on the user's input to the image search device 1.

[0019] The configuration of the image retrieval system 100 shown in Fig. 1 is an example, and various modifications may be made to the configuration. For example, the image retrieval device 1, the storage device 2, the display device 3, and the input device 4 may be integrated into any combination. The image retrieval system 100 may also include a sound output device such as a speaker. The image retrieval device 1 may also be composed of multiple devices. In this case, the multiple devices that make up the image retrieval device 1 exchange information required to execute pre-assigned processes between these multiple devices.

[0020] (2) Hardware Configuration 2 shows the hardware configuration of the image retrieval device 1. The image retrieval device 1 includes, as hardware components, a processor 11, a memory 12, and an interface 13. The processor 11, the memory 12, and the interface 13 are connected via a data bus 19.

[0021] The processor 11 executes a predetermined process by executing a program stored in the memory 12. The processor 11 is a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). The processor 11 may be composed of multiple processors. The processor 11 is an example of a computer.

[0022] The memory 12 is composed of various types of volatile and non-volatile memories such as RAM (Random Access Memory) and ROM (Read Only Memory). The memory 12 also stores programs for the image retrieval device 1 to execute various processes. The memory 12 is also used as a working memory, and temporarily stores information obtained from the storage device 2. The memory 12 may also function as the storage device 2. Similarly, the storage device 2 may also function as the memory 12 of the image retrieval device 1. The programs executed by the image retrieval device 1 may be stored in a storage medium other than the memory 12.

[0023] The interface 13 is an interface for electrically connecting the image retrieval device 1 to other devices. These interfaces may be wireless interfaces such as network adapters for wirelessly transmitting and receiving data to and from other devices, or may be hardware interfaces for connecting to other devices via cables or the like.

[0024] The hardware configuration of the image search device 1 is not limited to the configuration shown in Fig. 2. For example, the image search device 1 may include at least one of a display device 3 and an input device 4. The image search device 1 may also be connected to or built-in with a sound display device such as a speaker.

[0025] (3) Image Search Processing Overview An overview of the image retrieval process executed by the image retrieval device 1 will be described. In general, the image retrieval device 1 generates object region information regarding the region of an object detected from each candidate image, and calculates image features by integrating the local features extracted from each candidate image with the object region information. In this way, the image retrieval device 1 generates image features that take into account the region of the detected object, and improves search accuracy even when searching for objects with small regions in the candidate images. Hereinafter, the region of an object on an image will also be simply referred to as the "object region."

[0026] Fig. 3 shows an example of functional blocks of the image retrieval device 1. As shown in Fig. 3, the processor 11 of the image retrieval device 1 functionally includes a search input information acquisition unit 30, a search input feature extraction unit 31, an image acquisition unit 32, an object region information generation unit 33, an image feature extraction unit 34, a similarity calculation unit 35, and a search unit 36. Note that in Fig. 3, blocks where data is exchanged are connected by solid lines, but the combination of blocks where data is exchanged is not limited to this. The same applies to other functional block diagrams described later.

[0027] The search input information acquisition unit 30 acquires search input information based on an input signal supplied from the input device 4 via the interface 13. The search input information is any information specifying an image to be searched for, and includes, for example, text information specifying the image to be searched for. Furthermore, in addition to or instead of the text information, the search input information may also be information specifying a position on the image where an object exists (e.g., "the right half of the image," "the center of the image," etc.) (also referred to as "position specification information"). In this case, the search input information acquisition unit 30 may display a GUI (Graphical User Interface) for specifying a position on the image and accept an input specifying an arbitrary position (including a region) on the image. The search input information acquisition unit 30 supplies the acquired search input information to the search input feature extraction unit 31 and the object region information generation unit 33.

[0028] Search input feature extraction unit 31 extracts features of the search input information supplied from search input information acquisition unit 30 and generates search input features, which are features of the search input information. In this case, search input feature extraction unit 31 acquires, for example, features output by a feature extraction model when search input information is input to the feature extraction model, as the search input features. The feature extraction model described above may be any feature extraction model applied to input text information in a VLM (Vision-Language Model) such as BLIP-2. Alternatively, search input feature extraction unit 31 may calculate the search input features using only text information included in the search input. Search input feature extraction unit 31 supplies the generated search input features to similarity calculation unit 35.

[0029] The image acquisition unit 32 acquires each candidate image registered in the image DB 21, and supplies the acquired candidate images to the object region information generation unit 33 and the image feature extraction unit 34. Note that the image feature amount of each candidate image registered in the image DB 21 is calculated by the image feature extraction unit 34.

[0030] The object region information generation unit 33 generates object region information, which is information about object regions in the candidate images, based on the search input information supplied from the search input information acquisition unit 30 and the candidate images supplied from the image acquisition unit 32. The object region information includes information identifying the detected object region (including parameters representing the position, size, etc.), and may further include information identifying the type of the detected object (i.e., the classified class). The object region information generation unit 33 supplies the generated object region information to the image feature extraction unit 34.

[0031] The image feature extraction unit 34 generates image features based on the candidate images supplied from the image acquisition unit 32 and the object region information supplied from the object region information generation unit 33. Functionally, the image feature extraction unit 34 has a local feature extraction unit 341 and an integration unit 342.

[0032] The local feature extraction unit 341 extracts local features of the candidate image supplied from the image acquisition unit 32. The local features are features calculated from a partial region of the image, and may be vectors arranging pixel values ​​of small regions (e.g., lattice regions) regularly separated in the image, or feature maps output by a convolution layer of a convolutional neural network. The local features calculated by the local feature extraction unit 341 are not limited to the above-mentioned examples, and may be any local features that are commonly used, such as HOG (Histograms of Oriented Gradients), SIFT (Scaled Invariance Feature Transform), or HaarWavelet. Hereinafter, the regions on the image used to calculate each local feature will also be referred to as "local regions."

[0033] The integrating unit 342 generates image features based on the local features of a candidate image and object region information of the candidate image. In this case, when the integrating unit 342 generates image features by adding together local features using an attention mechanism, the integrating unit 342 determines at least one of a key and a query to be used in the attention mechanism based on the object region information. The integrating unit 342 supplies the image features generated for each candidate image to the similarity calculating unit 35.

[0034] The similarity calculation unit 35 calculates the similarity between the search input feature supplied from the search input feature extraction unit 31 and the image feature of each candidate image supplied from the image feature extraction unit 34. Here, any similarity index calculated by comparing features may be used as the similarity. For example, the similarity may be cosine similarity, or a value obtained by inputting a constant multiple of cosine similarity into a softmax function. The similarity calculation unit 35 supplies the above-mentioned similarity calculated for each candidate image to the search unit 36.

[0035] The search unit 36 ​​generates search results for candidate images related to the search input information based on the similarity calculated for each candidate image by the similarity calculation unit 35. The search unit 36 ​​then transmits a display signal representing the generated search results to the display device 3 via the interface 13, thereby displaying the search results on the display device 3. In this case, the search results may be, for example, a list of candidate images in which a predetermined number of candidate images with the highest similarity are arranged according to similarity or other criteria, or a list of candidate images with the highest similarity based on criteria other than similarity. The search unit 36 ​​may also generate captions (i.e., explanatory text) for each candidate image (also referred to as a "display image") displayed as a search result and display the generated explanatory text on the display device 3 together with the display image. In this case, for example, when using a VLM such as BLIP-2, the search unit 36 ​​generates a caption for the display image by inputting the image features of the display image generated by the integration unit 342 into a large language model (LLM).

[0036] Here, each of the components, including the search input information acquisition unit 30, the search input feature extraction unit 31, the image acquisition unit 32, the object region information generation unit 33, the image feature extraction unit 34, the similarity calculation unit 35, and the search unit 36, can be realized, for example, by the processor 11 executing a program. Alternatively, each component may be realized by recording the necessary program on any non-volatile storage medium and installing it as needed. Note that at least a portion of each component may not be realized by software programs, but may be realized by any combination of hardware, firmware, and software. Also, at least a portion of each component may be realized using a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or a microcontroller. In this case, the integrated circuit may be used to realize a program consisting of each of the above components. Also, at least a portion of each component may be configured by an ASSP (Application Specific Standard Produce), an ASIC (Application Specific Integrated Circuit), or a quantum processor (quantum computer control chip). In this way, each component may be realized by various hardware. The same applies to other embodiments described below. Furthermore, each of these components may be realized by the cooperation of multiple computers, for example, using cloud computing technology.

[0037] (4) Generation of object region information The generation of object region information by the object region information generation unit 33 will be described. The object region information generation unit 33 extracts an object region of each candidate image selected based on the search input information. By selecting an object region based on the search input information, the object region information generation unit 33 can generate object region information that focuses on a portion corresponding to the search input information. Then, by calculating image features that take such object region information into consideration, image search according to the user's intention becomes possible.

[0038] In a first example of selecting an object region based on search input information, when the search input information includes text information, the object region information generation unit 33 extracts a noun phrase from the text information and extracts an object region of an object related to the extracted noun phrase from each candidate image. In this case, the object region information generation unit 33 may extract the noun phrase by any morphological analysis, or may extract the noun phrase from the text information using any machine learning model (e.g., a language model) that has been trained to extract noun phrases from input text. Note that the trained parameters of the machine learning model are stored in the storage device 2, memory 12, or the like.

[0039] In a second example of selecting an object region based on search input information, when the search input information includes position designation information that designates the position on the image of the object to be detected, the object region information generation unit 33 extracts from each candidate image an object region that exists at the position indicated by the position designation information. Note that when the search input information includes both text information and position designation information, the object region information generation unit 33 extracts from each candidate image an object region that is selected based on the text information and that exists at the position indicated by the position designation information.

[0040] Next, a specific method for extracting an object region from a candidate image will be described. For example, the object region information generation unit 33 performs a process of extracting an object region corresponding to a predetermined class from the candidate image using an arbitrary object detector (object detection model). Examples of such object detectors include object detectors such as Grounding DINO, which is used in OVD (Open-Vocabulary Object Detection), a task of detecting unknown object classes specified in arbitrary text. Another example of an object detector includes YOLO (You Only Look Once). Further examples of object detectors include any model that uses segmentation results such as semantic segmentation, instance segmentation, and panoptic segmentation. Furthermore, the object region information generation unit 33 may determine the final object region to be output based on the results output by multiple object detectors. In this case, the object region information generation unit 33 may integrate the results output by the multiple object detectors by union, or may integrate them using any integration method such as non-maximum suppression.

[0041] Then, the object region information generation unit 33 generates object region information including parameters indicating the position and size of the object region from each candidate image, for example. Hereinafter, as an example, the parameters indicating the position and size of the object region will be described as being the image coordinate values ​​"(x, y)" of the representative point of the object region and four parameters "(x, y, w, h)" representing the horizontal length (w) and vertical length (h) of the object region. Note that the parameters indicating the position and size of the object region are not limited to parameters when the object region is regarded as a rectangle, but may be any combination of parameters for identifying the object region. Furthermore, the object region information is not limited to including both parameters of the position and size of the object region, and the object region information generation unit 33 may include at least a parameter indicating the position of the object region in the object region information.

[0042] The object region information generation unit 33 may generate object region information for each candidate image without using search input information. In this case, the object region information generation unit 33 generates object region information representing object regions, which are characteristic regions other than background regions, from each candidate image using an object detector that has been trained in advance by machine learning to detect object regions of specific classes such as cars, buildings, and people. Even when image features are calculated based on object region information that does not take into account such search input information, highly accurate image search is possible.

[0043] (5) Image feature generation based on object region information Next, the generation of image features based on object region information by the integration unit 342 of the image feature extraction unit 34 will be described. When generating image features by adding local features together using an attention mechanism, the integration unit 342 determines at least one of the key and query to be used in the attention mechanism based on the object region information. The attention mechanism performs integration using a weighted sum, and determines a local region in a candidate image to focus on based on the similarity between the query and the key. Here, the attention mechanism may be self-attention or cross-attention. Similar embodiments can be applied to other mechanisms as long as they use an attention mechanism.

[0044] Fig. 4 is a diagram showing an outline of the processing flow for generating image features. In the example of Fig. 4, first, local features are calculated from local regions set in the candidate images.

[0045] Thereafter, the integrating unit 342 sets a key and a query based on each local feature, and determines at least one of the set key and query based on the object region information. This determination method will be described later. In this way, the integrating unit 342 sets a key and a query.

[0046] In one example of a method for determining a key and a query, the integrating unit 342 provisionally determines a key and a query without relying on object region information, and then corrects the provisionally determined key and query based on the object region information. Hereinafter, a key calculated without relying on object region information is referred to as an "original key," and a query calculated without relying on object region information is referred to as an "original query." The same number of original keys as local features are set, and the original keys are generated from each local feature using a neural network such as a multilayer perceptron or a combination of a neural network and PE (Positional Embeddings). The method for determining the original query differs depending on whether self-attention or cross-attention is used. In the case of self-attention, the original query is calculated from local features using a neural network or the like. In the case of cross-attention, the original query is calculated from learning parameters and local features using a neural network or the like. The original key and the original query may be calculated using any calculation method used in a Vision-Language Model (VLM) such as BLIP-2.

[0047] Next, the integration unit 342 calculates the similarity between the key and the query as a weight, and generates a weight map for each query, having the same number of weight elements as the number of local features. In this case, the similarity may be a cosine similarity, or a value obtained by inputting a constant multiple of the cosine similarity into a softmax function. Each weight in the weight map increases as the similarity between the corresponding key and query increases.

[0048] The integrating unit 342 then calculates a weighted sum of the local features using the weight map, and converts the calculated weighted sum using a neural network to calculate the value as the image feature. Here, since at least one of the key and the query is determined based on the object region information, the image feature is calculated taking the object region information into consideration.

[0049] Next, aspects (first and second aspects) of determining a key and a query based on object region information will be described.

[0050] In a first aspect, the integrating unit 342 assigns to the original query information indicating the position (size may also be considered) of the object region obtained by converting the object region information, and assigns to the original key information regarding the relative position (size may also be considered) of the local region corresponding to the local special information amount. In a second aspect, the integrating unit 342 adjusts the size of the component of the original query projected onto the subspace of the original key corresponding to the object region indicated by the object region information.

[0051] First, a method for determining a query in the first mode will be described. Hereinafter, the “i”th original query to be corrected will be referred to as “q i ” and the final query obtained by correcting the original query is “qa i " The original query "q i The local features and learning parameters corresponding to i ", "φ i "

[0052] The integration unit 342 integrates the original query q from the parameters (x, y, w, h) of the position and size of the object region indicated by the object region information. i The information (more specifically, a vector) "p" to be added to the original query q is calculated using a neural network or similar. i and the calculated additional information p are integrated. The integration may be performed by adding vectors or by combining vectors to expand the dimension of the vectors.

[0053] For example, when vector addition is used for integration, the integration unit 342 performs the following operation on the query qa i is calculated based on the following formula (1). qa i =q i +p (1)

[0054] Here, if a function that takes parameters of the position and size of the object region as input and outputs additional information p is defined as "f", then equation (1) can be rewritten as equation (2) below. qa i =q i +f1(x,y,w,h) (2)

[0055] Here, the function f1 is, for example, a neural network or a combination of a neural network and a PE, and the learned parameters thereof are stored in advance in the storage device 2 or the memory 12. The function f1 further takes, as an argument, a local feature m i or / and the learning parameter φ i may further comprise:

[0056] Original query q i In the case of Self-Attention, the local feature m i and is expressed as follows using the function "g". q i =g(m i ) The function g is, for example, a neural network or a combination of a neural network and a PE, and these learned parameters are stored in advance in the storage device 2 or the memory 12 or the like.

[0057] On the other hand, the original query q i In the case of Cross-Attention, the local feature m i and the learning parameter φ i and is expressed as follows using the function g. q i =g(m i ,φ i )

[0058] In the first aspect, the integration unit 342 integrates the original query q using a function value based on parameters of the position and size of the object region, as in the above-described formula (1). i Instead of correcting the original query q i Query directly without using qa i may be calculated.

[0059] Specifically, in the case of Self-Attention, the integration unit 342 integrates the parameters (x, y, w, h) of the position and size of the object region and the local feature m i and function f2, the query qai Determine. qa i =f2(x,y,w,h,m i ) (3)

[0060] On the other hand, in the case of Cross-Attention, the parameters (x, y, w, h) of the position and size of the object region and the local feature m i and the learning parameter φ i and function f3, as shown in the following formula (4), query qa i Determine. qa i =f3(x,y,w,h,m i ,φ i ) (4)

[0061] The functions f2 and f3 used in equations (3) and (4) are respectively a neural network or a combination of a neural network and a PE, and their trained parameters are stored in advance in the storage device 2 or the memory 12. For example, an architecture such as DAB-DETR (Dynamic Anchor Boxes are Better Queries for DETR) can be applied to the functions f2 and f3.

[0062] Here, a supplementary explanation will be given of the method for determining a query based on the first mode when there are multiple object regions indicated by the object region information.

[0063] First, a case where there are multiple object regions indicated by the object region information and the attached information p is calculated will be described. In this case, the integration unit 342 calculates the attached information p for each of the multiple object regions, and calculates the average of all the calculated attached information p (a statistical representative value other than the average may be used; the same applies below). Then, the integration unit 342 compares the calculated average of the attached information p with the original query q i By integrating with the query QA i The integration unit 342 extracts features from the attribute information p calculated for each object region, and calculates the average of the feature amounts of the obtained attribute information p and the original query q. iBy integrating with the query QA i Alternatively, the integrating unit 342 may classify the objects represented by the object regions and calculate the average of the attribute information p or the average of the feature amounts of the attribute information p for each object class. In this case, the integrating unit 342 calculates the average of the attribute information p or the average of the feature amounts of the attribute information p for each object class, and then calculates the average for all object types from the calculated average for each object class. Then, the integrating unit 342 applies the calculated average for all object types to the original query q i By integrating with query qa i Ask for.

[0064] Next, when there are multiple object regions indicated by the object region information, the query qa i In this case, the integration unit 342 calculates a tentative query for each object by using formula (3) or formula (4) for each of the multiple object regions. Then, the integration unit 342 calculates the average (or a statistical representative value other than the average; the same applies below) of all the calculated tentative queries as the final query qa i The integration unit 342 extracts features from the provisional queries calculated for each object region, and calculates the final query qa i Alternatively, the integrating unit 342 may classify the objects represented by the object regions and calculate the average of the provisional queries or the average of the feature quantities of the provisional queries for each object class. In this case, the integrating unit 342 calculates the average of the provisional queries or the average of the feature quantities of the provisional queries for each object class, and then averages the calculated averages for each object class for all object types, and calculates the final query qa i You may ask for:

[0065] Next, a method for determining a key based on the first aspect will be described. Hereinafter, the "i"th original key to be corrected will be referred to as "k i " and the final key obtained by correcting the original key is "ka i " The original key "k i The local feature corresponding to "m i "

[0066] In this case, the integration unit 342 integrates the local feature m i The parameter indicating the relative position (i.e., the position in the candidate image) of the original key k i For example, the local feature m i The coordinate position of the local region in the candidate image used to calculate is (x i ,y i ), and width "w i ", and the vertical width is "h i ” then the integration unit 342 calculates the original key k as shown in the following equation (4): i local feature m i The value output by function f4, which takes as an argument a parameter indicating the relative position of ka i =k i +f4(x i ,y i ,w i ,h i ) (4) Here, the function f4 is, for example, a neural network or a combination of a neural network and a PE, and the learned parameters thereof are stored in advance in the storage device 2 or the memory 12. i is the local feature m i It is calculated using a neural network or the like.

[0067] Next, a second aspect of the query determination method based on object region information will be described. In the second aspect, the integrating unit 342 adjusts the information of the original key corresponding to the object region included in the original query. Specifically, the integrating unit 342 corrects the original query so as to strengthen the information of the original key corresponding to the object region included in the original query. Note that, as will be described later, the integrating unit 342 may correct the original query for a specific object region so as to weaken the information of the original key corresponding to the object region included in the original query.

[0068] 5 is a diagram showing an overview of correction of an original query in the second mode. First, the integrating unit 342 extracts an original key corresponding to an object region (also called an "object region corresponding key") from all original keys generated from candidate images. In this case, the integrating unit 342 refers to object region information corresponding to the candidate image, and extracts, as the object region corresponding key, an original key corresponding to a local feature calculated from the object region represented by the object region information.

[0069] Next, the integrating unit 342 applies principal component analysis to the object region corresponding keys to generate a subspace, and projects the corresponding original query onto the generated subspace to extract information on the object region corresponding keys originally included in the original query. Note that before performing the above-mentioned principal component analysis, the integrating unit 342 may multiply each key corresponding to the object region by a weight according to its importance. Here, the "weight according to importance" may be, for example, a value based on the size in the candidate image of the object region to which each key belongs (e.g., the area of ​​the object region), or may be a value based on the distance between the center position of the object region to which each key belongs and the position on the candidate image corresponding to each key.

[0070] Next, the integrating unit 342 multiplies the information of the object region corresponding key contained in the original query by a predetermined coefficient α (α is a value greater than 1), and adds the information of the object region corresponding key contained in the original query after multiplication by the coefficient α to the original query. In this way, the integrating unit 342 can correct the original query so as to strengthen the information of the object region corresponding key contained in the original query. Note that the coefficient α may be a default value stored in advance in the storage device 2 or memory 12, or may be set from a predetermined formula or lookup table based on the size of the object region.

[0071] If an object region representing an object that is undesirable to be included in the search results is present in the candidate image, the integrating unit 342 may correct the original query to weaken the information of the object region corresponding key held by the original query. In this case, the integrating unit 342 first refers to the object region information, and if it determines that an object region representing an object that is undesirable to be included in the search results is present, extracts the object region corresponding key corresponding to the object region. The "object that is undesirable to be included in the search results" may be an object that falls under a predetermined class, or may be an object designated by the search input information as an object to be excluded. Then, the integrating unit 342 multiplies the information of the object region corresponding key held by the original query by a predetermined coefficient β (β is a value less than 0), and adds the information of the object region corresponding key held by the original query after multiplication by the coefficient β to the original query.

[0072] Next, a supplementary explanation will be given on the selection of the original query to be corrected based on the first or second aspect.

[0073] In the first selection example, the integration unit 342 regards all original queries generated from the candidate images as correction targets, and corrects each original query based on the first aspect or the second aspect.

[0074] In a second selection example, the integrating unit 342 determines an original query to be corrected based on a weight map calculated using the original key and the original query. FIG. 6 is a diagram illustrating an overview of a second selection example of an original query to be corrected. For ease of explanation, three original queries are clearly shown in FIG. 6, and a weight map is generated from each original query. Here, the weight map is the weight map shown in FIG. 4, and is a map in which the similarity between the original key and the original query is used as a weight.

[0075] The integrating unit 342 calculates the similarity between the weight map corresponding to each original query and the ideal weight map, and determines the similarity or dissimilarity between the weight map corresponding to each original query and the ideal weight map based on the calculated similarity. The similarity calculated by the integrating unit 342 may be any index used as an index of similarity between two images. For example, the integrating unit 342 calculates the similarity between the weight map corresponding to each original query and the ideal weight map based on cosine similarity.

[0076] Here, the ideal weight map is, for example, a map generated from object region information, in which the object region is assigned a high weight and the region other than the object region (i.e., the background region) is assigned a low weight. Here, as an example, the ideal weight map is a mask image representing the object region identified based on the object region information. In the mask image, the object region is assigned the maximum pixel value (white in the figure) and the background region is assigned the minimum pixel value (black in the figure).

[0077] The integrating unit 342 then determines that a weight map whose similarity to the ideal weight map is equal to or greater than a predetermined threshold or a weight map whose similarity is in the top M (M is a positive integer) ranks is similar to the ideal weight map, and determines that other weight maps are dissimilar to the ideal weight map. The integrating unit 342 then considers a query corresponding to a weight map determined to be similar to the ideal weight map to be a correction target, and corrects the original query based on the first or second aspect described above. Furthermore, the integrating unit 342 considers a query corresponding to a weight map determined to be dissimilar to the ideal weight map to be an exception to the correction target, and leaves the original query as is without performing correction based on the first or second aspect described above.

[0078] (6) Processing flow FIG. 7 is an example of a flowchart showing an outline of the image search process executed by the image search device 1.

[0079] First, the image retrieval device 1 acquires search input information based on an input signal supplied by the input device 4 (step S11). In this case, the image retrieval device 1 may cause the display device 3 to display an input screen for accepting user input (external input) related to the search input information. Furthermore, the image retrieval device 1 calculates search input feature quantities, which are feature quantities of the search input information, at any timing after step S11 and before the start of the processing of step S18.

[0080] Next, the image retrieval device 1 acquires candidate images from the image DB 21 (step S12). Then, the image retrieval device 1 acquires object region information of the candidate images acquired in step S12 (step S13). In this case, the image retrieval device 1 detects an object region from the candidate images acquired in step S12, for example, using an object detector that has been trained by machine learning in advance, and generates object region information based on the detection result of the object region. In this case, the image retrieval device 1 may identify an object to be detected based on the search input information acquired in step S11, and generate object region information related to the identified object.

[0081] Next, the image retrieval device 1 calculates local features of the candidate image acquired in step S12 (step S14). Then, the image retrieval device 1 generates image features based on the local features calculated in step S14 and the object region information acquired in step S13 (step S15). In this case, the image retrieval device 1 determines at least one of a key and a query to be used in integrating local features based on the attention mechanism, based on the object region information.

[0082] Next, the image retrieval device 1 determines whether image features have been generated for all images registered in the image DB 21 (step S16). If image features have not been generated for all images registered in the image DB 21 (step S16; No), the image retrieval device 1 returns the process to step S12 and acquires candidate images for which image features have not been generated from the image DB 21.

[0083] On the other hand, if image features have been generated for all images registered in the image DB 21 (step S16; Yes), the image retrieval device 1 outputs image search results based on the similarity between the image features of each candidate image and the search input features (step S17). In this case, the image retrieval device 1 determines that the candidate image with a higher similarity between its image features and the search input features has a higher priority for output as an image search result, and determines the candidate images to be output as search results and their output priority. The image retrieval device 1 then transmits a display signal indicating the image search results to the display device 3, and causes the display device 3 to display information about the search results.

[0084] (7) Image feature extraction training Next, we will explain the learning (i.e., machine learning) of the image feature extraction unit 34 that is performed before the image search process by the image search device 1. Hereinafter, it is assumed that the image search device 1 performs the learning of the image feature extraction unit 34, but this is not limiting, and the image feature extraction unit 34 may be trained by a device other than the image search device 1. When the image search device 1 or a device other than the image search device 1 performs the learning, the parameters obtained by the learning are stored in the storage device 2 or memory 12 before the image search process so that the image search device 1 can refer to them in the image search process.

[0085] In the training of the image feature extraction unit 34, a training data set is stored in the storage device 2 or the memory 12, and the image search device 1 updates the parameters of the image feature extraction unit 34 using the training data set.

[0086] Here, the training dataset has multiple records, and each record corresponds to a set of training images used for training and input information suitable for searching the training images (a so-called set of positive examples). Note that the input information in this case may be text information or any information that can be specified as search input information (for example, position specification information that specifies the position of an image).

[0087] The image retrieval device 1 regards training images as candidate images, generates object region information using an object region information generation unit 33, and calculates image features using an image feature extraction unit 34. The image retrieval device 1 then determines parameters for the image feature extraction unit 34 so as to minimize a loss calculated from the search input features and image features calculated by the search input feature extraction unit 31 from input information paired with the training images. The algorithm for determining the parameters may be any learning algorithm used in machine learning, such as gradient descent or backpropagation. In this case, any optimization method, such as stochastic gradient descent (SGD) or Adam, may also be used. The loss function for determining the loss may be any loss function that increases as the similarity between the search input features and the image features decreases, or may be a loss function that identifies whether the training images and input information are a correct pair.

[0088] Furthermore, the parameters to be learned in the image feature extraction unit 34 may be all of the parameters of the image feature extraction unit 34, or may be some of the parameters of the image feature extraction unit 34. In a first example in which some of the parameters of the image feature extraction unit 34 are to be learned, the image retrieval device 1 determines the parameters of the function used in the first mode of correcting the original key and the original query as the parameters to be learned. In a second example in which some of the parameters of the image feature extraction unit 34 are to be learned, the image retrieval device 1 adds learnable parameters to the image feature extraction unit 34 and learns the added parameters. To add such parameters, a method such as Visual Prompt Tuning may be used. Note that the first and second examples described above may be executed in combination.

[0089] (8) Variations Next, preferred modifications of the above-described embodiment will be described. The following modifications may be applied to the above-described embodiment in combination.

[0090] (Variation 1) The image search device 1 may recursively repeat the process corresponding to the integration unit 342 multiple times.

[0091] 8 shows an example of functional blocks of the image feature extraction unit 34. The image feature extraction unit 34 has a local feature extraction unit 341 and n integrating units 342 (a first integrating unit 3421 to an n-th integrating unit 342n, where n is an integer of 2 or more).

[0092] The first integrating unit 3421 performs the same processing as the integrating unit 342 described in FIG. 3, generating image features based on local features and object region information. The second integrating unit 3422 regards the image features output by the first integrating unit 3421 as a query (original query) and generates image features based on the local features calculated by the local feature extraction unit 341 and the object region information. The third integrating unit 3423 to the n-th integrating unit 342n also regard the image features output by the integrating unit with the previous serial number as a query (original query) and generate image features based on the local features calculated by the local feature extraction unit 341 and the object region information. The similarity calculating unit 35 calculates the similarity between the image features output by the n-th integrating unit 342n and the search input features. By repeating such recursive processing by the integrating units, it is possible to generate image features with higher accuracy.

[0093] It is sufficient that at least one of the first integrating unit 3421 to the n-th integrating unit 342n uses object region information. In this case, the integrating unit that does not use object region information does not perform a process of correcting the original key and the original query using the object region information, but generates a weight map based on the similarity between the original key and the original query, and generates image features from the weight map.

[0094] (Variation 2) The image retrieval device 1 may perform the image retrieval process using the object region information generated before the image retrieval process.

[0095] FIG. 9 shows an example of functional blocks of the image retrieval device 1. The processor 11 of the image retrieval device 1 functionally includes a search input information acquisition unit 30, a search input feature extraction unit 31, an image acquisition unit 32, an object region information selection unit 33A, an image feature extraction unit 34, a similarity calculation unit 35, and a search unit 36. The search input information acquisition unit 30, the search input feature extraction unit 31, the image acquisition unit 32, the image feature extraction unit 34, the similarity calculation unit 35, and the search unit 36 ​​shown in FIG. 9 perform the same processes as the search input information acquisition unit 30, the search input feature extraction unit 31, the image acquisition unit 32, the image feature extraction unit 34, the similarity calculation unit 35, and the search unit 36 ​​shown in FIG. 3, respectively, and therefore descriptions of these units will be omitted as appropriate. The storage device 2 also stores an image DB 21 and an object region information DB 22.

[0096] The object region information DB22 is a database that stores object region information corresponding to each candidate image registered in the image DB21. The object region information is information that is generated, before image retrieval processing, by the image retrieval device 1 or another device based on the object detection results for each candidate image registered in the image DB21. Note that when a candidate image contains multiple objects, object region information corresponding to each object is registered in the object region information DB22 for that candidate image.

[0097] In addition, in the object region information DB22, meta information regarding each object included in each candidate image is added to the object region information. The above-mentioned meta information is, for example, text information that indicates a noun related to the object in the object region. Note that the meta information may include information other than text information (for example, information specifying a position on the image) instead of or in addition to the text information. The meta information also includes identification information (for example, an image ID) of the candidate image in which the object indicated by the object region information is detected. Note that the identification information is added to each candidate image, for example.

[0098] The object region information selection unit 33A extracts related object region information from the object region information DB 22 based on the candidate images supplied from the image acquisition unit 32 and the search input information supplied from the search input information acquisition unit 30. In this case, the object region information selection unit 33A extracts object region information associated with the identification information of the candidate images and whose meta information is related to (i.e., matches or is similar to) the search input information from the object region information DB 22. Then, the object region information selection unit 33A supplies the selected object region information to the image feature extraction unit 34.

[0099] If the meta information and the search input information include text information, the object region information selection unit 33A may determine whether the meta information and the search input information are similar to each other using a similarity scale that evaluates the degree of matching of any text, such as BLUE, CIDEr, or SPICE. The object region information selection unit 33A may also use features generated by Word2Vec, Doc2Vec, or the VLM used to determine whether the meta information and the search input information are similar to each other based on the similarity of the features. The object region information selection unit 33A may select only object region information having meta information whose similarity to the search input information is equal to or greater than a predetermined threshold, or may select only object region information having meta information whose similarity to the search input information is in the top M (M is a positive integer) order.

[0100] Note that the object region information selection unit 33A does not need to use the search input information. In this case, the object region information selection unit 33A selects object region information associated with the candidate image supplied from the image acquisition unit 32 from the object region information DB 22. Alternatively, the object region information selection unit 33A may select object region information related to the search input information based on the search input information, regardless of the candidate image supplied from the image acquisition unit 32, and supply the selected object region information to the image feature extraction unit 34. In this case, the image feature extraction unit 34 may acquire from the image acquisition unit 32 only candidate images related to the object region information received from the object region information selection unit 33A as candidates to be included in the search results, and calculate image feature amounts of the candidate images.

[0101] According to this modification, the image retrieval device 1 can preferably reduce the processing load required to generate object region information in the image retrieval process.

[0102] (Variation 3) The image search device 1 may divide the search results into multiple classes according to the size of the object area contained in the positive example images of the search input, calculate an evaluation index value for each divided class, and then calculate a final evaluation index value by integrating the calculated evaluation indexes through statistical processing such as averaging.

[0103] Any recommendation evaluation index such as Recall@K or Median Rank may be used as the evaluation index. Furthermore, the "size of the object region" may be, for example, the proportion of the object's area in the image. In one example of a method for dividing search results, the image retrieval device 1 determines the division to which the search results belong based on the maximum or minimum value of the object region size in the positive example images. For example, the image retrieval device 1 may divide the search results into 10% divisions based on the proportion of the object's area in the image. In this case, 10 classes are generated, including a class with a proportion of 0% to 10%, a class with a proportion of 10% to 20%, and a class with a proportion of 90% to 100%, and an evaluation index value corresponding to each class is calculated.

[0104] Second Embodiment 10 is a block diagram of an image retrieval device 1X. The image retrieval device 1X includes a first acquisition unit 30X, a second acquisition unit 33X, an integration unit 342X, and a search unit 36X. The image retrieval device 1X may be composed of multiple devices.

[0105] The first acquiring means 30X acquires input information related to a search. The first acquiring means 30X can be, for example, the search input information acquiring unit 30 in the first embodiment (including modifications, the same applies below).

[0106] The second acquisition unit 33X acquires object region information relating to the region of an object included in an image in the image database to be searched. The second acquisition unit 33X can be, for example, the object region information generation unit 33 and the object region information selection unit 33A in the first embodiment.

[0107] The integrating means 342X integrates the local feature extracted from the image and the object region information to calculate the image feature. The integrating means 342X can be, for example, the integrating unit 342 in the first embodiment.

[0108] The search means 36X searches the image database for images related to the input information based on the similarity between the image features and the features of the input information. The "features of the input information" are an example of the "search input features" in the first embodiment. The search means 36X can be, for example, the search unit 36 ​​in the first embodiment.

[0109] FIG. 11 is an example of a flowchart showing the processing procedure of the image retrieval device 1X. First, the first acquisition means 30X acquires input information related to the search (step S21). Next, the second acquisition means 33X acquires object region information related to the region of an object included in an image in the image database to be searched (step S22). Next, the integration means 342X calculates an image feature by integrating the local feature extracted from the image and the object region information as the feature of the image (step S23). The search means 36X searches the image database for images related to the input information based on the similarity between the image feature and the feature of the input information (step S24).

[0110] According to the second embodiment, the image retrieval device 1X can accurately retrieve images related to input information related to a search from an image database.

[0111] In each of the above-described embodiments, the program can be stored using various types of non-transitory computer-readable media and supplied to a computer processor or the like. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The program may also be supplied to a computer by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable medium can supply the program to a computer via a wired communication path such as an electric wire or optical fiber, or via a wireless communication path.

[0112] In addition, part or all of the above-described embodiments (including variations, the same applies below) may also be described as, but are not limited to, the following supplementary notes. Furthermore, not only the devices, methods, and storage media described in the supplementary notes, but also various hardware, software, various recording means (including storage media) for recording software, or systems may be made to depend on part or all of the configurations described in the supplementary notes, as long as they do not deviate from the above-described embodiments.

[0113] [Appendix 1] a first acquisition means for acquiring input information related to a search; a second acquisition means for acquiring object region information relating to a region of an object included in an image in the image database that is the target of the search; an integration means for integrating the local feature extracted from the image and the object region information to calculate an image feature as a feature of the image; a search means for searching the image database for an image related to the input information based on the similarity between the image feature amount and the feature amount of the input information; An image search device having the above configuration. [Appendix 2] The image search device according to claim 1, wherein the integration means determines at least one of a key and a query to be used in the integration of the local features based on an attention mechanism, based on the object region information. [Appendix 3] 4. The image search device according to claim 2, wherein the integration means determines the query based on the position of the region indicated by the object region information. [Appendix 4] The image retrieval device according to claim 3, wherein the integrating means determines the key corresponding to the local feature based on a position of a local region corresponding to the local feature. [Appendix 5] The image search device according to any one of appendices 2 to 4, wherein the integration means adjusts the size of the components of the query projected into the subspace of the key corresponding to the region indicated by the object region information. [Appendix 6] 6. The image retrieval device according to any one of claims 1 to 5, wherein the second obtaining means obtains the object region information relating to a region of the object selected based on the input information. [Appendix 7] 7. The image retrieval device according to any one of claims 1 to 6, wherein the second acquisition means acquires the object region information corresponding to the image from a database that stores the object region information. [Appendix 8] the database stores the object region information to which meta information relating to each object included in the image is added, 8. The image search device according to claim 7, wherein the second acquisition means acquires the object region information to which the meta information related to the input information is added. [Appendix 9] The computer Get search input information, Obtaining object region information relating to the region of an object included in an image of the image database to be searched; calculating an image feature amount obtained by integrating the local feature amount extracted from the image and the object region information as a feature amount of the image; searching the image database for images related to the input information based on the similarity between the image feature amount and the feature amount of the input information; How to search for images. [Appendix 10] Get search input information, Obtaining object region information relating to the region of an object included in an image of the image database to be searched; calculating an image feature amount obtained by integrating the local feature amount extracted from the image and the object region information as a feature amount of the image; A storage medium storing a program that causes a computer to execute a process of searching the image database for images related to the input information based on the similarity between the image features and the features of the input information. [Appendix 11] the input information includes at least text information; 9. The image search device according to any one of claims 1 to 8, wherein the search means searches for images related to the input information based on a similarity between the image feature amount and a feature amount of the text information. [Appendix 12] An image search device described in any one of Appendices 1 to 8 and 11, wherein the search means generates a caption for the display image based on the image features of the display image displayed as a result of the search, and displays the display image and the caption on a display device. [Appendix 13] The image search device according to any one of appendices 1 to 8, 11, or 12, wherein the search means divides the search results into a plurality of classes based on the size of the area of ​​the object contained in the positive example image corresponding to the input information, calculates an evaluation index value for each of the plurality of classes, and integrates the calculated evaluation index values.

[0114] Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications within the scope of the present invention that would be understood by those skilled in the art can be made to the configuration and details of the present invention. In other words, the present invention naturally includes various modifications and alterations that would be possible for those skilled in the art based on the entire disclosure, including the claims, and the technical ideas. Furthermore, the disclosures of the above-cited patent and non-patent documents are incorporated herein by reference. [Explanation of symbols]

[0115] 1. 1X image search device 2 Storage device 3 Display device 4 Input Devices 11 processors 12 Memory 13 Interface 21 Image DB 22 Object region information DB 100 Image Search System

Claims

1. a first acquisition means for acquiring input information related to a search; a second acquisition means for acquiring object region information relating to a region of an object included in an image in the image database to be searched; an integration means for integrating the local feature extracted from the image and the object region information to calculate an image feature as a feature of the image; a search means for searching the image database for an image related to the input information based on the similarity between the image feature amount and the feature amount of the input information; An image search device having the above configuration.

2. The image search device according to claim 1 , wherein the integration means determines at least one of a key and a query used in integration of the local feature amounts based on an attention mechanism, based on the object region information.

3. The image search device according to claim 2 , wherein the integration means determines the query based on the position of the region indicated by the object region information.

4. The image search device according to claim 3 , wherein the integrating means determines the key corresponding to the local feature based on a position of the local region corresponding to the local feature.

5. The image search device according to claim 2 , wherein the integration means adjusts a size of the component of the query projected onto a subspace of the key corresponding to the region indicated by the object region information.

6. The image search device according to claim 1 , wherein the second acquisition means acquires the object region information relating to a region of the object selected based on the input information.

7. The image search device according to claim 1 , wherein the second acquisition means acquires the object region information corresponding to the image from a database that stores the object region information.

8. the database stores the object region information to which meta information relating to each object included in the image is added, The image search device according to claim 7 , wherein the second acquisition means acquires the object region information to which the meta information related to the input information has been added.

9. The computer Get search input information, Obtaining object region information relating to the region of an object included in an image of the image database to be searched; calculating an image feature amount obtained by integrating the local feature amount extracted from the image and the object region information as a feature amount of the image; searching the image database for images related to the input information based on the similarity between the image feature amount and the feature amount of the input information; How to search for images.

10. Get search input information, Obtaining object region information relating to the region of an object included in an image of the image database to be searched; calculating an image feature amount obtained by integrating the local feature amount extracted from the image and the object region information as a feature amount of the image; A storage medium storing a program that causes a computer to execute a process of searching the image database for images related to the input information based on the similarity between the image features and the features of the input information.

Images