Search Device

Abstract

To solve a problem that there are cases in which a user is hard to find a desired image.SOLUTION: A device for retrieving an image related to a keyword from retrieval object images, includes: means for acquiring a reliability value of tag information related to the key word from the retrieval object images; means for acquiring area information of a subject related to the keyword from the retrieval object images, and calculating a ratio of the subject occupying in the total image; and means for calculating an evaluation value by using the reliability value of the tag information and the ratio of the subject occupying in the total image, and returns the image retrieval result in which the evaluation value is reflected.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a search device, and more particularly to a keyword search. [Background technology]

[0002] In conventional keyword-based image search devices, tags are linked to images registered in a database, and search results are displayed based on the degree of match and relevance between the keywords and tags entered during the search.

[0003] In recent years, with the development of AI, tagging of images has become more common automatically, and Patent Document 1 discloses a method for automatically generating tags by performing image analysis. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2015-501982 Summary of the Invention [Problem to be solved by the invention]

[0005] However, many automatic tagging techniques, including the conventional techniques disclosed in the above-mentioned patent documents, often use a threshold value to associate only highly reliable tags with an image, and tags with low reliability may not be associated with the image.

[0006] Depending on the image, the reliability of the correct tag indicating the subject may not be high, so the correct tag for the subject may not be associated with the image. In such cases, even if you perform an image search using a keyword such as the name of the subject, the image will not be found because the correct tag for the subject is not associated with the image.

[0007] Furthermore, even if all tags are linked to an image without using a threshold, a composition that zooms in on the subject tends to have lower reliability of the correct tags that identify the subject than a composition that pulls back and captures the entire image. In such cases, the image will not appear at the top of keyword image searches.

[0008] In this way, it may be difficult for users to find the desired image. Therefore, an object of the present invention is to make it easier for users to find the desired image. [Means for solving the problem]

[0009] In order to solve the above problems, the present invention provides: Multiple Keywords from the search target images Images with associated tag information matching A search device for searching for an image to be searched. Tag information associated with the search target image and a means for acquiring a reliability value of the image to be searched from the image to be searched. Subjects appearing in the search target image and a means for acquiring area information of the subject and calculating the ratio of the subject to the entire image, using the reliability value of the tag information and the ratio of the subject to the entire image. For the search target image means for calculating an evaluation value; An image associated with tag information matching the keyword is searched from among the plurality of search target images, and the images included in the image search results are sorted based on the evaluation value for the image. A search device that returns image search results. [Effects of the Invention]

[0010] According to the present invention, it is possible to make it easier for a user to find a desired image. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 2 is a block diagram of a user terminal and a server according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating a connection configuration of a user terminal and a server according to an embodiment of the present invention. [Figure 3] This is an image of a raccoon, with the face taken as the focus. [Figure 4] This is an image of a raccoon taken so that its entire body is visible. [Figure 5]10 is an image of tag information for an image. [Figure 6] 10 is an image of tag information for an image. [Figure 7] This is an image that visualizes region information for an image. [Figure 8] This is an image that visualizes region information for an image. [Figure 9] FIG. 4 is a sequence diagram of image registration between a user terminal and a server in the first embodiment. [Figure 10] FIG. 4 is a sequence diagram of an image search performed by a user terminal and a server in the first embodiment. [Figure 11] 4 is a flowchart of a server according to the first embodiment. [Figure 12] 4 is a flowchart of a user terminal according to the first embodiment. [Figure 13] 10 is a flowchart of a user terminal according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0013] (First embodiment) In this embodiment, the image search system searches for images that match the keywords sent from the user terminal among the images managed in the server's database, and returns the search results to the user terminal.

[0014] The images managed in the server's database are registered images sent by users. After receiving the images from users, they are analyzed using AI, tagged using the results, and then registered in the database.

[0015] In addition, the above tagging utilizes area information recorded in the image meta information.

[0016] Area information is data that has become increasingly popular in recent years due to advances in AI and improvements in camera sensors, and it indicates the boundaries of areas that represent the same thing within an image.

[0017] This area information is used to calculate the percentage of the image that the subject occupies, and this value is reflected in the image search results, along with the reliability of the image analysis results. This makes it possible to make images that could not be found using conventional search methods or that did not appear at the top of search results more likely to appear at the top of search results, making them easier to find.

[0018] The configuration of the present invention will be described below with reference to FIG.

[0019] 1 is a block diagram of a user terminal and a server according to an embodiment of the present invention. In this embodiment, the server A100 is assumed to be a general server PC, and the user terminal B100 is assumed to be a PC found in an ordinary home. Note that the user terminal B100 may also be a smartphone, a notebook PC, a tablet PC, or the like.

[0020] The server terminal A100 is composed of a control unit A101, a storage unit A102, a memory A103, a display unit A104, an operation unit A105, and a communication unit A106, and the control unit A101 controls each unit of the server A100. Note that instead of the control unit A101 controlling the entire device, the entire device may be controlled by multiple hardware devices sharing the processing load.

[0021] The storage unit A102 is an electrically erasable and recordable non-volatile storage medium such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), etc. The storage unit A102 stores programs executed by the control unit A101, databases, images, etc.

[0022] The memory A103 is, for example, a RAM (Random Access Memory), and is used as a buffer memory for temporarily storing various data, a work area for the control unit A101, and the like.

[0023] The display unit A104 displays a GUI (Graphical User Interface) for interactive operations. The display unit A104 does not necessarily have to be built into the server A100.

[0024] The operation unit A105 is used to input instructions to the server A100. The operation unit A105 includes, for example, a power button for instructing ON / OFF of the power supply of the server A100 and operation buttons for instructing screen transitions. It also includes a touch panel provided on the display unit A104. Note that the operation unit A105 does not necessarily have to be built into the server A100.

[0025] The communication unit A106 is, for example, a NIC (Network Interface Card) incorporating a connector (RJ45) for connecting to Ethernet, a communication IC, etc., and is a communication unit for connecting to the Internet. The communication unit A107 is connected to the Internet C100 and communicates with the user terminal B100 via the Internet C100. Note that the communication unit A106 may be configured to connect to the Internet C100 via a router.

[0026] The user terminal B100 is composed of a control unit B101, a storage unit B102, a memory B103, a display unit B104, an operation unit B105, and a communication unit B106, and the control unit B101 controls each unit of the user terminal B100. Note that instead of the control unit B101 controlling the entire device, the entire device may be controlled by multiple hardware devices sharing the processing load.

[0027] The storage unit B102 is an electrically erasable and recordable non-volatile storage medium such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), etc. The storage unit B102 stores programs executed by the control unit B101, as well as databases, images, etc.

[0028] The memory B103 is, for example, a RAM (Random Access Memory), and is used as a buffer memory for temporarily storing various data, a work area for the control unit B101, and the like.

[0029] The display unit B104 displays a GUI (Graphical User Interface) for interactive operations. The display unit B104 does not necessarily have to be built into the user terminal B100.

[0030] The operation unit B105 is used to input instructions to the user terminal B100. The operation unit B105 includes, for example, a power button for instructing ON / OFF of the power of the user terminal B100 and operation buttons for instructing screen transitions. Note that the operation unit B105 does not necessarily have to be built into the user terminal B100.

[0031] The communication unit B106 is, for example, a NIC (Network Interface Card) incorporating a connector (RJ45) for connecting to Ethernet, a communication IC, etc., and is a communication unit for connecting to the Internet. The communication unit B107 is connected to the Internet C100 and communicates with the server A100 via the Internet C100. Note that the communication unit B106 may be configured to connect to the Internet C100 via a router.

[0032] The connection configuration of the present invention will be described below with reference to FIG. 2. FIG. 2 is a diagram showing the connection configuration according to this embodiment.

[0033] 2, the system of this embodiment includes a server A100 and a user terminal B100. The server A100 and the user terminal B100 will be described as having the same hardware configuration as the server A100 and the user terminal B100 shown in FIG.

[0034] The server A100 is a terminal that holds a database and a group of images managed by the database, and is made publicly available on the Internet C100.

[0035] User terminal B100 is a terminal of a person who uses this system to register or search for images, and can be connected to server A100 via the Internet C100. User terminal B100 can register images to server A100 by sending the images to server A100 via the Internet C100. Then, by sending a search keyword, it is possible to search for images related to the keyword from among the images registered on server A100.

[0036] Below, we will explain the information obtained by image analysis using AI and its characteristics with reference to Figures 3, 4, 5, and 6.

[0037] Figure 3 is an image taken with the raccoon's face in the center, and Figure 4 is an image taken so that the raccoon's entire body is visible.

[0038] As shown in Patent Document 1, tag information can be automatically generated from an image through image analysis using AI. For example, by performing image analysis on the image of the raccoon in Figure 3, tag information such as that shown in Figure 5 can be generated.

[0039] The tag information in Figure 5 includes information indicating the position of the subject within the image, as well as information indicating what the subject is and its reliability. In the tag information shown in Figure 5, the subject in Figure 3 is a raccoon, but the reliability value indicating a cat is higher than that of a raccoon. This is a characteristic of the AI ​​that performed this image analysis, and is presumed to be because the image data used for learning contained more images of cats than raccoons.

[0040] Similarly, by performing image analysis on the image of the raccoon in Figure 4, tag information like that shown in Figure 6 can be generated.

[0041] The reliability value for raccoon contained in the tag information in Figure 6 is higher than the reliability value for raccoon contained in the tag information in Figure 5. This is presumably because the raccoon image in Figure 3 used for image analysis shows only part of the body, centered on the face, whereas the raccoon image in Figure 4 shows the entire body, which is a distinctive feature.

[0042] It should be noted that the results of image analysis shown in FIGS. 5 and 6 are merely examples, and the results obtained by image analysis may be in other formats or have other contents.

[0043] Hereinafter, the region information of an image will be described with reference to FIGS.

[0044] FIG. 7 is a visualization of the region information for the raccoon image of FIG. 3, and FIG. 8 is a visualization of the region information for the raccoon image of FIG.

[0045] The area information may be generated by image analysis using AI, or may be generated using an optical sensor or acceleration sensor with technology such as SLAM (Simulataneous Localization and Mapping), or may be generated using a ranging sensor such as LiDAR (Light Detection and Ranging) or ToF (Time of Flight).

[0046] By using this area information, it is possible to obtain the position and size of the subject in the image.

[0047] The following describes how to calculate the evaluation value related to the display order of image search results.

[0048] In conventional image search systems, the reliability values ​​included in tag information, such as those shown in Figures 5 and 6, are often used directly to display image search results. In such systems, images with more reliable tags related to the search word are displayed higher in the search results, and images with less reliability are displayed lower in the search results. In contrast, in this embodiment, the image search results are displayed using evaluation values, which will be described below.

[0049] The proportion of the subject in the image is represented by α (0.0 to 1.0), and the reliability of the tag information obtained by AI image analysis is represented by β (0.0 to 1.0).

[0050] In this embodiment, the evaluation value is defined as α×n+β×m. The value of n is a coefficient that indicates how much the subject's proportion of the image is taken into consideration when calculating the evaluation value, and can be any numerical value. The value of m is a coefficient that indicates how much the reliability of the tag information is taken into consideration when calculating the evaluation value, and can be any numerical value.

[0051] The reason why the evaluation value is not determined solely based on the proportion α of the subject in the entire image is to prevent the image from appearing at the top of the search results when an image is searched using a tag that has been mistakenly attached to the subject. For example, when an image search is performed using the keyword "cat," if the evaluation value were determined solely based on the proportion α of the subject in the entire image, a raccoon image like the one shown in Figure 3 may appear at the top of the search results. However, by also taking into account the reliability value β of the tag information, the evaluation value of the raccoon image will be lower than that of other cat images with a similar proportion α of the subject in the entire image, and the raccoon image should appear lower than cat images during image searches.

[0052] A method for calculating the evaluation value for the image in Fig. 3 will be specifically described below with reference to Fig. 5 and Fig. 7. In the following calculations, the coefficients n and m of the evaluation value are set to n=0.7 and m=0.3, respectively.

[0053] First, the proportion α of the subject to the entire image is calculated from the region information shown in Fig. 7. Here, the explanation will be given assuming α=0.8.

[0054] Next, a reliability value β is obtained for the tag information of the subject shown in Fig. 5. The reliability value β of the tag "raccoon" is β=0.321568727.

[0055] Using the above values, the evaluation value of the tag "raccoon" for the subject in Figure 3 can be calculated as α×n+β×m=0.8×0.7+0.321568727×0.3=0.6564706181.

[0056] The tag information of the subject shown in FIG. 5 includes multiple tags in addition to raccoon, so the evaluation value is calculated for each tag using the method described above.

[0057] Similarly, if we calculate the evaluation value for the image in Figure 4, the evaluation value for the tag "raccoon" is α×n+β×m=0.1×0.7+0.7493764×0.3=0.29481292, which is a smaller evaluation value than the raccoon image in Figure 3.

[0058] As described above, even if the tag reliability value β is small for an image, the evaluation value in this embodiment may be large depending on how much consideration is given to the proportion of the subject in the entire image.

[0059] The above-described method for calculating the evaluation value is merely an example, and the definition and formula of the evaluation value may be different.

[0060] Hereinafter, an overview of the operations of the server A100 and the user terminal B100 in this embodiment will be described with reference to FIGS.

[0061] In the following explanation, the operation of each device will be explained using an example in which the user terminal B100 registers an image in the server A100.

[0062] FIG. 9 is a sequence diagram showing the flow of processing in which the user terminal B100 registers an image in the server A100.

[0063] The sequence shown in FIG. 9 starts in step S901 when an image registration start operation is received from the user.

[0064] In step S902, the user terminal B100 selects an image to be registered in the server A100 via the display unit B104 and the operation unit B105.

[0065] In step S903, the user terminal B100 transmits an image registration request to the server A100 through the communication unit B106. The parameters of the image registration request include at least the image data selected in step S902.

[0066] In step S904, the server A100 performs image analysis on the image data included in the image registration request received from the user terminal B100 in step S903, and acquires tag information.

[0067] In step S905, the server A100 acquires the area information of the image data.

[0068] In step S906, the server A100 calculates an evaluation value of the tag based on the tag information acquired in step S904 and the area information acquired in step S905.

[0069] In step S907, the server A100 associates the image data, tag information, and the evaluation value calculated in step S906, and registers them in a database managed by the storage unit A102.

[0070] In step S908, the server A100 transmits an image registration response to the user terminal B100 through the communication unit A106.

[0071] In the following explanation, the operation of each device will be explained using an example in which the user terminal B100 searches for an image on the server A100.

[0072] FIG. 10 is a sequence diagram showing the flow of processing in which the user terminal B100 searches for images on the server A100.

[0073] The sequence shown in FIG. 10 starts in step S1001 when an image search start operation is received from the user.

[0074] In step S1002, the user terminal B100 receives keywords relating to the image to be searched for from the user via the operation unit B105.

[0075] In step S1003, the user terminal B100 transmits an image search request to the server A100 through the communication unit B106. The parameters of the image search request include at least the keyword input in step S1002.

[0076] In step S1004, the server A100 searches the database managed by the storage unit A102 for images having tag information related to the keywords included in the image search request received from the user terminal B100 in step S1003.

[0077] In step S1005, the server A100 transmits an image search response to the user terminal B100 through the communication unit A106. The parameters of the image search response include at least the image search results acquired in step S1004.

[0078] Next, detailed operations of the server A100 and the user terminal B100 for realizing the above sequence will be described with reference to FIGS.

[0079] 11 is a flowchart of the server A100 in this embodiment. This flowchart starts when the system on the server A100 starts up.

[0080] In step S1101, the control unit A101 determines whether to end the process based on the status of each unit of the server A100. For example, the control unit A101 ends the process of this flowchart when an application stop operation is performed via the operation unit A105. If the process is to continue, the control unit A101 proceeds to step S1102.

[0081] In step S1102, the control unit A101 determines whether an image registration request has been received from the user terminal B100 via the communication unit A106. If the control unit A101 determines through the above processing that an image registration request has been received, the control unit A101 proceeds to step S1103. If the control unit A101 determines through the above processing that an image registration request has not been received, the control unit A101 proceeds to step S1112.

[0082] In step S1103, the control unit A101 determines whether tag information is included in the image data included in the image registration request received in step S1102. If the control unit A101 determines through the above processing that tag information is included in the image data, it proceeds to step S1104. If the control unit A101 determines through the above processing that tag information is not included in the image data, it proceeds to step S1105.

[0083] In step S1104, the control unit A101 acquires tag information included in the image data.

[0084] In step S1105, the control unit A101 reads out the AI ​​for generating tag information recorded in the storage unit A102 into the memory A103 and executes it on the image data. The control unit A101 obtains tag information through the process.

[0085] The processing in steps S1104 and S1105 corresponds to the processing in S904 in FIG.

[0086] In step S1106, the control unit A101 determines whether the image data included in the image registration request received in step S1102 includes region information. If the control unit A101 determines through the above processing that the image data includes region information, the control unit A101 proceeds to step S1107. If the control unit A101 determines through the above processing that the image data does not include region information, the control unit A101 proceeds to step S1108.

[0087] In step S1107, the control unit A101 acquires area information contained in the image data.

[0088] In step S1108, the control unit A101 reads out the AI ​​for generating region information recorded in the storage unit A102 into the memory A103 and executes it on the image data. The control unit A101 obtains region information through the process.

[0089] The processing in steps S1107 and S1108 corresponds to the processing in S905 in FIG.

[0090] In step S1109, the control unit A101 calculates an evaluation value based on the tag information acquired in step S1104 or step S1105 and the area information acquired in step S1107 or step S1108. The processing in this step corresponds to the processing in S906 in FIG. 9.

[0091] In step S1110, the control unit A101 associates the image data included in the image registration request received in step S1102 with the tag information acquired in step S1104 or step S1105 and the evaluation value acquired in step S1109. Then, the association is registered in a database managed by the storage unit A102. The processing in this step corresponds to the processing in S907 in FIG. 9.

[0092] In step S1111, the control unit A101 sends an image registration response to the user terminal B100 via the communication unit A106 to notify the user terminal B100 that the image registration process has been completed. After this step is completed, the process proceeds to step S1101. The process of this step corresponds to the process of S908 in FIG. 9.

[0093] In step S1112, the control unit A101 determines whether an image search request has been received from the user terminal B100 via the communication unit A106. If the control unit A101 determines through the above processing that an image search request has been received, the control unit A101 proceeds to step S1113. If the control unit A101 determines through the above processing that an image search request has not been received, the control unit A101 proceeds to step S1101.

[0094] In step S1113, the control unit A101 acquires the keyword included in the image search request received in step S1112, and searches the database managed by the storage unit A102 for image data associated with tag information that matches the keyword.

[0095] In step S1114, the control unit A101 sorts the search results acquired in step S1113 based on the evaluation values ​​associated with the image data.

[0096] The processing in steps S1113 and S1114 corresponds to the processing in step S1004 in FIG.

[0097] In step S1115, the control unit A101 transmits an image search response to the user terminal B100 via the communication unit A106. The image search response includes at least the search result values ​​sorted in step S1114. After this step is completed, the process proceeds to step S1101. The process of this step corresponds to the process of step S1005 in FIG. 10.

[0098] 12 is a flowchart of the user terminal B100 in this embodiment. This flowchart starts when the system is started in the user terminal B100.

[0099] In step S1201, the control unit B101 determines whether to end the process based on the status of each unit of the user terminal B100. For example, the control unit B101 ends the process of this flowchart when an application stop operation is performed via the operation unit B105. If the process is to continue, the control unit B101 proceeds to step S1202.

[0100] In step S1202, the control unit B101 determines whether the user has performed an image registration start operation via the operation unit B105. If the control unit B101 determines through the above processing that an image registration start operation has been performed, the control unit B101 proceeds to step S1203. If the control unit B101 determines through the above processing that an image registration start operation has been performed, the control unit B101 proceeds to step S1205.

[0101] In step S1203, the control unit B101 selects image data to be transmitted to the server A100 via the display unit B104 and operation unit B105. The processing in this step corresponds to the processing in step S902 in FIG.

[0102] In step S1204, the control unit B101 transmits an image registration request to the server A100 via the communication unit B106. The image registration request includes at least the image data selected in step S1203. After this step is completed, the process proceeds to step S1201. The process of this step corresponds to the process of step S903 in FIG. 9.

[0103] In step S1205, the control unit B101 determines whether the user has performed an image search start operation via the operation unit B105. If the control unit B101 determines through the above processing that an image search start operation has been performed, the control unit B101 proceeds to step S1206. If the control unit B101 determines through the above processing that an image search start operation has been performed, the control unit B101 proceeds to step S1201.

[0104] In step S1206, the control unit B101 receives input of search keywords to be sent to the server A100 via the display unit B104 and operation unit B105. The processing in this step corresponds to the processing in step S1002 in FIG.

[0105] In step S1207, the control unit B101 transmits an image search request to the server A100 via the communication unit B106. The image search request includes at least the keywords input in step S1206. After this step is completed, the process proceeds to step S1201. The process of this step corresponds to the process of step S1003 in FIG. 10.

[0106] As described above, in the system of this embodiment, which is composed of the server A100 and the user terminal B100, the server A100 calculates an evaluation value from the tag information and area information of an image, and then sorts and returns image search results to the user terminal B100 based on the evaluation value. This makes it possible for images that could not be found using conventional search methods or images that did not appear at the top of search results to appear at the top of search results, making them easier to find in searches.

[0107] (Second embodiment) In the first embodiment, the user terminal performs an image search for images managed on the server, but it is also possible to perform a search for images stored in the user terminal on the user terminal.

[0108] In this embodiment, the image search system is closed to the user terminal, and searches for images that match the keywords entered into the user terminal from among the images stored in the user terminal, and displays the search results on the user terminal.

[0109] Since this embodiment has many parts in common with the first embodiment, the following description will focus on the parts unique to this embodiment.

[0110] Hereinafter, a method for calculating evaluation values ​​relating to the display order of image search results in this embodiment, which is different from that in the first embodiment, will be described.

[0111] If the image for which the evaluation score is calculated contains a certain amount of background, it is possible that the image was not primarily taken of the subject in the image, but rather a landscape photo. On the other hand, if the proportion of background in the image is less than a certain amount, it is likely that the photographer tried to capture only the subject in the image, and we believe that such images should appear higher in image searches.

[0112] Therefore, in this embodiment, the value of the proportion α of the subject area is calculated after excluding the background area.

[0113] A specific method for calculating the evaluation value for the image in Figure 3 will be described below using Figures 5 and 7. In the following calculations, the coefficients n and m of the evaluation value are set to n = 0.7 and m = 0.3, respectively. Regarding whether to calculate the value of α excluding the background region, in this embodiment, if the proportion θ of the background to the entire image is less than half (θ < 0.5), the value of α is calculated excluding the background region.

[0114] First, the ratio θ of the background to the entire image is calculated from the region information shown in Fig. 7. Here, the explanation will be given assuming θ=0.2.

[0115] Next, from the area information shown in Figure 7, the proportion α that the subject occupies in the entire image is calculated. However, since the proportion θ that the background occupies in the entire image is less than half of the total (θ<0.5), the value of α is calculated excluding the background area, resulting in α=1.0.

[0116] Next, a reliability value β is obtained for the tag information of the subject shown in Fig. 5. The reliability value β of the tag "raccoon" is β=0.321568727.

[0117] Using the above values, the evaluation value of the tag "raccoon" for the subject in Figure 3 can be calculated as α×n+β×m=1.0×0.7+0.321568727×0.3=0.7964706181.

[0118] The evaluation value of the tag "raccoon" in this embodiment is higher than that in the first embodiment by the amount of the ratio α of the subject to the total other objects excluding the background area.

[0119] The above-described method for calculating the evaluation value is merely an example, and the definition and formula of the evaluation value may be different.

[0120] The configuration of each device is the same as that of the user terminal B100 in FIG. 1 described in the first embodiment, and therefore a description thereof will be omitted.

[0121] Since only the user terminal B100 appears in this embodiment, the connection configuration of each device will not be described here.

[0122] Next, detailed operation of the user terminal B100 will be described with reference to Fig. 13. Fig. 13 is a flowchart of the user terminal B100 in this embodiment. This flowchart starts when the system is started up in the user terminal B100.

[0123] In step S1301, the control unit B101 determines whether to end the process based on the status of each unit of the user terminal B100. For example, the control unit B101 ends the process of this flowchart when an application stop operation is performed via the operation unit B105. If the process is to continue, the control unit B101 proceeds to step S1302.

[0124] In step S1302, the control unit B101 determines whether the user has performed an image search start operation via the operation unit B105. If the control unit B101 determines through the above processing that an image search start operation has been performed, the control unit B101 proceeds to step S1303. If the control unit B101 determines through the above processing that an image search start operation has been performed, the control unit B101 proceeds to step S1301.

[0125] In step S1303, the control unit B101 receives a search keyword input via the display unit B104 and the operation unit B105.

[0126] In step S1304, the control unit B101 acquires image data to be searched that is recorded in the storage unit B102.

[0127] In step S1305, the control unit B101 determines whether the image data acquired in step S1304 includes tag information. If the control unit B101 determines through the above processing that the image data includes tag information, the control unit B101 proceeds to step S1306. If the control unit B101 determines through the above processing that the image data does not include tag information, the control unit B101 proceeds to step S1307.

[0128] In step S1306, the control unit B101 acquires tag information included in the image data.

[0129] In step S1307, the control unit B101 reads out the AI ​​for generating tag information recorded in the storage unit B102 into the memory B103 and executes it on the image data. The control unit B101 obtains tag information through the process.

[0130] In step S1308, the control unit B101 registers the tag information acquired in step S1307 in the image data as meta information.

[0131] In step S1309, the control unit B101 determines whether the image data acquired in step S1304 includes region information. If the control unit B101 determines through the above processing that the image data includes region information, the control unit B101 proceeds to step S1310. If the control unit B101 determines through the above processing that the image data does not include region information, the control unit B101 proceeds to step S1311.

[0132] In step S1310, the control unit B101 acquires area information contained in the image data.

[0133] In step S1311, the control unit B101 reads out the AI ​​for generating region information recorded in the storage unit B102 into the memory B103 and executes it on the image data. The control unit B101 obtains region information through the process.

[0134] In step S1312, the control unit B101 registers the area information acquired in step S1311 in the image data as meta information.

[0135] In step S1313, the control unit B101 calculates an evaluation value based on the tag information acquired in step S1306 or step S1307 and the area information acquired in step S1310 or step S1311.

[0136] The processes in steps S1305 to S1313 are performed on all images to be searched acquired in step S1304.

[0137] In step S1314, the control unit B101 searches for images having tag information related to the keyword input in step S1303 from among the search target images acquired in step S1304.

[0138] In step S1315, the control unit B101 rearranges the search results acquired in step S1314 based on the evaluation values ​​acquired in step S1313.

[0139] In step S1316, the control unit B101 displays the search result values ​​sorted in step S1315 via the display unit B104.

[0140] As described above, in a system consisting only of the user terminal B100 of this embodiment, the user terminal B100 calculates an evaluation value from the tag information and area information of an image, and displays image search results based on the evaluation value. This makes it possible to make images that could not be found using conventional search methods or that did not appear in the top search results appear more likely to appear in the top search results, making them easier to find in searches.

[0141] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0142] The present invention can also be realized by providing a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having the computer of the system or device read and execute the program. The computer has one or more processors or circuits, and may include multiple separate computers or a network of multiple separate processors or circuits to read and execute computer-executable instructions.

[0143] The processor or circuitry may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gateway (FPGA), a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

Claims

1. A search device that searches for images associated with tag information that matches a keyword from among a plurality of search target images, means for acquiring, from the search target image, a reliability value of tag information associated with the search target image; a means for acquiring area information of a subject appearing in the search target image from the search target image and calculating the proportion of the subject occupying the entire image; a means for calculating an evaluation value for the search target image using the reliability value of the tag information and the proportion of the subject in the entire image; A search device that searches for images associated with tag information that matches the keyword from among multiple search target images, sorts the images included in the image search results based on the evaluation value for the image, and returns the sorted image search results.

2. A means for calculating the ratio of the background to the entire image for the search target image, 2. The search device according to claim 1, wherein, if the ratio of the background to the entire image is less than a certain value, the ratio of the subject to the entire image is calculated by excluding the background from the image.

3. A search device as described in Claim 2, characterized in that if the proportion of the background in the entire image is equal to or greater than a certain value, the background is included in the image when calculating the proportion of the subject in the entire image.

4. 4. The search device according to claim 1, further comprising: a receiving unit for receiving an input of the keyword from a user.

5. 5. The search device according to claim 4, wherein the receiving means is a touch panel.

6. A method executed by a search device for searching for an image associated with tag information matching a keyword from among a plurality of search target images, comprising: obtaining, from the search target image, a reliability value of tag information associated with the search target image; a step of acquiring area information of a subject appearing in the search target image from the search target image and calculating a ratio of the subject to the entire image; calculating an evaluation value for the search target image using the reliability value of the tag information and the ratio of the subject to the entire image; A method executed by a search device, comprising the steps of searching for images associated with tag information that matches the keyword from among the plurality of search target images, sorting the images included in the image search results based on the evaluation value for the image, and returning the sorted image search results.

7. The method further comprises a step of calculating a ratio of the background to the entire image for the search target image, 7. The method executed by the search device according to claim 6, wherein if the proportion of the background in the entire image is less than a certain value, the proportion of the subject in the entire image is calculated by excluding the background from the image.

8. A method executed by the search device described in Claim 7, characterized in that if the proportion of the background in the entire image is equal to or greater than a certain value, the background is included in the image when calculating the proportion of the subject in the entire image.

9. 9. The method executed by the search device according to claim 6, further comprising a receiving step of receiving an input of the keyword from a user.

10. A computer-readable program for causing a computer to function as each of the means of the search device according to any one of claims 1 to 5.

Images