Image recognition support device, image recognition support method, and image recognition support program

The image recognition support device generates summarized sentences using a language model to enhance image search accuracy by converting search queries into features, addressing the limitation of existing systems that do not adapt linguistic features to queries.

JP2024165586APending Publication Date: 2024-11-28HITACHI LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023081892
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-17
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Existing image search systems using multimodal features fail to generate linguistic features in response to a search query, leading to suboptimal search accuracy when the accompanying text is unrelated to the query.

Method used

An image recognition support device that includes a text input section, storage section, image feature conversion, text processing unit, and image search unit, which generates summarized sentences based on search queries using a language model like GPT-3, converts these into features, and performs accurate image searches.

Benefits of technology

Enables highly accurate image searches by generating summarized sentences according to the search query, improving search precision, especially for complex queries.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024165586000001_ABST
    Figure 2024165586000001_ABST
Patent Text Reader

Abstract

To provide a technique which can perform image retrieval with high accuracy.SOLUTION: An image recognition support device 100 includes: a text input section 111 which receives inputs of a retrieval query and an instruction sentence; a retrieval object image DB 121 which holds a retrieval object image; an image feature amount conversion section 114 which converts the retrieval object image into image feature amount; an image recognition section 112 which outputs image associated text information related to the retrieval object image; a text processing section 113 which outputs a summarized sentence according to the retrieval query on the basis of the instruction sentence from the image associated text information; a sentence feature amount conversion section 115 which converts the output summarized sentence into summarized sentence feature amount; and an image retrieval section 116 which acquires a retrieval result by retrieving images using the summarized sentence, the summarized sentence feature amount, the retrieval object image, and image feature amount.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an image recognition support device, an image recognition support method, and an image recognition support program. [Background technology]

[0002] In image search systems that use text as a search query, there is known a search system that uses multimodal features, in which not only the image features of the search target but also the linguistic features of the text accompanying the search target image are used to assist in the search. For example, in a product image search system where the search target image is a product image, examples of the text accompanying the search target image in this case include product description text, text embedded in the Exif information of the product image, and text extracted from the product image by Image Captioning technology. By converting the text accompanying the search target image into linguistic features and using them together with the image features for the search, it becomes possible to perform a search with higher accuracy than when only image features are used.

[0003] Here, image features represent feature vectors extracted by inputting an image into a neural network model such as CNN or Transformer, and similarly, linguistic features represent feature vectors extracted by inputting text into a neural network model such as CNN or Transformer.

[0004] However, the text accompanying the image may be unrelated to the search query, and the added linguistic features may not improve the accuracy of the search. Therefore, in order to improve the accuracy of the search, it is necessary to generate linguistic features according to the search query.

[0005] For example, Patent Document 1 is known as a prior patent relating to a search system using multimodal features that uses linguistic features of text accompanying an image to be searched for in addition to image features of the image to be searched for to assist in the search.

[0006] In addition, Non-Patent Documents 1 and 2 propose a technology that calculates the labels and coordinate information of various objects in an image as text, inputs this together with instructions into a language model such as GPT-3, and creates a caption that is more suitable for describing the image. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2012-003357 A [Non-patent literature]

[0008] [Non-Patent Document 1] Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language, Arxiv, 2022. [Non-Patent Document 2] Visual Clues: Bridging Vision and Language Foundations for Image Paragraph Captioning, NeurIPS, 2022. Summary of the Invention [Problem to be solved by the invention]

[0009] However, since the technique disclosed in Patent Document 1 uses linguistic features of text prepared in advance, it is not possible to generate linguistic features corresponding to a search query.

[0010] In addition, the technologies in Non-Patent Documents 1 and 2 can generate caption text that contains more information than conventional Image Captioning technologies, and therefore the linguistic features of the generated text may lead to improved search accuracy compared to conventional techniques. However, they do not describe a method for generating linguistic features in response to a search query.

[0011] Therefore, an object of the present invention is to provide a technique that enables highly accurate image search. [Means for solving the problem]

[0012] In order to solve the above-mentioned problems, one representative image recognition assistance device of the present invention comprises a text input unit that accepts input of a search query and an instruction text, a memory unit that holds an image to be searched for, an image feature conversion unit that converts the image to be searched for into an image feature, an image recognition unit that outputs image-accompanying text information related to the image to be searched for, a text processing unit that outputs a summarized sentence corresponding to the search query based on the instruction text from the image-accompanying text information, a sentence feature conversion unit that converts the output summarized sentence into summarized sentence feature text, and an image search unit that searches for images using the summarized sentence, the summarized sentence feature text, the image to be searched for, and the image feature text to obtain search results. Effect of the Invention

[0013] According to the present invention, a highly accurate image search can be performed.

[0014] Problems, configurations and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0015] [Figure 1] 1 is a diagram illustrating an overview of a configuration of an image recognition support device according to an embodiment of the present invention. [Diagram 2] FIG. 11 is a diagram showing an example of image-accompanying text information generated by an image recognition unit. [Diagram 3] FIG. 13 is a diagram showing an example of information stored in a search target image DB. [Figure 4] 1 is a block diagram showing an example of a function of an image recognition support device according to a first embodiment. [Diagram 5] 4 is a flowchart illustrating an example of a process executed by the image recognition support device according to the first embodiment. [Figure 6]FIG. 11 is a block diagram showing an example of functions of an image recognition support device according to a second embodiment. [Figure 7] 13 is a flowchart illustrating an example of a process executed by the image recognition support device according to the second embodiment. [Figure 8] FIG. 11 is a block diagram showing an example of the functions of an image recognition support device according to a third embodiment. [Figure 9] 11 is a flowchart illustrating an example of a process executed by the image recognition support device according to the third embodiment. [Figure 10] FIG. 13 is a block diagram showing an example of the functions of an image recognition support device according to a fourth embodiment. [Figure 11] 13 is a flowchart showing an example of a process executed by the image recognition support device according to the fourth embodiment. [Figure 12] FIG. 13 is a block diagram showing an example of the functions of an image recognition support device according to a fifth embodiment. [Figure 13] 13 is a flowchart showing an example of a process executed by the image recognition support device according to the fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0017] For clarity of explanation, the following description and drawings are omitted and simplified as appropriate. In addition, the present invention is not limited to the present embodiment, and any application example that meets the idea of ​​the present invention is included in the technical scope of the present invention. In addition, unless otherwise specified, the number of each component mentioned may be singular or plural. <System configuration> FIG. 1 is a diagram illustrating an outline of the configuration of an image recognition support device according to this embodiment.

[0018] The image recognition assistance device 100 includes a control unit 110 that controls various processes such as image recognition and text processing, a memory unit 120 that stores an image DB and a trained model, and a display unit 180 that displays detection results, etc.

[0019] The control unit 110 includes the following functional units: a text input unit 111, an image recognition unit 112, a text processing unit 113, an image feature conversion unit 114, a text feature conversion unit 115, an image search unit 116, a detailed information attachment unit 117, and an image conversion unit 118.

[0020] The memory unit 120 stores a search target image DB 121 (DB: database), an image recognition model 122, a language model 123, an image feature conversion model 124, a text feature conversion model 125, an image search model 126, a detailed information assignment model 127, an image conversion model 128, and various programs 129.

[0021] The image recognition model 122, the language model 123, the image feature conversion model 124, the sentence feature conversion model 125, the image retrieval model 126, the detailed information assignment model 127, and the image conversion model 128 are trained models used in the image recognition unit 112, the text processing unit 113, the image feature conversion unit 114, the sentence feature conversion unit 115, the image retrieval unit 116, the detailed information assignment unit 117, and the image conversion unit 118, respectively.

[0022] The image feature conversion model 124, the text feature conversion model 125, the image search model 126, the detailed information assignment model 127, and the image conversion model 128 are configured using neural network models such as CNN and Transformer.

[0023] The image recognition model 122 used by the image recognition unit 112 is configured with a neural network such as CNN or Transformer. Examples of these include models such as CLIP (Contrastive Language-Image Pre-Training) that can output attributes on an image by using a similarity comparison with text, GLIP (Grounded Language-Image Pre-training) that detects various objects in an image, and BLIP (Bootstrapping Language-Image Pre-training) that outputs a caption sentence for an image, but various other models can also be used.

[0024] As the language model 123 used by the text processing unit 113, a language model capable of summarizing text information such as GPT-3 is used. Note that the language model is not limited to GPT-3, and various other models can be used as long as they can appropriately summarize text information.

[0025] The text input unit 111 accepts input from a user. For example, the text input unit 111 accepts input of a search query and an instruction, and edits of a summarized sentence output from the text processing unit 113. The text input unit 111 also accepts input to the search target image DB 121.

[0026] The image recognition unit 112 extracts image recognition unit extracted text and metadata from the image acquired from the search target image DB 121. The image recognition unit 112 also combines the image recognition unit extracted text and metadata to generate image-accompanying text information 140 as shown in FIG.

[0027] FIG. 2 is a diagram showing an example of image-accompanying text information 140 generated by image recognition unit 112. As shown in FIG.

[0028] The image-accompanying text information 140 includes metadata of the image obtained from the search target image DB 121, and text extracted by the image recognition unit 112 including object labels, positional information of objects on the image, captioning text, etc., output by the image recognition unit 112 for the image obtained from the search target image DB 121.

[0029] Returning to Fig. 1, the search query and instruction sentence input by the text input unit 111, and the image-accompanying text information 140 are input to the text processing unit 113. In order to output a summarized sentence corresponding to the search query, an instruction sentence for summarization is input to the text processing unit 113 together with the search query. The text processing unit 113 uses the language model 123 to generate a summarized sentence corresponding to the search query.

[0030] The image feature conversion unit 114 extracts image features of the images to be searched during image search.

[0031] The text feature conversion unit 115 converts the summarized text generated by the text processing unit 113 into text feature data for use in search.

[0032] The image search unit 116 searches for images based on the image features, summarized text features, the search query, and the instruction text, and obtains the search results. The image search unit 116 uses the summarized text features and image features related to the search target images outputted above to perform a search using a feature matching method that is typically used in image searches using neural networks.

[0033] The detailed information adding unit 117 adds detailed information to the images in the search results. The image conversion unit 118 converts the images in the search results.

[0034] The search target image DB 121 holds image data of search target images.

[0035] FIG. 3 is a diagram showing an example of information stored in the search target image DB 121. As shown in FIG.

[0036] The search target image DB 121 holds metadata of the search target images as well as image data of the search target images.

[0037] Moreover, the image recognition unit extracted text extracted by the image recognition unit 112 is appropriately stored in the search target image DB 121. Furthermore, the image feature extracted by the image feature conversion unit 114 is also appropriately stored in the search target image DB 121.

[0038] If the text extracted by the image recognition unit is stored in the search target image DB 121, the text processing unit 113 reads it out and uses it to generate summarized text. If the image feature amount is stored in the search target image DB 121, the image search unit 116 reads out the image feature amount and uses it for image search. This makes it possible to reduce the processing time when performing multiple searches.

[0039] An example of an instruction sentence input to the text input unit 111 is, for example, "Regarding <text information>, if a word similar to <search query> is included, extract that word, otherwise output None." Here, <text information> refers to image-accompanying text information 140. Moreover, actual text information and a search query are input in <> according to a search. By providing such an instruction sentence to a language model capable of summarizing sentences such as GPT-3, it becomes possible to generate summarized sentences according to a search query from text information containing various information.

[0040] Examples of metadata for images to be searched include text information accompanying the image as EXIF ​​information, latitude and longitude information, image size information, and other information previously held about the image (e.g., the actual size of buildings in the image).By using this information together with the image recognition unit extracted text output from image recognition unit 112, text processing unit 113 can output summarized text such as "images containing buildings within 10 meters of a river," and image searches using the summarized text can be used to support accurate searches.

[0041] The search target images held in the search target image DB 121 may be either color images or monochrome images. Each image may be an RGB image, an IR image (IR: Infrared), a CG image (CG: Computer Graphics), or any other type of image.

[0042] Furthermore, the image recognition support device 100 may be provided with a GUI that displays, on the display unit 180, the image-accompanying text information before processing by the text processing unit 113 and the summarized sentences after being summarized by the text processing unit 113, allowing the user to compare them. This enables the user to check whether the text processing unit 113 has correctly summarized the image-accompanying text information and output appropriate summarized sentences.

[0043] In addition to the above, the user may be allowed to edit the summarized sentence using the text input unit 111 while displaying on the display unit 180 the image-accompanying text information before processing by the text processing unit 113 and the summarized sentence after summary by the text processing unit 113 for comparison. This makes it possible to output the summarized sentence and summarized sentence feature quantities appropriate to the user's intention, enabling more accurate image search. <Hardware related configuration> Various hardware components constitute the image recognition support device 100. As an example, the processing of the control unit 110 is implemented by a processing device such as a central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), or a field-programmable gate array (FPGA).

[0044] The processing of the storage unit 120 is realized by a memory device or a storage medium such as a ROM (Read Only Memory), a RAM (Random Access Memory), a HDD (Hard Disk Drive), an SSD (Solid State Drive), etc. The display unit 180 is realized by a liquid crystal display, an organic EL (Electro-Luminescence) display, etc. The text input unit 111 is realized by a mouse, a keyboard, etc.

[0045] Also, for example, if the memory unit 120 and the control unit 110 are on separate devices and need to be connected, network communication between the devices is performed by a communication device consisting of a NIC (Network Interface Card), a wireless communication module, a USB (Universal Serial Interface) module, a serial communication module, or the like.

[0046] Each function of the image recognition support device 100 is realized by the control unit 110 reading and executing a program 129 stored in the storage unit 120. Moreover, each process described below is realized by the control unit 110 executing the program 129 held in the storage unit 120. Moreover, the above-mentioned program 129 can be distributed by being recorded on a recording medium, for example. <Image search processing> FIG. 4 is a block diagram illustrating an example of functions of the image recognition support apparatus according to the first embodiment, and FIG. 5 is a flowchart illustrating an example of processing executed by the image recognition support apparatus according to the first embodiment.

[0047] As a process flow, first, the text input unit 111 accepts a search query and an instruction input by the user (step S11).

[0048] Next, the image acquired from the search target image DB 121 is input to the image recognition unit 112, which extracts image recognition unit extracted text from the input image (step S12). The image recognition unit 112 also extracts metadata of the input image (step S13). The image recognition unit extracted text and metadata are combined to generate image-accompanying text information 140 as shown in FIG. 2 (step S14).

[0049] Thereafter, the search query and instruction received in step S11 and the image-accompanying text information 140 generated in step S14 are input to the text processing unit 113 (step S15). The text processing unit 113 uses the language model 123 to generate a summarized sentence based on the instruction from the image-accompanying text information 140 in accordance with the search query (step S16).

[0050] To use these summarized sentences in searches, the sentence feature converter 115 converts them into sentence features. The image feature converter 114 also converts the image into image features (step S17). The image, image feature, summarized sentence, and sentence feature are then input to the image search unit 116 together with the search query and instruction text (step S18), and the image search unit 116 obtains the search results (step S19), terminating the process.

[0051] 5, the image recognition support device 100 outputs summarized text corresponding to the search query by the text processing unit 113, converts it into summarized text features by the text feature conversion unit 115, and uses the converted summarized text features together with image features to support accurate search. This makes it possible to perform accurate image search even for detailed search query text that was difficult to search for in the past, such as "an image of a man in red clothing standing by a river."

[0052] Note that the processing from step S11 to step S18 in FIG. 5 is also executed in the processing of FIGS. 7, 9, 11, and 13 described below. Therefore, in the following explanation, the processing from step S11 to step S18 is collectively referred to as step S001, and the explanation thereof will be omitted. <Question-answering processing> FIG. 6 is a block diagram illustrating an example of functions of the image recognition support apparatus according to the second embodiment, and FIG. 7 is a flowchart illustrating an example of processing executed by the image recognition support apparatus according to the second embodiment.

[0053] The process in FIG. 7 differs from that in FIG. 5 in that a question answering process in steps S60 to S62 is executed after step S19.

[0054] After obtaining the search results from image search unit 116 in step S19, text input unit 111 accepts input of a question about the search results by the user (eg, "What object is near the target image?") (step S60).

[0055] The input question text and the image-accompanying text information 140 acquired in step S14 of Fig. 5 are input to the text processing unit 113, which generates a question-response sentence based on the question text and the image-accompanying text information 140 (step S61). Next, the search result including the question-response sentence is displayed on the display unit 180 (step S62), and the process ends.

[0056] According to the process of FIG. 7, the text processing unit 113 generates a question response sentence based on the question sentence and the image-accompanying text information 140, and displays the search results including the question response sentence on the display unit 180, so that the user can carry out an operation to ask a detailed question about the searched image.

[0057] Regarding question answering, in addition to the language model 123 used in the text processing unit 113, a separate question answering model may be prepared and used. <Detailed information addition process> FIG. 8 is a block diagram illustrating an example of functions of the image recognition support apparatus according to the third embodiment, and FIG. 9 is a flowchart illustrating an example of processing executed by the image recognition support apparatus according to the third embodiment.

[0058] The process in FIG. 9 differs from that in FIG. 5 in that detailed information adding processing in steps S70 to S71 is executed after step S19.

[0059] In FIG. 8, for example, attribute list text such as "yellow, blue, circle, square, . . . " is prepared in advance in text input unit 111.

[0060] After obtaining the search results from the image search unit 116 in step S19, the attribute list text and the search results are input to the detailed information adding unit 117, which generates detailed information (e.g., 20s to 30s, wearing green clothing, wearing a hat, male) for the image of the search results based on the attribute list text and the search results (step S70). Next, the search results including the detailed information are displayed on the display unit 180 (step S71), and the process ends.

[0061] Here, as the detailed information attachment model 127, CLIP (Contrastive Language-Image Pre-Training) or the like that can output various attributes on an image by using a similarity comparison with text can be used.

[0062] According to the process of FIG. 9, the detailed information adding unit 117 generates detailed information of the images in the search results based on the attribute list text and the search results, and displays the search results including the detailed information on the display unit 180, so that it becomes possible to display more detailed information about the search results, such as listing detailed attributes of the search result images. <Image conversion processing> FIG. 10 is a block diagram illustrating an example of functions of the image recognition support apparatus according to the fourth embodiment, and FIG. 11 is a flowchart illustrating an example of processing executed by the image recognition support apparatus according to the fourth embodiment.

[0063] The process in FIG. 11 differs from that in FIG. 5 in that image conversion processes in steps S80 to S82 are performed after step S19.

[0064] After obtaining the search results from image search unit 116 in step S19, text input unit 111 accepts an input of an image conversion instruction sentence by the user (for example, the search result image is morning, but convert it to look like evening) (step S80).

[0065] The input image conversion instruction sentence and the search result acquired in step S19 are input to image conversion unit 118, which converts the image of the search result based on the image conversion instruction sentence (step S81). Next, the search result after image conversion is displayed on display unit 180 (step S82), and the process ends.

[0066] Here, a neural network model such as Stable Diffusion can be used as the image transformation model 128. In addition, various other image transformation models such as a model derived from a Generative Adversarial Network, a model derived from a Variational Auto-Encoder, or a model derived from Flow-Based Generative Models can also be used.

[0067] According to the process of Fig. 11, the image conversion unit 118 converts the image of the search result based on the image conversion instruction sentence, so that the search result image can be converted to match the image conversion instruction sentence entered by the user. This makes it easier to imagine the searched image in a different state, for example. <Image conversion processing for specific areas> FIG. 12 is a block diagram illustrating an example of functions of the image recognition support apparatus according to the fifth embodiment, and FIG. 13 is a flowchart illustrating an example of processing executed by the image recognition support apparatus according to the fifth embodiment.

[0068] The process in FIG. 13 differs from that in FIG. 5 in that image conversion processes in steps S90 to S92 are performed after step S19.

[0069] After obtaining the search results from the image search unit 116 in step S19, the text input unit 111 accepts input of image conversion instruction text from the user (e.g., what would it look like if there was a car next to this man?) and conversion target position information (e.g., the user specifies an arbitrary position next to the man in the search result image) (step S90).

[0070] The input image conversion instruction sentence, the conversion target position information, and the search results acquired in step S19 are input to the image conversion unit 118, which converts a specific portion of the image in the search result based on the image conversion instruction sentence and the conversion target position information (step S91). Next, the search results after image conversion are displayed on the display unit 180 (step S92), and the process ends.

[0071] As the image transformation model 128, a model such as Stable Diffusion as in the third embodiment or various other image transformation models can be used.

[0072] According to the process of Fig. 13, the image conversion unit 118 converts a specific part of the image of the search result based on the image conversion instruction sentence and the conversion target position information, so that it is possible to convert the designated area of ​​the search result image to the image conversion instruction sentence. This makes it easier to imagine another state of the search image, for example.

[0073] As described above, the image recognition support device 100 of this embodiment generates summarized sentences according to a search query using a language model, and uses the summarized sentences for a search, thereby enabling highly accurate image search.

[0074] In addition, various processes can be performed on the images in the search results, such as question answering, adding detailed information, image conversion, and image conversion of specific parts.

[0075] Furthermore, if the summarized sentences generated by the language model are inappropriate for retrieval, they can be appropriately corrected by user input, enabling more accurate image retrieval.

[0076] The present invention is not limited to the above-described embodiment, and can be implemented using any components without departing from the scope of the present invention. The above-described embodiments and modifications are merely examples, and the present invention is not limited to these contents as long as the features of the invention are not impaired. In addition, although various embodiments and modifications have been described above, the present invention is not limited to these contents. Other aspects conceivable within the scope of the technical idea of ​​the present invention are also included in the scope of the present invention.

[0077] For example, some of the functions provided in each device in the embodiments may be provided in another device, and functions provided in another device may be provided in the same device. [Explanation of symbols]

[0078] 100 Image recognition support device 110 Control section 120 Storage section 180 Display section

Claims

1. a text input section for accepting input of a search query and a command; A storage unit for storing search target images; an image feature quantity conversion unit that converts the search target image into an image feature quantity; an image recognition unit that outputs image-accompanying text information related to the search target image; a text processing unit that outputs a summarized sentence corresponding to the search query from the image-accompanying text information based on the instruction sentence; a text feature quantity conversion unit for converting the output summarized text into summarized text feature quantities; an image search unit that searches for images using the summarized text, the summarized text feature amount, the search target image, and the image feature amount to obtain search results; An image recognition assistance device comprising:

2. 2. The image recognition support device according to claim 1, The storage unit stores the image feature converted by the image feature conversion unit, The image search unit is an image recognition support device that searches for images using image features stored in the memory unit.

3. 2. The image recognition support device according to claim 1, The storage unit stores the image-accompanying text information output by the image recognition unit, The text processing unit is an image recognition support device that outputs the summarized sentence from the image-accompanying text information stored in the storage unit.

4. 2. The image recognition support device according to claim 1, Further comprising a display unit, the text input unit accepts input of a question regarding the search results; the text processing unit generates a response sentence to the question sentence based on the question sentence and the image-accompanying text information; The display unit is an image recognition support device that displays the search results and the response sentence.

5. 2. The image recognition support device according to claim 1, a detailed information adding unit that generates detailed information about the search results based on the search results and a previously prepared attribute list text; a display unit that displays the search results and the detailed information; The image recognition support device further comprises:

6. 2. The image recognition support device according to claim 1, Further comprising an image conversion unit, The text input unit accepts input of an image conversion instruction sentence, The image conversion unit is an image recognition support device that converts the search results based on the image conversion instruction sentence.

7. 2. The image recognition support device according to claim 1, Further comprising an image conversion unit, The text input unit receives input of an image conversion instruction sentence and conversion target position information; The image conversion unit is an image recognition support device that converts a specific portion of the search result based on the image conversion instruction sentence and the conversion target position information.

8. 2. The image recognition support device according to claim 1, The image recognition support device further comprises a display unit that displays the image-accompanying text information and the summarized sentence.

9. 9. The image recognition support device according to claim 8, The text input unit is an image recognition support device that accepts editing of the summarized text displayed by the display unit.

10. 1. An image recognition support method for an image recognition support device having a storage unit for storing a search target image, comprising: a text input step for accepting input of a search query and a directive; an image feature conversion step of converting the search target image into an image feature; an image recognition step of outputting image-accompanying text information related to the search target image; a text processing step of outputting a summarized text corresponding to the search query from the image-accompanying text information based on the instruction sentence; a text feature conversion step of converting the output summarized text into summarized text feature values; an image retrieval step of searching for images using the summarized text, the summarized text feature amount, the search target image, and the image feature amount to obtain search results; The image recognition assistance method includes:

11. An image recognition support program executed by an image recognition support device having a storage unit for storing a search target image, The image recognition support device includes: a text input process for accepting input of a search query and a command; an image feature conversion process for converting the search target image into an image feature; an image recognition process for outputting image-accompanying text information related to the search target image; a text processing step of outputting a summarized text corresponding to the search query from the image-accompanying text information based on the instruction sentence; a text feature quantity conversion process for converting the output summarized text into summarized text feature quantities; an image search process for searching for images using the summarized text, the summarized text feature amount, the search target image, and the image feature amount to obtain search results; An image recognition assistance program that executes the above.

Citation Information

Patent Citations

  • Feature information creation device, method and program

    JP2012003357A