Image recognition support device, image recognition support method, and image recognition support program

Abstract

To provide a technique which allows for multi-label inference with high accuracy.SOLUTION: An image recognition support device includes: a multi-label inference unit which uses a multi-label inference model to output a first label candidate group and respective confidence levels of the first label candidate group with respect to an input image; a multi-label editing unit which uses a multi-label editing model to output a second label candidate group by adding or deleting labels on the basis of the first label candidate group and the respective confidence levels of the first label candidate group; and a multi-level shaping unit which uses a multi-label shaping model and uses label relevant information indicating an inputted detection object to convert the second label candidate group and outputs a third label candidate to be given to the input image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image recognition support device, an image recognition support method, and an image recognition support program that support the recognition of attributes of each subject in an image. [Background technology]

[0002] One type of technology for quickly understanding the situation in a captured image is technology that automatically identifies the attributes (hereinafter referred to as "labels") of each subject in the image. By using this technology, for example, it becomes possible to quickly understand the disaster situation over a wide area by automatically recognizing the damage-related attributes of each subject in a wide-area image taken from the air of a disaster site.

[0003] One attribute recognition technology used in this case is a multi-label recognition model that estimates and outputs multiple attributes of a subject. For example, the abstract of Patent Document 1 states, "A method for performing multi-label classification includes: extracting a feature vector from an input vector containing input data using a feature extractor; determining a related vector containing related labels having relevance scores based on the feature vector using a label predictor; updating a binary masking vector by masking pre-selected labels selected in a previous label selection; applying the updated binary masking vector to the related vector so that the related label vector is updated to exclude the pre-selected labels from the related labels; and selecting related labels from the updated related label vector based on the relevance scores of the updated related label vector." FIG. 1 discloses a configuration in which multiple binary classifiers are provided before the feature extractor. Thus, Patent Document 1 describes a technology for efficiently identifying multiple attributes by using multiple binary classifiers.

[0004] However, the technology in Patent Document 1 only recognizes pre-trained attributes (labels for which a binary classifier has already been prepared) and cannot recognize untrained attributes. On the other hand, in recent years, a multi-label recognition model called the Recognize Anything Model (https: / / arxiv.org / abs / 2306.03514) has become popular, which enables the recognition of a variety of attributes by training a large amount of image data with text information and using a large-scale extracted vocabulary as attribute candidates.

[0005] Furthermore, as described in Non-Patent Documents 1 and 2, a technique is also known in which text information such as attributes of the subject in an image is input into a language model such as GPT (registered trademark)-3 along with instructional statements to automatically create explanatory text that describes the situation in an image. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Special Publication No. 2019-527440 [Non-patent literature]

[0007] [Non-Patent Document 1] Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language, Arxiv, 2022. [Non-patent document 2] Visual Clues: Bridging Vision and Language Foundations for Image Paragraph Captioning, NeurIPS, 2022. Summary of the Invention [Problem to be solved by the invention]

[0008] However, since a multi-label recognition model is a model that outputs all attributes for each subject that have a reliability above a certain threshold, using a multi-label recognition model with a large vocabulary database may result in an increase in the output of miscellaneous attributes other than those that are useful for understanding the situation, which may make it more difficult to accurately understand the situation in the image.

[0009] Furthermore, the technologies in Non-Patent Documents 1 and 2 cannot create explanatory text while ignoring miscellaneous attributes other than those useful for understanding the situation, so if a large number of miscellaneous attributes are input, it may not be possible to create a text that adequately describes the situation in the image.

[0010] Therefore, an object of the present invention is to provide an image recognition support device, an image recognition support method, and an image recognition support program that can accurately output attributes that are useful for understanding the situation in an image, even when using a multi-label recognition model with a large vocabulary database. [Means for solving the problem]

[0011] An image recognition support device comprising a control unit and a storage unit, wherein the storage unit includes: a multi-label inference model used when applying multi-label inference to an input image to output a first label candidate group and respective reliabilities of the first label candidate group; a multi-label editing model used when adding or deleting labels based on the first label candidate group and respective reliabilities of the first label candidate group to output a second label candidate group; and a multi-label shaping model used when converting the second label candidate group using label-related information indicating an input detection target to output a third label candidate to be assigned to the input image. the control unit comprises: a multi-label inference unit that uses the multi-label inference model to output, for an input image, a first set of label candidates and a reliability of each of the first label candidates; a multi-label editing unit that uses the multi-label editing model to add or delete labels based on the first label candidates and the reliability of each of the first label candidates, and outputs a second set of label candidates; and a multi-label shaping unit that uses the multi-label shaping model to convert the second set of label candidates using label-related information that indicates an input detection target, and outputs a third label candidate to be assigned to the input image. [Effects of the Invention]

[0012] According to the image recognition support device, image recognition support method, and image recognition support program of the present invention, it is possible to accurately output attributes that are useful for understanding the situation in an image, even when using a multi-label recognition model with a large vocabulary database.

[0013] Other objects, configurations and effects will become apparent from the following examples. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram illustrating an outline of an image recognition support device according to a first embodiment. [Figure 2] FIG. 2 is a diagram showing an example of an image captured by an imaging device. [Figure 3]FIG. 4 is a diagram showing an example of search area information according to the first embodiment. [Figure 4] FIG. 3 is a diagram showing an example of label information according to the first embodiment. [Figure 5] 3 is a flowchart of a process executed by the image recognition support device according to the first embodiment. [Figure 6] 10 is a flowchart of a process executed by an image recognition support device according to a second embodiment. [Figure 7] FIG. 10 is a diagram illustrating an outline of an image recognition support device according to a third embodiment. [Figure 8] 11 is a flowchart of a process executed by an image recognition support device according to a third embodiment. [Figure 9] FIG. 10 is a diagram illustrating an outline of an image recognition support device according to a fourth embodiment. [Figure 10] FIG. 10 is a diagram showing an example of a dialogue performed by the image recognition support device according to the fourth embodiment. [Figure 11] FIG. 10 is a diagram illustrating an outline of an image recognition support device according to a fifth embodiment. [Figure 12] FIG. 10 is a diagram showing an example of a dialogue performed by the image recognition support device according to the fifth embodiment. [Figure 13] FIG. 10 is a diagram illustrating an outline of an image recognition support device according to a sixth embodiment. [Figure 14] 13 is a flowchart of a process executed by an image recognition support device according to a sixth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that for clarity of explanation, the following description and drawings have been omitted or simplified as appropriate. Furthermore, the present invention is not limited to the following embodiments, and all application examples that conform to the concept of the present invention are included in the technical scope of the present invention. Furthermore, unless otherwise specified, the number of each component referred to may be singular or plural. [Example]

[0016] First, an image recognition support device 1 according to a first embodiment will be described with reference to FIGS.

[0017] 1 is a diagram illustrating the configuration of an image recognition support device 1 according to this embodiment and an overview of its peripheral devices. As illustrated here, an imaging device 2, a text input device 3, and a display device 4 are connected to the image recognition support device 1. Note that although the imaging device 2, the text input device 3, and the display device 4 are directly connected to the image recognition support device 1 here, they do not necessarily have to be directly connected, and a configuration may be adopted in which information exchange with the image recognition support device 1 is achieved via a wireless communication network or a storage medium. Below, an overview of the imaging device 2, the text input device 3, and the display device 4 will be provided, followed by a detailed description of the image recognition support device 1.

[0018] <Imaging device 2> The imaging device 2 is a camera that captures a wide area and outputs an image, and may be not only a normal camera that captures in the visible light range and outputs a color image or a monochrome image, but also an infrared camera that captures in the infrared range and outputs an IR image (IR: Infrared). Note that the imaging device 2 in this embodiment also includes a device that creates CG images (CG: Computer Graphics) based on the various images described above.

[0019] 2 is an example of a disaster site image captured by the imaging device 2. As shown in the figure, this disaster site image records not only river flooding due to heavy rain and fallen trees due to strong winds, but also the resulting damages such as flooded, collapsed, and fire-damaged houses, as well as people evacuated from houses where fires broke out. Therefore, it is expected that the attributes of flooded, fire, and evacuees will be identified for house H1 in the image, the attributes of flooded and collapsed will be identified for house H2, and the attribute of proximity to a burned house will be identified for house H3.

[0020] <Text Input Device 3> The text input device 3 is a keyboard, a mouse, a touch panel, a microphone, or the like that is used by the user to input text information.

[0021] <Display device 4> The display device 4 is a display that displays the output of the image recognition support device 1. The information displayed here may be, for example, an image in which attribute text information is superimposed on each house in the image of Fig. 2, or equivalent information displayed in a chat format.

[0022] <Image Recognition Support Device 1> The image recognition assistance device 1 is a device that assists a user in quickly understanding the situation in an image when an image such as that shown in Figure 2 is input, and is specifically a computer equipped with a control unit 11 and a memory unit 12.

[0023] The control unit 11 is a functional unit that performs image recognition processing, text processing, etc., and includes functional units such as an image input unit 11a, a text input unit 11b, an image recognition unit 11c, a multi-label inference unit 11d, a multi-label editing unit 11e, a multi-label shaping unit 11f, and a display unit 11g. Specifically, the control unit 11 is an arithmetic device such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), a GPU (Graphics Processing Unit), or an FPGA (Field-Programmable Gate Array).

[0024] The storage unit 12 is a functional unit that stores information and stores various data such as a program 12a, an image database 12b, a label information database 12c, an image recognition model 12d, a multi-label inference model 12e, a multi-label editing model 12f, and a multi-label shaping model 12g. Each of the above models (and each model appearing in the following examples) is a trained model used by the control unit 11 and is configured using a neural network such as a convolutional neural network (CNN) or a transformer. Specifically, the storage unit 12 is a storage medium such as a read-only memory (ROM), a random access memory (RAM), a hard disk drive (HDD), or a solid state drive (SSD).

[0025] Below, we will explain in detail each functional part of the control unit 11, which is realized when the arithmetic device that is the entity of the control unit 11 executes the program 12a loaded from the memory unit 12, and the details of the various data in the memory unit 12.

[0026] <<Image input unit 11a>> The image input unit 11a is a functional unit that transmits an image received from the imaging device 2 to the image recognition unit 11c, the image database 12b, and the like.

[0027] <<Text input section 11b>> The text input unit 11b is a functional unit that transmits text information received from the text input device 3 to the multi-label shaping unit 11f, the image database 12b, the label information database 12c, etc., depending on the situation and content.

[0028] <<Image recognition unit 11c>> The image recognition unit 11c is a functional unit that processes the image received from the image input unit 11a using the image recognition model 12d to recognize objects, etc. in the image, and then outputs information about the recognized objects, etc. to the multi-label inference unit 11d, etc. The image recognition model 12d is, for example, a model such as the Grounding DINO or YOLO series that detects various objects in an image.

[0029] <<Multi-label inference unit 11d>> The multi-label inference unit 11d is a functional unit that processes an image of an object or the like detected by the image recognition unit 11c using the multi-label inference model 12e to infer multiple attributes related to the object or the like, and then outputs the inferred attribute group to the multi-label editing unit 11e, etc. The multi-label inference model 12e is, for example, a Recognize Anything Model that includes a large-scale vocabulary database to list various attributes of a subject image.

[0030] <<Multilabel Editorial Department 11e>> The multi-label editing unit 11e is a functional unit that processes the attribute group received from the multi-label inference unit 11d using the multi-label editing model 12f to add other attributes to the attribute group or delete some attributes from the attribute group, and then outputs the processed attribute group to the multi-label shaping unit 11f, etc. Here, the multi-label editing unit 11e determines from what perspective attributes are added or deleted based on instructions (prompt information) received from the text input unit 11b. The multi-label editing model 12f is, for example, a language model such as LLaMA that processes text information.

[0031] <<Multi-label forming section 11f>> The multi-label shaping unit 11f is a functional unit that uses the multi-label shaping model 12g to shape the attribute group processed by the multi-label editing unit 11e into a predetermined sentence format and outputs the result to the display unit 11g. Here, the multi-label shaping unit 11f determines from what perspective the attribute group will be shaped into a sentence, taking into account the instruction sentence (prompt information) received from the text input unit 11b. The multi-label shaping model 12g is, for example, a language model such as LLaMA that processes text information, and may use the same language model as the multi-label editing model 12f or a different language model.

[0032] <<Display section 11g>> The display unit 11g is a functional unit that outputs to the display device 4 a video signal or the like that includes the text information shaped by the multi-label shaping unit 11f.

[0033] <<Image Database 12b>> The image database 12b is a database that stores images received from the image input unit 11a and holds information about a search area within each image as search area information I1.

[0034] 3 is a diagram showing an example of search area information I1. As shown in this example, search area information I1 is a table in which area IDs, metadata, image features, and image-related text are registered for each search area within an image. Of these, the area ID is a unique ID assigned to each search area, and the metadata includes, for example, the time and location of the image capture, and coordinate information for the search area within the image. Furthermore, the image-related text includes, for example, captioning text and attribute information for the search area output from image recognition model 12d.

[0035] <<Label Information Database 12c>> The label information database 12c is a database that stores information about the label for each search area in the image as label information I2.

[0036] 4 is a diagram showing an example of label information I2. As shown in this example, label information I2 is a table in which an area ID, multi-label inference label information, multi-label editing label information, multi-label shaping label, and user instruction information can be registered for each search area. Of these, the multi-label inference label information, multi-label editing label information, and multi-label shaping label are label information for each search area output by the multi-label inference unit 11d, the multi-label editing unit 11e, and the multi-label shaping unit 11f, respectively. Furthermore, prompt information is an instruction sentence to be given to a language model such as LLaMA, and is, for example, label information and instruction content using the label information.

[0037] <Processing in this Example> Next, the image recognition support process executed by the image recognition support device 1 of this embodiment will be described with reference to the flowchart of FIG.

[0038] <<Step S1>> First, in step S1, the image recognition unit 11c performs object detection on the image input from the image input unit 11a using the image recognition model 12d, and then outputs position information of a search area including the detected object. For example, if a house H1 is detected from the image of Figure 2, the image recognition unit 11c outputs coordinate information within the image of a rectangular area including the house H1 as position information of the search area corresponding to the house H1. Note that the entire image input from the image input unit 11a may be set as one search area.

[0039] In this step, the image recognition unit 11c assigns a unique area ID to each search area and registers the area ID together with the metadata (coordinate information, etc.) of the search area in the search area information I1 (see FIG. 3). <<Step S2>> Next, in step S2, the multi-label inference unit 11d performs multi-label inference using the multi-label inference model 12e on the search area detected in step S1. As a result, label information is output that lists multiple attributes (first label candidate group) possessed by the image in the search area, along with the reliability of each attribute. For example, if label information is output in a format in which the reliability of each attribute is listed in parentheses following each attribute, label information such as "building (0.88) | damage (0.76) | debris (0.79) | flood (0.90) | house (0.75) | house roof (0.98) | mud (0.79)" is output. Here, the higher the reliability value, the higher the reliability. Label information is not generated for attributes whose reliability is below a predetermined detection threshold (e.g., 0.70).

[0040] In this step, the multi-label inference unit 11d also registers the output label information together with the area ID of the corresponding search area in the multi-label inference label information field of the label information I2 in FIG. 4 (see FIG. 4). <<Step S3>> In step S3, the multi-label editing unit 11e uses the multi-label editing model 12f to edit the label information generated in step S2, such as adding or deleting labels. For example, if you want to know only about houses and flood conditions, you can give the multi-label editing unit 11e an instruction (prompt information) such as "Please extract only attributes related to houses and flood conditions. Please group similar attributes together." The label information exemplified above as the output of step S2 is edited into label information containing only labels related to houses and flood conditions (second label candidate group), such as "flood (0.90) | house (0.75)."

[0041] On the other hand, if the user only wants to know about the house and the flooding situation, by giving the multi-label editing unit 11e an instruction (prompt information) such as "Please provide some attributes similar to flood," the multi-label editing unit 11e will be presented with phrases such as floodwater, inundation, and overflow in addition to flood (0.90) | house (0.75), and it is possible to prevent missed detections due to vocabulary variations even for the phrase inundation that the user wants to know.

[0042] In addition, in this step, the multi-label editing unit 11e registers the instruction sentence (prompt information) used in this step in the prompt information field of the label information I2, and also registers the label information that is the output of this step in the multi-label editing label information field (see FIG. 4).

[0043] <<Step S4>> In step S4, the multi-label shaping unit 11f uses the multi-label shaping model 12g to shape the label information generated in step S3 into label information in a format desired by the user. For example, if the multi-label shaping unit 11f receives an instruction (prompt information) such as "Please display Object and Attribute separately as items. Also, please tell us whether there is a damage situation in an item such as Damage," the label information of the second label candidate group exemplified in step S3, "flood (0.90) | house (0.75)," is shaped into label information of the third label candidate group in the format of "Object: house, Attribute: flood, Damage: Yes."

[0044] At this time, the multi-label shaping unit 11f refers to preset label information defined as "flood (0.80 or more) | house (0.7 or more)" as a criterion for label shaping. This allows the generation of label information in this step only for objects that satisfy the conditions that the reliability of being a "flood" is 0.80 or more and the reliability of being a "house" is 0.7 or more.

[0045] In this step, the multi-label formatting unit 11f registers the instruction sentence (prompt information) used in this step in the prompt information field of the label information I2, and also registers the formatted label information in the multi-label formatted label information field (see FIG. 4). <<Step S5>> In step S5, the display unit 11g generates a video signal in which the label information output from step S4 is superimposed on the search area in the image captured by the imaging device 2, and transmits the video signal to the display device 4. This allows the user viewing the video on the display device 4 to quickly identify only the objects that meet the criteria defined by the preset label information.

[0046] At this time, the display unit 11g may transmit various information held in the label information I2 to the display device 4. This allows the user to check the label contents at each stage of the flowchart in Fig. 5 after the fact, and to check after the fact whether the instruction sentence (prompt information) used is appropriate or not, and the validity of the criteria set in the preset label information, etc.

[0047] <Effects of this Example> As described above, according to the image recognition support device 1 of this embodiment, even when a multi-label recognition model such as the Recognize Anything Model equipped with a large-scale vocabulary database is used, attributes useful for understanding the situation in the image can be accurately output, thereby making it possible to quickly understand the situation in the image. [Example]

[0048] Second Embodiment Next, an image recognition support device 1 according to a second embodiment will be described with reference to Fig. 6. Note that a duplicated description of points common to the first embodiment will be omitted.

[0049] Fig. 6 is a flowchart of the process executed by the image recognition support device 1 of this embodiment. As is obvious when compared with Fig. 5, this flowchart adds step S4a after step S4 of Fig. 5, and also provides a path that returns to step S2 after executing step S4b if the requirements of step S4a are not met.

[0050] In step S4a, the multi-label shaping unit 11f determines whether a detection target that satisfies the criteria defined in the preset label information has been found or whether the detection threshold of the multi-label inference unit 11d has fallen below a certain level. If the criteria are met, the process proceeds to step S5; if the criteria are not met, the process proceeds to step S4b.

[0051] In step S4b, the multi-label inference unit 11d lowers the detection threshold in multi-label inference. For example, if the initial value of the detection threshold is 0.70, it is lowered to 0.60. As a result, in the next step S2, the multi-label inference unit 11d can generate label information including attributes that were not detected in the previous step S2 and have a reliability of, for example, 0.65.

[0052] That is, in this embodiment, when the label information is shaped in step S4, if a detection target that satisfies the conditions defined in the preset label information is not found, the detection threshold in the multi-label inference unit 11d is successively lowered and the processing of steps S2 to S4 is repeated, thereby making it easier to find the detection target.

[0053] For example, if the target to be detected is "forest," the lower limit of the detection threshold to be repeated is 0.5, the initial value of the detection threshold is 0.7, and the initial output by the multi-label inference unit 11d is "flood (0.90) | house (0.75)," lowering the detection threshold by 0.1 and repeating steps S2 to S4 will output label information such as "flood (0.90) | house (0.75) | tower (0.65)." In this case, since the detection threshold is still above the detection threshold and the target has not been found, the processing of steps S2 to S4 is repeated. Next, if "flood (0.90) | house (0.75) | tower (0.65) | forest (0.54)" is output, the target has been found, so the repeated processing is terminated and the process proceeds to step S5. In this way, this embodiment can provide a function for efficiently finding targets that are difficult to find. [Example]

[0054] Next, an image recognition support device 1 according to a third embodiment will be described with reference to Figures 7 and 8. Note that a duplicated description of points common to the above-mentioned embodiments will be omitted.

[0055] 7 is a diagram illustrating the configuration of the image recognition support device 1 according to this embodiment and an overview of its peripheral devices. As is obvious from a comparison with FIG. 1, the image recognition support device 1 of this embodiment is obtained by adding a label information determination unit 11h to the control unit 11.

[0056] The label information determination unit 11h is a functional unit that compares the label information output by the multi-label inference unit 11d with past label information registered in the label information database 12c (label information in the multi-label inference label information column in FIG. 4), and determines that the processes of the multi-label editing unit 11e and the multi-label shaping unit 11f should be omitted if the two pieces of label information match or are similar by a certain threshold or more. Note that various models, such as a model that associates images with languages, such as CLIP (Contrastive Language-Image Pre-Training), or a language model, may be used to determine the similarity between label information pieces.

[0057] Fig. 8 is a flowchart of the process executed by the image recognition support device 1 of this embodiment. As is obvious when compared with Fig. 5, this flowchart adds step S2a after step S2 in Fig. 5, and also provides a path to bypass steps S3 and S4 when the requirements of step S2a are met.

[0058] In step S2a, the label information determination unit 11h determines whether the label information output from the multi-label inference unit 11d is a perfect match with past multi-label inference label information retrieved from the label information in the label information database 12c, or whether it is similar by a certain threshold or more. If either of these conditions is met, the process bypasses steps S3 and S4 and proceeds to step S5. In this way, in this embodiment, omitting some of the processes speeds up the overall process.

[0059] When evaluating the degree of agreement of the label information in step S2a, the label information determination unit 11h can also evaluate the prompt information (see FIG. 4) stored in the label information database 12c. This allows the degree of agreement of the multi-label inference label information to be evaluated by calculating linguistic features for the label information and measuring the similarity between them. Models such as CNN and Transformer can be used to calculate the linguistic features. [Example]

[0060] Next, an image recognition support device 1 according to a fourth embodiment will be described with reference to Figures 9 and 10. Note that a duplicated description of points common to the above-mentioned embodiments will be omitted.

[0061] 9 is a diagram illustrating the configuration of the image recognition support device 1 according to this embodiment and an overview of its peripheral devices. As is obvious from a comparison with FIG. 1, the image recognition support device 1 of this embodiment is obtained by adding a dialogue response processing model 12h to the storage unit 12.

[0062] The dialogue response processing model 12h is a processing model that enables user-friendly dialogue-style question and answering during the inference processing by the multi-label inference unit 11d (step S2), the editing processing by the multi-label editing unit 11e (step S3), and the shaping processing by the multi-label shaping unit 11f (step S4). Note that the dialogue response processing model 12h may be any language model having a dialogue function, and therefore may use the dialogue function of the multi-label editing model 12f or the multi-label shaping model 12g.

[0063] FIG. 10 is an example of a dialogue video using the dialogue response processing model 12h. In this figure, an instruction (prompt information) input by the user is displayed on the left side of the screen, and a response from the image recognition support device 1 using the dialogue response processing model 12h is displayed on the right side of the screen. Here, the user may freely input an instruction (prompt information), or may input an instruction (prompt information) based on the response from the image recognition support device 1. Furthermore, the image recognition support device 1 may respond by considering only the immediately preceding instruction (prompt information), or by considering all preceding instruction (prompt information). Furthermore, if an AI (Assistant) model capable of processing images and other modal information is used as the dialogue response processing model 12h, images and other modal information may be accepted as instructions (prompt information). [Example]

[0064] Next, an image recognition support device 1 according to a fifth embodiment will be described with reference to Figures 11 and 12. Note that a duplicated description of points common to the above-mentioned embodiments will be omitted.

[0065] 11 is a diagram illustrating the configuration of the image recognition support device 1 according to this embodiment and an overview of its peripheral devices. As is obvious from a comparison with FIG. 1, the image recognition support device 1 of this embodiment is obtained by adding a coordinate information processing language model 12i to the storage unit 12.

[0066] The coordinate information processing language model 12i is a language model that enables the positional relationship between detection targets to be described using the coordinate information of each detection target on an image during inference processing by the multi-label inference unit 11d (step S2) and during editing processing by the multi-label editing unit 11e (step S3). If the search area for the detection target is rectangular, the position and size of the area on the image can be specified using the x and y coordinates of the upper left and the lower right of the area. For example, if the x coordinate of the upper left of a rectangular area is 0, the x coordinate of the lower right is 100, the y coordinate of the upper left is 20, and the y coordinate of the lower right is 80, the coordinate information of the area can be defined as [0, 100, 20, 80].

[0067] Furthermore, even if the instruction sentence (prompt information) input by the user includes coordinate information, the image recognition support device 1 of this embodiment can further implement the dialogue response processing model 12h of the fourth embodiment to make it possible to respond appropriately to the instruction sentence (prompt information). When inputting the coordinate information of a desired area into the instruction sentence (prompt information), it is convenient to specify the area on the image using a mouse, which is a type of text input device 3.

[0068] 12 is an example of a video of a dialogue between a user and the image recognition support device 1 when coordinate information is included in an instruction sentence (prompt information). In this example, the image recognition support device 1 responds appropriately to the positional relationship between two houses that the user specified using the coordinate information. [Example]

[0069] Sixth Embodiment Next, an image recognition support device 1 according to a sixth embodiment will be described with reference to Figures 13 and 14. Note that overlapping descriptions of points common to the above-mentioned embodiments will be omitted.

[0070] Fig. 13 is a diagram illustrating the configuration of an image recognition support device 1 according to this embodiment and an overview of its peripheral devices. As is obvious from a comparison with Fig. 1, the image recognition support device 1 of this embodiment has an image search unit 11i added to the control unit 11 and an image search model 12j added to the storage unit 12. Here, the image search model 12j is, for example, a CLIP that is trained by associating images with languages.

[0071] Fig. 14 is a flowchart of the process executed by the image recognition support device 1 of this embodiment. As is obvious when compared with Fig. 5, this flowchart includes step S6 after step S5 of Fig. 5 for extracting past images similar to the input image.

[0072] In step S6, image search unit 11i uses image search model 12j to extract, from the group of past images registered in image database 12b, past images whose image features or metadata (such as shooting time and shooting location) are similar to those of the most recently input image, or past images having label information similar to the label information resulting from the processing of step S4. Then, display unit 11g converts the extracted past images into video signals and displays them on display device 4. This allows the user to easily view past images captured in a similar situation to the input image.

[0073] The present invention is not limited to the above-described embodiments, and can be implemented using any components within the scope of the present invention. The above-described embodiments are merely examples, and the present invention is not limited to these contents as long as the features of the invention are not impaired. Furthermore, although various embodiments have been described above, the present invention is not limited to these contents. Other embodiments conceivable within the technical spirit of the present invention are also included within the scope of the present invention. For example, some of the functions of each device in the embodiments may be provided in another device, or functions provided in another device may be provided in the same device. [Explanation of symbols]

[0074] 1. Image recognition support device 11 Control Unit 11a Image input unit 11b Text input section 11c Image recognition section 11d Multi-label inference part 11e Multilabel Editorial Department 11F Multi-Label Forming Department 11g display 11h Label information judgment section 11i Image Search Section 12 Storage Unit 12a Program 12b Image Database 12c Label Information Database 12d image recognition model 12e Multi-label inference model 12f Multi-label Editing Model 12g Multi-label shaping model 12h dialogue response processing model 12i Coordinate Information Processing Language Model 12j Image Search Model 2. Imaging device 3 Text Input Devices 4 Display device

Claims

1. An image recognition assistance device comprising a control unit and a storage unit, The storage unit comprises: a multi-label inference model used when applying multi-label inference to an input image and outputting a first label candidate group and respective reliabilities of the first label candidate group; a multi-label editing model used to add or delete labels and output a second set of label candidates based on the first set of label candidates and the respective reliabilities of the first set of label candidates; a multi-label shaping model that is used to convert the second label candidate group using label-related information indicating the input detection target and output a third label candidate to be assigned to the input image; The control unit a multi-label inference unit that uses the multi-label inference model to output a first label candidate group and a reliability of each of the first label candidate group for an input image; a multi-label editing unit that uses the multi-label editing model to add or delete labels based on the first label candidate set and the respective reliabilities of the first label candidate set, and outputs a second label candidate set; a multi-label shaping unit that uses the multi-label shaping model and label-related information indicating the input detection target to convert the second label candidate group and output a third label candidate to be assigned to the input image; An image recognition support device comprising:

2. 2. The image recognition support device according to claim 1, When the multi-label shaping unit does not output a detection target attribute that meets a preset criterion, the multi-label inference unit performs multi-label inference by lowering the reliability threshold used to identify a first label candidate.

3. 2. The image recognition support device according to claim 1, An image recognition support device characterized by omitting multi-label editing and multi-label shaping when information previously output in multi-label inference and currently output information are similar to each other by a certain threshold or more, or when the attribute to be detected has already been output in a previous detection.

4. 2. The image recognition support device according to claim 1, An image recognition support device that presents information between detected objects by using coordinate information of the detected objects and a language model that processes the coordinate information in multi-label inference and multi-label editing.

5. 2. The image recognition support device according to claim 1, An image recognition support device that enables question and answer sessions by using a language model capable of interactive response in response to multi-label information output in each process of multi-label inference, multi-label editing, and multi-label shaping.

6. 2. The image recognition support device according to claim 1, This image recognition support device is characterized by having a similar image search function that presents similar images and images taken around the same time for the multi-label information and detection result images output in the processes of multi-label inference, multi-label editing, and multi-label shaping.

7. 2. The image recognition support device according to claim 1, The image recognition support device is characterized in that the control unit further comprises a display unit that displays multi-label information output in each process of multi-label inference, multi-label editing, and multi-label shaping, and the processing process on a display device.

8. a multi-label inference step of applying multi-label inference to an input image and outputting a set of first label candidates and respective confidence levels of the first label candidates; a multi-label editing step of adding or deleting labels based on the first label candidate set and the respective reliabilities of the first label candidate set to output a second label candidate set; a multi-label shaping step of converting the second label candidate group using input label-related information indicating the detection target, and outputting a third label candidate to be assigned to the input image; An image recognition support method comprising:

9. applying multi-label inference to the input image to output a first set of label candidates and respective confidence levels of the first set of label candidates; a multi-label editing process for adding or deleting labels based on the first label candidate set and the respective reliabilities of the first label candidate set, and outputting a second label candidate set; multi-label shaping that converts the second label candidate group using label-related information indicating the input detection object and outputs a third label candidate to be assigned to the input image; An image recognition support program characterized by causing a computer to execute the above.

Images