Program, information processing apparatus, and information processing method

Abstract

To obtain a higher accurate processing result with a smaller processing amount even when a plurality of objects are included in an image.SOLUTION: A program according to an embodiment of the present invention makes a computer function as a detection unit, a cutting unit, an acquiring unit, and a VQA processing unit. The detection unit detects, from an image, at least one piece of target information including a target region with a detection target and target identification information for use in discrimination of the detection target. The cutting unit cuts, from the image, at least one of the target regions to generate at least one target image. The acquiring unit acquires at least one question responding to the target identification information. The VQA processing unit carries out VQA (Visual Question Answering) processing for the at least one target image with the at least one question.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] An embodiment of the present invention relates to a program, an information processing device, and an information processing method. [Background technology]

[0002] AI (Artificial Intelligence) technology for Visual Question Answering (VQA) has been known for some time. VQA is a task that returns a correct answer when an image and a question related to the image are given. This task is a cross-modal task that requires the correct handling of both image knowledge and text knowledge. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] GLIP BLIP,[online],[Retrieved June 15, 2023],Internet〈URL:https: / / huggingface.co / spaces / Pinwheel / GLIP-BLIP-Object-Detection-VQA> [Non-Patent Document 2] Faster R-CNN,Girshick Ross,Proceedings of the IEEE international conference on computer vision. 2015 [Non-Patent Document 3] Visual Question Answering, [online], [Retrieved June 15, 2023], Internet <URL: https: / / visualqa.org / > [Non-Patent Document 4] MSCOCO,[online],[Retrieved June 15, 2023],Internet〈URL:https: / / tech.amikelive.com / node-718 / what-object-categories-labels-are-in-coco-dataset / > [Non-Patent Document 5] Anderson, Peter, et al. “Bottom-up and top-down attention for image captioning and visual question answering.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2018 [Non-Patent Document 6] Gu,Xiuye,et al. “Open-vocabulary object detection via vision and language knowledge distillation.” arXiv preprint arXiv:2104.13921 (2021) Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional techniques, when an image contains multiple objects, it is difficult to obtain highly accurate processing results with a smaller amount of processing. [Means for solving the problem]

[0005] A program according to an embodiment causes a computer to function as a detection unit, a cutout unit, an acquisition unit, and a VQA processing unit. The detection unit detects at least one object information including an object region including a detection object and object identification information for identifying the detection object from an image. The cutout unit generates at least one object image by cutting out at least one object region from the image. The acquisition unit acquires at least one question according to the object identification information. The VQA processing unit executes a VQA (Visual Question Answering) process using the at least one question for each of the at least one object image. [Brief description of the drawings]

[0006] [Figure 1] Schematic diagram showing an example of VQA. [Diagram 2] FIG. 1 shows an example of an image analysis system using VQA. [Diagram 3] FIG. 1 is a diagram for explaining example 1 of a VQA task. [Figure 4] A diagram to explain example 2 of a VQA task. [Diagram 5] FIG. 2 is a diagram showing an example of the functional configuration of the information processing apparatus according to the first embodiment. [Figure 6] FIG. 4 is a schematic diagram showing a processing example of a VQA processing unit in the first embodiment. [Figure 7] FIG. 4 is a diagram showing an example of a format of a question ID for identifying a question in the first embodiment. [Figure 8] FIG. 2 is a diagram showing an overview of VQA processing according to the first embodiment. [Figure 9] FIG. 4 is a diagram showing an example of display information according to the first embodiment. [Figure 10] 4 is a flowchart showing an example of an information processing method according to the first embodiment. [Figure 11] FIG. 4 is a diagram for explaining Example 1 of effects according to the first embodiment. [Figure 12] FIG. 11 is a diagram for explaining Example 2 of the effect according to the first embodiment. [Figure 13] FIG. 13 is a diagram showing an example of the functional configuration of an information processing apparatus according to a second embodiment. [Figure 14]FIG. 11 is a schematic diagram showing an example of a voting process according to the second embodiment. [Figure 15] FIG. 11 is a graph showing the relationship between the number of votes and recognition accuracy in the second embodiment. [Figure 16] 10 is a flowchart showing an example of an information processing method according to a second embodiment. [Figure 17] 10 is a flowchart showing an example of a voting process in step S18 in the second embodiment. [Figure 18] 10 is a flowchart showing an example of a determination process in step S18 in the second embodiment. [Figure 19] FIG. 13 is a diagram showing an example of the functional configuration of an information processing apparatus according to a third embodiment. [Figure 20] 13A to 13C are views for explaining a processing example of a deformation unit according to the third embodiment. [Figure 21] FIG. 13 is a diagram showing an example of the functional configuration of an information processing apparatus according to a fourth embodiment. [Figure 22] FIG. 1 is a diagram showing an example of a hardware configuration of an information processing apparatus according to first to fifth embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, embodiments of a program, an information processing device, and an information processing method will be described in detail with reference to the accompanying drawings.

[0008] (First embodiment) First, an overview of VQA will be provided.

[0009] Figure 1 is a schematic diagram showing an example of VQA. VQA is an AI that responds to any question by determining the content from an image. The greatest feature of VQA is that the question is given in free-form natural language text. This gives it a high level of versatility, so that it can theoretically handle anything that can be expressed in text.

[0010] Figure 2 is a diagram showing an example of an image analysis system using VQA. By applying a list of questions prepared in advance to the image to be analyzed, a system capable of analyzing any image can be constructed. For example, as shown in Figure 2, a system for danger detection (automated system for calling attention) and a system for analyzing television images (automated system for extracting arbitrary scenes) can be constructed.

[0011] Figure 3 is a diagram for explaining VQA problem example 1. One of the major problems with VQA is that, as shown in Figure 3, when multiple objects (buses in the example of Figure 3) are captured in an image, it is not possible to identify which object the question is directed to.

[0012] For example, if the question is "What color is the bus?", it is unclear which bus to answer because three buses are captured in the left image of Fig. 3 and two buses are captured in the right image of Fig. 3. For example, questions such as "What color is the left bus?", "What color is the middle bus?", and "What color is the right bus?" can be answered in the left image of Fig. 3, but because there are only two buses in the right image of Fig. 3, the answer is still unclear.

[0013] In other words, if individual images could be viewed in advance, appropriate questions could be asked that identified the target object; however, in a system that targets arbitrary images, it is not possible to view images in advance, and therefore it is not possible to ask appropriate questions.

[0014] Figure 4 is a diagram to explain VQA problem example 2. Another problem with VQA is that, as shown in Figure 4, when there are multiple analysis targets, such as people, dogs, trains, buses, tables, and airplanes, in order to cover all of them, a question list containing 23 questions about all targets must be applied to all images.

[0015] For example, in the example in Figure 4, the leftmost image of the five images does not show a train, a bus, a table, or an airplane. However, questions about trains, buses, tables, and airplanes must also be processed using the leftmost image, which increases the amount of processing unnecessarily.

[0016] That is, as the number of images to be analyzed increases, the number of questions also increases, and questions unrelated to the images to be processed are also processed, resulting in an increased amount of processing.

[0017] [Example of functional configuration] 5 is a diagram illustrating an example of a functional configuration of the information processing device 1 according to the first embodiment. The information processing device 1 according to the first embodiment includes a detection unit 11, a clipping unit 12, an acquisition unit 13, a storage unit 14, a VQA processing unit 15, and a display control unit 16.

[0018] The detection unit 11 detects at least one piece of target information including a target area including a detection target (e.g., an object, a person, etc.) and a target ID (an example of target identification information) that identifies the detection target from the image. The target ID includes a label indicating the type of the detection target (an example of a target type) and information that uniquely identifies the detection target (e.g., a number, etc.). For example, when the detected target is a person, the target is identified by assigning a unique target ID to each target, such as "object (person) 1" and "object (person) 2."

[0019] For the detection process, for example, Non-Patent Document 2 is used. In normal target detection, the detection process is performed for all learned labels, but in the first embodiment, the detection process is performed for labels related to a preset question.

[0020] When the detection engine used in the detection unit 11 finds the detection target, it returns the coordinates and label of the box containing the detection target. The box coordinate format is, for example, coordinates that specify a rectangular area (x and y coordinates of the upper left and x and y coordinates of the lower right).

[0021] The cutout unit 12 cuts out at least one object region from the image to generate at least one object image. For example, the cutout unit 12 cuts out a rectangular region including each detected object as the object region.

[0022] The acquiring unit 13 acquires, from the storage unit 14, at least one question corresponding to a label indicating the type of the detection target.

[0023] The storage unit 14 stores a question list (at least one question) for each label indicating the type of detection target.

[0024] The VQA processing unit 15 executes VQA processing using at least one question for at least one target image.

[0025] The VQA processing unit 15 uses an AI that judges the image content and answers any question, as shown in FIG. 1 above. There are various possible configurations for the AI, but a representative configuration is as shown in the lower part of FIG. 1. The image is converted into a feature vector by an image encoder. Similarly, the question (text) is converted into a feature vector by a text encoder. The answer is obtained by a decoder that combines both feature vectors to generate the answer.

[0026] The VQA AI model is trained using, for example, a VQA dataset (Non-Patent Document 3).

[0027] The VQA processing unit 15 inputs the selected image and question to the AI ​​model of VQA and obtains an answer from the AI ​​model. Note that the VQA processing unit 15 may obtain the predicted probability (confidence) of the answer from the AI ​​model at the same time as the answer, and output the predicted probability.

[0028] For example, labels based on Microsoft Common Objects in Context (MSCOCO) are used as labels associated with questions. MSCOCO is the most standard benchmark dataset for object detection. It uses 80 labels related to common objects, such as people, cars, dogs, cats, trains, buses, and tables (see, for example, Non-Patent Document 4).

[0029] In this embodiment, a label called "image" may be prepared as a special label. This label represents the entire image, and the "image" label is used for questions about the entire scene.

[0030] Also, it is not necessary to use all the labels provided by MSCOCO, but only the labels necessary for the question (e.g., people and cars, etc.).

[0031] Fig. 6 is a schematic diagram showing an example of processing by the VQA processing unit 15 of the first embodiment. In consideration of the problems in Figs. 3 and 4 described above, the VQA processing unit 15 applies VQA to each object detected from an image, as shown in Fig. 6. By applying VQA to each object, the problem in Fig. 3 described above can be solved.

[0032] Furthermore, questions for images should differ depending on the type (label) of the detected object. For example, a person may be asked about the color of clothes they are wearing, but this question is not appropriate for a car. Conversely, a car may be asked about the number of doors it has, but this is not appropriate for a person. Therefore, as shown in FIG. 6, a list of questions is prepared for each label (cars, people, and dogs in the example of FIG. 6), and the VQA processing unit 15 applies appropriate questions to each label of the detected object.

[0033] Specifically, in the example of Fig. 6, multiple questions appropriate for each label that identifies a car, a person, and a dog are applied. By preparing a different question list for each label that identifies an object, the problem in Fig. 4 described above can be solved.

[0034] The questions are managed by a question ID (an example of question identification information) that uniquely identifies the question. The acquisition unit 13 assigns the question identification information to at least one question that is applied to each target image generated by the cropping unit 12.

[0035] Fig. 7 is a diagram showing an example of a format of a question ID for identifying a question in the first embodiment. Since multiple objects with the same label may be detected from an image, it is necessary to be able to uniquely identify the object and the question and answer for that object. The example in Fig. 7 is an example of a format for managing questions and answers for each detected object.

[0036] FIG. 7 shows an example in which there are two objects ("Object (Person) 1" and "Object (Person) 2") and "Person-Question 1" and "Person-Question 2" regarding labels that identify the objects.

[0037] In the example of FIG. 7, "Person-Question 1" and "Person-Question 2" are IDs that identify questions about people. "Person-Question 1" indicates that it is the first question to a person. "Person-Question 2" indicates that it is the second question to a person. In this case, there are a total of four combinations, each of which is assigned a unique question ID, "ID1" to "ID4."

[0038] By linking the VQA answer with these question IDs, it becomes possible to uniquely associate the answer, the target, and the question (see FIG. 8). The answer, the target, and the question may be output in a data format such as a file as VQA processing result information. That is, the VQA processing unit 15 may output VQA processing result information in which a target ID (an example of target identification information), a question ID (an example of question identification information), and the answer to the question identified by the question ID are associated with each other.

[0039] The question ID for identifying a question may be any word such as "gender" or "color" instead of a number, as long as it is uniquely identifiable information. There is no limit to the number of labels for identifying detection targets or the number of questions.

[0040] Returning to FIG. 5, the display control unit 16 displays, on the display device, display information in which the VQA processing result information is superimposed on the image, thereby visualizing the VQA processing result information in an easily understandable manner.

[0041] Fig. 9 is a diagram showing an example of display information in the first embodiment. Fig. 9 shows an example in which one car, three people, and one dog are detected. In the example of Fig. 9, the detected objects are displayed as boxes, and the object ID, the question ID, and the answer to the question are arranged above the boxes, thereby visualizing the VQA processing result information in an easily understandable manner. That is, the display control unit 16 displays, on the display device, display information in which the VQA processing result information including the object ID, the question ID, and the answer to the question is assigned to the detected object identified by the object ID included in the VQA processing result information.

[0042] [Examples of information processing methods] 10 is a flowchart showing an example of the information processing method according to the first embodiment. First, the detection unit 11 detects at least one object from an image (step S1), and obtains object information including information specifying an object region (for example, the above-mentioned box coordinates) and a label identifying the object (step S2).

[0043] Next, the cutout unit 12 acquires a target image by cutting out a target area from the image based on the target information acquired in step S2 (step S3).

[0044] Next, the acquisition unit 13 acquires a question list (at least one question) corresponding to the label that identifies the target from the storage unit 14 based on the target information acquired in step S2 (step S4).

[0045] Next, the VQA processing unit 15 acquires a question from the question list acquired in step S4 (step S5). Next, the VQA processing unit 15 executes VQA processing using the target image acquired in step S3 and the question acquired in step S5 (step S6).

[0046] Next, the VQA processor 15 determines whether or not the entire question list has been processed (step S7). If the entire question list has not been processed (step S7, No), the process returns to step S5.

[0047] If the entire question list has been processed (step S7, Yes), the detection unit 11 determines whether or not all the objects detected in step S1 have been processed (step S8). If all the objects detected in step S1 have not been processed (step S8, No), the process returns to step S2. If all the objects detected in step S1 have been processed (step S8, Yes), the process ends.

[0048] As described above, in the information processing device 1 of the first embodiment, the detection unit 11 detects at least one piece of target information including a target region including a detection target and target identification information for identifying the detection target from an image. The cutout unit 12 generates at least one target image by cutting out at least one target region from the image. The acquisition unit 13 acquires at least one question according to the target identification information. The VQA processing unit 15 executes VQA processing using at least one question for each at least one target image.

[0049] As a result, according to the first embodiment, even when an image contains multiple objects, it is possible to obtain more accurate processing results with a smaller amount of processing.

[0050] Fig. 11 is a diagram for explaining Example 1 of the effect of the first embodiment. According to the first embodiment, questions can be applied to each target, and answers to the questions can be obtained. In the example of Fig. 11, a list of questions about buses is applied to each of three buses.

[0051] Fig. 12 is a diagram for explaining Example 2 of the effect according to the first embodiment. As shown in Fig. 12, the amount of VQA processing can be reduced by applying a question list according to the target. In the example of Fig. 12, VQA only needs to be applied 2+2+2+2+3+2=13 times for five images.

[0052] Although the features of detected objects were also utilized in conventional VQA (Non-Patent Document 5), the features of all objects in an image corresponded one-to-one to the questions. Therefore, in the example of Fig. 4 above, since there are 23 questions for six types of objects, if these were applied to the same five images as in Fig. 12, the VQA process had to be performed 23*5=115 times.

[0053] Second embodiment Next, a second embodiment will be described. In the description of the second embodiment, the same description as in the first embodiment will be omitted, and only the differences from the first embodiment will be described. In the second embodiment, a case in which voting behavior is incorporated in order to stabilize answers to questions about videos will be described.

[0054] [Example of functional configuration] 13 is a diagram showing an example of the functional configuration of an information processing device 1-2 according to the second embodiment. The information processing device 1-2 according to the second embodiment includes a detection unit 11, a clipping unit 12, an acquisition unit 13, a storage unit 14, a VQA processing unit 15, a display control unit 16, and a voting unit 17. In the second embodiment, the voting unit 17 is added to the configuration of the first embodiment.

[0055] The detection unit 11 detects at least one of the above-mentioned target information from a frame included in a moving image.

[0056] The voting unit 17 votes for the answers to the questions obtained by performing VQA processing on the target images generated for each frame included in the video, and determines the answers to the questions for each target image based on the voting results.

[0057] Fig. 14 is a schematic diagram showing an example of the voting process of the second embodiment. When the detection unit 11 finds the target, it returns the coordinates of the box containing the target and the target ID that identifies the target being tracked. In the example of Fig. 14, the box coordinate format is a rectangle (top left x coordinate, top left y coordinate, bottom right x coordinate, bottom right y coordinate).

[0058] The voting unit 17 prepares a voting box (answer bin) for each target ID and realizes statistical VQA.

[0059] The cutout unit 12 cuts out a target region specified by the box coordinates, and inputs the cutout to the VQA processing unit 15. In addition, the acquisition unit 13 acquires a question list corresponding to the label of the target region from the storage unit 14, and inputs the question list to the VQA processing unit 15.

[0060] The VQA processing unit 15 returns the most likely answer for the target image and the predicted probability of that answer.

[0061] The voting unit 17 performs a voting action of adding a predicted probability to the answer bin in the voting box. The more frames in which the same answer is predicted, the larger the bin of that answer will be. Finally, the voting unit 17 performs a determination based on the determination conditions, and if determined, outputs the answer with the most votes as the final answer. If the answer is not determined, a new frame is input to the detection unit 11, and processing is performed again from target detection.

[0062] For tracking of targets (e.g., objects and people), a method is used that uses, for example, Intersection over Union (IoU) to track the target that has the largest overlap with a bounding box (bbox) containing the target between frames.

[0063] Fig. 15 is a diagram showing the relationship between the number of votes and the recognition accuracy in the second embodiment. As shown in Fig. 15, by voting for answers in multiple frames included in a video, it becomes possible to make a statistical judgment. The example in Fig. 15 shows the recognition accuracy in the case where each time an answer for one frame is obtained by VQA, this answer is voted for, and the most voted answer for each frame is set as the final answer result.

[0064] In the case of 2-frame voting, the accuracy drops because the votes are split and the decision-making power decreases, but it can be seen that the accuracy improves significantly with 3 frames or more. Of course, the higher the accuracy of the engine (VQA AI model) alone (Engine Accuracy), the faster the final accuracy converges. For this reason, the voting unit 17 of the second embodiment performs voting using a voting method of 3 frames or more.

[0065] [Examples of information processing methods] 16 is a flowchart showing an example of an information processing method according to the second embodiment. The processes in steps S11 to S17 and S19 are similar to the processes in steps S1 to S8 in the first embodiment, and therefore will not be described.

[0066] In the second embodiment, after the VQA process is performed on the detection target in each frame included in the moving image, a voting process for an answer to the detection target tracked in a plurality of frames is executed (step S18).

[0067] 17 is a flowchart showing an example of the voting process in step S18 of the second embodiment. First, the voting unit 17 extracts one target o from a tracking list including at least one tracking target (step S21).

[0068] Next, the voting unit 17 calculates the IoU between the bbox of the previous frame of the object o and the bbox of the frame to be processed (step S22). Next, the voting unit 17 determines whether the IoU is equal to or greater than a threshold and whether the labels of the two bboxes used in the IoU calculation are the same (step S23).

[0069] If the IoU is greater than or equal to a threshold and the labels of the two bboxes used in the IoU calculation are identical (step S23, Yes), the predicted probability is added to the target voting box (answer bin) identified by that label (step S26).

[0070] If the IoU is less than the threshold or the labels of the two bboxes used in the IoU calculation are not the same (step S23, No), the voting unit 17 determines whether or not all tracking targets included in the tracking list have been processed (step S24). If all tracking targets included in the tracking list have not been processed (step S24, No), the process returns to step S21.

[0071] When all tracking targets included in the tracking list have been processed (step S24, Yes), the voting unit 17 adds the target included in the bbox of the frame to be processed as a new tracking target to the tracking list (step S25) since the target included in the bbox of the frame to be processed does not exist in the tracking list. Next, the voting unit 17 adds a predicted probability to the voting box (answer bin) of the target included in the bbox of the frame to be processed (step S26).

[0072] 18 is a flowchart showing an example of the determination process in step S18 of the second embodiment. First, the voting unit 17 calculates the total number of votes for the answers to the question (step S31). Next, the voting unit 17 calculates the voting probability by dividing the bin of each answer by the total number of votes calculated in step S31 (step S32). Next, the voting unit 17 identifies the bin with the most votes, and obtains the number of votes for the bin with the most votes and the voting probability of the bin with the most votes (step S33).

[0073] Next, the voting unit 17 judges whether the number of votes obtained in step S33 is equal to or greater than a threshold (step S34). If the number of votes is not equal to or greater than the threshold (step S34, No), the voting unit 17 leaves the answer unconfirmed and ends the process.

[0074] If the number of votes is equal to or greater than the threshold (step S34, Yes), the voting unit 17 determines whether the voting probability obtained in step S33 is equal to or greater than the threshold (step S35). If the voting probability is not equal to or greater than the threshold (step S35, No), the voting unit 17 leaves the answer unconfirmed and ends the process.

[0075] If the voting probability is equal to or greater than the threshold (step S35, Yes), the voting unit 17 confirms the answer and ends the process. In other words, if a certain degree of confidence is obtained (the number of votes is equal to or greater than the threshold and the voting probability is equal to or greater than the threshold), the process is interrupted midway and the answer is determined.

[0076] According to the flowchart in FIG. 18, when a video (camera footage) is input, it is possible to determine what the answer to a question is.

[0077] Third embodiment Next, a third embodiment will be described. In the description of the third embodiment, the same description as in the first embodiment will be omitted, and only the differences from the first embodiment will be described. In the third embodiment, a case where a target image is deformed in order to improve the accuracy of answering a question will be described.

[0078] [Example of functional configuration] 19 is a diagram showing an example of the functional configuration of an information processing device 1 according to a third embodiment. An information processing device 1-3 according to the third embodiment includes a detection unit 11, a clipping unit 12, an acquisition unit 13, a storage unit 14, a VQA processing unit 15, a display control unit 16, and a deformation unit 18. In the third embodiment, the deformation unit 18 is added to the configuration of the first embodiment.

[0079] The deformation unit 18 deforms the object region (for example, a box) detected by the detection unit 11 according to at least one of the label that identifies the object and the question.

[0080] For example, the transformation unit 18 merges a box of a nearby person with a box of a dog to transform it into a new box. In addition, for example, when the transformation unit 18 asks a person a question such as "Are you standing on a stepladder?", it is assumed that the stepladder is below the person, so the transformation unit 18 extends the box including the person downward.

[0081] The transformation method is stored, for example, in association with each question in the question list. Specifically, a question such as "What is your gender?" is not accompanied by a transformation method for the box. On the other hand, a question such as "What are you doing?" is accompanied by a transformation method such as enlarging the box by 1.5 times, since it is necessary to check the surroundings to determine the action.

[0082] Fig. 20 is a diagram for explaining a processing example of the transformation unit 18 in the third embodiment. Fig. 20 shows an example of asking the question "What are you doing?" in a scene where a dog is being walked. If the target area is cut out individually for each target, as shown in the upper part of Fig. 20, only the person will be cut out, and it is expected that the answer given will be "walking" rather than "walking the dog." On the other hand, if the transformation unit 18 transforms the target area by merging a person box and a dog box that are close to each other, as shown in the lower part of Figure 20, and the cut-out unit 12 cuts them out as a new box, the answer obtained is "walking the dog."

[0083] (Fourth embodiment) Next, a fourth embodiment will be described. In the description of the fourth embodiment, the same description as in the first embodiment will be omitted, and only the differences from the first embodiment will be described. In the fourth embodiment, a case where target detection is performed using an open vocabulary will be described.

[0084] Ordinary object detection can only detect objects of a type that corresponds to the label that was determined during learning. For example, if you want to ask a question about sports cars, you need to use the label "car," and cars other than sports cars will also be detected as targets. In response to such issues, open vocabulary object detection has been proposed in recent years (Non-Patent Document 6). This is a type of object detection that allows the user to specify the label at the time of detection, and is capable of detecting sports cars.

[0085] [Example of functional configuration] 21 is a diagram illustrating an example of a functional configuration of an information processing device 1 according to the fourth embodiment. An information processing device 1-4 according to the fourth embodiment includes a detection unit 11-2, a clipping unit 12, an acquisition unit 13, a storage unit 14-2, a VQA processing unit 15, a display control unit 16, and a deformation unit 18.

[0086] The detection unit 11-2 performs object detection using an open vocabulary from an image. In object detection using an open vocabulary, the label of the object to be detected must be given at the time of inference. Therefore, as shown in Fig. 21, the detection unit 11-2 reads the label specified together with the question from the storage unit 14-2 and sets it as the input label for object detection using the open vocabulary.

[0087] That is, the detection unit 11-2 detects, from the image, at least one object area including a detection object of the input label (an example of an object type) read out from the storage unit 14-2, and object information including an object ID (an example of object identification information).

[0088] Fifth embodiment Next, a fifth embodiment will be described. In the description of the fifth embodiment, the description of the same parts as those in the first embodiment will be omitted, and only the parts different from the first embodiment will be described. The functional configuration of the information processing device 1 of the fifth embodiment is the same as that of the first embodiment.

[0089] The detection unit 11 of the fifth embodiment receives a question from a user and detects an object according to the question. For example, when the detection unit 11 receives the question "What is your gender?", it identifies the label (one example of an object type) of the detection object as "person" and detects the detection object classified into the label from the image. Then, the VQA processing unit 15 applies the question "What is your gender?" to the object image detected by the detection unit 11 and cut out by the cutout unit 12, and outputs the answer.

[0090] Finally, an example of the hardware configuration of the information processing devices 1 (1-2 to 1-4) according to the first to fifth embodiments will be described.

[0091] [Example of hardware configuration] 22 is a diagram showing an example of a hardware configuration of an information processing device 1 (1-2 to 1-4) according to the first to fifth embodiments. The information processing device 1 includes a processor 201, a main storage device 202, an auxiliary storage device 203, a display device 204, an input device 205, and a communication device 206. The processor 201, the main storage device 202, the auxiliary storage device 203, the display device 204, the input device 205, and the communication device 206 are connected via a bus 210.

[0092] Note that the information processing device 1 may not include some of the above configurations. For example, if the information processing device 1 can use the input function and display function of an external device, the information processing device 1 may not include the display device 204 and the input device 205.

[0093] The processor 201 executes a program read from the auxiliary storage device 203 to the main storage device 202. The main storage device 202 is a memory such as a read only memory (ROM) and a random access memory (RAM). The auxiliary storage device 203 is a hard disk drive (HDD), a memory card, or the like.

[0094] The display device 204 is, for example, a liquid crystal display. The input device 205 is an interface for operating the information processing device 1. The display device 204 and the input device 205 may be realized by a touch panel or the like having a display function and an input function. The communication device 206 is an interface for communicating with other devices.

[0095] For example, the program executed by the information processing device 1 is provided as a computer program product in the form of a file in an installable or executable format, recorded on a computer-readable storage medium such as a memory card, a hard disk, CD-RW, CD-ROM, CD-R, DVD-RAM, or DVD-R.

[0096] Also, for example, the program executed by the information processing device 1 may be configured to be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network.

[0097] Also, for example, the program executed by the information processing device 1 may be provided via a network such as the Internet without being downloaded. Specifically, for example, the program may be configured as an ASP (Application Service Provider) type cloud service.

[0098] Also, for example, the program for the information processing device 1 may be provided by being pre-installed in a ROM or the like.

[0099] The program executed by the information processing device 1 has a modular configuration including functions that can be realized by the program among the above-mentioned functional configurations. As for each function, as actual hardware, the processor 201 reads the program from a storage medium and executes it, and the above-mentioned functional blocks are loaded onto the main storage device 202. In other words, the above-mentioned functional blocks are generated on the main storage device 202.

[0100] Note that some or all of the above-described functions may be realized by hardware such as an integrated circuit (IC) instead of by software.

[0101] Furthermore, each function may be realized using a plurality of processors 201. In this case, each processor 201 may realize one of the functions, or may realize two or more of the functions.

[0102] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0103] 1. Information processing device 11 Detection section 12 Cut-out section 13 Acquisition Department 14 Storage section 15 VQA Processing Section 16 Display control section 17 Voting Department 18 Deformation section 201 Processor 202 Main storage 203 Auxiliary storage device 204 Display device 205 Input Device 206 Communication Equipment 210 Bus

Claims

1. Computer, A detection unit that detects at least one piece of object information including an object region including a detection object and object identification information that identifies the detection object from an image; a cropping unit for cropping at least one of the target regions from the image to generate at least one target image; an acquisition unit that acquires at least one question corresponding to the target identification information; a VQA processing unit that executes a VQA (Visual Question Answering) process using the at least one question for each of the at least one target image; A program to function as a

2. The computer includes: A storage unit is further provided for storing at least one question for each target type indicating a type of the detection target, The object identification information includes information indicating the object type, The acquisition unit acquires at least one question corresponding to the target identification included in the target identification information from the storage unit. The program according to claim 1.

3. The detection unit detects, from the image, at least one object area including a detection object of the object type read from the storage unit and object information including the object identification information. The program according to claim 2.

4. The detection unit receives a question regarding the image from a user, identifies the target type from the question, and detects at least one target area including a detection target of the identified target type and target information including the target identification information. The program according to claim 2 or 3.

5. The computer, a deformation unit that deforms the target area in accordance with at least one of the target type and the question; The cutout unit generates at least one target image by cutting out at least one of the target regions or at least one of the target regions deformed by the deformation unit from the image. The program according to claim 2 or 3.

6. the acquiring unit assigns question identification information to at least one question applied to each of the target images; the VQA processing unit outputs VQA processing result information in which the target identification information, the question identification information, and an answer to the question identified by the question identification information are associated with each other. The program according to any one of claims 1 to 3.

7. The computer, a display control unit that displays, on a display device, display information assigned to a detection target identified by the target identification information included in the VQA processing result information; The program according to claim 6, further functioning as:

8. the image is a frame included in a video, The detection unit detects at least one of the target information from the frame, The computer, a voting unit that votes for an answer to a question obtained by performing a VQA process on the target image generated for each frame, and determines an answer to the question for each target image based on a result of the voting; The program according to claim 1 , further comprising:

9. A detection unit that detects at least one piece of object information including an object region including a detection object and object identification information that identifies the detection object from an image; a cropping unit for cropping at least one of the target regions from the image to generate at least one target image; an acquisition unit that acquires at least one question corresponding to the target identification information; a VQA processing unit that executes a VQA (Visual Question Answering) process using the at least one question for each of the at least one target image; An information processing device comprising:

10. A step in which an information processing device detects, from an image, at least one piece of object information including an object region including a detection object and object identification information that identifies the detection object; generating at least one target image by cutting out at least one target region from the image; The information processing device acquires at least one question corresponding to the target identification information; The information processing device executes a visual question answering (VQA) process using the at least one question for each of the at least one target image; An information processing method comprising:

Images