Program, information processing device, and information processing method

Abstract

To obtain a higher accurate processing result with a smaller processing amount even when a plurality of objects are included in an image.SOLUTION: A program according to an embodiment of the present invention makes a computer function as a detection unit, a cutting unit, an acquiring unit, and a VQA processing unit. The detection unit detects, from an image, at least one piece of target information including a target region with a detection target and target identification information for use in discrimination of the detection target. The cutting unit cuts, from the image, at least one of the target regions to generate at least one target image. The acquiring unit acquires at least one question responding to the target identification information. The VQA processing unit carries out VQA (Visual Question Answering) processing for the at least one target image with the at least one question.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] Embodiments of the present invention relate to a program, an information processing apparatus, and an information processing method.

Background Art

[0002] The AI (Artificial Intelligence) technology of Visual Question Answering (VQA) has been conventionally known. VQA is a task of returning a correct answer when a single image and a question related to the image are given. This task is a cross-modal task that requires correctly handling both knowledge about images and knowledge about text.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Non-Patent Document 2

Non-Patent Document 3

Non-Patent Document 4

[0004] However, with conventional techniques, it was difficult to obtain more accurate processing results with less processing effort when an image contained multiple objects. [Means for solving the problem]

[0005] The program of the embodiment causes the computer to function as a detection unit, an extraction unit, an acquisition unit, and a VQA processing unit. The detection unit detects at least one piece of target information from an image, which includes a target region containing the object to be detected and target identification information that identifies the object to be detected. The extraction unit generates at least one target image by extracting at least one of the target regions from the image. The acquisition unit acquires at least one question corresponding to the target identification information. The VQA processing unit performs VQA (Visual Question Answering) processing for each of the at least one target images using the at least one question. [Brief explanation of the drawing]

[0006] [Figure 1] A schematic diagram showing an example of VQA. [Figure 2] A diagram showing an example of an image analysis system using VQA. [Figure 3] A diagram illustrating example 1 of the VQA challenges. [Figure 4] A diagram illustrating example 2 of the VQA problem. [Figure 5] A diagram showing an example of the functional configuration of the information processing device according to the first embodiment. [Figure 6] A schematic diagram showing an example of the processing of the VQA processing unit in the first embodiment. [Figure 7] A diagram showing an example of the format of a question ID for identifying a question in the first embodiment. [Figure 8] A diagram showing an overview of the VQA processing in the first embodiment. [Figure 9] A diagram showing an example of display information in the first embodiment. [Figure 10] A flowchart illustrating an example of the information processing method of the first embodiment. [Figure 11] A diagram illustrating an example of the effect of the first embodiment. [Figure 12] A diagram illustrating an example of the effect of the first embodiment, Part 2. [Figure 13] A diagram showing an example of the functional configuration of the information processing device according to the second embodiment. [Figure 14]Schematic diagram showing an example of voting processing according to the second embodiment. [Figure 15] Diagram showing the relationship between the number of votes and recognition accuracy according to the second embodiment. [Figure 16] Flowchart showing an example of the information processing method according to the second embodiment. [Figure 17] Flowchart showing an example of the voting processing in step S18 according to the second embodiment. [Figure 18] Flowchart showing an example of the confirmation processing in step S18 according to the second embodiment. [Figure 19] Diagram showing an example of the functional configuration of the information processing apparatus according to the third embodiment. [Figure 20] Diagram for explaining an example of the processing of the modified part according to the third embodiment. [Figure 21] Diagram showing an example of the functional configuration of the information processing apparatus according to the fourth embodiment. [Figure 22] Diagram showing an example of the hardware configuration of the information processing apparatus according to the first to fifth embodiments.

Embodiments for Carrying Out the Invention

[0007] Hereinafter, embodiments of a program, an information processing apparatus, and an information processing method will be described in detail with reference to the accompanying drawings.

[0008] (First Embodiment)<U+ First, an overview of VQA will be described.

[0009] FIG. 1 is a schematic diagram showing an example of VQA. VQA is an AI that determines and answers the content from an image in response to an arbitrary question. The greatest feature of VQA is that the question is given as free-form natural language text. As a result, theoretically, it has a high degree of versatility that can handle all matters that can be expressed in text.

[0010] Figure 2 shows an example of an image analysis system using VQA. By applying a pre-prepared list of questions to the image to be analyzed, a system capable of analyzing any image can be constructed. For example, as shown in Figure 2, a system for detecting danger (an automated warning system) and a system for analyzing television images (an automated scene extraction system) can be constructed.

[0011] Figure 3 is a diagram illustrating example 1 of the challenges of VQA. One of the major challenges of VQA is that, as shown in Figure 3, when multiple objects (in the example in Figure 3, a bus) are visible in an image, it is impossible to determine which object the question was directed at.

[0012] For example, if the question is "What color is the bus?", the left image in Figure 3 shows three buses, while the right image in Figure 3 shows two buses, making it unclear which bus to answer. For example, questions like "What color is the bus on the left?", "What color is the bus in the middle?", and "What color is the bus on the right?" can be answered using the left image in Figure 3, but the right image in Figure 3 only shows two buses, making the answer ambiguous.

[0013] In other words, if individual images can be reviewed in advance, appropriate questions can be asked that identify the target object. However, with a system that targets arbitrary images, it is not possible to review the images in advance, making it impossible to ask appropriate questions.

[0014] Figure 4 is a diagram illustrating example 2 of the challenges of VQA. Another challenge of VQA is that, as shown in Figure 4, when there are multiple objects to be analyzed, such as people, dogs, trains, buses, tables, and airplanes, in order to cover all of them, a question list containing 23 questions about all objects must be applied to all images.

[0015] For example, in the example in Figure 4, the leftmost of the five images does not show a train, bus, table, or airplane. However, questions about trains, buses, tables, and airplanes must also be processed using the leftmost image, resulting in an unnecessary increase in processing load.

[0016] In other words, the more images to be analyzed, the more questions are asked, and questions unrelated to the images being processed are also processed, thus increasing the processing load.

[0017] [Example of functional configuration] Figure 5 shows an example of the functional configuration of the information processing device 1 of the first embodiment. The information processing device 1 of the first embodiment includes a detection unit 11, a cutting unit 12, an acquisition unit 13, a storage unit 14, a VQA processing unit 15, and a display control unit 16.

[0018] The detection unit 11 detects at least one piece of object information from the image, which includes a target region containing the object to be detected (e.g., an object and a person) and an object ID (an example of object identification information) that identifies the object to be detected. The object ID includes a label indicating the type of object to be detected (an example of an object type) and information that uniquely identifies the object to be detected (e.g., a number). For example, if the detected object is a person, the object is identified by assigning a unique object ID to each object, such as "object (person) 1" and "object (person) 2".

[0019] For the detection process, for example, Non-Patent Document 2 is used. In normal object detection, the detection process is performed on all learned labels, but in the first embodiment, the detection process is performed on labels related to pre-set questions.

[0020] The detection engine used in the detection unit 11 returns the coordinates of the box containing the detected object and a label when a detection target is found. The format of the box coordinates is, for example, the coordinates that identify the rectangular area (the x and y coordinates of the top left and the x and y coordinates of the bottom right).

[0021] The cropping unit 12 generates at least one target image by cropping at least one target region from the image. For example, for each detected target, the cropping unit 12 crops a rectangular region containing that target as the target region.

[0022] The acquisition unit 13 acquires at least one question from the storage unit 14 that corresponds to a label indicating the type of object to be detected.

[0023] The memory unit 14 stores a list of questions (at least one question) for each label indicating the type of object to be detected.

[0024] The VQA processing unit 15 performs VQA processing with at least one question for at least one target image.

[0025] The VQA processing unit 15 uses an AI that judges the image content and provides an answer to an arbitrary question, as shown in Figure 1 above. Various configurations of the AI ​​are possible, but a typical configuration is shown at the bottom of Figure 1. The image is converted into a feature vector by an image encoder. Similarly, the question (text) is converted into a feature vector by a text encoder. The answer is obtained by a decoder that fuses both feature vectors to generate the answer.

[0026] VQA's AI model is trained using, for example, the VQA dataset (Non-Patent Document 3).

[0027] The VQA processing unit 15 inputs the selected image and question into the VQA AI model and obtains an answer from the AI ​​model. The VQA processing unit 15 may also obtain the predicted probability (confidence level) of the answer from the AI ​​model at the same time as the answer and output this predicted probability as well.

[0028] Labels associated with the questions may, for example, conform to MSCOCO (Microsoft® Common Objects in Context). MSCOCO is the most standard benchmark dataset for object detection. It uses 80 labels for common objects such as people, cars, dogs, cats, trains, buses, and tables (see, for example, Non-Patent Document 4).

[0029] In this embodiment, a special label called "image" may be prepared. This label represents the entire image, and questions about the entire scene can be asked using this "image" label.

[0030] Furthermore, you don't need to use all the labels provided by MSCOCO; you only need to use the labels necessary for your question (for example, "person" or "car").

[0031] Figure 6 is a schematic diagram showing an example of processing by the VQA processing unit 15 of the first embodiment. In view of the problems described in Figures 3 and 4 above, as shown in Figure 6, the VQA processing unit 15 applies VQA to each object detected from the image. By applying VQA to each object, the problem described in Figure 3 above can be solved.

[0032] Furthermore, the questions asked about an image should vary depending on the type of object being detected (label). For example, one might ask a person about the color of their clothes, but this question is not appropriate for a car. Conversely, one might ask a car how many doors it has, but this is not appropriate for a person. Therefore, as shown in Figure 6, a list of questions is prepared for each label (in the example in Figure 6, car, person, and dog), and the VQA processing unit 15 applies the appropriate question for each label of the detected object.

[0033] Specifically, in the example in Figure 6, multiple questions appropriate to each label are applied to identify cars, people, and dogs. By providing a different list of questions for each label that identifies the subject, the problem described in Figure 4 above can be solved.

[0034] Furthermore, each question is managed by a question ID (an example of question identification information) that uniquely identifies the question. The acquisition unit 13 assigns question identification information to at least one question applied to each target image generated by the cropping unit 12.

[0035] Figure 7 shows an example of a question ID format for identifying questions in the first embodiment. Since multiple objects with the same label may be detected from an image, it is necessary to uniquely identify each object and the question and answer for that object. The example in Figure 7 is an example of a format for managing questions and answers for each detected object.

[0036] Figure 7 shows an example where there are two objects ("Object (Person) 1" and "Object (Person) 2") and "Person-Question 1" and "Person-Question 2" related to labels that identify those objects.

[0037] In the example in Figure 7, "Person-Question 1" and "Person-Question 2" are IDs that identify questions about a person. "Person-Question 1" indicates that it is the first question asked to the person. "Person-Question 2" indicates that it is the second question asked to the person. In this case, there are a total of four possible combinations, each assigned a unique question ID from "ID1" to "ID4".

[0038] By associating the VQA responses with these question IDs, it becomes possible to uniquely match the responses, subjects, and questions (see Figure 8). These responses, subjects, and questions may be output as VQA processing result information in data format such as a file. That is, the VQA processing unit 15 may output VQA processing result information in which the subject ID (an example of subject identification information), the question ID (an example of question identification information), and the answer to the question identified by the question ID are associated.

[0039] Furthermore, the question ID used to identify a question may be any word, such as "gender" or "color," rather than a number, as long as it provides unique identification information. There are also no restrictions on the number of labels used to identify the target of detection or the number of questions.

[0040] Returning to Figure 5, the display control unit 16 visualizes the VQA processing result information in an easy-to-understand manner by displaying the VQA processing result information superimposed on the image on the display device.

[0041] Figure 9 shows an example of display information according to the first embodiment. Figure 9 shows an example in which one car, three people, and one dog are detected. In the example in Figure 9, the detected objects are displayed as boxes, and the object ID, question ID, and the answer to the question are arranged above the boxes, making the VQA processing result information easy to understand and visualize. That is, the display control unit 16 displays the display information attached to the detected object identified by the object ID included in the VQA processing result information, which includes the object ID, question ID, and the answer to the question, on the display device.

[0042] [Examples of information processing methods] Figure 10 is a flowchart showing an example of an information processing method according to the first embodiment. First, the detection unit 11 detects at least one object from the image (step S1) and obtains object information including information that identifies the object region (for example, the box coordinates described above) and a label that identifies the object (step S2).

[0043] Next, the cropping unit 12 acquires a target image by cropping the target region from the image based on the target information acquired in step S2 (step S3).

[0044] Next, the acquisition unit 13 acquires a list of questions (at least one question) corresponding to a label identifying the target from the storage unit 14, based on the target information acquired in step S2 (step S4).

[0045] Next, the VQA processing unit 15 retrieves a question from the question list obtained in step S4 (step S5). Then, the VQA processing unit 15 performs VQA processing using the target image obtained in step S3 and the question obtained in step S5 (step S6).

[0046] Next, the VQA processing unit 15 determines whether or not it has processed all of the question list (step S7). If it has not processed all of the question list (step S7, No), the process returns to step S5.

[0047] If all questions on the list have been processed (Step S7, Yes), the detection unit 11 determines whether all the targets detected in Step S1 have been processed (Step S8). If not all the targets detected in Step S1 have been processed (Step S8, No), the process returns to Step S2. If all the targets detected in Step S1 have been processed (Step S8, Yes), the process ends.

[0048] As described above, in the information processing device 1 of the first embodiment, the detection unit 11 detects at least one piece of target information from an image, which includes a target region containing the object to be detected and target identification information that identifies the object to be detected. The cropping unit 12 generates at least one target image by cropping at least one target region from the image. The acquisition unit 13 acquires at least one question corresponding to the target identification information. The VQA processing unit 15 performs VQA processing with at least one question for each of the at least one target images.

[0049] As a result, according to the first embodiment, even when an image contains multiple objects, a more accurate processing result can be obtained with less processing effort.

[0050] Figure 11 is a diagram illustrating Example 1 of the effects of the first embodiment. According to the first embodiment, questions can be applied to each subject, and answers to those questions can be obtained. In the example in Figure 11, a list of questions about buses is applied to each of the three buses.

[0051] Figure 12 is a diagram illustrating an example of the effect of the first embodiment. As shown in Figure 12, the amount of VQA processing can be reduced by applying a question list tailored to the subject. In the example in Figure 12, 2+2+2+2+3+2=13 VQA applications are required for 5 images.

[0052] While conventional VQA also utilized the features of detected objects (Non-Patent Literature 5), there was a one-to-one correspondence between the features of all objects in the image and the questions. Therefore, in the example in Figure 4 above, since there are 23 questions for the 6 types of objects, applying them to the same 5 images as in Figure 12 would require 23 * 5 = 115 VQA processes.

[0053] (Second Embodiment) Next, a second embodiment will be described. In the description of the second embodiment, explanations similar to those of the first embodiment will be omitted, and the differences from the first embodiment will be explained. In the second embodiment, a case in which voting behavior is incorporated in order to stabilize the answers to questions about the video will be described.

[0054] [Example of functional configuration] Figure 13 shows an example of the functional configuration of the information processing device 1-2 of the second embodiment. The information processing device 1-2 of the second embodiment includes a detection unit 11, a cutting unit 12, an acquisition unit 13, a storage unit 14, a VQA processing unit 15, a display control unit 16, and a voting unit 17. In the second embodiment, the voting unit 17 is added to the configuration of the first embodiment.

[0055] The detection unit 11 detects at least one of the above-mentioned target information from the frames included in the video.

[0056] The voting unit 17 votes on the answers to questions obtained by performing VQA processing on target images generated for each frame included in the video, and determines the answers to the questions for each target image based on the voting results.

[0057] Figure 14 is a schematic diagram showing an example of the voting process in the second embodiment. If a target is found, the detection unit 11 returns the coordinates of the box containing the target and the target ID that identifies the target being tracked. In the example in Figure 14, the box coordinates are in the format of a rectangle (x coordinate of the top left, y coordinate of the top left, x coordinate of the bottom right, y coordinate of the bottom right).

[0058] The voting unit 17 prepares a voting box (response bin) for each target ID, enabling statistical VQA.

[0059] The extraction unit 12 extracts the target area identified by the box coordinates and inputs it to the VQA processing unit 15. The acquisition unit 13 also acquires a list of questions corresponding to the labels of the target area from the storage unit 14 and inputs the question list to the VQA processing unit 15.

[0060] The VQA processing unit 15 returns the most likely answer for the target image and the predicted probability of that answer.

[0061] The voting unit 17 performs a voting action by adding the predicted probability to the answer bin in the voting box. The more frames in which the same answer is predicted, the longer the bin for that answer will grow. Finally, the voting unit 17 makes a determination judgment based on the determination conditions, and if it is determined, it outputs the answer with the most votes as the final answer. If the answer is not determined, a new frame is input to the detection unit 11, and processing is performed again from target detection.

[0062] Furthermore, for tracking objects (e.g., objects and people), a method is used that tracks the object with the largest overlap with the bounding box (bbox) containing the object between frames, for example, using Intersection over Union (IoU).

[0063] Figure 15 shows the relationship between the number of votes and recognition accuracy in the second embodiment. As shown in Figure 15, statistical judgments can be made by voting on answers for multiple frames included in the video. The example in Figure 15 shows the recognition accuracy when, each time one answer is obtained for a frame by VQA, this answer is voted for once, and the answer with the most votes for each frame is used as the final answer result.

[0064] In the case of 2-frame voting, the accuracy decreases because the votes are split and the decisiveness is reduced, but it can be seen that the accuracy improves significantly with 3 frames or more. Of course, the higher the accuracy of the engine (VQA's AI model) itself, the faster the final accuracy converges. For this reason, the voting unit 17 of the second embodiment performs voting using a voting method of 3 frames or more.

[0065] [Examples of information processing methods] Figure 16 is a flowchart showing an example of the information processing method of the second embodiment. The processing in steps S11 to S17 and S19 is the same as the processing in steps S1 to S8 of the first embodiment, so the explanation is omitted.

[0066] In the second embodiment, after VQA processing is performed on the target to be detected in each frame included in the video, voting processing for responses to the target to be detected that has been tracked across multiple frames is performed (step S18).

[0067] Figure 17 is a flowchart showing an example of the voting process in step S18 of the second embodiment. First, the voting unit 17 takes one target o from a tracking list which includes at least one tracking target (step S21).

[0068] Next, the voting unit 17 calculates the IoU between the bbox of the previous frame of target o and the bbox of the frame to be processed (step S22). Next, the voting unit 17 determines whether the IoU is greater than or equal to a threshold and whether the labels of the two bboxes used to calculate the IoU are the same (step S23).

[0069] If IoU is above a threshold and the labels of the two bboxes used to calculate IoU are the same (Step S23, Yes), the predicted probability is added to the target voting box (response bin) identified by that label (Step S26).

[0070] If IoU is below the threshold, or if the labels of the two bboxes used to calculate IoU are not the same (step S23, No), the voting unit 17 determines whether all tracking targets included in the tracking list have been processed (step S24). If not all tracking targets included in the tracking list have been processed (step S24, No), the process returns to step S21.

[0071] If all tracking targets included in the tracking list have been processed (Step S24, Yes), the voting unit 17 adds the targets included in the bbox of the frame to be processed to the tracking list as new tracking targets because the targets included in the bbox of the frame to be processed do not exist in the tracking list (Step S25). Next, the voting unit 17 adds the predicted probability to the voting box (response bin) of the target included in the bbox of the frame to be processed (Step S26).

[0072] Figure 18 is a flowchart showing an example of the finalization process in step S18 of the second embodiment. First, the voting unit 17 calculates the total number of votes for each answer to the question (step S31). Next, the voting unit 17 calculates the voting probability by dividing the bin for each answer by the total number of votes calculated in step S31 (step S32). Next, the voting unit 17 identifies the bin with the most votes and obtains the number of votes for the bin with the most votes and the voting probability for the bin with the most votes (step S33).

[0073] Next, the voting unit 17 determines whether the number of votes obtained in step S33 is equal to or greater than a threshold (step S34). If the number of votes is not equal to or greater than the threshold (step S34, No), the voting unit 17 leaves the answer undetermined and terminates the process.

[0074] If the number of votes is equal to or greater than the threshold (Step S34, Yes), the voting unit 17 determines whether the voting probability obtained in Step S33 is equal to or greater than the threshold (Step S35). If the voting probability is not equal to or greater than the threshold (Step S35, No), the voting unit 17 leaves the answer undetermined and terminates the process.

[0075] If the voting probability is above the threshold (step S35, Yes), the voting unit 17 confirms the answer and terminates the process. In other words, if a certain degree of confidence is obtained (the number of votes is above the threshold, AND the voting probability is above the threshold), the process is interrupted midway to determine the answer.

[0076] The flowchart in Figure 18 allows us to determine what the answer to a question is when video (camera footage) is used as input.

[0077] (Third embodiment) Next, a third embodiment will be described. In the description of the third embodiment, explanations similar to those of the first embodiment will be omitted, and the differences from the first embodiment will be explained. In the third embodiment, a case in which the target image is deformed in order to improve the accuracy of answering questions will be described.

[0078] [Example of functional configuration] Figure 19 shows an example of the functional configuration of the information processing device 1 of the third embodiment. The information processing device 1-3 of the third embodiment includes a detection unit 11, a cutting unit 12, an acquisition unit 13, a storage unit 14, a VQA processing unit 15, a display control unit 16, and a deformation unit 18. In the third embodiment, the deformation unit 18 is added to the configuration of the first embodiment.

[0079] The deformation unit 18 deforms the target area (e.g., a box) detected by the detection unit 11 according to at least one of the label and question used to identify the target.

[0080] For example, the deformation unit 18 merges the box containing the person and the box containing the dog, which are nearby, to transform into a new box. Also, for example, if the deformation unit 18 asks the person a question such as "Are you standing on a stepladder?", it assumes that the stepladder is below the person, so it expands the box containing the person downwards.

[0081] The transformation method is stored, for example, associated with each question in a question list. Specifically, a question like "What is your gender?" does not have a transformation method associated with it. On the other hand, a question like "What are you doing?" has a transformation method associated with it, such as expanding the box by 1.5 times, because it is necessary to check the surroundings to determine the action.

[0082] Figure 20 is a diagram illustrating an example of processing of the deformation part 18 in the third embodiment. Figure 20 shows an example of asking the question "What are you doing?" in a scene of walking a dog. If the target area is cut out individually for each object, as shown in the upper part of Figure 20, only the person will be cut out, so it is expected that the answer will be "walking" rather than "walking a dog". On the other hand, if the deformation unit 18 deforms the target area by merging the person box and the dog box, which are close to each other, as shown in the lower part of Figure 20, and the cutting unit 12 cuts it out as a new box, then "walking a dog" can be obtained as the answer.

[0083] (Fourth Embodiment) Next, the fourth embodiment will be described. In the description of the fourth embodiment, explanations similar to those of the first embodiment will be omitted, and the differences from the first embodiment will be described. In the fourth embodiment, the case in which the target is detected using an open vocabulary will be described.

[0084] Conventional object detection can only detect objects of a type that matches the labels defined during training. For example, if you want to ask a question about sports cars, you need to use the label "car," which will also detect cars other than sports cars. To address this problem, open vocabulary object detection has been proposed in recent years (Non-Patent Literature 6). This is a type of object detection where the user can specify the label at the time of detection, and it can detect sports cars.

[0085] [Example of functional configuration] Figure 21 is a diagram showing an example of the functional configuration of the information processing device 1 of the fourth embodiment. The information processing device 1-4 of the fourth embodiment includes a detection unit 11-2, a cutting unit 12, an acquisition unit 13, a storage unit 14-2, a VQA processing unit 15, a display control unit 16, and a deformation unit 18.

[0086] The detection unit 11-2 performs object detection from the image using open vocabulary. In open vocabulary object detection, it is necessary to provide the label of the object to be detected during inference. Therefore, as shown in Figure 21, the detection unit 11-2 reads the label specified along with the question from the storage unit 14-2 and uses it as the input label for open vocabulary object detection.

[0087] In other words, the detection unit 11-2 detects at least one piece of target information from the image, which includes a target area containing the target of the input label (an example of target type) read from the storage unit 14-2, and a target ID (an example of target identification information).

[0088] (Fifth embodiment) Next, the fifth embodiment will be described. In the description of the fifth embodiment, explanations similar to those of the first embodiment will be omitted, and the differences from the first embodiment will be described. The functional configuration of the information processing device 1 in the fifth embodiment is the same as that of the first embodiment.

[0089] In the fifth embodiment, the detection unit 11 receives a question from the user and detects a target corresponding to the question. For example, when the detection unit 11 receives the question "What is your gender?", it identifies that the label of the target to be detected (an example of a target type) is "person" and detects a target classified under that label from the image. The VQA processing unit 15 then applies the question "What is your gender?" to the target image detected by the detection unit 11 and extracted by the cropping unit 12, and outputs the answer.

[0090] Finally, we will describe examples of the hardware configuration of the information processing device 1 (1-2 to 1-4) according to the first to fifth embodiments.

[0091] [Example hardware configuration] Figure 22 shows examples of the hardware configuration of the information processing device 1 (1-2 to 1-4) according to the first to fifth embodiments. The information processing device 1 comprises a processor 201, a main memory 202, an auxiliary storage device 203, a display device 204, an input device 205, and a communication device 206. The processor 201, the main memory 202, the auxiliary storage device 203, the display device 204, the input device 205, and the communication device 206 are connected via a bus 210.

[0092] Furthermore, the information processing device 1 may not be equipped with some of the above-described components. For example, if the information processing device 1 can utilize the input and display functions of an external device, the information processing device 1 may not be equipped with the display device 204 and the input device 205.

[0093] The processor 201 executes the program read from the auxiliary storage device 203 into the main memory 202. The main memory 202 is memory such as ROM (Read Only Memory) and RAM (Random Access Memory). The auxiliary storage device 203 is such as an HDD (Hard Disk Drive) and a memory card.

[0094] The display device 204 is, for example, a liquid crystal display. The input device 205 is an interface for operating the information processing device 1. The display device 204 and the input device 205 may be implemented by a touch panel or the like that has both display and input functions. The communication device 206 is an interface for communicating with other devices.

[0095] For example, a program executed by the information processing device 1 is provided as a computer program product, recorded in an installable or executable file format on a computer-readable storage medium such as a memory card, hard disk, CD-RW, CD-ROM, CD-R, DVD-RAM, or DVD-R.

[0096] Alternatively, for example, the program executed by the information processing device 1 may be stored on a computer connected to a network such as the Internet, and provided by allowing users to download it via the network.

[0097] Alternatively, the information processing device 1 may be configured to provide programs via a network such as the Internet without requiring users to download them. Specifically, it may be configured, for example, as an ASP (Application Service Provider) type cloud service.

[0098] Alternatively, for example, the program for the information processing device 1 may be pre-installed and provided in ROM or the like.

[0099] The program executed by the information processing device 1 has a modular configuration that includes functions that can also be implemented by the program, as described above. In actual hardware terms, each of these functions is loaded onto the main memory 202 by the processor 201 reading and executing the program from the storage medium. In other words, each of the above function blocks is generated on the main memory 202.

[0100] Furthermore, some or all of the above-mentioned functions may be implemented using hardware such as an IC (Integrated Circuit) instead of software.

[0101] Alternatively, multiple processors 201 may be used to implement each function, in which case each processor 201 may implement one of the functions, or two or more of the functions.

[0102] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of Symbols]

[0103] 1. Information Processing Device 11 Detection Unit 12 Cutting section 13 Acquisition Department 14 Storage section 15 VQA Processing Unit 16 Display Control Unit 17 Voting Department 18 Deformed part 201 Processor 202 Main storage 203 Auxiliary storage device 204 Display device 205 Input device 206 Communication equipment 210 Bus

Claims

1. Computers, A detection unit that detects at least one piece of target information from an image, which includes a target region containing the target to be detected and target identification information that identifies the target to be detected. A cropping unit that generates at least one target image by cropping out at least one target region from the aforementioned image, An acquisition unit that acquires at least one question corresponding to the aforementioned target identification information, For each of the at least one target image, a VQA (Visual Question Answering) processing unit executes VQA processing using the at least one question. A program designed to function as such.

2. The aforementioned computer, The system further comprises a storage unit that stores at least one question for each target type indicating the type of object to be detected, The aforementioned target identification information includes information indicating the type of target, The acquisition unit acquires from the storage unit at least one question corresponding to the target identification included in the target identification information. The program according to claim 1.

3. The detection unit detects at least one piece of target information from the image, which includes a target region containing a target of a specific type read from the storage unit, and the target identification information. The program according to claim 2.

4. The detection unit receives a question from the user regarding the image, identifies the target type from the question, and detects at least one piece of target information including a target region containing the detection target of the identified target type and the target identification information. The program according to claim 2 or 3.

5. The aforementioned computer, The aforementioned target area is further made to function as a deformation part that deforms according to at least one of the aforementioned target type and the aforementioned question, The cropping unit generates at least one target image by cropping out at least one target region, or at least one target region that has been deformed by the deformation unit, from the image. The program according to claim 2 or 3.

6. The acquisition unit assigns question identification information to at least one question applied to each of the target images. The VQA processing unit outputs VQA processing result information that associates the target identification information, the question identification information, and the answer to the question identified by the question identification information. The program according to any one of claims 1 to 3.

7. The aforementioned computer, A display control unit that displays on a display device the display information assigned to the detection target identified by the target identification information included in the VQA processing result information. The program according to claim 6, which further functions as such.

8. The aforementioned image is a frame included in the video. The detection unit detects at least one of the target information from the frame, The aforementioned computer, A voting unit that votes on the target image generated for each frame, assigning answers to questions obtained by performing VQA processing, and determines the answers to the questions for each target image based on the voting results. The program according to any one of claims 1 to 3, which further functions as such.

9. A detection unit that detects at least one piece of target information from an image, which includes a target region containing the target to be detected and target identification information that identifies the target to be detected. A cropping unit that generates at least one target image by cropping out at least one target region from the aforementioned image, An acquisition unit that acquires at least one question corresponding to the aforementioned target identification information, A VQA processing unit that performs Visual Question Answering (VQA) processing with the at least one question for each of the at least one target images, An information processing device equipped with the following features.

10. The information processing device detects at least one piece of target information from an image, which includes a target region containing the object to be detected and object identification information that identifies the object to be detected. The information processing device generates at least one target image by extracting at least one target region from the image, The information processing device takes the step of obtaining at least one question corresponding to the target identification information, The information processing device performs a Visual Question Answering (VQA) process for each of the at least one target images, using the at least one question. Information processing methods including

Images