Information processing program, information processing method, and information processing device

Abstract

This information processing program causes a computer to execute processing of: acquiring a video to be monitored; identifying a first area where a first object is located in a predetermined video frame among a plurality of video frames included in the acquired video and a question related to the first object existing in the first area; analyzing the acquired video to identify a second object related to the first object existing in the first area among a plurality of objects respectively included in the plurality of video frames; and generating an answer to the question on the basis of the question related to the first object and image feature amounts of the first object and the second object.

Description

Information processing program, information processing method, and information processing apparatus

[0001] This invention relates to an information processing program, an information processing method, and an information processing apparatus.

[0002] In recent years, advancements in LMM (Large Multi-modal Model) technologies such as GPT®-4o and Gemini®-1.5 Pro have led to remarkable improvements in the image and video comprehension capabilities of information processing devices. This improved image and video comprehension capability has enabled information processing devices to perform practical tasks such as generating captions and conducting visual question answering (VQA) related to input images and videos.

[0003] However, while many conventional LMMs excel at understanding broad spatial and temporal information, they struggle to understand local spatial and temporal information. For example, spatially, conventional LMMs are adept at understanding what is happening throughout an entire image, but they are less adept at understanding finer details within the image, such as specific locations or people. Temporally, because conventional LMMs smooth and store a video over time, they are good at retaining information about events with significant visual changes. However, if the visual changes of a particular object are relatively small, information about that object's events, even if important, is easily lost. In other words, conventional LMMs are poor at understanding events involving specific objects with relatively small temporal visual changes.

[0004] Therefore, in order to further improve comprehension capabilities, it is important that LMM has a mechanism that can prioritize the extraction and processing of information on user-specified targets in order to accurately understand spatially and temporally localized video information. The following technologies exist for prioritizing the extraction and processing of information on user-specified targets.

[0005] For example, there is a technology that uses visual prompts for images. A visual prompt is a visual instruction written directly on an image by the user. By using technology that uses visual prompts for images, LMM can perform image comprehension and VQA under conditions that focus on specified areas.

[0006] There are also technologies that use visual prompts for videos. These technologies allow LMM to propagate a visual prompt for one video frame to all frames of the video, enabling video understanding and VQA (Visual Quality Analysis) focused on specified objects. This technology performs ROI (Regions of Interest) tracking and ROI selection. ROI tracking is the process of tracking user-specified ROIs using the HQ (High Quality) track framework, a high-quality tracking framework for videos. ROI selection, on the other hand, is the process of reducing redundancy in the tracked ROI sequence using the K-means method.

[0007] Zeyi Sun, Ye Fang, Tong Wu, Pan Zhang, Yuhang Zang, Shu Kong, Yuanjun Xiong, Dahua Lin, Jiaqi Wang, ”Alpha-CLIP: A CLIP Model Focusing on Wherever You Want”, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 13019-13029Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P. Meyer, Yuning Chai, Dennis Park, Yong Jae Lee, ”ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts”, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 12914-12923Jihao Qiu, Yuan Zhang, Xi Tang, Lingxi Xie, Tianren Ma, Pengyu Yan, David Doermann, Qixiang Ye, Yunjie Tian, ”Artemis: Towards Referential Understanding in Complex Videos”, arXiv Accessibility Forum 2024, 1 Jun 2024

[0008] However, when using visual prompts for images, it is practically difficult for the user to provide visual prompts for every frame of a video. Furthermore, if a visual prompt is specified for a specific frame of a video, it is not obvious whether the LMM can interpret that visual prompt similarly for all frames of the video. Therefore, it is difficult to improve the LMM's ability to understand video using visual prompts for images.

[0009] Furthermore, while conventional visual prompts for videos consider the target specified by the visual prompt, they do not consider or retain surrounding information about the target. This leads to a decrease in accuracy in video comprehension and VQA, which require surrounding information. Therefore, it is difficult to improve LMM's ability to understand video using conventional visual prompt technology.

[0010] The disclosed technology was made in view of the above, and aims to provide an information processing program, an information processing method, and an information processing device that improve the ability to understand images and videos.

[0011] In one embodiment, the information processing program, information processing method, and information processing apparatus disclosed in this application acquire video footage of a subject to be monitored, identify a first region in which a first object is located within a predetermined video frame among a plurality of video frames contained in the acquired video, and identify a question relating to the first object located in the first region, respectively, analyze the acquired video footage to identify a second object related to the first object located in the first region among a plurality of objects in each of the plurality of video frames, and cause a computer to perform a process to generate an answer to the question based on the question relating to the first object, the image features of the first object and the second object.

[0012] According to one embodiment of the information processing program, information processing method, and information processing apparatus disclosed in this application, the ability to understand images and videos is improved.

[0013] Figure 1 is a block diagram of the VQA device according to the embodiment. Figure 2 is a block diagram of the related area estimation unit. Figure 3 is a diagram showing an overview of the question-and-answer processing by the VQA device according to the embodiment. Figure 4 is a diagram showing an overview of the related area estimation processing by the related area estimation unit. Figure 5 is a diagram showing an example of the related area information generation process. Figure 6 is a diagram for explaining the learning of the estimation module of the related area estimation unit. Figure 7 is a diagram showing a scene graph. Figure 8 is a diagram showing an example of the degree of relevance. Figure 9 is a diagram for explaining the learning of the question-and-answer processing by the LLM decoder. Figure 10 is a flowchart of the question-and-answer processing by the VQA device according to the embodiment. Figure 11 is a flowchart of the related area estimation processing by the related area estimation unit according to the embodiment. Figure 12 is a diagram showing an example of a factory image with a visual prompt set. Figure 13 is a hardware configuration diagram of the VQA device.

[0014] The following describes in detail, with reference to the drawings, embodiments of the information processing program, information processing method, and information processing apparatus disclosed in this application. However, the following embodiments do not limit the information processing program, information processing method, and information processing apparatus disclosed in this application.

[0015] Figure 1 is a block diagram of a VQA device according to an embodiment. As shown in Figure 1, the VQA device 1 is connected to a video output device 2 and a user terminal device 3.

[0016] The video output device 2 is a device that outputs long-duration video of a monitored object, such as more than one hour. As an example, the video output device 2 acquires and outputs video that is continuously captured by a fixed-point video camera placed inside a factory.

[0017] The user terminal device 3 is used by users who request answers to video-based questions using the VQA device 1. Specifically, the user terminal device 3 receives video input from the video output device 2. The user refers to the display screen of the user terminal device 3 and uses the user terminal device 3 to select a frame from the video acquired by the user terminal device 3 to specify a visual prompt indicating the object of their interest. Hereinafter, the frame selected by the user as the frame for specifying the visual prompt will be referred to as the "selected frame." The user then uses the user terminal device 3 to specify the area in the selected frame that contains the object of their interest using a visual prompt. Furthermore, the user terminal device 3 receives input from the user in the form of questions related to the area of ​​interest indicated by the visual prompt.

[0018] The user terminal device 3 then outputs information about the selected frame and a visual prompt indicating the object of the user's attention to the designated area extraction unit 104 of the VQA device 1. The user terminal device 3 also outputs a text prompt containing a question about the object of the user's attention to the text conversion unit 111 of the VQA device 1. Here, the object specified by the user using the visual prompt is an example of the "first object," and the frame in which the object of attention is specified using the visual prompt is an example of the "predetermined video frame." Furthermore, the text prompt containing a question about the object of the user's attention is an example of the "question about the first object."

[0019] Furthermore, users can use the display screen of the user terminal device 3 to check the answers from the VQA device 1 to questions about the subject they are interested in.

[0020] The VQA device 1 includes a visual encoder 101, a spatiotemporal feature calculation unit 102, an overall projector 103, a specified region extraction unit 104, an ROI tracker 105, a related region estimation unit 106, a sub-region feature calculation unit 107, a selection unit 108, and a projector 109. Furthermore, the VQA device 1 includes an LLM (Large Language Models) decoder 110, a text conversion unit 111, and an embedding unit 112.

[0021] The visual encoder 101 receives the input of the video output by the video output device 2. Then, for each frame of the video, the visual encoder 101 calculates the feature amount of the entire image. Here, the picture represented by the entire frame of the video is called an image. That is, the video is a continuous set of images for each frame. Also, hereinafter, the feature amount of the entire image is called the image feature amount. The visual encoder 101 outputs the image feature amount for each frame to the spatio-temporal feature amount calculation unit 102 and the partial region feature amount calculation unit 107.

[0022] The spatio-temporal feature amount calculation unit 102 receives the input of the image feature amount for each frame of the video from the visual encoder 101. Then, based on the temporal and spatial relationships of each object shown in the image for each frame, the spatio-temporal feature amount calculation unit 102 calculates the spatial feature amount and the temporal feature amount of the entire video. Then, the spatio-temporal feature amount calculation unit 102 outputs the spatial feature amount and the temporal feature amount of the entire video to the overall projector 103.

[0023] Here, in the present embodiment, the spatio-temporal feature amount calculation unit 102 calculates both the spatial feature amount and the temporal feature amount of the entire video, but either one may be used. That is, the spatio-temporal feature amount calculation unit 102 calculates the spatial or temporal image feature amount of the video.

[0024] The overall projector 103 receives the input of the spatial feature amount and the temporal feature amount of the entire video from the spatio-temporal feature amount calculation unit 102. Then, the overall projector 103 performs an embedding for matching the spatial feature amount of the feature amount of the LLM decoder 110 with respect to the spatial feature amount and the temporal feature amount of the entire video. For example, the overall projector 103 performs processing such as matching the number of dimensions of the spatial feature amount and the temporal feature amount of the entire video with the number of dimensions of the feature amount space of the LLM decoder 110. After that, the overall projector 103 outputs the embedded data of the spatial feature amount and the temporal feature amount of the entire video to the LLM decoder 110.

[0025] The designated area extraction unit 104 receives, from the user terminal device 3, the information of the selected frame of the video output from the video output device 2 and the input of the information of the visual prompt specified by the user for the image of the selected frame. Then, the designated area extraction unit 104 extracts, as the ROI, the partial area on the image indicated by the visual prompt for the image of the selected frame. For example, the designated area extraction unit 104 can set the X-axis and Y-axis for the image of the selected frame, and represent the partial area using the X coordinate and Y coordinate representing each point of the image.

[0026] In this embodiment, the designated area extraction unit 104 extracts the ROI as an area called a BBox (Bounding Box). A BBox is a rectangular partial area that encloses the object area of interest with the smallest rectangle and delimits it with a boundary with respect to the external area. For example, the BBox is represented as a rectangle enclosing a predetermined area on the image of the selected frame. The designated area extraction unit 104 can represent the BBox by the XY coordinates of two vertices on the diagonal line, and sets the area enclosed by the BBox as the ROI. The designated area extraction unit 104 outputs the information of the ROI to the ROI tracker 105.

[0027] Thus, the designated area extraction unit 104 receives an operation of designating the first area where the first object is located in a predetermined video frame displayed on the display screen of the user terminal device 3, that is, an input of the information of the visual prompt.

[0028] The ROI tracker 105 receives the input of the video output by the video output device 2. The ROI tracker 105 also receives, from the designated area extraction unit 104, the input of the BBox indicating the information of the ROI corresponding to the visual prompt specified by the user.

[0029] Next, the ROI tracker 105 searches for and tracks the subregion on the image of the selected frame that corresponds to the ROI for each frame of the video. This allows the ROI tracker 105 to extract the subregion corresponding to the visual prompt specified by the user for each frame of the video output by the video output device 2. Subsequently, the ROI tracker 105 outputs information about the ROI and the subregions for each frame of the video to the related region estimation unit 106 and the subregion feature calculation unit 107. Hereafter, the ROI and the subregions for each frame of the video will be collectively referred to as the "ROI-corresponding subregion."

[0030] This ROI tracker 105 is an example of a "region identification unit." Furthermore, the ROI-corresponding region extracted by the ROI tracker 105 is an example of a "first region in which a first object is located within a predetermined video frame among multiple video frames included in the acquired video." In other words, the designated region extraction unit 104 acquires the video to be monitored and, based on the processing of the designated region extraction unit 104, identifies the first region in which a first object is located within a predetermined video frame among multiple video frames constituting the acquired video.

[0031] The related region estimation unit 106 has a machine learning model that estimates related regions associated with ROI-corresponding subregions in the image of each frame. The related region estimation unit 106 receives video input from the video output device 2. The related region estimation unit 106 also receives information about ROI-corresponding subregions from the ROI tracker 105.

[0032] The related region estimation unit 106 uses a machine learning model to estimate a predetermined number of related regions in descending order of relevance to the ROI-corresponding partial region in each frame, taking the overall image and the image of the ROI-corresponding partial region as input for each frame in which the ROI-corresponding partial region has been extracted. The related region estimation unit 106 then outputs information on the related regions that have a high degree of relevance to the estimated ROI-corresponding partial region as the estimation result.

[0033] Figure 2 is a block diagram of the related region estimation unit. The details of the related region estimation unit 106 will be described below with reference to Figure 2. As shown in Figure 2, the related region estimation unit 106 includes a preprocessing unit 161, a visual encoder 162, a partial region projector 163, a whole projector 164, a synthesis unit 165, a normalization unit 166, a decoding unit 167, and a region generation unit 168. Here, the estimation module 160 includes the visual encoder 162, the partial region projector 163, the whole projector 164, the synthesis unit 165, the normalization unit 166, and the decoding unit 167. This estimation module 160 is a machine learning model that estimates the related region associated with the ROI-corresponding partial region in the image of each frame.

[0034] The preprocessor 161 receives input from the video output by the video output device 2 and information on the ROI-compatible region output from the ROI tracker 105. Next, the preprocessor 161 identifies the frame from which the ROI-compatible region has been extracted from the video output by the video output device 2. Hereinafter, the frame from which the ROI-compatible region has been extracted will be referred to as the "target frame".

[0035] Then, the preprocessing unit 161 extracts a partial image corresponding to the ROI-compatible region from each of the target frame images. After that, the preprocessing unit 161 outputs the image of each target frame and the partial image corresponding to the ROI-compatible region in that frame to the visual encoder 162.

[0036] The visual encoder 162 receives, for each target frame, the image of that frame and a partial image corresponding to the ROI-compatible partial region in that frame as input from the preprocessor 161. Next, the visual encoder 162 calculates the feature quantity of the ROI-compatible partial region from the partial image corresponding to the ROI-compatible partial region in the target frame. Hereafter, the feature quantity of the ROI-compatible partial region will be referred to as the "partial region feature quantity". Subsequently, the visual encoder 162 outputs the partial region feature quantity of the target frame to the partial region projector 163.

[0037] Furthermore, the visual encoder 162 calculates the overall image features for the target frame. Hereafter, the overall image features extracted by the visual encoder 162 will be referred to as "overall features". Subsequently, the visual encoder 162 outputs the overall features of the target frame to the overall projector 164.

[0038] The sub-region projector 163 receives input of sub-region features in the target frame from the visual encoder 162. The sub-region projector 163 then performs conversion processing of the information representing each sub-region feature in order to make it easier to compare the sub-region features with the overall features. For example, the sub-region projector 163 performs image space aligning processing, which includes processing to match the dimensionality of the sub-region features and the overall features, and processing to align the parts of interest. After that, the sub-region projector 163 outputs the converted sub-region features of the target frame to the synthesis unit 165.

[0039] The overall projector 164 receives the overall feature quantities of the target frame as input from the visual encoder 162. The overall projector 164 then performs a conversion process for the information representing each overall feature quantity. After that, the overall projector 164 outputs the converted overall feature quantities for each target frame to the synthesis unit 165.

[0040] The synthesis unit 165 receives input of sub-region features for each target frame that has undergone transformation processing from the sub-region projector 163. The synthesis unit 165 also receives input of overall features for each target frame that has undergone transformation processing from the overall projector 164. Next, the synthesis unit 165 synthesizes the sub-region features and overall features for each target frame. For example, the synthesis unit 165 performs a matrix calculation to multiply the sub-region features and overall features. Hereafter, the result of synthesizing the sub-region features and overall features will be called the "synthesized features". Finally, the synthesis unit 165 outputs the synthesized features for each target frame to the normalization unit 166.

[0041] In this way, the synthesis unit 165 can obtain a feature related to the partial feature from the overall feature by synthesizing the partial feature and the overall feature. That is, this feature corresponds to a feature that indicates the region related to the ROI-corresponding partial region.

[0042] The normalization unit 166 receives the composite feature quantities for each target frame from the synthesis unit 165. Next, the normalization unit 166 normalizes each composite feature quantity using a function such as Softmax. After that, the normalization unit 166 outputs the normalized composite feature quantities to the decoding unit 167.

[0043] The decoding unit 167 receives normalized composite feature quantities for the target frame from the normalization unit 166. Next, the decoding unit 167 generates a relevance attention map from the composite feature quantities for the target frame, representing subregions with a high degree of relevance to the ROI-corresponding subregion. In this embodiment, the decoding unit 167 generates a predetermined number of relevance attention maps representing subregions in descending order of relevance. After that, the decoding unit 167 outputs the relevance attention map for each target frame to the region generation unit 168.

[0044] The region generation unit 168 receives the relevance attention map for each target frame as input from the decoding unit 167. The region generation unit 168 then generates related region information indicating the related region for each target frame from each relevance attention map. For example, the region generation unit 168 generates a BBox for the related region for each target frame. After that, the region generation unit 168 outputs the related region information for each target frame to the sub-region feature calculation unit 107.

[0045] Here, the related region is an example of the "second region," and the object included in the related region is an example of the "second object." That is, the related region estimation unit 106 analyzes the acquired video to identify the second object that is related to the first object that exists in the first region, which is a BBox representing the ROI, from among the multiple objects that each of the multiple video frames has. More specifically, the related region estimation unit 106 searches the first region for multiple video frames and identifies the second region containing the second object for each video frame. Furthermore, the related region estimation unit 106 uses an estimation module 160 that generates a relevance attention map that displays the surrounding regions related to the first object according to the strength of the relevance, and identifies the second object based on the attention map generated by the estimation module 160.

[0046] Returning to Figure 1, the explanation continues. The subregion feature calculation unit 107 receives image feature quantities for each frame of the video calculated by the visual encoder 101. The subregion feature calculation unit 107 also receives information on ROI-corresponding subregions output from the ROI tracker 105. Furthermore, the subregion feature calculation unit 107 receives related region information for each target frame estimated by the related region estimation unit 106.

[0047] Next, the subregion feature calculation unit 107 calculates the feature quantities of the ROI-corresponding subregion from the image feature quantities of each frame of the video. The subregion feature calculation unit 107 also calculates the feature quantities of the related region from the image feature quantities of each frame of the video. Finally, the subregion feature calculation unit 107 outputs the feature quantities of the ROI-corresponding subregion and the feature quantities of the related region to the selection unit 108.

[0048] The selection unit 108 receives feature quantities for the ROI-corresponding subregion and related regions as input from the subregion feature quantity calculation unit 107. The selection unit 108 selects the ROI-corresponding subregion features and related region features to be used to generate answers to questions by removing duplicate or unimportant features from among the ROI-corresponding subregion features and related region features. For example, the selection unit 108 can classify the ROI-corresponding subregion features and related region features using the K-means method, select groups considering the similarity of each group, and then select a predetermined number of features that match specific conditions from the selected groups. After that, the selection unit 108 outputs the selected ROI-corresponding subregion features and related region features to the projector 109.

[0049] The selection unit 108 selects features based on both the features of the ROI-corresponding subregion and the features of the related region, thereby taking into account the state of the related region as well as the ROI-corresponding subregion. For example, the selection unit 108 can select features even if there is little change in the ROI-targeted subregion, but there is a large change in the related region. This makes it possible to include important information about the related region in the questions. In this way, the VQA device 1 selects multiple image features from the image features of the first object and the second object using the K-means method.

[0050] The projector 109 receives input of feature quantities for the ROI-corresponding subregion and related regions selected by the selection unit 108. The projector 109 then performs an embedding process on the feature quantities for the ROI-corresponding subregion and related regions to match the feature quantity space of the LLM decoder 110. After that, the projector 109 outputs the embedded data of the ROI-corresponding subregion and related region feature quantities to the LLM decoder 110.

[0051] The text conversion unit 111 receives a text prompt input from the user terminal device 3 containing a question about the video, including the ROI. The text conversion unit 111 then performs text conversion processing according to the format of the question to the LLM decoder 110, such as dividing the text prompt into vocabulary. This allows the text conversion unit 111 to identify what kind of question was input to the LLM decoder 110. Subsequently, the text conversion unit 111 outputs the converted text prompt to the embedding unit 112.

[0052] In this way, the text conversion unit 111 identifies a question about a first object specified by the user using a visual prompt. More specifically, the text conversion unit 111 receives a question document from the user about a first object located in a first area, and identifies a question based on the question document.

[0053] The embedding unit 112 receives the text prompt input, which has undergone text conversion, from the text conversion unit 11. The embedding unit 112 then performs embedding processing, such as converting the text prompt into a vector, to convert it into a format that can be input to the LLM decoder 110. After that, the embedding unit 112 outputs the embedded text prompt to the LLM decoder 110.

[0054] The LLM decoder 110 is a machine learning model that receives input of image features and text prompts for questions about the image, and outputs a response to those questions. The LLM decoder 110 receives embedded data of spatial and temporal features of the entire video from the overall projector 103. The LLM decoder 110 also receives embedded data of features of ROI-corresponding subregions and related regions from the projector 109. Furthermore, the LLM decoder 110 receives embedded data of text prompts from the embedding unit 112.

[0055] The LLM decoder 110 then generates an answer to the question indicated by the text prompt, based on the embedded data of spatial and temporal features of the entire video, features of the ROI-corresponding subregion, and features of the related region. Subsequently, the LLM decoder 110 outputs the generated answer to the user terminal device 3.

[0056] In this way, the LLM decoder 110 generates an answer based on the spatial and temporal features of the entire video, as well as the features of the ROI-corresponding subregion, and the features of the related region. That is, the LLM decoder 110 can generate an answer to a question by considering events that occurred in the related region. The answer generated by the LLM decoder 110 is transmitted to the user terminal device 3 and displayed on the display screen.

[0057] In this way, the LLM decoder 110 generates an answer to a question based on the first object specified by the user using the visual prompt, which is the object specified by the visual prompt, and the image features of a second object present in the related region. More specifically, the LLM decoder 110 generates an answer based on a plurality of image features selected by the selection unit 108. The LLM decoder 110 is an example of a "large-scale multimodal model".

[0058] In other words, the VQA device 1 generates an answer to a question by inputting a prompt containing the question, the image features of the first object, and the second object into a large-scale multimodal model. The VQA device 1 also calculates embeddings for spatial or temporal image features, multiple image features selected by the selection unit 108, and the question, and inputs the calculated embeddings into the large-scale multimodal model to generate an answer.

[0059] Figure 3 is a diagram illustrating the overview of the question-and-answer processing performed by the VQA device according to the embodiment. Next, referring to Figure 3, the overall picture of the question-and-answer processing performed by the VQA device 1 will be explained. Figure 3 also shows the data used in each process. Each piece of data will be explained using the name indicated in Figure 3.

[0060] The video output device 2 outputs video V. Video V contains a large number of consecutive frames. The user uses the user terminal device 3 to select a frame F from video V and sets a visual prompt P for the selected frame F.

[0061] The visual encoder 101 extracts image feature quantities f from the video V for each frame.ｔ Calculate it.

[0062] The spatio-temporal feature quantity calculation unit 102 calculates the spatial feature quantity f of the video V and the temporal feature quantity f from the image feature quantity f of each frame calculated by the visual encoder 101. ｔ from the image feature quantity f of each frame calculated by the visual encoder 101. ｓｐａｔｉａｌ and the temporal feature quantity f ｔｅｍｐｏｒａｌ Calculate it.

[0063] The overall projector 103 performs an embedding process to match the spatial feature quantity f to the spatial feature quantity of the LLM decoder 110, and generates the embedded data e of the spatial feature quantity. Similarly, the overall projector 103 performs an embedding process to match the temporal feature quantity f to the spatial feature quantity of the LLM decoder 110, and generates the embedded data e of the temporal feature quantity. ｓｐａｔｉａｌ to match the spatial feature quantity f to the spatial feature quantity of the LLM decoder 110, and generates the embedded data e of the spatial feature quantity. ν ｓｐａｔｉａｌ Similarly, the overall projector 103 performs an embedding process to match the temporal feature quantity f to the spatial feature quantity of the LLM decoder 110, and generates the embedded data e of the temporal feature quantity. ｔｅｍｐｏｒａｌ to match the temporal feature quantity f to the spatial feature quantity of the LLM decoder 110, and generates the embedded data e of the temporal feature quantity. ν ｔｅｍｐｏｒａｌ Generate it.

[0064] The specified region extraction unit 104 generates a BBox 21 indicating the ROI, which is the partial region specified by the visual prompt P, based on the visual prompt P for the selected frame F.

[0065] The ROI tracker 105 searches each frame of the video V using the BBox 21, and generates a BBox 22 indicating the ROI corresponding partial region of each frame.

[0066] The related region estimation unit 106 estimates the related regions in each target frame from which the ROI corresponding partial region was extracted, from the BBox 22 indicating the ROI corresponding partial region of each frame and the video V. Here, the related region estimation unit 106 estimates L related regions in descending order of the degree of relevance.

[0067] The partial region feature quantity calculation unit 107 calculates the feature quantity f of the ROI corresponding partial region of each target frame from the BBox 22 indicating the ROI corresponding partial region of each target frame. Ｒｏｉ ｔ,０ Calculate it.

[0068] Furthermore, the sub-region feature calculation unit 107 calculates the feature f of each related region of each target frame from the information indicating the related region in each target frame. ＲＲｏｉ ｔ,１ ~f ＲＲｏｉ ｔ,Ｌ The subregion feature calculation unit 107 calculates the feature f. ＲＲｏｉ ｔ,１ ~f ＲＲｏｉ ｔ,Ｌ We calculate L feature quantities for each related region.

[0069] The selection unit 108 selects the feature quantity f of the ROI-corresponding subregion of each target frame. Ｒｏｉ ｔ,０ , and the feature quantities f of each related domain ＲＲｏｉ ｔ,１ ~f ＲＲｏｉ ｔ,Ｌ Select the features to use in your question from the options provided.

[0070] The projector 109 performs embedding processing on the feature quantities selected by the selection unit 108 to embed data e related to the ROI-corresponding subregion and related regions. ＲｏＩ ０ and embedded data e ＲｏＩ １ ~e ＲｏＩ Ｌ Generates.

[0071] The text conversion unit 111 performs text conversion processing on the text prompt T according to the format of the question to the LLM decoder 110.

[0072] The embedding unit 112 performs an embedding process on the text prompt T that has undergone text conversion to embed data e ｔ Generates.

[0073] The LLM decoder 110 uses the embedded spatial feature data e ν ｓｐａｔｉａｌ , embedded data of time features e ν ｔｅｍｐｏｒａｌ , embedded data e related to ROI-compatible subregion and related regions ＲｏＩ ０ and embedded data e ＲｏＩ １ ~e ＲｏＩＬ , and embedded data e indicating the question ｔ The LLM decoder 110 receives the input. Based on the input data, it generates an answer A to the question regarding the subject specified in the video and visual prompt.

[0074] Figure 4 shows an overview of the related region estimation process performed by the related region estimation unit. Next, we will refer to Figure 4 to provide a summary of the overall related region estimation process performed by the related region estimation unit. Figure 4 also shows the data used in each process. Each data point will be explained using the name shown in the figure.

[0075] The preprocessing unit 161 generates a cropped image 32 by cutting out the region indicated by the ROI-corresponding partial region R from the image 31 of each frame included in the video V.

[0076] The visual encoder 162 calculates the subregion feature quantities of the ROI-corresponding subregion from the cropped image 32. The visual encoder 162 also calculates the overall feature quantities of each frame from the image 31 of each frame.

[0077] The subregion projector 163 performs a conversion process on the ROI-compatible subregion to generate subregion feature quantities 33.

[0078] The overall projector 164 performs a conversion process on each frame to generate the overall feature vector 34.

[0079] The synthesis unit 165 generates a composite feature by performing matrix integration of the subregion feature 33 and the overall feature 34.

[0080] The normalization unit 166 performs a normalization process on the composite feature quantities.

[0081] The decoding unit 167 performs a decoding process on the normalized composite features to generate a relevance attention map 35.

[0082] The region generation unit 168 generates related region information 36, which indicates related regions for each frame, from the relatedness attention map 35.

[0083] Figure 5 shows an example of the process for generating related region information. Here, we will explain the case where an ROI-corresponding subregion 210 is extracted in a specific frame 200.

[0084] The related region estimation unit 106 generates a relevance attention map 201 of the ROI-corresponding subregion 210 from a specific frame 200 and the ROI-corresponding subregion 210. For example, in the relevance attention map 201, darker colors indicate a higher degree of relevance. In this case, the related region estimation unit 106 generates a relevance attention map 201 showing the related regions 211 to 214 for the ROI-corresponding subregion 210.

[0085] The related area estimation unit 106 then generates related area information indicating two related areas with a high degree of relevance from among the related areas 211 to 214 shown in the relevance attention map 201. In the case of Figure 5, the related area estimation unit 106 generates BBox, which indicates related areas 211 and 212, as related area information. In this case, related areas 213 and 214 are not considered related areas because their degree of relevance is low.

[0086] Figure 6 is a diagram illustrating the learning of the estimation module in the related domain estimation unit. Next, the learning of the estimation module 160 will be explained with reference to Figure 6.

[0087] In order for the estimation module 160 to generate training data to be used for learning, a training data synthesis process is performed (step S1). The training data synthesis process may be performed when training the estimation module 160, or it may be performed in advance. The training data synthesis process is performed in the following procedure. Here, we will explain the case where the training data synthesis process is performed using a training information processing device (not shown) different from the VQA device 1.

[0088] The learning information processing device acquires learning data 300. The learning data 300 includes an image 301 of a specific frame of a video and its scene graph 302. The scene graph 302 includes multiple BBoxes 321 and multiple relational elements 322. The scene graph 302 is data that graphically represents the relationships between objects, including people, in the image.

[0089] Figure 7 is a diagram of a scene graph. Here, we will briefly explain the scene graph with reference to Figure 7.

[0090] For example, in image 330 of Figure 7, there are objects such as a man represented by BBox331, a box represented by BBox332, a man represented by BBox333, a woman represented by BBox334, and a shelf represented by BBox335. The scene graph 350 shown in Figure 7 is data that shows these objects and the relationships between them.

[0091] Scene graph 350 includes relation 341, which shows the relationship between man 351, represented by BBox 331, and box 352, represented by BBox 332. Relation 341 indicates that man 351 is holding box 352. Similarly, scene graph 350 includes relation 342, which shows the relationship between man 351 and man 353, represented by BBox 333, and relation 344, which shows the relationship between man 351 and shelf 355, represented by BBox 335. Furthermore, scene graph 350 includes relation 343, which shows the relationship between box 352 and woman 354, represented by BBox 334. That is, man 351 is related to woman 354 through box 352.

[0092] Here, scene graph 350 focuses on the man shown in BBox 331, but from the same image 330, there are also scene graphs that focus on other objects such as the box shown in BBox 332 and the man shown in BBox 333.

[0093] Returning to Figure 6, let's continue the explanation. The scene graph 302 of the training data 300 is the scene graph of image 301. Each of the multiple objects contained in image 301 corresponds to a different object. Relation 322 indicates the relationship between each object shown in Relation 321. Here, in the scene graph 302, objects that are directly or indirectly connected by Relation 322 are considered to have a "relationship." Furthermore, the fewer Relation 322s connecting the objects, the higher the "degree of relationship."

[0094] The learning information processing device then selects a region of interest from among the BBox321. The learning information processing device sequentially selects each object corresponding to the BBox321 as a region of interest. Then, the learning information processing device selects a related region corresponding to the region of interest (step S11). Based on this selection of related regions, the learning information processing device generates a focus BBox304 representing the region of interest and a related BBox303 representing the related region for the region of interest.

[0095] Next, the learning information processing device performs a synthesis process on the attention BBox 304 and related BBox 303 based on the degree of relevance obtained from the scene graph 303 to generate a degree of relevance attention map 305 that displays related regions according to the degree of relevance (step S12).

[0096] Figure 8 shows an example of relevance. Figure 8 shows the relevance region and relevance when focusing on the man shown in BBox 331 in Figure 7. In this case, BBox 331 is the region of focus. BBoxes 332 to 335 are all relevance regions to the region of focus. The objects shown in BBoxes 332, 323, and 325 are all connected to the man shown in BBox 331 by one relation 341, 342, or 344, so their relevance is high. In contrast, the woman shown in BBox 334 is connected to the man shown in BBox 331 via two relations, 341 and 343, so her relevance is medium. In this way, the relevance is obtained from the scene graph 303, so the learning information processing device can generate a relevance attention map 305 using the relevance obtained from the scene graph 303.

[0097] Furthermore, the learning information processing device may generate a relevance attention map 305 by applying attention, which is information indicating the degree of relevance, to the regions of the related BBox 303 using a Gaussian distribution or similar method. The learning information processing device can generate a more natural relevance attention map 305 by applying attention using a Gaussian distribution rather than uniformly considering all regions of BBox 303 as related. A more natural relevance attention map 305 is thought to contribute to improving the accuracy of the VQA device 1's responses.

[0098] Returning to Figure 6, the explanation continues. The learning information processing device can generate a relevance attention map 305, with each BBox 321 corresponding to each object in image 301 as a region of interest. Furthermore, since the video contains multiple frames, the learning information processing device can obtain image 301 for each frame.

[0099] For training the estimation module 160, the image 301 used to generate the relevance attention map 305 and the cropped image 306 generated by cutting out the attention BBox 304 from the image 301 are used.

[0100] The visual encoder 162 and the partial region projector 163 calculate partial region features from the cropped image 306. The visual encoder 162 and the whole region projector 164 also calculate partial region features from the cropped image 306. The partial region features and whole region features are then combined in the synthesis unit 165 and normalized in the normalization unit 166. The normalized features are then decoded in the decoding unit 167 to generate the attention map 307.

[0101] The learning execution unit 310 adjusts the parameters of the estimation module 160. The learning execution unit 310 may be located in the VQA device 1, in the learning information processing device described above, or in any other computer.

[0102] The learning execution unit 310 compares the relevance attention map generated by the estimation module 160 with the training data relevance attention map 305 and performs loss calculations such as the mean squared error (MSE) loss. Subsequently, the learning execution unit 310 adjusts the parameters of the estimation module 160 according to the calculated error. For example, the learning execution unit 310 adjusts the parameters of the partial region projector 163, the whole projector 164, and the decoding unit 167 to reduce the error. With the above steps, the learning of the estimation module 160 can be performed.

[0103] In this embodiment, the description was given as a case where a learning information processing device (not shown), different from the VQA device 1, is used to perform the synthesis of training data. However, the synthesis of training data and the training process may also be performed by the VQA device 1. That is, the VQA device 1 generates a relevance attention map of training data based on a dataset that includes each image of multiple frames contained in a predetermined video and a scene graph for each image. Then, the VQA device 1 performs training of the estimation module 160 based on the generated relevance attention map of training data.

[0104] Figure 9 is a diagram illustrating the learning process for question-and-answer processing using an LLM decoder. In the learning of the LLM decoder 110, the spatial and temporal features of the entire specific video, and the features of the ROI-corresponding subregion and related region in the specific video become explanatory variables. The answers to the questions become the target variables. The learning execution unit 14 then adjusts the parameters of the LLM decoder 110 based on the error information between the prediction results when the explanatory variables are input to the LLM decoder 110 and the target variables, thereby causing the LLM decoder 110 to learn.

[0105] Figure 10 is a flowchart of the question and answer processing by the VQA device according to the embodiment. Next, the flow of the question and answer processing by the VQA device 1 according to the embodiment will be explained with reference to Figure 10.

[0106] The user terminal device 3 receives instructions from the user and inputs the video output from the video output device 2, the visual prompts set by the user, and the text prompts indicating questions to the VQA device 1 (step S101).

[0107] The visual encoder 101 extracts the overall feature quantities for each frame of the video. The spatiotemporal feature calculation unit 102 calculates the spatial and temporal feature quantities of the entire video based on the temporal and spatial relationships of each object in the video for each frame (step S102).

[0108] The specified region extraction unit 104 extracts a portion of the image indicated by a visual prompt for the selected frame as an ROI. The ROI tracker 105 tracks the ROI specified by the visual prompt across the entire video and extracts the ROI-corresponding portion (step S103).

[0109] The related region estimation unit 106 estimates the related region of the ROI-corresponding subregion for each frame and generates related region information (step S104).

[0110] The sub-region feature calculation unit 107 calculates the features of the ROI-corresponding sub-region and the features of the related region (step S105).

[0111] The selection unit 108 selects features from the features of the ROI-corresponding subregion and the features of the related region by thinning out similar features (step S106).

[0112] The overall projector 103 calculates embedded data for spatial and temporal features of the entire image. The projector 109 calculates embedded data for features selected from the features of the ROI-corresponding subregion and related regions. The embedding unit 112 calculates embedded data for text prompts that have undergone text conversion processing by the text conversion unit 111 (step S107).

[0113] The LLM decoder 110 receives embedded data of spatial and temporal features of the entire video, embedded data of features selected from the ROI-corresponding subregions and related regions, and embedded data of text prompts. The LLM decoder 110 then generates an answer to the question written in the text prompt (step S108). After that, the LLM decoder 110 transmits the answer to the user terminal device 3.

[0114] Figure 11 is a flowchart of the related region estimation process by the related region estimation unit according to the embodiment. The flowchart in Figure 11 is an example of the process performed in step S104 of Figure 10. Next, referring to Figure 11, the flow of the related region estimation process by the related region estimation unit 106 according to the embodiment will be explained.

[0115] The preprocessing unit 161 selects one frame from which a ROI-corresponding subregion has been extracted (step S111).

[0116] Next, the preprocessing unit 161 cuts out the ROI-corresponding region from the image of the selected frame to generate a cropped image (step S112).

[0117] The visual encoder 162 calculates partial region features from the cropped image (step S113).

[0118] The sub-region projector 163 performs a transformation process on the sub-region features to make it easier to compare the sub-region features with the overall features (step S114).

[0119] Furthermore, the visual encoder 162 calculates the overall feature vector from the image of the selected frame (step S115).

[0120] The overall projector 164 performs a transformation process on the overall feature vector to make it easier to compare the subregion feature vectors with the overall feature vectors (step S116).

[0121] The synthesis unit 165 performs matrix calculations on the subregion features and the overall features, and synthesizes the subregion features and overall features to calculate the synthesized features (step S117).

[0122] The normalization unit 166 performs normalization on the composite features (step S118).

[0123] The decoding unit 167 performs decoding on the normalized composite features to generate a relevance attention map (step S119).

[0124] The region generation unit 168 generates related region information indicating related regions for each frame from the relatedness attention map (step S120).

[0125] The preprocessing unit 161 determines whether the extraction of related regions has been completed for all frames from which ROI-corresponding partial regions have been extracted (step S121).

[0126] If there are frames for which related regions have not been extracted (step S121: negative), the preprocessing unit 161 returns to step S111. Conversely, if the extraction of related regions for all frames is completed (step S121: positive), the related region estimation unit 106 terminates the related region estimation process.

[0127] Figure 12 shows an example of a factory image with visual prompts set. Next, with reference to Figure 12, a specific example of question and answer by the VQA device 1 will be explained. The following scenarios can be considered as use cases for question and answer performed by the VQA device 1. Here, we consider the case where the image 400 shown in Figure 12 and the visual prompt indicating ROI 401 are input.

[0128] Users may ask questions like the following to confirm the details of the work performed on an individual basis. These questions are entered into the VQA device 1 as text prompts containing their content.

[0129] For example, a user might input a question into VQA device 1 such as, "Please summarize the work performed by the person in question in the video." In this case, VQA device 1 would respond with something like, "After observing the surrounding workers, the person helps with the loading of cargo... (omitted)." Alternatively, a user might input a question into VQA device 1 such as, "Does the person in question have any contact with forklift operators?" In this case, VQA device 1 would respond with something like, "There is a scene where the person helps with the loading of cargo." Through these questions, users can confirm the work performed by the person they are interested in.

[0130] Furthermore, users may perform safety management by asking questions such as the following. These questions, along with their content, are input to the VQA device 1 as text prompts.

[0131] For example, a user might input a question into VQA device 1 such as, "Are there instances where equipment such as forklifts are approaching the worker in question?" In this case, VQA device 1 would respond, "There have been three such instances." The user might also input a question into VQA device 1 such as, "What kinds of dangerous situations could occur to the person in question?" In this case, VQA device 1 would respond, "Collisions with equipment, objects falling from shelves, etc. are possible." Furthermore, the user might input a question into VQA device 1 such as, "Please list areas for improvement to ensure the safety of the person in question." In this case, VQA device 1 would respond, "Clearly define the paths for forklifts. Communication from those around is also important." Through these questions, users can implement safety management in the factory.

[0132] In this way, the VQA device 1 can perform the following processing. The VQA device 1 acquires video footage in which a person has been filmed and questions about events that pose a risk to the person and are contained in a first region. Next, the VQA device 1 analyzes the video footage to generate identification results that identify the region containing the person in the video, the region containing the object in the video, and the type of relationship between the person and the object. Based on the generated identification results, the VQA device 1 identifies the first region where the person is located. Next, based on the identification results, the VQA device 1 identifies a second region where the object surrounding the person is located. Next, the VQA device 1 inputs questions about the events and prompts containing image features of the first and second regions into a large-scale multimodal model to generate information about events that pose a risk to the person as answers to the questions about the events.

[0133] Furthermore, the VQA device 1 can perform the following processing. The VQA device 1 acquires the video in which multiple objects are captured and the question concerning events related to the objects included in the first region. The VQA device 1 then generates temporal or spatial image features using the estimation results of the second region for each of the multiple video frames. Based on the question and the temporal or spatial image features, the VQA device 1 then generates an answer to the question concerning events that occurred due to temporal or spatial changes in objects related to the objects included in the first region among the multiple objects present in the multiple video frames.

[0134] When generating answers to questions like those described above, it is important to consider the relationship with other objects displayed on the screen. In this regard, the VQA device 1 according to this embodiment can decompress the image while considering the relationship with other objects on the screen, thereby improving the accuracy of the answers.

[0135] As described above, the VQA device 1 according to this embodiment tracks the designated ROI for all frames to extract the ROI-corresponding region, and extracts related regions in each frame that are related to the ROI-corresponding region and have a high degree of relevance. The VQA device 1 then generates an answer using the spatial and temporal features of the entire video, the features of the ROI-corresponding region, and the features of the related regions.

[0136] In this way, the VQA device 1 can automatically capture surrounding information related to its designated target and provide it to the LMM. This allows the VQA device 1 to consider not only the temporal and spatial changes in importance of the entire video and the important changes of the target of interest, but also the important changes of related objects such as people and objects that are highly relevant to the target of interest. Therefore, its ability to understand images and videos can be improved. Furthermore, by considering the important changes of related objects such as people and objects that are highly relevant to the target of interest when conducting question and answer sessions, the accuracy of the VQA responses can be improved.

[0137] (Hardware Configuration) Figure 13 is a hardware configuration diagram of the VQA device. Next, an example of a hardware configuration for realizing each function of the VQA device 1 will be described with reference to Figure 13.

[0138] As shown in Figure 13, the VQA device 1 includes, for example, a CPU (Central Processing Unit) 91, memory 92, a hard disk 93, and a network interface 94. The CPU 91 is connected to the memory 92, the hard disk 93, and the network interface 94 via a bus.

[0139] The network interface 94 is an interface for communication between the VQA device 1 and external devices. For example, the network interface 94 relays communication between the video output device 2 or the user terminal device 3 and the CPU 91.

[0140] The hard disk 93 is an auxiliary storage device. The hard disk 93 stores various programs, including the programs described below. For example, the hard disk 93 stores programs for realizing the functions of the visual encoder 101, the spatiotemporal feature calculation unit 102, and the overall projector 103, as illustrated in Figure 1. Also, for example, the hard disk 93 stores programs for realizing the functions of the specified region extraction unit 104, the ROI tracker 105, the related region estimation unit 106, the partial region feature calculation unit 107, the selection unit 108, the projector 109, the LLM decoder 110, the text conversion unit 111, and the embedding unit 112.

[0141] Memory 92 is the main memory. Memory 92 can be, for example, DRAM (Dynamic Random Access Memory).

[0142] The CPU 91 reads various programs from the hard disk 93, loads them into memory 92, and executes them. This allows the CPU 91 to implement the functions of the visual encoder 101, the spatiotemporal feature calculation unit 102, and the overall projector 103, as illustrated in Figure 1. Furthermore, the CPU 91 implements the functions of the specified region extraction unit 104, the ROI tracker 105, the related region estimation unit 106, the partial region feature calculation unit 107, the selection unit 108, the projector 109, the LLM decoder 110, the text conversion unit 111, and the embedding unit 112, as illustrated in Figure 1.

[0143] 1 VQA device 2 Video output device 3 User terminal device 101 Visual encoder 102 Temporal and spatiotemporal feature calculation unit 103 Overall projector 104 Specified region extraction unit 105 ROI tracker 106 Related region estimation unit 107 Partial region feature calculation unit 108 Selection unit 109 Projector 110 LLM decoder 111 Text conversion unit 112 Embedding unit 160 Estimation module 161 Preprocessing unit 162 Visual encoder 163 Partial region projector 164 Overall projector 165 Synthesis unit 166 Normalization unit 167 Decoding unit 168 Region generation unit

Claims

1. An information processing program characterized by: acquiring video footage of a target being monitored; identifying a first region in a predetermined video frame among a plurality of video frames contained in the acquired video where a first object is located, and a question concerning the first object present in the first region; analyzing the acquired video to identify a second object related to the first object present in the first region among a plurality of objects present in each of the plurality of video frames; and causing a computer to perform a process to generate an answer to the question based on the question concerning the first object, the image features of the first object and the second object.

2. The information processing program according to claim 1, characterized in that the process for identifying the first region and the question includes an operation to specify the first region in which the first object is located within the predetermined video frame displayed on the display screen, and a process for receiving a question document from the user regarding the first object located in the first region, and identifying the question based on the question document; the process for generating the answer includes a process for generating an answer to the question by inputting a prompt including the question, the first object and the second object image features into a large-scale multimodal model; and the computer further causes the computer to perform a process for displaying the answer to the question on the display screen.

3. The information processing program according to claim 1, characterized in that the process for identifying the second object includes searching the first region for each of the plurality of video frames and identifying the second region containing the second object for each video frame.

4. The information processing program according to claim 1, characterized in that the computer is further instructed to perform a process of selecting a plurality of image features from the image features of the first object and the second object using the K-means method, and the process of generating the answer includes a process of generating the answer based on the plurality of selected image features.

5. The information processing program according to claim 4, wherein the process of having the computer further perform a process to calculate spatial or temporal image features of the video, and generating an answer to the question, includes a process of calculating the spatial or temporal image features, the selected plurality of image features, and the respective embeddings of the question, and inputting the calculated embeddings into a large-scale multimodal model to generate the answer.

6. The information processing program according to claim 1, characterized in that the process for identifying the second object includes a process for identifying the second object based on the attention map generated by an estimation module, which generates an attention map that displays the surrounding area associated with the first object according to the strength of the association.

7. The information processing program according to claim 6, characterized in that it generates a relevance attention map of training data based on a dataset including each image of multiple frames contained in a predetermined video and a scene graph for each image, and further causes the computer to perform a process of training the estimation module based on the generated relevance attention map of training data.

8. The information processing program according to claim 1, wherein the process for identifying the first region and the question includes obtaining the video in which the person is filmed and the question concerning an event that poses a risk to the person and is included in the first region, and analyzing the video to generate an identification result that identifies the region in the video containing the person, the region in the video containing an object, and the type of relationship between the person and the object, and based on the generated identification result, the process for identifying the first region where the person is located; the process for identifying the second object includes identifying the second region where the object surrounding the person is located, based on the identification result; and the process for generating the answer includes inputting the question concerning the event and a prompt including image features of the first region and the second region into a large-scale multimodal model to generate information concerning an event that poses a risk to the person as an answer to the question concerning the event.

9. The information processing program according to claim 1, wherein the process for identifying the first region and the question includes obtaining the video in which a plurality of objects are captured and the question relating to an event concerning an object included in the first region, and causing the computer to further perform a process for generating temporal or spatial image features using the estimation results of the second region for each of the plurality of video frames, and the process for generating the answer includes, based on the question and the temporal or spatial image features, generating the answer relating to an event caused by a temporal or spatial change of an object among the plurality of objects present in the plurality of video frames that is related to the object included in the first region.

10. An information processing method characterized in that an information processing device acquires video of a subject to be monitored, identifies a first region in which a first object is located within a predetermined video frame among a plurality of video frames constituting the acquired video, and identifies a question concerning the first object located in the first region, and analyzes the acquired video to identify a second object related to the first object located in the first region among a plurality of objects having each of the plurality of video frames, and performs a process to generate an answer to the question based on the question concerning the first object, the image features of the first object and the second object.

11. An information processing device comprising: a region identification unit that acquires video of a monitored object and identifies a first region in which a first object is located within a predetermined video frame among a plurality of video frames constituting the acquired video; a text conversion unit that identifies a question relating to the first object that exists in the first region; a related region estimation unit that analyzes the video and identifies a second object that is related to the first object that exists in the first region among a plurality of objects that each of the plurality of video frames has; and an answer generation unit that generates an answer to the question based on the question relating to the first object, the image features of the first object and the second object.

Images