Information processing program, information processing method, and information processing device

Abstract

This information processing device: acquires a video to be monitored; analyzes a video frame constituting the video to extract, for each type of context indicating attributes of objects included in the video frame constituting the video and / or the relationship between the objects, a first feature amount related to the context from the video frame; executes, on the basis of information on a topic of interest related to the video, sampling processing on the extracted first feature amount related to the context; and outputs, on the basis of the first feature amount on which the sampling processing has been executed, an answer to a request related to the video.

Description

Information processing program, information processing method, and information processing device.

[0001] This invention relates to an information processing program, an information processing method, and an information processing apparatus.

[0002] In recent years, Visual Question Answering (VQA) has become known as a task that generates answers from images and natural language questions. For example, VQA performs the task of obtaining answers to questions by inputting compressed features obtained by compressing features acquired from video and the question text related to the video into a Large Language Model (LLM).

[0003] For example, known feature compression techniques used in VQA include MA-LMM and Flash-VStream. MA-LMM is a compression technique that merges the features of adjacent frames by taking the weighted average of the frames with the highest cosine similarity. Flash-VStream is a compression technique that comprehensively preserves the features of long-duration video by utilizing multiple memory mechanisms with different granularities and lengths.

[0004] Patent No. 7509972

[0005] However, with the above compression techniques, the accuracy of the VQA output may be low for long video durations, such as one hour.

[0006] For example, MA-LMM extracts information based on the magnitude of change between adjacent frames by simply compressing the images contained in the video in a spatiotemporal manner. As a result, even important information may be significantly smoothed out, potentially leading to the loss of crucial details. Furthermore, Flash-VStream mimics human memory, emphasizing longer or more frequent scenes while neglecting shorter ones. Therefore, even important scenes may not be emphasized if they are sudden or brief. Thus, to obtain high-precision results with the above compression techniques, understanding a video of approximately 10 minutes is the limit.

[0007] Furthermore, even if a task is performed in which compressed features obtained from video and a request for the video are input into LLM, for example, the accuracy of the response to the request will be low.

[0008] In one aspect, the objective is to provide an information processing program, information processing method, and information processing device that can improve the accuracy of responses to video-related requests.

[0009] In the first proposal, the information processing program is characterized by causing a computer to perform the following processes: acquire video of a monitored object, analyze the video frames constituting the video, extract a first feature quantity related to the context from the video frames for each type of context indicating the attributes of objects and / or relationships between objects that the video frames constituting the video have, perform a sampling process on the extracted first feature quantity related to the context based on information on topics of interest related to the video, and output a response to a request related to the video based on the first feature quantity after the sampling process.

[0010] According to one embodiment, the accuracy of responses to video-related requests can be improved.

[0011] Figure 1 is a diagram illustrating the information processing device according to Embodiment 1. Figure 2 is a diagram comparing general processing with the processing of the information processing device according to Embodiment 1. Figure 3 is a diagram illustrating the detailed processing of the information processing device according to Embodiment 1 (before question input). Figure 4 is a diagram illustrating the detailed processing of the information processing device according to Embodiment 1 (when question input). Figure 5 is a functional block diagram showing the functional configuration of the information processing device according to Embodiment 1. Figure 6 is a diagram illustrating encoder training. Figure 7 is a diagram illustrating an example of compression mechanism training. Figure 8 is a diagram illustrating an example of first topic extraction mechanism training. Figure 9 is a diagram illustrating an example of second topic extraction mechanism training. Figure 10 is a diagram illustrating the context feature extraction process. Figure 11 is a diagram illustrating structured context features. Figure 12 is a diagram illustrating the initial generation process. Figure 13 is a diagram illustrating the processing when a question is input. Figure 14 is a diagram illustrating the updating of context features when a new question is input. Figure 15 is a diagram illustrating the answer output for a question. Figure 16 is a flowchart of the processing flow. Figure 17 is a diagram illustrating an example of hardware configuration.

[0012] The following describes in detail, with reference to the drawings, embodiments of the information processing program, information processing method, and information processing apparatus according to the present invention. However, the present invention is not limited by these embodiments. Each embodiment can be combined as appropriate within a non-consistent range.

[0013] (Description of Information Processing Device 10) Figure 1 is a diagram illustrating the information processing device 10 according to Embodiment 1. As shown in Figure 1, the information processing device 10 is an example of a computer device that inputs video data (hereinafter sometimes simply referred to as video) and question texts, which are requests regarding the video, into a VQA and outputs answers to the question texts. The question texts are, for example, text-based input documents related to the video.

[0014] Here, we will explain general VQA processing and the VQA processing performed by the information processing device 10 shown in Figure 1. Figure 2 is a diagram comparing general processing with the processing of the information processing device 10 according to Example 1. As shown in Figure 2(a), in general processing, when video and question text are input, feature quantities are obtained from the video and compressed, and the compressed feature quantities and question text are input to the LLM to obtain the answer to the question text. MA-LMM or Flash-VStream are used for compression.

[0015] In contrast, the information processing device 10 according to Embodiment 1, upon receiving video and a question, acquires feature quantities from the video and compresses them into feature quantities based on context, which represents the attributes of objects in each frame of the video and / or the relationships between objects, and is necessary (information to be retained) to answer the user's question. On the other hand, the information processing device 10 selects a topic of interest (important topic) based on the content of the question. Then, using the topic of interest, the information processing device 10 samples the compressed feature quantities and inputs them into an LLM composed of the sampled feature quantities and the question to obtain an answer to the question.

[0016] As shown in Figure 2(b), for example, the information processing device 10 according to Embodiment 1 takes a question as input and selects a topic of interest based on the content of the input question. The information processing device 10 also takes a video as input and acquires the feature quantities of the input video. The information processing device 10 then compresses the feature quantities based on context. Next, the information processing device 10 samples feature quantities using the selected topic of interest and the compressed feature quantities. After that, the information processing device 10 generates an answer to the question by inputting a prompt, which consists of the sampled feature quantities and the input question, into the LLM.

[0017] LLM is, for example, a pre-trained language model composed of a neural network. Furthermore, LLM is a language model that learns using large computational resources, data, and parameters, and then takes natural language as input, executes the learned processing, and returns a response.

[0018] Thus, the information processing device 10 according to Embodiment 1 can perform VQA (Video Quality Analysis) that appropriately recognizes and selects context even for long-duration videos of one hour or more, by compressing video information using context as the basis for compression decisions. Furthermore, even if the monitored video is long-duration, the information processing device 10 can appropriately output answers to questions about events that occurred within the monitored video, thereby improving the accuracy of the answers.

[0019] (Explanation of the processing of the information processing device 10) Next, the processing of the information processing device 10 according to Example 1 will be explained. Figure 3 is a diagram illustrating the detailed processing of the information processing device according to Example 1 (before question input), and Figure 4 is a diagram illustrating the detailed processing of the information processing device according to Example 1 (when question input is performed).

[0020] As shown in Figure 3, the information processing device 10 inputs each frame of the video (video frame) into an encoder, extracts the visual features of each video frame, and stores each visual feature. Subsequently, the information processing device 10 inputs each visual feature into a compression mechanism, such as an autoencoder, and extracts contextual features from each visual feature.

[0021] The information processing device 10 then inputs the feature quantities of each context into a first topic extraction mechanism, which is a mechanism that predicts, extracts, and prioritizes objects and topics that may be the subject of questions, such as site characteristics and people involved, to generate topics of interest (hereinafter sometimes simply referred to as topics) and stores them in a topic bank. Subsequently, the information processing device 10 performs sampling to extract features corresponding to topics from the feature quantities of the contexts and stores the sampled feature quantities of the contexts in a memory bank.

[0022] In other words, during the initial video input stage, since no questions have been entered, the information processing device 10 extracts candidate topics that appear to be important from the video alone and generates the initial state of the topic bank using Topic extraction, which is an example of a first topic extraction mechanism. Furthermore, the information processing device 10 stores information that has a high degree of relevance to the feature quantities of the topic bank in the memory bank, such as when the number of frames exceeds the memory bank length.

[0023] Subsequently, when a question is input, as shown in Figure 4, the information processing device 10 inputs the question into an analysis mechanism to decompose it into morphemes. Next, the information processing device 10 inputs the obtained morphemes into a second topic extraction mechanism, which extracts the objects and topics that are currently the subject of the question from the question and updates the topic bank, to extract topics. Then, the information processing device 10 inputs the extracted topics into a first conversion mechanism, which is an example of a projector that performs shape conversion to topics to be stored in the topic bank, and updates the topic bank with the shape-converted topics.

[0024] Subsequently, the information processing device 10 performs sampling to extract features corresponding to the topics in the updated topic bank, and stores the sampled context features in the memory bank. In other words, the information processing device 10 can update (regenerate) the memory bank using the stored (stocked) image features. The information processing device 10 can also establish update criteria for situations such as when a question on a previously unexplored topic is input. Furthermore, the information processing device 10 extracts context features that have a high degree of relevance to the top K topics (K is any number) in the topic bank.

[0025] The information processing device 10 then repeats the process shown in Figure 4 each time a question is input. When the question is finished, the information processing device 10 inputs the morphemes obtained from the question into a second conversion mechanism (Embedding) that converts it into an input format for LLM, and converts it into a numerical vector. Similarly, the information processing device 10 inputs the context features stored in the memory bank into a projector and converts (restores) them into features (visual embeddings) that the LLM can understand. After that, the information processing device 10 inputs the numerical vector of the question and the features (visual embeddings) into the LLM and outputs the answer obtained.

[0026] In this way, the information processing device 10 performs feature compression based on context and extracts candidate important topics from the video information. Subsequently, the information processing device 10 updates the topics of interest according to the content of the questions that are input each time, and performs information compression or extracts information from stored video features based on the topics of interest, updating the memory so that it contains more information related to the topics of interest. After that, the information processing device 10 restores the compressed features so that the LLM can understand them and inputs them into the LLM.

[0027] Therefore, the information processing device 10 can store important information in long-duration video based on the context of the video and the content of the questions, and can also compress the features, thereby improving the accuracy of the VQA output.

[0028] (Functional Configuration) Figure 5 is a functional block diagram showing the functional configuration of the information processing device 10 according to Embodiment 1. As shown in Figure 5, the information processing device 10 has a communication unit 11, a first storage unit 12, and a control unit 20.

[0029] The communication unit 11 is a processing unit that controls communication with other devices, and can be implemented, for example, by a communication interface. For example, the communication unit 11 may receive video from a camera or accept input of a question from the administrator's terminal. The question may be in text or voice format.

[0030] The first storage unit 12 is a processing unit that stores various data and programs executed by the control unit 20, and is implemented, for example, by a hard disk. This first storage unit 12 has a video DB 13 and a frame DB 14.

[0031] The video database 13 is a database that stores video data, which is the subject of the question. The video database 13 can store video acquired by the communication unit 11, and it can also store video manually stored by the user. The frame database 14 is a database that stores each frame that makes up the video stored in the video database 13.

[0032] For example, the video footage would include scenes of a warehouse work site featuring a forklift driver and workers, such as the driver taking cardboard boxes from shelves to move them with the forklift. Later in the video, workers appear and are shown carrying cardboard boxes by hand, but approaching the forklift due to poor visibility.

[0033] The control unit 20 is the processing unit that manages the entire information processing device 10, and is implemented by a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). This control unit 20 has a second storage unit 30, a pre-processing unit 40, and a question answering unit 50. The second storage unit 30, the pre-processing unit 40, and the question answering unit 50 are implemented by electronic circuits and processes executed by the processor.

[0034] The second memory unit 30 is a memory unit implemented by the processor's memory, such as GPU memory. The second memory unit 30 has a memory bank 31, which is an example of a second memory, and a topic bank 32, which is an example of a first memory. The memory bank 31 is an area for storing context feature quantities, which will be described later, and the topic bank 32 is an area for storing topics, which will be described later.

[0035] The preprocessing unit 40 is a processing unit that trains various mechanisms and learning models constituting the VQA. Also, the preprocessing unit 40 trains the learning models and the like in the order in which data is input in the VQA, and trains the subsequent learning models using the previously trained models, thereby improving the accuracy of the VQA. Here, an example in which the preprocessing unit 40 performs training will be described, but it is also possible to perform training using a different device from the information processing device 10. Also, the information processing device 10 can acquire and use various mechanisms and learning models trained using a different device.

[0036] For example, the preprocessing unit 40 executes training of an encoder that extracts image feature amounts from video frames. FIG. 6 is a diagram for explaining the training of the encoder. As shown in FIG. 6, the preprocessing unit 40 inputs training data in which a video frame as an explanatory variable and an image feature amount as a target variable are associated to the encoder to obtain the output result of the encoder, and executes machine learning of the encoder so that the error between the image feature amount as the target variable and the output result becomes small.

[0037] Also, the preprocessing unit 40 executes training of a compression mechanism (for example, an autoencoder, Compressor) that extracts context feature amounts from the image feature amounts. FIG. 7 is a diagram for explaining a training example of the compression mechanism. Here, an example in which a pre-trained encoder located in the previous stage of the compression mechanism is used will be described.

[0038] As shown in FIG. 7, the preprocessing unit 40 inputs training data having the image feature amount (f) output from the encoder as an explanatory variable and the context feature amount (f') as a target variable to the compression mechanism to obtain an output result (f' ｔ ) and acquires it. Then, the preprocessing unit 40 executes machine learning of the compression mechanism so that the error between the context feature amount (f') as the target variable and the output result (f' ｔ ) becomes small.

[0039] Also, the preprocessing unit 40 executes training of a first topic extraction mechanism (e.g., Topic extraction) that extracts topics such as objects or topics that can be question targets from the feature amounts of the context. FIG. 8 is a diagram for explaining a training example of the first topic extraction mechanism. Here, an example using a learned compression mechanism located in front of the first topic extraction mechanism will be described.

[0040] As shown in FIG. 8, the preprocessing unit 40 inputs training data having the feature amounts (f') of the context output from the compression mechanism as explanatory variables and topics as objective variables to the first topic extraction mechanism to obtain an output result. Then, the preprocessing unit 40 executes machine learning of the first topic extraction mechanism so that the error between the topic as the objective variable and the output result becomes small.

[0041] Also, the preprocessing unit 40 executes training of a second topic extraction mechanism (e.g., Topic extraction) that extracts topics such as objects or topics that can be question targets from the morphological analysis results of the question sentences. FIG. 9 is a diagram for explaining a training example of the second topic extraction mechanism.

[0042] As shown in FIG. 9, the preprocessing unit 40 inputs training data having the morphemes obtained by morphologically analyzing the question sentence as explanatory variables and topics as objective variables to the second topic extraction mechanism to obtain an output result. Then, the preprocessing unit 40 executes machine learning of the second topic extraction mechanism so that the error between the topic as the objective variable and the output result becomes small. Note that the preprocessing unit 40 can also add the priority order of each topic as an objective variable and train the second topic extraction mechanism to output topics and priority values.

[0043] Returning to FIG. 5, the question answering unit 50 is a processing unit that generates an answer to a question about the video using the VQA including various mechanisms and learning models generated by the preprocessing unit 40. This question answering unit 50 includes a feature amount extraction unit 51, an initial generation unit 52, an extraction unit 53, an update unit 54, and an answer output unit 55.

[0044] The feature extraction unit 51 is a processing unit that acquires contextual features from video frames stored in the frame DB 14. Specifically, the feature extraction unit 51 analyzes the video frames that make up the video to identify the type of context that indicates the attributes of objects and / or relationships between objects that the video frames possess. Then, for each type of context, the feature extraction unit 51 extracts contextual features (first features) from the video frames.

[0045] For example, the feature extraction unit 51 pre-stores a list of contexts categorized as "forklift," "worker," and "forklift and worker approaching." The feature extraction unit 51 then performs image analysis on the video frames, extracts frames containing the predetermined context categories, and extracts regions containing the contexts from the extracted frames. The feature extraction unit 51 then inputs the extracted regions into an autoencoder or the like to compress them, thereby extracting the context features from the video frames.

[0046] To illustrate with another example, the feature extraction unit 51 can also extract contextual features using the trained encoder and trained compression mechanism described above. Figure 10 is a diagram illustrating the contextual feature extraction process. As shown in Figure 10, the feature extraction unit 51 inputs each video frame to the encoder and extracts image features (f) for all frames in a format that the LLM can understand. ｔ The feature extraction unit 51 extracts image features (f) from all extracted frames until the answer to the question is output. ｔ The image features (f) for all frames are stored in memory or elsewhere. The feature extraction unit 51 then extracts the image features (f) for all frames. ｔ The context features (f') for all frames, which are in a format that LLM cannot understand, are input to the compression mechanism. ｔ Extract the following:

[0047] Furthermore, the feature extraction unit 51 can also extract contextual features by structuring the information contained in the image features into elements related to information transmission and compressing the structured information while retaining it. Specifically, the feature extraction unit 51 extracts contextual features that retain the information of 5W1H (When, Where, Who, What, Why, How) + Atmosphere.

[0048] Furthermore, the feature extraction unit 51 can also extract feature quantities for one context from one frame, where the element "5W1H (When, Where, Who, What, Why, How) + Atmosphere" is treated as one type. In addition, the feature extraction unit 51 can also extract feature quantities for each type of context from one frame, where the element "5W1H (When, Where, Who, What, Why, How) + Atmosphere" is treated as a separate category.

[0049] Figure 11 illustrates the features of a structured context. As shown in Figure 11, the feature extraction unit 51 extracts information such as time, aisles, and boxes as features from images taken at times T1 and T2 that show a warehouse with shelves and aisles where boxes are stacked. As a result, the feature extraction unit 51 extracts the features of a context that includes "Where = Aisle, When = T1 (T2), Atmosphere = Warehouse with boxes".

[0050] Furthermore, for the images taken at times T3 and T4, which show "a forklift entering the aisle of a warehouse where boxes are stacked," the feature extraction unit 51 extracts information such as the warehouse, the forklift, and the entry into the aisle as features. As a result, the feature extraction unit 51 extracts contextual features that include "What = The forklift is entering the aisle, Where = Aisle, Who = Forklift driver, When = T3 (T4), How = Turning the corner, Atmosphere = Warehouse with boxes, forklift."

[0051] Furthermore, for the image taken at time T5, which shows "a forklift driving down an aisle in a warehouse where boxes are stacked, and workers performing tasks in the warehouse," the feature extraction unit 51 extracts information such as the warehouse, forklift, and workers as features. As a result, the feature extraction unit 51 extracts contextual features that include "What = The forklift is in the aisle and the worker is about to enter, Where = Aisle, Who = Forklift driver, Worker, When = T5, How = The worker is entering the aisle, Atmosphere = Warehouse with boxes, Workers, forklift."

[0052] Furthermore, for the image taken at time T6, which shows "a forklift driving down an aisle in a warehouse where boxes are stacked, and a worker working near the forklift," the feature extraction unit 51 extracts features such as the warehouse, forklift, worker, and the proximity of the forklift and worker. As a result, the feature extraction unit 51 extracts contextual features that include "What = The forklift is entering the aisle, Where = Aisle, Who = Forklift driver, Worker, When = T6, How = The worker is approaching to the forklift, Atmosphere = Machines and people crowded together in the aisle."

[0053] Furthermore, the feature extraction unit 51 can also perform the extraction process using a compression mechanism trained to extract features from a structured context, and can perform the extraction process by extracting predetermined elements or structured contexts through image analysis.

[0054] Returning to FIG. 5, the initial generation unit 52 is a processing unit that stores various data in the topic bank 32 and the memory bank 31 before a question sentence is input. Specifically, before a question sentence, which is an example of an input document related to video, is acquired, the initial generation unit 52 analyzes video frames to generate topic candidates. Then, the initial generation unit 52 extracts the feature amounts of the context from the video frames for each type of context. After that, the initial generation unit 52 executes sampling processing on the feature amounts of the context based on the information related to the topic candidates, and stores the sampled feature amounts of the context in the memory bank 31.

[0055] FIG. 12 is a diagram for explaining the initial generation process. As shown in FIG. 12, the initial generation unit 52 inputs the feature amounts of the context (f´ ｔ ) for all the frames extracted by the feature extraction unit 51 into the learned first topic extraction mechanism, and extracts topic A "forklift movement" of candidate 1, topic B "cardboard movement" of candidate 2, and topic C "action of worker X" of candidate 3. Then, the initial generation unit 52 stores each extracted topic in the topic bank 32. At this time, the initial generation unit 52 can also select a topic with a high priority so that the number is less than the threshold among the plurality of extracted topics.

[0056] Subsequently, the initial generation unit 52 samples the feature amounts of the context (f´ ｔ ) for all the frames using the topic candidates in the topic bank 32, and stores them in the memory bank 31. For example, the initial generation unit 52 extracts the feature amount A1 of the context as the feature amount including topic A, the feature amount whose similarity to topic A is below the threshold, or the feature amount corresponding to topic A from the feature amounts of the context (f´ ｔ ) for all the frames, and stores it in the memory bank 31.

[0057] Similarly, the initial generation unit 52 uses the feature amounts of the context (f´ ｔFrom these, the feature quantities B1, B2, and B3 of the context corresponding to topic B, and the feature quantities C1 and C2 of the context corresponding to topic C are extracted and stored in memory bank 31. The initial generation unit 52 stores them in order of priority. If there is no free space in memory bank 31, the initial generation unit 52 prioritizes storing the feature quantities of the context corresponding to the higher priority topics.

[0058] The extraction unit 53 is a processing unit that, when a question is input, extracts topics from the question and updates the topic bank 32 as necessary. Figure 13 is a diagram illustrating the processing when a question is input. As shown in Figure 13, when the question "Tell me what a forklift does" is input, the extraction unit 53 inputs the question into the analysis mechanism and obtains the morphological analysis result "Forklift, what, doing, tell me".

[0059] The extraction unit 53 then inputs the morpheme "Forklift, what, is, doing, tell me" into the second topic extraction mechanism, which has already been trained, and extracts the topic candidate "Forklifts carry loads". If the topic candidate is a new topic not already stored in topic bank 32, the extraction unit 53 stores it in topic bank 32. If topic bank 32 is updated, the processing by the update unit 54, which will be described later, is executed.

[0060] The update unit 54 is a processing unit that updates the memory bank 31 when a question is input and the topic bank 32 is updated. For example, each time a question is acquired, the update unit 54 updates the topic in the topic bank 32, performs sampling on the context features based on the updated topic, and updates the memory bank 31 with the context features from which the sampling process was performed.

[0061] Figure 14 illustrates the updating of contextual features when a new question is input. As shown in Figure 14, when a new question, "Tell me about instances where a forklift and a worker came close together," is input into the update unit 54, the question is input into the analysis mechanism and the morphological analysis result "Tell me about instances where a forklift, and, a worker, came close together" is obtained.

[0062] The update unit 54 then inputs the morpheme "Tell me about cases where a forklift and a worker approached each other" into the second topic extraction mechanism, which has already been trained, and extracts a new topic candidate, "approach between a forklift and a worker". Since the new topic candidate does not exist in the memory bank 31, the update unit 54 stores the new topic candidate D as new topic D, "approach between a forklift and a worker," in the topic bank 32. At this time, if there is no free space in the topic bank 32, the update unit 54 deletes the topic with the lowest priority, the topic with the oldest storage time, or the topic with the fewest extractions, and then stores the new topic.

[0063] On the other hand, the update unit 54 acquires image features for all frames that have been extracted by the feature extraction unit 51 and stored in memory, and inputs them to the compression mechanism to extract contextual features for all frames.

[0064] The update unit 54 then uses topics D, B, and C stored in the updated topic bank 32 to sample context features for all frames. Specifically, the update unit 54 obtains context features D1 and D2 corresponding to the new topic D and stores them in the memory bank 31. If the memory bank 31 is full, for example, the update unit 54 removes context feature A1 corresponding to the deleted topic A and the oldest stored context feature B1 from the topic bank 32, and stores the context features D1 and D2 corresponding to the new topic D in the topic bank 32. The update unit 54 also rearranges the topics in order of priority.

[0065] In this way, the update unit 54 updates the topic bank 32 each time a new topic with a new question is extracted, and updates the memory bank 31 with the new topic. For example, when a new question with low relevance to a topic stored in the topic bank 32 is entered, the update unit 54 increases the priority of the topic of the question and updates the topic bank 32. Since user questions become more refined over time, prioritizing the updating of the topic bank 32 with topics extracted from the most recent question is effective in improving the accuracy of the answers to user questions.

[0066] The response output unit 55 is a processing unit that outputs an output result for the question based on the feature quantities of the context in which the sampling process was performed. Specifically, the response output unit 55 inputs the feature quantities of the context in which the sampling process was performed and the question to the LLM, generates an output result for the question, and outputs the output result to a display screen such as a display.

[0067] Figure 15 illustrates the output of a response to a question. As shown in Figure 15, the response output unit 55 inputs the latest question, "Tell me about instances where a forklift and a worker came close together," into the analysis mechanism and obtains the morphological analysis result, "Tell me about instances where a forklift, and, a worker, came close together." Subsequently, the update unit 54 inputs the morphological elements "Tell me about instances where a forklift, and, a worker, came close together" into a second conversion mechanism (Embedding) that converts them into an input format for LLM. Subsequently, the update unit 54 obtains the output result (numerical vector) of the second conversion mechanism (Embedding).

[0068] In parallel, the response output unit 55 inputs the "context features D1, D2, B2, C1, B3, C2" stored in the memory bank 31 to the projector and converts the features (f) into an LLM that can be understood. Ｂ ...f Ｅ Convert (restore) to ).

[0069] Subsequently, the information processing device 10 processes the numerical vector of the question and each feature (f Ｂ ...f ＥThe information processing device 10 inputs the following into the LLM and obtains the LLM output result: "There was one instance where a forklift and a worker came into close proximity. The worker was carrying a lot of cargo in their arms and appeared to be lacking attention to the forklift." The information processing device 10 then outputs this output result to a display unit such as a display or sends it to the user's terminal as an answer to the question.

[0070] In other words, the information processing device 10 according to Embodiment 1 acquires video footage of a target to be monitored. Next, the information processing device 10 analyzes the video frames that make up the video and extracts first feature quantities related to the context from the video frames for each type of context that indicates the attributes of objects and / or relationships between objects that the video frames possess. The information processing device 10 also performs sampling on the extracted first feature quantities related to the context based on information on topics of interest related to the video. Then, the information processing device 10 outputs a response to a request related to the video based on the first feature quantities on which the sampling process was performed. After that, the information processing device 10 displays the output results on a display screen.

[0071] Specifically, for example, the information processing device 10 according to Embodiment 1 acquires a video of the monitored object and a question about an event that occurred in the monitored area within the video. Next, the information processing device 10 analyzes the video frames that make up the video and extracts a first feature quantity related to the context from the video frame for each type of context that indicates the attributes of the objects in the video frame and / or the relationships between objects in the video frame. The information processing device 10 also generates information on a topic of interest using the content of the acquired question about the event, and performs a sampling process on the extracted first feature quantity related to the context based on the generated topic of interest information. Then, based on the first feature quantity from which the sampling process was performed and the question about the event, the information processing device 10 generates information about the event that occurred in the monitored area as an answer to the question.

[0072] (Processing Flow) Figure 16 is a flowchart showing the processing flow. As shown in Figure 16, when the user instructs the information processing device 10 to start processing (S101: Yes) and when video is input (S102: Yes), the device extracts image features from each video frame that makes up the video (S103) and stores the image features in memory or the like (S104).

[0073] Next, the information processing device 10 extracts context features from image features (S105), extracts topics from context features, and stores them in the topic bank 32 (S106). Then, the information processing device 10 uses each topic stored in the topic bank 32 to sample context features and stores them in the memory bank 31 (S107).

[0074] Subsequently, when a question is input (S108: Yes), the information processing device 10 extracts morphemes from the question (S109), extracts topics from the morphemes, and updates the topic bank 32 (S110). Next, the information processing device 10 uses each topic stored in the topic bank 32 to sample contextual features and updates the memory bank 31 (S111).

[0075] Subsequently, when a new question is entered (S112: Yes), the information processing device 10 repeats steps S109 onwards. On the other hand, when no new question is entered and the questioning ends (S112: No), the information processing device 10 inputs the "numerical vector of the question" obtained from the latest question and the feature quantities (visual embeddings) obtained from the feature quantities of each context stored in the latest memory bank 31, converted into an input format for the LLM, into the LLM (S113). Then, the information processing device 10 outputs the answer from the LLM (S114).

[0076] (Effects) As described above, the information processing device 10 can compress information without losing any information necessary to answer the questions, even in environments where memory usage is limited and it is necessary to retain a large amount of information, such as in VQA for long-duration video. As a result, the information processing device 10 can retain information while keeping memory usage low. Therefore, the information processing device 10 can improve the accuracy of the VQA output results even for long-duration video, such as one hour.

[0077] Furthermore, the information processing device 10 can structure the image features of a video frame into elements to consider for information transmission (such as 5W1H), and extract compressed features while retaining the structured information. As a result, the information processing device 10 can suppress information loss regarding relationships with people such as workers and objects such as forklifts, and efficiently extract useful information, thereby improving the accuracy of VQA.

[0078] Furthermore, the information processing device 10 extracts contextual features from the video. As a result, the information processing device 10 can extract features that are expected to be important for the answer, even when the question text has not been entered.

[0079] Furthermore, the information processing device 10 updates the topic each time a question is input, and updates the feature quantities of the context that are the input to the LLM. As a result, the information processing device 10 can keep up with refined question sentences, thereby improving the accuracy of the answers to the questions.

[0080] Now, although embodiments of the present invention have been described, the present invention may be implemented in various other forms besides those described above.

[0081] (Numerical values, etc.) The machine learning model, context, topic, features, images, etc. used in the above example are merely examples and can be changed at will. Also, the processing flow explained in each flowchart can be changed as appropriate within a range that does not contradict each other. Note that "forklift" and "worker" are examples of object attributes, and "forklift and worker approaching" is an example of a relationship between objects.

[0082] (System) Unless otherwise specified, the processing procedures, control procedures, specific names, and information including various data and parameters shown in the above documents and drawings may be changed at will.

[0083] Furthermore, the specific forms of distribution and integration of the components of each device are not limited to those shown in the diagram. For example, the pre-processing unit 40 and the question answering unit 50 may be integrated. In other words, all or part of the components may be functionally or physically distributed and integrated in any unit depending on various loads and usage conditions. Moreover, all or any part of the processing functions of each device may be implemented by a CPU and a program executed for analysis by that CPU, or as hardware using wired logic.

[0084] Furthermore, each processing function performed by each device can be implemented, in whole or in part, by a CPU and a program executed for analysis by that CPU, or by hardware using wired logic.

[0085] (Hardware) Figure 17 is a diagram illustrating an example of hardware configuration. As shown in Figure 17, the information processing device 10 includes a communication device 10a, an HDD (Hard Disk Drive) 10b, memory 10c, and a processor 10d. Furthermore, each of the parts shown in Figure 17 is interconnected by a bus or the like.

[0086] The communication device 10a is a network interface card or the like, and communicates with other devices. The HDD 10b stores programs and databases that operate the functions shown in Figure 5.

[0087] The processor 10d operates a process that performs the functions described in Figure 5 by reading a program that performs the same processing as each processing unit shown in Figure 5 from the HDD 10b or the like and loading it into memory 10c. For example, this process performs the same functions as each processing unit of the information processing device 10. Specifically, the processor 10d reads a program that has the same functions as the pre-processing unit 40 and the question answering unit 50 from the HDD 10b or the like. Then, the processor 10d executes a process that performs the same processing as the pre-processing unit 40 and the question answering unit 50.

[0088] Thus, the information processing device 10 operates as an information processing device that executes an information processing method by reading and executing a program. Furthermore, the information processing device 10 can also achieve the same functionality as in the above-described embodiment by reading the program from a recording medium using a media reader and executing the read program. Note that the program referred to in this other embodiment is not limited to being executed by the information processing device 10. For example, the above embodiment may also be applied similarly when another computer or server executes the program, or when they collaborate to execute the program.

[0089] This program may be distributed via a network such as the Internet. Alternatively, this program may be recorded on a computer-readable recording medium such as a hard disk, flexible disk (FD), CD-ROM, MO (Magneto-Optical disk), or DVD (Digital Versatile Disc), and executed by being read from the recording medium by a computer.

[0090] 10 Information Processing Unit 11 Communication Unit 12 First Storage Unit 13 Video DB 14 Frame DB 20 Control Unit 30 Second Storage Unit 31 Memory Bank 32 Topic Bank 40 Pre-processing Unit 50 Question Answering Unit 51 Feature Extraction Unit 52 Initial Generation Unit 53 Extraction Unit 54 Update Unit 55 Answer Output Unit

Claims

1. An information processing program characterized by causing a computer to perform the following processes: acquire video of a monitored object; analyze the video frames constituting the video to extract a first feature quantity related to the context from the video frames for each type of context indicating the attributes of objects and / or relationships between objects that the video frames constituting the video have; perform a sampling process on the extracted first feature quantity related to the context based on information on a topic of interest related to the video; and output a response to a request related to the video based on the first feature quantity on which the sampling process has been performed.

2. The information processing program according to claim 1, characterized in that the extraction process extracts feature quantities from video frames constituting the video, structures the information contained in the extracted feature quantities into elements related to information transmission, and extracts the first feature quantities relating to the context by compressing the structured information while retaining it.

3. The information processing program according to claim 1, characterized in that the request related to the video is a question about an event that occurred in the monitored area within the video, the acquisition process acquires a question about the event, and the output process inputs a prompt composed of the first feature quantity obtained by the sampling process and the question into a large-scale language model to generate information about the event that occurred in the monitored area as an answer to the question, and outputs the generated information about the event that occurred in the monitored area to a display screen.

4. The information processing program according to claim 1, characterized in that the acquisition process includes a process for acquiring an input document relating to the video, the output process includes a process for generating an output result for the input document as a response to the request based on the first feature quantity on which the sampling process was performed and the input document, and the computer further causes the computer to perform a process for generating information on the topic of interest.

5. The information processing program according to claim 4, characterized in that the generation process analyzes the video frames to generate candidate topics of interest before the input document is acquired; the extraction process extracts a first feature quantity related to the context from the video frames for each type of context; and the execution process performs a sampling process on the first feature quantity based on information about the candidate topics of interest.

6. The information processing program according to claim 5, characterized in that, when the input document is acquired, the generation process analyzes the input document to generate the topic of interest, and updates the candidate topic of interest with the generated topic of interest.

7. The information processing program according to claim 4, characterized in that the generation process updates the topic of interest each time the input document is acquired; the extraction process extracts a first feature quantity relating to the context from the video frame each time the input document is acquired; the execution process performs a sampling process on the first feature quantity based on the updated topic of interest information; and the output process generates an output result for the input document based on the first feature quantity on which the sampling process has been performed.

8. The information processing program according to claim 7, characterized in that the extraction process updates the first feature quantity stored in a first memory of the processor, the execution process performs a sampling process on the first feature quantity based on the updated information on the topic of interest, stores the first feature quantity obtained by the sampling process in a second memory of the processor, and the output process generates an output result for the input document based on the first feature quantity stored in the second memory.

9. An information processing method characterized by a computer acquiring video of a target to be monitored, analyzing the video frames constituting the video, extracting a first feature quantity related to the context from the video frames for each type of context indicating the attributes of objects and / or relationships between objects that the video frames constituting the video possess, performing a sampling process on the extracted first feature quantity related to the context based on information on a topic of interest related to the video, and outputting a response to a request related to the video based on the first feature quantity on which the sampling process has been performed.

10. An information processing apparatus comprising a control unit that acquires video of a target to be monitored, analyzes the video frames constituting the video, extracts a first feature quantity related to the context from the video frames for each type of context indicating the attributes of objects and / or relationships between objects that the video frames constituting the video possess, performs a sampling process on the extracted first feature quantity related to the context based on information on a topic of interest related to the video, and outputs a response to a request related to the video based on the first feature quantity on which the sampling process has been performed.

Images