Long video question and answer method based on multi-round reasoning of large model agent

Abstract

The invention relates to a long video question and answer method based on large model agent multi-round reasoning, an initial key frame sequence is determined according to questions and options, confidence scores of the options are generated, and the confidence scores are used for prompting that a large language model can still obtain relatively reliable answers under the condition that input information is insufficient. The positioned key frames are input into a large model agent together with a text context in a grid splicing form to deduce and predict answers. And the large model agent carries out evaluation based on the currently provided information and the answer prediction process, and determines whether the next round of iteration needs to be carried out to supplement missing visual information or not. According to the self-adaptive multi-round reasoning question and answer, different numbers of video frames required by different types of questions are considered, so that the accuracy and the quality of video question and answer are effectively improved.

Description

Technical Field

[0001] The present invention relates to a long video question-answering method based on multi-round reasoning of a large model intelligent agent. The large model intelligent agent evaluates the predicted answer and adaptively determines whether it is necessary to iteratively identify and locate key information to answer the question, belonging to the technical field of video question-answering. Background Art

[0002] Video Question Answering (Video QA) is a multimodal task of computer vision and natural language processing that aims to reason based on video content and predict answers to given questions. Unlike traditional image question answering, Video QA tasks not only need to understand the information of static images, but also must handle dynamic changes in videos, including visual information and their association in the temporal dimension.

[0003] Although Video QA has made good progress in short videos of a few seconds to more than ten seconds, understanding long videos of up to several minutes is still a major challenge in the field of computer vision. This task requires a model that can handle multimodal information, process very long sequences, and effectively reason about these sequences. For a long time, the mainstream solution in the field of video question answering has been end-to-end methods. The main problem with this type of method is that the black box nature of the model leads to poor interpretability. For long video question answering, the model is expected to understand events across different time scales, which is a major challenge for end-to-end models.

[0004] Recently, large language models (LLMs) have demonstrated impressive complex reasoning capabilities on a wide range of tasks. The baseline approach for large language models in video question-answering tasks is to sample a large number of video frames from the video, convert the image information of these frames into text descriptions respectively, and concatenate all the frame description information into a context input into the LLM to predict the answer. However, this approach will result in lengthy contexts and may introduce some noise and interference factors, which will affect the accuracy and reliability of the generated answers. In addition, compared to images, text is a highly condensed representation of information and may miss detailed information on the image, which is not conducive to answering complex visual reasoning questions. Finally, the long video question-answering task needs to solve several challenging problems:

[0005] 1) Locate key information in the video

[0006] Long videos contain a lot of redundant information. Existing models cannot accept complete video frame sequences as input in terms of storage capacity or processing power. Most questions in video question-answering datasets are based on the content of several scene clips in the video. Therefore, how to locate the relevant clips from long video sequences is the primary goal of solving this task. Long videos usually contain more complex sequences of actions and events, requiring stronger temporal modeling capabilities. Previous work uniformly sampled frames or selected frames in a single iteration, which could not locate key information more effectively.

[0007] 2) Limitations of single-round reasoning video question answering

[0008] Traditional video question-answering methods usually generate answers based on a single round of reasoning, and are unable to evaluate the validity of the answers and make corrections. One of the most notable features of large language model agents is their excellent self-reflection ability, that is, after generating answers or performing tasks, they can evaluate and correct their own outputs to improve overall performance and accuracy. The self-reflection ability of large language models has been fully verified in the field of natural language processing. At the same time, compared to getting answers through a single round of reasoning, video question-answering based on multi-round reasoning is more in line with the process of humans quickly locating effective information from long video clips.

[0009] This paper aims to propose an effective solution to the challenging problems in the field of long video question answering mentioned above. BLIP-2 is a multimodal model that effectively integrates the pre-trained visual encoder and language model through two-stage training and cross-modal attention mechanism, significantly improving the performance of image-text understanding tasks. Inspired by the SeViLA method, two modules are designed based on BLIP-2, namely the key frame positioning module F loc and video question and answer module F ans They share the same BLIP-2 structure but store two different copies of Q-Former weights. Q-Former is a lightweight transformer that extracts visual features from a frozen visual model using a learnable set of query vectors. Summary of the invention

[0010] In order to overcome the shortcomings of existing research, the present invention provides a long video question answering method based on multi-round reasoning of a large model agent, which mainly includes the following two stages:

[0011] 1. Generate input for the large language model agent. Locate the initial language-aware keyframe sequence based on the question and options, and calculate the confidence scores of the options. The keyframes combined by the gridding method are used as visual information, and the text context consisting of the question, option confidence scores, and instruction prompts is input into the multimodal large language model.

[0012] 2. The large language model agent iteratively locates key information. The large language model agent is prompted to reason through the thought chain, generate answers and evaluate the answers. Adaptively decide whether to perform the next round of iterative positioning based on the evaluation results of the answers to supplement key visual information. When the confidence of the generated answer is low, the multimodal large language model is prompted to retrieve video clips related to the question and locate key information that helps answer the question, and then update the status to continue the next round of reasoning question and answer.

[0013] The specific steps of a long video question answering method based on multi-round reasoning of a large model agent are as follows:

[0014] Step 1: Fine-tune the keyframe localization module and video question answering module on the downstream dataset; use the pre-trained keyframe localization module F loc to locate key frames and train the video question answering module F on the downstream dataset ans Fine-tune the frozen F based on QA task prompts ans The module generates a frame-level prediction answer for each uniformly sampled frame, and compares the predicted answer with the true answer to obtain a pseudo label as F loc Keyframe annotations of a module to fine-tune the module;

[0015] First, use the pre-trained keyframe module F loc The localized keyframes are used in the downstream dataset for the video question answering module F ans Then, use the frozen F ans Module to train F loc Module. F ans The module predicts the frame-level answer for the input uniformly sampled video frame and compares it with the true answer to generate a binary frame-level pseudo label as F loc Specifically, if F ans The module uses this frame to output the correct answer, and the video frame is marked as a key frame.

[0016] Step 2: Locate the initial key frame sequence;

[0017] The goal of the key frame localization module is to locate the language-aware key frames from a given video frame sequence. The module accepts n video frames uniformly sampled from the original video, and a text context that combines questions, options, and localization prompts as input. A score is generated for each video frame, and the score of each frame indicates the degree of association between the frame and the question. The higher the score, the stronger the association between the video frame content and the question. Finally, the k video frames with the highest scores are selected as the output of the module. In order to ensure that the multimodal large language model can accept a longer video frame sequence, the present invention adopts a grid splicing method to splice the located k key frames into a grid map, and mark the video frame number on each sub-map to retain the timing information between different frames as the visual input content of the large language model agent in the subsequent steps.

[0018] Step 3: Generate confidence scores for options;

[0019] Video Q&A Module F ans The key frame sequence located in step 2 is used as input to generate the confidence score of the option. First, Q-Former Q ans Process key frames and extract query features of key frames. Then, these query features are input into the large language model LLM together with the text context consisting of questions, options and instruction prompts. Finally, this module generates a confidence score for each option. When insufficient information is provided, option confidence can be used to prompt the large language model agent to generate relatively reliable answers.

[0020] Step 4: Generate answers and evaluate their confidence;

[0021] Describe the workflow as based on the current state t The state is the sequence of all video frames that the model can see in the current iteration round. The Multimodal Large Language Model (MLLM) is used as an agent to decide the next action to be performed based on the current state. t : Whether to answer the question directly or continue searching for new information. If you continue searching for new frames, you will retrieve new observations through subsequent steps. t , that is, the retrieved new video frame. Finally, based on the observation value o t To update the status t , we get the input of MLLM in the next iteration. Where t is the iteration number, 1≤t≤T, and T is the highest iteration number.

[0022] Phase 1: Construct the current state. Step 2 obtains the key frames of problem perception, allowing MLLM to have a preliminary understanding of the video content related to the problem. Since the existing multimodal large model cannot process more image sequences at the same time, but can accept images with larger resolutions, the current state is stored by gridding and splicing video frames. At the same time, based on the option confidence generated in step 3, the options are reorganized into the form of "option + confidence score". Subsequently, the grid map containing k frames of key frames is input into MLLM together with the question, options, and instruction prompts.

[0023] The second stage: decide the next action. First, the thought chain prompts MLLM to infer and predict the answer based on the current state and question. And evaluate the confidence of the predicted answer, which is divided into: level 1: completely unreliable, level 2: partially reliable, level 3: completely reliable. If the confidence of the generated answer is high, it means that MLLM believes that the current state is sufficient to predict a reliable answer, so the generated answer is returned as the final answer. Otherwise, the next action is executed to further search for new valid information to supplement the current state. The confidence score is used as the judgment condition for whether to perform multi-round positioning. MLLM adaptively selects the rounds that need to be iterated according to different problems, and then finds the most relevant information at a lower cost.

[0024] Step 5: Identify video clips containing key information;

[0025] In this step, it is necessary to further retrieve visual information related to the question based on the current state. Since some information may appear multiple times in the video, segment-level information retrieval is performed instead of video-level retrieval to enhance temporal reasoning capabilities. First, the k key frames located in step 2 divide the initial video frame sequence into several paragraphs. Then, the MLLM is prompted to locate the paragraph where the video frame that helps answer the question is located by designing instructions. For example, suppose the question is "How does the boy react when the woman takes the measuring spoon from him?" If it is detected that the content of the i-th frame is a woman holding a measuring spoon, it can be inferred that the answer frame corresponding to the question may appear in the video paragraph after the i-th frame.

[0026] Step 6: Locate key frames from the video clip to supplement the visual information.

[0027] Based on the video segment located in step 5, the video frame sequence in the segment is used as the key frame positioning module F loc The input is then used to select the m frames with the highest relevance to the question as supplementary visual information. These supplementary video frames are then integrated with the initial k frames in chronological order. This results in richer visual information, which is then input into the multimodal large language model for the next round of reasoning question answering.

[0028] The step 2 specifically includes: uniformly sampling n frames {f1,...,f n}, through the image encoder E v Extract the visual features of the frame and obtain a set of video frame feature sequences V = {h1,...,h n}, key frame positioning module F loc is to select k language-aware keyframe features from V, compose the question, options and positioning prompts into a text context L, and Q-FormerQ loc The visual query features v are extracted independently from V i The features of the top-k key frames with the highest scores are selected as the output of the module. The k key frames are combined into a grid image, and the number of the current video frame is marked in the upper left corner of each frame to preserve the timing information.

[0029] K=LLM(concat(Q loc (E v (f1,...,f n )),L)),|K|=k<<n (1).

[0030] The step three specifically includes: based on the key frame feature set K obtained in step two, through the Q-former Q ans To generate the key frame query feature v i , the key frame query feature v i By inputting LLM together with the text context, we can get the confidence score s for each option. i , for c options, get the confidence score set S of all options ans ={s1,...,s c} as the output of the current module;

[0031]

[0032] The step 4 specifically includes:

[0033] The confidence score set S obtained in step 3 ans By option i (s i ) is in the form of each option option iAdd confidence scores, combine questions, options, and instruction prompts as text inputs for the multimodal large language model MLLM, and use the key frame splicing image obtained in step 2 as the visual input of MLLM. Use MLLM's multimodal reasoning ability to fuse information from different modalities, find the answer that best fits the question from the options, and prompt MLLM to evaluate based on the currently provided information and answer prediction process. The next steps of the model are:

[0034] Action 1: If the information provided in the current state is sufficient to answer the question, the multimodal large language model MLLM generates the answer to the question and exits the iteration process;

[0035] Action 2: If the information in the current state is not sufficient, it is necessary to continue searching for new information. Then, the MLLM is prompted to infer which video segment contains the necessary information to answer the question based on the current state, and further search for video frame information in that segment;

[0036] Finally, MLLM decides whether to execute Action 1 or Action 2 based on the answer confidence score.

[0037] The step 5 specifically includes: prompting MLLM to divide the original n-frame video frame sequence into several paragraphs based on the k-frame video frame IDs initially provided. MLLM can infer and predict whether the paragraph may contain content related to the question based on the content of the starting frame and the ending frame of the paragraph. Finally, the frame ID sequence S contained in the located video clip is returned.

[0038] The step six specifically includes: using the key frame positioning module F loc Compared with the prior art, the invention has the following advantages:

[0039] The present invention determines the initial key frame sequence based on the question and options, and generates a confidence score for the option. The confidence score is used to prompt the large language model to still obtain a relatively reliable answer when the input information is insufficient. The located key frames are input into the large model agent in the form of grid splicing together with the text context to infer and predict the answer. The large model agent evaluates based on the currently provided information and the answer prediction process, and decides whether the next round of iteration is needed to supplement the missing visual information. The adaptive multi-round reasoning question and answer takes into account the different numbers of video frames required for different categories of questions, effectively improving the accuracy and quality of video question and answer.

[0040] The present invention proposes a long video question-answering method based on multi-round reasoning of a large model agent. Unlike the traditional method of generating answers through a single reasoning, the present invention evaluates the predicted answers through a large model agent to decide whether to perform multiple rounds of reasoning to locate key information to answer the question. The iterative rounds of the problem are determined by an adaptive selection strategy, and problems of different difficulty levels are solved at the lowest cost. In addition, the grid-like splicing of video frames not only solves the problem of lack of detailed information in text descriptions, but also overcomes the limitation that the multimodal large language model cannot accept longer image sequence inputs at the same time. The generated option confidence score is an effective means to assist the large language model in making correct choices, thereby improving the accuracy and quality of video question-answering. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 This is a flow chart of the long video question answering method based on multi-round reasoning of a large model agent of the present invention;

[0043] Figure 2 This is a diagram of the grid-based key frame splicing method of the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] The specific steps of a long video question answering method based on multi-round reasoning of a large model agent are as follows:

[0046] A long video question answering method based on multi-round reasoning of a large model agent includes the following steps:

[0047] Step 1: Fine-tune the keyframe localization module and video question answering module on the downstream dataset.

[0048] First, use the pre-trained keyframe localization module F loc to locate key frames and train the video question answering module F on the downstream dataset ans Next, based on the QA task prompts, the frozen F ansThe module generates a frame-level prediction answer for each uniformly sampled frame, and compares the predicted answer with the true answer to obtain a pseudo label as F loc Keyframe annotations of a module to fine-tune the module.

[0049] Step 2: Locate the initial key frame sequence.

[0050] uniformly sample n frames {f1,...,f n}, through the image encoder E v Extract the visual features of the frame, and then get a set of video frame feature sequences V = {h1,...,h n}. Key frame positioning module F loc The main goal of is to select k language-aware key frame features from V. Thus, the question, options, and positioning prompt “Does the information within the frame provide the necessary details to accurately answer the given question?” form the text context L, and Q-Former Q loc The visual query feature v is extracted independently from V i The LLM outputs the scores of each frame and selects the top-k keyframe features with the highest scores as the output of the module. Then, the k keyframes are combined into a grid image. The number of the current video frame is marked in the upper left corner of each frame to preserve the timing information.

[0051] K=LLM(concat(Q loc (E v (f1,...,f n )),L)),|K|=k< <n(1)

[0052] Step 3: Generate confidence scores for options.

[0053] Based on the key frame feature set K obtained in step 2, through the Q-former Q ans To generate the key frame query feature v i . The key frame query feature v i By inputting LLM together with the text context, we can get the confidence score s for each option. i Finally, for c options, we get the confidence score set S of all options. ans =s1,...,s c} as the output of the current module.

[0054]

[0055] Step 4: Generate answers and assess their confidence.

[0056] The confidence score set S obtained in step 3 ans By option i (s i ) is in the form of each option option i Add confidence scores. Combine questions, options, and instruction prompts as text inputs for the multimodal large language model MLLM, and use the key frame splicing image obtained in step 2 as the visual input for MLLM. Use MLLM's multimodal reasoning capabilities to fuse information from different modalities and find the answer that best fits the question from the options. Prompt MLLM to evaluate based on the currently provided information and answer prediction process. The next steps of the model are:

[0057] Action 1: If the information provided in the current state is sufficient to answer the question, the multimodal large language model MLLM generates the answer to the question and exits the iterative process.

[0058] Action 2: If the information in the current state is not sufficient, it is necessary to continue searching for new information. Then, the MLLM is prompted to infer which video segment contains the necessary information to answer the question based on the current state, and further search for video frame information in that segment.

[0059] Finally, MLLM decides whether to execute Action 1 or Action 2 based on the answer confidence score.

[0060] Step 5: Identify video clips containing key information.

[0061] In the current step, MLLM is needed to help locate the segment of the video frame related to the question, and then find the missing visual information from the segment. First, prompt MLLM to divide the original n-frame video frame sequence into several paragraphs based on the k-frame video frame IDs initially provided. MLLM can infer and predict whether the paragraph may contain content related to the question based on the content of the starting frame and the ending frame of the paragraph. Finally, the frame ID sequence S contained in the located video segment is returned.

[0062] Step 6: Locate frames from the video clip to supplement the visual information.

[0063] After obtaining the video clip located in step 5, use the key frame positioning module F loc To locate the key frames in a given clip. Different from step 2, the input of the key frame positioning module in the current step is the original video frame feature sequence V = {h1,...,h n} is a subset V_seg = {hseg_1 ,...,h seg_n}. Key frame positioning module F loc The input sequence is processed to obtain the score of each frame in the given segment, and the top-m key frame features with the highest scores are selected for output. Considering that the initial frames are not uniformly sampled, the lengths of the divided video segments are also inconsistent. Therefore, a length threshold is set. When the length of the video segment exceeds this threshold, the length of the supplementary key frame m is set to m1, otherwise it is set to m2, and m1>m2. After the current step, m frames of supplementary video frames are obtained and used to update the initial state. The specific operation is to splice the initial k frames and the supplementary m frames in chronological order to form the initial state of the next round of reasoning question answering.

[0064] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. A long video question answering method based on multi-round reasoning of a large model agent, characterized by: The following steps are involved: Step 1: Fine-tune the keyframe localization module and video question answering module on the downstream dataset; use the pre-trained keyframe localization module F loc to locate key frames and train the video question answering module F on the downstream dataset ans Fine-tune the frozen F based on QA task prompts ans The module generates a frame-level prediction answer for each uniformly sampled frame, and compares the predicted answer with the true answer to obtain a pseudo label as F loc Keyframe annotations of a module to fine-tune the module; Step 2: Locate the initial key frame sequence; Step 3: Generate confidence scores for options; Step 4: Generate answers and evaluate their confidence; Step 5: Identify video clips containing key information; Step 6: Locate key frames from the video clip to supplement the visual information.

2. According to claim 1, a long video question answering method based on multi-round reasoning of a large model agent is characterized by: The step 2 specifically includes: uniformly sampling n frames {f1, ..., f n }, through the image encoder E v Extract the visual features of the frame and obtain a set of video frame feature sequences V = {h1, ..., h n }, key frame positioning module F l oc selects k language-aware keyframe features from V, composes the question, options and positioning prompts into a text context L, and Q-FormerQ loc The visual query features v are extracted independently from V i The features of the top-k key frames with the highest scores are selected as the output of the module. The k key frames are combined into a grid image, and the number of the current video frame is marked in the upper left corner of each frame to preserve the timing information. K=LLM(concat(Q loc (E v (f1,...,f n )),L)),|K|=k<<n (1)。 3. According to claim 2, a long video question answering method based on multi-round reasoning of a large model agent is characterized by: The step three specifically includes: based on the key frame feature set K obtained in step two, through the Q-former Q ans To generate the key frame query feature v i , the key frame query feature v i By inputting LLM together with the text context, we can get the confidence score s for each option. i , for c options, get the confidence score set S of all options ans ={s1, ..., s c } as the output of the current module; 4. According to claim 3, a long video question answering method based on multi-round reasoning of a large model agent is characterized by: The step 4 specifically includes: The confidence score set S obtained in step 3 ans By option i (s i ) is in the form of each option option i Add confidence scores, combine questions, options, and instruction prompts as text inputs for the multimodal large language model MLLM, and use the key frame splicing image obtained in step 2 as the visual input of MLLM. Use MLLM's multimodal reasoning ability to fuse information from different modalities, find the answer that best fits the question from the options, and prompt MLLM to evaluate based on the currently provided information and answer prediction process. The next steps of the model are: Action 1: If the information provided in the current state is sufficient to answer the question, the multimodal large language model MLLM generates the answer to the question and exits the iteration process; Action 2: If the information in the current state is not sufficient, it is necessary to continue searching for new information, prompting the MLLM to infer which video segment contains the necessary information to answer the question based on the current state, and further search for video frame information in the segment; Finally, MLLM decides whether to execute Action 1 or Action 2 based on the answer confidence score.

5. According to claim 4, a long video question answering method based on multi-round reasoning of a large model agent is characterized by: The step five specifically includes: prompting MLLM to divide the original n-frame video frame sequence into several paragraphs based on the initially provided k-frame video frame IDs, and MLLM can infer and predict whether the paragraph may contain content related to the question based on the starting frame content and the ending frame content of the paragraph, and return the frame ID sequence S contained in the located video clip.

6. According to claim 5, a long video question answering method based on multi-round reasoning of a large model agent is characterized by: The step six specifically includes: using the key frame positioning module F l oc to locate the keyframes in a given clip.

Images