A video question and answer optimization method based on multi-agent cooperation
By using a multi-agent collaborative framework, the problems of video data alignment and computational burden of long videos in video question answering tasks are solved, achieving an efficient video question answering process and improving accuracy and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- LANZHOU UNIV
- Filing Date
- 2026-02-13
- Publication Date
- 2026-05-29
AI Technical Summary
Existing multimodal large language models have several drawbacks in video question answering tasks, including the difficulty in efficiently aligning and processing visual and textual information due to the temporal complexity of video data, the high computational burden of long video inputs, the increased difficulty in model understanding due to redundant user queries, and the tendency for models to fall into suboptimal inference paths.
A multi-agent collaborative framework is adopted, including video localization, segment verification, question and answer generation, answer verification, and query rewriting. The localization agent filters key video segments, the verification agent filters relevant segments, the answering agent generates the initial answer, the verification agent verifies the correctness of the answer, the rewriting agent optimizes the query, and the answering agent generates the final answer, forming a closed-loop process.
It significantly improves the accuracy of video question answering tasks, optimizes long video processing capabilities, reduces computational costs, and enhances the robustness and consistency of reasoning.
Smart Images
Figure CN122114167A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of multimodal large language models and video understanding, and in particular to a video question answering optimization method based on multi-agent collaboration. Background Technology
[0002] 1. Current Technological Progress
[0003] Video question answering is a classic research area in computer vision, requiring multimodal large language models (MLLMs) to combine multimodal video input with text commands to generate accurate answers. With the development of large language models (LLMs), MLLMs have achieved unified modeling of text, images, and videos, making significant progress in multimodal tasks such as visual question answering and narrative construction. Numerous studies have validated the application potential of MLLMs in video QA tasks, effectively improving task processing capabilities and enabling cross-modal perception and reasoning fusion.
[0004] 2. Deficiencies in existing technology
[0005] Although MLLMs provide technical support for VideoQA, their accuracy in practical applications has not yet met expectations. The core issues are as follows:
[0006] (1) Video data has a unique temporal dimension. The complexity of multimodal input makes it difficult to efficiently align and process visual and textual information. Existing models lack explicit temporal localization, frame revisit and memory aggregation mechanisms, which easily produce incomplete inference trajectories and make it difficult to capture dynamic interactions and causal dependencies in videos.
[0007] (2) Long video input will bring high computational burden, making it difficult for the model to focus on key video segments, diluting attention and thus reducing the accuracy of the answer. Moreover, the existing methods show significant performance degradation in long video understanding scenarios.
[0008] (3) User text queries have problems such as unclear expression and redundancy, which increases the difficulty of model understanding. Excessive cognitive load can easily lead to incorrect answers. At the same time, the existing model lacks effective query optimization and answer verification mechanisms.
[0009] (4) Large language models often exhibit “thinking degradation”. Once they fall into a suboptimal reasoning path, they will continue to reinforce errors. A single model architecture is difficult to balance reasoning accuracy and efficiency. Summary of the Invention
[0010] To address the shortcomings of existing VideoQA technologies, this invention proposes the MVRR multi-agent collaborative framework, integrating five core capabilities: video localization, segment verification, question-and-answer generation, answer validation, and question rewriting. This framework first generates an initial answer by locating key video segments and combining them with the original query. Then, a validation agent judges the correctness of the initial answer. If the answer is incorrect, a rewriting agent simplifies redundant query information. Finally, a secondary question-and-answer session is completed based on the rewritten question, forming a closed-loop process of "localization-verification-initial question-and-answer-validation-rewriting-secondary question-and-answer," significantly improving the accuracy of VideoQA tasks and demonstrating excellent performance in multiple benchmark tests.
[0011] Includes the following steps:
[0012] S1) Video Segment Localization: The localization agent receives video data and user queries, performs layer normalization and linear projection on the token embedding and frame embedding of the video, adds position encoding and modality encoding to achieve feature fusion, and then optimizes the model through total loss to output key video segments that match the user query.
[0013] S2) Segment Relevance Verification: The verification agent filters the localized video segments, selects Top-K (K=5) segments, and expands both ends of each segment by 50% to supplement the context, inserting...<SEG_START> and<SEG_END> Identify segment boundaries; process segments using MLLM and extract "yes / no" binary decision probabilities, calculate segment relevance scores, and select the most relevant video segments;
[0014] S3) Initial Question and Answer Generation: The answering agent generates initial answers based on the most relevant video clips and the user's original query, relying on the Qwen2-VL basic model.
[0015] S4) Initial Answer Validation: The validation agent takes the complete video content, the original query, and the initial answer as input, and calculates the answer confidence based on the Qwen2-VL model.
[0016] S5) Query Rewrite Optimization: The rewrite agent receives the original query and, guided by preset prompts, simplifies it into a highly focused, non-redundant rewrite problem, outputting a JSON structure. ,
[0017] S6) Secondary Question Answering Output: The responding agent generates the final answer based on the most relevant video clip and a rewritten question.
[0018] Furthermore, the formula for calculating the segment relevance score in S2 is as follows:
[0019]
[0020] In the formula, Indicates the fragment relevance score, This represents the log probability that the model determines a video segment is relevant to a user query. This represents the log probability that the model determines the video clip is irrelevant to the user's query. The function maps the differences to the 0-1 interval to quantify the degree of correlation.
[0021] Furthermore, the formula for calculating the initial answer in S3 is as follows:
[0022]
[0023] In the formula, This represents the initial answer generated based on the original query. The function representing the response of the intelligent agent. This represents the video segment with the highest relevance selected by the validation agent. This represents the user's original query.
[0024] Furthermore, the formula for calculating the confidence level of the S4 answer is as follows:
[0025]
[0026] In the formula, Indicates the confidence level. Indicates the preset threshold. This represents the logarithmic probability that the model determines the initial answer is correct. This represents the log-probability that the model determines the initial answer is incorrect; the confidence level... With preset threshold In comparison, if > Then the initial answer will be used as the final answer. < The initial answer is then deemed incorrect, triggering the subsequent process.
[0027] Furthermore, the simplified query result obtained in S5 is as follows:
[0028]
[0029] In the formula, This indicates a simplified query after rewriting. The function that represents rewriting the agent. This represents the user's original query.
[0030] Furthermore, the formula for generating the final answer in S6 is as follows:
[0031]
[0032] In the formula, This represents the final correct answer. The function representing the response of the intelligent agent. This represents the video segment with the highest relevance selected by the validation agent. This indicates the rewritten query.
[0033] Furthermore, the localization agent, verification agent, response agent, validation agent, and rewriting agent share the Qwen2-VL backbone network based on the Chain-of-LoRA strategy, and the specific implementation is as follows:
[0034] Freeze the large weight matrix of the backbone network Configure an independent low-rank matrix for each agent. and (rank The intelligent agent's functionality is adapted using a formula, and each intelligent agent is configured with unique prompts to define task boundaries. The formula for adapting the intelligent agent's functionality is as follows:
[0035]
[0036] In the formula, This represents the frozen large weight matrix of the Qwen2-VL backbone network. Indicates input features, and These are low-rank matrices (rank) ), This is the output of the low-rank adapter.
[0037] Furthermore, the specific prompts for each agent must meet the following requirements:
[0038] (1) The prompt words for the localization agent clearly define its core task of "matching user queries with video clips";
[0039] (2) Verify that the agent's prompts limit its attention to only those words.<SEG_START> and<SEG_END> The identified video segment range;
[0040] (3) Verify that the agent's prompts constrain its output to be only a binary result of "yes" or "no";
[0041] (4) The prompt words for rewriting the agent include successful and failed cases of query rewriting, guiding it to generate concise questions that meet the requirements.
[0042] The beneficial effects of this invention are as follows:
[0043] 1. This invention can improve question-and-answer accuracy. Through a closed-loop process, it achieves answer correction and query optimization. In benchmark tests such as Video-MME (long), LongVideoBench, and CG-Bench, the MVRR model with 2B parameters achieves accuracies of 46.6%, 51.4%, and 31.9% respectively, all outperforming the VideoMind model with the same parameters and surpassing most existing models with 7B or more parameters.
[0044] 2. This invention optimizes long video processing capabilities. The localization and verification agent can accurately filter key segments in long videos, reducing the computational and cognitive burden of long videos. On the CG-Bench dataset with an average duration of 27 minutes, the MVRR model achieves a question-answering accuracy of 31.9% for long videos, significantly outperforming traditional methods.
[0045] 3. This invention can reduce computational costs. By adopting a Chain-of-LoRA strategy to share the backbone network, multi-agent functions are achieved only through lightweight LoRA adapters, without the need for large-scale model reconstruction. Excellent results can be achieved with a 2B parameter model, combining performance and efficiency.
[0046] 4. This invention enhances the robustness of reasoning. The multi-agent collaborative mechanism effectively avoids the problem of "thinking degradation," verifies the agent's secondary review, and rewrites the agent's query optimization, providing dual protection for the reasoning process, improving the reliability and consistency of the answer, and simultaneously achieving the interpretability of the reasoning process. Attached Figure Description
[0047] Figure 1 This is a comparison diagram between end-to-end methods and multi-agent methods;
[0048] Figure 2 This is a schematic diagram of the overall workflow of the MVRR framework;
[0049] Figure 3 This is a diagram illustrating the working principle and architecture of the LoRA localization agent. Figure 3 (a) is the LoRA parameter update mechanism. Figure 3 (b) is the feature processing and loss calculation process for the localization agent;
[0050] Figure 4 Visualization of actual question-and-answer examples using the MVRR framework;
[0051] Figure 5 The graph shows a comparison of model performance and an analysis of the impact of threshold values. Figure 5 (a) Performance comparison of MVRR with other models on the Video-MME (Long) dataset. Figure 5 (b) shows the impact of threshold adjustment on multi-benchmark performance. Detailed Implementation
[0052] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings:
[0053] Example 1
[0054] As a preferred embodiment, such as Figure 1 As shown, this embodiment proposes a video question-answering optimization method (MVRR) based on multi-agent collaboration. This method constructs a five-agent collaborative system including a localization agent, a verification agent, a response agent, a validation agent, and a rewriting agent. Each agent shares the Qwen2-VL backbone network based on the Chain-of-LoRA strategy. Functional division is achieved by switching dedicated LoRA adapters and prompt words, replacing the traditional end-to-end single VLM model architecture and realizing a closed-loop process of "localization-verification-initial question-answering-validation-rewriting-secondary question-answering".
[0055] As a preferred embodiment, such as Figure 3 As shown in (b), the localization agent receives video and user queries. First, it performs layer normalization and linear projection on the video token embedding and frame embedding, and adds position encoding and modality encoding to complete feature fusion. Then, it processes the fused features through a dual-branch network. The first branch generates evidence classification features and calculates the classification loss. The second branch aggregates multi-scale context through a time pyramid encoder, via... <reg>Head regression start and end timestamps and calculate regression loss At the same time, consistency loss is introduced. Finally, the model is optimized using the total loss, and the key video segments that match the query are output. The formula for calculating the total loss is:
[0056]
[0057] In the formula, This represents the classification loss (used to determine the degree of matching between a video clip and the query category). This represents the regression loss (used to optimize the temporal positioning accuracy of video segments). This represents the consistency loss (used to ensure consistency in the fusion of multimodal features).
[0058] As a preferred embodiment, such as Figure 3 (b) As shown in the method principle, the verification agent scores and ranks the fragments output by the localization agent, selects the top-5 highly relevant fragments, expands both ends of each fragment by 50% to supplement the context, and inserts...<SEG_START> and<SEG_END> Identify boundaries; process segments using MLLM and extract "yes / no" binary decision probabilities; calculate relevance scores according to a formula; select the most relevant video segments; during training, segments with IoU greater than 0.5 are considered positive examples; and a lightweight LoRA adapter is used for fine-tuning. The formula for calculating the relevance score is:
[0059]
[0060] In the formula, This represents the relevance score between the video clip and the user's query. This represents the log probability that the model determines the fragment is relevant to the query. This represents the log-probability of determining that the two are unrelated. The function maps the differences to the 0-1 interval to quantify the degree of correlation.
[0061] As a preferred embodiment, such as Figure 2 As shown, the answering agent generates an initial answer (e.g., "They are glowing") based on the highest relevance video clips selected by the validation agent and the user's original query (e.g., "What did Layna and Bell do in the video clip?") using the Qwen2-VL basic model. The calculation formula is as follows:
[0062]
[0063] In the formula, This represents the initial answer generated based on the original query. To answer the agent's inference function, This refers to the video segment with the highest relevance selected by the verification agent. Query the raw text entered by the user.
[0064] As a preferred embodiment, such as Figure 2 As shown, the validation agent takes the complete video, the original query, and the initial answer as input, and calculates the answer confidence based on the Qwen2-VL model:
[0065]
[0066] In the formula, This represents the confidence level of the initial answer (range 0-1). This represents the logarithmic probability that the model determines the initial answer is correct. This represents the logarithmic probability of an incorrect judgment; more will follow. With preset threshold The comparison determines whether to trigger the query rewrite process.
[0067] As a preferred embodiment, such as Figure 2 As shown, the rewritten agent addresses redundant original queries by extracting core information through plain text processing. The original query "What did Layna and Bell do in the video clip?" is simplified into a JSON-formatted rewritten question, "What did the two partners do?", effectively reducing the model's cognitive load and focusing on the core query intent. Its calculation formula is:
[0068]
[0069] In the formula, This indicates that the simplified query generated by the agent will be rewritten (output format is JSON). To rewrite the text optimization function for the agent, The original redundant query entered by the user.
[0070] As a preferred embodiment, such as Figure 2 As shown, the answering agent, based on the most relevant video clip and the rewritten question, re-reasons and generates the final answer (e.g., "They decided to attack the enemy"), completing the error correction and optimization of the entire video question-answering process. The computational formula used is:
[0071]
[0072] In the formula, This represents the final answer after two rounds of optimization. To answer the agent's inference function, The most relevant video segment. To rewrite the simplified queries generated by the agent.
[0073] As a preferred embodiment, such as Figure 3 As shown in (a), the functional adaptation of each agent in this embodiment relies on the LoRA mechanism. This mechanism enables agent function switching without modifying the backbone network, significantly reducing storage and computational overhead. Its mathematical expression is:
[0074]
[0075] In the formula, This represents the output of the adapted agent. This is the frozen weight matrix of the Qwen2-VL backbone network. For the input feature vector, and A low-rank trainable matrix (rank) ), This is the output of the low-rank adapter.
[0076] As a preferred embodiment, as shown in Figure 4, this figure is a visualization of an actual question-and-answer example of the MVRR framework, which intuitively presents the reasoning process and error correction effect of the model in a specific video question-and-answer task. The corresponding core data is shown in Table 1 below.
[0077] Table 1 Figure 4 Actual Question and Answer Example Core Data Details
[0078]
[0079] In the case study, after the answering agent generates an initial answer based on the most relevant fragment and the original query, the verification agent determines that its confidence level has not reached the threshold, triggering the rewriting agent to simplify and optimize the query. Finally, the correct answer is output through a second question-and-answer session, highlighting the error correction capability of the MVRR framework's "location-verification-rewriting-second question-and-answer" closed loop, effectively avoiding answer errors caused by fragment location deviations or query comprehension redundancy in traditional models.
[0080] As a preferred embodiment, as shown in Figure 5, the figure includes two parts: model performance comparison and threshold influence analysis. The core data is shown in Table 2 below.
[0081] Table 2 Figure 5 Model performance and threshold affect core data
[0082]
[0083] Figure 5(a) is a bar chart comparing the performance of MVRR with other models on the Video-MME (Long) dataset. It can be seen that the MVRR model with 2B parameters achieves an accuracy of 46.6%, which not only outperforms the VideoMind model with the same parameters, but also surpasses most existing models with 7B or more parameters, fully verifying the performance advantages of the multi-agent collaborative framework. Figure 5(b) is a line graph showing the impact of threshold adjustment on the performance of multiple benchmark tests. It shows that when the preset threshold for the validation agent is 0.48, the overall performance of the model reaches its peak. Thresholds that are too high or too low will lead to a decrease in performance. This conclusion provides an important reference for parameter configuration in practical applications, ensuring that the model achieves the optimal balance between accuracy and computational efficiency.
[0084] Experimental comparison:
[0085] This embodiment verifies the performance of the method in Embodiment 1 using five video question-answering benchmarks: Video-MME, MLVU, LVBen, LongVideoBench, and CG-Bench, comprehensively evaluating the effectiveness of MVRR in video question-answering scenarios with different durations and task types.
[0086] This embodiment uses a 2B parameter MVRR model, and compares it with models including VideoMind (2B), VideoChat-TPO (7B), and PLLAVA (34B). The maximum number of frames for the localization agent is set to 100, the frame resolution for the response agent is set to 64×28×28 to 128×28×28, and the maximum number of frames for the validation agent is set to 100. The framework is built based on the Qwen2-VL model. During the training phase, the video resolution is 128×28×28 per frame, and during the inference phase, it is increased to 256×28×28, while maintaining a maximum of 16 frames. The question-answering accuracy (acc) is used as the core indicator, and the impact of adjusting the validation agent threshold on the model performance is also analyzed.
[0087] This embodiment verifies the performance of the MVRR method in Embodiment 1 on five types of video question answering benchmarks, and comprehensively evaluates its effectiveness in video question answering scenarios with different durations and task types. The experimental results are shown in Table 3 below.
[0088] Table 3 Performance comparison of the Model in Example 1 on video inference and general video benchmarks.
[0089]
[0090] As shown in Table 3, the MVRR model significantly outperforms existing methods in both long video question answering and general video understanding tasks.
[0091] To visually and intuitively evaluate the reasoning accuracy and error correction capabilities of the method of this invention in video question answering tasks, the MVRR model of Example 1 and the VideoMind model were qualitatively compared on a character behavior attribution task in the CG-Bench dataset. The qualitative results show that the MVRR model, through a closed-loop process of "verifying the agent's initial answer as incorrect—rewriting the agent to simplify the query—generating the correct conclusion through secondary question answering," demonstrates a significant improvement in accurate causal reasoning based on key segments of long videos and correcting erroneous answers. Compared to traditional models, it effectively avoids reasoning biases caused by query redundancy and segment misjudgment.
[0092] Table 4. Results of MVRR ablation experiments (Video-MME(long) baseline)
[0093]
[0094] As shown in Table 4, enabling both the Reviser and Rewriter agents simultaneously is key to improving performance.
[0095] When only the rewritten agent is enabled, the Video-MME(long) baseline accuracy is 43.2%, which is only 0.9 percentage points higher than the baseline VideoMind model (42.3%).
[0096] Simultaneously, after enabling dual agents, the accuracy rate jumped to 46.6%, an improvement of 4.3 percentage points compared to the baseline, proving that the collaborative strategy of "verify first and then rewrite" can effectively avoid the loss of key query information and achieve dual optimization of answer correction and reasoning focus.
[0097] The embodiments and descriptions above are merely illustrative of the principles and preferred embodiments of the present invention. Various changes and modifications may be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed.< / reg>
Claims
1. A video question answering optimization method (MVRR) based on multi-agent collaboration, characterized in that, Includes the following steps: S1) Video Segment Localization: The localization agent receives video data and user queries, performs layer normalization and linear projection on the token embedding and frame embedding of the video, adds position encoding and modality encoding to achieve feature fusion, and then optimizes the model through total loss to output key video segments that match the user query. S2) Segment Relevance Verification: The verification agent filters the localized video segments, selects Top-K (K=5) segments, and expands both ends of each segment by 50% to supplement the context, inserting...<SEG_START> and<SEG_END> Identify segment boundaries; process segments using MLLM and extract "yes / no" binary decision probabilities, calculate segment relevance scores, and select the most relevant video segments; S3) Initial Question and Answer Generation: The answering agent generates initial answers based on the most relevant video clips and the user's original query, relying on the Qwen2-VL basic model. S4) Initial Answer Validation: The validation agent takes the complete video content, the original query, and the initial answer as input, and calculates the answer confidence based on the Qwen2-VL model. S5) Query Rewrite Optimization: The rewrite agent receives the original query and, guided by preset prompts, simplifies it into a highly focused, non-redundant rewrite problem, outputting a JSON structure. , S6) Secondary question answering output: The answering agent generates the final answer based on the most relevant video clip and the rewritten question.
2. The video question-answering optimization method based on multi-agent collaboration according to claim 1, characterized in that, The formula for calculating the fragment relevance score in S2 is as follows: In the formula, Indicates the fragment relevance score, This represents the log probability that the model determines a video segment is relevant to a user query. This represents the log probability that the model determines the video clip is irrelevant to the user's query. The function maps the differences to the 0-1 interval to quantify the degree of correlation.
3. The video question-answering optimization method based on multi-agent collaboration according to claim 1, characterized in that, The formula for calculating the initial answer in S3 is as follows: In the formula, This represents the initial answer generated based on the original query. The function representing the response of the intelligent agent. This represents the video segment with the highest relevance selected by the validation agent. This represents the user's original query.
4. The video question-answering optimization method based on multi-agent collaboration according to claim 1, characterized in that, The formula for calculating the confidence level of the S4 answer is as follows: In the formula, Indicates the confidence level. Indicates the preset threshold. This represents the logarithmic probability that the model determines the initial answer is correct. This represents the log-probability that the model determines the initial answer is incorrect; the confidence level... With preset threshold In comparison, if > Then the initial answer will be used as the final answer. < The initial answer is then deemed incorrect, triggering the subsequent process.
5. The video question-answering optimization method based on multi-agent collaboration according to claim 1, characterized in that, The simplified query result obtained from the rewritten S5 is as follows: In the formula, This indicates a simplified query after rewriting. The function that represents rewriting the agent. This represents the user's original query.
6. The video question-answering optimization method based on multi-agent collaboration according to claim 1, characterized in that, The formula for generating the final answer in S6 is as follows: In the formula, This represents the final correct answer. The function representing the response of the intelligent agent. This represents the video segment with the highest relevance selected by the validation agent. This indicates the rewritten query.
7. The video question-answering optimization method based on multi-agent collaboration according to claims 1 to 6, characterized in that, The localization agent, verification agent, response agent, validation agent, and rewriting agent share the Qwen2-VL backbone network based on the Chain-of-LoRA strategy. The specific implementation method is as follows: Freeze the large weight matrix of the backbone network Configure an independent low-rank matrix for each agent. and (rank The intelligent agent's functionality is adapted using a formula, and each intelligent agent is configured with unique prompts to define task boundaries. The formula for adapting the intelligent agent's functionality is as follows: In the formula, This represents the frozen large weight matrix of the Qwen2-VL backbone network. Indicates input features, and These are low-rank matrices (rank) ), This is the output of the low-rank adapter.
8. The video question-answering optimization method based on multi-agent collaboration according to claim 7, characterized in that, Each agent's unique cue words must meet the following requirements: (1) The prompt words for the localization agent clearly define its core task of "matching user queries with video clips"; (2) Verify that the agent's prompts limit its attention to only those words.<SEG_START> and<SEG_END> The identified video segment range; (3) Verify that the agent's prompts constrain its output to be only a binary result of "yes" or "no"; (4) The prompt words for rewriting the agent include successful and failed cases of query rewriting, guiding it to generate concise questions that meet the requirements.