Multi-modal video reasoning method and system based on hierarchical multi-agent

Abstract

The invention discloses a hierarchical multi-agent-based multi-modal video reasoning method and system, and belongs to the technical field of artificial intelligence and multi-modal video understanding. The method comprises the following steps: receiving an input video and a natural language query, and analyzing a complex query into a plurality of logically coherent sub-questions through a question decomposition agent; distributing the sub-questions to multi-source answer generation agents, wherein the multi-source answer generation agents comprise an answer generation agent based on Web, an answer generation agent based on time sequence memory and a video-language answer generation agent; the intelligent agent based on time sequence memory models video long-range dependence through a visual memory library and a query memory library, and adopts a cross-attention mechanism to fuse time sequence features; performing consistent voting and fusion on the multi-source answers through the decision agent to generate a final answer; and carrying out training optimization on the system based on a cross entropy loss function.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and video question answering technology, and in particular to a multimodal video reasoning method and system based on hierarchical multi-agent systems. Background Technology

[0002] Multimodal video reasoning has become a fundamental capability of artificial intelligence, enabling systems to understand dynamic visual content, model temporal dependencies, and integrate cross-modal semantics. It supports a wide range of applications, including video question answering, event localization, and scene understanding. Compared to static image or text data, video is inherently dynamic and multimodal, encapsulating visual, auditory, and occasional textual streams while exhibiting complex temporal dependencies. Reasoning on such data requires understanding long-range temporal structures and using external, often domain-specific, knowledge to locate ambiguous content.

[0003] Existing video language models have the following limitations:

[0004] 1) Limitations of closed-world training data: While independent models such as video language agents have demonstrated strong temporal localization performance, these models are inherently limited by closed-world training data and are often unable to answer queries that rely on external knowledge, such as regulatory standards, scientific principles, or real-world events.

[0005] 2) Insufficient generalization ability on small datasets: Models trained on small datasets, even with specialized temporal architectures, lack the generalization ability required to adapt across different video domains and contextual scenarios.

[0006] 3) Video Adaptation Issues in Retrieval-Enhanced RAG Generation: Most RAG methods treat videos as static collections of frames, ignoring the temporal dynamics and long-range context necessary for comprehensive video understanding. Retrieval strategies developed for images or text often fail to capture video-specific features such as variable-length temporal segments, evolving scenes, and continuous motion.

[0007] 4) Limitations of Heterogeneous Input Integration in Single-Source Retrieval: Many RAG frameworks rely on single-source retrieval, which limits their ability to integrate heterogeneous inputs, such as web-based knowledge and temporal video memory, both of which are crucial for overall video reasoning.

[0008] Therefore, addressing these shortcomings is crucial for improving the accuracy, scalability, and robustness of multimodal video inference systems. Summary of the Invention

[0009] To address the shortcomings of existing technologies, this invention provides a multimodal video reasoning method and system based on hierarchical multi-agent systems, which can significantly improve the accuracy, scalability, and robustness of video reasoning.

[0010] This invention provides a multimodal video inference method based on hierarchical multi-agent systems, comprising the following steps:

[0011] S1: Receives input video and natural language queries;

[0012] S2: The problem decomposition agent parses complex queries into multiple logically coherent sub-problems;

[0013] S3: Distribute sub-problems to multi-source answer generation agents, including web-based answer generation agents, time-series memory-based answer generation agents, and video-language answer generation agents;

[0014] S4: The agent based on temporal memory models long-range video dependencies through visual memory and query memory, and uses a cross-attention mechanism to fuse temporal features;

[0015] S5: Generate the final answer by performing consensus voting and fusion of multiple source answers through a decision-making intelligent agent;

[0016] S6: Train and optimize the system based on the cross-entropy loss function.

[0017] Furthermore, the multi-source answer generation agent includes:

[0018] A web-based answer generation agent is used to perform external knowledge retrieval and generate web-based answers.

[0019] A temporal memory-based answer generation agent is used to model the long-range temporal dependencies of videos and generate temporal-based answers.

[0020] The video-language answer generation agent is used to reason directly based on video content and generate video-language answers.

[0021] Furthermore, the answer generation agent based on temporal memory includes: a visual memory bank and a query memory bank, used to store the visual features of the video and historical query features;

[0022] By fusing features from the visual memory and query memory through a dual-memory cross-attention mechanism, a temporally-aware representation relevant to the current query is generated.

[0023] Furthermore, the decision-making agent includes: a consensus voting mechanism for evaluating the consistency of multiple source answers;

[0024] If the consistency of multiple source answers meets the preset threshold, the wording of the answers will be optimized and unified through a large language model to generate the final answer;

[0025] If the consistency of multiple answers does not meet the preset threshold, an expert model is invoked to perform multi-source evidence fusion, conflict identification and resolution, and generate the final answer.

[0026] Furthermore, the system training optimization includes training the system using the cross-entropy loss function, as shown in the following formula:

[0027]

[0028] in The total number of real text tags. For the first A real marker, For the first The sequence of tags preceding each tag, To input video data, Indicates a given video Generate tags when using pre-marking The probability is calculated, and all parameters are optimized using the AdamW optimizer.

[0029] A multimodal video reasoning system based on hierarchical multi-agent retrieval enhancement generation, for implementing the above method, includes: an input module for receiving input video and natural language queries;

[0030] Problem decomposition agents are used to parse complex queries into multiple logically coherent sub-problems;

[0031] Multi-source answer generation agents include web-based answer generation agents, time-series memory-based answer generation agents, and video-language answer generation agents;

[0032] A decision-making agent is used to perform consensus voting and fusion of multiple source answers to generate the final answer.

[0033] The training module is used to train and optimize the system based on the cross-entropy loss function.

[0034] Furthermore, the multi-source answer generation agent includes: a Web-based answer generation agent configured with an external retrieval interface for retrieving external knowledge and generating Web-based answers;

[0035] A temporal memory-based answer generation agent is configured with a visual memory bank and a query memory bank to model long-range video dependencies and generate temporal-based answers.

[0036] The video-language answer generation agent is used to reason directly based on video content and generate video-language answers.

[0037] Furthermore, the answer generation agent based on temporal memory includes: a visual memory bank and a query memory bank, used to store the visual features of the video and historical query features;

[0038] A cross-attention mechanism is used to fuse features from the visual memory bank and the query memory bank to generate a temporally aware representation relevant to the current query.

[0039] Furthermore, the decision-making intelligent agent includes:

[0040] The consistency voting module is used to evaluate the consistency of answers from multiple sources.

[0041] The answer fusion module is used to generate the final answer based on the consistency assessment results.

[0042] Furthermore, the training module includes:

[0043] The cross-entropy loss function is used to calculate the difference between the system output and the true label;

[0044] The AdamW optimizer is used to optimize system parameters.

[0045] The beneficial effects of this invention are:

[0046] Significantly improves reasoning accuracy: By integrating internal temporal understanding of videos with external knowledge retrieval, it achieves state-of-the-art performance in multiple benchmark tests such as CLEVRER, NExT-GQA, MMVBench, and MLVU, surpassing mainstream 7B / 9B models with only 2B parameters;

[0047] Effective handling of complex queries: Through a problem decomposition mechanism, complex queries with multiple intents are processed in a structured manner, supporting cross-source joint retrieval, which solves the problem of insufficient ability of traditional methods to handle compound queries;

[0048] Enhanced temporal modeling capabilities: The temporal memory agent effectively captures long-range video dependencies through a dual-memory mechanism, achieving an accuracy of 62.30% on the MLVU long video inference benchmark, which is significantly better than existing methods;

[0049] Improve system scalability: Adopt an agent-based plug-and-play architecture, each agent can be optimized or replaced independently, and the Web agent integrates mechanisms such as real-time fact verification and attribute-aware generation, which improves the quality of external knowledge retrieval.

[0050] Ensuring answer reliability: The decision-making agent effectively resolves conflicts between multiple sources of answers by using ROUGE-L and BLEU dual-indicator consistency voting, combined with expert model refinement, ensuring the consistency and accuracy of the final output;

[0051] It combines parameter efficiency and generalization ability: while maintaining high performance, it only requires 2B parameters, which significantly reduces the computational cost, and through hyperparameter optimization (such as k=10), it shows good robustness on different tasks. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of the overall architecture of the hierarchical multi-agent framework of the present invention;

[0053] Figure 2 This is a schematic diagram of the video understanding agent based on temporal memory according to the present invention;

[0054] Figure 3 This is a visual schematic diagram of the multimodal video reasoning process of the present invention. Detailed Implementation

[0055] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0056] Example 1: A multimodal video reasoning system based on hierarchical multi-agent retrieval enhancement generation, such as... Figure 1 As shown, this embodiment provides a multimodal video reasoning system based on hierarchical multi-agent retrieval-enhanced generation (HM-RAG). This embodiment focuses on multimodal video reasoning tasks, including video question answering, event reasoning, and temporal understanding; specifically, it includes the following steps:

[0057] S1: System initialization and parameter setting: Load pre-trained video - language base model, initialize three dedicated agents:

[0058] Web agent (WA), configures external search interface, number of search results. ,in Indicates the number of search results;

[0059] Temporal memory agent (TA), configured with visual memory bank and query memory bank, memory compression threshold. ,in The similarity threshold representing frame feature merging;

[0060] Video-Language Agent (VA), Initialize Decision Agent (DA), Set Consistency Threshold ,in Indicates the consistency judgment threshold; fusion weight ,in These represent the weighting coefficients for the ROUGE-L and BLEU scores, respectively.

[0061] S2: Video and Query Input

[0062] The system receives video input and complex natural language queries as input. The video input consists of a sequence of multiple video frames, some of which may contain key objects or events (such as...). Figure 1 (As shown in the video frame sequence). Complex queries may contain multiple sub-questions, such as "What is the little boy in the video doing? What color is the motorcycle in the video? How much does this motorcycle cost?", requiring the system to decompose and perform multi-source reasoning.

[0063] S3: Problem Decomposition The agent determines the query intent:

[0064] If it is a multi-intent query, it is broken down into a sequence of sub-questions. ;

[0065] If it is a single-intent query, it will be treated as a single sub-question. .

[0066] S4: Multi-source answer generation. A multi-source agent receives the sub-problem sequence and processes it in parallel through three dedicated agents to generate multi-source answers.

[0067] Web-based intelligent agents: perform external knowledge retrieval for each sub-problem, and the retrieval process is implemented using the following formula:

[0068]

[0069] in This represents the set of external knowledge results obtained through retrieval. Indicates a sub-question query. This indicates the retrieval of a set of configuration parameters, specifically the values... ={numResults=k,language=en,type=web}, where k is a positive integer. numResults=k indicates that the number of search results is k, language=en indicates that the search language is English, and type=web indicates that the search type is web page search.

[0070] The agent comprises three main operating modes: real-time fact verification, which evaluates the validity of retrieved facts by querying neural memory; attribute-aware generation, which ensures the traceability of results through a two-stage attention mechanism; and adaptive query expansion, which mitigates word mismatch by dynamically adjusting term weights. The agent interacts with multiple search engines (such as Google, Bing, Baidu, Yahoo, etc.) to perform website search operations and generate web-based answers.

[0071] Temporal Agent: Models long-range temporal dependencies based on a video-based temporal memory. Video frames are processed by an encoder to extract visual features and store them in a memory; text queries are also processed by an encoder; video features from the memory and encoded text are input into an attention mechanism, combined with prompt words, and then input into a large language model to generate a text answer based on the video's temporal context.

[0072] Visual memory bank interaction:

[0073]

[0074]

[0075] in: , , These represent the query matrix, key matrix, and value matrix, respectively. Represents the query vector at time step t; This represents the temporal-aware feature matrix stored in the visual memory bank; , , This is the trainable projective weight matrix corresponding to the visual memory bank; The dimension of the key vector in the visual memory bank; This is the output representation of the visual memory bank.

[0076] Query memory interaction:

[0077]

[0078]

[0079] in: , , These represent the query matrix, key matrix, and value matrix, respectively. This represents the historical query feature matrix stored in the query memory; , , To query the trainable projective weight matrix corresponding to the memory; To query the key vector dimension of the memory; The output representation of the query memory.

[0080] Memory compression strategy: Calculate the similarity between adjacent frames ,in Let represent the cosine similarity of features between frame t and frame (t+1). Let these represent the feature vectors of frame t and frame (t+1) respectively; if Then merge features ,in This represents the similar frame features after fusion (see Example 2 for specific implementation details).

[0081] Video Language Agent: It performs reasoning directly based on video content, uses a pre-trained video-language model to process video frames and queries end-to-end, and generates video language answers.

[0082] The set of answers generated by the three agents { A multi-answer set is formed, including web-based answers, time-series answers, and video-language answers, and then passed to the decision-making agent.

[0083] like Then merge features ,in This represents the similar frame features after fusion.

[0084] S6: Answer Consistency and Decision-Making Intelligent Agent (Performs consistency evaluation and fusion of multi-source answers. First, a voting operation is performed to initially aggregate the multi-answer set; then, in the refinement stage, a consistency voting mechanism is executed. A summary is generated for the multi-source answers.) } Calculate the consistency score: ROUGE-L score:

[0085]

[0086] in They represent the first The and the first Summary text of each candidate answer; Indicates the length of the longest common subsequence of the two summaries; These represent the character lengths of the two summaries, respectively.

[0087] BLEU score:

[0088]

[0089] in: express -gram precision, i.e., the matching accuracy between the candidate abstract and the reference abstract. -gram ratio; express The weighting coefficients of -grams are usually taken as equal weights. ; express The maximum order of a gram is usually taken as... ; These represent the lengths of the candidate abstract and the reference abstract, respectively.

[0090] Fusion Consistency Score ,in These represent the average ROUGE-L and BLEU scores, respectively.

[0091] If the consensus vote is positive (C≥τ), then the multiple source answers are essentially consistent, and the process enters the refinement stage. A large-scale language model is used to optimize and unify the wording of the answers, generating the final answer. If the consensus vote is negative (C<τ), then there is a substantial conflict between the answers. An expert model is invoked to perform multi-source evidence fusion, conflict identification and resolution, and the final answer is generated through thought chain reasoning. The correct answer is then output.

[0092] S6: The training objective is to train the system by minimizing the cross-entropy loss.

[0093]

[0094] in The total number of real text tags. For the first A real marker, For the first The sequence of tags preceding each tag, To input video data, Indicates a given video Generate tags when using pre-marking The probability of.

[0095] Optimize all parameters using the AdamW optimizer.

[0096] Example 2: Temporal Memory Intelligent Agent

[0097] S1: Video Input and Visual Encoding

[0098] S1.1: Input video sequence containing T frames, where T represents the total number of frames.

[0099] S1.2: The video frames are processed by two pre-trained visual encoders to extract visual features and query relevant features, respectively.

[0100] The visual encoder extracts visual features from each frame and outputs a visual feature sequence. ,in Represents the visual features of frame t. This indicates the number of features after processing. It also generates a query-related feature sequence. ,in This represents the query feature at time step t.

[0101] S2: Long-Term Memory Construction and Maintenance

[0102] The features output by the encoder are stored in a long-term memory, which contains two sub-memories.

[0103] Visual memory bank Store the sequence of visual features extracted by the visual encoder, each element Represents the visual features of the nth frame, with maximum capacity. Frame, in which This indicates the maximum capacity of the visual memory bank.

[0104] Query memory Store the query-related feature sequence, each element This represents the query feature representation at the nth time step, with a maximum capacity. Query, among which This indicates the maximum capacity of the query memory.

[0105] S3: Dual-Memory Bank Cross-Attention Mechanism in Sequential Memory Agents

[0106] The core of the Temporal Memory Agent (TA) is its dual-memory bank cross-attention mechanism, which retrieves relevant information from the visual memory bank and the query memory bank in parallel and fuses the retrieved features to form a comprehensive and coherent temporal perception representation.

[0107] S3.1: Visual memory bank cross-attention interaction with the current query vector As a "query", the visual memory bank The temporal-aware features stored in the database are used for cross-attention retrieval, focusing on the spatiotemporal context of the video relevant to the current query.

[0108]

[0109] In the formula:

[0110] : Represents the query matrix, key matrix, and value matrix, respectively;

[0111] : No. The query feature vector at each time step;

[0112] The temporal-aware feature matrix stored in the visual memory bank;

[0113] : The trainable projective weight matrix corresponding to the attention mechanism of the visual memory bank;

[0114] : The dimension of the visual memory key vector;

[0115] The output representation of the visual memory bank across attention interactions encodes the temporal context information of the video most relevant to the current query.

[0116] S3.2: While querying the memory across attention interactions, use the current query vector As a "query", the query memory The historical query features stored in the database are used for cross-attention retrieval, aiming to discover historical question patterns or semantic contexts related to the current query.

[0117]

[0118] In the formula:

[0119] : Represents the query matrix, key matrix, and value matrix, respectively;

[0120] : Query the historical query feature matrix stored in the memory;

[0121] : Query the trainable projective weight matrix corresponding to the attention mechanism in the memory bank;

[0122] : Query the dimensions of the key vector in the memory;

[0123] The output representation of the query memory across attention interactions encodes historical query context information related to the semantics of the current query.

[0124] S4: Output of the fully connected layer and the feature transformation visual memory bank through cross-attention interaction Output of cross-attention interaction with query memory After feature fusion, feature transformation is performed through a fully connected layer (FC) to obtain the final temporal-aware representation. This is used for subsequent language model generation.

[0125] S5: Large-scale language model generation will generate the results from S4. Combined with text prompts, the input is a large language model (LLM). The LLM generates answers based on the fused multimodal representation and the temporal context of the video. It completes the entire reasoning process from video input to text output.

[0126] Example 3

[0127] like Figure 3 As shown, this embodiment details the complete process from query input to final answer generation, including question decomposition, multi-source answer generation, consensus voting, and expert refinement mechanisms. The specific steps are as follows:

[0128] S1: Query Input and Problem Breakdown

[0129] Receive natural language query input ,in This represents the original query text; it also receives video input. ,in This indicates the input video data, which consists of a sequence of frames across multiple time segments, with timestamps as follows. ,in Indicates the first A time marker, This indicates the total number of timestamps.

[0130] The problem-decomposition agent determines the intent of the input query:

[0131] If the query is determined to be a single-intent query, the query content is not broken down and is directly treated as a single sub-problem. Transmission, among which Represent subproblems;

[0132] If it is determined to be a multi-intent query, it is decomposed into a sequence of multiple sub-problems. ,in Indicates the number of subproblems. Indicates the first The issue is about size.

[0133] In this embodiment, the query is determined to be a single intent, so the original query is kept unchanged and directly passed to the multi-source agent.

[0134] S2: Multi-source answer generation

[0135] Multi-source intelligent agent receiving query and video input It is processed in parallel by three dedicated sub-agents:

[0136] 1. Temporal Intelligent Agent: Based on a video temporal memory, it records the sequence of events in the video and generates answers. ,in This represents the answer generated by the sequential intelligent agent;

[0137] 2. Web-based intelligent agent: Performs external knowledge retrieval and generates answers. ,in This represents the answer generated by the Web agent;

[0138] 3. Video Language Intelligent Agent: Performs reasoning based on video content to generate answers. ,in This represents the answer generated by the video language agent.

[0139] The set of answers generated by the three agents is This information is then transmitted to the decision-making agent for consistency evaluation.

[0140] S3: Consensus Voting and Decision Making

[0141] The decision-making agent receives answers from multiple sources. Implement a consensus voting mechanism:

[0142] S3.1: Summary Generation Generates a summary of uniform length for each answer. Set the summary length Words, among which Indicates the length of the abstract. They represent Summary text.

[0143] S3.2: Calculation of Consistency Score Calculate the ROUGE-L and BLEU consistency scores for each pair of answers:

[0144] ROUGE-L score:

[0145]

[0146] in: Indicates the first The and the first ROUGE-L scores among the summaries; They represent the first The and the first Summary text of each answer; Indicates two summaries and The length of the longest common subsequence; Each represents a summary and The character length.

[0147] BLEU score:

[0148]

[0149] in: Indicates the first The and the first BLEU scores between summaries; express -gram precision, which is the percentage of candidate abstracts that match the reference abstract. -gram ratio; express -gram weighting coefficients, usually taken as ; express - The order of the gram, with values ​​ranging from 1 to 2. ; These represent the character lengths of the candidate abstract and the reference abstract, respectively.

[0150] S3.3: Overall Consistency Judgment: Calculate the average consistency score of the three pairs of answers.

[0151]

[0152] in: This represents the average ROUGE-L score; Indicates the average BLEU score; superscript These represent three pairs of answers: Web-Time Series, Web-Video, and Time Series-Video.

[0153] The overall consistency score obtained by fusion:

[0154]

[0155] in: Indicates the overall consistency score; These represent the weighting coefficients for the ROUGE-L and BLEU scores, respectively, and are typically taken as... .

[0156] Set a consistency threshold ,in The consistency judgment threshold is indicated as follows:

[0157] like If the answers from multiple sources are considered to be largely consistent, then a large-scale language model is used to refine the wording.

[0158] like Then it enters the expert refinement process.

[0159] In this embodiment, the calculation is as follows: Therefore, it undergoes refining.

[0160] S4: Final Answer Generation and Interpretable Output

[0161] Based on the consensus voting result, the decision-making agent executes the following process:

[0162] S4.1: Concise Answer Due to Using a large language model (to refine wording, integrate key information from multiple source answers, and generate a unified answer). ,in This indicates a refined answer.

[0163] S4.2: Generating a detailed explanation of the reasoning process. ,in Text representing the reasoning process includes:

[0164] Analysis of primary sources of evidence;

[0165] Reasons for conflict resolution;

[0166] The basis for the final selection.

[0167] S4.3: Final Output - Output the final answer ,in This represents the final answer text, with an optional confidence score. ,in This indicates the confidence level of the answer, with a value ranging from 0 to 1. A higher value indicates a higher confidence level.

[0168] For example, output the final answer. "B. Send a stop signal to others", confidence level and provide reasoning. "The woman gestured, the man performed, and the band stopped. The chain of motion shows that the man signaled for the others to stop."

[0169] The final result provides users with transparent and interpretable multimodal video inference results.

[0170] Example 4

[0171] This embodiment verifies the performance of Embodiment 1 on multiple multimodal video inference benchmarks, including NExT-QA, Video-MME, MVBench, and MLVU.

[0172] Dataset description:

[0173] 1. NExT-QA: A video question-and-answer benchmark focused on short videos (1-3 minutes) for open-world reasoning, containing 20,000+ video clips and 100,000+ QA pairs, covering everyday activities, scientific experiments and public events; 25% of the questions require integrating video content with external factual knowledge.

[0174] 2. Video-MME: A benchmark for long-form video multimodal inference, evaluating a model’s ability to model long-range temporal dependencies and integrate cross-modal semantics. It includes 50,000+ annotated video clips (1-10 minutes) from heterogeneous domains such as sports, news, and entertainment; 40% of the test queries require the fusion of video frame content and external knowledge.

[0175] 3. MVBench: A fine-grained multimodal video understanding benchmark, containing 30,000+ high-resolution videos (integrating audio, visual, and text modalities), with target tasks including attribute recognition, motion localization, and cross-modal retrieval; 30% of the test queries require combining cross-segment video cues with external knowledge.

[0176] 4. MLVU: Multimodal long video understanding benchmark, focusing on long-range temporal relationship reasoning across extended content, including 40,000+ video clips (5-15 minutes) from educational lectures, documentaries and everyday life scenarios; 45% of the test queries require linking non-continuous video clips (spanning 3+ minutes) with domain-specific background knowledge.

[0177] Training settings:

[0178] This embodiment is based on the Qwen2-VL-2B model for fine-tuning and ensemble. The main training was performed on the NExT-QA and Video-MME datasets.

[0179] Problem decomposition agent: Using Qwen2.5-7B, fine-tuned for 5 epochs on a self-built query decomposition dataset (containing 10K single / multiple intent query pairs) with a learning rate of 2e-5;

[0180] Temporal memory agent: Based on Qwen2-VL-2B, trained on Video-MME and MLVU training sets, batch size 16, learning rate 1e-5, training 30K steps; maximum memory capacity set to 1000 frames, similarity compression threshold 0.95;

[0181] Web agent: Uses Google Serper API, no training required; number of search results k=10;

[0182] Answer aggregation agent: consensus voting threshold θ=0.6, ROUGE-L and BLEU fusion weight α=0.5; expert model uses GPT-4 for thought chain reasoning.

[0183] Mixed precision training was used, with AdamW as the optimizer and a learning rate that decayed cosine after linear warm-up (1K steps).

[0184] Experimental results:

[0185] The experimental results of this embodiment on four benchmark tests verify the effectiveness of HM-RAG in multimodal video inference tasks:

[0186] Table 1 Performance comparison of the multimodal video inference model on four benchmark datasets.

[0187]

[0188] As can be seen from Table 1, the HM-RAG framework of this invention (2B parameters only) achieves leading performance on all four benchmark datasets:

[0189] 1. Video-MME (Long Video Multimodal Inference): Acc reached 53.26%, which is 3.16 percentage points higher than the second-place ST-LLM (50.10%) and 20.76 percentage points higher than the lightweight FrozenBiLM (32.50%);

[0190] 2. NExT-QA (Video QA requiring external knowledge): Acc reached 66.62%, exceeding the runner-up ST-LLM (59.60%) by 7.02 percentage points, and higher than Video-ChatGPT (55.20%) by 11.42 percentage points.

[0191] 3. MVBench (Fine-grained video understanding): Acc reached 52.80%, slightly lower than Chat-UniVi (53.30%) but better than the specialized models mPLUG-Owl (51.40%) and ST-LLM (51.90%).

[0192] 4. MLVU (Long-Term Temporal Relational Reasoning): Acc reached 62.30%, which is 12.50 percentage points higher than the second-ranked ST-LLM (49.80%) and 22.90 percentage points higher than Video-ChatGPT (39.40%), fully demonstrating its advantage in long-term video reasoning.

[0193] These results systematically confirm the effectiveness of the hierarchical multi-agent RAG mechanism. Its core advantages are: first, specialized Web agents and temporal memory agents address the pain points of mainstream models (such as Otter's insufficient external knowledge and LLaMA-VID's weak long-range temporal modeling); second, efficient parameter design (2B vs. 7B / 9B) avoids the high computational cost of large models while ensuring inference performance.

[0194] Ablation experiment:

[0195] Table 2 Performance Comparison of Different Variants

[0196]

[0197] Note: WA = Web Agent, TA = Temporal Memory Agent, VA = Video Language Agent, S = Answer Aggregation Agent

[0198] Table 2 details the ablation study of the four core agents of the framework, quantifying the contribution of each component to multimodal inference performance. A clear hierarchical structure of component importance emerges, and the complete system (with all agents enabled) achieves the highest average accuracy (58.75%), validating the synergistic effect among agents.

[0199] The Temporal Memory Agent (TA) is most critical to baseline performance: its removal caused the most dramatic drop in mean accuracy (31.41 percentage points to 27.34%), exhibiting consistent degradation across all benchmarks (GQA: 39.61%, MME: 23.67 percentage points, MLVU: 30.35%, MVBench: 31.99%). This confirms the irreplaceable role of TA in modeling temporal dependencies and contextual continuity, which are fundamental to multimodal tasks.

[0200] Web agents (WAs) are crucial for knowledge-intensive tasks, and their discontinuation reduced average accuracy by 10.88% (down to 47.87%). The impact was most significant in GQA (a 31.16% decrease, from 71.19% to 40.03%), while decreases in MME (10.90%) and MLVU (9.87%) further validated the value of WAs in supplementing domain-specific / external knowledge.

[0201] The Video Language Agent (VA) supports video-centric reasoning; its removal resulted in an average decrease of 1.29% (down to 57.46%), but its unique value was demonstrated in MVBench (a 3.99% gain when enabled, from 48.81% to 52.80%) and MLVU (a 5.62% gain, from 56.68% to 62.30%), proving its effectiveness in video-language semantic alignment.

[0202] The answer-aggregating agent (S) acts as an integration hub, ensuring framework robustness by aggregating multi-source outputs and enforcing cross-modal consistency. All effective configurations (S enabled) outperform the assumed S disabled setting, and the coordination of S is key to optimal performance of the complete system.

[0203] The full framework (with all agents enabled) outperforms all ablation settings: it surpasses the second-best configuration by 1.29% and the weakest configuration by 31.41%. It also achieves top performance on MME (53.26%), MLVU (62.30%), and MVBench (52.80%), highlighting agent collaboration.

[0204] These results confirm the complementary roles of the agents: the TA performs temporal reasoning, the WA provides external knowledge, the VA bridges video and language, and the S integrates these components. This collaborative design effectively addresses the multimodal challenges from temporal modeling to cross-modal alignment, driving superior performance.

[0205] The embodiments and descriptions above are merely illustrative of the principles and preferred embodiments of the present invention. Various changes and modifications may be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed.

Claims

1. A multimodal video reasoning method based on hierarchical multi-agent systems, characterized in that, Includes the following steps: S1: Receives input video and natural language queries; S2: The problem decomposition agent parses complex queries into multiple logically coherent sub-problems; S3: Distribute sub-problems to multi-source answer generation agents, including web-based answer generation agents, time-series memory-based answer generation agents, and video-language answer generation agents; S4: The agent based on temporal memory models long-range video dependencies through visual memory and query memory, and uses a cross-attention mechanism to fuse temporal features; S5: Generate the final answer by performing consensus voting and fusion of multiple source answers through a decision-making intelligent agent; S6: Train and optimize the system based on the cross-entropy loss function.

2. The method according to claim 1, characterized in that, The multi-source answer generation agent includes: A web-based answer generation agent is used to perform external knowledge retrieval and generate web-based answers. A temporal memory-based answer generation agent is used to model the long-range temporal dependencies of videos and generate temporal-based answers. The video-language answer generation agent is used to reason directly based on video content and generate video-language answers.

3. The method according to claim 2, characterized in that, The answer generation agent based on temporal memory includes: a visual memory bank and a query memory bank, used to store the visual features of the video and historical query features; By fusing features from the visual memory and query memory through a dual-memory cross-attention mechanism, a temporally-aware representation relevant to the current query is generated.

4. The method according to claim 1, characterized in that, The decision-making agent includes: a consensus voting mechanism for evaluating the consistency of multiple answers; If the consistency of multiple source answers meets the preset threshold, the wording of the answers will be optimized and unified through a large language model to generate the final answer; If the consistency of multiple answers does not meet the preset threshold, an expert model is invoked to perform multi-source evidence fusion, conflict identification and resolution, and generate the final answer.

5. The method according to claim 1, characterized in that, The system training optimization includes training the system using the cross-entropy loss function, as shown in the following formula: in The total number of real text tags. For the first A real marker, For the first The sequence of tags preceding each tag, To input video data, Indicates a given video Generate tags when using pre-marking The probability is calculated, and all parameters are optimized using the AdamW optimizer.

6. A hierarchical multi-agent multimodal video reasoning system, used to implement the method according to any one of claims 1 to 5, characterized in that, include: The input module is used to receive input video and natural language queries; Problem decomposition agents are used to parse complex queries into multiple logically coherent sub-problems; Multi-source answer generation agents include web-based answer generation agents, time-series memory-based answer generation agents, and video-language answer generation agents; A decision-making agent is used to perform consensus voting and fusion of multiple source answers to generate the final answer. The training module is used to train and optimize the system based on the cross-entropy loss function.

7. The system according to claim 6, characterized in that, The multi-source answer generation agent includes: a web-based answer generation agent configured with an external retrieval interface for retrieving external knowledge and generating web-based answers; A temporal memory-based answer generation agent is configured with a visual memory bank and a query memory bank to model long-range video dependencies and generate temporal-based answers. The video-language answer generation agent is used to reason directly based on video content and generate video-language answers.

8. The system according to claim 7, characterized in that, The answer generation agent based on temporal memory includes: a visual memory bank and a query memory bank, used to store the visual features of the video and historical query features; A cross-attention mechanism is used to fuse features from the visual memory bank and the query memory bank to generate a temporally aware representation relevant to the current query.

9. The system according to claim 6, characterized in that, The decision-making agent includes: The consistency voting module is used to evaluate the consistency of answers from multiple sources. The answer fusion module is used to generate the final answer based on the consistency assessment results.

10. The system according to claim 6, characterized in that, The training module includes: The cross-entropy loss function is used to calculate the difference between the system output and the true label; The AdamW optimizer is used to optimize system parameters.

Images