Long video question answering method and system based on visual language model and causal reasoning tree
By using a visual language model and causal reasoning tree-based approach, a training-free long-video question-answering system is constructed, solving the efficiency and accuracy problems of long-video causal question answering and achieving an efficient and interpretable question-answering process.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- EAST CHINA NORMAL UNIV
- Filing Date
- 2026-02-09
- Publication Date
- 2026-05-08
AI Technical Summary
Existing long-video question answering technologies suffer from low computational efficiency, blurred focus, and difficulty in modeling causal relationships. Existing models are costly to apply to long videos and have limited generalization capabilities.
By employing a visual language model and causal reasoning tree-based approach, and through video parsing, causal unit extraction, hierarchical aggregation, and lightweight indexing, a training-free causal reasoning tree is constructed to achieve efficient retrieval and reasoning.
It enables structured understanding of long video content, supports efficient and accurate complex question answering, reduces deployment overhead and has plug-and-play capabilities, and the answer generation process has good interpretability.
Smart Images
Figure CN121999415A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, specifically to computer vision, natural language processing and video content understanding technology, and particularly to a long video question answering method and system based on visual language models and causal reasoning trees. Background Technology
[0002] With the widespread adoption of video and surveillance applications, enabling machines to automatically understand long video content and answer complex user questions has become an important research topic. Existing long video question answering technologies face two main challenges: First, long videos contain a massive amount of redundant information, making end-to-end question answering using the entire video computationally inefficient and blurry; second, many questions involve deep reasoning about the causal relationships behind events, requiring models not only to recognize objects and actions but also to understand the logical connections between them.
[0003] Currently, some methods attempt to preprocess videos using video captioning or scene segmentation, but the generated descriptions are often linear and flat, lacking explicit modeling of hierarchical and causal relationships between events, making it difficult to support "why" reasoning. Other studies introduce graph neural networks to model relationships in videos, but this usually requires extensive supervised training for specific tasks, which is costly and has limited generalization ability.
[0004] Furthermore, visual language models and large language models based on large-scale pre-training have demonstrated powerful cross-modal understanding and generation capabilities, showing excellent performance on short video question-answering tasks under 5 minutes. However, these models cannot be effectively applied to long videos of 30 minutes or even longer, due to the O(n) time complexity of the Transformer core module of the large language model. 2 The decision is made based on the existing, untrained foundational models. Therefore, how to effectively organize these readily available, training-free basic models to solve causal question-answering problems in long videos through a structured mechanism remains an underexplored area. Current technologies lack a unified, training-free framework capable of dynamically organizing video content, building causal memories, and supporting efficient and accurate retrieval and reasoning. Summary of the Invention
[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a long video question answering method based on visual language models and causal inference trees. This method achieves a structured understanding of long video content through hierarchical causal inference trees without any training and utilizes lightweight indexes and intelligent agents to achieve more accurate and complex question answering.
[0006] To achieve the above objectives, the present invention adopts the following technical solution: A long-video question answering method based on visual language models and causal inference trees includes the following steps: Step S1: The video parsing module performs time-series segmentation on the input long video to obtain multiple consecutive short video segments; Step S2: The video parsing module calls the visual language model to generate a text description for each short video segment; Step S3: Causal Unit Extraction Unit calls the large language model to parse each text description into structured causal units; Step S4: The hierarchical aggregation unit constructs leaf nodes based on each short video segment and its causal unit, and constructs intermediate nodes and root nodes from bottom to top through semantic clustering to form a hierarchical causal reasoning tree; Step S5: The index building unit generates a text embedding vector for each node in the causal inference tree and builds a global lightweight vector index; Step S6: The intelligent question-answering agent receives user questions and locates key nodes in the causal reasoning tree by querying the vector index; Step S7: Based on the type of question, the intelligent question-answering agent executes the corresponding search strategy on the causal reasoning tree and integrates the information of relevant nodes; Step S8: Based on the integrated information, the intelligent question-answering agent generates and outputs the final natural language answer.
[0007] Furthermore, in step S1, the temporal segmentation adopts uniform segmentation or an adaptive segmentation method based on scene change detection, and the length of the segmented fragments is within 60 seconds.
[0008] Furthermore, in step S4, the aggregation process is implemented by calling a large language model. The large language model generates a content summary and a coherent causal chain description of the parent node based on the description and causal unit of the child node, guided by prompt words.
[0009] Furthermore, in step S6, the lightweight vector index uses the output of a text embedding model as the vector index, and the text embedding model uses a pre-trained model based on the Transformer architecture, such as BERT.
[0010] Furthermore, the search strategy described in step S7 includes: If the question is a global overview type, the global summary and causal chain stored in the root node of the causal reasoning tree are read directly to answer it; if the question is a specific content-focused type, the description, causal unit, and information tracing back to the parent node are read from the relevant nodes returned by the vector index to integrate and answer the question.
[0011] A long-video question-answering system based on a visual language model and a causal reasoning tree includes: Video parsing module: It receives long input videos, divides them into multiple consecutive short video segments, and calls a visual language model to generate text descriptions for each video segment, including visual entities, actions, and events. The causal tree construction module, connected to the video parsing module, is responsible for receiving the text descriptions of all video segments. It calls a large language model to parse each description into one or more causal units representing atomic relationships between subjects, actions, and objects. Based on the semantic and temporal coherence of the causal units, it aggregates multiple nodes into higher-level event nodes from bottom to top, ultimately constructing a hierarchical causal reasoning tree containing leaf nodes, intermediate nodes, and a root node. Simultaneously, it generates an embedding vector of the descriptive text for each node in the causal reasoning tree and constructs a lightweight vector index based on the embedding vector. The intelligent question-answering agent module, connected to the causal tree construction module, receives user questions, locates the set of nodes most relevant to the question in the causal inference tree by querying the lightweight vector index, and adaptively triggers different traversal strategies based on the question type. It performs heuristic search and information integration on the causal inference tree, ultimately generating and outputting the answer. This invention has significant advantages and beneficial effects compared to existing technologies.
[0012] By employing the above technical solution, the long video question-answering system provided by this invention, based on a visual language model and a causal reasoning tree, achieves considerable technological advancement and practicality, and has at least the following advantages: The system described in this invention employs a training-free framework entirely based on pre-trained models. By constructing hierarchical causal reasoning trees and lightweight vector indexes, it effectively solves the challenges of causal reasoning and efficient retrieval in long video understanding. Compared to traditional long video understanding models that require extensive labeled data for training, this invention avoids cumbersome training processes and the risk of overfitting, significantly reduces deployment overhead, and possesses the ability to replace pre-trained models on demand and is plug-and-play. Furthermore, the system uses intelligent agents to perform heuristic searches on the causal trees, accurately supporting various complex question-answering tasks ranging from global overviews to deep causal reasoning, and its answer generation process exhibits good interpretability. Attached Figure Description
[0013] Figure 1 This is a flowchart of the overall process of the present invention; Figure 2 This is a schematic diagram of the overall construction of the causal reasoning tree of the present invention; Figure 3 This is a schematic diagram of the intelligent question-answering agent workflow according to an embodiment of the present invention; Figure 4 This is an interaction diagram between the various modules of the present invention. Detailed Implementation
[0014] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the following embodiments are only for explaining the invention and do not constitute a limitation on the scope of protection of this invention.
[0015] A long-video question answering method based on visual language models and causal inference trees includes the following steps: Step S1: The video parsing module performs time-series segmentation on the input long video to obtain multiple consecutive short video segments; Step S2: The video parsing module calls the visual language model to generate a text description for each short video segment; Step S3: Causal Unit Extraction Unit calls the large language model to parse each text description into structured causal units; Step S4: The hierarchical aggregation unit constructs leaf nodes based on each short video segment and its causal unit, and constructs intermediate nodes and root nodes from bottom to top through semantic clustering to form a hierarchical causal reasoning tree; Step S5: The index building unit generates a text embedding vector for each node in the causal inference tree and builds a global lightweight vector index; Step S6: The intelligent question-answering agent receives user questions and locates key nodes in the causal reasoning tree by querying the vector index; Step S7: Based on the type of question, the intelligent question-answering agent executes the corresponding search strategy on the causal reasoning tree and integrates the information of relevant nodes; Step S8: Based on the integrated information, the intelligent question-answering agent generates and outputs the final natural language answer.
[0016] This invention also relates to a long-video question-answering system based on a visual language model and a causal inference tree. This system requires no fine-tuning of the base model and includes the following three core modules: The video parsing module is responsible for segmenting the input long video into temporally continuous segments. Then, it calls a pre-trained visual language model to generate detailed text descriptions for each segment, including the objects, people, actions, and interactions that appear.
[0017] Causal Tree Construction Module: This module is the core of the system. It receives descriptions of all segments and uses a pre-trained large language model to parse each segment into multiple structured "causal units." Each unit describes an atomic-level causal relationship, such as "A causes B." Then, based on these causal units, the system aggregates segments (leaf nodes) into smaller events (intermediate nodes) from the bottom up, according to semantic relevance and temporal proximity, and further aggregates them into a global narrative (root node), thus constructing a multi-level causal reasoning tree. Simultaneously, it generates text embeddings for each node in the tree and constructs a lightweight vector index based on an efficient approximate nearest neighbor search algorithm to achieve millisecond-level node retrieval.
[0018] Intelligent Question Answering Proxy Module: This module serves as the system's interactive interface. It first understands the user's intent in the question, such as determining whether it's a causal, localization, or global question. Next, it transforms the question into a text embedding vector and quickly locates the K most relevant nodes in the tree by querying the lightweight vector index in the causal tree construction module. Finally, based on the question type, the proxy intelligently determines which part of the tree to search; for example, for causal questions, it traces upwards to either the cause or the effect, integrating information from multiple nodes to generate an accurate and coherent natural language answer. Example
[0019] This embodiment provides a long-video question-answering system based on a visual language model and a causal reasoning tree. The method and steps are as follows: Figure 1 As shown. The system uses Qwen2.5-VL as the visual language model, Qwen2 as the large language model, BERT as the text embedding model, and the HNSW algorithm for vector indexing. The input is a 60-minute video containing a 10-second clip of a water glass breaking on the floor. The user's question to the system is: "Has anything unexpected happened in the office?" The specific implementation steps are as follows: Step S1: The input for this embodiment is a 60-minute long video, including a segment where a water glass is smashed on the ground, which occupies 10 seconds of the complete video. Using a uniform segmentation strategy, the video is divided into approximately 57 consecutive segments, each 64 seconds long. For each segment S... i One frame is sampled per second. These short video clips are used as the original frame sequence for each leaf node.
[0020] Step S2: Input the frame sequences obtained in S1 into the Qwen2.5-VL model, along with the prompt "Please describe in detail the objects, people, actions, and interactions between them in this video", to obtain S for each segment. i Natural Language Description D i .
[0021] Step S3: For each description D obtained in S2 i Input the Qwen2 model and guide it to output structured causal units using carefully crafted prompts. Prompt: "Please extract all explicit causal relationships from the following description. Each relationship should be output strictly according to the format: [Subject] [Action] [Object]. Description: {D i The generated key description is "A person runs quickly toward a table, knocks over a white cup on the table, and the cup shatters on the ground." The model outputs causal units: "[person] [running and colliding] [table]", "[white cup] [knocked over and fell] [ground]", and "[white cup] [impacted the ground] [shattered]". Each video clip, its description, and the causal unit together constitute a leaf node of the causal inference tree.
[0022] Step S4: As Figure 2 As shown, the system starts from all leaf nodes and uses a time-order-based clustering algorithm. Three leaf nodes that are temporally continuous and share the same subject (in this example, "cup"), or exhibit clear causal transmission (in this example, "knocked over" -> "fell"), are aggregated into an intermediate node. The Qwen2 model is invoked to generate a summary for this intermediate node, "the process of a person bumping into a table, causing a cup to break," and a coherent causal chain, based on the descriptions and causal units of these leaf nodes. This process is recursively executed, aggregating lower-level intermediate nodes into higher-level nodes, until a root node is generated, which contains a global summary of the entire video (in this example, "recorded an event where an office accident caused damage to an item").
[0023] Step S5: Generate index text for each node obtained in S4. This text is composed of a node summary, causal chain, and keywords. Use the BERT model to convert the index text of all nodes into embedding vectors. Then, use the HNSW library to build an efficient vector index, adding the vectors of all nodes to it. This index supports approximate nearest neighbor search, enabling it to quickly find the node most similar to the query vector in constant time.
[0024] Step S6: The intelligent question-answering agent receives the user's question "Has anything unexpected happened in the office?", queries the vector index, finds that the most similar index is "cup broken", and locates the key node in the causal reasoning tree.
[0025] Step S7: The intelligent question-answering agent determines the question as "content-focused" and executes the corresponding search strategy on the causal reasoning tree, integrating the information of adjacent nodes and parent nodes, obtaining a complete description of the "cup breaking" event in the parent node, and then integrating the relevant information.
[0026] Step S8: The intelligent question-answering agent inputs the integrated information and user question into the Qwen2 model, generates a fluent and complete final answer and returns it to the user: "An accident did happen in the office. Someone bumped into the table, causing the cup on the table to fall to the ground and break."
[0027] As can be seen from the above embodiments, the system of the present invention effectively solves the problem of causal question answering in long videos through a structured, training-free approach. It has significant advantages in terms of reasoning depth and interpretability, and can replace the visual language model and large language model or fine-tune the model, thus having the advantage of plug-and-play functionality.
[0028] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.
Claims
1. A long-video question answering method based on visual language models and causal reasoning trees, comprising the following steps: Step S1: The video parsing module receives the long video, performs time-series segmentation, and obtains multiple consecutive short video segments; Step S2: Invoke the visual language model to generate a text description for each short video segment; Step S3: Causal Unit Extraction Unit calls the large language model to parse each text description into structured causal units; Step S4: Formation of hierarchical causal reasoning tree: The hierarchical aggregation unit constructs leaf nodes based on each short video segment and its causal unit, and constructs intermediate nodes and root nodes from bottom to top through semantic clustering to form a hierarchical causal reasoning tree; Step S5: The index building unit generates a text embedding vector for each node in the causal inference tree and builds a global lightweight vector index; Step S6: The intelligent question-answering agent receives user questions and locates key nodes in the causal reasoning tree by querying the vector index; Step S7: Based on the type of question, the intelligent question-answering agent executes the corresponding search strategy on the causal reasoning tree and integrates the information of relevant nodes; Step S8: Based on the integrated information, the intelligent question-answering agent generates and outputs the final natural language answer.
2. The long video question answering method based on a visual language model and causal reasoning tree according to claim 1, characterized in that, The temporal segmentation described in step S1 uses either uniform segmentation or an adaptive segmentation method based on scene change detection, and the length of the segmented fragments is within 60 seconds.
3. The long video question-answering method based on a visual language model and causal inference tree according to claim 1, characterized in that, The aggregation process described in step S4 is implemented by calling a large language model. The large language model generates a content summary and a coherent causal chain description of the parent node based on the description and causal unit of the child node, guided by prompt words.
4. The long video question-answering method based on a visual language model and causal reasoning tree according to claim 1, characterized in that, The lightweight vector index in step S5 uses the output of the text embedding model as the vector index, and the text embedding model is a pre-trained model based on the Transformer architecture.
5. The long video question answering method based on a visual language model and causal inference tree according to claim 1, characterized in that, The search strategy described in step S7 includes: If the question is a global overview type, the global summary and causal chain stored in the root node of the causal reasoning tree are read directly to answer it; if the question is a specific content-focused type, the description, causal unit, and information tracing back to the parent node are read from the relevant nodes returned by the vector index to integrate and answer the question.
6. A long-video question-answering system based on a visual language model and a causal reasoning tree, characterized in that, The system includes: Video parsing module: It receives long input videos, divides them into multiple consecutive short video segments, and calls a visual language model to generate text descriptions for each video segment, including visual entities, actions, and events. The causal tree construction module, connected to the video parsing module, is responsible for receiving text descriptions of all video segments. The causal unit extraction unit calls a large language model to parse each description into one or more causal units representing atomic relationships between subjects, actions, and objects. Based on the semantic and temporal coherence of the causal units, multiple nodes are aggregated into higher-level event nodes from bottom to top, ultimately constructing a hierarchical causal reasoning tree containing leaf nodes, intermediate nodes, and a root node. Simultaneously, an embedding vector of the descriptive text is generated for each node in the causal reasoning tree, and a lightweight vector index is constructed based on the embedding vector. Intelligent question answering agent module: connected to the causal tree construction module, used to receive user questions, locate the set of nodes most relevant to the question in the causal inference tree by querying the lightweight vector index, and adaptively trigger different traversal strategies according to the question type, perform heuristic search and information integration on the causal inference tree, and finally generate and output the answer.
7. A long video question-answering system based on a visual language model and a causal inference tree according to claim 6, characterized in that, The causal tree construction module specifically includes: Causal unit extraction unit: This unit is used to input the text description of the video clip into the large language model. The model output format is guided by structured prompts: [Subject] Causal units that cause / affect [Object / Result] due to [Action / State]. Hierarchical aggregation unit: used to cluster leaf nodes representing video segments based on the continuity of the main body in their causal units and the causal and temporal tightness of events, forming intermediate nodes representing sub-scenes or composite events; and recursively aggregate intermediate nodes to form root nodes representing global video content. Index building unit: Used to generate high-dimensional vectors for each node's summary, causal chain, and keyword information using a text embedding model, and to build a full-tree vector index using an approximate nearest neighbor search algorithm.
8. A long video question-answering system based on a visual language model and a causal inference tree according to claim 6, characterized in that, The intelligent question-answering agent module also includes: Question classification unit: Used to input user questions and specific prompts into the large language model, classify questions into global overview type or specific content focus type, and design different causal tree search strategies; Index retrieval unit: used to calculate the embedding vector of the user question, and quickly retrieve the K most relevant nodes in the causal reasoning tree by querying the lightweight vector index, as the nodes to be queried later; Strategy Execution Unit: Used to execute the corresponding tree search strategy based on the results of the question classification unit; for global overview questions, the root node is directly accessed to obtain the summary; for specific content-focused questions, the node returned by the index retrieval unit is used as the starting point to trace the cause and effect upwards along the tree structure and integrate information from multiple nodes to form the answer.
9. A long video question-answering system based on a visual language model and a causal inference tree according to claim 6, characterized in that, The system uses both a visual language model and a large language model, both of which are pre-trained models. It relies on the knowledge and prompt words of the pre-trained models for reasoning.
10. A long video question-answering system based on a visual language model and a causal inference tree according to claim 7, characterized in that, The causal unit extraction unit further includes a causal unit: used to store structured information output by the causal unit extraction unit, including subject, action / state, object / result.