A two-stage inference-based video question answering method and system

Abstract

The application belongs to but is not limited to the cross-modal learning field of computer vision and natural language processing, and discloses a video question answering method based on two-stage reasoning, which solves the complex space-time modeling problem in the video question answering task through the iterative idea of 'initial solution-refinement'. In the first stage, the video frame sequence is decomposed into independent image question answering tasks, and a pre-trained multi-modal model is used to generate a preliminary semantic description and predict answer features; in the second stage, the original video features, general description features and the predicted features in the first stage are element-level fused, a language-guided residual cross-attention module is used to perform secondary reasoning, and a time sequence mean pooling is combined to output the final answer. Experiments show that the application achieves an accuracy of 54.9% on the MSVD-QA dataset, and the trainable parameter amount is only 7M, which significantly reduces the computing resource demand and improves the interpretability of reasoning.

Description

Technical Field

[0001] This invention belongs to (but is not limited to) the field of cross-modal learning in computer vision and natural language processing, and particularly relates to a video question answering method and system based on two-stage reasoning. Background Technology

[0002] Video question answering (VideoQA), as a core task of multimodal understanding, has attracted widespread attention in recent years. Existing research mainly focuses on three aspects: methods based on traditional deep learning, methods based on graph neural networks, and methods based on vision-language pre-training.

[0003] Early VideoQA methods often employed a combination of CNN and RNN architectures. They used convolutional neural networks to extract frame-level features and recurrent neural networks (such as LSTM) to encode the video and questions. Finally, they generated the answer through attention mechanisms or multimodal fusion modules. However, simple sequence models struggle to capture the complex interactions and spatiotemporal structures of objects in videos.

[0004] To better model fine-grained relationships in videos, graph neural networks (GNNs) have been widely adopted. For example, HGA aligns video shots and question words using heterogeneous graphs for cross-modal reasoning. In recent years, numerous improvements to GNNs have emerged: STCT proposed the Spatiotemporal Graph Convolution Transformer, which enhances dynamic interaction by explicitly modeling the spatiotemporal relationships between visual objects; KRST introduced a keyword-aware relevance spatiotemporal graph, guiding graph construction with keywords from the question and modeling the relative relationships between objects; HSSHG designed a heuristically semantically constrained spatiotemporal heterogeneous graph, using plot summaries and object positions as priors to constrain edge weights. While these graph-based methods perform well in structured reasoning, their graph construction processes are often complex, and they still have limitations when dealing with long-tailed distributions or semantic ambiguity.

[0005] With the rise of large-scale pre-trained models (such as CLIP and BLIP), vision-language pre-training (VLP) methods have gradually become mainstream. These methods utilize massive amounts of text-image pairs for contrastive learning or generative pre-training, significantly bridging the semantic gap between multimodalities. For example, VisualBERT and VideoBERT apply the Transformer architecture to multimodal sequences. FrozenBiLM proposed a zero-shot VideoQA framework that achieves efficient cross-modal transfer by freezing the bidirectional language model and training only a lightweight adapter. Furthermore, V-CAT enhances the semantic and content alignment between videos and questions by introducing contextual information and utilizing contrastive learning. Although pre-trained models provide powerful feature representations, directly applying them to complex VideoQA tasks often lacks deep reasoning capabilities for video temporal logic, and end-to-end training computation is expensive.

[0006] In summary, current video question-answering technology has the following main problems:

[0007] Insufficient reasoning depth: Most current VideoQA models adopt a single-stage reasoning architecture, which results in poor performance when dealing with deep semantic problems.

[0008] Performance and cost imbalance: Although methods based on large model pre-training (VLP) and complex graph neural networks (GNN) have improved performance, their computational and resource costs are too high, making it difficult to balance efficiency. Summary of the Invention

[0009] To address the problems existing in the prior art, this invention provides a video question answering method based on two-stage reasoning.

[0010] This invention is implemented as follows: a video question-answering method based on two-stage reasoning, the method comprising:

[0011] S1: Video Question Answering Dataset Preprocessing. In the data preprocessing stage, sparse sampling is performed on the original video to obtain a frame sequence, and an encoder is used to extract the visual features of the video and the textual features of the questions;

[0012] S2: Single-frame image question answering and description generation. The first stage of reasoning deconstructs the sampled video frames and input questions into multiple independent image question answering tasks. A pre-trained large language model is used to generate a preliminary semantic description for each frame, and the description is encoded to obtain the initial solution features.

[0013] S3: Multi-source feature fusion and temporal reasoning. In the feature pre-fusion stage, the visual appearance features obtained in step S1 are combined with the initial solution features and general image title features obtained in step S2 through element-level addition and fusion to obtain multi-source fused features;

[0014] S4: Two-stage video question answering model training and validation. The second stage of reasoning uses question text features as guidance to perform cross-attention reasoning on multi-source fusion features, captures video temporal context information, and generates enhanced frame-by-frame features; temporal aggregation and answer prediction: temporal pooling is performed on the enhanced frame-by-frame features, the cosine similarity between the obtained video representation and the candidate answer set is calculated, and the final answer is output.

[0015] Further, in step S1, the number of sampling frames L is set to 16, and the visual encoder of the CLIP model is used to extract D-dimensional appearance embedding features. .

[0016] Furthermore, S1 specifically includes:

[0017] For videos, considering the variation in video length, L frames are uniformly and sparsely extracted for each video, i.e., L≪|V|, where |V| is the length of the video; for each sampled frame, CLIP is used to extract features as the frame appearance embedding. = ,in This represents the l-th frame in the sparsely sampled video; the feature size is D, consistent with the CLIP setting.

[0018] For text, to obtain a visually aligned language representation, CLIP's text encoder BERT was used to extract different embeddings; the [EOT] embedding was extracted for the question. ∈ As a global representation; for all answer candidates, extract sentence-level embeddings. = Where |A| is the number of candidate options for each question; to supplement additional linguistic information, a pre-trained image title converter is used to convert all frames into titles and extract features, resulting in = .

[0019] Furthermore, in step S2, a BLIP-2 model with frozen parameters is used as the initial inference engine, through a condition generation process. We obtain a natural language description for each frame and map it to a high-dimensional embedding space using the CLIP text encoder.

[0020] Furthermore, in step S2, in the first stage of the two-stage architecture, the goal is to utilize the zero-shot reasoning capability of the pre-trained Large Language Model (LLM) to generate a preliminary semantic description related to the question for each frame in the video; this step initially transforms high-dimensional visual information into a compact text semantic space, thereby alleviating the pressure of directly performing long video temporal modeling.

[0021] Furthermore, step S3 specifically includes:

[0022] (1) Multi-source feature pre-fusion

[0023] To fully utilize semantic information from different dimensions, we first perform element-level fusion on three features of the same dimension; given the original video appearance features... General image description features and the strongly related question-guided descriptive features generated in the first phase The fusion process is as follows:

[0024] ,

[0025] This additive fusion approach can initially align the original visual signals, general semantics, and problem-oriented semantics, providing a rich contextual basis for subsequent attention mechanisms;

[0026] (2) Language-guided cross-attention reasoning

[0027] To further enhance the model's focus on the core elements of the problem, this invention introduces a cross-attention module based on residual connections; in this module, the fused video frame features... As a query vector, and global problem features As key vectors and value vectors:

[0028] ,

[0029] This design allows the fused features of each frame to be dynamically adjusted according to the global semantics of the question; through cross attention, the model can automatically identify and enhance those frame features that contribute more to answering the current question, while suppressing irrelevant noise.

[0030] (3) Temporal aggregation and final representation

[0031] Obtain enhanced frame-by-frame features Then, they need to be aggregated into a unified video-level representation; to balance computational efficiency and performance, a temporal average pooling strategy is adopted. Normalization is performed on L sampled frames:

[0032] ,

[0033] in, Represents the enhanced frame-by-frame features The feature vector of the l-th frame in the dataset.

[0034] The final video representation It integrates cross-modal reasoning results; during the reasoning stage, this invention calculates the similarity between this feature and the encoded features of the candidate answer set;

[0035] For open-ended question-and-answer or multiple-choice tasks, this invention utilizes computational video representation. Features of all candidate answers The cosine similarity between the candidates is used to select the candidate with the highest score as the final output.

[0036] .

[0037] in, Represents the features of all candidate answers The Middle The encoded feature representation corresponding to each candidate answer.

[0038] Furthermore, in step S4, a cross-attention module based on residual connections is used, wherein the fused features... As a query vector, global problem characteristics The key vector Key and value vector V are given as the global video representation generated in the second stage. Feature representation of the candidate answer set (in (To find the total number of candidate answers), first calculate the cosine similarity score between the video features and the features of each candidate answer. ,in, Represents the set of candidate answers The Middle The feature representation of each answer. This score reflects the degree of matching between the video semantics and a specific candidate answer;

[0039] (2) Temperature coefficient and probability distribution

[0040] To control the sharpness of the predicted distribution and enhance the model's ability to mine difficult samples, a learnable temperature parameter was introduced. During the experimental initialization phase, Set to 0.04; predict the video corresponding to the [number]th [section]. Probability distribution of candidate answers Calculated using the Softmax function: ,in, Representing video features and the first The and the first A similarity score between candidate answers. This is calculated by dividing the similarity score by a smaller factor. The model can amplify the difference in scores between correct and incorrect predictions, thereby accelerating convergence;

[0041] (3) Optimization Objective

[0042] Finally, standard cross-entropy loss is used as the overall optimization objective; for a training sample with a batch size of B, the loss function is... Defined as: in Indicates the first Each sample corresponds to a real answer index; in this way, the model not only learns how to generate video descriptions related to the question, but also learns how to accurately align video content with the correct answer in a multimodal space.

[0043] Another object of the present invention is to provide a computer device including a memory and a processor, the memory storing a computer program, which, when executed by the processor, causes the processor to perform the steps of the video question answering method based on two-stage reasoning.

[0044] Another object of the present invention is to provide a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the steps of the video question-answering method based on two-stage reasoning.

[0045] Based on the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by this invention are as follows:

[0046] First, this invention addresses the shortcomings of existing video question answering methods, such as "insufficient reasoning depth and performance-cost imbalance," by providing a two-stage reasoning-based video question answering method, specifically including:

[0047] Regarding reasoning accuracy: This invention effectively solves the problem of logical discontinuity in traditional models when processing complex long videos through a two-stage reasoning framework. By using ImageQA prior knowledge as the reasoning anchor, it significantly enhances the model's understanding of video content and the accuracy of its answers.

[0048] Regarding the number of parameters: The lightweight fusion module proposed in this invention, compared with traditional bilinear pooling or complex Transformer fusion layers, reduces the number of model parameters by 93% while maintaining high performance through simplified attention and mean pooling strategies, which greatly lowers the hardware deployment threshold and meets the real-time requirements of mobile or edge computing devices.

[0049] Regarding interpretability and transparency: This invention decouples the complex video question-answering task into two explicit steps—"static visual perception" and "dynamic logical deduction"—through a two-stage reasoning architecture, making the intermediate reasoning results observable and verifiable. This layered design transforms the end-to-end "black box" model into a logically clear white box reasoning process, facilitating subsequent error tracing and model optimization.

[0050] Table 1 Performance comparison on the MSVD-QA dataset

[0051]

[0052] Note: The representative used a large-scale video-language pre-training method; This represents a method that uses CLIP or BLIP as the visual backbone network.

[0053] like Figure 2 As shown, to verify the effectiveness of C-BLIP, we conducted a broad comparison with mainstream methods in recent years on the MSVD-QA dataset. These comparison methods cover spatiotemporal graph convolution-based methods (KRST, HSSHG), attention-based and context-aligned methods (PEANUT, V-CAT), complex bidirectional inference networks (CACR), and methods based on fine-tuning of large-scale pre-trained models (LGVA, FrozenBiLM).

[0054] Experimental results show that our C-BLIP achieves an accuracy of 54.9% on the MSVD-QA dataset, surpassing all comparable methods and reaching the current state-of-the-art (SOTA) level, while achieving a minimum parameter count of 7M.

[0055] Compared to traditional spatiotemporal reasoning methods, C-BLIP demonstrates a significant performance advantage (approximately 8.6%-13.4% improvement) compared to methods such as KRST (41.5%), PEANUT (42.8%), and HSSHG (46.3%). While these methods employ sophisticated graph networks or attention modules to capture spatiotemporal relationships, they are often limited by the semantic gap between visual and textual features, and struggle to handle complex open-domain question answering without large-scale graph-text pre-training guidance. In contrast, C-BLIP leverages the powerful multimodal alignment capabilities of CLIP and BLIP, effectively addressing this shortcoming through a two-stage strategy.

[0056] Comparing semantic alignment and reasoning enhancement methods: V-CAT (45.2%) enhances video context alignment through contrastive learning, while CACR (51.1%) introduces bidirectional semantic reasoning to eliminate spurious associations. Although CACR achieved good results, C-BLIP still leads by 3.8%. This suggests that instead of designing complex bidirectional reasoning modules, it is more effective to adopt our proposed "initial solution-refinement" strategy: first, use image question answering capabilities to obtain intuitive answers, and then introduce video context for correction. This iterative process, which conforms to human cognitive habits, is more effective in the VideoQA task.

[0057] Comparing large-scale pre-training and fine-tuning methods: The most noteworthy comparison is with LGVA (52.8%) and FrozenBiLM (54.8%). LGVA also uses CLIP features, but relies primarily on language-guided visual aggregation, neglecting the iterative optimization process for answer generation. FrozenBiLM, on the other hand, is a strong baseline pre-trained on a large-scale video-language platform. Our C-BLIP narrowly outperforms FrozenBiLM with 54.9% accuracy, establishing its leadership in MSVD-QA. This demonstrates that directly adapting image question-answering models to video tasks (stage 1) and supplementing them with lightweight spatiotemporal context fusion (stage 2) can capture more crucial question-answering cues than blindly performing large-scale end-to-end fine-tuning.

[0058] Secondly, as supplementary evidence of the inventive step of the claims of this invention, it is also reflected in the following important aspects:

[0059] (1) The expected benefits and commercial value of the technical solution of this invention after transformation are as follows:

[0060] The lightweight and high-precision characteristics of this invention give it enormous commercial potential and a very high return on investment in the following areas:

[0061] 1. Significantly Reduced Computing Costs and Operating Expenses: Compared to existing mainstream large-scale video pre-trained models (such as FrozenBiLM), this invention reduces the number of model parameters to 7M. ​​In actual commercial deployments, this translates to a significant reduction in cloud inference costs. For short video platforms or content moderation service providers that process hundreds of millions of videos daily, adopting this solution can save millions in GPU server procurement and energy consumption costs.

[0062] 2. Empowering Edge Computing and Mobile Applications: By eliminating complex graph computations and bulky Transformer stacks, this solution's lightweight nature makes real-time video semantic understanding possible on mobile terminals (phones, automotive chips, drones). This provides core algorithmic support for hardware products with strong real-time and offline processing requirements, such as smart home interaction, autonomous driving scene understanding, and assistive glasses for the blind, demonstrating extremely high embedded market value.

[0063] 3. Improved conversion rates for video retrieval and content distribution: This invention achieves state-of-the-art (SOTA) performance on datasets such as MSVD-QA, signifying more accurate video content understanding capabilities. When applied to video search engines, intelligent ad delivery, and personalized recommendation systems, it can significantly improve the click-through rate (CTR) of user search results and the accuracy of ad delivery, thereby directly boosting the platform's traffic monetization capabilities.

[0064] (II) The technical solution of this invention fills a technological gap both domestically and internationally:

[0065] 1. This invention fills the technological gap in low-cost migration from large-scale static image and text models to dynamic video tasks: Currently, the industry mostly uses methods such as "pre-training from scratch" or "full fine-tuning" to adapt image models such as CLIP / BLIP to videos, which consumes huge computational resources. This invention innovatively proposes a zero-shot / few-shot transfer paradigm based on two-stage inference, filling the technological gap of efficiently utilizing prior knowledge of static images to process dynamic video tasks without large-scale video pre-training, using only a lightweight adaptation module.

[0066] 2. This invention fills the gap in the field of "interpretable reasoning architecture" for video question answering: Existing video question answering technologies are mostly end-to-end "black box" models, lacking transparency in the intermediate reasoning process. This invention constructs a white-box reasoning link of "visual perception - initial semantic understanding - logical correction" by fusing the "intermediate semantic description" generated in the first stage with the video semantic features in the second stage. This fills the technical gap in logical tracing and interpretability in high-precision video understanding models, and is of great significance for fields with extremely high security requirements such as medical video analysis and security monitoring.

[0067] (III) The technical solution of this invention solves a technical problem that people have long desired to solve but have never been able to achieve:

[0068] 1. Overcoming the paradox of "high accuracy and low parameter count being mutually exclusive": For a long time, the field of video understanding has faced a common technical bottleneck: improving accuracy requires stacking parameters and computing power (such as introducing large graph networks or increasing the number of Transformer layers), while lightweighting inevitably sacrifices accuracy. This invention, through an ingenious two-stage design, successfully breaks this "zero-sum game" between performance and efficiency, surpassing the performance of complex models with several times more parameters for the first time using only 7M parameters, thus solving this long-standing performance balance problem that has plagued the industry.

[0069] 2. This invention addresses the issues of "semantic forgetting" and "logical discontinuity" in long-form video reasoning: When processing long videos or complex temporal logic, traditional RNNs or simple Transformers are prone to losing key information as the sequence grows (semantic forgetting). There has been a long-standing desire to find a method that can effectively capture fine-grained spatiotemporal cues. This invention introduces a "question-guided cross-attention mechanism" combined with a "coarse-to-fine" reasoning strategy, allowing the model to actively ignore redundant frames and focus on key frames highly relevant to the question. This effectively solves the problem of logical reasoning failure caused by information overload in long-form video understanding. Attached Figure Description

[0070] Figure 1This is a flowchart of a video question-answering method based on two-stage reasoning provided in an embodiment of the present invention; Figure 2 This is a C-BLIP architecture diagram provided in an embodiment of the present invention; Figure 3 This is a flowchart of the multi-source feature fusion and temporal reasoning method provided in the embodiments of the present invention; Figure 4 This is a performance comparison chart on the MSVD-QA dataset provided in this embodiment of the invention. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0072] like Figure 1 As shown, this embodiment of the invention provides a video question-answering method based on two-stage reasoning, the method comprising:

[0073] S1: Video Question Answering Dataset Preprocessing. In the data preprocessing stage, sparse sampling is performed on the original video to obtain a frame sequence, and an encoder is used to extract the visual features of the video and the textual features of the questions;

[0074] S2: Single-frame image question answering and description generation. The first stage of reasoning deconstructs the sampled video frames and input questions into multiple independent image question answering tasks. A pre-trained large language model is used to generate a preliminary semantic description for each frame, and the description is encoded to obtain the initial solution features.

[0075] S3: Multi-source feature fusion and temporal reasoning. In the feature pre-fusion stage, the visual appearance features obtained in step S1 are combined with the initial solution features and general image title features obtained in step S2 through element-level addition and fusion to obtain multi-source fused features;

[0076] S4: Two-stage video question answering model training and validation. The second stage of reasoning uses question text features as guidance to perform cross-attention reasoning on multi-source fusion features, captures video temporal context information, and generates enhanced frame-by-frame features; temporal aggregation and answer prediction: temporal pooling is performed on the enhanced frame-by-frame features, the cosine similarity between the obtained video representation and the candidate answer set is calculated, and the final answer is output.

[0077] In step S1, the number of sampling frames L is set to 16, and the visual encoder of the CLIP model is used to extract D-dimensional appearance embedding features. .

[0078] S1 specifically includes:

[0079] For videos, considering the variation in video length, L frames are uniformly and sparsely extracted for each video, i.e., L≪|V|, where |V| is the length of the video; for each sampled frame, CLIP is used to extract features as the frame appearance embedding. = ,in This represents the l-th frame in the sparsely sampled video; the feature size is D, consistent with the CLIP setting.

[0080] For text, to obtain a visually aligned language representation, CLIP's text encoder BERT was used to extract different embeddings; the [EOT] embedding was extracted for the question. ∈ As a global representation; for all answer candidates, extract sentence-level embeddings. = Where |A| is the number of candidate options for each question; to supplement additional linguistic information, a pre-trained image title converter is used to convert all frames into titles and extract features, resulting in = .

[0081] In step S2, a BLIP-2 model with frozen parameters is used as the initial inference engine, through a condition generation process. Natural language descriptions for each frame are obtained, and the CLIP text encoder is used to map them to a high-dimensional embedding space to obtain preliminary response features. ,Right now:

[0082] ,

[0083] In step S2, the first stage of the two-stage architecture aims to utilize the zero-shot reasoning capability of the pre-trained Large Language Model (LLM) to generate a preliminary semantic description related to the question for each frame in the video. This step initially transforms high-dimensional visual information into a compact text semantic space, thereby alleviating the pressure of directly performing long video temporal modeling.

[0084] like Figure 3 As shown, step S3 specifically includes:

[0085] S31: Multi-source feature prefusion

[0086] To fully utilize semantic information from different dimensions, we first perform element-level fusion on three features of the same dimension; given the original video appearance features... General image description features and the strongly related question-guided descriptive features generated in the first phase The fusion process is as follows:

[0087] ,

[0088] This additive fusion approach can initially align the original visual signals, general semantics, and problem-oriented semantics, providing a rich contextual basis for subsequent attention mechanisms;

[0089] S32: Language-guided cross-attention reasoning

[0090] To further enhance the model's focus on the core elements of the problem, a cross-attention module based on residual connections was introduced; in this module, the fused video frame features... As a query vector, and global problem features As key vectors and value vectors:

[0091] ,

[0092] This design allows the fused features of each frame to be dynamically adjusted according to the global semantics of the question; through cross attention, the model can automatically identify and enhance those frame features that contribute more to answering the current question, while suppressing irrelevant noise.

[0093] S33: Temporal Aggregation and Final Representation

[0094] Obtain enhanced frame-by-frame features Then, they need to be aggregated into a unified video-level representation; to balance computational efficiency and performance, a temporal average pooling strategy is adopted. Each sampled frame is normalized:

[0095] ,

[0096] The final video representation It incorporates cross-modal reasoning results; during the reasoning phase, we calculate the similarity between this feature and the encoded features of the candidate answer set;

[0097] For open-ended question-and-answer or multiple-choice tasks, video representation is computed. Features of all candidate answers The cosine similarity between the candidates is used to select the candidate with the highest score as the final output.

[0098] .

[0099] In step S4, a cross-attention module based on residual connections is used, wherein the fused features As a query vector, global problem characteristics As a key vector (Key) and a value vector (Value).

[0100] Step S4 specifically includes:

[0101] (1) Similarity score calculation

[0102] Given the global video representation generated in the second stage Feature representation of the candidate answer set (in (To find the total number of candidate answers), first calculate the cosine similarity score between the video features and the features of each candidate answer. This score reflects the degree of match between the video's semantics and a specific candidate answer;

[0103] (2) Temperature coefficient and probability distribution

[0104] To control the sharpness of the predicted distribution and enhance the model's ability to mine difficult samples, a learnable temperature parameter was introduced. During the experimental initialization phase, Set to 0.04; predict the video corresponding to the [number]th [section]. Probability distribution of candidate answers Calculated using the Softmax function: By dividing the similarity by a smaller The model can amplify the difference in scores between correct and incorrect predictions, thereby accelerating convergence;

[0105] (3) Optimization Objective

[0106] Finally, standard cross-entropy loss is used as the overall optimization objective; for a training sample with a batch size of B, the loss function is... Defined as: in Indicates the first Each sample corresponds to a real answer index; in this way, the model not only learns how to generate video descriptions related to the question, but also learns how to accurately align video content with the correct answer in a multimodal space.

[0107] Example 1: Two-stage inference implementation based on a general video question-answering dataset

[0108] In this embodiment, a video question-and-answer dataset containing video, question text, and candidate answers is selected as the input sample.

[0109] First, uniform sparse sampling is performed on each video, extracting 16 frames at equal intervals from the complete video sequence as the analysis object. For each sampled frame, its visual appearance features are extracted, and a global semantic representation of the input question is also extracted.

[0110] Subsequently, each frame is combined with the question into an independent image question-answering input. A pre-trained language generation model with frozen parameters is used to generate a natural language description that is directly related to the semantics of the question, and this description is encoded into a high-dimensional semantic vector.

[0111] In the feature fusion stage, the original visual appearance features, general image semantic features, and question-guided descriptive features are summed element by element to form multi-source aligned fused features.

[0112] Guided by the semantics of the question, cross-attention inference is performed on the fused features to strengthen the frame information related to the question. Finally, a video-level representation is obtained through temporal mean aggregation, and the answer is predicted.

[0113] This implementation enables accurate video question-answering reasoning without explicitly modeling the complete video timing.

[0114] Example 2: Decomposition-based Semantic Reasoning Implementation for Long Video Scenarios

[0115] This embodiment is designed for application scenarios where the video length is much greater than the range that conventional time series models can process.

[0116] By employing sparse sampling, long videos are compressed into a fixed number of keyframes, allowing each frame to participate in reasoning as an independent semantic unit. In the first stage, semantic descriptions guided by frame-by-frame questions are generated, mapping high-dimensional visual content to a compact textual semantic space.

[0117] The second stage no longer processes the original video sequence directly, but performs cross-frame attention inference based on the already aligned question-related semantic features.

[0118] This approach effectively avoids the computational complexity explosion problem caused by direct temporal modeling of long videos, while maintaining continuous attention to the key semantics of the problem.

[0119] Example 3: Implementation method for multi-select video question answering tasks

[0120] In this embodiment, multiple candidate answers are set for each question.

[0121] After obtaining the video-level semantic representation, semantic encoding is performed on all candidate answers, and the similarity between the video representation and the representation of each candidate answer is calculated.

[0122] By normalizing the similarity distribution, the matching probability between the video and each candidate answer is obtained, and the candidate answer with the highest probability is selected as the output result.

[0123] This implementation fully leverages the advantages of multimodal semantic alignment, enabling the video content, question semantics, and answer semantics to be discriminated within a unified space.

[0124] Example 4: Training Implementation Method Using Temperature Regulation Mechanism

[0125] During the model training phase, a learnable temperature parameter is introduced to adjust the similarity distribution.

[0126] By dividing the similarity between the video representation and the candidate answer representation by the temperature parameter, the difference between the correct and incorrect answers is amplified in the probability space.

[0127] By combining the cross-entropy loss function for end-to-end optimization, the model can accelerate convergence and improve discrimination ability while maintaining inference stability.

[0128] This implementation improves the model's discrimination accuracy on complex samples.

[0129] Example 5: Implementation of Coordination between General Image Semantics and Question-Guided Semantics

[0130] In this embodiment, in addition to the question-guided descriptive semantics, general image semantic features generated from video frames are also introduced.

[0131] By using element-level fusion, the original visual information, general semantic information, and problem-related semantics are aligned within the same feature space.

[0132] In subsequent cross-attention inference, the question semantics uniformly schedules the three types of information, thereby taking into account both the overall semantics of the video and the question-specific semantics.

[0133] This collaborative mechanism significantly improves the robustness of the model in complex semantic scenarios.

[0134] Example 6: Systematically Deployed Video Question Answering Implementation Method

[0135] In this embodiment, the above method is deployed as a modular system structure, including a data preprocessing module, a single-frame semantic generation module, a multi-source feature fusion module, a question-guided reasoning module, and an answer prediction module.

[0136] The modules are connected through standardized feature interfaces, supporting independent deployment or joint operation on different computing power platforms.

[0137] This systematic implementation approach enables the two-stage reasoning mechanism to be stably applied to practical video question-answering systems, demonstrating good engineering feasibility.

[0138] Evidence related to the technical effects obtained by the embodiments of the present invention:

[0139] To analyze in depth the effectiveness of each module in the C-BLIP framework and its contribution to model performance, we conducted a series of ablation experiments on the MSVD-QA validation set.

[0140] 1. BLIP-2 Feature Validity Analysis

[0141] The core idea of ​​C-BLIP is to utilize the image question-answering capabilities of BLIP-2 as the initial solution in the first stage. We explored the impact of introducing image captions generated by BLIP-2 and BLIP-2 VQA predicted features on the final accuracy. The baseline model only uses visual features and question features extracted by CLIP. As shown in Table 2, the accuracy of the baseline model is 49.3%. The effect of image captions is evident: after introducing only image caption features, the accuracy improved to 52.9%. This indicates that the scene description in natural language form supplements the high-level semantic information missing in the visual features.

[0142] The dominance of VQA features: After introducing only VQA features (i.e., the predicted answer embedding in the first stage), the accuracy jumped significantly to 56.3%. This strongly proves our core hypothesis: inputting the "preliminary answer for a single frame" as prior knowledge into the second stage can greatly reduce the difficulty of reasoning.

[0143] Combined effect: When both are introduced simultaneously, performance increases slightly to 56.4%. Although VQA features already provide most of the key information, the image caption still provides subtle background supplementation.

[0144] Table 2. The impact of different features of BLIP-2 on accuracy.

[0145]

[0146] 2. Impact of Time-Sequence Pooling Strategy

[0147] In the second stage of video context fusion, it is crucial to aggregate frame-level features into video-level representations. We compare four common pooling strategies: Attention Pooling, Mean Pooling, LSTM+Mean Pooling, and Max Pooling.

[0148] The experimental results (Table 3) show that mean pooling achieved the highest accuracy (56.4%).

[0149] Simplicity is effectiveness: Although mean pooling is simple, it can retain information from all frames in a balanced way, avoiding the loss of key clues due to incorrect weight allocation by the attention mechanism.

[0150] Overfitting of complex models: Introducing LSTM actually led to a performance drop to 54.0%. We speculate that superimposing complex recurrent neural networks on high-level semantic features extracted by CLIP and BLIP can easily cause overfitting on small datasets and may also destroy the original linear separability of the feature space.

[0151] Table 3. Impact of different pooling methods on accuracy

[0152]

[0153] 3. Comparison of Feature Fusion Methods

[0154] In the model architecture, we need to fuse the "initial answer features from the first stage" with the "video context features from the second stage." We tested three fusion operations: element-wise summation, element-wise product, and concatenation. As shown in Table 4, element-wise summation performed best (56.4%). Element-wise summation is similar to a residual connection, effectively adding contextual information as a "correction term" to the initial answer features, preserving not only the original inference result but also enriching the semantics. Element-wise product performed the worst (39.8%), possibly because the feature vector contains zero or minimum values, leading to gradient vanishing or information loss during multiplication. While feature concatenation preserves all information, it increases the number of parameters in subsequent fully connected layers, increasing training difficulty and resulting in slightly lower performance than element-wise summation.

[0155] Table 4. Impact of different fusion methods on accuracy

[0156]

[0157] 4. The necessity of supervised training

[0158] Finally, to verify that C-BLIP does not solely rely on the zero-shot capability of pre-trained models, we compared the performance of direct inference (unsupervised) with that of supervised fine-tuning.

[0159] Table 5 shows that the untuned model achieved only 22.8% accuracy, while supervised training improved this to 56.4%. This indicates that although CLIP and BLIP possess strong feature representation capabilities, cross-modal mapping relationships and temporal context reasoning specific to the VideoQA task still require supervised learning to establish. C-BLIP's lightweight training module effectively accomplishes this task.

[0160] Table 5. Impact of supervised training on model performance

[0161]

[0162] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented using hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or using software executed by various types of processors, or using a combination of the above-described hardware circuitry and software, such as firmware.

[0163] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention and within the spirit and principles of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A video question-answering method based on two-stage reasoning, characterized in that, Includes the following steps: The input video is sparsely sampled to obtain a frame sequence consisting of multiple video frames, and the visual appearance features of each video frame and the question text features corresponding to the video are extracted respectively. Using each video frame and the question as input, multiple independent single-frame image question answering tasks are constructed. Through a pre-trained language generation model, a natural language description related to the semantics of the question is generated for each video frame, and the natural language description is mapped to the first-stage semantic features. The visual appearance features, general image semantic features, and the first-stage semantic features are element-level summed and fused to form multi-source fusion features; Using the multi-source fusion features as query information and the question text features as guiding information, enhanced frame-by-frame features are obtained through cross-attention reasoning; The enhanced frame-by-frame features are temporally aggregated to obtain a video-level semantic representation. Based on the similarity relationship between the video-level semantic representation and the semantic representation of the candidate answer, the video question answering result is output.

2. The method according to claim 1, characterized in that, The number of frames for sparse sampling is 16, and the visual appearance features are vector representations with consistent dimensions.

3. The method according to claim 1, characterized in that, The question text features are global semantic representations extracted from the question text, and the candidate answer semantic representations are sentence-level semantic representations obtained by encoding each candidate answer.

4. The method according to claim 1, characterized in that, The semantic features in the first stage are generated by a pre-trained language generation model with frozen parameters. This model is used to transform the visual information of video frames into a text semantic space representation related to the semantics of the question, thereby reducing the computational complexity of subsequent temporal reasoning.

5. A problem-guided multi-source feature collaborative reasoning method, characterized in that, Includes the following steps: For multiple sparsely sampled video frames in the video, the corresponding visual appearance features, general semantic features, and semantic description features related to the problem are obtained respectively. The above three types of features are summed element by element to form the aligned fused features; A cross-attention relationship is constructed between the fused features and the global semantic features of the question, so that the fused features are dynamically reweighted under the guidance of the question semantics, resulting in enhanced frame-by-frame semantic features. The enhanced frame-by-frame semantic features are temporally aggregated to generate a unified video-level semantic representation for answer discrimination in video question answering tasks.

6. The method according to claim 5, characterized in that, The cross-attention inference employs a residual connection structure, which enables the question-guided information to enhance the semantics of keyframes while maintaining the stability of the original fused features.

7. The method according to claim 5, characterized in that, The temporal aggregation is achieved by averaging all frame-by-frame semantic features to obtain a normalized representation of the overall semantics of the video.

8. A video question-answering system based on two-stage reasoning, characterized in that, include: The data preprocessing module is used to perform sparse sampling on the input video and extract the visual appearance features of the video frames and the features of the problem text. The single-frame semantic generation module is used to generate a natural language description related to the question for each video frame and encode it as the first-stage semantic feature. The multi-source feature fusion module is used to perform element-level fusion of visual appearance features, general semantic features, and first-stage semantic features to form multi-source fused features. The question-guided reasoning module is used to perform cross-attention reasoning between the multi-source fused features and the question text features to obtain enhanced frame-by-frame features. The answer prediction module is used to perform temporal aggregation of the enhanced frame-by-frame features and output video question-and-answer answers based on the aggregation results.

9. The system according to claim 8, characterized in that, The answer prediction module determines the final output answer by calculating the cosine similarity between the video-level semantic representation and the semantic representation of each candidate answer.

10. The system according to claim 8, characterized in that, During the training phase, the system adjusts the similarity distribution by introducing a temperature parameter and optimizes the prediction results using a cross-entropy loss function.

Images