Video Question Answering Methods Based on Cue Enhancement and Sparse Attention

By employing a video question answering method based on prompt enhancement and sparse attention, and utilizing a pre-trained CLIP model and a probabilistic sparse self-attention mechanism, the problem of redundant information interference in video question answering is solved, achieving an efficient and low-cost video question answering task.

CN117649628BActive Publication Date: 2026-11-17XIDIAN UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202311681462.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-08
Publication Date
2026-11-17
Estimated Expiration
2043-12-08

AI Technical Summary

Technical Problem

Existing technologies suffer from redundant visual information interference in video question answering tasks, which is difficult to eliminate, resulting in high model training costs, low efficiency, and an inability to effectively utilize key information related to the question in the video.

Method used

We employ a cue-based enhancement and sparse attention approach, using a pre-trained CLIP model with frozen parameters to extract video features, and then using a probabilistic sparse self-attention mechanism to filter key features, combined with textual prompts for answer prediction.

Benefits of technology

It effectively eliminates interference from redundant video information, reduces training costs and GPU resource requirements, and improves the accuracy and efficiency of question answering.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117649628B_ABST
    Figure CN117649628B_ABST
Patent Text Reader

Abstract

This invention discloses a video question answering method based on cue enhancement and sparse attention, comprising the following steps: Step 1: Feature extraction and cue construction using a pre-trained CLIP model; Step 2: Inputting the features extracted in Step 1, constructing a probabilistic sparse self-attention (PSSA) model to extract key video features; Step 3: Fusing video features with text cue using the key video features extracted in Step 2, and determining the answer based on the similarity between the fused features and candidate answer features. This invention leverages the powerful visual-text generalization ability of the pre-trained CLIP model to address the cost issue of video-text pre-training. By fully utilizing the contrastive learning characteristics of CLIP and considering probabilistic sparse self-attention and visual-language cross-attention, redundant information in video features is effectively eliminated.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of video question answering technology, specifically relating to a video question answering method based on cue enhancement and sparse attention. Background Technology

[0002] Video question answering tasks require answering natural language questions based on the content of a given video. This demands that the model understand both visual and linguistic information and accurately identify question-relevant visual information from information-rich videos. However, video acquisition and processing are more complex and time-consuming than image processing, and the amount of information contained in videos far exceeds that in text, making video language pre-training more expensive than image language training.

[0003] Meanwhile, the excessive visual information contained in videos creates redundancy for the question-answering task. Most existing technologies cannot eliminate this redundancy from interfering with the task, making it difficult to explicitly link visual objects with entities in the text; or they design complex modules to eliminate redundant semantic information, but these methods significantly increase the cost of training the model.

[0004] Video question answering is a crucial task in the field of multimodal video understanding, aiming to enable computers to deeply understand the semantic information of videos and thus answer natural language questions related to the video content. These questions typically cover various aspects of the video, including objects, actions, locations, and context. Unlike image question answering tasks, video question answering needs to process features in the temporal dimension, requiring analysis of different frames of the video. In recent years, with the continuous development of deep learning technology, techniques such as cross-modal attention and graph neural networks have been widely used to model the information content of videos to answer questions. However, the complexity of video content in natural scenes, the large number of entities, and the complex relationships between entities hinder the improvement of question answering accuracy. To address these issues, how to extract key information from a large number of video frames based on the information in the question answer text is a key problem that current video question answering technologies need to solve.

[0005] In their paper "Just Ask: Learning to Answer Questions from Millions of Narrated Videos," published at ICCV, Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, Cordelia Schmid, and others proposed an automated annotation method to generate a large amount of video question-answering training data based on HowTo100M. The authors utilized answer generation and question generation models pre-trained on the text-based question-answering dataset SQuADv1 to automatically generate questions and answers from the video captions of HowTo100M, ultimately obtaining 69 million video-question-answer triples. Simultaneously, they performed cross-modal encoding on the videos and questions, while encoding the answers separately, and then employed contrastive learning and masked language modeling for pre-training.

[0006] In their paper "(2.5+1)D Spatio-Temporal Scene Graphs for Video Question Answering," published at AAAI, Anoop Cherian, Chiori Hori, Tim K. Marks, Jonathan Le Roux, and others proposed a (2.5+1)D scene graph representation to better capture the spatiotemporal information flow within videos. The authors used an off-the-shelf 2D-to-3D conversion module to transform each 2D frame into a 2.5D (pseudo-3D) scene graph with an inferred 3D structure, and registered video frames to a shared (2.5+1)D spatiotemporal space, where the location of each 2D scene graph was determined. The (2.5+1)D graph was then separated into static and dynamic subgraphs based on whether objects within it typically move in the world, with each dynamic subgraph enriching its motion features by capturing its interactions with other graph nodes. Finally, the (2.5+1)D graph was embedded into a spatiotemporally hierarchical latent space, where subgraphs of different granularities and their interactions were captured, enabling accurate video question answering.

[0007] While Antoine Yang et al.'s method achieved good results, directly training video language models requires a large number of video-text pairs and a significant amount of GPU resources. Furthermore, video acquisition and processing are much more complex and time-consuming than image processing. Moreover, the amount of information in videos is too large and inconsistent with text, making video language pre-training far more expensive than image language pre-training. In addition, the authors did not consider that questions in video question answering tasks may only be related to some video segments, and irrelevant visual information may have an adverse effect on answer prediction.

[0008] Although the (2.5+1)D scene graph representation method by Anoop Cherian et al. can eliminate the interference of redundant information on question answers to a certain extent, its processing of visual information is complex and inefficient, making it difficult to apply directly in real-world scenarios. Summary of the Invention

[0009] To overcome the shortcomings of the existing technologies, the present invention aims to provide a video question answering method based on cue enhancement and sparse attention. This method leverages the powerful visual-text generalization ability of the pre-trained CLIP model to address the cost issue of video-text pre-training. By fully utilizing the contrastive learning characteristics of CLIP and considering probabilistic sparse self-attention and visual-language cross-attention, redundant information in video features can be effectively eliminated.

[0010] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0011] A video question answering method based on cue enhancement and sparse attention includes the following steps;

[0012] Step 1: Use the pre-trained CLIP model for feature extraction and text prompt construction;

[0013] Step 2: Construct a probability sparse self-attention mechanism (PSSA) and input the features extracted in Step 1 to obtain key video features;

[0014] Step 3: Using the key video features extracted in Step 2, fuse the video features with the text prompts, and determine the answer by the similarity between the fused features and the candidate answer features.

[0015] Step 1 specifically involves:

[0016] (1a) Sample L frames from the original video V, and use a pre-trained CLIP model with frozen parameters as a feature extractor to extract the visual features F of the L frames. v As a video feature;

[0017] (1b) Use CLIP text decoder to extract features f corresponding to the video problem q The question-answering model selects the correct answer from N candidate answers, given a video, a question related to the video, and several candidate answers. a and by using f q With F a Adding them together yields the corresponding N text prompt features F. t .

[0018] Step 1 involves characterizing the original video image information and question-answer text for subsequent operations. The frozen parameters lock the model's parameters, preventing gradient updates during the training phase.

[0019] In step 2, the video features F v The key video features F' are obtained by feeding them into a multi-head probabilistic sparse self-attention layer. v Multi-head refers to using multiple self-attention mechanisms on an input sequence to obtain multiple sets of attention results, and then concatenating and linearly projecting these results to obtain the final output; each head is a probabilistic sparse self-attention mechanism.

[0020] F v ←F v +MultiHead PSSA (LN(F v ),LN(F v ),LN(F v ))

[0021] F' v =F v +FFN(LN(F v ))

[0022] Where ← represents the assignment operation, and LN and FFN represent the layer normalization operation and a feedforward neural network layer, respectively.

[0023] The multi-head probabilistic sparse self-attention layer is constructed using the following method:

[0024] (2a) Construct a self-attention mechanism based on query Q, key K, and value V:

[0025]

[0026] Where softmax() represents the softmax function, and d represents the dimension of the feature. These are feature vectors belonging to a feature space of dimension L×d;

[0027] (2b) Modify the self-attention mechanism to construct a probabilistic sparse self-attention mechanism, where each row q in the query Q of the self-attention mechanism is... i The attention given to all keys K is defined as a probability distribution:

[0028]

[0029] Among them, L K It is the length of the bond K, k(q) i ,k j ) is an asymmetric exponential kernel function;

[0030] (2c) At the same time Defined as a uniform distribution, representing unrelated attention, the similarity of the dot product pairs is measured by calculating the KL divergence between distributions p and q:

[0031]

[0032] (2d) Select M(q) i The highest u queries in K) are obtained pass We obtain the probabilistic sparse self-attention (PSSA), where u is determined by the sampling factor c, and u = clnL Q :

[0033]

[0034] (2e) Replace the probabilistic sparse attention layer with multi-head attention. In the multi-head perspective, this attention generates different sparse query key pairs for each head, thereby avoiding severe information loss:

[0035] MultiHead PSSA (Q,K,V)=W c ([h1;…;h e ])

[0036] Among them W c It is a linear transformation, [;] denotes the feature concatenation operation, e is the number of multi-heads, and h is the output of each attention head:

[0037]

[0038] These are linear transformations of the i-th attention head query Q, key K, and value V, respectively.

[0039] Video features are filtered according to their popularity over time to remove redundant information.

[0040] Step 3 specifically involves:

[0041] (3a) Obtain the video key features F' v With text prompt F t Perform cross-attention to obtain the fused feature F' t ;

[0042] (3b) The output logical value is obtained by using mean pooling, and the final answer prediction is obtained by optimizing the cosine similarity.

[0043] The specific steps (3a) are as follows:

[0044] The obtained video key features F′ v With text prompt F t Cross-attention is performed through a cross-modal interaction module to obtain the fused feature F'. t ;

[0045] F' v ←F' v +MultiHead(LN(F' v ),LN(F t ),LN(F t ))

[0046] F' t =F' v +FFN(LN(F' v ))

[0047] The specific steps (3b) are as follows:

[0048] The output logical value f' is obtained using mean pooling. t =mean(F' t Using the cosine similarity function sim(f') t ,f a Calculate the similarity between the features of the predicted answer and the true answer.

[0049]

[0050] Where f a The answer features obtained in (1b) are: <,> represents the scalar multiplication operation between two features, and ||| represents the modulus of the feature.

[0051] The model is optimized by treating the remaining question-answer pairs in the same batch as negative samples, maximizing the similarity between the matched samples and minimizing the similarity with the negative samples. The loss function is defined as:

[0052]

[0053] Where τ is the temperature hyperparameter used to scale feature similarity, and N is the number of candidate answers or class categories.

[0054] The beneficial effects of this invention are:

[0055] First, this invention uses a pre-trained CLIP model with frozen parameters as a feature extractor, which not only ensures the powerful extraction capability of the pre-trained model for visual text information, but also requires only limited parameter training and extremely low GPU memory consumption, eliminating the need to retrain a video text model, making video understanding tasks more economical.

[0056] Secondly, compared to methods that design complex modules to eliminate redundant information, this invention directly uses a pre-trained CLIP model to extract aligned modal features. By searching static video frames using aligned text features, redundant information is eliminated, which simply and effectively solves the problem of redundant information in video information interfering with answering questions. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0058] The present invention will now be described in further detail with reference to the accompanying drawings.

[0059] like Figure 1 As shown, the video question answering method based on cue enhancement and sparse attention includes the following steps;

[0060] Step 1: Use a pre-trained CLIP model for feature extraction and cue construction;

[0061] Step 2: Input the features extracted in Step 1, construct a probabilistic sparse self-attention PSSA to extract key features from the video;

[0062] Step 3: Using the key video features extracted in Step 2, fuse the video features with text prompts, and determine the answer based on the similarity between the fused features and the candidate answer features. Step 1 specifically involves:

[0063] (1a) Sample L frames from video V, and use a pre-trained CLIP model with frozen parameters (locking the model's parameters and not updating gradients during training) as a feature extractor to extract visual features F from the L frames. v As a video feature;

[0064] (1b) Use CLIP text decoder to extract the problem features corresponding to the video. q The features F of the N candidate answers to the question a and by using f q With F a Adding them together yields the corresponding N text prompt features F. t .

[0065] The original video image information and question answer text are characterized for subsequent operations.

[0066] Step 2 specifically involves:

[0067] (2a) Self-attention mechanism based on query Q, key K, and value V:

[0068]

[0069] Where sofrmax() represents the doftmax function, and d represents the dimension of the feature. These are feature vectors belonging to a feature space of dimension L×d;

[0070] (2b) Modify the attention mechanism to construct a probabilistic sparse self-attention mechanism, where each row of the query Q in the self-attention mechanism is a subset of the query Q. i The attention given to all keys K is defined as a probability distribution:

[0071]

[0072] Among them, L K It is the length of the bond K, k(q) i ,k j ) is an asymmetric exponential kernel function;

[0073] (2c) At the same time Defined as a uniform distribution, representing unrelated attention, the similarity of the dot product pairs is measured by calculating the KL divergence between distributions p and q:

[0074]

[0075] (2d) Select M(q) i The highest u queries in K) are obtained pass We obtain the probabilistic sparse self-attention (PSSA), where u is determined by the sampling factor c, and u = clnL Q :

[0076]

[0077] (2e) Replace the probabilistic sparse attention layer with multi-head attention. In the multi-head perspective, this attention generates different sparse query key pairs for each head, thereby avoiding severe information loss:

[0078] MultiHead PSSA (Q,K,V)=W c ([h1;…;h e ])

[0079] Among them W c It is a linear transformation, [;] denotes the feature concatenation operation, e is the number of multi-heads, and h is the output of each attention head:

[0080]

[0081] These are linear transformations of the i-th attention head query Q, key K, and value V, respectively.

[0082] (2f) Feed the video features Fv into a multi-head probabilistic sparse self-attention layer to obtain the video key features F'. v 。:

[0083] F v ←F v +MultiHead PSSA (LN(F v ),LN(F v ),LN(F v ))

[0084] F' v =F v +FFN(LN(F v ))

[0085] Where ← represents the assignment operation, and LN and FFN represent the layer normalization operation and a feedforward neural network layer, respectively.

[0086] Video features are filtered according to their popularity over time to remove redundant information.

[0087] Step 3 specifically involves:

[0088] (3a) Obtain the video key features F' v With text prompt F t Cross-attention is performed through a cross-modal interaction module to obtain the fused feature F'. t ;

[0089] F' v ←F' v +MultiHead(LN(F' v ),LN(F t ),LN(F t ))

[0090] F' t =F' v +FFN(LN(F' v ))

[0091] (3b) Use mean pooling to obtain the output logical value f' t =mean(F' t Using the cosine similarity function sim(f') t ,f a Calculate the similarity between the features of the predicted answer and the true answer.

[0092]

[0093] Where f aThe answer features obtained in (1b) are where <,> represent the scalar multiplication operation between two features, and ‖‖ represents the modulus of the feature. (3c) The model is optimized by maximizing the similarity between the matching samples and minimizing the similarity between the matching samples and the negative samples, using the remaining question-answer pairs in the same batch as negative samples. The loss function is defined as:

[0094]

[0095] Where τ is the temperature hyperparameter used to scale feature similarity, and N is the number of candidate answers or class categories.

[0096] This invention designs a probabilistic sparse attention mechanism by leveraging the local query importance distribution. This mechanism sparsely samples the most important video frames for video question answering tasks, effectively and simply removing the interference of redundant information in the video on the question answer.

[0097] To effectively integrate information from question-answer pairs and adapt the pre-trained CLIP to video question-answering tasks with minimal modifications, appropriate text prompts need to be designed. A common approach is to use a large-scale language model to convert each question into a statement text template based on different question types and insert candidate answers. However, for open-ended question answering, this requires generating N prompts for each question.

[0098] This invention uses a simple and effective method to construct hints from the question answer text, requiring only O(1) feedforward computation time, avoiding the O(N) feedforward propagation time cost of using large-scale language models to build templates to generate hints.

[0099] This invention trains video question answering tasks using the same contrastive loss method as CLIP, maintaining alignment with downstream tasks and upstream cross-modal pre-training. On the NExT-QA validation set, it achieves improvements of 3.4%, 1.11%, 11.29%, and 4.05% in causal accuracy, temporal accuracy, descriptive accuracy, and overall accuracy compared to Anoop Cherian et al.'s P3D-G model, respectively. On the MSVD-QA and ActivityNet-QA datasets, it achieves improvements of 1.0% and 1.1% in accuracy compared to Antoine Yang et al.'s Just Ask model, respectively.

Claims

1. A video question-answering method based on cue enhancement and sparse attention, characterized in that, Includes the following steps; Step 1: Use the pre-trained CLIP model for feature extraction and text prompt construction; Step 2: Construct a probabilistic sparse self-attention mechanism (PSSA) and input the features extracted in Step 1 to obtain the key features of the extracted video; Step 3: Using the key video features extracted in Step 2, fuse the video features with the text prompts, and determine the answer based on the similarity between the fused features and the candidate answer features; Step 1 specifically involves: (1a) From the original video Medium sampling Frames, using a pre-trained CLIP model with frozen parameters as a feature extractor, extract... Visual features of frames As a video feature; (1b) Use CLIP text decoder to extract problem features corresponding to the video. Corresponding to the question Features of candidate answers and by making and Adding them together, we get the corresponding... Text prompt features ; Step 3 specifically involves: (3a) Obtain key features from the video With text prompts Perform cross-attention to obtain fused features. (3b) The output logical value is obtained by using mean pooling, and the final answer prediction is obtained by optimizing the cosine similarity. The model is optimized by treating the remaining question-answer pairs in the same batch as negative samples, maximizing the similarity between the matched samples and minimizing the similarity with the negative samples. The loss function is defined as: in It is a temperature hyperparameter used to scale feature similarity. It is the number of candidate answers or categories; To predict the similarity between the features of the answer and the true answer.

2. The video question answering method based on cue enhancement and sparse attention according to claim 1, characterized in that, Step 1 involves characterizing the original video image information and question-answer text for subsequent operations. The parameter freezing step locks the model's parameters, preventing gradient updates during the training phase.

3. The video question answering method based on cue enhancement and sparse attention according to claim 1, characterized in that, Step 2, video features Key features of the video are obtained by feeding them into a multi-head probabilistic sparse self-attention layer. ; in This represents an assignment operation. and These represent layer normalization operations and a feedforward neural network layer, respectively.

4. The video question answering method based on cue enhancement and sparse attention according to claim 3, characterized in that, The multi-head probabilistic sparse self-attention layer is constructed using the following method: (2a) Construct a query-based system ,key Sum Self-attention mechanism: in express function, Indicates the dimensionality of the feature. , , They belong to the dimension of Feature vectors in feature space; (2b) Modify the self-attention mechanism to construct a probabilistic sparse self-attention mechanism, and incorporate the query in the self-attention mechanism. Each of its rows For all keys Attention level is defined as a probability distribution: in, Is a key Length, It is an asymmetric exponential kernel function; (2c) At the same time Defined as a uniform distribution, representing irrelevant attention, by calculating the distribution and The KL divergence measure is the similarity of the dot product pairs: (2d) selection highest among A query yielded ,pass We obtained probabilistic sparse self-attention PSSA, where By sampling factor Sure, : (2e) Replace the probabilistic sparse attention layer with multi-head attention. In the multi-head perspective, the attention generates different sparse query key pairs for each head: in It is a linear transformation. This represents the concatenation operation of features. It's the number of bullish positions. The output of each attention head: They are respectively for the first Individual attention head query ,key Sum Linear transformation.

5. The video question answering method based on cue enhancement and sparse attention according to claim 4, characterized in that, The specific steps (3a) are as follows: Obtain key features from the video With text prompts Cross-attention is performed through a cross-modal interaction module to obtain fused features. 。 6. The video question answering method based on cue enhancement and sparse attention according to claim 5, characterized in that, The specific steps (3b) are as follows: The output logical value is obtained using mean pooling. Using the cosine similarity function Calculate the similarity between the features of the predicted answer and the true answer. in The answer characteristics obtained in (1b) are... This represents the multiplication operation of two characteristic numbers. The modulus of the feature.

Citation Information

Patent Citations

  • Video question answering method and system for end-to-end training based on sparse sampling

    CN113807222A

  • Dual-channel semantic positioning multi-granularity attention mutual enhancement video question answering method and system

    CN114020891A