A video question-answering method based on layered and overall multimodal feature combination
Patent Information
- Application Number
- CN202311801840.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-26
- Publication Date
- 2026-10-27
- Estimated Expiration
- 2043-12-26
AI Technical Summary
然而将视频和问题整体编码，不仅失去了视频中所包含的不同种类的信息，同时减少了视频中不同层次的内容和问题的交互，而如果仅仅使用层次信息，则会使模型更加离散化，无法完整的理解视频信息
[0049]与现有技术相比，本发明基于视频具有不同种类的信息，对其进行外观、运动和空间特征的提取，以此获得了全局特征，在此过程中，本发明可以保留视频中的语义和几何信息，然后本发明基于特征融合的方式将整体多种模态的特征进行结合，能够使得视频中不同层次的内容和问题产生交互，便于理解视频信息，同时更好地获取到视频中的关键信息，最后再将相关的最终特征输入至预测模型中，进而由预测模型，从而能够使得预测模型能够正分析问题并提供准确的答案，具有结果准确，适应性强的优点。
Smart Images
Figure CN117764086B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of video question answering technology, and in particular to a video question answering method based on a combination of hierarchical and holistic multimodal features. Background Technology
[0002] For video question answering, despite increasing research interest, existing models still fall short of satisfactory performance. These models can be broadly categorized into early attention-based models, memory network-based models, transformer-based models, graph neural network-based models, and others. One researcher proposed a bidirectional LSTM-based method, which incorporates spatial and temporal attention mechanisms to better focus on keyframes and key regions within frames. Another researcher incorporated and modified memory networks into video question answering to store video and caption features in a memory bank. Yet another researcher pioneered the introduction of a Transformer architecture into video question answering without pre-training. This architecture consists of two location-based self-attention blocks and a video-question co-attention block; the former replaces the LSTM, while the latter processes both visual and textual information simultaneously. Graph structure techniques have been shown to improve the reasoning ability of video question answering models. However, encoding the video and question as a whole not only loses the diverse information contained within the video but also reduces the interaction between different levels of content and the question. Furthermore, using only hierarchical information makes the model more discretized and unable to fully understand the video information. Summary of the Invention
[0003] The purpose of this invention is to provide a video question-answering method based on a combination of hierarchical and holistic multimodal features. This invention can integrate features from multiple aspects to correctly analyze questions and provide accurate answers.
[0004] The technical solution of this invention: A video question answering method based on the combination of hierarchical and holistic multimodal features, comprising the following steps:
[0005] Step 1: Extract appearance features, motion features, and spatial features from the video to obtain the global features of the video;
[0006] Step 2: Extract features from the question-and-answer statements to obtain text features;
[0007] Step 3: Construct a deep learning hierarchical model, which has a multi-layer conditional bidirectional attention graph convolutional module and a prediction module;
[0008] Step 4: Input the spatial features and text features into the first layer conditional bidirectional attention graph convolutional module of the deep learning hierarchical model to obtain the first output feature. Concatenate the first output feature with the appearance feature and then output it into the second layer conditional bidirectional attention graph convolutional module to obtain the second output feature. Combine the second output feature with the motion feature and then input it into the third layer conditional bidirectional attention graph convolutional module to obtain the third output feature.
[0009] Step 5: Concatenate the third output feature with the global feature to obtain the first image data. Then, use a random masking method to augment the image data to obtain the second image data. Finally, convolve the first image data and the second image data to obtain the first final feature and the second final feature, respectively.
[0010] Step 6: Using the first and second final features as input, the prediction module outputs the final answer.
[0011] The video question answering method described above, which combines hierarchical and holistic multimodal features, first divides the video into k segments, each containing L frames, and then uses ResNet-101 to extract appearance features F from the video frames. a ResNeXt-101 was used to extract motion features F from the video. m A pre-trained object detector is used to extract n interest spaces from video frames, which serve as the spatial features F of the video. i .
[0012] In the aforementioned video question answering method based on the combination of hierarchical and holistic multimodal features, in step 1, after extracting the three types of features, the three features are projected onto a d-dimensional space along the time dimension through a one-dimensional convolutional layer; wherein, for each object in the spatial features, a bounding box position f is added respectively. s and time position f t To enhance the spatial feature representation, the three components are concatenated and projected into a d-dimensional space using a linear transformation, followed by activation using ELU. For appearance and motion features, two one-dimensional convolutions are applied along the time dimension to achieve projection, with a sliding window size of 3, followed by activation using ELU. Then, the three features in the d-dimensional space are concatenated and activated using ELU to obtain the global features f of the video. aim .
[0013] In the aforementioned video question answering method based on the combination of hierarchical and holistic multimodal features, step 2 uses Bi-GRU to project the words in the question-and-answer statement into a d-dimensional space as partial visual input, represented as a language query Q. The language query Q is represented by connecting the forward and backward hidden states of each word in the sentence, and the last hidden state is used as the global text vector, representing the text feature f. Q .
[0014] The aforementioned video question answering method based on a combination of hierarchical and holistic multimodal features, wherein the conditional bidirectional attention graph convolutional module first processes a set of input video nodes X in This is achieved by expanding the representation using language queries Q:
[0015]
[0016] In the formula, σ is the softmax normalization function; T represents the transpose matrix; α m Let q be the m-th normalization function; m Let Q be the query for the m-th language, and M be the total number of language queries.
[0017] Then, the similarity between node pairs is dynamically calculated and the edges are represented by the adjacency matrix A:
[0018]
[0019] In the formula: Represents the learnable parameter W av and W ak Linear transformation;
[0020] Then, H-layer graph attention aggregation with skip connections is applied to refine the nodes associated with their neighbors based on the adjacency matrix A:
[0021]
[0022] In the formula: Q h and Let be the parameters and output of the attention layer h; I is the identity matrix of the skip connections;
[0023] The output node is obtained through the last hop connection:
[0024]
[0025] A self-attention pool is applied to the output node to obtain the final aggregation, represented as xf:
[0026]
[0027] In the formula: W f These are learnable linear mapping weights.
[0028] In the aforementioned video question answering method based on the combination of hierarchical and holistic multimodal features, step 4 first involves combining spatial features F... i As input, along with the text features, it is fed into the conditional bidirectional attention map convolutional module to obtain the first output feature:
[0029] F oi =CBAGC(F i );
[0030] In the formula, CBAGC represents the calculation process of the conditional bidirectional attention graph convolution module;
[0031] Obtain output features With appearance feature F a Series combination:
[0032]
[0033] In the formula: W ai The parameters are for the linear layer; t is the current frame.
[0034] Then The input is fed into the second-layer conditional bidirectional attention map convolutional module to obtain the second output feature F. S :
[0035]
[0036] Then the second output feature F S With motion characteristics F m Serial combination to obtain feature F ms The input is then fed into the third-layer conditional bidirectional attention map convolution module to obtain the third output feature F. l :
[0037] F l =CBAGC(F ms ).
[0038] In the aforementioned video question answering method based on the combination of hierarchical and holistic multimodal features, step 5 involves the third output feature F... l With global features f aim The data f in the first figure is obtained by combining the series and cascades. la Graph data augmentation was performed by multiplying a random mask with the adjacency matrix A to obtain the second graph data f. mask Then f la with f mask The first final feature f is obtained by graph convolution. V With the second final feature
[0039] In the aforementioned video question answering method based on a combination of hierarchical and holistic multimodal features, in step 6, for multiple-choice questions, the prediction model will use the first final feature f V With text features f Q Perform dot products and use a fully connected layer with softmax as a classifier:
[0040]
[0041] In the formula, s is the predicted score; W a W b and b are learnable parameters, where W a W b Here, b represents the weight parameters, and b is the bias.
[0042] For open-ended questions, the prediction model will use text features f Q and the first final feature f V Connect them and categorize them:
[0043]
[0044] In the formula: W qv W qvm W a and W b These are learnable weight parameters;
[0045] During training, optimization is achieved by minimizing the cross-entropy loss.
[0046]
[0047] In the formula: s i A is the predicted score of the i-th sample; oe The size of the predefined answer set;
[0048] If the answer of the i-th sample corresponding to the answer index matches, then y i It is 1 if it is true, otherwise it is 0.
[0049] Compared with existing technologies, this invention extracts appearance, motion, and spatial features from videos based on different types of information, thereby obtaining global features. In this process, this invention can preserve the semantic and geometric information in the video. Then, based on feature fusion, this invention combines the features of multiple modalities, enabling different levels of content and questions in the video to interact, facilitating the understanding of video information and better obtaining key information in the video. Finally, the relevant final features are input into the prediction model, which can then analyze the problem and provide accurate answers, with the advantages of accurate results and strong adaptability. Attached Figure Description
[0050] Figure 1 This is a schematic diagram of the conditional bidirectional attention graph convolution module in an embodiment of the present invention;
[0051] Figure 2 This is a schematic diagram illustrating the implementation of video question-and-answer in an embodiment of the present invention.
[0052] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this should not be construed as limiting the present invention.
[0053] Example: A video question answering method based on a combination of hierarchical and holistic multimodal features, characterized by the following steps:
[0054] Step 1: Extract appearance, motion, and spatial features from the video to obtain global features. In this step, given a video, it is first divided into k segments, each containing L frames. Then, ResNet-101 (a fast-training residual network) is used to extract appearance features F from the video frames. a ResNeXt-101 (a variant of the fast-training residual network with high-performance classification capabilities) is used to extract motion features F from the video. m A pre-trained object detector (which can be used to upload and label images using the Azure AI Custom Vision Portal, then train, evaluate, test, and deploy the model; or code to perform the training task can be written using the REST API or SDK) is used to extract n spaces of interest from video frames as spatial features F of the video. i ;
[0055] After extracting the three features, the three features are projected onto a d-dimensional space along the time dimension using a one-dimensional convolutional layer. For each object in the spatial features, a bounding box position f is added to preserve its semantic and geometric information. s and time position f t To enhance the spatial feature representation, the three components are concatenated and projected into a d-dimensional space using a linear transformation, followed by activation using ELU. For appearance and motion features, two one-dimensional convolutions are applied along the time dimension to achieve projection, with a sliding window size of 3, followed by activation using ELU. Then, the three features in the d-dimensional space are concatenated and activated using ELU to obtain the global features f of the video. aim .
[0056] Step 2: Extract features from the question-and-answer statements to obtain text features. In this step, the penultimate layer of the BERT model is used to extract word-by-word sentence embeddings. To facilitate cross-modal interaction, Bi-GRU is used to project the words in the question-and-answer statements into a d-dimensional space as part of the visual input, represented as a language query Q. The language query Q is represented by connecting the forward and backward hidden states of each word in the sentence. The last hidden state is used as the global text vector, representing the text feature f. Q .
[0057] Step 3: Construct a deep learning hierarchical model, which has a multi-layered conditional bidirectional attention graph convolutional module (CBAGC) and a prediction module; in this step, as... Figure 1 As shown, the conditional bidirectional attention graph convolution module first processes a set of input video nodes X in Expanded by querying Q using language (i.e., a set of input visual nodes X) in The contextualization associated with their neighbors in semantic and geometric space is represented as follows:
[0058]
[0059] In the formula, σ is the softmax normalization function; T represents the transpose matrix; α m Let q be the m-th normalization function; m Let Q be the query for the m-th language, and M be the total number of language queries.
[0060] By combining the cross-attention aggregation of words in the question with visual nodes, the video elements mentioned in the question are highlighted. Then, the similarity between node pairs is dynamically calculated, and edges are represented by an adjacency matrix A.
[0061]
[0062] In the formula: Represents the learnable parameter W av and W ak Linear transformation;
[0063] Then, H-layer graph attention aggregation with skip connections is applied to refine the nodes associated with their neighbors based on the adjacency matrix A:
[0064]
[0065] In the formula: W h and Let be the parameters and output of the attention layer h; I is the identity matrix of the skip connections;
[0066] The output node is obtained through the last hop connection:
[0067]
[0068] Self-attention pooling is applied to the output node to obtain the final aggregation, representing x. f :
[0069]
[0070] In the formula: W f These are the learnable linear mapping weights.
[0071] In this step, since video question answering is divided into multiple-choice questions and open-ended questions, for multiple-choice questions, the prediction model will use the first final feature f. V With text features f Q Perform dot products and use a fully connected layer with softmax as a classifier:
[0072]
[0073] In the formula, S is the predicted score; W a W b and b are learnable parameters; where W a W b Here, b represents the weight parameters, and b is the bias.
[0074] For open-ended questions, the prediction model will use text features f Q and the first final feature f V Connect them and categorize them:
[0075]
[0076] In the formula: W qv W qvm W a and W b These are learnable weight parameters, each corresponding to the weight of the content in parentheses in the formula;
[0077] During training, optimization is achieved by minimizing the cross-entropy loss.
[0078]
[0079] In the formula: s i A is the predicted score of the i-th sample; oe The size of the predefined answer set;
[0080] If the answer of the i-th sample corresponding to the answer index matches, then yi is 1; otherwise, it is 0.
[0081] Step 4: Input the spatial features and text features into the first layer of the deep learning hierarchical model's conditional bidirectional attention map convolutional module to obtain the first output feature. Concatenate the first output feature with the appearance feature and then input it into the second layer of the conditional bidirectional attention map convolutional module to obtain the second output feature. Combine the second output feature with the motion feature and then input it into the third layer of the conditional bidirectional attention map convolutional module to obtain the third output feature. Specifically, first input the spatial features F... i As input, along with the text features, it is fed into the conditional bidirectional attention map convolutional module to obtain the first output feature:
[0082] F oi =CBAGC(F i );
[0083] In the formula, CBAGC represents the calculation process of the conditional bidirectional attention graph convolution module;
[0084] Obtain output features With appearance feature F a Series combination:
[0085]
[0086] In the formula: W ai The parameters are for the linear layer; t is the current frame.
[0087] Then The input is fed into the second-layer conditional bidirectional attention map convolutional module to obtain the second output feature F. S :
[0088]
[0089] Then the second output feature F S With motion characteristics F m Serial combination to obtain feature F ms (Steps and Output Characteristics) With appearance feature F a The data is then fed into a third-layer conditional bidirectional attention map convolutional module (combined with a cascaded array) to obtain the third output feature F. l :
[0090] F l =CBAGC(F ms );
[0091] Thus, the above steps combine all the local discrete features.
[0092] Step 5: To enhance the coherence and completeness of video features, the third output feature is concatenated with the global features to obtain the first image data. To improve the model's generalization ability and robustness, a random masking method is used to augment the image data, resulting in the second image data. The first and second image data are then convolved with graphs to obtain the first and second final features, respectively. Specifically, the third output feature F... l With global features f aim The data f in the first figure is obtained by combining the series and cascades. la Graph data augmentation was performed by multiplying a random mask with the adjacency matrix A to obtain the second graph data f. mask Then f la with f mask The first final feature f is obtained by graph convolution. V With the second final feature
[0093] Step 6: Using the first and second final features as input, the prediction module outputs the final answer.
[0094] The hardware in this embodiment uses an Intel Xeon(R) Platinum 8350C processor equipped with an NVIDIA RTX 2080 Ti graphics card; the operating system is Ubuntu 20.04, based on the open-source deep learning framework PyTorch, version 1.10.2, with CUDA version 11.3.
[0095] Furthermore, comparisons were conducted using various comparative models, which were selected as follows:
[0096] 1) ST-VQA model based on cross-attention.
[0097] 2) Models based on dynamic appearance and memory, Co-Mem[ and HME].
[0098] 3) Graph-based models L-GCN and HGA.
[0099] 4) HCRN based on hierarchical model.
[0100] 5) The method of this invention is abbreviated as HHBG.
[0101] The models were compared using the MSVD-QA test set, and the results are as follows:
[0102] Co-Mem 19.6 48.7 81.6 74.1 31.7 31.7 HME 22.4 50.1 73.0 70.7 42.9 33.7 HGA 23.5 50.4 83.0 72.4 46.4 34.7 GMIN 24.8 49.9 78.6 75.9 53.6 35.4 HCRN - - - - - 36.1 HHBG 30.05 55.29 84.1 76.14 56.14 40.35
[0103] As can be seen from the table above, this invention has the advantage of higher accuracy in word recognition and analysis compared to traditional models. Furthermore, as... Figure 2As shown in the first row, HHBG's use of global variables is particularly helpful for "Where" type questions. Global variables allow the model to integrate features from multiple aspects, thereby correctly analyzing the question and providing accurate answers. The second row provides an example of the model's correct answer; even without using global variables, it can still correctly answer object recognition questions. The third row shows an example of a model's incorrect answer. In this example, although the invention does not accurately answer questions involving motion information, the result is quite close, demonstrating the invention's adaptability to a general language of questions involving motion information.
[0104] In summary, this invention extracts appearance, motion, and spatial features from videos based on different types of information, thereby obtaining global features. During this process, the invention preserves semantic and geometric information within the video. Then, based on feature fusion, it combines features from multiple modalities, enabling interaction between different levels of content and questions within the video, facilitating understanding of video information and better capturing key information. Finally, the relevant final features are input into a prediction model, which then analyzes the problem and provides accurate answers, offering advantages such as accurate results and strong adaptability.
[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A video question answering method based on a combination of hierarchical and holistic multimodal features, characterized in that: Includes the following steps: Step 1: Extract appearance features, motion features, and spatial features from the video to obtain the global features of the video; Step 2: Extract features from the question-and-answer statements to obtain text features; Step 3: Construct a deep learning hierarchical model, which has a multi-layer conditional bidirectional attention graph convolutional module and a prediction module; Step 4: Input the spatial features and text features into the first layer conditional bidirectional attention graph convolutional module of the deep learning hierarchical model to obtain the first output feature. Concatenate the first output feature with the appearance feature and then output it into the second layer conditional bidirectional attention graph convolutional module to obtain the second output feature. Combine the second output feature with the motion feature and then input it into the third layer conditional bidirectional attention graph convolutional module to obtain the third output feature. Step 5: Concatenate the third output feature with the global feature to obtain the first image data. Then, use a random masking method to augment the image data to obtain the second image data. Finally, convolve the first image data and the second image data to obtain the first final feature and the second final feature, respectively. Step 6: Using the first and second final features as input, the prediction module outputs the final answer.
2. The video question answering method based on the combination of hierarchical and holistic multimodal features according to claim 1, characterized in that: In step 1, the video is first divided into k segments, each containing L frames. Then, ResNet-101 is used to extract the appearance features F from the video frames. a ResNeXt-101 was used to extract motion features F from the video. m A pre-trained object detector is used to extract n interest spaces from video frames, which serve as the spatial features F of the video. i .
3. The video question answering method based on the combination of hierarchical and holistic multimodal features according to claim 2, characterized in that: In step 1, after extracting the three features, the three features are projected onto a d-dimensional space along the time dimension using a one-dimensional convolutional layer; wherein, for each object in the spatial features, a bounding box position f is added. s and time position f t To enhance the spatial feature representation, the three components are concatenated and projected into a d-dimensional space using a linear transformation, followed by activation using ELU. For appearance and motion features, two one-dimensional convolutions are applied along the time dimension to achieve projection, with a sliding window size of 3, followed by activation using ELU. Then, the three features in the d-dimensional space are concatenated and activated using ELU to obtain the global features f of the video. aim .
4. The video question answering method based on the combination of hierarchical and holistic multimodal features according to claim 1, characterized in that: In step 2, Bi-GRU is used to project the words in the question-and-answer statement into a d-dimensional space as part of the visual input, which is represented as a language query Q. The language query Q is represented by connecting the forward hidden state and the backward hidden state of each word in the sentence. The last hidden state is used as the global vector of the text, representing the text feature fQ.
5. The video question answering method based on the combination of hierarchical and holistic multimodal features according to claim 4, characterized in that: The conditional bidirectional attention graph convolution module first processes a set of input video nodes X in Expanded by using the language query Q, it can be represented as: In the formula, σ is the softmax normalization function; T represents the transpose matrix; α m Let q be the m-th normalization function; m Let Q be the query for the m-th language, and M be the total number of language queries. Then, the similarity between node pairs is dynamically calculated and the edges are represented by the adjacency matrix A: In the formula: Represents the learnable parameter W av and W ak Linear transformation; Then, H-layer graph attention aggregation with skip connections is applied to refine the nodes associated with their neighbors based on the adjacency matrix A: In the formula: W h and Let be the parameters and output of the attention layer h; I is the identity matrix of the skip connections; The output node is obtained through the last hop connection: Self-attention pooling is applied to the output node to obtain the final aggregation, representing x. f : In the formula: W f These are learnable linear mapping weights.
6. The video question answering method based on the combination of hierarchical and holistic multimodal features according to claim 5, characterized in that: In step 4, the spatial features F are first... i As input, along with the text features, it is fed into the conditional bidirectional attention map convolutional module to obtain the first output feature: F oi =CBAGC(F i ); In the formula, CBAGC represents the calculation process of the conditional bidirectional attention graph convolution module; Obtain output features With appearance feature F a Series combination: In the formula: Wai is the parameter of the linear layer; t is the current frame; Then The input is fed into the second-layer conditional bidirectional attention map convolutional module to obtain the second output feature F. S : Then the second output feature F S With motion characteristics F m Serial combination to obtain feature F ms The input is then fed into the third-layer conditional bidirectional attention map convolution module to obtain the third output feature F. l : F l =CBAGC(F ms )。 7. The video question answering method based on the combination of hierarchical and holistic multimodal features according to claim 5, characterized in that: In step 5, the third output feature F l With global features f aim The data f in the first figure is obtained by combining the series and cascades. la Graph data augmentation was performed by multiplying a random mask with the adjacency matrix A to obtain the second graph data f. mask Then f la with f mask The first final feature f is obtained by graph convolution. V With the second final feature 8. The video question answering method based on the combination of hierarchical and holistic multimodal features according to claim 1, characterized in that: In step 6, for multiple-choice problems, the prediction model will use the first final feature f V With text features f Q Perform dot products and use a fully connected layer with softmax as a classifier: In the formula, S is the predicted score; W a W b and b are learnable parameters, where W a W b Here, b represents the weight parameters, and b is the bias. For open-ended questions, the prediction model will use text features f Q and the first final feature f V Connect them and categorize them: In the formula: W qv W qvm W a and W b These are learnable weight parameters; During training, optimization is achieved by minimizing the cross-entropy loss. In the formula: s i A is the predicted score of the i-th sample; oe The size of the predefined answer set; If the answer of the i-th sample corresponding to the answer index matches, then y i It is 1 if it is true, otherwise it is 0.