An open video question-answering method based on external knowledge constraints

Abstract

The application discloses an open video question and answer method based on external knowledge constraint, which mainly relates to an open video question and answer framework capable of effectively generating out-of-domain answers, and external common sense knowledge is introduced through an adaptive external knowledge module and a multi-stream information interaction mechanism to provide rich semantic support for the generation of out-of-domain answers. The application comprises the following steps: 1. data preprocessing is performed on a video and a question described in natural language and an answer text; 2. external common sense knowledge is introduced and feature interaction is performed through the adaptive external knowledge module and the multi-stream information interaction mechanism; 3. an answer is generated by using a decoder structure; and 4. model training is performed, and a model parameter is trained by using a back propagation algorithm. The application can effectively generate the open video question and answer framework of out-of-domain answers, and can perform feature fusion and answer generation by using external common sense knowledge, and the best effect in the field of open video question and answer is obtained.

Description

Technical Field

[0001] This invention relates to a method for generating out-of-domain answers for open-ended video question answering (Open-ended VideoQA), and more particularly to an open-ended video question answering method based on external knowledge constraints, an open-ended video question answering framework that can effectively generate out-of-domain answers, and a method that uses adaptive external knowledge modules and multi-stream information interaction mechanisms to introduce external common sense knowledge to provide rich semantic support for the generation of out-of-domain answers. Background Technology

[0002] Open-ended video question answering aims to generate answers from scratch based on a given video and question, without having to choose from several pre-provided candidate answers or fill in the missing parts of an incomplete answer. To achieve the ability to freely generate answers, open-ended video question answering requires not only a good understanding of the key content of the video and the semantic information of the question, but also the ability to combine this multimodal information to perform comprehensive cross-modal reasoning. Therefore, open-ended video question answering is more challenging than other types of video question answering tasks.

[0003] In recent years, many excellent works have treated open-ended video question answering as a multi-label classification task, using attention mechanisms, graph networks, causal analysis, or modular neural networks to study it. This setup requires forming a candidate answer set from the most frequent answers and then inferring the specific answer to the corresponding question from the candidate set. Since the candidate set only contains answers that appear in the training set, these methods can only predict in-domain answers, not out-of-domain answers (answers that do not appear in the training set). Furthermore, due to the structural characteristics of the answers in the training set, the candidate set basically contains very short answers of length 1 or 2. Therefore, the predicted answers often lack rich semantics and detail. In contrast, freely generated out-of-domain answers usually contain more detailed, longer, and semantically complete answers. Based on this, some works have addressed open-ended video question answering in a generative way. They rely on the entire training set instead of the candidate set; however, the training set may still not provide enough semantics to generate high-quality answers.

[0004] Knowledge is defined as a high-level awareness and understanding of input information and its surrounding environment. It includes universally accepted facts and rules, often referred to as common sense, which are beneficial for human and neural network learning, communication, and reasoning. This invention argues that knowledge can provide detailed and comprehensive key information for generating external answers. Therefore, how to correctly and effectively utilize external knowledge is a problem worthy of in-depth research.

[0005] In conclusion, open-ended video question answering based on external knowledge constraints is a direction worthy of in-depth research. This project intends to start from several key issues in this task, solve some problems existing in the current work, and ultimately form an open-ended video question answering framework that can effectively generate answers from outside the domain.

[0006] The complexity and variety of video content in application scenarios, along with its varying lengths and the flexibility of natural language questions, present significant challenges to video question-answering reasoning algorithms. In general, the main difficulties lie in the following two aspects:

[0007] (1) How to build a cross-modal framework for open-ended video question answering: Open-ended video question answering has received more attention as a cross-modal computer vision task. In the past few years, many works have explored this task using attention mechanisms. Multi-stream spatiotemporal attention network models have been constructed to learn joint representations of dynamic video content and context-aware question embeddings. In recent years, the in-depth development of graph networks, neural modules, and memory networks has also provided more methods for video question answering. To date, many high-performing models have explored this task from multiple perspectives, but these models define the question as a discriminative task of multi-label classification. In real-world applications, it is difficult to define a candidate set that includes all possible answers.

[0008] (2) How to provide effective support for the generation of out-of-domain answers: Most current works define open-ended video question answering tasks as multi-label classification tasks, which requires defining a finite candidate set. Due to the structural characteristics of the answers in the training set, the candidate set basically contains extremely short answers of length 1 or 2. Such predicted answers usually lack rich details and semantics. In contrast, freely generated out-of-domain answers usually contain more detailed, longer, and semantically complete answers. Recently, some works have used generative methods to address open-ended video question answering tasks, but because they fundamentally do not provide rich semantic support for the generation of out-of-domain answers, they cannot effectively generate out-of-domain answers. Summary of the Invention

[0009] The purpose of this invention is to address the shortcomings of existing technologies by providing an open-ended video question-answering method based on external knowledge constraints.

[0010] This invention provides an open video question-answering framework capable of effectively generating out-of-domain answers, addressing two challenges mentioned in the background section. First, this invention proposes an overall open video question-answering framework that effectively generates out-of-domain answers through an encoder-decoder structure. Second, this invention introduces external common-sense knowledge through adaptive external knowledge modules and a multi-stream information interaction mechanism to provide rich semantic support for the generation of out-of-domain answers.

[0011] The technical solution adopted by this invention to solve its technical problem is as follows:

[0012] An open video question answering framework capable of effectively generating out-of-domain answers includes a video-text encoder, an adaptive external knowledge module, a multi-stream information control mechanism, and a language decoder.

[0013] The video-text encoder extracts video question features based on the contextual information of the video and the question. The adaptive external knowledge module attaches detected video objects to a knowledge base to obtain external knowledge related to the video content, providing implicit constraints for answer generation. The multi-stream information control mechanism fuses video question features and external knowledge through local and global attention to obtain knowledge-enhanced video question features, which are then input into the language decoder to generate answers freely. The core of the multi-stream information control mechanism is to capture key information between video question features and external knowledge features through an attention mechanism that focuses on different patterns. "Multi-stream" refers to the module taking three sets of stream information as input: the video question stream, the external knowledge stream, and the feature fusion stream.

[0014] The specific implementation process of the framework of this invention is as follows:

[0015] Step (1): Data preprocessing, feature extraction from video and problem data:

[0016] 1-1. Randomly divide the L-frame video V into N clips of equal length C = {c1,...,c...} N The question text is then embedded into a 100-dimensional word vector Q. The WordPiece word embedding method is used here to improve semantic distinction between different words in the question text.

[0017] 1-2. Using the baseline model ClipBERT as a video-text encoder, we extract common contextual information from the video and the question with good performance. Furthermore, ClipBERT is efficient not only in visual and linguistic feature extraction but also in terms of memory and computational power consumption. The feature representation F of the video question... V for:

[0018] FV =E ClipBERT (C,Q) (Formula 1)

[0019] Among them, E ClipBERT (·) represents the video-text encoder model.

[0020] Step (2): Introduce external knowledge features using an adaptive external knowledge module, and capture key information between video problem features and external knowledge features through a multi-stream information interaction mechanism.

[0021] 2-1. To provide robust semantic support for generating out-of-domain answers, an adaptive external knowledge module is proposed to introduce external knowledge features. To represent the external knowledge of the video, this paper proposes fitting the video frame f... i Visual knowledge in (i = 1, ..., L) is used to query the corresponding external knowledge k in the external common sense knowledge base. E Furthermore, based on external knowledge k E Knowledge features F are constructed using an iterative memory network. K .

[0022] Therefore, this paper first uses Faster R-CNN with ResNet-101 to fit each video frame f i Extract video entity objects and represent the text description of the entity objects as o. i However, even a pre-trained Faster R-CNN model can extract objects from a video that are irrelevant to the video content; therefore, a CLIP model is further used to supplement the text description. i Denoising is performed by calculating the fitted video frame f. i and text description o i The similarity between objects is used to filter out irrelevant objects. This reduces the impact of irrelevant noise on subsequent video reasoning and other processes. The denoising process is described as follows:

[0023]

[0024] The output entity object label is denoted as visual knowledge k. V (Visual Knowledge), k V It still involves visual knowledge of entities in the video after denoising. CLIP(·) represents the CLIP model, and α is a predefined parameter, which is set to 0.2 in this paper.

[0025] To acquire external knowledge, a triple <o,r,s> was retrieved from the commonsense knowledge base ConceptNet, where o and s are text labels of objects, and r represents the relationship between objects. Visual knowledge k was then used to acquire this knowledge.V Text description in o i The text label of each object is compared with o and s in all triples, and o or s is selected to match the text description o. i Collect all similar triples, and then recombine the o and s in these similar triples into sets. and set It also uses CLIP for noise reduction:

[0026]

[0027] Among them, set and set Both represent objects described by external knowledge, k e This represents the denoised external knowledge, and β = 0.19 is the threshold. For further calculation, k... e Embedded in WordPiece, the corresponding vectorized representation is obtained:

[0028]

[0029] To fully utilize the external knowledge k between adjacent frames E This invention employs an attention-based GRU algorithm to compute context-aware knowledge representations. In this part, the invention uses "memory" to provide more original and direct external knowledge for answer generation. Simultaneously, due to the richness of video content, this invention assumes that the model may focus on different but key elements in the video each time it is watched, forming a final perception of the video after multiple playbacks. External knowledge can also be perceived in a similar way. For this purpose, a set of contextual memory units {m} is used. 1 ,m 2 ,…,m T} to record external knowledge k E Attention at each time t. Specifically, the following calculations are performed:

[0030] m 0 =tanh(W p k E +b p ), (Formula 5)

[0031] z t =[k E ·m 0 ,k E ·m t-1 ,|k E -m 0 |,|k E -m t-1 |], (Formula 6)

[0032]

[0033]

[0034]

[0035] Among them, z t Capturing the similarity between external knowledge and memory, g t The attention operation performed by the model on each knowledge is defined, |·| is the absolute value, · represents the dot product between elements, and e t Let e ​​be the state vector of the GRU at time t. 0 =0. By appropriately setting the parameters and the number of GRU states, g t and e t Having and k E For the same size, e is calculated one by one using formula (8). t Elements, among which k r E and This corresponds to an element in the GRU. The relevant information is then updated using formula (9), where... This is the final state of GRU. After the iteration is complete, the external knowledge feature F K It can be represented as:

[0036] F K =ReLU(W f [m 1 ,...,m t ]+b f ) (Formula 10)

[0037] Where, m t This represents the final memory information. (b) f b g1 b g2 b p Both refer to the hyperparameters that need to be learned.

[0038] 2-2. To discover the complementarity between video problem features and knowledge features, this invention designs a multi-stream information control mechanism. The core of this mechanism is to capture key information between video problem features and external knowledge features through an attention mechanism, which focuses on different patterns. Multi-stream refers to the module taking three sets of stream information as input, namely, the video problem feature F... V External knowledge characteristics F K and integration features.

[0039] First, calculate the video problem features F. V and external knowledge characteristics F K Global attention F between C:

[0040] F C =Sigmoid(BN(W) c [F V ,F K ]+b c )) (Formula 11)

[0041] Here, BN represents standardized batch processing. Then, based on global attention F... C Local attention is calculated separately based on different features, and then the local attention is integrated to obtain the fused knowledge-enhanced video problem features F. M The fusion feature is calculated as follows:

[0042] F M =ReLU(W m [(F C ) T F V ,(F C ) T F K ]+b m ) (Formula 12)

[0043] Among them, b c b m All represent the hyperparameters to be learned. Due to the attention mechanism, the important content of the video, the question, and external knowledge can be well aligned, thus enabling the answer generation module to obtain semantically correct feature representations. To verify the superiority of the multi-stream information control mechanism, this invention was also compared with four other multimodal fusion methods. These five different cross-modal fusion mechanisms in Figure 2 The relevant examples are shown in the diagram. Detailed comparisons will be presented in the experimental section.

[0044] Step (3), Answer Generation:

[0045] The Transformer is used as the answer generation module, and its parameters are initialized using a pre-trained GPT2 model. This allows the model to selectively focus on the input features most relevant to the answer. The answer generation process is described as follows:

[0046] y=D GPT2 (F M ) (Formula 13)

[0047] Where y is the predicted answer, and D is the predicted answer. GPT2 (·) is a decoder structure based on GPT2 initialization.

[0048] Step (4) Model Training

[0049] The predicted answer from step (3) and the standard answer 'a' provided in the dataset are input together into the predefined loss function to obtain the corresponding loss, as shown in the following formula:

[0050] Loss=H(y,a) (Formula 14)

[0051] Where H(·) represents the cross-entropy loss function, and a represents the standard answer.

[0052] The specific cross-entropy loss function is shown below:

[0053]

[0054] Where p and q are the predicted answer and the standard answer, respectively, and n is the sample size. The loss value (LOSS) calculated according to the above formula is then used to adjust the parameters in the network using the back-propagation (BP) algorithm until training converges, thus obtaining the optimal model.

[0055] The beneficial effects of this invention are as follows:

[0056] This invention proposes a method for effectively generating out-of-domain answers in open-ended video question answering tasks. Specifically, it proposes an open-ended video question answering framework that can effectively generate out-of-domain answers, and utilizes external common-sense knowledge for feature fusion and answer generation, achieving state-of-the-art results in the open-ended video question answering field. This invention uses an adaptive external knowledge module and a multi-stream information interaction mechanism to introduce external common-sense knowledge, providing rich semantic support for the generation of out-of-domain answers. Attached Figure Description

[0057] Figure 1 This is a framework diagram of the present invention.

[0058] Figure 2 This is a schematic diagram of five different cross-modal fusion mechanisms of the present invention. Detailed Implementation

[0059] This invention provides an efficient framework for generating out-of-domain answers for open-ended video question answering. The detailed parameters of this invention are further described below.

[0060] Its architecture is as follows Figure 1 As shown, the video-text encoder ( Figure 1 (as shown in section (a)) Extracts video question features based on the contextual information of the video and the question; Adaptive external knowledge module ( Figure 1 (As shown in section (b)) the detected video objects are fitted into the knowledge base to obtain external knowledge related to the video content, providing implicit constraints for answer generation; multi-stream information control mechanism ( Figure 1As shown in section (c), video problem features and external knowledge are fused through local and global attention to obtain knowledge-enhanced video problem features, which are finally input into the language decoder. Figure 1 (As shown in section (d)) The answer can be generated freely.

[0061] The specific implementation steps of this invention are as follows:

[0062] 1. Perform data preprocessing on the video and the questions and answers described in natural language.

[0063] 2. Adaptive external knowledge modules and multi-stream information interaction mechanisms are used to introduce external common sense knowledge and feature interaction.

[0064] 3. Use a decoder structure to generate the answer.

[0065] 4. Model training: Use the backpropagation algorithm to train the model parameters.

[0066] Regarding the dataset, this invention primarily uses the Open-QA dataset for both training and testing.

[0067] In the input representation, this invention references the ClipBERT settings in VideoQA tasks to extract video question features. During the external knowledge acquisition stage, in addition to the visual and external knowledge obtained through CLIP denoising, this invention also excludes some information-poor knowledge, such as "person," "hair," and "sky." Furthermore, when retrieving knowledge bases containing video objects, this invention only retains the top three weighted triads to represent knowledge confidence; the total number of knowledge base objects retrieved for each video is 30. This invention extracts external knowledge from videos, not from video clips. During knowledge feature extraction, this invention sets the dimension of the contextual memory unit to 768.

[0068] During the training phase, this invention sets the initial learning rate to 5e-5, warms it up in the first 10% of training steps, and then linearly decays it to 0. The batch size is set to 256. For each task, this invention trains the model for 50 epochs.

[0069] For the training model described in step (4), the specific steps are as follows: For the answer vector generated in step (3), compare it with the correct answer to the question, calculate the difference between the predicted value and the actual correct value through the defined loss function XE, and form the loss value. Then, based on the loss value, use the back-propagation algorithm (BP) to adjust the parameter values ​​of the entire network until the network converges.

[0070] Table 1 shows the accuracy of the method described in this paper on the Open-QA dataset. KcGA represents the method of this invention, and Acc represents the accuracy of this invention. KcGA achieves an overall accuracy of 27.04%, which is up to 13.61% higher and down to 6.68% higher than other methods.

[0071] Table 1: Results (accuracy) of the method of this invention on the Open-QA dataset.

[0072]

[0073] Table 2 shows the experimental results of different multi-stream information interaction mechanisms on the Open-QA dataset (see table for details). Figure 2 The methods in Table 2 are listed in top-to-bottom order, corresponding to... Figure 2 Methods (a) to (d) are used in the study. KcGA achieved an overall accuracy of 27.04%, outperforming other methods by a maximum of 2.03% and a minimum of 0.93%.

[0074] Table 2: Results (accuracy) of different multi-stream information interaction mechanisms on the Open-QA dataset.

[0075]

Claims

1. An open video question answering method based on external knowledge constraints, characterized in that, Comprising the following steps: Step (1), data preprocessing, feature extraction of video and question data; Step (2), introducing external knowledge features with adaptive external knowledge module, capturing key information between video question features and external knowledge features through multi-stream information interaction mechanism; Step (3), answer generation: using transformer as answer generation module, and using pre-trained model GPT2 for parameter initialization, so as to allow the model to selectively focus on the most relevant input feature part of the predicted answer; Step (4), model training: input the predicted answer in step (3) and the standard answer provided in the data set into the pre-defined loss function to obtain the corresponding LOSS; The adaptive external knowledge module introduced in step (2) to introduce external knowledge features is implemented as follows: 2-1. To represent the external knowledge of the video, it is proposed to fit the visual knowledge in the video frame f i to query the corresponding external knowledge k E in the external common sense knowledge base; and then based on the external knowledge k E to construct the knowledge feature F K , where i = 1,..., L through the iterative memory network. 2-2. Extract video entity objects from each fitted video frame f i using Faster R-CNN with ResNet-101, and represent the text description of the entity objects as o i ; Denoising the text description o i using the CLIP model, filtering out irrelevant objects by calculating the similarity between the fitted video frame f i and the text description o i ; the denoising process is described as: wherein the output entity object label is denoted as visual knowledge k V , CLIP(·) represents a CLIP model, and a is a predefined parameter; 2-3. To obtain external knowledge, a set of triples <o, r, s> are retrieved from the common sense knowledge base ConceptNet, where o and s are text labels of objects and r represents the relationship between the objects. By comparing the text labels of each object o V in the visual knowledge k i with o and s in all triples, all triples whose o or s are similar to the text description o i are selected, and then o and s in these similar triples are reorganized into sets and respectively. Meanwhile, CLIP denoising is also performed: where the set and the set both represent the objects described by the external knowledge, k e is the denoised external knowledge, and β = 0.19 is the threshold value; 2-4. Set k e Embedding in WordPiece, get the corresponding vector representation: To fully exploit the external knowledge k E between adjacent frames, a context-aware knowledge representation is computed using a GRU algorithm based on attention; for this a set of episodic memory units {m 1 , m 2 ,..., m T} is used to record the external knowledge k E at each time instant t; the specific computation is as follows: m 0 = tanh(W p k E + b p ), (Equation 5) z t = [k E ·m 0 , k E ·m t-1 , |k E -m 0 |, |k E -m t-1 |], (Equation 6) Among them, z t Capturing the similarity between external knowledge and memory, g t The attention operation performed by the model on each knowledge is defined, |·| is the absolute value, · represents the dot product between elements, and e t Let e ​​be the state vector of the GRU at time t. 0 =0; By appropriately setting the parameters and the number of GRU states, g t and e t Having and k E For the same size, e is calculated one by one using formula (8). t Elements, among which k r E and This corresponds to an element in the GRU; subsequently, the relevant information is updated using formula (9), where This is the final state of GRU; after the iteration is complete, the external knowledge feature F K It can be represented as: F K = ReLU(W f [m 1 ,...,m t ]+ b f ) (Equation 10) where m t represents the final memory information; b f , b g1 , b g2 , b p all represent hyperparameters to be learned.

2. The open video question answering method based on external knowledge constraint according to claim 1, characterized in that Step (1) is implemented as follows: 1-1. Randomly divide the video V of the L-frame into N clips C = {c1,..., c N} of equal length, and embed the problem text into a vector Q of 100-dimensional word vectors; 1-2. Use the baseline model ClipBERT as a video-text encoder to extract good performance of video and question public context information; video question feature representation F V is: F V = E ClipBERT (C, Q) (Equation 1) where E ClipBERT (·) denotes the video-text encoder model.

3. The open video question answering method based on external knowledge constraints according to claim 2, characterized in that The multi-stream information interaction mechanism in step (2) for capturing key information between video question features and external knowledge features is implemented as follows: 2-5. The core of the multi-flow information control mechanism is to capture the key information between the video problem features and the external knowledge features through the attention mechanism, which pays attention to different patterns; multi-flow refers to that the module takes three groups of flow information as input, i.e., the video problem features F V , the external knowledge features F K , and the fusion features; 2-6. Computing video question features F V and external knowledge features F K between global attention F C : F C = Sigmoid(BN(W c [F V ,F K + b c ))(Equation 11) Wherein, BN represents the standardized batch processing; 2-7. Based on global attention F C , the local attention is calculated respectively according to different features, and the local attention is integrated to obtain the video question feature F after fusion of knowledge enhancement M , that is, the fusion feature, and the specific calculation is as follows: F M = ReLU(W m F C ) T F V ,(F C ) T F K + b m )(Equation 12) where b c , b m all represent hyperparameters to be learned.

4. The open video question answering method based on external knowledge constraints according to claim 3, characterized in that The answer generation process of step (3) is described as follows: y = D GPT2 (F M )(Equation 13) where y is the predicted answer, and D GPT2 (·) is a decoder structure based on GPT2 initialization.

5. An open video question answering system capable of effectively generating out-of-domain answers, the system is implemented based on the method of any one of claims 1-4, characterized in that, It includes video-text encoder, adaptive external knowledge module, multi-stream information control mechanism and language decoder; The video-text encoder is used to extract video question features according to the context information of video and question; The adaptive external knowledge module is used to fit the detected video object to the knowledge base to obtain external knowledge related to the video content and provide implicit constraints for answer generation; The multi-stream information control mechanism is used to fuse video question features and external knowledge through local and global attention to obtain knowledge-enhanced video question features, and finally input the language decoder to freely generate answers, wherein the core of the multi-stream information control mechanism is to capture the key information between the video question features and the external knowledge features through the attention mechanism, and the attention mechanism pays attention to different patterns.

Images