A video question-answer modeling method based on a hierarchical selection framework

Abstract

The application discloses a video question and answer modeling method based on a hierarchical selection framework, characterized in that: firstly, video data and text data are preprocessed to obtain video segment features, object features, question features, question word features and candidate answer features as inputs of a video question and answer model; video segment features, video frame features and object features are selected hierarchically from large to small according to time span and space span, and then are fused with question features to realize prediction of answers. The method can be combined with different pre-training models for training and fine-tuning, that is, plug and play; the video segment features, video frame features and object features related to questions can be selected hierarchically to realize extraction of question-related key features and filtering of irrelevant redundant information.

Description

Technical Field

[0001] The present invention belongs to the multimodal field combining computer vision and natural language processing, and specifically relates to a video question-answering modeling method based on a hierarchical selection mechanism. Background Art

[0002] In recent years, the widespread popularity of self-media has generated a vast amount of video data, sparking research and development of video understanding models. Video question answering is one of the primary tasks in video understanding, and current video question answering models are widely used in fields such as autonomous driving, education and training, and interactive entertainment. With the development of deep learning methods, the Transformer architecture has become capable of unifying multiple modalities (such as images, language, and audio). In the field of video question answering, a mainstream framework consists of a visual backbone network and a text backbone network. The visual backbone network processes the video data, while the text backbone network processes the text information (such as questions and answers).

[0003] Early video question-answering models primarily processed short video clips (less than 10 seconds in length). These videos generally contained information about objects and actions, and the corresponding questions were in the form of questions like whether an object appeared in the video or whether a certain action occurred in the video. However, in addition to these short videos, there is also a large amount of long video data. Compared to short videos, these videos contain richer temporal information. For example, a video may contain multiple events and interactions between multiple objects. The corresponding questions are also more complex, such as what someone did after an event. Compared to image data, the biggest difference lies in the inter-frame relationships and temporal information of videos. Therefore, modeling these long video data can better reflect the model's ability to understand videos.

[0004] Since long-duration video data contains richer information, this means that the model needs to sample (extract) more video frames (such as 16 frames and 32 frames) to retain as much of the original information of the video as possible. However, as the number of sampled video frames increases, the input of the model will also become larger, and subsequent processing will become more complicated. Therefore, for video question answering tasks, the model needs to focus on the part of the data in the video that is related to a specific question. This part of the data may be a video clip, a video frame, or information about an object, or the interaction between them. This seems to be a contradiction: for long video data, we hope to avoid losing information as much as possible in the preprocessing stage, and we also want the model to pay attention to the part of the data related to the specific question in the data processing stage.

[0005] To address the potential problems of computational inefficiency and a lack of focus on key information in long video question answering, some existing methods have designed specific feature selection modules and embedded them into video question answering models to select question-relevant features. However, these methods only sparsely extract question-relevant signals from the video and fail to fully consider the multi-granularity nature of long video question answering. Consequently, these methods still have limitations when dealing with complex problems involving long time series and multiple granularities. Summary of the Invention

[0006] In response to the existing steps in the prior art, the present invention provides a video question answering modeling method based on a hierarchical selection framework. Considering that the question may be directly related to a certain clip, frame, or object in the video, it may also reflect the interaction between them, for example:

[0007] Related fragment: "Determine whether event B occurs after event A".

[0008] Frame related: "Where was this video taken?"

[0009] Object related: "Is object A on top of object B?"

[0010] Interaction relationship: "Does object A appear in event B?"

[0011] This method can hierarchically select video clip features, video frame features, and object features based on time span and spatial span from large to small, and then fuse them with question features to predict the answer. The specific implementation steps are as follows:

[0012] Step 1: Preprocess the video data to obtain video clip features, object features, and object features.

[0013] Step 2: Preprocess text data to obtain question features, question word features, and candidate answer features.

[0014] Step 3: The video clip features, video frame features, object features, question features, question word features, and candidate answer features obtained above are used as input to the video question answering model. After entering the video question answering model, these features are first passed through a linear layer to uniformly map their feature dimensions to d dimensions.

[0015] Step 4: Video segment selection. The video segment selection module of the video question answering model selects the question feature X output by the linear layer. q , video clip features X s , video frame features X f and object features X o Processing to obtain the selected video clip features Video frame features and object features

[0016] Step 5: Video frame selection. q , video frame features selected by the video clip selection module and object features Input into the video frame selection module to obtain the selected video frame features and object features

[0017] Step 6: Object selection. q and object features selected by the video frame selection module Input into the object selection module to obtain the selected object features

[0018] Step 7: Perform cross-modal interaction and fusion on the selected multi-granularity visual features and question word features.

[0019] Step 8: Perform an average pooling operation on the fusion output obtained in step 7 to obtain a global fusion feature. Subsequently, the similarity between the global fusion feature and the candidate answer feature is calculated, and the candidate answer with the highest similarity is used as the final answer prediction.

[0020] Furthermore, the specific operations of step 1 are as follows:

[0021] (1-1) For a dataset with n videos, each video contains several questions and answers. Set the number of sampling frames to n f , averagely sample n videos and convert the sampled video frames into RGB images with a width of w, a height of h and 3 channels. The final output size is (n,n f ,3,w,h) video frame data.

[0022] (1-2) Select a pre-trained visual encoder F v Perform feature extraction on each image of size (3, w, h) in the video frame data. Let the feature dimension be d v , then the video frame data is extracted by features and the size is (n,n f ,d v ) video frame features X f0 . Set the segment length to l s , then the number of fragments is n s =n f / l s , where n s Take an integer. At this time, the video frame features can be further divided into (n,n s ,ls ,d v ) of X f .

[0023] (1-3) For the size (n,n s ,l s ,d v ) video frame features X f , in l s The average pooling method is used to transform the dimension into a size of (n,n s ,d v ) video clip feature X s .

[0024] (1-4) Set the number of detection targets n o , select a pre-trained target detection model F o Perform target detection on each image of size (3, w, h) in the video frame data, and then convert the detected target object area into an image of the same size as the video frame, and finally obtain an image of size (n, n f ,n o ,3,w,h) object data. Then, the visual encoder F v Extract features from each object in the object data and finally get a f ,n o ,d v ) object feature X o .

[0025] Therefore, the features obtained after data preprocessing include video frame features X f , video clip features X s and object features X o It should be noted that, in order to reduce the amount of computation, the pre-trained visual encoder and target detection model used in video data preprocessing do not participate in subsequent training.

[0026] Furthermore, the specific operations of step 2 are as follows:

[0027] The text data contains questions and answers. Select a pre-trained language encoder F t Extract features for each question and answer. Let the feature dimension be d t , then for each question, extract the corresponding question feature X q (The feature dimension is d t ); For each answer, extract the corresponding answer feature X a (The feature dimension is d t ). Let the length of the problem be l q , then for each word in the question, extract the corresponding question word feature Xw , the size is (l q ,d t ). Different from the visual part, the language encoder F t Requires participation in subsequent training.

[0028] Furthermore, the specific operations of step 3 are as follows:

[0029] The above features are used as the input of the video question answering model, and the above features include the video clip feature X s , video frame features X f , object feature X o 、Problem feature X q , question word feature X w and candidate answer features X a After being input into the video question answering model, these features first pass through a linear layer to uniformly map their feature dimensions to d dimensions. For convenience, the symbolic representation of the output of this linear layer is still defined as X s ,X f ,X o ,X q ,X w and X a .

[0030] Furthermore, the specific method of step 4 is as follows:

[0031] (4-1) The video segment selection module has four inputs, namely the problem feature X output by the linear layer q , video clip features X s , video frame features X f and object features X o .

[0032] (4-2) Set the number k of video clips to be selected s .

[0033] Calculate the problem feature X q and video clip feature X s The similarity score is converted into a probability distribution logit using the softmax function s , the formula is as follows:

[0034]

[0035] Then, the Gumbel-Softmax method is used to analyze the logit s Perform k s Subsampling, get the selection matrix Logit' about the video clip features s , the formula is as follows:

[0036]

[0037] (4-3) Video segment features selected by the video segment selection module Video frame features and object features By inputting X s 、X f 、X o and Logit' s The matrix multiplication is obtained.

[0038]

[0039] Furthermore, the specific method of step 5 is as follows:

[0040] (5-1) The video frame selection module has three inputs, namely, the problem feature X q , video frame features selected by the video clip selection module and object features

[0041] (5-2) Set the number k of video frames to be selected f Calculate the problem feature X q and video frame features The similarity score is converted into a probability distribution logit using the softmax function f , the formula is as follows:

[0042]

[0043] Then, the Gumbel-Softmax method is used to analyze the logit f Perform k f Subsampling, get the selection matrix Logit' about the video frame features f , the formula is as follows:

[0044]

[0045] (5-3) The video frame features and object features selected by the video frame selection module are respectively input through and Logit' f The matrix multiplication of is:

[0046]

[0047] Furthermore, the specific method of step 6 is as follows:

[0048] (6-1) The object selection module has two inputs, namely the problem feature X q and object features selected by the video frame selection module

[0049] (6-2) Set the number k of video frames to be selected o Calculate the problem feature X q and video frame features The similarity score is converted into a probability distribution logit using the softmax function o , the formula is as follows:

[0050]

[0051] Then, the Gumbel-Softmax method is used to analyze the logit o Perform k o Sampling, get the selection matrix Logit' about the object features o , the formula is as follows:

[0052]

[0053] Logit' obtained by fusion o Contains selected information about object features.

[0054] (6-3) The object features selected by the object selection module are input and Logit' o The matrix multiplication of is:

[0055]

[0056] Furthermore, the specific method of step 7 is as follows:

[0057] The video segment features obtained by the video segment selection module, video frame selection module and object selection module in steps 4 to 6 are respectively Video frame features and object features Collectively referred to as multi-granularity visual features, and combined with the question word feature X w After concatenation, an L-layer Transformer layer is introduced to perform cross-modal interaction and fusion of the concatenated question word features and multi-granularity visual features, thereby building a deep correspondence between questions and answers:

[0058]

[0059] Z out =TransformerLayer(Z in )

[0060] Furthermore, during the video question answering model training process, the modules involved in the training include the language encoder, linear mapping layer, video clip selection module, video frame selection module, object selection module, and Transformer layer. The visual encoder and object detection model in step 1 are not involved in the training.

[0061] The training objective of the model is to minimize the predicted answer The cross entropy loss between the correct answer a is:

[0062]

[0063] The advantages and beneficial results of the method of the present invention are as follows:

[0064] (1) The hierarchical selection framework designed by the method of the present invention has good versatility and can be combined with different pre-trained models for training and fine-tuning, and can be plug-and-play.

[0065] (2) The hierarchical selection framework designed by the method of the present invention is optimized for the long time series and multi-granularity problems in video question answering tasks. It can hierarchically select video clips, video frames and object features related to the question, extract key features related to the question, and filter irrelevant redundant information.

[0066] (3) The hierarchical selection framework designed by the method of the present invention reduces the amount of data input to the Transformer layer by first selecting the input data, thereby improving the computational efficiency of the video question answering model and reducing the training cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 This is the overall idea diagram of the present invention.

[0068] Figure 2 Graph showing the accuracy change of the model in the embodiment. DETAILED DESCRIPTION

[0069] The present invention will be described in detail below with reference to specific embodiments.

[0070] The overall idea of ​​the present invention is as follows Figure 1 As shown in the figure, it is mainly reflected in the design and introduction of three feature selection modules and a hierarchical selection framework, which enables the model to select features of video clips, video frames and objects according to the problem.

[0071] A video question answering modeling method based on a hierarchical selection framework, the steps are as follows:

[0072] Step 1: Video data preprocessing.

[0073] (1-1) Construct a video question answering dataset, which contains n videos, each video contains several questions and answers. Set the number of sampling frames n f =32, averagely sample n videos, and convert the sampled video frames into RGB images with width w, height h, and 3 channels. The final output video frame data is (n, 32, 3, 224, 224).

[0074] (1-2) Select the image encoder part of the pre-trained CLIP (ViT-B / 32) model as the visual encoder F v , used to extract features from each image of size (3,224,224) in the video frame data. Assuming the feature dimension is 512, the video frame data is extracted to obtain a video frame feature of size (n,32,512). Set the segment length to 8, and the number of segments is 4. At this time, the video frame features can be further divided into data X of size (n, 8, 4, 512) f CLIP does not participate in the training of subsequent models.

[0075] (1-3) For video frame features X of size (n, 8, 4, 512) f , using the average pooling method in the video frame dimension to convert the data into a video clip feature X of size (n, 8, 512) s .

[0076] (1-4) Set the number of detection targets to 16 and select the pre-trained Faster-RCNN model as the target detection model F o , used to detect the target in each image of size (3,224,224) in the video frame data, and then convert the detected target object area into an image of the same size as the video frame, obtaining object data of size (n,32,16,3,224,224). Subsequently, the image model F v Extract features from each object in the object data and finally obtain object features X of size (n, 32, 16, 512) o Faster-RCNN does not participate in the training of subsequent models.

[0077] Step 2: Text data preprocessing. The text data contains questions and candidate answers. The pre-trained DeBERTa-base model is selected as the language encoder F t , used to extract features for each question and candidate answer, the feature dimension is 768, that is, the question feature X corresponding to each question q The feature dimension is 768, and each candidate answer corresponds to the answer feature X aThe feature dimension of is 768. Let the length of the problem be l q , then for each word in the question, extract the corresponding question word feature X w , the size is (l q ,768). Deberta participated in the training of the video question answering model.

[0078] Step 3: The video clip feature X obtained in steps 1 and 2 s , video frame features X f , object feature X o 、Problem feature X q , question word feature X w and candidate answer features X a They are input into their respective single-layer linear layers, and their feature dimensions are uniformly mapped to 512 dimensions. For convenience, the symbolic representation of the linear layer output is still represented by X s ,X f ,X o ,X q ,X w and X a express.

[0079] Step 4: Video clip selection. q , video clip features X s , video frame features X f and object features X o Input to the video segment selection module, and its output is the video segment features selected by the video segment selection module Video frame features and object features

[0080] (4-1) The video segment selection module has four inputs, namely the problem feature X output by the linear layer q , video clip features X s , video frame features X f and object features X o .

[0081] (4-2) Set the number k of video clips to be selected s .

[0082] Calculate the problem feature X q and video clip feature X s The similarity score is converted into a probability distribution logit using the softmax function s , the formula is as follows:

[0083]

[0084] Then, the Gumbel-Softmax method is used to analyze the logit s Perform k s Subsampling, get the selection matrix Logit' about the video clip features s , the formula is as follows:

[0085]

[0086] (4-3) Video segment features selected by the video segment selection module Video frame features and object features By inputting X s 、X f 、X o and Logit' s The matrix multiplication is obtained.

[0087]

[0088] After the video clip is selected, subsequent video frame selection and object selection will be performed among the video frames contained in the selected video clip.

[0089] Step 5: Video frame selection. q , video frame features selected by the video clip selection module and object features Input into the video frame selection module, and its output is the video frame features selected by the video frame selection module and object features

[0090]

[0091] (5-1) The video frame selection module has three inputs, namely, the problem feature X q , video frame features selected by the video clip selection module and object features

[0092] (5-2) Set the number k of video frames to be selected f Calculate the problem feature X q and video frame features The similarity score is converted into a probability distribution logit using the softmax function f , the formula is as follows:

[0093]

[0094] Then, the Gumbel-Softmax method is used to analyze the logit fPerform k f Subsampling, get the selection matrix Logit' about the video frame features f , the formula is as follows:

[0095]

[0096] (5-3) The video frame features and object features selected by the video frame selection module are respectively input through and Logit' f The matrix multiplication of is:

[0097]

[0098] Step 6: Object selection. q and object features selected by the video frame selection module Input to the object selection module, and its output is the object features selected by the object selection module

[0099] (6-1) The object selection module has two inputs, namely the problem feature X q and object features selected by the video frame selection module

[0100] (6-2) Set the number k of video frames to be selected o Calculate the problem feature X q and video frame features The similarity score is converted into a probability distribution logit using the softmax function o , the formula is as follows:

[0101]

[0102] Then, the Gumbel-Softmax method is used to analyze the logit o Perform k o Sampling, get the selection matrix Logit' about the object features o , the formula is as follows:

[0103]

[0104] Logit' obtained by fusion o Contains selected information about object features.

[0105] (6-3) The object features selected by the object selection module are input and Logit' o The matrix multiplication of is:

[0106]

[0107] Step 7: The video segment features obtained by the video segment selection module, video frame selection module and object selection module in steps 4 to 6 are respectively selected Video frame features Object characteristics Collectively referred to as multi-granularity visual features, and combined with the question word feature X w Then, L layers of Transformer are introduced to perform cross-modal interaction and fusion of the spliced ​​question word features and multi-granularity visual features, and a deep correspondence relationship from question to answer is constructed to obtain the fused feature Z out .

[0108]

[0109] Z out =TransformerLayer(Z in )

[0110] Step 8: The fusion feature Z obtained in step 7 out Implement average pooling operation to obtain global fusion features Afterwards, With candidate answer feature X a Calculate the similarity and take the candidate answer with the highest similarity as the final answer. The prediction formula is as follows:

[0111]

[0112] Obtain an existing dataset to train the video question answering model. The modules involved in the training include the language encoder, linear mapping layer, video clip selection module, video frame selection module, object selection module, and Transformer layer. The visual encoder and object detection model in step 1 are not included in the training.

[0113] The training objective of the model is to minimize the predicted answer The cross entropy loss between the correct answer a is:

[0114]

[0115] Example

[0116] This example uses the publicly available Next-QA dataset, a benchmark dataset for multiple-choice video question answering (QA) for causal and temporal reasoning. It contains 5,440 videos with an average length of 44 seconds and approximately 52,000 questions.

[0117] According to the steps described in the specific implementation, in the data preprocessing stage, the visual encoder used in this embodiment is the pre-trained CLIP (ViT-B / 32) model, the target detection model is Faster-RCNN, and the language model is DeBERTa-base. In the data selection stage, this embodiment selects the number of video clip features k s Set to 2, select the number k of video frame features f Set to 2, select the number of object features k o Set to 12. During the data fusion phase, this embodiment sets the number of Transformer layers to 4. In terms of model training settings, the loss function is set to the cross-entropy loss function, the learning rate is set to 0.00003, the dropout probability is set to 0.3, the training optimizer is set to the Adam optimizer, the data mini-batch size is set to 64, and the iteration period is set to 30. In addition, during the training process, gradient clipping is applied to the model parameter optimization (the maximum norm is set to 12).

[0118] During the training phase, the Logit output of the selection module is shown in Table 1:

[0119] Table 1 Logit output of each selection module

[0120]

[0121] The accuracy change of model training is as follows Figure 2 As shown, the accuracy of the training data set reaches 91.53%, and the accuracy of the test data set can reach 58.80%. The detailed results are shown in Table 2:

[0122] Table 2 Test dataset results

[0123] Dataset ACC@C ACC@T ACC@D ACC Next-QA 57.51％ 57.09％ 66.19％ 58.80％

[0124] Among them, ACC@C, ACC@T and ACC@D indicate that the model achieved a test accuracy of 57.51%, 57.09% and 66.19% on causal, temporal and descriptive questions respectively, and ACC indicates that the model achieved a test accuracy of 58.80% on all questions.

[0125] The above description is a further detailed description of the present invention in conjunction with specific / preferred embodiments, and the specific implementation of the present invention should not be considered to be limited to these descriptions. Those skilled in the art of the present invention may make various substitutions or modifications to the described embodiments without departing from the scope of the present invention, and such substitutions or modifications should be considered to fall within the scope of protection of the present invention.

[0126] Parts of the present invention that are not described in detail belong to the common knowledge of those skilled in the art.

Claims

1. A video question answering modeling method based on a hierarchical selection framework, characterized in that: Please follow the steps below to implement: Step 1: Preprocess the video data to obtain video clip features, video frame features, and object features; Step 2: Preprocess the text data to obtain question features, question word features, and candidate answer features; Step 3: The video clip features, video frame features, object features, question features, question word features and candidate answer features obtained above are used as the input of the video question answering model; after inputting these features into the video question answering model, they are firstly passed through a linear layer to uniformly map the feature dimensions of the video clip features, video frame features, object features, question features, question word features and candidate answer features to dimension; Step 4: Video clip selection; the video clip selection module of the video question answering model selects the question features output by the linear layer , video clip features , video frame features and object features Processing to obtain the selected video clip features , video frame features and object features ; Step 5: Video frame selection; problem features , video frame features selected by the video clip selection module and object features Input into the video frame selection module to obtain the selected video frame features and object features ; Step 6: Object selection; characterize the problem and object features selected by the video frame selection module Input into the object selection module to obtain the selected object features ; Step 7: Perform cross-modal interaction and fusion on the selected multi-granularity visual features and question word features; Step 8: Perform average pooling on the fusion output obtained in step 7 to obtain the global fusion feature; Calculate the similarity between the global fusion features and the candidate answer features, and take the candidate answer with the highest similarity as the final answer prediction; Step 1: (1-1) For a dataset with n videos, each video contains several questions and answers; set the number of sampling frames to ,right The video is sampled evenly and the sampled video frames are converted into frames with a width of Gao Wei And the number of channels of RGB image is 3, the output size is Video frame data; (1-2) Select a pre-trained visual encoder For each frame in the video frame data, the size is Extract features from images; set up The feature dimension is , then the video frame data is extracted by features and the size is Video frame features ; Set the fragment length to , then the number of fragments is ,in Take an integer; at this time, the video frame features can be further divided into of ; (1-3) For the size of Video frame features ,exist The dimension is converted into a size of Video clip features ; (1-4) Set the number of detection targets , select a pre-trained target detection model For each frame in the video frame data, the size is The detected target object area is converted into an image of the same size as the video frame, and the size is obtained. object data; using visual encoder Extract features from each object in the object data and get a size of Object features .

2. A video question answering modeling method based on a hierarchical selection framework according to claim 1, characterized in that: Step 2: The text data contains questions and answers; select a pre-trained language encoder Perform feature extraction on each question and answer; set up The feature dimension is , then for each question, the corresponding question features are extracted ; For each answer, extract the corresponding answer features ; Assume the length of the problem is , then for each word in the question, extract the corresponding question word features , the size is ; Unlike the visual part, the language encoder Requires participation in subsequent training.

3. The video question answering modeling method based on a hierarchical selection framework according to claim 1 is characterized in that: Step 3: The obtained video clip features , video frame features , object features , Problem Characteristics , question word features and candidate answer features As the input of the video question answering model; after entering the video question answering model, these features first pass through a linear layer to uniformly map the feature dimensions of video clip features, video frame features, object features, question features, question word features, and candidate answer features to Dimension; the symbolic representation of the linear layer output is still defined as and .

4. A video question answering modeling method based on a hierarchical selection framework according to claim 3, characterized in that: Step 4: (4-1) The video segment selection module has four inputs, which are the problem features output by the linear layer , video clip features , video frame features and object features ; (4-2) Set the number of video clips to be selected ; Computational Problem Characteristics and video clip features The similarity score is converted into a probability distribution using the softmax function , the formula is as follows: The Gumbel-Softmax method is used to conduct Subsampling, get the selection matrix of video clip features , the formula is as follows: (4-3) Video clip features selected by the video clip selection module , video frame features and object features By input 、 、 and The matrix multiplication of is obtained; 。 5. The video question answering modeling method based on a hierarchical selection framework according to claim 4 is characterized in that: Step 5: (5-1) The video frame selection module has three inputs, namely, the problem features , video frame features selected by the video clip selection module and object features ; (5-2) Set the number of video frames to be selected ; Computational problem characteristics and video frame features The similarity score is converted into a probability distribution using the softmax function , the formula is as follows: The Gumbel-Softmax method is used to conduct Subsampling, get the selection matrix of video frame features , the formula is as follows: (5-3) The video frame features and object features selected by the video frame selection module are respectively input 、 and The matrix multiplication of is: 。 6. The video question answering modeling method based on a hierarchical selection framework according to claim 5 is characterized in that: Step 6: (6-1) The object selection module has two inputs, namely the problem features and object features selected by the video frame selection module ; (6-2) Set the number of video frames to be selected ; Computational problem characteristics and video frame features The similarity score is converted into a probability distribution using the softmax function , the formula is as follows: The Gumbel-Softmax method is used to conduct Subsampling, get the selection matrix of object features , the formula is as follows: The fusion Contains selected information about object features; (6-3) The object features selected by the object selection module are input and The matrix multiplication of is: 。 7. The video question answering modeling method based on a hierarchical selection framework according to claim 6 is characterized in that: Step 7: The video segment features obtained by the video segment selection module, video frame selection module and object selection module in steps 4 to 6 are respectively , video frame features and object features Collectively referred to as multi-granularity visual features, and combined with question word features After splicing, an L-layer Transformer layer is introduced to perform cross-modal interaction and fusion of the spliced ​​question word features and multi-granularity visual features, building a deep correspondence between questions and answers: 。 8. The video question answering modeling method based on a hierarchical selection framework according to claim 1 is characterized in that: During the video question answering model training process, the modules involved in the training include the language encoder, linear mapping layer, video clip selection module, video frame selection module, object selection module, and Transformer layer; the visual encoder and object detection model in step 1 are not involved in the training; The training objective of the model is to minimize the predicted answer and the correct answer The cross entropy loss between: 。

Images