Video question and answer modeling method based on hierarchical selection framework

Abstract

The invention discloses a video question and answer modeling method based on a hierarchical selection framework, which is characterized by comprising the following steps: firstly, preprocessing video data and text data to obtain video clip features, object features, question features, question word features and candidate answer features as input of a video question and answer model; video clip features, video frame features and object features are selected hierarchically from large to small according to the time span and the space span, and then feature fusion with question features is carried out to realize prediction of answers. The method can be combined with different pre-training models for training and fine tuning, and is plug-and-play; video clips, video frames and object features related to problems can be selected hierarchically, key features related to the problems can be extracted, and irrelevant redundant information can be filtered.

Description

Technical Field

[0001] The present invention belongs to the multimodal field of combining computer vision with natural language processing, and specifically relates to a video question-answering modeling method based on a hierarchical selection mechanism. Background Art

[0002] In recent years, the widespread popularity of self-media has brought a large amount of video data, and has also aroused the research and development of video understanding models by relevant workers. Video question answering is one of the main tasks of video understanding. Currently, video question answering models are widely used in fields such as autonomous driving, education and training, and interactive entertainment. With the development of deep learning related methods, the Transformer architecture can achieve the unification of multiple modalities (such as images, languages, audio, etc.). In the field of video question answering, a mainstream framework contains a visual backbone network and a text backbone network. Among them, the visual backbone network is responsible for processing video data, and the text backbone network is responsible for processing text information (such as questions and answers).

[0003] Early video question-answering models mainly processed short video clips (less than 10 seconds in length). In these video data, the information contained is generally objects, action information, etc., and the corresponding question forms are such as whether a certain object appears in the video, whether a certain action appears in the video, etc. However, in addition to these short videos, there are also a large number of long-duration video data. Compared with short videos, these videos contain richer temporal information, such as the occurrence of multiple events and interactions between multiple objects in the video, and the corresponding questions are also more complex, such as what someone did after an event occurred. Compared with image data, the biggest difference lies in the relationship between frames and temporal information of the video. Therefore, modeling these long-duration video data can better reflect the model's ability to understand the video.

[0004] Since long-duration video data contains richer information, this means that the model needs to sample (extract) more video frames (such as 16 frames and 32 frames) to retain as much of the original information of the video as possible. However, as the number of sampled video frames increases, the input of the model will also become larger, and subsequent processing will become more complicated. Therefore, for video question-answering tasks, the model needs to focus on the part of the data in the video that is related to a specific question. This part of the data may be the information of a video clip, a video frame, or an object, or it may be the interaction between them. This seems to be a contradiction: for long video data, we hope not to lose information as much as possible in the preprocessing stage, but also let the model pay attention to the part of the data related to the specific question in the data processing stage.

[0005] In order to solve the problems of low computational efficiency and lack of prominence of key information in long video question answering, some existing methods have designed specific feature selection modules and embedded them into the video question answering model to select question-related features. However, these methods only sparsely extract question-related signals from the video and do not fully consider the multi-granularity characteristics in long video question answering. Therefore, these methods still have limitations in dealing with complex problems involving long time series and multiple granularities. Summary of the invention

[0006] In view of the existing steps in the prior art, the present invention provides a video question answering modeling method based on a hierarchical selection framework. Considering that the question may be directly related to a certain clip, a certain frame or a certain object in the video, it may also reflect the interaction between them, for example:

[0007] Fragment related: "Determine whether event B occurs after event A".

[0008] Frame related: "Where was this video taken?"

[0009] Object related: "Is object A on top of object B?"

[0010] Interaction relationship: "Does object A appear in event B?"

[0011] This method can hierarchically select video clip features, video frame features, and object features from large to small according to the time span and space span, and then fuse them with the question features to predict the answer. The specific implementation steps are as follows:

[0012] Step 1: Preprocess the video data to obtain video clip features, object features, and object characteristics.

[0013] Step 2: Preprocess text data to obtain question features, question word features, and candidate answer features.

[0014] Step 3: The video clip features, video frame features, object features, question features, question word features, and candidate answer features obtained above are used as the input of the video question answering model. After being input into the video question answering model, these features are firstly passed through a linear layer to uniformly map their feature dimensions to d dimensions.

[0015] Step 4: Video segment selection. The video segment selection module of the video question answering model selects the question feature X output by the linear layer. q , Video clip features X s , video frame feature X f and object feature X o Processing to obtain the selected video clip features Video frame features and object features

[0016] Step 5: Video frame selection. q , Video frame features selected by the video segment selection module and object features Input into the video frame selection module to obtain the selected video frame features and object features

[0017] Step 6: Object selection. q And the object features selected by the video frame selection module Input into the object selection module to obtain the selected object features

[0018] Step 7: Perform cross-modal interaction and fusion on the selected multi-granularity visual features and question word features.

[0019] Step 8: Perform an average pooling operation on the fusion output obtained in step 7 to obtain a global fusion feature. Subsequently, the similarity between the global fusion feature and the candidate answer feature is calculated, and the candidate answer with the highest similarity is used as the final answer prediction.

[0020] Furthermore, the specific operations of step 1 are as follows:

[0021] (1-1) For a dataset with n videos, each video contains several questions and answers. Set the number of sampling frames to n f , averagely sample n videos, and convert the sampled video frames into RGB images with a width of w, a height of h, and 3 channels. The final output size is (n,n f ,3,w,h) video frame data.

[0022] (1-2) Select a pre-trained visual encoder F v Perform feature extraction on each image of size (3, w, h) in the video frame data. Let the feature dimension be d v , then the video frame data after feature extraction has a size of (n,n f ,d v ) video frame features X f0 . Set the segment length to l s , then the number of fragments is n s =n f / l s , where n s Take an integer. At this time, the video frame features can be further divided into (n,n s ,ls ,d v ) f .

[0023] (1-3) For the size (n,n s ,l s ,d v ) video frame features X f , in l s The dimension is converted into a size of (n,n s ,d v ) video clip feature X s .

[0024] (1-4) Set the number of detection targets n o , select a pre-trained target detection model F o Perform target detection on each image of size (3,w,h) in the video frame data, and then convert the detected target object area into an image of the same size as the video frame, and finally obtain a (n,n f ,n o ,3,w,h) object data. Then, the visual encoder F v Extract features for each object in the object data, and finally get a size of (n,n f ,n o ,d v ) of the object feature X o .

[0025] Therefore, the features obtained after data preprocessing include video frame features X f , Video clip features X s and object feature X o It should be noted that in order to reduce the amount of computation, the pre-trained visual encoder and object detection model used in video data preprocessing do not participate in subsequent training.

[0026] Furthermore, the specific operations of step 2 are as follows:

[0027] The text data contains questions and answers. Select a pre-trained language encoder F t Extract features for each question and answer. Let the feature dimension be d t , then for each question, extract the corresponding question feature X q (The feature dimension is d t );For each answer, extract the corresponding answer feature X a (The feature dimension is d t ). Let the length of the problem be l q , then for each word in the question, extract the corresponding question word feature Xw , the size is (l q ,d t ). Different from the visual part, the language encoder F t Need to participate in follow-up training.

[0028] Furthermore, the specific operations of step 3 are as follows:

[0029] The above features are used as the input of the video question answering model, and the above features include the video clip feature X s , video frame feature X f 、Object feature X o 、Problem Features X q 、Question word feature X w And the candidate answer feature X a After being input into the video question answering model, these features first pass through a linear layer to uniformly map their feature dimensions to d dimensions. For convenience, the symbolic representation of the output of this linear layer is still defined as X s ,X f ,X o ,X q ,X w and X a .

[0030] Furthermore, the specific method of step 4 is as follows:

[0031] (4-1) The video segment selection module has four inputs, which are the problem features X output by the linear layer q , Video clip features X s , video frame feature X f and object feature X o .

[0032] (4-2) Set the number k of video clips to be selected s .

[0033] Calculate the problem feature X q and video clip feature X s The similarity score is converted into a probability distribution logit using the softmax function s , the formula is as follows:

[0034]

[0035] Then, the Gumbel-Softmax method is used to analyze the logit s Carry out k s Sub-sampling, get the selection matrix Logit' about the video clip features s , the formula is as follows:

[0036]

[0037] (4-3) Video segment features selected by the video segment selection module Video frame features and object features By inputting X s , X f , X o With Logit' s It is obtained by multiplying the matrices of .

[0038]

[0039] Furthermore, the specific method of step 5 is as follows:

[0040] (5-1) The video frame selection module has three inputs, namely, the problem feature X q , Video frame features selected by the video segment selection module and object features

[0041] (5-2) Set the number k of video frames to be selected f . Calculate the problem feature X q and video frame features The similarity score is converted into a probability distribution logit using the softmax function f , the formula is as follows:

[0042]

[0043] Then, the Gumbel-Softmax method is used to analyze the logit f Carry out k f Sub-sampling, get the selection matrix Logit' about the video frame features f , the formula is as follows:

[0044]

[0045] (5-3) The video frame features and object features selected by the video frame selection module are respectively input through With Logit' f Matrix multiplication of :

[0046]

[0047] Furthermore, the specific method of step 6 is as follows:

[0048] (6-1) The object selection module has two inputs, namely, the problem feature X q and object features selected by the video frame selection module

[0049] (6-2) Set the number k of video frames to be selected o . Calculate the problem feature X q and video frame features The similarity score is converted into a probability distribution logit using the softmax function o , the formula is as follows:

[0050]

[0051] Then, the Gumbel-Softmax method is used to analyze the logit o Carry out k o Sampling, get the selection matrix Logit' about the object features o , the formula is as follows:

[0052]

[0053] Logit' obtained by fusion o Contains selected information about the features of the object.

[0054] (6-3) The object features selected by the object selection module are input With Logit' o Matrix multiplication of :

[0055]

[0056] Furthermore, the specific method of step 7 is as follows:

[0057] The video segment features selected by the video segment selection module, the video frame selection module and the object selection module in steps 4 to 6 are respectively Video frame features and object features Collectively referred to as multi-granular visual features, and combined with the question word feature X w After concatenation, an L-layer Transformer layer is introduced to perform cross-modal interaction and fusion of the concatenated question word features and multi-granularity visual features, thus building a deep correspondence between questions and answers:

[0058]

[0059] Z out = TransformerLayer(Z in )

[0060] Furthermore, during the video question answering model training process, the modules involved in the training include the language encoder, linear mapping layer, video clip selection module, video frame selection module, object selection module, and Transformer layer. The visual encoder and object detection model in step 1 are not involved in the training.

[0061] The training objective of the model is to minimize the predicted answer The cross entropy loss between the correct answer a is:

[0062]

[0063] The advantages and beneficial results of the method of the present invention are as follows:

[0064] (1) The hierarchical selection framework designed by the method of the present invention has good versatility and can be combined with different pre-trained models for training and fine-tuning, and can be plug-and-play.

[0065] (2) The hierarchical selection framework designed by the method of the present invention is optimized for long-term, multi-granular, and other problems in video question-answering tasks. It can hierarchically select video clips, video frames, and object features related to the question, extract key features related to the question, and filter irrelevant redundant information.

[0066] (3) The hierarchical selection framework designed by the method of the present invention reduces the amount of data input to the Transformer layer by first selecting the input data, thereby improving the computational efficiency of the video question answering model and reducing the training cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 This is the overall idea diagram of the present invention.

[0068] Figure 2 Graph showing the accuracy change of the model in the embodiment. DETAILED DESCRIPTION

[0069] The present invention is described in detail below in conjunction with specific implementation modes.

[0070] The general idea of ​​the present invention is as follows Figure 1 As shown in the figure, it is mainly reflected in the design and introduction of three feature selection modules and a hierarchical selection framework, which enables the model to select features of video clips, video frames and objects according to the problem.

[0071] A video question answering modeling method based on a hierarchical selection framework, the steps are as follows:

[0072] Step 1: Video data preprocessing.

[0073] (1-1) Construct a video question-answering dataset, which contains n videos, each of which contains several questions and answers. Set the number of sampling frames n f =32, sample n videos on average, and convert the sampled video frames into RGB images with a width of w, a height of h, and 3 channels. Finally, output video frame data of size (n, 32, 3, 224, 224).

[0074] (1-2) Select the image encoder part of the pre-trained CLIP (ViT-B / 32) model as the visual encoder F v , used to extract features from each image of size (3,224,224) in the video frame data. Assuming the feature dimension is 512, the video frame data is extracted to obtain a video frame feature of size (n,32,512). Set the segment length to 8, and the number of segments is 4. At this time, the video frame features can be further divided into data X of size (n, 8, 4, 512) f . CLIP does not participate in the training of subsequent models.

[0075] (1-3) For a video frame feature X of size (n, 8, 4, 512) f , using the average pooling method in the video frame dimension to convert the data into a video segment feature X of size (n, 8, 512) s .

[0076] (1-4) Set the number of detection targets to 16 and select the pre-trained Faster-RCNN model as the target detection model F o , used to detect the target in each image of size (3,224,224) in the video frame data, and then convert the detected target object area into an image of the same size as the video frame, and obtain object data of size (n,32,16,3,224,224). Then, use the image model F v Extract features from each object in the object data, and finally obtain object features X of size (n, 32, 16, 512) o . Faster-RCNN does not participate in the training of subsequent models.

[0077] Step 2: Text data preprocessing. The text data contains questions and candidate answers. The pre-trained DeBERTa-base model is selected as the language encoder F t , used to extract features for each question and candidate answer, the feature dimension is 768, that is, the question feature X corresponding to each question q The feature dimension of is 768, and each candidate answer corresponds to the answer feature X aThe feature dimension of is 768. Let the length of the problem be l q , then for each word in the question, extract the corresponding question word feature X w , the size is (l q ,768). Deberta participated in the training of the video question answering model.

[0078] Step 3: The video clip features X obtained in steps 1 and 2 s , video frame feature X f 、Object feature X o 、Problem Features X q 、Question word feature X w And the candidate answer feature X a They are input into their respective single-layer linear layers, and their feature dimensions are uniformly mapped to 512 dimensions. For convenience, the symbolic representation of the linear layer output is still represented by X s ,X f ,X o ,X q ,X w and X a express.

[0079] Step 4: Video clip selection. q , Video clip features X s , video frame feature X f and object feature X o Input to the video segment selection module, and its output is the video segment features selected by the video segment selection module Video frame features and object features

[0080] (4-1) The video segment selection module has four inputs, which are the problem features X output by the linear layer q , Video clip features X s , video frame feature X f and object feature X o .

[0081] (4-2) Set the number k of video clips to be selected s .

[0082] Calculate the problem feature X q and video clip feature X s The similarity score is converted into a probability distribution logit using the softmax function s , the formula is as follows:

[0083]

[0084] Then, the Gumbel-Softmax method is used to analyze the logit s Carry out k s Sub-sampling, get the selection matrix Logit' about the video clip features s , the formula is as follows:

[0085]

[0086] (4-3) Video segment features selected by the video segment selection module Video frame features and object features By inputting X s , X f , X o With Logit' s It is obtained by multiplying the matrices of .

[0087]

[0088] After the video segment is selected, the subsequent video frame selection and object selection will be selected from the video frames contained in the selected video segment.

[0089] Step 5: Video frame selection. q , Video frame features selected by the video segment selection module and object features Input to the video frame selection module, and its output is the video frame features selected by the video frame selection module and object features

[0090]

[0091] (5-1) The video frame selection module has three inputs, namely, the problem feature X q , Video frame features selected by the video segment selection module and object features

[0092] (5-2) Set the number k of video frames to be selected f . Calculate the problem feature X q and video frame features The similarity score is converted into a probability distribution logit using the softmax function f , the formula is as follows:

[0093]

[0094] Then, the Gumbel-Softmax method is used to analyze the logit fCarry out k f Sub-sampling, get the selection matrix Logit' about the video frame features f , the formula is as follows:

[0095]

[0096] (5-3) The video frame features and object features selected by the video frame selection module are respectively input through With Logit' f Matrix multiplication of :

[0097]

[0098] Step 6: Object selection. q And the object features selected by the video frame selection module Input to the object selection module, and its output is the object features selected by the object selection module

[0099] (6-1) The object selection module has two inputs, namely, the problem feature X q and object features selected by the video frame selection module

[0100] (6-2) Set the number k of video frames to be selected o . Calculate the problem feature X q and video frame features The similarity score is converted into a probability distribution logit using the softmax function o , the formula is as follows:

[0101]

[0102] Then, the Gumbel-Softmax method is used to analyze the logit o Carry out k o Sampling, get the selection matrix Logit' about the object features o , the formula is as follows:

[0103]

[0104] Logit' obtained by fusion o Contains selected information about the features of the object.

[0105] (6-3) The object features selected by the object selection module are input With Logit' o Matrix multiplication of :

[0106]

[0107] Step 7: The video segment features obtained by the video segment selection module, the video frame selection module and the object selection module in steps 4 to 6 are respectively selected Video frame features Object characteristics Collectively referred to as multi-granular visual features, and combined with the question word feature X w Then, L layers of Transformer are introduced to perform cross-modal interaction and fusion of the concatenated question word features and multi-granular visual features, and a deep correspondence relationship from question to answer is constructed to obtain the fused feature Z out .

[0108]

[0109] Z out = TransformerLayer(Z in )

[0110] Step 8: The fusion feature Z obtained in step 7 out Implement average pooling operation to obtain global fusion features Afterwards, With candidate answer feature X a Calculate the similarity and take the candidate answer with the highest similarity as the final answer prediction formula as follows:

[0111]

[0112] Obtain an existing dataset to train the video question answering model. The modules involved in the training include the language encoder, linear mapping layer, video clip selection module, video frame selection module, object selection module, and Transformer layer. The visual encoder and object detection model in step 1 are not involved in the training.

[0113] The training objective of the model is to minimize the predicted answer The cross entropy loss between the correct answer a is:

[0114]

[0115] Example

[0116] The dataset used in this example is the public Next-QA dataset, which is a multi-choice video question answering benchmark dataset for causal and temporal reasoning. It contains a total of 5440 videos with an average length of 44 seconds and approximately 52,000 questions.

[0117] According to the steps described in the specific implementation, in the data preprocessing stage, the visual encoder used in this embodiment is the pre-trained CLIP (ViT-B / 32) model, the target detection model is Faster-RCNN, and the language model is DeBERTa-base. In the data selection stage, this embodiment selects the number of video clip features k s Set to 2, select the number k of video frame features f Set to 2, select the number of object features k o Set to 12. In the data fusion stage, this embodiment sets the number of Transformer layers to 4. In terms of model training settings, the loss function is set to the cross entropy loss function, the learning rate is set to 0.00003, the dropout probability is set to 0.3, the training optimizer is set to the Adam optimizer, the data mini-batch size is set to 64, and the iteration cycle is set to 30. In addition, during the training process, gradient clipping is applied to the model parameter optimization (the maximum norm is set to 12).

[0118] In the training phase, the Logit output of the selection module is shown in Table 1:

[0119] Table 1 Logit output of each selection module

[0120]

[0121] The accuracy change of model training is as follows Figure 2 As shown, the accuracy of the training data set reaches 91.53%, and the accuracy of the test data set can reach 58.80%. The detailed results are shown in Table 2:

[0122] Table 2 Test dataset results

[0123] Dataset ACC@C ACC@T ACC@D ACC Next-QA 57.51％ 57.09％ 66.19％ 58.80％

[0124] Among them, ACC@C, ACC@T and ACC@D mean that the model achieved a test accuracy of 57.51%, 57.09% and 66.19% on causal, temporal and descriptive questions respectively, and ACC means that the model achieved a test accuracy of 58.80% on all questions.

[0125] The above contents are further detailed descriptions of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, they can also make several substitutions or modifications to these described embodiments without departing from the concept of the present invention, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention.

[0126] Parts of the present invention that are not described in detail belong to the well-known technologies of those skilled in the art.

Claims

1. A video question answering modeling method based on a hierarchical selection framework, characterized in that: Follow the steps below to implement it: Step 1: Preprocess the video data to obtain video clip features, object features, and object characteristics; Step 2: Preprocess the text data to obtain question features, question word features, and candidate answer features; Step 3: The video clip features, video frame features, object features, question features, question word features, and candidate answer features obtained above are used as the input of the video question answering model; after being input into the video question answering model, these features are firstly passed through a linear layer respectively to uniformly map their feature dimensions to d dimensions; Step 4: Video segment selection: The video segment selection module of the video question answering model selects the question feature X output by the linear layer. q , Video clip features X s , video frame feature X f and object feature X o Processing to obtain the selected video clip features Video frame features and object features Step 5: Video frame selection; problem feature X q , Video frame features selected by the video segment selection module and object features Input into the video frame selection module to obtain the selected video frame features and object features Step 6: Object selection; feature X q And the object features selected by the video frame selection module Input into the object selection module to obtain the selected object features Step 7: Perform cross-modal interaction and fusion on the selected multi-granularity visual features and question word features; Step 8: Perform an average pooling operation on the fusion output obtained in step 7 to obtain a global fusion feature; Subsequently, the similarity between the global fusion features and the candidate answer features is calculated, and the candidate answer with the highest similarity is used as the final answer prediction.

2. A video question answering modeling method based on a hierarchical selection framework according to claim 1, characterized in that: Step 1: (1-1) For a dataset with n videos, each video contains several questions and answers; set the number of sampling frames to n f , averagely sample n videos, and convert the sampled video frames into RGB images with a width of w, a height of h, and 3 channels. The final output size is (n,n f ,3,w,h) video frame data; (1-2) Select a pre-trained visual encoder F v Perform feature extraction on each image of size (3, w, h) in the video frame data; set up The feature dimension is d v , then the video frame data after feature extraction has a size of (n,n f ,d v ) video frame features X f0 ; Set the fragment length to l s , then the number of fragments is n s =n f / l s , where n s Take an integer; at this time, the video frame features can be further divided into (n,n s ,l s ,d v ) f . (1-3) For the size (n,n s ,l s ,d v ) video frame features X f , in l s The dimension is converted into a size of (n,n s ,d v ) video clip feature X s ; (1-4) Set the number of detection targets n o , select a pre-trained target detection model F o Perform target detection on each image of size (3,w,h) in the video frame data, and then convert the detected target object area into an image of the same size as the video frame, and finally obtain a size of (n,n f ,n o ,3,w,h) object data; then, use the visual encoder F v Extract features for each object in the object data, and finally get a size of (n,n f ,n o ,d v ) of the object feature X o ; Therefore, the features obtained after data preprocessing include video frame features X f , Video clip features X s and object feature X o ; It should be noted that in order to reduce the amount of computation, the pre-trained visual encoder and target detection model used in video data preprocessing do not participate in subsequent training.

3. The video question answering modeling method based on a hierarchical selection framework according to claim 1 is characterized in that: Step 2: The text data contains questions and answers; select a pre-trained language encoder F t Perform feature extraction on each question and answer; set up The feature dimension is d t , then for each question, extract the corresponding question feature X q (The feature dimension is d t );For each answer, extract the corresponding answer feature X a (The feature dimension is d t ); Let the length of the problem be l q , then for each word in the question, extract the corresponding question word feature X w , the size is (l q ,d t ); Different from the visual part, the language encoder F t Need to participate in follow-up training.

4. The video question answering modeling method based on a hierarchical selection framework according to claim 1, characterized in that: Step 3: The obtained video clip feature X s , video frame feature X f 、Object feature X o 、Problem Features X q 、Question word feature X w And the candidate answer feature X a As the input of the video question answering model; after entering the video question answering model, these features first pass through a linear layer to uniformly map their feature dimensions to d dimensions; for convenience, the symbolic representation of the output of this linear layer is still defined as X s ,X f ,X o ,X q ,X w and X a .

5. The video question answering modeling method based on a hierarchical selection framework according to claim 4 is characterized in that: Step 4: (4-1) The video segment selection module has four inputs, which are the problem features X output by the linear layer q , Video clip features X s , video frame feature X f and object feature X o ; (4-2) Set the number k of video clips to be selected s ; Calculate the problem feature X q and video clip feature X s The similarity score is converted into a probability distribution logit using the softmax function s , the formula is as follows: Then, the Gumbel-Softmax method is used to quantify the logit s Carry out k s Sub-sampling, get the selection matrix Logit' about the video clip features s , the formula is as follows: (4-3) Video segment features selected by the video segment selection module Video frame features and object features By inputting X s , X f , X o With Logit' s The matrix multiplication of is obtained; 6. A video question answering modeling method based on a hierarchical selection framework according to claim 5, characterized in that: Step 5: (5-1) The video frame selection module has three inputs, namely, the problem feature X q , Video frame features selected by the video segment selection module and object features (5-2) Set the number k of video frames to be selected f ; Calculate the problem feature X q and video frame features The similarity score is converted into a probability distribution logit using the softmax function f , the formula is as follows: Then, the Gumbel-Softmax method is used to quantify the logit f Carry out k f Sub-sampling, get the selection matrix Logit' about the video frame features f , the formula is as follows: (5-3) The video frame features and object features selected by the video frame selection module are respectively input through With Logit' f Matrix multiplication of :

7. The video question answering modeling method based on a hierarchical selection framework according to claim 6 is characterized in that: Step 6: (6-1) The object selection module has two inputs, namely, the problem feature X q and object features selected by the video frame selection module (6-2) Set the number k of video frames to be selected o ; Calculate the problem feature X q and video frame features The similarity score is converted into a probability distribution logit using the softmax function o , the formula is as follows: Then, the Gumbel-Softmax method is used to quantify the logit o Carry out k o Sampling, get the selection matrix Logit' about the object features o , the formula is as follows: Logit' obtained by fusion o Contains selected information about the object's features; (6-3) The object features selected by the object selection module are input With Logit' o Matrix multiplication of :

8. The video question answering modeling method based on a hierarchical selection framework according to claim 7 is characterized in that: Step 7: The video segment features selected by the video segment selection module, the video frame selection module and the object selection module in steps 4 to 6 are respectively Video frame features and object features Collectively referred to as multi-granular visual features, and combined with the question word feature X w After concatenation, an L-layer Transformer layer is introduced to perform cross-modal interaction and fusion of the concatenated question word features and multi-granularity visual features, thus building a deep correspondence between questions and answers: WITH out =TransformerLayer(Z in )。 9. The video question answering modeling method based on a hierarchical selection framework according to claim 1, characterized in that: During the video question answering model training process, the modules involved in the training include the language encoder, linear mapping layer, video clip selection module, video frame selection module, object selection module, and Transformer layer; the visual encoder and target detection model in step 1 do not participate in the training; The training objective of the model is to minimize the predicted answer The cross entropy loss between the correct answer a is:

Images