A zero-shot video question answering method based on temporal information-guided hybrid experts

Abstract

The present invention provides a zero-sample video question-answering method based on temporal information-guided hybrid experts. The model of the present invention is first trained: the first feature information of the original video, the second feature information of the video text, and the temporal feature information of the first feature information are extracted, the above three feature information are used as input of a large language model, and the temporal feature information is analyzed by using the temporal module in the large language model. According to the analysis result, the three feature information are routed to the corresponding expert module for processing, the output result is input into the classification head, and a probability vector is obtained after processing, and the word corresponding to the maximum probability is selected as the output result; the training task is to predict the randomly masked words in the descriptive text related to the video, and when migrating to the video question-answering template, the corresponding answer is given according to the set question. The present invention has strong generalization and reasoning capabilities, and effectively improves the generalization performance in the video question-answering task.

Description

Technical Field

[0001] The present invention relates to the technical field of video question answering, and specifically to a zero-sample video question answering method based on temporal information-guided hybrid experts. Background Art

[0002] Currently, the methods used in most video question-answering tasks rely on a fully supervised setting, that is, they need to be trained on a well-labeled video question-answering dataset. This approach has two disadvantages: first, in order to obtain a well-labeled video question-answering dataset, a lot of manpower and material resources are required; second, when the model trained under the fully supervised setting encounters data from other domains that are different from the training set data, the results will perform very poorly and are not very generalizable.

[0003] Zero-shot video question answering can, to a certain extent, get rid of the dependence on well-labeled video question answering datasets. Due to the powerful generalization and reasoning capabilities of large-scale language models, current zero-shot video question answering mostly combines large-scale language models for reasoning; however, this method cannot well combine the language model with the ability to perceive video information, and at the same time cannot effectively improve the generalization of video question answering tasks. Summary of the Invention

[0004] In response to the above-mentioned deficiencies in the prior art, the present invention provides a zero-shot video question-answering method based on temporal information-guided hybrid experts. A temporal module is introduced to extract temporal information from videos, thereby improving the perception of temporal information in videos by large-scale language models. At the same time, a hybrid expert module guided by video temporal information is proposed to improve the generalization ability of the model in zero-shot video question-answering tasks.

[0005] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is: a zero-sample video question answering method based on temporal information-guided hybrid experts, comprising the following steps:

[0006] S1: Extract the first feature information of the original video and the second feature information of the video text, randomly select some words in the second feature information for masking, and obtain the third feature information;

[0007] S2: Extracting time series feature information of the first feature information using a time series module;

[0008] S3: using the first feature information, the third feature information, and the temporal feature information as input to the large language model;

[0009] S4: Analyze the temporal feature information using the linear router in the large language model, and route the temporal feature information, the first feature information, and the third feature information to the corresponding expert module for processing based on the analysis results to obtain features corresponding to the masked word;

[0010] S5: The features corresponding to each masked word are input into the classification head. After processing, a probability vector with a dimension size equal to the vocabulary size is obtained. The word with the highest probability is selected as the output result. After multiple predictions, a pre-trained video question answering dataset is obtained.

[0011] S6: Transfer the model from the pre-trained video question answering dataset to the video question answering task to reason about and answer video questions.

[0012] The beneficial effects of the technical solution of the present invention are as follows: the present invention is based on a large language model, and introduces a timing module and a hybrid expert module, which can improve the large language model's perception of timing information in videos and the model's generalization ability in zero-sample video question-answering tasks. At the same time, during training, masked language modeling is used as a training task for pre-training, so that the model is trained on a video text dataset to obtain a model with strong understanding and generalization capabilities of video language.

[0013] Furthermore, the specific steps of S1 are as follows:

[0014] S11: using a visual encoder to extract image features of a video frame in the original video to obtain first feature information;

[0015] S12: Segment the video text description into words to obtain multiple tokens;

[0016] S13: Use the word embedding converter in the large language model to convert each word into second feature information with semantic information;

[0017] S14: Randomly select words from the second feature information and perform masking processing to obtain third feature information.

[0018] The above further beneficial effect is: converting video and text into feature vectors with semantic information, so that the model can understand the data information of the two modalities, which is helpful for model training and migration.

[0019] Furthermore: the specific steps of S2 are:

[0020] S21: Using the first feature information as input to a time series module;

[0021] S22: Performing a convolution operation on the first feature information along the time dimension using a one-dimensional convolutional network to obtain convolution features of the first feature information;

[0022] S23: Add explicit temporal position embedding to the convolutional features of the first feature information, and input it into the multi-head attention module together with a learnable CLS vector to obtain temporal feature information.

[0023] The beneficial effect of the above further solution is: the first feature information is processed by the time sequence module to obtain time sequence feature information, which serves as the basis for the subsequent linear router to select the expert module.

[0024] Furthermore, the expression of the convolution operation is as follows:

[0025]

[0026] in, is the feature of the t-th frame after the convolution operation, Δt is a time step, t is the current t-th frame, f t+Δt is the first feature information of the t+Δt frame, W conv is the parameter of the one-dimensional convolution kernel, b conv is the offset of the one-dimensional convolution kernel.

[0027] The beneficial effect of the above further solution is that the timing module uses convolution operation to enable each frame feature in the video to capture the information of the two adjacent frames, thereby extracting local information of the video.

[0028] Furthermore, the calculation expression of the time series feature information is as follows:

[0029]

[0030] f p =f conv +P temp

[0031] Among them, f temp is the time series feature, W q , W K and W V are all learnable mapping parameters, q is a learnable CLS vector, and f p To add the convolutional features of temporal position embedding, f conv is the convolution feature of the first feature information, P temp For temporal position embedding, the softmax(.) function converts the input into a probability vector.

[0032] The beneficial effect of the above further scheme is: by inputting the CLS vector and the convolution feature into the multi-head attention, the CLS vector interacts with the convolved features of each frame, and finally obtains the global features of the video, which is convenient for subsequent analysis and processing.

[0033] Further: the large language model includes a word transformation embedder, a classification head, and a Transformer layer with a mixture of experts module;

[0034] The word conversion embedder is used to segment the video text description into multiple word units, and convert each word unit into second feature information with semantic information;

[0035] The classification head is used to finally map the output features corresponding to the masked words into probability vectors corresponding to the vocabulary;

[0036] The Transformer layer with the hybrid expert module is used to route the first feature information, the third feature information and the temporal feature information to different expert modules for processing according to the extracted video temporal features, so as to obtain the corresponding features of the masked words.

[0037] The beneficial effects of the above further solution are: based on the large language model, a Transformer layer with a hybrid expert module is used and the analysis of temporal features is introduced to improve the large language model's ability to perceive temporal information in videos and the model's generalization ability in zero-sample video question-answering tasks.

[0038] Furthermore: the Transformer layer with the hybrid expert module includes at least one expert module and a linear router; the linear router is used to route the first feature information, the third feature information and the timing feature information to different expert modules for processing according to the extracted video timing features; each expert module is used to analyze the first feature information, the third feature information and the timing feature information to obtain the corresponding feature information of the masked word.

[0039] The above further beneficial effects are: using linear routers to process temporal feature information, introducing video temporal perception capabilities for large language models, and using hybrid expert modules to adopt different expert modules based on the analysis results of the linear routers, thereby improving the generalization ability of the model at the temporal level.

[0040] Furthermore: the expression of the linear router is as follows:

[0041] s gate =softmax(G(f temp ))

[0042] G(f temp )=W g f temp +δ·softplus(W noise f temp )

[0043] Among them, s gate is the probability vector for selecting each expert module. The probability vector is 4-dimensional, corresponding to 4 expert modules. temp is the time series feature, W g and Wnoise are all learnable parameters in the linear layer, δ is Gaussian noise, G(.) function is the function of the linear router, and softmax(.) is the softmax function.

[0044] The beneficial effect of the above further scheme is: the probability vector of each expert module is selected according to the temporal feature calculation, and the first feature information is input into the selected expert module according to the probability vector, so that videos with different temporal characteristics can be input into the corresponding specific expert module for processing.

[0045] Furthermore, the objective optimization function of the classification head is expressed as follows:

[0046]

[0047] in, is the value of the cross entropy loss function, x represents the masked word, p(x|S \m(s) ) represents the probability vector output by the classification head, S \m(s) Represents all words in the input sentence except the masked words.

[0048] A further beneficial effect of the above is that the formula is used to convert the high-dimensional feature vector into the dimension of the vocabulary size, which facilitates the selection of the most appropriate answer as the model output based on the probability value of the probability vector.

[0049] The beneficial effects of the present invention are as follows: the present invention does not need to be trained on a video question-answering dataset, but is trained on other datasets, and then directly performs inference on the video question-answering task without the need for sample training; based on the drawbacks of current video question-answering work that is overly dependent on well-labeled datasets, and the problem that current zero-sample video question-answering methods fail to well combine video timing information and fail to study how to improve the generalization ability of the model, the present invention proposes a module that can extract video timing information, and introduces video timing perception capabilities for large language models; at the same time, the proposed hybrid expert module improves the generalization ability of the model at the timing level; the present invention achieves optimal performance on multiple traditional multimodal tasks, and has strong understanding and generalization capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 Flowchart of the zero-shot video question answering method based on temporal information to guide hybrid experts;

[0051] Figure 2 This is a framework diagram of the zero-shot video question answering method based on temporal information to guide hybrid experts;

[0052] Figure 3 It is the structural diagram of the timing module;

[0053] Figure 4This is the structural diagram of the hybrid expert module. DETAILED DESCRIPTION

[0054] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0055] like Figure 1 As shown in the figure, a zero-shot video question answering method based on temporal information-guided hybrid experts is proposed. The pre-trained model is transferred to the template of the video question answering task for video question answering. The training method includes the following steps:

[0056] S1: Extract the first feature information of the original video and the second feature information of the video text, randomly select some words in the second feature information for masking, and obtain the third feature information;

[0057] S2: Extracting time series feature information of the first feature information using a time series module;

[0058] S3: using the first feature information, the third feature information, and the temporal feature information as input to the large language model;

[0059] S4: Analyze the temporal feature information using the linear router in the large language model, and route the temporal feature information, the first feature information, and the third feature information to the corresponding expert module for processing based on the analysis results to obtain features corresponding to the masked word;

[0060] S5: The features corresponding to each masked word are input into the classification head. After processing, a probability vector with a dimension size equal to the vocabulary size is obtained. The word with the highest probability is selected as the output result. After multiple predictions, a pre-trained video question answering dataset is obtained.

[0061] S6: Transfer the model from the pre-trained video question answering dataset to the video question answering task to reason about and answer video questions.

[0062] In this invention, the training tasks of S1-S5 are to predict randomly masked words in the description text related to the video. S6 is to migrate the model to video question answering for question reasoning. The model is based on the template pre-set in the video question answering, namely prompt: [CLS] Question: <question>? Answer:[MASK].[SEP]

[0063] To output the answer, this is a template for open video question answering tasks. For an input sample (video, question), the model replaces the question in the prompt <question>part, and then output the word corresponding to [MASK] as the answer.

[0064] The specific steps of S1 are as follows:

[0065] S11: using a visual encoder to extract image features of a video frame in the original video to obtain first feature information;

[0066] S12: Segment the video text description into words to obtain multiple tokens;

[0067] S13: Use the word embedding converter in the large language model to convert each word into second feature information with semantic information;

[0068] S14: Randomly select words from the second feature information and perform masking processing to obtain third feature information.

[0069] The specific steps of S2 are:

[0070] S21: Using the first feature information as input to a time series module;

[0071] S22: Performing a convolution operation on the first feature information along the time dimension using a one-dimensional convolutional network to obtain convolution features of the first feature information;

[0072] S23: Add explicit temporal position embedding to the convolutional features of the first feature information, and input it into the multi-head attention module together with a learnable CLS vector to obtain temporal feature information.

[0073] The large language model in the present invention includes a word conversion embedder, a linear router, a hybrid expert block and a classification head. In addition, the present invention also includes a visual encoder and a timing module; the word conversion embedder is used to convert words into word embeddings with semantic information; the linear router is used to route input data to the corresponding expert module for processing according to the extracted video timing features; the hybrid expert block is used to use different expert modules for processing according to videos with different timing characteristics; the classification head is used to ultimately map the output features corresponding to the masked words into probability vectors of the corresponding vocabulary; the visual encoder is used to extract video frames from the original video; and the timing module is used to extract timing information from the video frames.

[0074] The expression of the module operation in the present invention is as follows:

[0075] The expression of the convolution operation is as follows:

[0076]

[0077] in, is the feature of the t-th frame after the convolution operation, Δt is a time step, t is the current t-th frame, f t+Δt is the first feature information of the t+Δt-th frame, W conv is the parameter of the one-dimensional convolution kernel, b conv is the offset of the one-dimensional convolution kernel.

[0078] The calculation expression of time series feature information is as follows:

[0079]

[0080] f p =f conv +P temp

[0081] Among them, f temp is the time series feature, W q , W K and W V are all learnable mapping parameters, q is a learnable CLS vector, and f p To add the convolutional features of temporal position embedding, f conv is the convolution feature of the first feature information, P temp For temporal position embedding, the softmax(.) function converts the input into a probability vector.

[0082] The expression of the linear router is as follows:

[0083] s gate =softmax(G(f temp ))

[0084] G(f temp )=W g f temp +δ·softplus(W nopse f temp )

[0085] Among them, s gate is the probability vector for selecting each expert module. The probability vector is 4-dimensional, corresponding to 4 expert modules. temp is the time series feature, W g and W noise are all learnable parameters in the linear layer, δ is Gaussian noise, G(.) function is the function of the linear router, and softmax(.) is the softmax function.

[0086] The calculation formula / expression of the probability vector is as follows:

[0087] p(x|S \m(s) )=W h f x +b h

[0088] Among them, p(x|S \m(s) ) is the probability vector output by the classification head, x is the masked word, S \m(S) W is all the words in the input sentence except the masked words. h is the parameter of the classification head, f x is the corresponding output feature of the masked word, b h is the offset of the classification header.

[0089] One embodiment of the present invention: Figure 2 As shown, the input includes video frames, a description of the video text as Amuzzle of a horse close up at sunset in slow motion, and a CLS vector; the video frame is processed by the visual encoder and input into the large language model in a linear mapping manner, which is the first feature information; at the same time, it is input into the timing module together with the CLS vector, and after analysis and processing, the timing feature information is obtained, and finally enters the large language model; the description of the video text is input into the word embedder of the large language model, and some words are randomly selected for masking and replaced with [MASK] to obtain the following description: Amuzzle of a [MASK] close up at sunset in slow motion, which is the third feature information; the linear router in the large language model analyzes the timing features and routes the input information to different expert modules for processing according to the analysis results. Finally, the large language model outputs the feature corresponding to [MASK], and then inputs the feature into the classification head to obtain a probability vector, and the word with the highest probability, horse, is obtained through the probability vector.

[0090] After multiple predictions of masked words, a trained video question-answering dataset is obtained; the model is migrated to the video question-answering task, and video questions are inferred and answered based on the trained video question-answering dataset.

[0091] like Figure 3 As shown, the timing module of the present invention includes timing convolution, timing position embedding, multi-head attention, layer normalization and feedforward layer; wherein the timing convolution enables each frame in the video to perceive its two adjacent frames before and after, so that each frame feature of the video has a local receptive field, containing local information of the video; timing position embedding can explicitly indicate the timing relationship between frames; multi-head attention enables [CLS] to collect global information of the video by interacting [CLS] token with each frame of the video; layer normalization standardizes the data in the network to make the data distribution more stable, which is beneficial to the stability and generalization ability of network training; the feedforward layer performs feature space transformation to provide more nonlinearity and learning ability.

[0092] like Figure 4 As shown in the figure, the hybrid expert module includes a linear router and four expert modules. The structure of the expert module is a modulator, including a down-mapping linear layer, an activation function ReLU and an up-mapping linear layer. The linear router can select the corresponding expert module to process the current input data according to the extracted video timing features; the down-mapping linear layer can map the current data dimension to another lower dimension; the activation function ReLU can increase the network sparsity and alleviate overfitting; the up-mapping linear layer can restore the down-mapped data to the original dimension.

[0093] The beneficial effects of the present invention are as follows: the present invention does not need to be trained on a video question-answering dataset, but is trained on other datasets, and then directly performs inference on the video question-answering task without the need for sample training; based on the drawbacks of current video question-answering work that is overly dependent on well-labeled datasets, and the problem that current zero-sample video question-answering methods fail to well combine video timing information and fail to study how to improve the generalization ability of the model, the present invention proposes a module that can extract video timing information, and introduces video timing perception capabilities for large language models; at the same time, the proposed hybrid expert module improves the generalization ability of the model at the timing level; the present invention achieves optimal performance on multiple traditional multimodal tasks, and has strong understanding and generalization capabilities.

[0094] Example 1

[0095] The effects of the present invention will be described below with reference to examples:

[0096] First, explain the following parameter nouns:

[0097] Table 1 Description of relevant terms in the present invention

[0098]

[0099]

[0100] The experimental conditions are set as follows: system: Ubuntu 18.04, programming language: Python 3.8, deep learning framework: Pytorch 11.6, GPU: NVIDIA GeForce RTX A6000.

[0101] Input a video, sample it evenly by 10 frames, and fill the video with 0 to make it 10 frames long if it is less than 10 frames long. For the input text, limit its length to less than 256 words, and then perform token parsing.

[0102] The following is a comparative analysis of this method with other methods:

[0103] For the open-ended video question answering task, the datasets selected were MSVD-QA and MSRVTT-QA. The two currently best-performing methods, FrozenBilm and Flamingo-3B, were selected. As can be seen from Table 2, our method achieved an accuracy 2.2 percentage points higher than the current best method, FrozenBilm, on the MSVD-QA dataset, and 2.3 percentage points higher on the MSRVTT-QA dataset.

[0104] Table 2 Performance of T-MoENet and other methods on open-ended video question answering

[0105] method MSVD-QA MSRVTT-QA Flamingo-3B 27.5 11.0 FrozenBilm 33.8 16.7 T-MoENet 36.0 19.0

[0106] For the multiple-choice video question answering task, we used two multiple-choice video question answering datasets, NExT-QA and STAR, and selected the two best-performing methods, FrozenBilm and Flamingo-3B. The results are shown in Table 3. We can see that T-MoENet outperformed the previous best model by 4.3 percentage points on NExT-QA, and also demonstrated competitive performance with the best model on another dataset, STAR.

[0107] Table 3 Performance of T-MoENet and other methods on multi-classification video question answering

[0108] method NExT-QA STAR Flamingo-3B 21.3 39.6 FrozenBilm 29.6 28.1 T-MoENet 33.9 33.1

[0109] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.< / question> < / question>

Claims

1. A zero-shot video question answering method based on temporal information-guided hybrid experts, characterized by: The following steps are involved: S1: Extract the first feature information of the original video and the second feature information of the video text, randomly select some words in the second feature information for masking, and obtain the third feature information; S2: Extracting time series feature information of the first feature information using a time series module; S3: using the first feature information, the third feature information, and the temporal feature information as input to the large language model; S4: Analyze the temporal feature information using the linear router in the large language model, and route the temporal feature information, the first feature information, and the third feature information to the corresponding expert module for processing based on the analysis results to obtain features corresponding to the masked word; S5: The features corresponding to each masked word are input into the classification head. After processing, a probability vector with a dimension size equal to the vocabulary size is obtained. The word with the highest probability is selected as the output result. After multiple predictions, a pre-trained video question answering dataset is obtained. S6: Transfer the model from the pre-trained video question answering dataset to the video question answering task to reason about and answer video questions.

2. According to the zero-shot video question answering method based on temporal information-guided hybrid experts according to claim 1, the specific steps of S1 are as follows: S11: using a visual encoder to extract image features of a video frame in the original video to obtain first feature information; S12: Segment the video text description into words to obtain multiple tokens; S13: Use the word embedding converter in the large language model to convert each word into second feature information with semantic information; S14: Randomly select words from the second feature information and perform masking processing to obtain third feature information.

3. According to the zero-shot video question answering method based on temporal information-guided hybrid experts according to claim 1, the specific steps of S2 are: S21: Using the first feature information as input to a time series module; S22: Performing a convolution operation on the first feature information along the time dimension using a one-dimensional convolutional network to obtain convolution features of the first feature information; S23: Add explicit temporal position embedding to the convolutional features of the first feature information, and input it into the multi-head attention module together with a learnable CLS vector to obtain temporal feature information.

4. In the zero-shot video question answering method based on temporal information-guided hybrid experts according to claim 3, the expression of the convolution operation is as follows: in, is the feature of the t-th frame after the convolution operation, is a time step, t is the current t-th frame, For the The first feature information of the frame, is the parameter of the one-dimensional convolution kernel, is the offset of the one-dimensional convolution kernel.

5. According to the zero-shot video question answering method based on temporal information-guided hybrid experts according to claim 3, the calculation expression of the temporal feature information is as follows: in, is the time series feature, , and are all learnable mapping parameters, is the learnable CLS vector, To add the convolutional features of temporal position embedding, is the convolution feature of the first feature information, is the temporal position embedding, is the softmax function, which converts the input into a probability vector.

6. The zero-shot video question answering method based on temporal information-guided hybrid experts according to claim 1, wherein the large language model includes a word transformation embedder, a classification head, and a Transformer layer with a hybrid expert module; The word conversion embedder is used to segment the video text description into multiple word units, and convert each word unit into second feature information with semantic information; The classification head is used to finally map the output features corresponding to the masked words into probability vectors corresponding to the vocabulary; The Transformer layer with the hybrid expert module is used to route the first feature information, the third feature information and the temporal feature information to different expert modules for processing according to the extracted video temporal features, so as to obtain the corresponding features of the masked words.

7. The zero-shot video question answering method based on temporal information-guided hybrid experts according to claim 6, wherein the Transformer layer with the hybrid expert module includes at least one expert module and a linear router; The linear router is used to route the first feature information, the third feature information and the time sequence feature information to different expert modules for processing according to the extracted video time sequence features; Each expert module is used to analyze the first feature information, the third feature information and the temporal feature information of the video temporal feature to obtain corresponding feature information of the masked word.

8. According to the zero-shot video question answering method based on temporal information-guided hybrid experts according to claim 7, the linear router is expressed as follows: in, The probability vector for selecting each expert module is 4-dimensional, corresponding to 4 expert modules. is the time series feature, and are all learnable parameters in the linear layer, is Gaussian noise, Function is the function of linear router, is the softmax function, which converts the input into a probability vector.

9. According to the zero-shot video question answering method based on temporal information-guided hybrid experts according to claim 1, the calculation formula / expression of the probability vector is as follows: in, is the probability vector output by the classification head, x is the masked word, For all words in the input sentence except the masked words, is the parameter of the classification head, is the corresponding output feature of the masked word, is the offset of the classification header.

10. According to the zero-shot video question answering method based on temporal information-guided hybrid experts according to claim 1, the objective optimization function expression of the classification head is as follows: in, is the value of the cross entropy loss function, x represents the masked word, represents the probability vector output by the classification head, Represents all words in the input sentence except the masked words.

Images