A video question answering method based on object-oriented two-stream attention network

Abstract

The present invention discloses a video question-answering method based on an object-oriented dual-stream attention network. A dual-stream mechanism is used to represent the visual content of a video, wherein one stream is a static appearance stream of a foreground object, and the other stream is a dynamic behavior stream of a foreground object. In each stream, the features of the object include not only the features of the object itself, but also the spatiotemporal encoding of the object and the contextual information features of the scene in which the object is located. When performing deep feature extraction in subsequent graph convolution operations, the relative spatiotemporal relationships and context-aware relationships between objects can be explored. At the same time, the use of a dual-stream mechanism solves the problem that previous video question-answering models only consider static features of objects but lack analysis of dynamic information. The present invention improves the ability to explore intra-modal interactions and inter-modal semantic alignment, and has achieved good results on relevant video question-answering datasets.

Description

Technical Field

[0001] The present invention relates to the field of video question answering, and in particular to a video question answering method based on an object-oriented dual-stream attention network including a static appearance stream and a dynamic behavior stream. Background Art

[0002] Video question answering is an interdisciplinary task between computer vision and natural language processing. It requires the system to output the correct answer based on a given video and a question described in natural language. This task requires the system to fully understand the information of the two modalities, namely the appearance characteristics, behavioral characteristics of the foreground object in the video and the content of the question, and also to have the ability to explore the semantic interaction between the two modalities. Most existing video question answering tasks extract visual features in frames. This feature extraction method does not learn the subject of the question and answer, namely the foreground object in the video, in detail. At the same time, this feature extraction method also ignores the modeling of the interaction between objects. Therefore, it is crucial to design a model that can explore the features of the target objects in the video in detail and the interaction between them.

[0003] Video question answering and image question answering belong to the same branch of visual question answering. Both of them output the correct answer based on a given piece of visual information and a question. Although video question answering is an extended task of image question answering, there are many differences between them. Therefore, the image question answering method cannot be simply applied to the video question answering task. The main difference between video question answering and image question answering is that the video question answering task processes long sequence images instead of single static images. These sequence images contain richer appearance and action information. Therefore, in the video question answering task, the system needs to model the appearance information and action information of the objects in the video at the same time to improve the accuracy of the model's answering questions.

[0004] Attention network is a commonly used object relationship modeling mechanism, which is widely used in various types of machine learning tasks such as natural language processing, image recognition and speech recognition. The attention mechanism can help the model assign different weights to each part of the input, extract more critical and important information, and enable the model to make more accurate judgments without incurring more overhead in the calculation and storage of the model. The present invention proposes two variants of the attention mechanism to achieve modeling of intra-modal interaction relationships and inter-modal interaction relationships. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a video question answering method based on an object-oriented dual-stream attention network, which can learn the static appearance features and dynamic behavior features of the foreground objects in the video in parallel, accurately model the interaction relationship between the objects in the video, and realize cross-modal semantic alignment. Good results have been achieved on relevant video question answering datasets.

[0006] A video question answering method based on an object-oriented two-stream attention network, the steps are as follows:

[0007] Step (1), data preprocessing:

[0008] Input the video clip to be answered, first use the convolutional network to extract the static appearance features and dynamic behavior features of the video frame, then use the target detection algorithm to extract the target object in the video, and use the convolutional network to extract the static appearance features and dynamic behavior features of the object at the same time; use the extracted static appearance features and dynamic behavior features of the video frame as the context representation features of the target object;

[0009] Step (2), video feature encoding:

[0010] A dual-stream mechanism is used to encode video features, where the two streams are the static appearance stream and the dynamic behavior stream of the target object.

[0011] In each stream, the features of the target object, the spatiotemporal encoding of the target object, and the contextual representation features of the target object are combined as nodes to construct a feature graph; a graph convolution operation is performed on the constructed feature graph to obtain high-order features of the target object in the video;

[0012] Step (3), problem feature coding:

[0013] Use a recurrent neural network to encode the question to obtain the local encoding of each word and the global encoding of the question sentence;

[0014] Step (4), cross-modal fusion of video and question:

[0015] In each stream, the high-order features of the target object are first input into the self-attention network to obtain the intra-modal interaction relationship between objects, and the high-order features of the target object in the video are reconstructed according to the intra-modal interaction relationship between objects. Similarly, the local features of each word are input into the self-attention network to obtain the intra-modal interaction relationship between words, and the question features are reconstructed according to the intra-modal interaction relationship between words. Then, the reconstructed high-order features of the target object and the reconstructed question features are input into the question-guided attention network to explore the cross-modal semantic connection between vocabulary and object features, and update the high-order features of the video.

[0016] The high-order features of the static appearance stream and the dynamic behavior stream of the target object are fused to obtain the dual-stream high-order features of the video.

[0017] The dual-stream high-order features of the video are combined with the global encoding of the question to obtain the answer encoding;

[0018] Step (5): Decode the answer code obtained in step (4) to obtain the final answer:

[0019] Different decoding methods are used for different question types. Linear classifiers are used to decode open-ended questions and multiple-choice questions, and linear regressors are used to decode counting questions.

[0020] Furthermore, the specific method of step (1) is as follows:

[0021] Input the video clip to be answered, divide the video clip into a fixed number of video frames, and perform corresponding static appearance feature extraction and dynamic behavior feature extraction on each frame to obtain the static appearance feature of the video frame. and dynamic behavior characteristics As the context representation feature of the target object. Then, the target detection algorithm is used in each frame to locate the position information b of the target object, and then the RoIAlign alignment network is used to align the position information to the static appearance features and dynamic behavior features of the video frame to obtain the static appearance features O of each object. a and dynamic behavior characteristics O m :

[0022] The object position information b is specifically formulated as follows:

[0023] b=[c1,c2,w,h] (Formula 1)

[0024] Among them, c1 and c2 are the center position coordinates of the target object, and w and h represent the length and width of the target object respectively.

[0025] Furthermore, the specific method of step (2) is as follows:

[0026] A dual-stream mechanism is used to represent the feature encoding of the video. The static appearance features and the dynamic behavior features of the object are encoded in parallel using the same encoding mechanism. The network that encodes the static appearance features is called the static appearance stream, and the network that encodes the dynamic behavior features is called the dynamic behavior stream. Finally, the high-order features of the static appearance stream and the high-order features of the dynamic behavior stream of the video are obtained respectively. The specific process is as follows:

[0027] The static appearance stream and the dynamic behavior stream use the same network structure, so in order to simplify the expression, the superscript a / m is used to represent the characteristics of both streams.

[0028] The characteristics of the target object in a given video and the location information collection of the target object First, the location information b of the target object is passed into the spatial position encoding network and the temporal position encoding network to obtain the spatial position encoding d of the object. s and time position code d t , where the spatial position encoding network is composed of a multi-layer perceptron, and the temporal position encoding network uses sine and cosine functions of different frequencies to encode the temporal position of the target object. The specific formula is as follows:

[0029] d s =MLP(b)(Formula 2)

[0030]

[0031] Where b represents the location information of the target object obtained by formula 1; n represents the number of target objects; d p Represents the dimension of the encoding, and 0≤2j≤d p .

[0032] Afterwards, the features of the target object and the spatiotemporal position encoding of the target object are combined to obtain the location-aware features of the target object. The specific formula is as follows:

[0033]

[0034] The location-aware features of the target object are combined with the features of the video frame to obtain the context-aware features. The specific formula is as follows:

[0035]

[0036] After obtaining the features of each object with location information and context information, an undirected complete graph is established with the features of each object as nodes and the interaction relationships between objects as edges. in represents the node set, and ε represents the edge set. The calculation formula of the interaction relationship matrix between objects is:

[0037] A a / m =softmax(φ(F a / m )φ(F a / m ) T ) (Formula 7)

[0038] Where φ(F a / m )=F a / m W, W is the learnable weight matrix.

[0039] Then, two graph convolution operations are performed on the constructed undirected complete graph to obtain the high-order features of the object. The graph convolution formula is:

[0040] GCN(F a / m ; A a / m )=A a / m (F a / m W) (Formula 8)

[0041] Furthermore, the feature encoding of the problem using a recurrent neural network described in step (3) is performed as follows:

[0042] First, each word in the question is mapped to a 300-dimensional word vector space initialized by GloVe to obtain the encoding of each word. Then, each encoding vector is passed into a bidirectional LSTM network to obtain the local encoding F of each word. Q ∈R L×d and the global encoding F of the question sentence q ∈R d .

[0043] Furthermore, the cross-modal fusion steps described in step (4) are as follows:

[0044] Similar to step (2), the two flows use the same network structure, so the superscripts a / m are used to represent the characteristics of the two flows.

[0045] Define a self-attention network to explore the interaction relationship within the modality. The self-attention network formula is as follows:

[0046] f=MultiHead(X,X,X)=[head1,head2,...,head h ]W O (Formula 9)

[0047]

[0048] in, d is the feature dimension; h means that the multi-head attention mechanism has h heads; X represents the visual feature: W O , are the learnable parameter matrices respectively.

[0049] A question-guided attention network is defined to explore the interaction between modalities. The formula of the question-guided attention network is as follows:

[0050] f=MultiHead(X, Y, Y)=[head1, head2,..., head h ]W O

[0051] (Formula 11)

[0052]

[0053] Among them, X represents the visual features and γ represents the question features.

[0054] Then, the high-order feature F obtained in step (2) is a / m The local feature F of the text obtained in step (3) is passed into the defined self-attention network to obtain the intra-modal interaction relationship between objects. Q ∈R L×d The outputs of the two self-attention networks are passed into the defined self-attention network to obtain the intra-modal interaction relationship between words, and then the outputs of the two self-attention networks are passed into the defined question-guided attention network to obtain the cross-modal semantic connection between words and objects. At the same time, the high-order features of each stream of the video are updated separately under the guidance of the question.

[0055] Finally, the global feature encoding of the question is fused with the dual-stream features of the video to obtain the encoding of the answer.

[0056] Furthermore, the feature decoding steps described in step (5) are as follows:

[0057] For open-ended questions, a linear classifier is used to predict the probability that each answer in the answer set is the correct answer, and the cross entropy function is used as the loss function for parameter training:

[0058]

[0059] in, is the probability that each answer is the correct answer, yi=1 means a i ∈A is the correct answer.

[0060] For counting problems, a linear regressor is used to predict the answer, and the mean square error is used as the loss function to train the parameters:

[0061]

[0062] For multiple-choice questions, a linear classifier is used to predict the probability that each candidate answer is the correct answer, and the hinge loss function is used as the lpss function to train the model:

[0063]

[0064] Among them, s p and n Represents the scores for correct answers and incorrect answers respectively.

[0065] Beneficial effects of the present invention:

[0066] Based on the work of predecessors, the present invention uses a dual-stream mechanism to represent the visual content of the video. One stream is the static appearance stream of the foreground object, and the other stream is the dynamic behavior stream of the foreground object. In each stream, the features of the object include not only the features of the object itself, but also the spatiotemporal encoding of the object and the contextual information features of the scene in which the object is located. In this way, when the subsequent graph convolution operation performs deep feature extraction, the relative spatiotemporal relationship and context-aware relationship between objects can be explored. At the same time, the use of a dual-stream mechanism solves the problem that the previous video question-answering model only considers the static features of the object and lacks analysis of dynamic information.

[0067] Aiming at the problem of cross-modal semantic alignment and intra-modal interaction relationship exploration, this paper proposes a question-guided attention network model, which improves the exploration ability of intra-modal interaction and inter-modal semantic alignment to a certain extent. This paper has achieved good results on relevant video question-answering datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a flow chart of a method according to an embodiment of the present invention.

[0069] Figure 2 It is a schematic diagram of a network framework of an embodiment of the present invention. DETAILED DESCRIPTION

[0070] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0071] The present invention provides a video question answering method based on an object-oriented dual-stream attention network, the steps of which are as follows:

[0072] Step (1), data preprocessing:

[0073] Input the video clip to be answered, divide the video clip into a fixed number of video frames, and perform corresponding static appearance feature extraction and dynamic behavior feature extraction on each frame to obtain the static appearance feature of the video frame. and dynamic behavior characteristics As the context representation feature of the target object. Then, the target detection algorithm is used in each frame to locate the position information b of the target object, and then the RoIAlign alignment network is used to align the position information to the static appearance features and dynamic behavior features of the video frame to obtain the static appearance features O of each object. a and dynamic behavior characteristics O m :

[0074] The object position information b is specifically formulated as follows:

[0075] b=[c1,c2,w,h] (Formula 1)

[0076] Among them, c1 and c2 are the center position coordinates of the target object, and w and h represent the length and width of the target object respectively.

[0077] Step (2), video feature encoding:

[0078] A dual-stream mechanism is used to represent the feature encoding of the video. The static appearance features and the dynamic behavior features of the object are encoded in parallel using the same encoding mechanism. The network that encodes the static appearance features is called the static appearance stream, and the network that encodes the dynamic behavior features is called the dynamic behavior stream. Finally, the high-order features of the static appearance stream and the high-order features of the dynamic behavior stream of the video are obtained respectively. The specific process is as follows:

[0079] The static appearance stream and the dynamic behavior stream use the same network structure, so in order to simplify the expression, the superscript a / m is used to represent the characteristics of both streams.

[0080] The characteristics of the target object in a given video and the location information collection of the target object First, the location information b of the target object is passed into the spatial position encoding network and the temporal position encoding network to obtain the spatial position encoding d of the object. s and time position code d t , where the spatial position encoding network is composed of a multi-layer perceptron, and the temporal position encoding network uses sine and cosine functions of different frequencies to encode the temporal position of the target object. The specific formula is as follows:

[0081] d s =MLP(b) (Formula 2)

[0082]

[0083] Where b represents the location information of the target object obtained by formula 1; n represents the number of target objects; d p Represents the dimension of the encoding, and 0≤2j≤d p .

[0084] Afterwards, the features of the target object and the spatiotemporal position encoding of the target object are combined to obtain the location-aware features of the target object. The specific formula is as follows:

[0085]

[0086] The location-aware features of the target object are combined with the features of the video frame to obtain the context-aware features. The specific formula is as follows:

[0087]

[0088] After obtaining the features of each object with location information and context information, an undirected complete graph is established with the features of each object as nodes and the interaction relationships between objects as edges. in represents the node set, and ε represents the edge set. The calculation formula of the interaction relationship matrix between objects is:

[0089] A a / m =softmax(φ(F a / m )φ(F a / m ) T ) (Formula 7)

[0090] Where φ(F a / m )=F a / m W, W is the learnable weight matrix.

[0091] Then, two graph convolution operations are performed on the constructed undirected complete graph to obtain the high-order features of the object. The graph convolution formula is:

[0092] GCN(F a / m ; A a / m )=A a / m (F a / m W) (Formula 8)

[0093] Step (3) uses a recurrent neural network to encode the features of the problem. The specific process is as follows:

[0094] First, each word in the question is mapped to a 300-dimensional word vector space initialized by GloVe to obtain the encoding of each word. Then, each encoding vector is passed into a bidirectional LSTM network to obtain the local encoding F of each word. Q ∈R L×d and the global encoding F of the question sentence q ∈R d .

[0095] Step (4), cross-modal fusion of video and question:

[0096] Similar to step (2), the two flows use the same network structure, so the superscripts a / m are used to represent the characteristics of the two flows.

[0097] Define a self-attention network to explore the interaction relationship within the modality. The self-attention network formula is as follows:

[0098] f=MultiHead(X,X,X)=[head1,head2,...,head h ]W O (Formula 9)

[0099]

[0100] in, d is the feature dimension; h means that the multi-head attention mechanism has h heads; X represents the visual feature: W O , are the learnable parameter matrices respectively.

[0101] A question-guided attention network is defined to explore the interaction between modalities. The formula of the question-guided attention network is as follows:

[0102] f=MultiHead(X, Y, Y)=[head1, head2,..., head h ]W O (Formula 11)

[0103]

[0104] Among them, X represents the visual features and γ represents the question features.

[0105] Then, the high-order feature F obtained in step (2) is a / m The local feature F of the text obtained in step (3) is passed into the defined self-attention network to obtain the intra-modal interaction relationship between objects. Q ∈R L×d The outputs of the two self-attention networks are passed into the defined self-attention network to obtain the intra-modal interaction relationship between words, and then the outputs of the two self-attention networks are passed into the defined question-guided attention network to obtain the cross-modal semantic connection between words and objects. At the same time, the high-order features of each stream of the video are updated separately under the guidance of the question.

[0106] Finally, the global feature encoding of the question is fused with the dual-stream features of the video to obtain the encoding of the answer.

[0107] Step (5): Decode the answer code obtained in step (4) to obtain the final answer:

[0108] For open-ended questions, a linear classifier is used to predict the probability that each answer in the answer set is the correct answer, and the cross entropy function is used as the loss function for parameter training:

[0109]

[0110] in, is the probability that each answer is the correct answer, yi=1 means a i ∈A is the correct answer.

[0111] For counting problems, a linear regressor is used to predict the answer, and the mean square error is used as the loss function to train the parameters:

[0112]

[0113] For multiple-choice questions, a linear classifier is used to predict the probability that each candidate answer is the correct answer, and the hinge loss function is used as the loss function to train the model:

[0114]

[0115] Among them, s p and n Represents the scores for correct answers and incorrect answers respectively.

[0116] Example

[0117] like Figure 1 , Figure 2 As shown in Figure 1, a video question answering method based on an object-oriented two-stream attention network is shown in Figure 1. The steps are as follows:

[0118] Step (1), preprocessing the input data. For an input video, firstly, the video frames are sampled by average sampling. In the present invention, the number of samples for each video is T=10 frames. Then, the Faster-RCNN target detection algorithm is used to generate the target object on each frame to obtain multiple candidate frames. In addition, a convolutional network is used to extract static appearance features and dynamic behavior features for each video frame. In the present invention, the ResNet-152 network trained on the ImageNet image library is used to extract static appearance features, and the I3D network trained on the Kinetics action recognition data set is used to extract dynamic behavior features of the video frame. Finally, the RoIAlign method is used to map the candidate frame of each target object to the feature map of the Conv5 layer of ResNet-152 and the feature map of the last convolution layer of I3D to obtain the static appearance features and dynamic behavior features of the target object.

[0119] Step (2), feature encoding of the video, adopts a dual-stream mechanism to encode the feature of the video, where one stream is the static appearance stream of the object and the other is the dynamic behavior stream of the object.

[0120] 2-1. Multi-dimensional feature aggregation and location information collection of a given target object The static appearance features and dynamic behavior features of the object are set O a / m = Static appearance features and dynamic behavior features of video frames Compute position-aware and context-aware features for each target object:

[0121] The spatial position coding and temporal position coding of each target object are calculated according to Formula 2, Formula 3, and Formula 4 respectively, and the position-aware features of each target object are calculated according to Formula 5.

[0122] According to Formula 6, the context-aware feature F of each target object is calculated. a / m .

[0123] 2-2 calculates the two-stream high-order feature representation of the target object, which is obtained by passing the position- and context-aware features obtained in 2-1 through two layers of graph convolution operations (Formula 7) and a residual network.

[0124] Step (3) encodes the input natural language question and embeds each word in the natural language question into the 300-dimensional feature space initialized by GloVe to obtain the local code of each word. Then, the local code of each word is sequentially passed into the bidirectional recurrent neural network to obtain the local code F of each word. Q ∈R L×d and the global encoding F of the problem q ∈R d .

[0125] Step (4) Cross-modal fusion of video and question: First, the interaction relationship analysis is performed within the modality, and the feature representation of the two modalities is updated according to the self-attention weight within the modality. The two-stream high-order feature F of the video obtained in step (2) is a / m And the word local feature F obtained in step (3) Q ∈R L×d Reconstruct features according to formula 9 and formula 10 respectively. Then, perform cross-modal semantic alignment on the reconstructed question features and target object high-order features according to formula 11 and formula 12, and update the dual-stream high-order features of the video at the same time. Finally, fuse the updated dual-stream high-order features and the global encoding of the question to obtain the encoding of the answer. The specific formula is as follows:

[0126] a=FC(FC(f a +f m )+FC(f q )) (Formula 16)

[0127] Among them, FC represents the fully connected layer, f a is the global static appearance feature encoding, f m Encodes the global dynamic behavior characteristics, f q Encode global problem characteristics.

[0128] Step (5) encodes and decodes the answer obtained in step (4) to obtain the final answer;

[0129] Different decoding methods are selected according to the type of question. For open-ended questions, a linear classifier is selected for decoding and the Loss defined in Formula 13 is used for parameter training. For counting questions, a linear regressor is selected for decoding and the Loss defined in Formula 14 is used for parameter training. For multiple-choice questions, a linear classifier is selected for decoding and the Loss defined in Formula 15 is used for parameter training.

[0130] In order to test the performance of the dual-stream mechanism and attention mechanism in the video question-answering method based on the object-oriented dual-stream attention network of the present invention, different model structures were adopted for ablation experiment testing. The experimental results are shown in Table 1. The data set used in the ablation experiment is the TGIF-QA data set, which contains four types of questions, namely Count counting problems, Action action recognition problems, Transition state conversion problems and FrameQA image question-answering problems. The first row Appearance and the third row Motion respectively indicate that the model only uses the static appearance stream and the dynamic behavior stream as the representation of the video and does not use the attention module for cross-modal semantic alignment during cross-modal fusion. The second row and the fourth row respectively indicate that only the static appearance stream and the dynamic behavior stream are used as the representation of the video while the attention module is used for cross-modal semantic alignment. The last row shows the performance of the dual-stream attention network model of the present invention.

[0131] Table 1. Comparison of ablation test performance of different model structures

[0132]

[0133] In order to further test the performance of the video question answering method based on the object-oriented two-stream attention network of the present invention, the present invention is compared with the L-GCN model which performs best on the TGIF-QA dataset. The experimental comparison results are shown in Table 2.

[0134] Table 2. Performance comparison between the present invention and L-GCN model

[0135]

Claims

1. A video question answering method based on an object-oriented two-stream attention network, characterized in that: Here are the steps: Step (1), data preprocessing: Input the video clip to be answered, first use the convolutional network to extract the static appearance features and dynamic behavior features of the video frame, then use the target detection algorithm to extract the target object in the video, and use the convolutional network to extract the static appearance features and dynamic behavior features of the object at the same time; use the extracted static appearance features and dynamic behavior features of the video frame as the context representation features of the target object; Step (2), video feature encoding: A dual-stream mechanism is used to encode video features, where the two streams are the static appearance stream and the dynamic behavior stream of the target object. In each stream, the features of the target object, the spatiotemporal encoding of the target object, and the contextual representation features of the target object are combined as nodes to construct a feature graph; a graph convolution operation is performed on the constructed feature graph to obtain high-order features of the target object in the video; Step (3), problem feature coding: Use a recurrent neural network to encode the question to obtain the local encoding of each word and the global encoding of the question sentence; Step (4), cross-modal fusion of video and question: In each stream, the high-order features of the target object are first input into the self-attention network to obtain the intra-modal interaction relationship between objects, and the high-order features of the target object in the video are reconstructed according to the intra-modal interaction relationship between objects. Similarly, the local features of each word are input into the self-attention network to obtain the intra-modal interaction relationship between words, and the question features are reconstructed according to the intra-modal interaction relationship between words. Then, the reconstructed high-level features of the target object and the reconstructed question features are input into the question-guided attention network to explore the cross-modal semantic connection between vocabulary and object features, and at the same time update the high-level features of the video; The high-order features of the static appearance stream and the dynamic behavior stream of the target object are fused to obtain the dual-stream high-order features of the video; The dual-stream high-order features of the video are combined with the global encoding of the question to obtain the answer encoding; Step (5), decode the answer code obtained in step (4) to obtain the final answer: Different decoding methods are used for different question types. Linear classifiers are used to decode open-ended questions and multiple-choice questions, and linear regressors are used to decode counting questions.

2. According to claim 1, a video question answering method based on an object-oriented dual-stream attention network is characterized in that: The specific method of step (1) is as follows: Input the video clip to be answered, divide the video clip into a fixed number of video frames, and perform corresponding static appearance feature extraction and dynamic behavior feature extraction on each frame to obtain the static appearance feature of the video frame. and dynamic behavior characteristics as the contextual representation features of the target object; Then, the target detection algorithm is used in each frame to locate the position information b of the target object, and then the RoIAlign alignment network is used to align the position information to the static appearance features and dynamic behavior features of the video frame to obtain the static appearance features O of each object. a and dynamic behavior characteristics O m : The object position information b is specifically formulated as follows: b=[c1,c2,w,h] (Formula 1) Among them, c1 and c2 are the center position coordinates of the target object, and w and h represent the length and width of the target object respectively.

3. The video question answering method based on object-oriented dual-stream attention network according to claim 2 is characterized in that: The specific method of step (2) is as follows: A dual-stream mechanism is used to represent the feature coding of the video. The static appearance features and the dynamic behavior features of the object are encoded in parallel using the same coding mechanism. The network that encodes the static appearance features is called the static appearance stream, and the network that encodes the dynamic behavior features is called the dynamic behavior stream. Finally, the high-order features of the static appearance stream and the high-order features of the dynamic behavior stream of the video are obtained respectively. The specific process is as follows: The static appearance stream and the dynamic behavior stream use the same network structure, so in order to simplify the expression, the superscript a / m is used to represent the characteristics of both streams; The characteristics of the target object in a given video and the location information collection of the target object First, the location information b of the target object is passed into the spatial position encoding network and the temporal position encoding network to obtain the spatial position encoding d of the object. s and time position code d t , where the spatial position encoding network is composed of a multi-layer perceptron, and the temporal position encoding network uses sine and cosine functions of different frequencies to encode the temporal position of the target object; the specific formula is as follows: d s =MLP(b) (Formula 2) Where b represents the location information of the target object obtained by formula 1; n represents the number of target objects; d p Represents the dimension of the encoding, and 0≤2j≤d p ; Afterwards, the features of the target object and the spatiotemporal position encoding of the target object are combined to obtain the location-aware features of the target object. The specific formula is as follows: The location-aware features of the target object are combined with the features of the video frame to obtain the context-aware features. The specific formula is as follows: After obtaining the features of each object with location information and context information, an undirected complete graph is established with the features of each object as nodes and the interaction relationships between objects as edges. in represents the node set, ε represents the edge set; the calculation formula of the interaction relationship matrix between objects is: Where φ(F a / m )=F a / m W, W is the learnable weight matrix; Then, two graph convolution operations are performed on the constructed undirected complete graph to obtain the high-order features of the object. The graph convolution formula is: GCN(F a / m ; A a / m ) = A a / m (F a / m W) (Equation 8).

4. The video question answering method based on an object-oriented dual-stream attention network according to claim 3 is characterized in that: The specific process of step (3) is as follows: First, each word in the question is mapped to a 300-dimensional word vector space initialized by GloVe to obtain the encoding of each word. Then, each encoding vector is passed into a bidirectional LSTM network to obtain the local encoding F of each word. Q ∈R L×d and the global encoding F of the question sentence q ∈R d .

5. The video question answering method based on object-oriented dual-stream attention network according to claim 4 is characterized in that: The cross-modal fusion steps described in step (4) are as follows: Similar to step (2), the two flows use the same network structure, so the superscripts a / m are used to represent the characteristics of the two flows; Define a self-attention network to explore the interaction relationship within the modality. The self-attention network formula is as follows: f=MultiHead(X,X,X)=[head1,head2,...,head h ]W O (Formula 9) in, d is the feature dimension; h means the multi-head attention mechanism has h heads; X represents the visual feature: W O , are respectively the learnable parameter matrices; A question-guided attention network is defined to explore the interaction between modalities. The formula of the question-guided attention network is as follows: f=MultiHead(X,Y,Y)=[head1,head2,...,head h ]W O (Formula 11) Among them, X represents visual features, and Y represents problem features; Then, the high-order feature F obtained in step (2) is a / m The local feature F of the text obtained in step (3) is passed into the defined self-attention network to obtain the intra-modal interaction relationship between objects. Q ∈R L×d The outputs of the two self-attention networks are passed into the defined self-attention network to obtain the intra-modal interaction relationship between words. Then, the outputs of the two self-attention networks are passed into the defined question-guided attention network to obtain the cross-modal semantic connection between words and objects. At the same time, the high-order features of each stream of the video are updated separately under the guidance of the question. Finally, the global feature encoding of the question is fused with the dual-stream features of the video to obtain the encoding of the answer.

6. The video question answering method based on object-oriented dual-stream attention network according to claim 5, characterized in that: Step (5) The decoding steps are as follows: For open-ended questions, a linear classifier is used to predict the probability that each answer in the answer set is the correct answer, and the cross entropy function is used as the loss function for parameter training: in, The probability that each answer is the correct answer, y i =1 means a i ∈A is the correct answer; For counting problems, a linear regressor is used to predict the answer, and the mean square error is used as the loss function to train the parameters: For multiple-choice questions, a linear classifier is used to predict the probability that each candidate answer is the correct answer, and the hinge loss function is used as the loss function to train the model: Among them, s p and n Represents the scores for correct answers and incorrect answers respectively.

Images