A spatio-temporal graph convolution-based video question answering method and system

Abstract

The application discloses a video question answering method and system based on a space-time graph convolution, wherein the method comprises the following steps: preprocessing video data and question text data; performing visual feature extraction based on the video data to generate a target matrix; performing space information aggregation based on the target matrix by using a gated multi-layer perceptron layer to obtain visual feature representation; constructing a space-time graph based on the visual feature representation, performing space graph convolution processing on the space graph, and performing time dynamic mining on the time graph; generating word-level question embedding based on the question text data to construct a question graph; generating video multi-level feature representation based on the target space-time graph by using a hierarchical aggregation algorithm, and performing feature aggregation on the question graph; fusing visual aggregation features and question text aggregation features, and generating a predicted answer based on an answer decoder. The application can dynamically adjust the attention weight of a visual object, enhances multi-modal collaborative perception capability, and improves the reliability of video question answering.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a video question answering method and system based on spatio-temporal graph convolution. BACKGROUND

[0002] Video question answering is a technology that answers questions about a given video in natural language form. In the past few years, thanks to the introduction of the Transformer model, a key algorithm for unified modeling of multi-modal information in cross-modal fields, and the continuous emergence of large-scale visual-text data sets, research on video question answering has been greatly promoted.

[0003] However, the modal interaction module of the Transformer model is relatively simple in design, so more attention is paid to overall recognition and shallow description, and less attention is paid to deep semantic interaction between visual and textual information. In more complex scenarios and problems involving visual reasoning, performance may decline. Some new research work adopts a more complex modal interaction module, which allows the model to maintain high accuracy while having high reasoning speed. However, due to the overly simple visual encoder, it cannot handle continuous video input, and such models require a large amount of data for pre-training, requiring high-quality data and large-scale computing power, which limits the practical application of such models.

[0004] Video question answering algorithms based on graph neural networks have also received much attention in recent years, but the ability of graph neural networks to capture spatio-temporal information of videos and the relatively simple modal interaction unit limit their reasoning ability. Such models do not make good use of the interaction between visual and textual information, and cannot achieve collaborative perception between modalities. Moreover, the graph structure of such models is simple and cannot capture video information well. SUMMARY

[0005] The present application aims to overcome the shortcomings of the prior art, and provides a video question answering method and system based on spatio-temporal graph convolution, which can dynamically adjust the attention weight of visual objects, enhance the multi-modal collaborative perception ability, and improve the reliability of video question answering.

[0006] To solve the above technical problems, the present application provides a video question answering method based on spatio-temporal graph convolution, which comprises:

[0007] Obtaining video data and question text data, and preprocessing the video data and question text data to obtain preprocessed video data and question text data;

[0008] Extracting visual features based on the preprocessed video data to obtain target visual features, and generating a target matrix based on the target visual features;

[0009] A gated multilayer perceptron layer is used to aggregate spatial information using the target matrix to obtain a visual feature representation;

[0010] Constructing a spatiotemporal graph based on the visual feature representation, the spatiotemporal graph includes a time graph and a space graph, performing spatial graph convolution processing on the spatial graph to obtain a target spatial graph, performing temporal dynamic mining on the time graph to obtain a target time graph, and generating a target spatiotemporal graph based on the target spatial graph and the target time graph;

[0011] Generate word-level question embeddings based on the preprocessed question text data, and construct a question graph based on the word-level question embeddings;

[0012] Generate a multi-level feature representation of the video based on the target spatiotemporal graph using a hierarchical aggregation algorithm, and perform feature aggregation on the question graph and the multi-level feature representation of the video based on a dynamic graph attention module to obtain visual aggregation features and question text aggregation features;

[0013] The visual aggregate features and the question text aggregate features are fused to obtain a joint feature representation, and the joint feature representation is used to generate a predicted answer based on an answer decoder.

[0014] Optionally, extracting visual features based on the preprocessed video data to obtain target visual features, and generating a target matrix based on the target visual features includes:

[0015] Extract appearance features from preprocessed video data based on the pre-trained ResNet-152 model;

[0016] splicing eight adjacent frames around each sampling frame in the preprocessed video data to obtain a spliced ​​frame, and extracting motion features from the spliced ​​frame based on a two-stream dilated 3D convolutional network model;

[0017] Performing projection processing on the appearance features and motion features based on a linear feature transformation layer to obtain appearance features and motion features after projection processing;

[0018] The projected appearance features and motion features are spliced ​​together to obtain the target matrix.

[0019] Optionally, the performing spatial information aggregation based on the gated multilayer perceptron layer using the target matrix to obtain a visual feature representation includes:

[0020] Based on the gated multilayer perceptron layer, the target matrix is ​​used in combination with the gating mechanism to perform linear mapping to obtain a linear mapping output. The expression of the linear mapping output is:

[0021] f W, (Z) = WZ + b,

[0022]

[0023] wherein f W, (Z) is a linear mapping output, W is a mapping parameter of spatial interaction, is a visual representation formed by the target matrix, b is a bias term, σ is an activation function, U is a linear transformation matrix, Z is an input vector;

[0024] generate a visual feature representation based on the linear mapping output using a spatial gating unit module, and an expression of the visual feature representation is:

[0025] O = Z * f W, (Z) * V,

[0026] wherein O is a visual feature representation, Z is an input vector, f W, (Z) is a linear mapping output, V is a linear mapping reflecting spatial dependency.

[0027] Optionally, the constructing a spatio-temporal graph based on the visual feature representation comprises:

[0028] taking each frame of object in the visual feature representation as a node, calculating a relative relationship between nodes, calculating a relative interaction relationship between each frame of object and neighbor based on the relative relationship between nodes, and constructing a spatial graph based on the relative interaction relationship, and an expression of the relative relationship between nodes is:

[0029]

[0030] wherein, is a relative relationship between an i-th node and a j-th node, W a is a parameter for modeling an absolute relationship, W r is a parameter for relative relationship reasoning, is an i'-th node in a t'-th frame, is a neighbor set of the i-th node in the t-th frame, is the i-th node in the t-th frame;

[0031] an expression of the relative interaction relationship between the object and the neighbor is:

[0032]

[0033] wherein, is a relative interaction relationship between an object and a neighbor, is a relative relationship between the i-th node and the j-th node in the t-th frame, and N(i) is all nodes interacting with the i-th node, a relative relationship between the i-th node and the m-th node in the t-th frame;

[0034] calculating a link score between adjacent frames in a visual feature representation, taking the link score as a relationship between adjacent frames, the relationship between adjacent frames being an edge of a temporal graph, and the calculation expression of the link score being:

[0035]

[0036] wherein s i, is a link score between the i-th object and the j-th object, ψ is a cosine similarity function between adjacent frames, is an appearance feature of the i-th object in the t-th frame, is an appearance feature of the j-th object in the t+1-th frame, is a motion feature of the i-th object in the t-th frame, is a motion feature of the j-th object in the t+1-th frame, λ is a hyperparameter, and IoU is an intersection over union;

[0037] calculating a node connection weight matrix based on the appearance feature and the motion feature after projection processing in a visual feature representation, and calculating edge connectivity based on the node connection weight matrix, constructing a temporal graph based on the edge connectivity and edges of the temporal graph, and the expression of the node connection weight matrix being:

[0038]

[0039] wherein, is a node connection weight matrix, i and j are node numbers, λ1 and λ2 are scaling factors, is an appearance feature after projection processing of the i-th node, T is time, is an appearance feature after projection processing of the j-th node, is a motion feature after projection processing of the i-th node, is a motion feature after projection processing of the j-th node, and L is the number of frames;

[0040] the expression of the edge connectivity being:

[0041]

[0042] wherein R t is edge connectivity, σ is an activation function, is a node connection weight matrix, and is a linear variation of different node connection weight matrices, is an appearance feature after projection processing of the i-th node, T is time, is an appearance feature after projection processing of the j-th node, the projected motion feature of the i-th node, the projected motion feature of the j-th node.

[0043] Optionally, the spatial graph convolution processing on the spatial graph is performed to obtain a target spatial graph, including:

[0044] The spatial graph convolution processing on the spatial graph is performed based on an attention mechanism to obtain a target spatial graph, and an expression of the spatial graph convolution processing is:

[0045]

[0046]

[0047] wherein, is an output of the spatial graph convolution processing, ReLU is an activation function, MHSA is multi-head self-attention, H is a number of attention heads, W () is a graph weight matrix of the u-th layer, u is a layer number, is an output node representation of the (u-1)-th layer, R t is edge connectivity, l is a frame length, n is a number of clips, and I is a convolution dimension.

[0048] Optionally, the time dynamic mining on the time graph is performed to obtain a target time graph, including:

[0049] The node feature processing on the time graph is performed based on a node Transformer to obtain a time graph after node feature processing;

[0050] The edge feature processing on the time graph after node feature processing is performed based on an edge Transformer to obtain a target time graph.

[0051] Optionally, the word-level question embedding is generated based on the preprocessed question text data, and a question graph is constructed based on the word-level question embedding, including:

[0052] The word embedding of the preprocessed question text data is obtained based on a bidirectional encoder representation transformer model;

[0053] The attention weight of each word is calculated based on a natural language processing toolkit using the word-level question embedding, and a target word-level question embedding is generated based on the attention weight, and an expression of the calculation of the attention weight is:

[0054] A w = softmax MLP(E s ),

[0055] wherein, A wis the attention weight, MLP is a multi-layer perceptron, E s is the softmax function,

[0056] The expression of the target word-level question embedding is:

[0057]

[0058] wherein, is the target word-level question embedding, is the attention weight of the corresponding word, Q w is the word-level question embedding;

[0059] The relevance between question nodes is calculated, the target weight is calculated, and a question graph is constructed based on the attention weight, the target word-level question embedding, the relevance between question nodes, and the target weight, and the expression of the relevance between question nodes is:

[0060]

[0061] wherein, e ij is the relevance between the i-th question node and the j-th question node, and λ is a scaling factor, is the attention output of the i-th question word, is the attention output of the j-th question word, and K is the number of question words.

[0062] Optionally, the video multi-level feature representation is generated based on the target spatio-temporal graph using a hierarchical aggregation algorithm, and the question graph and the video multi-level feature representation are aggregated based on a dynamic graph attention module to obtain visual aggregated features and question text aggregated features, including:

[0063] The node is aggregated based on the target spatio-temporal graph using an attention mechanism to obtain a frame-level feature representation, and the expression of the frame-level feature representation is:

[0064]

[0065]

[0066] wherein, f F is the frame-level feature representation, and N is the number of clips, is a linear transformation of the learnable parameter W G , is the i-th node representation of the spatio-temporal graph, and σ is an activation function;

[0067] The frame-level feature representation is projected based on a self-attention layer to obtain a projected frame-level feature representation, and the expression of the projected frame-level feature representation is:

[0068] f′ F =φW m [MHSA(φW f (f O ))],

[0069] Among them, f′ F is the frame-level feature representation after projection, W m and W f is a linear change matrix, MHSA is a multi-head self-attention mechanism, φ is a linear transformation parameter, and f O Identifies the object;

[0070] The fragment-level feature representation is generated based on the maximum pooling, and the expression of the fragment-level feature representation is:

[0071] f C =MPool(f′ F ),

[0072] Among them, f C is the segment-level feature representation, f′ F is the frame-level feature representation after projection, M is the segment-based motion feature vector, and Pool is the pooling function;

[0073] The multi-level feature representation of the video is used as a video graph, and the target attention weight between the video graph and the question graph is calculated. The calculation expression of the target attention weight is:

[0074]

[0075] φ(u i ,v t )=W2·σ(W1·[u i ;v t ]+b1)+b2,

[0076] Among them, a i, is the attention weight between the i-th node in the question graph and the t-th node in the video graph, exp is the exponential function, φ is the score function, and u i is the i-th node in the problem graph, v t is the t-th node in the video graph, W1, W2, b1 and b2 are the learnable parameters of the feedforward function, and σ is the nonlinear activation function;

[0077] Based on the target attention weight, feature aggregation is performed to obtain visual aggregation features and question text aggregation features. The expression of the visual aggregation feature is:

[0078]

[0079] in, is the visual aggregation feature, N is the number of clips, a i, is the attention weight between the i-th node in the question graph and the t-th node in the video graph, u i is the i-th node in the question graph;

[0080] The expression of the question text aggregation feature is:

[0081]

[0082] wherein, is the question text aggregation feature, T is the total number of words, a i, is the attention weight between the i-th node in the question graph and the t-th node in the video graph, v t is the t-th node in the video graph.

[0083] Optionally, the visual aggregation feature and the question text aggregation feature are fused to obtain a joint feature representation, and the answer decoder generates a predicted answer based on the joint feature representation, including:

[0084] The visual aggregation feature and the video multi-level feature representation are combined to obtain a combined visual feature, and the question text aggregation feature and the question graph are combined to obtain a combined question text feature, and the expression of the combined visual feature is:

[0085]

[0086] wherein, h t is the combined visual feature, ReLU is an activation function, φW v is a linear change with a learnable parameter, v t is the video multi-level feature representation, is the visual aggregation feature, b v is a bias term;

[0087] The expression of the combined question text feature is:

[0088]

[0089] wherein, h i is the combined question text feature, ReLU is an activation function, φW u is a linear change with a learnable parameter, u i is the question graph, is the question text aggregation feature, b u is a bias term;

[0090] The combined visual feature and the combined question text feature are fused to obtain a joint feature representation, and the expression of the joint feature representation is:

[0091]

[0092] f joint =Fusion(X),

[0093] wherein h t is the combined visual feature, h i is the combined question text feature, N is the number of clips, T is the total number of words, f joint is the joint feature representation, and Fusion is a fusion function.

[0094] inputting the joint feature representation into an answer decoder, performing prediction calculation based on the answer decoder to obtain a predicted answer, wherein the expression of the prediction calculation is:

[0095] y'=sigma(W o f joint +b o ),

[0096] s=softmax(W y′ y'+b),

[0097] wherein y' is a question mapping, sigma is an activation function, f joint is the joint feature representation, W o and W y′ are weights of the answer decoder, b o and b are bias terms of the answer decoder, softmax is a normalized exponential function, and s is a mapping probability of the answer.

[0098] In addition, the application also provides a video question answering system based on a spatio-temporal graph convolution, which comprises:

[0099] a data acquisition and preprocessing module, which is used for acquiring video data and question text data, and pre-processing the video data and the question text data to obtain pre-processed video data and question text data;

[0100] a target matrix generation module, which is used for performing visual feature extraction based on the pre-processed video data to obtain target visual features, and generating a target matrix based on the target visual features;

[0101] a visual feature representation acquisition module, which is used for performing spatial information aggregation based on a gated multi-layer perceptron layer and the target matrix to obtain a visual feature representation;

[0102] The target spatio-temporal graph generation module is configured to construct a spatio-temporal graph based on the visual feature representation, the spatio-temporal graph comprising a time graph and a space graph, perform space graph convolution processing on the space graph to obtain a target space graph, perform time dynamic mining on the time graph to obtain a target time graph, and generate a target spatio-temporal graph based on the target space graph and the target time graph.

[0103] The question graph construction module is configured to generate a word-level question embedding based on the preprocessed question text data, and construct a question graph based on the word-level question embedding.

[0104] The feature aggregation module is configured to generate a video multi-level feature representation based on the target spatio-temporal graph using a hierarchical aggregation algorithm, and perform feature aggregation on the question graph and the video multi-level feature representation based on a dynamic graph attention module to obtain visual aggregated features and question text aggregated features.

[0105] The answer prediction module is configured to fuse the visual aggregated features and the question text aggregated features to obtain a joint feature representation, and generate a predicted answer based on an answer decoder using the joint feature representation.

[0106] In the embodiment of the present application, the target matrix is used to aggregate spatial information based on the gated multi-layer perceptron layer to obtain the visual feature representation, which enhances the spatial information while keeping a relatively low number of parameters. The time graph is constructed based on the visual feature representation, and the space graph convolution processing is performed on the space graph to obtain the target space graph, which can capture the spatial dependency between the visual features within the frame, thereby enriching the feature representation of the spatio-temporal graph, which helps to improve the understanding of the video content by the model and improve the performance of the object spatial variation video question answering task. The target time graph is obtained by performing time dynamic mining on the time graph, which can effectively capture the time dynamics within the video, thereby improving the performance on tasks involving temporal changes in video question answering. The word-level question embedding is generated based on the preprocessed question text data, which can obtain context-aware word-level question embedding. The video multi-level feature representation is generated based on the target spatio-temporal graph using the hierarchical aggregation algorithm, which forms complete frame-level interactions and effectively captures multi-granularity information in the video, thereby enhancing the understanding of visual content by the model and improving the performance in the video question answering task. The question graph and the video multi-level feature representation are aggregated based on the dynamic graph attention module to obtain the visual aggregated features and the question text aggregated features, which can focus on relevant visual objects according to the key words in the question. The joint feature representation is obtained by fusing the visual aggregated features and the question text aggregated features, which contains more rich information and facilitates more accurate decision making in subsequent steps. The predicted answer is generated based on the answer decoder using the joint feature representation, which can dynamically adjust the attention weight of the visual object, enhance the multi-modal collaborative perception ability, and improve the reliability of the video question answering. BRIEF DESCRIPTION OF DRAWINGS

[0107] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0108] Figure 1 is a flowchart of a video question answering method based on spatio-temporal graph convolution in an embodiment of the present application;

[0109] Figure 2 is a structural composition diagram of a video question answering system based on spatio-temporal graph convolution in an embodiment of the present application. DETAILED DESCRIPTION

[0110] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort fall within the scope of the present application.

[0111] Embodiment one

[0112] Please refer to Figure 1 , Figure 1 is a flowchart of a video question answering method based on spatio-temporal graph convolution in an embodiment of the present application, the method comprising:

[0113] S11: obtaining video data and question text data, and pre-processing the video data and question text data to obtain pre-processed video data and question text data;

[0114] In the specific implementation of the present application, video data and question text data are obtained, and the question text data can include text data obtained through voice recognition. The video data and the question text data are preprocessed. The video data is denoised, that is, the noise in the video is removed from the video data, to obtain denoised video data. The question text data is denoised. Text denoising is cleaning the text data to remove web page tags, non-semantic characters, and garbled codes, etc. in the text data, to obtain denoised question text data. The denoised video data and the denoised question text data are de-redundant. Since the multi-modal data contains some redundant data of repeated dimensions, which is meaningless for subsequent feature extraction and analysis processing, the redundant data of repeated dimensions is removed. The multi-modal data is divided into a plurality of data blocks. The hash values in the data blocks are calculated. The code distance between data and data is calculated according to the hash values. If the code distance is less than a preset threshold, the redundant data is removed. The de-redundant video data and the de-redundant question text data are obtained. The de-redundant video data and the de-redundant question text data are normalized. The order of magnitude difference between each bit of data is eliminated to avoid too large data level difference, which causes too large prediction error. Normalized video data and normalized question text data are obtained. The normalized video data is subjected to key frame extraction to obtain preprocessed video data. After the question text data is normalized, the preprocessing is completed, and the preprocessed question text data is obtained.

[0115] S12: visual feature extraction based on the preprocessed video data is performed to obtain target visual features, and a target matrix is generated based on the target visual features;

[0116] In the specific implementation of the present application, the visual feature extraction based on the preprocessed video data is performed to obtain target visual features, and a target matrix is generated based on the target visual features. This includes: extracting appearance features of the preprocessed video data based on a pre-trained ResNet-152 model; splicing eight adjacent frames around each sampling frame in the preprocessed video data to obtain spliced frames, and extracting motion features of the spliced frames based on a dual-stream dilated 3D convolutional network model; projecting the appearance features and the motion features based on a linear feature transformation layer to obtain projected appearance features and motion features; and splicing the projected appearance features and the motion features to obtain the target matrix.

[0117] Specifically, the preprocessed video data is divided into a plurality of clip segments, each of which contains a plurality of frames, in order to effectively represent the diversified information in the video, two different features are extracted: frame-based appearance features and segment-based motion features, the pre-trained ResNet-152 model is used to extract the appearance features of the preprocessed video data, and the ResNet-152 model is a deep residual network with powerful image classification capability. Eight adjacent frames around each sample frame in the preprocessed video data are spliced to obtain spliced frames, and the motion features of the spliced frames are extracted based on a two-stream inflated 3D convolution network (Two-Stream Inflated 3D ConvNets, I3D) model. The appearance features and motion features are projected by a linear feature transformation layer, the appearance features and motion features are projected into a multi-dimensional feature space by the linear feature transformation layer, and the projected appearance features and motion features are obtained. The projected appearance features and motion features are spliced to obtain a target matrix, also known as a visual representation, and its expression is:

[0118]

[0119] wherein, is the target matrix, is the projected appearance feature, is the projected motion feature, L is the number of video frames, and d' is the dimension of the projection space.

[0120] S13: performing spatial information aggregation based on the target matrix by using a gated multi-layer perceptron layer to obtain a visual feature representation.

[0121] In the specific implementation process of the present application, the spatial information aggregation based on the target matrix by using the gated multi-layer perceptron layer to obtain the visual feature representation includes: performing linear mapping based on the target matrix by using the gated multi-layer perceptron layer in combination with the gating mechanism to obtain a linear mapping output, and the expression of the linear mapping output is:

[0122] f W, (Z)=WZ+b,

[0123]

[0124] wherein, f W, (Z) is the linear mapping output, W is the mapping parameter of spatial interaction, is the visual representation formed by the target matrix, b is the bias term, σ is the activation function, U is the linear transformation matrix, and Z is the input vector.

[0125] generate a visual feature representation using a spatial gating unit module based on the linear mapping output, an expression of the visual feature representation being:

[0126] O = Z f (Z) V W, (Z) V,

[0127] wherein O is a visual feature representation, Z is an input vector, f W, (Z) is a linear mapping output, and V is a linear mapping reflecting spatial dependency.

[0128] Specifically, the target matrix is input into a gated multi-layer perceptron (gMLP) layer, in which spatial information aggregation can be achieved by simultaneously using static parameterized channel projection, spatial mapping and gating mechanism. In the spatial dimension, the target matrix is linearly mapped by the gMLP layer to obtain a linear mapping output, an expression of the linear mapping output being:

[0129] f W, (Z) = WZ + b,

[0130]

[0131] wherein f W, (Z) is a linear mapping output, W is a mapping parameter of spatial interaction, is a visual representation formed by the target matrix, b is a bias term, σ is an activation function, U is a linear transformation matrix, and Z is an input vector; a visual feature representation is generated using a spatial gating unit module based on the linear mapping output, an expression of the visual feature representation being:

[0132] O = Z f (Z) V W, (Z) V,

[0133] wherein O is a visual feature representation, Z is an input vector, f W, (Z) is a linear mapping output, and V is a linear mapping reflecting spatial dependency. The spatial gating unit module can capture spatial dependency, and in this case, to ensure stable training, W and b can be initialized as 0 and 1 respectively, so that the gMLP approximates to a feedforward neural network in the initial stage of training, and gradually learns the spatial information between features during the training process.

[0134] S14: constructing a spatio-temporal graph based on the visual feature representation, the spatio-temporal graph including a time graph and a space graph, performing spatial graph convolution processing on the space graph to obtain a target space graph, performing time dynamic mining on the time graph to obtain a target time graph, and generating a target spatio-temporal graph based on the target space graph and the target time graph;

[0135] In the implementation of the present application, the constructing the spatio-temporal graph based on the visual feature representation comprises: taking each frame of object in the visual feature representation as a node, calculating the relative relationship between nodes, calculating the relative interaction relationship between each frame of object and neighbor based on the relative relationship between nodes, and constructing a spatial graph based on the relative interaction relationship. The calculation expression of the relative relationship between nodes is:

[0136]

[0137] wherein, is the relative relationship between the i-th node and the j-th node, W a is a parameter for modeling the absolute relationship, W r is a parameter for relative relationship reasoning, is the i'-th node in the t' frame, is the neighbor set of the i-th node in the t frame, is the i-th node in the t frame;

[0138] The expression of the relative interaction relationship between the object and the neighbor is:

[0139]

[0140] wherein, is the relative interaction relationship between the object and the neighbor, is the relative relationship between the i-th node and the j-th node in the t frame, and N(i) is all nodes interacting with the i-th node, is the relative relationship between the i-th node and the m-th node in the t frame;

[0141] The link score between adjacent frames in the visual feature representation is calculated, and the link score is taken as the relationship between adjacent frames. The relationship between adjacent frames is the edge of the time graph, and the calculation expression of the link score is:

[0142]

[0143] wherein, s i, is the link score between the i-th object and the j-th object, and ψ is the cosine similarity function between adjacent frames, is the appearance feature of the i-th object in the t frame, is the appearance feature of the j-th object in the t+1 frame, is the motion feature of the i-th object in the t frame, is the motion feature of the j-th object in the t+1 frame, λ is a hyperparameter, and IoU is the intersection over union;

[0144] A node connection weight matrix is ​​calculated based on the appearance features and motion features after projection processing in the visual feature representation, and edge connectivity is calculated based on the node connection weight matrix. A time graph is constructed based on the edge connectivity and the edges of the time graph. The expression of the node connection weight matrix is:

[0145]

[0146] in, is the node connection weight matrix, i and j are node numbers, λ1 and λ2 are scaling factors, is the appearance feature of the i-th node after projection processing, T is the time, is the appearance feature of the j-th node after projection processing, is the motion feature of the i-th node after projection processing, is the motion feature after projection processing of the j-th node, and L is the number of frames;

[0147] The expression of the edge connectivity is:

[0148]

[0149] Among them, R t is the edge connectivity, σ is the activation function, is the node connection weight matrix, and is the linear change of the connection weight matrix of different nodes, is the appearance feature of the i-th node after projection processing, T is the time, is the appearance feature of the j-th node after projection processing, is the motion feature of the i-th node after projection processing, is the motion feature of the j-th node after projection processing.

[0150] Furthermore, performing spatial graph convolution processing on the spatial graph to obtain a target spatial graph includes: performing spatial graph convolution processing on the spatial graph based on an attention mechanism to obtain a target spatial graph, and the expression of the spatial graph convolution processing is:

[0151]

[0152]

[0153] in, is the output of the spatial graph convolution process, ReLU is the activation function, MHSA is multi-head self-attention, H is the number of attention heads, and W () is the graph weight matrix of the u-th layer, u is the number of layers, is the output node representation of the u-1 layer, Rt is the edge connectivity, l is the frame length, n is the number of clips, and I is the convolution dimension.

[0154] Further, the time dynamic mining is performed on the time graph to obtain a target time graph, including: performing node feature processing on the time graph based on a node Transformer to obtain a time graph after node feature processing; performing edge feature processing on the time graph after node feature processing based on an edge Transformer to obtain the target time graph.

[0155] Specifically, most existing video question answering methods conceptualize a video as a comprehensive spatio-temporal graph, covering the representation of all detected objects, and use a simple similarity measure for node aggregation, which usually ignores the interactive relationship between objects. Therefore, in the present application, the interactive relationship is modeled and integrated into the graph framework, and the visual feature representation is processed as a spatio-temporal sequence. In order to reduce the computational complexity, a spatial graph and a temporal graph are constructed independently. In the spatial graph, the object representation of each frame is used as a node, and the interaction between different objects is assumed to be represented by the dynamic flow of information. The relationship between objects is determined according to the position change and relative position between different frames. The object in each frame of the visual feature representation is taken as a node, the relative relationship between nodes is calculated, the relative interaction relationship between each frame of the object and the neighbor is calculated based on the relative relationship between nodes, and the spatial graph is constructed based on the relative interaction relationship. The calculation expression of the relative relationship between nodes is:

[0156]

[0157] wherein, is the relative relationship between the i-th node and the j-th node, W a is a parameter for absolute relationship modeling, W r is a parameter for relative relationship reasoning, is the i'-th node in the t'-th frame, is the neighbor set of the i-th node in the t-th frame, is the i-th node in the t-th frame;

[0158] The expression of the relative interaction relationship between the object and the neighbor is:

[0159]

[0160] wherein, is the relative interaction relationship between the object and the neighbor, is the relative relationship between the i-th node and the j-th node in the t-th frame, and N(i) is all nodes interacting with the i-th node, The relative relationship between the i-th node and the m-th node in the t-th frame is based on the calculated relationship, and the spatial graph is constructed by connecting nodes and edges.

[0161] Each frame in the visual feature representation is taken as a node, and the relationship between adjacent frames is represented as an edge of the graph. The link score between adjacent frames in the visual feature representation is calculated, and the link score is taken as the relationship between adjacent frames, which is the edge of the temporal graph. The calculation expression of the link score is:

[0162]

[0163] where s i, is the link score between the i-th object and the j-th object, ψ is the cosine similarity function between adjacent frames, is the appearance feature of the i-th object in the t-th frame, is the appearance feature of the j-th object in the t+1-th frame, is the motion feature of the i-th object in the t-th frame, is the motion feature of the j-th object in the t+1-th frame, λ is a hyperparameter, and IoU is the intersection over union. In order to effectively balance the contributions of the cosine similarity and the IoU, the hyperparameter of the link score can be set to 1. The link score is maximized by a greedy algorithm, thereby connecting the same objects on different frames.

[0164] Each node is intricately connected to every other node through edges. These edges represent the relationships between nodes, which are quantified using vectors containing relevant features of each node. A node connection weight matrix is calculated based on the projected appearance features and motion features in the visual feature representation, and edge connectivity is calculated based on the node connection weight matrix. A temporal graph is constructed based on the edge connectivity and the edges of the temporal graph. The expression of the node connection weight matrix is:

[0165]

[0166] wherein, is the node connection weight matrix, i and j are node numbers, and λ1 and λ2 are scaling factors, is the projected appearance feature of the i-th node, T is time, is the projected appearance feature of the j-th node, is the projected motion feature of the i-th node, is the projected motion feature of the j-th node, and L is the number of frames.

[0167] The expression of the edge connectivity is:

[0168]

[0169] wherein R t is the edge connectivity, σ is the activation function, is the node connection weight matrix, and is the linear variation of different node connection weight matrices, is the appearance feature of the i-th node after projection processing, T is time, is the appearance feature of the j-th node after projection processing, is the motion feature of the i-th node after projection processing, is the motion feature of the j-th node after projection processing, and different variations are used to capture the asymmetric nature of the interaction between the subject and the object in the real world, as well as the complementary characteristics of appearance and motion features. Based on the relationship between nodes and edges, a temporal graph is constructed.

[0170] For the constructed temporal graph and spatial graph, further separate processing is required. In the spatial graph, the importance of adjacent nodes in spatial dependency learning is emphasized in order to infer the spatial interaction between objects. To achieve this, an attention mechanism is integrated into node aggregation, and graph convolution is applied directly in the spatial domain to reduce computational complexity. Several layers of spatial graph attention convolution are used on the spatial graph, i.e., the spatial graph is processed using spatial graph convolution based on an attention mechanism to obtain a target spatial graph, and the expression of the spatial graph convolution processing is:

[0171]

[0172]

[0173] wherein, is the output of the spatial graph convolution processing, ReLU is the activation function, MHSA is multi-head self-attention, H is the number of attention heads, W () is the graph weight matrix of the u-th layer, u is the number of layers, is the output node representation of the u-1-th layer, R t is the edge connectivity, l is the frame length, n is the number of clips, I is the convolution dimension, and the spatial graph convolution is used to learn and represent the visual content in the video using the spatial relationship within the video frame. It can capture the spatial dependency between visual features within the frame by applying spatial graph convolution operations, thereby enriching the feature representation of the spatio-temporal graph, which helps to improve the understanding of video content by the model and improve the performance of the object spatial change video question answering task.

[0174] In the video question answering task, the unique temporal information of the video is a key challenge of the video question answering task, in order to overcome this challenge, effectively capture the temporal information in the video clip, and apply the graph Transformer unit to learn the long-term temporal dependence between objects. The graph Transformer unit is composed of node Transformer and edge Transformer, accepts a set of time graph sequences as input, and mines the temporal dynamics in the video. Based on the node Transformer, the node feature processing is performed on the time graph, and the time graph after node feature processing is obtained, the core component of the node Transformer is the time graph multi-head attention module, which is specially designed for structured input data, unlike the traditional attention mechanism, the traditional attention mechanism needs to vectorize the input, while the graph input is essentially non-vectorized structured data, therefore, a specific dynamic range expansion Message Passing Neural Network (dyMPN) is needed to extract vectors from the nodes of the graph as queries, keys and values, and then input them into the attention block, wherein for the dyMPN, the number of jumps in the training process is selected by a random strategy, and the random receptive field is enabled for each node in the graph convolution operation by randomizing the message passing scheme, which has better generalization, the expression of the time graph after node feature processing is:

[0175]

[0176] Wherein, F′ o is the time graph after node feature processing, N is the number of clip segments, Trans is the function used by the node Transformer, is the feature representation sequence corresponding to the i-th object in all frames, in the node Transformer, we use several self-attention blocks to enhance the object representation of the node, and realize it by aggregating the information from other nodes in all adjacent frames of the same object.

[0177] In order to better capture the motion changes between video frames and the evolution of visual content, so as to realize improved causal relationship and temporal reasoning, the edge Transformer is used on the updated edge, that is, the edge feature processing is performed on the time graph after node feature processing based on the edge Transformer, and the target time graph is obtained, the expression of the target time graph is:

[0178] R′=ETrans(R s ),

[0179] Wherein, R′ is the target time graph, E is the transmission efficiency, Trans is the edge Transformer function, and Rs As an updated edge relation matrix, the edge Transformer aims to process the edge features in the temporal graph, promoting the learning of the temporal relationship between objects, and generating a target spatio-temporal graph from the target spatial graph and the target temporal graph. By integrating the node and edge Transformers, the model can effectively capture the temporal dynamics within the video, thereby improving the performance on tasks involving temporal changes in video question answering.

[0180] S15: generating word-level question embeddings based on the preprocessed question text data, and constructing a question graph based on the word-level question embeddings;

[0181] In the specific implementation of the present application, generating word-level question embeddings based on the preprocessed question text data, and constructing a question graph based on the word-level question embeddings, comprises: embedding the preprocessed question text data based on a bidirectional encoder representation transformer model to obtain word-level question embeddings; using a natural language processing toolkit to calculate the attention weight of each word based on the word-level question embeddings, and generating target word-level question embeddings based on the attention weight, the calculation expression of the attention weight being:

[0182] A w =softmax MLP(E s )),

[0183] wherein A w is the attention weight, MLP is a multi-layer perceptron, E s is a word, and softmax is a normalization exponential function,

[0184] The expression of the target word-level question embedding is:

[0185]

[0186] wherein, is the target word-level question embedding, is the attention weight of the corresponding word, and Q w is the word-level question embedding;

[0187] The correlation between the question nodes is calculated, the target weight is calculated, and the question graph is constructed based on the attention weight, the target word-level question embedding, the correlation between the question nodes, and the target weight, the expression of the correlation between the question nodes being:

[0188]

[0189] wherein e ij is the correlation between the i-th question node and the j-th question node, and λ is a scaling factor, Attention output of the i-th question word, Attention output of the j-th question word, K is the number of question words.

[0190] Specifically, the preprocessed question text data word is embedded based on a bidirectional encoder representations from transformers (BERT) model to obtain a word-level question embedding, the context dependency relationship is established by using the bidirectional encoding context capability of the BERT model, and the context-aware word-level question embedding is generated wherein L represents the number of words in the question, C w represents the dimension of the question embedding. The CoreNLP is used to calculate the attention weight of each word based on the word-level question embedding, the CoreNLP is a natural language processing toolkit, the semantic dependency relationship between the question and the candidate answer is identified through the CoreNLP, the attention mechanism is used to focus on the features of the question, the weight is assigned according to the part of speech, the attention weight of each word is calculated, and the target word-level question embedding is generated based on the attention weight, and the calculation expression of the attention weight is:

[0191] A w =softmax MLP(E s )),

[0192] wherein A w is the attention weight, MLP is a multi-layer perceptron, E s is the word, and softmax is a normalized exponential function,

[0193] The expression of the target word-level question embedding is:

[0194]

[0195] wherein, is the target word-level question embedding, is the attention weight of the corresponding word, and Q w is the word-level question embedding.

[0196] The correlation between the question nodes is calculated, the target weight is calculated, each word is designated as a node, and the graph representation of the question is constructed, that is, the question graph is constructed based on the attention weight, the target word-level question embedding, the correlation between the question nodes and the target weight, and the expression of the correlation between the question nodes is:

[0197]

[0198] wherein e ijis the relevance between the ith question node and the jth question node, and λ is a scaling factor, is the attention output of the ith question word, is the attention output of the jth question word, and K is the number of question words.

[0199] S16: generating a video multi-level feature representation based on the target spatio-temporal graph using a hierarchical aggregation algorithm, and performing feature aggregation on the question graph and the video multi-level feature representation based on a dynamic graph attention module to obtain visual aggregated features and question text aggregated features;

[0200] In the specific implementation of the present application, the generation of a video multi-level feature representation based on the target spatio-temporal graph using a hierarchical aggregation algorithm, and the feature aggregation on the question graph and the video multi-level feature representation based on a dynamic graph attention module to obtain visual aggregated features and question text aggregated features, comprises: performing node aggregation based on the target spatio-temporal graph using an attention mechanism to obtain a frame-level feature representation, and the expression of the frame-level feature representation is:

[0201]

[0202]

[0203] wherein f F is the frame-level feature representation, N is the number of clips, is a linear transformation of the learnable parameter W G , is the ith node representation of the spatio-temporal graph, and σ is an activation function;

[0204] projecting the frame-level feature representation based on a self-attention layer to obtain a projected frame-level feature representation, and the expression of the projected frame-level feature representation is:

[0205] f′ F = φW m [MHSA(φW f (f O ))],

[0206] wherein f′ F is the projected frame-level feature representation, W m and W f are linear transformation matrices, MHSA is a multi-head self-attention mechanism, φ is a linear transformation parameter, and f O is the object identifier;

[0207] generating a clip-level feature representation based on maximum pooling, and the expression of the clip-level feature representation is:

[0208] f C = MPool(f′F ),

[0209] Among them, f C is the segment-level feature representation, f′ F is the frame-level feature representation after projection, M is the segment-based motion feature vector, and Pool is the pooling function;

[0210] The multi-level feature representation of the video is used as a video graph, and the target attention weight between the video graph and the question graph is calculated. The calculation expression of the target attention weight is:

[0211]

[0212] φ(u i ,v t )=W2·σ(W1·[u i ;v t ]+b1)+b2,

[0213] Among them, a i, is the attention weight between the i-th node in the question graph and the t-th node in the video graph, exp is the exponential function, φ is the score function, and u i is the i-th node in the problem graph, v t is the t-th node in the video graph, W1, W2, b1 and b2 are the learnable parameters of the feedforward function, and σ is the nonlinear activation function;

[0214] Based on the target attention weight, feature aggregation is performed to obtain visual aggregation features and question text aggregation features. The expression of the visual aggregation feature is:

[0215]

[0216] in, is the visual aggregation feature, N is the number of clips, a i, is the attention weight between the i-th node in the question graph and the t-th node in the video graph, u i is the i-th node in the problem graph;

[0217] The expression of the question text aggregation feature is:

[0218]

[0219] in, is the aggregated feature of the question text, T is the total number of words, a i, is the attention weight between the i-th node in the question graph and the t-th node in the video graph, v t is the tth node in the video graph.

[0220] Specifically, in the video question answering task, most methods process the video as a sequence of frames, and the hierarchical cognition of visual objects is insufficient. The visual elements in the semantic space are not continuous, but present a hierarchical distribution form. In order to align with the multi-granularity information in language, the visual features in the video are roughly modeled as a multi-level structure of object level, frame level and clip level. The current node representation has explicitly considered the temporal and spatial interaction between objects, and the interaction is basically individualized without forming a complete frame-level interaction. In order to aggregate the visual features and the interaction into high-level video elements, the object-level visual features and the interaction are aggregated by hierarchical aggregation to generate multi-level features. The target spatio-temporal graph is used to utilize the attention mechanism to aggregate nodes to obtain frame-level feature representation, and the expression of the frame-level feature representation is:

[0221]

[0222]

[0223] where f F is the frame-level feature representation, N is the number of clip segments, is a linear transformation of learnable parameters W G , is the i-th node representation of the spatio-temporal graph, and σ is an activation function.

[0224] Although the frame-level feature representation captures the interaction between local objects, it may ignore the global context of the frame. Especially considering the constraint of retaining only a few objects, it may not include all objects of interest in the frame. In order to solve this limitation, the object identifier is enhanced by a self-attention layer and projected into a linear space, that is, the frame-level feature representation is projected based on the self-attention layer to obtain the projected frame-level feature representation, and the expression of the projected frame-level feature representation is:

[0225] f′ F = φW m [MHSA(φW f (f O ))],

[0226] where f′ F is the projected frame-level feature representation, W m and W f are linear transformation matrices, MHSA is a multi-head self-attention mechanism, φ is a linear transformation parameter, and f O is an object identifier.

[0227] The clip-level feature representation is generated based on the maximum pooling, and the expression of the clip-level feature representation is:

[0228] f C =MPool(f′ F ),

[0229] where f C is the segment-level feature representation, f′ F is the projected frame-level feature representation, M is the segment-based motion feature vector, and Pool is the pooling function; this hierarchical aggregation mechanism effectively captures the multi-granularity information in the video, enhances the understanding of the visual content by the model, and improves the performance in the video question answering task.

[0230] Using a bidirectional long short-term memory network to aggregate the word-level question embeddings in the question graph into a sentence-level representation helps to fuse the features with the segment-level features, and the video multi-level feature representation is taken as a video graph, and the target attention weight between the video graph and the question graph is calculated, and the calculation expression of the target attention weight is:

[0231]

[0232] φ(u i ,v t )=W2·σ(W1·[u i ;v t ]+b1)+b2,

[0233] where a i, is the attention weight between the i-th node in the question graph and the t-th node in the video graph, exp is the exponential function, φ is the scoring function, u i is the i-th node in the question graph, v t is the t-th node in the video graph, W1, W2, b1 and b2 are learnable parameters of the feedforward function, and σ is a nonlinear activation function; the target attention weight is used to aggregate information between different layers of nodes in the graph, and can focus on the relevant visual objects according to the keyword set in the question. In order to be able to focus on the relevant visual objects according to the keyword set in the question, feature aggregation is performed based on the target attention weight to obtain visual aggregated features and question text aggregated features, and the expression of the visual aggregated features is:

[0234]

[0235] where, is the visual aggregated feature, N is the number of clips, a i, is the attention weight between the i-th node in the question graph and the t-th node in the video graph, and u i is the i-th node in the question graph.

[0236] The expression of the question text aggregated features is:

[0237]

[0238] in, is the aggregated feature of the question text, T is the total number of words, a i, is the attention weight between the i-th node in the question graph and the t-th node in the video graph, v t is the tth node in the video graph.

[0239] S17: Fusing the visual aggregate features and the question text aggregate features to obtain a joint feature representation, and generating a predicted answer based on the answer decoder using the joint feature representation.

[0240] In the specific implementation process of the present invention, the visual aggregate features and the question text aggregate features are fused to obtain a joint feature representation, and the joint feature representation is used to generate a predicted answer based on the answer decoder, including: combining the visual aggregate features with the multi-level feature representation of the video to obtain a combined visual feature, and combining the question text aggregate features with the question graph to obtain a combined question text feature. The expression of the combined visual feature is:

[0241]

[0242] Among them, h t To combine visual features, ReLU is the activation function, φW v is a linear variation with learnable parameters, v t is a multi-level feature representation of the video, is the visual aggregation feature, b v is the bias term;

[0243] The expression combining the question text features is:

[0244]

[0245] Among them, h i To combine the question text features, ReLU is the activation function, φW u is a linear variation with learnable parameters, u i For the problem diagram, is the aggregated feature of the question text, b u is the bias term;

[0246] The combined visual features and the combined question text features are fused to obtain a joint feature representation, and the expression of the joint feature representation is:

[0247]

[0248] f joint=Fusion(X),

[0249] Among them, h t To combine visual features, h i To combine the question text features, N is the number of clips, T is the total number of words, and f joint is the joint feature representation, and Fusion is the fusion function;

[0250] The joint feature representation is input into the answer decoder, and a prediction calculation is performed based on the answer decoder to obtain a predicted answer, wherein the expression of the prediction calculation is:

[0251] y′=σ(W o f joint +b o ),

[0252] s=softmax(W y′ y′+b),

[0253] Where y′ is the problem mapping, σ is the activation function, and f joint is the joint feature representation, W o and W y′ is the weight of the answer decoder, b o and b are the bias terms of the answer decoder, softmax is the normalized exponential function, and s is the mapping probability of the answer.

[0254] Specifically, the visual aggregate feature is combined with the multi-level feature representation of the video to obtain the combined visual feature, and the question text aggregate feature is combined with the question graph to obtain the combined question text feature. The expression of the combined visual feature is:

[0255]

[0256] Among them, h t To combine visual features, ReLU is the activation function, φW v is a linear variation with learnable parameters, v t is a multi-level feature representation of the video, is the visual aggregation feature, b v is the bias term;

[0257] The expression combining the question text features is:

[0258]

[0259] Among them, h i To combine the question text features, ReLU is the activation function, φW u is a linear variation with learnable parameters, u i For the problem diagram, b is the bias term; can accommodate more abundant information, and promote more accurate decision-making in subsequent steps u b is the bias term; can accommodate more abundant information, and promote more accurate decision-making in subsequent steps

[0260] The combination of visual features and combined question text features is fused to obtain a joint feature representation, and the expression of the joint feature representation is:

[0261]

[0262] f joint =Fusion(X),

[0263] wherein h t is the combined visual feature, h i is the combined question text feature, N is the number of clips, T is the total number of words, f joint is the joint feature representation, and Fusion is the fusion function.

[0264] The joint feature representation is input into an answer decoder, and a prediction calculation is performed based on the answer decoder to obtain a predicted answer, wherein the expression of the prediction calculation is:

[0265] y′=σ(W o f joint +b o ),

[0266] s=softmax(W y′ y′+b),

[0267] wherein y' is the question mapping, σ is the activation function, f joint is the joint feature representation, W o and W y′ are the weights of the answer decoder, b o and b are the bias terms of the answer decoder, softmax is the normalization exponential function, and s is the mapping probability of the answer.

[0268] In addition, for multiple-choice question types, multiple answer candidates are processed to generate a set of visual representations corresponding to each candidate, and a target function is used to solve such problems, and the target function is:

[0269]

[0270] wherein is the inferred answer in the answer space, is the answer space, is the mapping function, q is the relevant question, is the video, and a is the answer set.

[0271] In the embodiment of the present application, the target matrix is used for spatial information aggregation based on the gated multi-layer perceptron gMLP layer to obtain a visual feature representation, which enhances the spatial information while maintaining a relatively low number of parameters. A spatio-temporal graph is constructed based on the visual feature representation, and a spatial graph convolution is performed on the spatial graph to obtain a target spatial graph, which can capture the spatial dependency between visual features within a frame, thereby enriching the feature representation of the spatio-temporal graph, which helps to improve the understanding of video content by the model and improve the performance of the object spatial change video question answering task. The target time graph is obtained by performing time dynamic mining on the time graph, which can effectively capture the time dynamics within the video, thereby improving the performance on tasks involving video question answering with time changes. The word-level question embedding is generated based on the preprocessed question text data, which can obtain context-aware word-level question embedding. The video multi-level feature representation is generated based on the target spatio-temporal graph using a hierarchical aggregation algorithm, forming a complete frame-level interaction, effectively capturing multi-granularity information in the video, enhancing the understanding of visual content by the model, and improving the performance in the video question answering task. The visual aggregation feature and the question text aggregation feature are obtained by performing feature aggregation on the question graph and the video multi-level feature representation based on the dynamic graph attention module, which can focus on relevant visual objects according to the keyword set in the question. The joint feature representation is obtained by fusing the visual aggregation feature and the question text aggregation feature, which contains more rich information and promotes more accurate decision making in subsequent steps. The prediction answer is generated based on the answer decoder using the joint feature representation, which can dynamically adjust the attention weight of the visual object, enhance the multi-modal collaborative perception ability, and improve the reliability of the video question answering.

[0272] Embodiment two

[0273] Please refer to Figure 2 , Figure 2 is a structural composition schematic diagram of the video question answering system based on spatio-temporal graph convolution in the embodiment of the present application.

[0274] As Figure 2 shown, a video question answering system based on spatio-temporal graph convolution, the system comprises:

[0275] The data acquisition and preprocessing module 21 is used for acquiring video data and question text data, and preprocessing the video data and question text data to obtain preprocessed video data and question text data.

[0276] The target matrix generation module 22 is used for extracting visual features based on the preprocessed video data to obtain target visual features, and generating a target matrix based on the target visual features.

[0277] The visual feature representation acquisition module 23 is used for aggregating spatial information based on the target matrix using the gated multi-layer perceptron layer to obtain a visual feature representation.

[0278] The target spatio-temporal graph generation module 24 is configured to construct a spatio-temporal graph based on the visual feature representation, the spatio-temporal graph comprising a time graph and a space graph, perform spatial graph convolution processing on the space graph to obtain a target space graph, perform time dynamic mining on the time graph to obtain a target time graph, and generate a target spatio-temporal graph based on the target space graph and the target time graph.

[0279] The question graph construction module 25 is configured to generate a word-level question embedding based on the preprocessed question text data, and construct a question graph based on the word-level question embedding.

[0280] The feature aggregation module 26 is configured to generate a video multi-level feature representation based on the target spatio-temporal graph using a hierarchical aggregation algorithm, and perform feature aggregation on the question graph and the video multi-level feature representation based on a dynamic graph attention module to obtain visual aggregated features and question text aggregated features.

[0281] The answer prediction module 27 is configured to fuse the visual aggregated features and the question text aggregated features to obtain a joint feature representation, and generate a predicted answer based on an answer decoder using the joint feature representation.

[0282] In the specific implementation of the present application, the specific implementation of the system item can refer to the implementation of the method item described above, which will not be repeated here.

[0283] In the embodiment of the present application, the gated multi-layer perceptron gMLP layer based on the target matrix is used to aggregate spatial information, obtain a visual feature representation, enhance the spatial information, and meanwhile maintain a relatively low number of parameters. A spatio-temporal graph is constructed based on the visual feature representation, a spatial graph convolution is performed on the spatial graph, a target spatial graph is obtained, the spatial dependency between the visual features in the frame can be captured, thereby enriching the feature representation of the spatio-temporal graph, which helps to improve the understanding of the model for the video content and improve the performance of the object spatial change video question answering task. The time dynamics are mined from the time graph to obtain a target time graph, which can effectively capture the time dynamics in the video, thereby improving the performance on the task involving the temporal change in the video question answering. The word-level question embedding is generated based on the preprocessed question text data, which can obtain the context-aware word-level question embedding. The video multi-level feature representation is generated based on the target spatio-temporal graph using a hierarchical aggregation algorithm, forming a complete frame-level interaction, effectively capturing the multi-granularity information in the video, enhancing the understanding of the model for the visual content, and improving the performance in the video question answering task. The dynamic graph attention module is used to aggregate the question graph and the video multi-level feature representation to obtain visual aggregation features and question text aggregation features, which can focus on the relevant visual objects according to the key words in the question. The visual aggregation features and the question text aggregation features are fused to obtain a joint feature representation, which contains more rich information and promotes more accurate decision making in the subsequent steps. The answer decoder is used to generate a predicted answer based on the joint feature representation, which can dynamically adjust the attention weight of the visual object, enhance the multi-modal collaborative perception ability, and improve the reliability of the video question answering.

[0284] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiments can be completed by programs instructing the related hardware, and the programs can be stored in a computer readable storage medium, which can include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0285] In addition, the above describes in detail the video question answering method and system based on spatio-temporal graph convolution provided by the embodiment of the present application, and the principle and implementation manner of the present application are described by using specific examples in this paper. The above embodiment is only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range can be changed, and the above description should not be understood as a limitation of the present application.

Claims

1. A video question answering method based on spatiotemporal graph convolution, characterized in that: The method comprises: Acquire video data and question text data, and preprocess the video data and question text data to obtain preprocessed video data and question text data; Performing visual feature extraction based on the preprocessed video data to obtain target visual features, and generating a target matrix based on the target visual features; A gated multilayer perceptron layer is used to aggregate spatial information using the target matrix to obtain a visual feature representation; Constructing a spatiotemporal graph based on the visual feature representation, the spatiotemporal graph includes a time graph and a space graph, performing spatial graph convolution processing on the spatial graph to obtain a target spatial graph, performing temporal dynamic mining on the time graph to obtain a target time graph, and generating a target spatiotemporal graph based on the target spatial graph and the target time graph; Generate word-level question embeddings based on the preprocessed question text data, and construct a question graph based on the word-level question embeddings; Generate a multi-level feature representation of the video based on the target spatiotemporal graph using a hierarchical aggregation algorithm, and perform feature aggregation on the question graph and the multi-level feature representation of the video based on a dynamic graph attention module to obtain visual aggregation features and question text aggregation features; The visual aggregate features and the question text aggregate features are fused to obtain a joint feature representation, and the joint feature representation is used to generate a predicted answer based on an answer decoder.

2. The video question answering method based on spatiotemporal graph convolution according to claim 1, characterized in that The extracting visual features based on the preprocessed video data to obtain target visual features, and generating a target matrix based on the target visual features, includes: Extract appearance features from preprocessed video data based on the pre-trained ResNet-152 model; splicing eight adjacent frames around each sampling frame in the preprocessed video data to obtain a spliced ​​frame, and extracting motion features from the spliced ​​frame based on a two-stream dilated 3D convolutional network model; Performing projection processing on the appearance features and motion features based on a linear feature transformation layer to obtain appearance features and motion features after projection processing; The projected appearance features and motion features are spliced ​​together to obtain the target matrix.

3. The video question answering method based on spatiotemporal graph convolution according to claim 1, characterized in that The gated multilayer perceptron layer uses the target matrix to aggregate spatial information to obtain visual feature representation, including: Based on the gated multilayer perceptron layer, the target matrix is ​​used in combination with the gating mechanism to perform linear mapping to obtain a linear mapping output. The expression of the linear mapping output is: f W,b (Z)=WZ+b, Among them, f W,b (Z) is the linear mapping output, W is the mapping parameter of spatial interaction, is the visual representation formed by the target matrix, b is the bias term, σ is the activation function, U is the linear transformation matrix, and Z is the input vector; Based on the linear mapping output, a spatial gating unit module is used to generate a visual feature representation, and the expression of the visual feature representation is: O=(Z⊙f W,b (Z))*V, Among them, O is the visual feature representation, Z is the input vector, and f W,b (Z) is the linear mapping output, and V is the linear mapping that reflects the spatial dependency.

4. The video question answering method based on spatiotemporal graph convolution according to claim 1, characterized in that The constructing of a spatiotemporal graph based on the visual feature representation includes: The objects in each frame of the visual feature representation are regarded as nodes, and the relative relationships between the nodes are calculated. Based on the relative relationships between the nodes, the relative interaction relationships between the objects in each frame and their neighbors are calculated. A spatial graph is constructed based on the relative interaction relationships. The calculation expression of the relative relationships between the nodes is: in, is the relative relationship between the i-th node and the j-th node, W a is the parameter used to model the absolute relationship, W r is the parameter used for relative relationship reasoning, For the tth ′ The i-th frame ′ nodes, is the neighbor set of the i-th node in the t-th frame, is the i-th node in the t-th frame; The expression of the relative interaction relationship between the object and its neighbors is: in, is the relative interaction relationship between the object and its neighbors, is the relative relationship between the i-th node and the j-th node in the t-th frame, N(i) is all the nodes that interact with the i-th node, is the relative relationship between the i-th node and the m-th node in the t-th frame; Calculate the link score between adjacent frames in the visual feature representation, and use the link score as the relationship between adjacent frames. The relationship between adjacent frames is the edge of the time graph. The calculation expression of the link score is: Among them, s i,j is the link score between the i-th object and the j-th object, ψ is the cosine similarity function between adjacent frames, is the appearance feature of the i-th object in the t-th frame, is the appearance feature of the j-th object in the t+1-th frame, is the motion feature of the i-th object in the t-th frame, The motion features of the jth object in the t+1th frame, λ is a hyperparameter, and IoU is the intersection over union ratio; A node connection weight matrix is ​​calculated based on the appearance features and motion features after projection processing in the visual feature representation, and edge connectivity is calculated based on the node connection weight matrix. A time graph is constructed based on the edge connectivity and the edges of the time graph.

5. The video question answering method based on spatiotemporal graph convolution according to claim 1, characterized in that The performing spatial graph convolution processing on the spatial graph to obtain a target spatial graph includes: Based on the attention mechanism, the spatial graph is convolved to obtain the target spatial graph. The expression of the spatial graph convolution is: in, is the output of the spatial graph convolution process, ReLU is the activation function, MHSA is multi-head self-attention, H is the number of attention heads, and W (u) is the graph weight matrix of the u-th layer, u is the number of layers, is the output node representation of the u-1 layer, R t is the edge connectivity, l is the frame length, n is the number of clips, and I is the convolution dimension.

6. The video question answering method based on spatiotemporal graph convolution according to claim 1, characterized in that The performing time dynamic mining on the time graph to obtain a target time graph includes: Performing node feature processing on the time graph based on a node transformer to obtain a time graph after node feature processing; Based on the edge transformer, edge feature processing is performed on the time graph after node feature processing to obtain the target time graph.

7. The video question answering method based on spatiotemporal graph convolution according to claim 1, characterized in that Generating word-level question embeddings based on the preprocessed question text data, and constructing a question graph based on the word-level question embeddings, includes: The pre-processed question text data is word-embedded based on the bidirectional encoding representation transformer model to obtain word-level question embeddings; The attention weight of each word is calculated based on the natural language processing toolkit using the word-level question embedding, and the target word-level question embedding is generated based on the attention weight. The calculation expression of the attention weight is: A w =softmax(MLP(E s )), Among them, A w is the attention weight, MLP is the multi-layer perceptron, E s is a word, softmax is a normalized exponential function, The target word-level question embedding expression is: in, is the target word-level question embedding, is the attention weight of the corresponding word, Q w Embedding for word-level questions; The associations between question nodes are calculated, target weights are calculated, and a question graph is constructed based on the attention weights, the target word-level question embeddings, the associations between question nodes, and the target weights.

8. The video question answering method based on spatiotemporal graph convolution according to claim 1, characterized in that The method generates a multi-level feature representation of the video based on the target spatiotemporal graph using a hierarchical aggregation algorithm, and aggregates the question graph and the multi-level feature representation of the video based on a dynamic graph attention module to obtain visual aggregate features and question text aggregate features, including: Based on the target spatiotemporal graph, an attention mechanism is used to aggregate nodes to obtain a frame-level feature representation; Projecting the frame-level feature representation based on the self-attention layer to obtain a projected frame-level feature representation; Generate segment-level feature representation based on maximum pooling, and generate multi-level feature representation of video based on the projected frame-level feature representation and segment-level feature representation; Representing the multi-level features of the video as a video graph, and calculating the target attention weight between the video graph and the question graph; Feature aggregation is performed based on the target attention weight to obtain visual aggregate features and question text aggregate features.

9. The video question answering method based on spatiotemporal graph convolution according to claim 1, characterized in that The fusing of the visual aggregate features and the question text aggregate features to obtain a joint feature representation, and generating a predicted answer using the joint feature representation based on an answer decoder, includes: Based on the combination of visual aggregate features and video multi-level feature representation, a combined visual feature is obtained. Based on the combination of question text aggregate features and question graph, a combined question text feature is obtained. The expression of the combined visual feature is: Among them, h t To combine visual features, ReLU is the activation function, ΦW v is a linear variation with learnable parameters, v t is a multi-level feature representation of the video, is the visual aggregation feature, b v is the bias term; The expression combining the question text features is: Among them, h i To combine the question text features, ReLU is the activation function, ΦW u is a linear variation with learnable parameters, u i For the problem diagram, is the aggregated feature of the question text, b u is the bias term; fusing the combined visual features and the combined question text features to obtain a joint feature representation; The joint feature representation is input into the answer decoder, and a prediction calculation is performed based on the answer decoder to obtain a predicted answer, wherein the expression of the prediction calculation is: y ′ =σ(W o f joint +b o ), s=softmax(W y′ y ′ +b), Among them, y ′ is the problem mapping, σ is the activation function, f joint is the joint feature representation, W o and W y′ is the weight of the answer decoder, b o and b are the bias terms of the answer decoder, softmax is the normalized exponential function, and s is the mapping probability of the answer.

10. A video question answering system based on spatiotemporal graph convolution, characterized in that: The system comprises: Data acquisition and preprocessing module: used to acquire video data and question text data, and preprocess the video data and question text data to obtain preprocessed video data and question text data; Target matrix generation module: used to extract visual features based on the preprocessed video data, obtain target visual features, and generate a target matrix based on the target visual features; A visual feature representation acquisition module is configured to aggregate spatial information using the target matrix based on a gated multilayer perceptron layer to obtain a visual feature representation; Target spatiotemporal graph generation module: used to construct a spatiotemporal graph based on the visual feature representation, the spatiotemporal graph including a time graph and a space graph, perform spatial graph convolution processing on the space graph to obtain a target spatial graph, perform temporal dynamic mining on the time graph to obtain a target time graph, and generate a target spatiotemporal graph based on the target spatial graph and the target time graph; Question graph construction module: used to generate word-level question embeddings based on the preprocessed question text data, and construct a question graph based on the word-level question embeddings; Feature aggregation module: used to generate a multi-level feature representation of the video based on the target spatiotemporal graph using a hierarchical aggregation algorithm, and perform feature aggregation on the question graph and the multi-level feature representation of the video based on the dynamic graph attention module to obtain visual aggregation features and question text aggregation features; Answer prediction module: used to fuse the visual aggregate features and the question text aggregate features to obtain a joint feature representation, and generate a predicted answer based on the answer decoder using the joint feature representation.

Images