Video question answering method and system based on space-time diagram convolution

Abstract

The invention discloses a video question answering method and system based on space-time diagram convolution, and the method comprises the steps: carrying out the preprocessing of video data and question text data; performing visual feature extraction based on the video data to generate a target matrix; performing spatial information aggregation by using the target matrix based on the gated multi-layer perceptron layer to obtain visual feature representation; constructing a space-time diagram based on visual feature representation, carrying out space diagram convolution processing on the space diagram, and carrying out time dynamic mining on the time diagram; generating word-level question embedding based on the question text data to construct a question graph; video multi-level feature representation is generated based on the target space-time diagram by using a hierarchical aggregation algorithm, and feature aggregation is performed on the video multi-level feature representation and the problem diagram; and fusing the visual aggregation feature and the question text aggregation feature, and generating a predicted answer based on an answer decoder. According to the method, the attention weight of the visual object can be dynamically adjusted, the multi-modal cooperative perception capability is enhanced, and the reliability of video question answering is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a video question-answering method and system based on spatiotemporal graph convolution. Background Art

[0002] Video question answering is a technology that answers questions about a given video in natural language. In the past few years, thanks to the introduction of the Transformer model, a key algorithm for uniformly modeling multimodal information in the cross-modal field, and the continuous emergence of large-scale visual-text datasets, the research on video question answering has been greatly promoted.

[0003] However, the modal interaction module of the Transformer model is relatively simple in design, so it focuses more on overall recognition and shallow description issues, and pays little attention to the deep semantic interaction between visual and text information. In more complex scenarios and problems involving visual reasoning, performance will degrade. Some new research works use more complex modal interaction modules, which enable the model to have a higher reasoning speed while maintaining high accuracy. However, due to the overly simple visual encoder, it cannot process continuous video input and such models require a large amount of data for pre-training, which has high requirements for high-quality data and large-scale computing power, limiting the practical application of such models.

[0004] Video question-answering algorithms based on graph neural networks have also received much attention in recent years. However, the ability of graph neural networks to capture spatiotemporal information of videos and relatively simple modal interaction units limit their reasoning capabilities. Such models do not make good use of the interactive relationship between vision and text, and cannot achieve collaborative perception between modalities. In addition, the graph structure of such models is simple and cannot capture video information well. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art. The present invention provides a video question-answering method and system based on spatiotemporal graph convolution, which can dynamically adjust the attention weight of visual objects, enhance multimodal collaborative perception capabilities, and improve the reliability of video question-answering.

[0006] In order to solve the above technical problems, the present invention provides a video question answering method based on spatiotemporal graph convolution, the method comprising:

[0007] Acquire video data and question text data, and preprocess the video data and question text data to obtain preprocessed video data and question text data;

[0008] Extract visual features based on the preprocessed video data to obtain target visual features, and generate a target matrix based on the target visual features;

[0009] Based on the gated multi-layer perceptron layer, the target matrix is ​​used to aggregate spatial information to obtain visual feature representation;

[0010] Based on the visual feature representation, a spatiotemporal graph is constructed, the spatiotemporal graph includes a time graph and a space graph, and the space graph is subjected to a spatial graph convolution process to obtain a target spatial graph, the time graph is subjected to temporal dynamic mining to obtain a target time graph, and a target spatiotemporal graph is generated based on the target spatial graph and the target time graph;

[0011] Generate word-level question embeddings based on the preprocessed question text data, and construct a question graph based on the word-level question embeddings;

[0012] Generate a multi-level feature representation of the video using a hierarchical clustering algorithm based on the target spatiotemporal graph, and perform feature aggregation on the question graph and the multi-level feature representation of the video based on a dynamic graph attention module to obtain visual aggregation features and question text aggregation features;

[0013] The visual aggregate features and the question text aggregate features are fused to obtain a joint feature representation, and the joint feature representation is used to generate a predicted answer based on an answer decoder.

[0014] Optionally, performing visual feature extraction based on the preprocessed video data to obtain target visual features, and generating a target matrix based on the target visual features includes:

[0015] Extract appearance features from preprocessed video data based on the pre-trained ResNet-152 model;

[0016] Splicing eight adjacent frames around each sampling frame in the preprocessed video data to obtain a spliced ​​frame, and extracting motion features from the spliced ​​frame based on a two-stream dilated 3D convolutional network model;

[0017] Performing projection processing on the appearance features and motion features based on a linear feature transformation layer to obtain the appearance features and motion features after projection processing;

[0018] The projected appearance features and motion features are concatenated to obtain the target matrix.

[0019] Optionally, the gated multilayer perceptron layer uses the target matrix to aggregate spatial information to obtain visual feature representation, including:

[0020] Based on the gated multilayer perceptron layer, the target matrix is ​​used in combination with the gating mechanism to perform linear mapping to obtain a linear mapping output, and the expression of the linear mapping output is:

[0021] f W, (Z) = WZ + b,

[0022]

[0023] Among them, f W, (Z) is the linear mapping output, W is the mapping parameter of spatial interaction, is the visual representation formed by the target matrix, b is the bias term, σ is the activation function, U is the linear transformation matrix, and Z is the input vector;

[0024] Based on the linear mapping output, a spatial gating unit module is used to generate a visual feature representation, and the expression of the visual feature representation is:

[0025] O=Z⊙f W, (Z))*V,

[0026] Among them, O is the visual feature representation, Z is the input vector, and f W, (Z) is the linear mapping output, and V is the linear mapping that reflects the spatial dependency.

[0027] Optionally, constructing a spatiotemporal graph based on the visual feature representation includes:

[0028] The objects in each frame in the visual feature representation are taken as nodes, the relative relationship between the nodes is calculated, the relative interaction relationship between the objects in each frame and their neighbors is calculated based on the relative relationship between the nodes, and a spatial graph is constructed based on the relative interaction relationship. The calculation expression of the relative relationship between the nodes is:

[0029]

[0030] in, is the relative relationship between the i-th node and the j-th node, W a is the parameter used to model the absolute relationship, W r is the parameter used for relative relationship reasoning, is the i′th node in the t′th frame, is the neighbor set of the ith node in the tth frame, is the i-th node in the t-th frame;

[0031] The expression of the relative interaction relationship between the object and its neighbors is:

[0032]

[0033] in, is the relative interaction relationship between the object and its neighbors, is the relative relationship between the i-th node and the j-th node in the t-th frame, N(i) is all the nodes that interact with the i-th node, The relative relationship between the i-th node and the m-th node in the t-th frame;

[0034] The link score between adjacent frames in the visual feature representation is calculated, and the link score is used as the relationship between adjacent frames. The relationship between adjacent frames is the edge of the time graph. The calculation expression of the link score is:

[0035]

[0036] Among them, s i, is the link score between the i-th object and the j-th object, ψ is the cosine similarity function between adjacent frames, is the appearance feature of the i-th object in the t-th frame, is the appearance feature of the jth object in the t+1th frame, is the motion feature of the i-th object in the t-th frame, The motion features of the jth object in the t+1th frame, λ is a hyperparameter, and IoU is the intersection over union ratio;

[0037] The node connection weight matrix is ​​calculated based on the appearance features and motion features after projection processing in the visual feature representation, and the edge connectivity is calculated based on the node connection weight matrix. The time graph is constructed based on the edge connectivity and the edges of the time graph. The expression of the node connection weight matrix is:

[0038]

[0039] in, is the node connection weight matrix, i and j are the node numbers, λ1 and λ2 are scaling factors, is the appearance feature of the i-th node after projection processing, T is the time, is the appearance feature of the j-th node after projection processing, is the motion feature of the i-th node after projection processing, is the motion feature of the jth node after projection processing, and L is the number of frames;

[0040] The expression of the edge connectivity is:

[0041]

[0042] Among them, R t is the edge connectivity, σ is the activation function, is the node connection weight matrix, and is the linear change of the connection weight matrix of different nodes, is the appearance feature of the i-th node after projection processing, T is the time, is the appearance feature of the j-th node after projection processing, is the motion feature of the i-th node after projection processing, is the motion feature of the jth node after projection processing.

[0043] Optionally, performing spatial graph convolution processing on the spatial graph to obtain a target spatial graph includes:

[0044] Based on the attention mechanism, the spatial graph is subjected to spatial graph convolution processing to obtain a target spatial graph. The expression of the spatial graph convolution processing is:

[0045]

[0046]

[0047] in, is the output of the spatial graph convolution processing, ReLU is the activation function, MHSA is multi-head self-attention, H is the number of attention heads, and W () is the graph weight matrix of the u-th layer, u is the number of layers, is the output node representation of the u-1th layer, R t is the edge connectivity, l is the frame length, n is the number of clips, and I is the convolution dimension.

[0048] Optionally, performing time dynamic mining on the time graph to obtain a target time graph includes:

[0049] Performing node feature processing on the time graph based on a node transformer to obtain a time graph after node feature processing;

[0050] Based on the edge transformer, edge feature processing is performed on the time graph after node feature processing to obtain the target time graph.

[0051] Optionally, generating word-level question embeddings based on the preprocessed question text data, and constructing a question graph based on the word-level question embeddings, includes:

[0052] The preprocessed question text data is word-embedded based on the bidirectional encoding representation transformer model to obtain word-level question embedding;

[0053] The attention weight of each word is calculated based on the natural language processing toolkit using the word-level question embedding, and the target word-level question embedding is generated based on the attention weight. The calculation expression of the attention weight is:

[0054] A w =softmax MLP(E s )),

[0055] Among them, A wis the attention weight, MLP is the multi-layer perceptron, E s is a word, softmax is a normalized exponential function,

[0056] The expression of the target word-level question embedding is:

[0057]

[0058] in, is the target word-level question embedding, is the attention weight of the corresponding word, Q w Embedding for word-level questions;

[0059] The association between question nodes is calculated, the target weight is calculated, and a question graph is constructed based on the attention weight, the target word-level question embedding, the association between question nodes and the target weight. The expression of the association between question nodes is:

[0060]

[0061] Among them, e ij is the correlation between the i-th problem node and the j-th problem node, λ is the scaling factor, The attention output of the i-th question word, is the attention output of the jth question word, and K is the number of question words.

[0062] Optionally, the step of generating a multi-level feature representation of a video using a hierarchical clustering algorithm based on the target spatiotemporal graph, and performing feature aggregation on the question graph and the multi-level feature representation of the video based on a dynamic graph attention module to obtain visual aggregation features and question text aggregation features includes:

[0063] Based on the target spatiotemporal graph, the attention mechanism is used to perform node aggregation to obtain a frame-level feature representation, and the expression of the frame-level feature representation is:

[0064]

[0065]

[0066] Among them, f F is the frame-level feature representation, N is the number of clips, is the learnable parameter W G The linear transformation of is the i-th node representation of the space-time graph, σ is the activation function;

[0067] The frame-level feature representation is projected based on the self-attention layer to obtain the projected frame-level feature representation. The expression of the projected frame-level feature representation is:

[0068] f′ F =φW m [MHSA(φW f (f O ))],

[0069] Among them, f′ F is the frame-level feature representation after projection, W m and W f is the linear change matrix, MHSA is the multi-head self-attention mechanism, φ is the linear transformation parameter, f O Identify the object;

[0070] The fragment-level feature representation is generated based on the maximum pooling, and the expression of the fragment-level feature representation is:

[0071] f C =MPool(f′ F ),

[0072] Among them, f C is the segment-level feature representation, f′ F is the frame-level feature representation after projection, M is the segment-based motion feature vector, and Pool is the pooling function;

[0073] The multi-level feature representation of the video is used as a video graph, and the target attention weight between the video graph and the question graph is calculated. The calculation expression of the target attention weight is:

[0074]

[0075] φ(u i ,v t )=W2·σ(W1·[u i ;v t ]+b1)+b2,

[0076] Among them, a i, is the attention weight between the i-th node in the question graph and the t-th node in the video graph, exp is the exponential function, φ is the score function, and u i is the i-th node in the problem graph, v t is the tth node in the video graph, W1, W2, b1 and b2 are the learnable parameters of the feedforward function, and σ is the nonlinear activation function;

[0077] Based on the target attention weight, feature aggregation is performed to obtain visual aggregation features and question text aggregation features. The expression of the visual aggregation feature is:

[0078]

[0079] in, is the visual aggregation feature, N is the number of clips, a i, is the attention weight between the i-th node in the question graph and the t-th node in the video graph, u i is the i-th node in the problem graph;

[0080] The expression of the question text aggregation feature is:

[0081]

[0082] in, is the aggregated feature of the question text, T is the total number of words, and a i, is the attention weight between the i-th node in the question graph and the t-th node in the video graph, v t is the tth node in the video graph.

[0083] Optionally, fusing the visual aggregate features and the question text aggregate features to obtain a joint feature representation, and generating a predicted answer based on an answer decoder using the joint feature representation, includes:

[0084] Based on the combination of visual aggregation features and video multi-level feature representation, a combined visual feature is obtained, and based on the combination of question text aggregation features and question graph, a combined question text feature is obtained. The expression of the combined visual feature is:

[0085]

[0086] Among them, h t To combine visual features, ReLU is the activation function, φW v is a linear variation with learnable parameters, v t is a multi-level feature representation of the video. is the visual aggregation feature, b v is the bias term;

[0087] The expression combining the question text features is:

[0088]

[0089] Among them, h i To combine the question text features, ReLU is the activation function, φW u is a linear variation with learnable parameters, u i For the problem diagram, is the aggregated feature of the question text, b u is the bias term;

[0090] The combined visual features and the combined question text features are fused to obtain a joint feature representation, and the expression of the joint feature representation is:

[0091]

[0092] f joint =Fusion(X),

[0093] Among them, h t To combine visual features, h i To combine the question text features, N is the number of clips, T is the total number of words, and f joint is the joint feature representation, and Fusion is the fusion function;

[0094] The joint feature representation is input into the answer decoder, and a prediction calculation is performed based on the answer decoder to obtain a predicted answer, wherein the expression of the prediction calculation is:

[0095] y′=σ(W o f joint +b o ),

[0096] s=softmax(W y′ y′+b),

[0097] Among them, y′ is the problem mapping, σ is the activation function, and f joint is the joint feature representation, W o and W y′ is the weight of the answer decoder, b o and b are the bias terms of the answer decoder, softmax is the normalized exponential function, and s is the mapping probability of the answer.

[0098] In addition, the present invention also provides a video question answering system based on spatiotemporal graph convolution, the system comprising:

[0099] Data acquisition and preprocessing module: used to acquire video data and question text data, and preprocess the video data and question text data to obtain preprocessed video data and question text data;

[0100] A target matrix generation module is used to extract visual features based on the preprocessed video data, obtain target visual features, and generate a target matrix based on the target visual features;

[0101] A visual feature representation acquisition module: used for performing spatial information aggregation using the target matrix based on a gated multi-layer perceptron layer to obtain a visual feature representation;

[0102] A target spatiotemporal graph generation module is used to construct a spatiotemporal graph based on the visual feature representation, wherein the spatiotemporal graph includes a time graph and a space graph, and performs spatial graph convolution processing on the space graph to obtain a target spatial graph, performs temporal dynamic mining on the time graph to obtain a target time graph, and generates a target spatiotemporal graph based on the target spatial graph and the target time graph;

[0103] A question graph construction module: used to generate word-level question embeddings based on the preprocessed question text data, and to construct a question graph based on the word-level question embeddings;

[0104] Feature aggregation module: used to generate a multi-level feature representation of the video based on the target spatiotemporal graph using a hierarchical aggregation algorithm, and perform feature aggregation on the question graph and the multi-level feature representation of the video based on the dynamic graph attention module to obtain visual aggregation features and question text aggregation features;

[0105] Answer prediction module: used to fuse the visual aggregate features and the question text aggregate features to obtain a joint feature representation, and generate a predicted answer based on the answer decoder using the joint feature representation.

[0106] In an embodiment of the present invention, a target matrix is ​​used to aggregate spatial information based on a gated multilayer perceptron layer to obtain a visual feature representation, thereby enhancing spatial information while maintaining a relatively low number of parameters. A spatiotemporal graph is constructed based on the visual feature representation, and a spatial graph convolution is performed on the spatial graph to obtain a target spatial graph, which can capture the spatial dependencies between visual features within a frame, thereby enriching the feature representation of the spatiotemporal graph, which helps to improve the model's understanding of video content and improve the performance of object space change video question-answering tasks. Temporal dynamics mining is performed on the time graph to obtain a target time graph, which can effectively capture the temporal dynamics within the video, thereby improving the performance on tasks involving temporal changes in video question-answering. Word-level question embedding is generated based on preprocessed question text data, and context-aware word-level question embedding can be obtained. A hierarchical aggregation algorithm is used based on the target spatiotemporal graph to generate a multi-level feature representation of the video, forming a complete frame-level interaction, effectively capturing multi-granular information in the video, enhancing the model's understanding of visual content, and improving performance in video question-answering tasks. Based on the dynamic graph attention module, the multi-level feature representations of the question graph and the video are aggregated to obtain visual aggregate features and question text aggregate features, which can focus on relevant visual objects according to the keywords in the question. The visual aggregate features and question text aggregate features are fused to obtain a joint feature representation, which contains richer information and promotes more accurate decision-making in subsequent steps. Based on the answer decoder, the joint feature representation is used to generate predicted answers, which can dynamically adjust the attention weights of visual objects, enhance the multimodal collaborative perception ability, and improve the reliability of video question answering. BRIEF DESCRIPTION OF THE DRAWINGS

[0107] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0108] Figure 1 is a flowchart of a video question answering method based on spatiotemporal graph convolution in an embodiment of the present invention;

[0109] Figure 2 It is a schematic diagram of the structural composition of a video question-answering system based on spatiotemporal graph convolution in an embodiment of the present invention. DETAILED DESCRIPTION

[0110] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0111] Embodiment 1

[0112] See also Figure 1 , Figure 1 is a flow chart of a video question answering method based on spatiotemporal graph convolution in an embodiment of the present invention, the method comprising:

[0113] S11: acquiring video data and question text data, and preprocessing the video data and question text data to obtain preprocessed video data and question text data;

[0114] In the specific implementation process of the present invention, video data and question text data are obtained, and the question text data may include text data through voice recognition. The video data and question text data are preprocessed, and the video data is denoised, that is, the noise in the video data is removed to obtain the denoised video data, and the question text data is denoised. Text denoising is to clean the text data, remove web page tags, non-semantic characters and garbled characters, etc., to obtain the denoised question text data, and the denoised video data and question text data are de-redundant. Since the multimodal data contains some repeated dimensional data, it is meaningless for the subsequent feature extraction and analysis processing. The redundant data of this part of the repeated dimension is removed, and the multimodal data is divided into several data blocks, and the number of data blocks in the data blocks is calculated. The hash value of the data is calculated according to the hash value. If the code distance is less than the preset threshold, it is redundant data. The redundant data is cleared to obtain the video data and question text data after redundancy removal. The video data and question text data after redundancy removal are normalized to eliminate the order of magnitude difference between each bit of the data to avoid excessive data level difference, which may cause excessive subsequent prediction errors. Normalized video data and normalized question text data are obtained. Key frames are extracted from the normalized video data to obtain preprocessed video data. After the question text data is normalized, the preprocessing is completed to obtain the preprocessed question text data.

[0115] S12: extracting visual features based on the preprocessed video data to obtain target visual features, and generating a target matrix based on the target visual features;

[0116] In the specific implementation process of the present invention, the visual feature extraction is performed based on the preprocessed video data to obtain the target visual features, and the target matrix is ​​generated based on the target visual features, including: extracting appearance features from the preprocessed video data based on a pretrained ResNet-152 model; splicing eight adjacent frames around each sampling frame in the preprocessed video data to obtain a spliced ​​frame, and extracting motion features from the spliced ​​frame based on a dual-stream expanded 3D convolutional network model; projecting the appearance features and motion features based on a linear feature transformation layer to obtain the appearance features and motion features after projection processing; splicing the appearance features and motion features after projection processing to obtain a target matrix.

[0117] Specifically, the preprocessed video data is divided into several clips, each of which contains several frames. In order to effectively represent the diverse information in the video, two different features are extracted: frame-based appearance features and segment-based motion features. The appearance features of the preprocessed video data are extracted based on the pre-trained ResNet-152 model. The ResNet-152 model is a deep residual network with powerful image classification capabilities. The eight adjacent frames around each sampling frame in the preprocessed video data are spliced ​​to obtain a spliced ​​frame, and motion features are extracted from the spliced ​​frame based on the two-stream inflated 3D convolutional network (Two-Stream Inflated 3D ConvNets, I3D) model. The appearance features and motion features are projected based on the linear feature transformation layer, and the appearance features and motion features are projected into a multi-dimensional feature space through the linear feature transformation layer to obtain the appearance features and motion features after projection processing. The appearance features and motion features after projection processing are spliced ​​to obtain a target matrix, which can also be called a visual representation, and its expression is:

[0118]

[0119] in, is the target matrix, is the appearance feature after projection processing, is the motion feature after projection processing, L is the number of video frames, and d′ is the dimension of the projection space.

[0120] S13: performing spatial information aggregation using the target matrix based on a gated multilayer perceptron layer to obtain a visual feature representation;

[0121] In the specific implementation process of the present invention, the gated multilayer perceptron layer uses the target matrix to aggregate spatial information to obtain visual feature representation, including: based on the gated multilayer perceptron layer, the target matrix is ​​used in combination with a gating mechanism to perform linear mapping to obtain a linear mapping output, and the expression of the linear mapping output is:

[0122] f W, (Z) = WZ + b,

[0123]

[0124] Among them, f W, (Z) is the linear mapping output, W is the mapping parameter of spatial interaction, is the visual representation formed by the target matrix, b is the bias term, σ is the activation function, U is the linear transformation matrix, and Z is the input vector;

[0125] Based on the linear mapping output, a spatial gating unit module is used to generate a visual feature representation, and the expression of the visual feature representation is:

[0126] O=Z⊙f W, (Z))*V,

[0127] Among them, O is the visual feature representation, Z is the input vector, and f W, (Z) is the linear mapping output, and V is the linear mapping that reflects the spatial dependency.

[0128] Specifically, the target matrix is ​​input into the gated multilayer perceptron (GMLP) layer. In the GMLP layer, the aggregation of spatial information can be achieved by using static parameterized channel projection, spatial mapping and gate mechanism at the same time. In the spatial dimension, the target matrix is ​​linearly mapped through the GMLP layer to obtain a linear mapping output, and the expression of the linear mapping output is:

[0129] f W, (Z) = WZ + b,

[0130]

[0131] Among them, f W, (Z) is the linear mapping output, W is the mapping parameter of spatial interaction, is the visual representation formed by the target matrix, b is the bias term, σ is the activation function, U is the linear transformation matrix, and Z is the input vector; based on the linear mapping output, the spatial gating unit module is used to generate a visual feature representation, and the expression of the visual feature representation is:

[0132] O=Z⊙f W, (Z))*V,

[0133] Among them, O is the visual feature representation, Z is the input vector, and f W, (Z) is the linear mapping output, V is the linear mapping reflecting the spatial dependency, and the spatial dependency can be captured by using the spatial gating unit module. Here, in order to ensure stable training, W and b can be initialized to 0 and 1, respectively, so that the gMLP is similar to a feedforward neural network in the initial stage of training and gradually learns the spatial information between features during the training process.

[0134] S14: constructing a spatiotemporal graph based on the visual feature representation, the spatiotemporal graph including a time graph and a space graph, performing spatial graph convolution processing on the space graph to obtain a target spatial graph, performing temporal dynamic mining on the time graph to obtain a target time graph, and generating a target spatiotemporal graph based on the target spatial graph and the target time graph;

[0135] In the specific implementation process of the present invention, the construction of the spatiotemporal graph based on the visual feature representation includes: taking the object of each frame in the visual feature representation as a node, calculating the relative relationship between the nodes, calculating the relative interaction relationship between the object of each frame and the neighbor based on the relative relationship between the nodes, and constructing a spatial graph based on the relative interaction relationship. The calculation expression of the relative relationship between the nodes is:

[0136]

[0137] in, is the relative relationship between the i-th node and the j-th node, W a is the parameter used to model the absolute relationship, W r is the parameter used for relative relationship reasoning, is the i′th node in the t′th frame, is the neighbor set of the ith node in the tth frame, is the i-th node in the t-th frame;

[0138] The expression of the relative interaction relationship between the object and its neighbors is:

[0139]

[0140] in, is the relative interaction relationship between the object and its neighbors, is the relative relationship between the i-th node and the j-th node in the t-th frame, N(i) is all the nodes that interact with the i-th node, The relative relationship between the i-th node and the m-th node in the t-th frame;

[0141] The link score between adjacent frames in the visual feature representation is calculated, and the link score is used as the relationship between adjacent frames. The relationship between adjacent frames is the edge of the time graph. The calculation expression of the link score is:

[0142]

[0143] Among them, s i, is the link score between the i-th object and the j-th object, ψ is the cosine similarity function between adjacent frames, is the appearance feature of the i-th object in the t-th frame, is the appearance feature of the jth object in the t+1th frame, is the motion feature of the i-th object in the t-th frame, The motion features of the jth object in the t+1th frame, λ is a hyperparameter, and IoU is the intersection over union ratio;

[0144] The node connection weight matrix is ​​calculated based on the appearance features and motion features after projection processing in the visual feature representation, and the edge connectivity is calculated based on the node connection weight matrix. The time graph is constructed based on the edge connectivity and the edges of the time graph. The expression of the node connection weight matrix is:

[0145]

[0146] in, is the node connection weight matrix, i and j are the node numbers, λ1 and λ2 are scaling factors, is the appearance feature of the i-th node after projection processing, T is the time, is the appearance feature of the j-th node after projection processing, is the motion feature of the i-th node after projection processing, is the motion feature of the jth node after projection processing, and L is the number of frames;

[0147] The expression of the edge connectivity is:

[0148]

[0149] Among them, R t is the edge connectivity, σ is the activation function, is the node connection weight matrix, and is the linear change of the connection weight matrix of different nodes, is the appearance feature of the i-th node after projection processing, T is the time, is the appearance feature of the j-th node after projection processing, is the motion feature of the i-th node after projection processing, is the motion feature of the jth node after projection processing.

[0150] Further, performing spatial graph convolution processing on the spatial graph to obtain a target spatial graph includes: performing spatial graph convolution processing on the spatial graph based on an attention mechanism to obtain a target spatial graph, and the expression of the spatial graph convolution processing is:

[0151]

[0152]

[0153] in, is the output of the spatial graph convolution processing, ReLU is the activation function, MHSA is multi-head self-attention, H is the number of attention heads, and W () is the graph weight matrix of the u-th layer, u is the number of layers, is the output node representation of the u-1th layer, Rt is the edge connectivity, l is the frame length, n is the number of clips, and I is the convolution dimension.

[0154] Furthermore, the time graph is subjected to time dynamic mining to obtain a target time graph, including: performing node feature processing on the time graph based on a node Transformer to obtain a time graph after node feature processing; performing edge feature processing on the time graph after node feature processing based on an edge Transformer to obtain a target time graph.

[0155] Specifically, most existing video question-answering methods conceptualize the video as a comprehensive spatiotemporal graph that covers the representation of all detected objects and uses a simple similarity measure for node aggregation, which usually ignores the interactive relationship between objects. Therefore, in the present invention, interactive relationship modeling is integrated into the graph framework, and the visual feature representation is processed as a spatiotemporal sequence. In order to reduce the computational complexity, the spatial graph and the temporal graph are independently constructed. In the spatial graph, the object representation of each frame is used as a node, and it is assumed that the interaction between different objects can be represented by the dynamic flow of information, and their relationship is determined according to the position change and relative position of the objects between different frames. The objects in each frame in the visual feature representation are taken as nodes, and the relative relationships between nodes are calculated. Based on the relative relationships between nodes, the relative interactive relationship between the objects and neighbors in each frame is calculated, and a spatial graph is constructed based on the relative interactive relationship. The calculation expression of the relative relationship between the nodes is:

[0156]

[0157] in, is the relative relationship between the i-th node and the j-th node, W a is the parameter used to model the absolute relationship, W r is the parameter used for relative relationship reasoning, is the i′th node in the t′th frame, is the neighbor set of the ith node in the tth frame, is the i-th node in the t-th frame;

[0158] The expression of the relative interaction relationship between the object and its neighbors is:

[0159]

[0160] in, is the relative interaction relationship between the object and its neighbors, is the relative relationship between the i-th node and the j-th node in the t-th frame, N(i) is all the nodes that interact with the i-th node, The relative relationship between the i-th node and the m-th node in the t-th frame. Based on the calculated relationship, the spatial graph is constructed by connecting nodes and edges.

[0161] Each frame in the visual feature representation is taken as a node, and the relationship between adjacent frames is represented as the edge of the graph. The link score between adjacent frames in the visual feature representation is calculated, and the link score is taken as the relationship between adjacent frames. The relationship between adjacent frames is the edge of the time graph. The calculation expression of the link score is:

[0162]

[0163] Among them, s i, is the link score between the i-th object and the j-th object, ψ is the cosine similarity function between adjacent frames, is the appearance feature of the i-th object in the t-th frame, is the appearance feature of the jth object in the t+1th frame, is the motion feature of the i-th object in the t-th frame, The motion features of the jth object in the t+1th frame, λ is a hyperparameter, and IoU is the intersection over union ratio; in order to effectively balance the contribution of cosine similarity and IoU, the hyperparameter of the link score can be set to 1, and the link score is maximized through a greedy algorithm to connect the same objects on different frames.

[0164] Each node is intricately connected to every other node through edges. These edges represent the relationships between nodes, which are quantified using vectors containing relevant features of each node. The node connection weight matrix is ​​calculated based on the appearance features and motion features after projection processing in the visual feature representation, and the edge connectivity is calculated based on the node connection weight matrix. The time graph is constructed based on the edge connectivity and the edges of the time graph. The expression of the node connection weight matrix is:

[0165]

[0166] in, is the node connection weight matrix, i and j are the node numbers, λ1 and λ2 are scaling factors, is the appearance feature of the i-th node after projection processing, T is the time, is the appearance feature of the j-th node after projection processing, is the motion feature of the i-th node after projection processing, is the motion feature of the jth node after projection processing, and L is the number of frames;

[0167] The expression of the edge connectivity is:

[0168]

[0169] Among them, R t is the edge connectivity, σ is the activation function, is the node connection weight matrix, and is the linear change of the connection weight matrix of different nodes, is the appearance feature of the i-th node after projection processing, T is the time, is the appearance feature of the j-th node after projection processing, is the motion feature of the i-th node after projection processing, The motion feature after projection processing of the jth node is used to capture the asymmetric nature of the interaction between the subject and the object in the real world, as well as the complementary characteristics of appearance and motion features. Based on the relationship between nodes and edges, a temporal graph is constructed.

[0170] The constructed time graph and spatial graph need to be further processed separately. In the spatial graph, in order to reason about the spatial interactions between objects, the importance of neighboring nodes in spatial dependency learning is emphasized. To achieve this, the attention mechanism is integrated in the node aggregation, and graph convolution is directly applied in the spatial domain to reduce computational complexity. Several layers of spatial graph attention convolution are used on the spatial graph, that is, the spatial graph is convolved based on the attention mechanism to obtain the target spatial graph. The expression of the spatial graph convolution is:

[0171]

[0172]

[0173] in, is the output of the spatial graph convolution processing, ReLU is the activation function, MHSA is multi-head self-attention, H is the number of attention heads, and W () is the graph weight matrix of the u-th layer, u is the number of layers, is the output node representation of the u-1th layer, R t is edge connectivity, l is the frame length, n is the number of clips, and I is the convolution dimension. The role of spatial graph convolution is to use the spatial relationship within the video frame to learn and represent the features of the visual content in the video. By applying the spatial graph convolution operation, it can capture the spatial dependencies between the visual features within the frame, thereby enriching the feature representation of the spatiotemporal graph, which helps to improve the model's understanding of the video content and improve the performance of the object space change video question answering task.

[0174] In the video question answering task, the unique temporal information of the video is a key challenge in the video question answering task. In order to overcome this challenge and effectively capture the temporal information in the video clips, the graph Transformer unit is applied to learn the long-term temporal dependencies between objects. The graph Transformer unit consists of a node transformer and an edge transformer, and accepts a set of temporal graph sequences as input to mine the temporal dynamics in the video. Based on the node transformer, the node feature processing is performed on the time graph to obtain the time graph after the node feature processing. The core component of the node transformer is the graph multi-head attention module, which is specially designed for structured input data. Different from the traditional attention mechanism, the traditional attention mechanism requires vectorized input, while the graph input is essentially non-vectorized structured data. Therefore, a specific dynamic range expansion message passing neural network (dyMPN) is required to extract vectors from the nodes of the graph as queries, keys and values, and then input them into the attention block. For dyMPN, a random strategy is adopted to select the number of jumps in the training process. The randomized message passing scheme enables a random receptive field for each node in the graph convolution operation, which has better generalization. The expression of the time graph after the node feature processing is:

[0175]

[0176] Among them, F′ o is the time graph after node feature processing, N is the number of clips, Trans is the function used by the node Transformer, is the sequence of feature representations corresponding to the i-th object in all frames. In the node Transformer, we use several self-attention blocks to enhance the object representation of the node by aggregating information from other nodes in all adjacent frames of the same object.

[0177] In order to better capture the motion changes between video frames and the evolution of visual content, thereby achieving improved causality and temporal reasoning, edge transformers are used on the updated edges. That is, edge transformers are used to process the time graph after node feature processing to obtain the target time graph. The expression of the target time graph is:

[0178] R′=ETrans(R s ),

[0179] Among them, R′ is the target time graph, E is the transmission efficiency, Trans is the edge Transformer function, Rs The edge transformer is designed to process edge features in the temporal graph, which promotes the learning of temporal relationships between objects and generates a target spatiotemporal graph from the target spatial graph and the target temporal graph. By integrating node and edge transformers, the model can effectively capture the temporal dynamics within the video, thereby improving the performance on tasks involving temporal changes in video question answering.

[0180] S15: Generate word-level question embeddings based on the preprocessed question text data, and construct a question graph based on the word-level question embeddings;

[0181] In the specific implementation process of the present invention, the word-level question embedding is generated based on the preprocessed question text data, and the question graph is constructed based on the word-level question embedding, including: embedding the preprocessed question text data words based on the bidirectional encoding representation converter model to obtain the word-level question embedding; calculating the attention weight of each word based on the natural language processing toolkit using the word-level question embedding, and generating the target word-level question embedding based on the attention weight, and the calculation expression of the attention weight is:

[0182] A w =softmax MLP(E s )),

[0183] Among them, A w is the attention weight, MLP is the multi-layer perceptron, E s is a word, softmax is a normalized exponential function,

[0184] The expression of the target word-level question embedding is:

[0185]

[0186] in, is the target word-level question embedding, is the attention weight of the corresponding word, Q w Embedding for word-level questions;

[0187] The association between question nodes is calculated, the target weight is calculated, and a question graph is constructed based on the attention weight, the target word-level question embedding, the association between question nodes and the target weight. The expression of the association between question nodes is:

[0188]

[0189] Among them, e ij is the correlation between the i-th problem node and the j-th problem node, λ is the scaling factor, The attention output of the i-th question word, is the attention output of the jth question word, and K is the number of question words.

[0190] Specifically, we embed the preprocessed question text data into words based on the Bidirectional Encoder Representations from Transformers (BERT) model to obtain word-level question embeddings. We use the BERT model’s ability to bidirectionally encode contexts to establish context dependencies and generate context-aware word-level question embeddings. Where L represents the number of words in the question, C w Represents the dimension of question embedding. The attention weight of each word is calculated based on the natural language processing toolkit (CoreNLP) using the word-level question embedding. CoreNLP is a natural language processing toolkit that uses CoreNLP to identify the semantic dependency between questions and candidate answers, uses the attention mechanism to focus on the characteristics of the question, assigns weights according to the part of speech, calculates the attention weight of each word, and generates the target word-level question embedding based on the attention weight. The calculation expression of the attention weight is:

[0191] A w =softmax MLP(E s )),

[0192] Among them, A w is the attention weight, MLP is the multi-layer perceptron, E s is a word, softmax is a normalized exponential function,

[0193] The expression of the target word-level question embedding is:

[0194]

[0195] in, is the target word-level question embedding, is the attention weight of the corresponding word, Q w Embedding for word-level questions;

[0196] The association between the question nodes is calculated, the target weight is calculated, each word is designated as a node, and a graph representation of the question is constructed, that is, a question graph is constructed based on the attention weight, the target word-level question embedding, the association between the question nodes and the target weight. The expression of the association between the question nodes is:

[0197]

[0198] Among them, e ijis the correlation between the i-th problem node and the j-th problem node, λ is the scaling factor, The attention output of the i-th question word, is the attention output of the jth question word, and K is the number of question words.

[0199] S16: Generate a multi-level feature representation of the video based on the target spatiotemporal graph using a hierarchical clustering algorithm, and perform feature aggregation on the question graph and the multi-level feature representation of the video based on a dynamic graph attention module to obtain visual aggregation features and question text aggregation features;

[0200] In the specific implementation process of the present invention, the hierarchical aggregation algorithm is used to generate a multi-level feature representation of the video based on the target spatiotemporal graph, and the problem graph and the multi-level feature representation of the video are feature aggregated based on the dynamic graph attention module to obtain visual aggregation features and question text aggregation features, including: based on the target spatiotemporal graph, the attention mechanism is used to perform node aggregation to obtain a frame-level feature representation, and the expression of the frame-level feature representation is:

[0201]

[0202]

[0203] Among them, f F is the frame-level feature representation, N is the number of clips, is the learnable parameter W G The linear transformation of is the i-th node representation of the space-time graph, σ is the activation function;

[0204] The frame-level feature representation is projected based on the self-attention layer to obtain the projected frame-level feature representation. The expression of the projected frame-level feature representation is:

[0205] f′ F =φW m [MHSA(φW f (f O ))],

[0206] Among them, f′ F is the frame-level feature representation after projection, W m and W f is the linear change matrix, MHSA is the multi-head self-attention mechanism, φ is the linear transformation parameter, f O Identify the object;

[0207] The fragment-level feature representation is generated based on the maximum pooling, and the expression of the fragment-level feature representation is:

[0208] f C =MPool(f′F ),

[0209] Among them, f C is the segment-level feature representation, f′ F is the frame-level feature representation after projection, M is the segment-based motion feature vector, and Pool is the pooling function;

[0210] The multi-level feature representation of the video is used as a video graph, and the target attention weight between the video graph and the question graph is calculated. The calculation expression of the target attention weight is:

[0211]

[0212] φ(u i ,v t )=W2·σ(W1·[u i ;v t ]+b1)+b2,

[0213] Among them, a i, is the attention weight between the i-th node in the question graph and the t-th node in the video graph, exp is the exponential function, φ is the score function, and u i is the i-th node in the problem graph, v t is the tth node in the video graph, W1, W2, b1 and b2 are the learnable parameters of the feedforward function, and σ is the nonlinear activation function;

[0214] Based on the target attention weight, feature aggregation is performed to obtain visual aggregation features and question text aggregation features. The expression of the visual aggregation feature is:

[0215]

[0216] in, is the visual aggregation feature, N is the number of clips, a i, is the attention weight between the i-th node in the question graph and the t-th node in the video graph, u i is the i-th node in the problem graph;

[0217] The expression of the question text aggregation feature is:

[0218]

[0219] in, is the aggregated feature of the question text, T is the total number of words, and a i, is the attention weight between the i-th node in the question graph and the t-th node in the video graph, v t is the tth node in the video graph.

[0220] Specifically, in the video question-answering task, most methods process the video as a frame sequence, and lack the hierarchical cognition of the visual object itself. The visual elements are not continuous in the semantic space, but present a hierarchical distribution form. In order to align with the multi-granularity information in the language, the visual features in the video are roughly modeled as a multi-level structure at the object level, frame level, and clip level. The current node representation has explicitly considered the temporal and spatial interactions between objects. This interaction is basically individualized and does not form a complete frame-level interaction. In order to aggregate visual features and interactions into high-level video elements, object-level visual features and interactions are aggregated through hierarchical aggregation to generate multi-level features. Based on the target spatiotemporal graph, the attention mechanism is used to perform node aggregation to obtain a frame-level feature representation. The expression of the frame-level feature representation is:

[0221]

[0222]

[0223] Among them, f F is the frame-level feature representation, N is the number of clips, is the learnable parameter W G The linear transformation of is the i-th node representation of the space-time graph, σ is the activation function;

[0224] Although the frame-level feature representation captures the interaction between local objects, it may ignore the global context of the frame, especially considering that only a few objects are retained, which may not contain all objects of interest in the frame. In order to solve this limitation, the object identification is enhanced by the self-attention layer and projected into the linear space, that is, the frame-level feature representation is projected based on the self-attention layer to obtain the projected frame-level feature representation. The expression of the projected frame-level feature representation is:

[0225] f′ F =φW m [MHSA(φW f (f O ))],

[0226] Among them, f′ F is the frame-level feature representation after projection, W m and W f is the linear change matrix, MHSA is the multi-head self-attention mechanism, φ is the linear transformation parameter, f O Identify the object;

[0227] The fragment-level feature representation is generated based on the maximum pooling, and the expression of the fragment-level feature representation is:

[0228] f C =MPool(f′ F ),

[0229] Among them, f C is the segment-level feature representation, f′ F is the projected frame-level feature representation, M is the segment-based motion feature vector, and Pool is the pooling function; this hierarchical aggregation mechanism effectively captures the multi-granularity information in the video, enhances the model’s understanding of the visual content, and improves the performance in video question-answering tasks.

[0230] The word-level question embedding in the question graph is aggregated into a sentence-level representation using a bidirectional long short-term memory network, which helps to fuse features with segment-level features. The multi-level feature representation of the video is used as a video graph, and the target attention weight between the video graph and the question graph is calculated. The calculation expression of the target attention weight is:

[0231]

[0232] φ(u i ,v t )=W2·σ(W1·[u i ;v t ]+b1)+b2,

[0233] Among them, a i, is the attention weight between the i-th node in the question graph and the t-th node in the video graph, exp is the exponential function, φ is the score function, and u i is the i-th node in the problem graph, v t is the tth node in the video graph, W1, W2, b1 and b2 are learnable parameters of the feedforward function, and σ is a nonlinear activation function; the target attention weight is used to aggregate information between nodes in different layers of the graph, and can focus on relevant visual objects according to the keywords in the question. In order to focus on relevant visual objects according to the keywords in the question, feature aggregation is performed based on the target attention weight to obtain visual aggregation features and question text aggregation features. The expression of the visual aggregation feature is:

[0234]

[0235] in, is the visual aggregation feature, N is the number of clips, a i, is the attention weight between the i-th node in the question graph and the t-th node in the video graph, u i is the i-th node in the problem graph;

[0236] The expression of the question text aggregation feature is:

[0237]

[0238] in, is the aggregated feature of the question text, T is the total number of words, and a i, is the attention weight between the i-th node in the question graph and the t-th node in the video graph, v t is the tth node in the video graph.

[0239] S17: Fusing the visual aggregate features and the question text aggregate features to obtain a joint feature representation, and generating a predicted answer based on the answer decoder using the joint feature representation.

[0240] In the specific implementation process of the present invention, the visual aggregate features and the question text aggregate features are fused to obtain a joint feature representation, and the joint feature representation is used to generate a predicted answer based on the answer decoder, including: combining the visual aggregate features with the multi-level feature representation of the video to obtain a combined visual feature, and combining the question text aggregate features with the question graph to obtain a combined question text feature, and the expression of the combined visual feature is:

[0241]

[0242] Among them, h t To combine visual features, ReLU is the activation function, φW v is a linear variation with learnable parameters, v t is a multi-level feature representation of the video. is the visual aggregation feature, b v is the bias term;

[0243] The expression combining the question text features is:

[0244]

[0245] Among them, h i To combine the question text features, ReLU is the activation function, φW u is a linear variation with learnable parameters, u i For the problem diagram, is the aggregated feature of the question text, b u is the bias term;

[0246] The combined visual features and the combined question text features are fused to obtain a joint feature representation, and the expression of the joint feature representation is:

[0247]

[0248] f joint=Fusion(X),

[0249] Among them, h t To combine visual features, h i To combine the question text features, N is the number of clips, T is the total number of words, and f joint is the joint feature representation, and Fusion is the fusion function;

[0250] The joint feature representation is input into the answer decoder, and a prediction calculation is performed based on the answer decoder to obtain a predicted answer, wherein the expression of the prediction calculation is:

[0251] y′=σ(W o f joint +b o ),

[0252] s=softmax(W y′ y′+b),

[0253] Among them, y′ is the problem mapping, σ is the activation function, and f joint is the joint feature representation, W o and W y′ is the weight of the answer decoder, b o and b are the bias terms of the answer decoder, softmax is the normalized exponential function, and s is the mapping probability of the answer.

[0254] Specifically, the visual aggregation feature is combined with the multi-level feature representation of the video to obtain the combined visual feature, and the question text aggregation feature is combined with the question graph to obtain the combined question text feature. The expression of the combined visual feature is:

[0255]

[0256] Among them, h t To combine visual features, ReLU is the activation function, φW v is a linear variation with learnable parameters, v t is a multi-level feature representation of the video. is the visual aggregation feature, b v is the bias term;

[0257] The expression combining the question text features is:

[0258]

[0259] Among them, h i To combine the question text features, ReLU is the activation function, φW u is a linear variation with learnable parameters, u i For the problem diagram, is the aggregated feature of the question text, b u It is a bias term; it can contain richer information and promote more accurate decision making in subsequent steps

[0260] The combined visual features and the combined question text features are fused to obtain a joint feature representation, and the expression of the joint feature representation is:

[0261]

[0262] f joint =Fusion(X),

[0263] Among them, h t To combine visual features, h i To combine the question text features, N is the number of clips, T is the total number of words, and f joint is the joint feature representation, and Fusion is the fusion function;

[0264] The joint feature representation is input into the answer decoder, and a prediction calculation is performed based on the answer decoder to obtain a predicted answer, wherein the expression of the prediction calculation is:

[0265] y′=σ(W o f joint +b o ),

[0266] s=softmax(W y′ y′+b),

[0267] Among them, y′ is the problem mapping, σ is the activation function, and f joint is the joint feature representation, W o and W y′ is the weight of the answer decoder, b o and b are the bias terms of the answer decoder, softmax is the normalized exponential function, and s is the mapping probability of the answer.

[0268] Furthermore, for multiple-choice question types, multiple answer candidates are processed to generate a set of visual representations corresponding to each candidate, using an objective function to solve such problems, which is:

[0269]

[0270] in, is the answer inferred in the answer space, For the answer space, is the mapping function, q is the related question, is the video, and a is the answer set.

[0271] In an embodiment of the present invention, a target matrix is ​​used to aggregate spatial information based on a gated multilayer perceptron gMLP layer to obtain a visual feature representation, thereby enhancing spatial information while maintaining a relatively low number of parameters. A spatiotemporal graph is constructed based on the visual feature representation, and a spatial graph convolution process is performed on the spatial graph to obtain a target spatial graph, which can capture the spatial dependencies between visual features within a frame, thereby enriching the feature representation of the spatiotemporal graph, which helps to improve the model's understanding of video content and improve the performance of object space change video question-answering tasks. Temporal dynamics mining is performed on the time graph to obtain a target time graph, which can effectively capture the temporal dynamics within the video, thereby improving the performance on tasks involving temporal changes in video question-answering. Word-level question embedding is generated based on preprocessed question text data, and context-aware word-level question embedding can be obtained. A hierarchical aggregation algorithm is used to generate a multi-level feature representation of the video based on the target spatiotemporal graph, forming a complete frame-level interaction, effectively capturing multi-granular information in the video, enhancing the model's understanding of visual content, and improving performance in video question-answering tasks. Based on the dynamic graph attention module, the multi-level feature representations of the question graph and the video are aggregated to obtain visual aggregate features and question text aggregate features, which can focus on relevant visual objects according to the keywords in the question. The visual aggregate features and question text aggregate features are fused to obtain a joint feature representation, which contains richer information and promotes more accurate decision-making in subsequent steps. Based on the answer decoder, the joint feature representation is used to generate predicted answers, which can dynamically adjust the attention weights of visual objects, enhance the multimodal collaborative perception ability, and improve the reliability of video question answering.

[0272] Embodiment 2

[0273] See also Figure 2 , Figure 2 It is a schematic diagram of the structural composition of a video question-answering system based on spatiotemporal graph convolution in an embodiment of the present invention.

[0274] like Figure 2 As shown, a video question answering system based on spatiotemporal graph convolution, the system comprising:

[0275] Data acquisition and preprocessing module 21: used to acquire video data and question text data, and preprocess the video data and question text data to obtain preprocessed video data and question text data;

[0276] The target matrix generation module 22 is used to extract visual features based on the preprocessed video data, obtain target visual features, and generate a target matrix based on the target visual features;

[0277] Visual feature representation acquisition module 23: used to aggregate spatial information using the target matrix based on the gated multi-layer perceptron layer to obtain visual feature representation;

[0278] A target spatiotemporal graph generation module 24 is used to construct a spatiotemporal graph based on the visual feature representation, wherein the spatiotemporal graph includes a time graph and a space graph, and performs a spatial graph convolution process on the space graph to obtain a target spatial graph, performs temporal dynamic mining on the time graph to obtain a target time graph, and generates a target spatiotemporal graph based on the target spatial graph and the target time graph;

[0279] A question graph construction module 25 is used to generate word-level question embeddings based on the preprocessed question text data, and to construct a question graph based on the word-level question embeddings;

[0280] Feature aggregation module 26: used to generate a multi-level feature representation of the video based on the target spatiotemporal graph using a hierarchical aggregation algorithm, and perform feature aggregation on the question graph and the multi-level feature representation of the video based on the dynamic graph attention module to obtain visual aggregation features and question text aggregation features;

[0281] Answer prediction module 27: used to fuse the visual aggregate features and the question text aggregate features to obtain a joint feature representation, and generate a predicted answer based on the answer decoder using the joint feature representation.

[0282] In the specific implementation process of the present invention, the specific implementation method of the system item can refer to the implementation method of the above-mentioned method item, which will not be repeated here.

[0283] In an embodiment of the present invention, a target matrix is ​​used to aggregate spatial information based on a gated multilayer perceptron gMLP layer to obtain a visual feature representation, thereby enhancing spatial information while maintaining a relatively low number of parameters. A spatiotemporal graph is constructed based on the visual feature representation, and a spatial graph convolution process is performed on the spatial graph to obtain a target spatial graph, which can capture the spatial dependencies between visual features within a frame, thereby enriching the feature representation of the spatiotemporal graph, which helps to improve the model's understanding of video content and improve the performance of object space change video question-answering tasks. Temporal dynamics mining is performed on the time graph to obtain a target time graph, which can effectively capture the temporal dynamics within the video, thereby improving the performance on tasks involving temporal changes in video question-answering. Word-level question embedding is generated based on preprocessed question text data, and context-aware word-level question embedding can be obtained. A hierarchical aggregation algorithm is used to generate a multi-level feature representation of the video based on the target spatiotemporal graph, forming a complete frame-level interaction, effectively capturing multi-granular information in the video, enhancing the model's understanding of visual content, and improving performance in video question-answering tasks. Based on the dynamic graph attention module, the multi-level feature representations of the question graph and the video are aggregated to obtain visual aggregate features and question text aggregate features, which can focus on relevant visual objects according to the keywords in the question. The visual aggregate features and question text aggregate features are fused to obtain a joint feature representation, which contains richer information and promotes more accurate decision-making in subsequent steps. Based on the answer decoder, the joint feature representation is used to generate predicted answers, which can dynamically adjust the attention weights of visual objects, enhance the multimodal collaborative perception ability, and improve the reliability of video question answering.

[0284] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0285] In addition, the above is a detailed introduction to a video question-answering method and system based on spatiotemporal graph convolution provided by an embodiment of the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A video question answering method based on spatiotemporal graph convolution, characterized in that: The method comprises: Acquire video data and question text data, and preprocess the video data and question text data to obtain preprocessed video data and question text data; Extract visual features based on the preprocessed video data to obtain target visual features, and generate a target matrix based on the target visual features; Based on the gated multi-layer perceptron layer, the target matrix is ​​used to aggregate spatial information to obtain visual feature representation; Based on the visual feature representation, a spatiotemporal graph is constructed, the spatiotemporal graph includes a time graph and a space graph, and the space graph is subjected to a spatial graph convolution process to obtain a target spatial graph, the time graph is subjected to temporal dynamic mining to obtain a target time graph, and a target spatiotemporal graph is generated based on the target spatial graph and the target time graph; Generate word-level question embeddings based on the preprocessed question text data, and construct a question graph based on the word-level question embeddings; Generate a multi-level feature representation of the video using a hierarchical clustering algorithm based on the target spatiotemporal graph, and perform feature aggregation on the question graph and the multi-level feature representation of the video based on a dynamic graph attention module to obtain visual aggregation features and question text aggregation features; The visual aggregate features and the question text aggregate features are fused to obtain a joint feature representation, and the joint feature representation is used to generate a predicted answer based on an answer decoder.

2. The video question answering method based on spatiotemporal graph convolution according to claim 1, characterized in that: The method of extracting visual features based on the preprocessed video data to obtain target visual features and generating a target matrix based on the target visual features includes: Extract appearance features from preprocessed video data based on the pre-trained ResNet-152 model; Splicing eight adjacent frames around each sampling frame in the preprocessed video data to obtain a spliced ​​frame, and extracting motion features from the spliced ​​frame based on a two-stream dilated 3D convolutional network model; Performing projection processing on the appearance features and motion features based on a linear feature transformation layer to obtain the appearance features and motion features after projection processing; The projected appearance features and motion features are concatenated to obtain the target matrix.

3. The video question answering method based on spatiotemporal graph convolution according to claim 1, characterized in that: The gated multi-layer perceptron layer uses the target matrix to aggregate spatial information to obtain visual feature representation, including: Based on the gated multilayer perceptron layer, the target matrix is ​​used in combination with the gating mechanism to perform linear mapping to obtain a linear mapping output, and the expression of the linear mapping output is: f W,b (Z)=WZ+b, Among them, f W，b (Z) is the linear mapping output, W is the mapping parameter of spatial interaction, is the visual representation formed by the target matrix, b is the bias term, σ is the activation function, U is the linear transformation matrix, and Z is the input vector; Based on the linear mapping output, a spatial gating unit module is used to generate a visual feature representation, and the expression of the visual feature representation is: O=(Z⊙f W,b (Z))*V, Among them, O is the visual feature representation, Z is the input vector, and f W,b (Z) is the linear mapping output, and V is the linear mapping that reflects the spatial dependency.

4. The video question answering method based on spatiotemporal graph convolution according to claim 1, characterized in that: The constructing a spatiotemporal graph based on the visual feature representation comprises: The objects in each frame in the visual feature representation are taken as nodes, the relative relationship between the nodes is calculated, the relative interaction relationship between the objects in each frame and their neighbors is calculated based on the relative relationship between the nodes, and a spatial graph is constructed based on the relative interaction relationship. The calculation expression of the relative relationship between the nodes is: in, is the relative relationship between the i-th node and the j-th node, W a is the parameter used to model the absolute relationship, W r is the parameter used for relative relationship reasoning, is the i′th node in the t′th frame, is the neighbor set of the ith node in the tth frame, is the i-th node in the t-th frame; The expression of the relative interaction relationship between the object and its neighbors is: in, is the relative interaction relationship between the object and its neighbors, is the relative relationship between the i-th node and the j-th node in the t-th frame, N(i) is all the nodes that interact with the i-th node, The relative relationship between the i-th node and the m-th node in the t-th frame; The link score between adjacent frames in the visual feature representation is calculated, and the link score is used as the relationship between adjacent frames. The relationship between adjacent frames is the edge of the time graph. The calculation expression of the link score is: Among them, s i,j is the link score between the i-th object and the j-th object, ψ is the cosine similarity function between adjacent frames, is the appearance feature of the i-th object in the t-th frame, is the appearance feature of the jth object in the t+1th frame, is the motion feature of the i-th object in the t-th frame, The motion features of the jth object in the t+1th frame, λ is a hyperparameter, and IoU is the intersection over union ratio; The node connection weight matrix is ​​calculated based on the appearance features and motion features after projection processing in the visual feature representation, and the edge connectivity is calculated based on the node connection weight matrix. The time graph is constructed based on the edge connectivity and the edges of the time graph. The expression of the node connection weight matrix is: in, is the node connection weight matrix, i and j are the node numbers, λ1 and λ2 are scaling factors, is the appearance feature of the i-th node after projection processing, T is the time, is the appearance feature of the j-th node after projection processing, is the motion feature of the i-th node after projection processing, is the motion feature of the jth node after projection processing, and L is the number of frames; The expression of the edge connectivity is: Among them, R t is the edge connectivity, σ is the activation function, is the node connection weight matrix, and is the linear change of the connection weight matrix of different nodes, is the appearance feature of the i-th node after projection processing, T is the time, is the appearance feature of the j-th node after projection processing, is the motion feature of the i-th node after projection processing, is the motion feature of the jth node after projection processing.

5. The video question answering method based on spatiotemporal graph convolution according to claim 1, characterized in that: The performing spatial graph convolution processing on the spatial graph to obtain a target spatial graph includes: Based on the attention mechanism, the spatial graph is subjected to spatial graph convolution processing to obtain a target spatial graph. The expression of the spatial graph convolution processing is: in, is the output of the spatial graph convolution processing, ReLU is the activation function, MHSA is multi-head self-attention, H is the number of attention heads, W(u) is the graph weight matrix of the u-th layer, u is the number of layers, is the output node representation of the u-1th layer, R t is the edge connectivity, l is the frame length, n is the number of clips, and I is the convolution dimension.

6. The video question answering method based on spatiotemporal graph convolution according to claim 1, characterized in that: The performing time dynamic mining on the time graph to obtain a target time graph includes: Performing node feature processing on the time graph based on a node transformer to obtain a time graph after node feature processing; Based on the edge transformer, edge feature processing is performed on the time graph after node feature processing to obtain the target time graph.

7. The video question answering method based on spatiotemporal graph convolution according to claim 1, characterized in that: The step of generating word-level question embeddings based on the preprocessed question text data and constructing a question graph based on the word-level question embeddings comprises: The preprocessed question text data is word-embedded based on the bidirectional encoding representation transformer model to obtain word-level question embedding; The attention weight of each word is calculated based on the natural language processing toolkit using the word-level question embedding, and the target word-level question embedding is generated based on the attention weight. The calculation expression of the attention weight is: A w =softmax(MLP(E s )), Among them, A w is the attention weight, MLP is the multi-layer perceptron, E s is a word, softmax is a normalized exponential function, The expression of the target word-level question embedding is: in, is the target word-level question embedding, is the attention weight of the corresponding word, Q w Embedding for word-level questions; The association between question nodes is calculated, the target weight is calculated, and a question graph is constructed based on the attention weight, the target word-level question embedding, the association between question nodes and the target weight. The expression of the association between question nodes is: Among them, e ij is the correlation between the i-th problem node and the j-th problem node, λ is the scaling factor, The attention output of the i-th question word, is the attention output of the jth question word, and K is the number of question words.

8. The video question answering method based on spatiotemporal graph convolution according to claim 1, characterized in that: The method generates a multi-level feature representation of the video based on the target spatiotemporal graph using a hierarchical clustering algorithm, and aggregates the question graph and the multi-level feature representation of the video based on a dynamic graph attention module to obtain visual aggregation features and question text aggregation features, including: Based on the target spatiotemporal graph, the attention mechanism is used to perform node aggregation to obtain a frame-level feature representation, and the expression of the frame-level feature representation is: Among them, f F is the frame-level feature representation, N is the number of clips, is the learnable parameter W G The linear transformation of is the i-th node representation of the space-time graph, σ is the activation function; The frame-level feature representation is projected based on the self-attention layer to obtain the projected frame-level feature representation. The expression of the projected frame-level feature representation is: f′ F =φW m [MHSA(φW f (f O ))], Among them, f′ F is the frame-level feature representation after projection, W m and W f is the linear change matrix, MHSA is the multi-head self-attention mechanism, φ is the linear transformation parameter, f O Identify the object; The fragment-level feature representation is generated based on the maximum pooling, and the expression of the fragment-level feature representation is: f C =MPool(f′ F ), Among them, f C is the fragment-level feature representation, f′ F is the frame-level feature representation after projection, M is the segment-based motion feature vector, and Pool is the pooling function; The multi-level feature representation of the video is used as a video graph, and the target attention weight between the video graph and the question graph is calculated. The calculation expression of the target attention weight is: φ(u i ,in t )=W2·σ(W1·[u i ;in t ]+b1)+b2, Among them, a i,t is the attention weight between the i-th node in the question graph and the t-th node in the video graph, exp is the exponential function, φ is the score function, and u i is the i-th node in the problem graph, v t is the tth node in the video graph, W1, W2, b1 and b2 are the learnable parameters of the feedforward function, and σ is the nonlinear activation function; Based on the target attention weight, feature aggregation is performed to obtain visual aggregation features and question text aggregation features. The expression of the visual aggregation feature is: in, is the visual aggregation feature, N is the number of clips, a i,t is the attention weight between the i-th node in the question graph and the t-th node in the video graph, u i is the i-th node in the problem graph; The expression of the question text aggregation feature is: in, is the aggregated feature of the question text, T is the total number of words, and a i,t is the attention weight between the i-th node in the question graph and the t-th node in the video graph, v t is the tth node in the video graph.

9. The video question answering method based on spatiotemporal graph convolution according to claim 1, characterized in that: The step of fusing the visual aggregate features and the question text aggregate features to obtain a joint feature representation, and generating a predicted answer based on an answer decoder using the joint feature representation, includes: Based on the combination of visual aggregation features and video multi-level feature representation, a combined visual feature is obtained, and based on the combination of question text aggregation features and question graph, a combined question text feature is obtained. The expression of the combined visual feature is: Among them, h t To combine visual features, ReLU is the activation function, φW v is a linear variation with learnable parameters, v t is a multi-level feature representation of the video. is the visual aggregation feature, b v is the bias term; The expression combining the question text features is: Among them, h i To combine the question text features, ReLU is the activation function, φW u is a linear variation with learnable parameters, u i For the problem diagram, is the aggregated feature of the question text, b u is the bias term; The combined visual features and the combined question text features are fused to obtain a joint feature representation, and the expression of the joint feature representation is: f joint =Fusion(X), Among them, h t To combine visual features, h i To combine the question text features, N is the number of clips, T is the total number of words, and f joint is the joint feature representation, and Fusion is the fusion function; The joint feature representation is input into the answer decoder, and a prediction calculation is performed based on the answer decoder to obtain a predicted answer, wherein the expression of the prediction calculation is: y′=σ(W o f joint +b o ), s=softmax(W y′ y′+b), Among them, y′ is the problem mapping, σ is the activation function, and f joint is the joint feature representation, W o and W y′ is the weight of the answer decoder, b o and b are the bias terms of the answer decoder, softmax is the normalized exponential function, and s is the mapping probability of the answer.

10. A video question answering system based on spatiotemporal graph convolution, characterized in that: The system comprises: Data acquisition and preprocessing module: used to acquire video data and question text data, and preprocess the video data and question text data to obtain preprocessed video data and question text data; A target matrix generation module is used to extract visual features based on the preprocessed video data, obtain target visual features, and generate a target matrix based on the target visual features; A visual feature representation acquisition module: used for performing spatial information aggregation using the target matrix based on a gated multi-layer perceptron layer to obtain a visual feature representation; A target spatiotemporal graph generation module is used to construct a spatiotemporal graph based on the visual feature representation, wherein the spatiotemporal graph includes a time graph and a space graph, and performs spatial graph convolution processing on the space graph to obtain a target spatial graph, performs temporal dynamic mining on the time graph to obtain a target time graph, and generates a target spatiotemporal graph based on the target spatial graph and the target time graph; A question graph construction module: used to generate word-level question embeddings based on the preprocessed question text data, and to construct a question graph based on the word-level question embeddings; Feature aggregation module: used to generate a multi-level feature representation of the video based on the target spatiotemporal graph using a hierarchical aggregation algorithm, and perform feature aggregation on the question graph and the multi-level feature representation of the video based on the dynamic graph attention module to obtain visual aggregation features and question text aggregation features; Answer prediction module: used to fuse the visual aggregate features and the question text aggregate features to obtain a joint feature representation, and generate a predicted answer based on the answer decoder using the joint feature representation.

Images