A text-video temporal localization method based on feature reconstruction

Abstract

This invention provides a text-video temporal localization method based on feature reconstruction. The method includes: receiving text description and video data; extracting text features from the text description to obtain global text features; extracting video features V′ from the video data; reconstructing text features based on a graph convolutional neural network to obtain global text features Q″ with key semantic information; reconstructing video features based on an attention mechanism to obtain video features V″ guided by text features; fusing the global text features Q″ and the video features V″ guided by text features, and outputting the start and end time results of video localization through a two-dimensional convolutional neural network. This invention mines key information from text features using a graph convolutional neural network, and then strengthens the video features related to the text through an attention mechanism, which can effectively distinguish similar segments in the video and improve the accuracy of text-video temporal localization.

Description

Technical Field

[0001] This invention belongs to the field of deep learning and video understanding technology, and specifically relates to a text-video temporal localization method based on feature reconstruction. Background Technology

[0002] With the rapid development of communication technology and the internet, tens of millions of video data points are uploaded to online platforms every day. Faced with this massive amount of video data, how to efficiently and conveniently retrieve the desired content has become a pressing problem. Text-video temporal localization, a newly proposed task in the field of video understanding and analysis, aims to find the start and end times of corresponding segments in a video based on textual descriptions. This is of great significance for fields such as surveillance and security, traffic management, and public safety.

[0003] In recent years, numerous methods for text-video localization have been proposed, achieving some success, but a key problem remains to be solved. Due to the continuous nature of time, a video will contain segments that are temporally adjacent and content-similar. If only global information from the text is extracted while ignoring details such as keywords or limiting words, it will significantly impact the video temporal localization results.

[0004] In their 2020 paper "Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language," published at the Association for the Advancement of Artificial Intelligence (AAAI) conference, Zhang et al. were the first to use two-dimensional convolutional neural networks for video temporal localization. However, this method inputs the entire text into a bidirectional long short-term memory (LSTM) network to extract global features from the text, and inputs the video into a three-dimensional convolutional network (C3D) to extract global features from the video. Then, the video features are fused with the text features, which ignores key information in the video and text that is helpful for localization. (Zhang S, Peng H, Fu J, et al. Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language[J]. Proceedings of the AAAIConference on Artificial Intelligence,2020,34(7):12870-12877.). For example, due to the continuity of time, a video may contain segments that are temporally adjacent and similar in content. Figure 2 As shown, the texts "He took out a glass." and "Person sets smaller glass on table." are similar, and the corresponding video clips also contain the same person, actions, and similar objects, both indicating that someone took a glass from a cabinet and placed it on the table. The difference lies in "smaller," meaning that someone took a smaller glass. However, ignoring the details in the text and video content and failing to uncover the differences between the two video clips and text will significantly impact the localization results. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a text-video temporal localization method based on feature reconstruction.

[0006] To achieve the objectives of this invention, a text-video temporal localization method based on feature reconstruction is provided, comprising the following steps:

[0007] Receive text descriptions and video data;

[0008] Extract text features from the text description to obtain global text features;

[0009] Video features V′ are extracted from the video data;

[0010] Text features are reconstructed based on graph convolutional neural networks to obtain global text features Q″ with key semantic information;

[0011] Video features are reconstructed based on an attention mechanism to obtain video features V″ guided by text features.

[0012] The global text features Q″ and the video features V″ guided by the text features are fused, and the start and end times of video localization are output through a two-dimensional convolutional neural network.

[0013] Furthermore, the extraction of text features from the text description includes:

[0014] Each word q in the text Q i Encoded as M-dimensional word vectors q i ′, i represents the i-th word in text Q;

[0015] Text vectors that have undergone word embedding techniques As input, N q The number of words in the text is used as the basis for generating the corresponding hidden layer states as preliminary text features that are context-aware.

[0016] The initial text features are input into the attention mechanism to generate a global text feature representation Q″. g :

[0017]

[0018] In the formula, W g For learnable parameters, α g,i N represents the weight coefficient of words in the text. q α represents the number of words in the text. g,i Let h represent the proportion of the i-th word in the current text, i.e., the weight coefficient. j This represents the hidden layer state of all words except the i-th word.

[0019] Furthermore, the neural network is any one of Bi-Short Memory Network (Bi-LSTM), Long Short Memory Network (LSTM), Gated Recurrent Unit (GRU), and Bi-Gated Recurrent Unit (Bi-GRU).

[0020] Furthermore, when the neural network is a bidirectional long short-term memory network (Bi-LSTM), the corresponding hidden layer states are generated as preliminary text features with contextual awareness using the following formula:

[0021]

[0022] In the formula, W q , and All are learnable parameters, h i This represents the hidden state of the i-th word. This represents the hidden layer state obtained from the forward computation. This represents the hidden layer state obtained by reverse computation.

[0023] Furthermore, the step of extracting video features from video data includes: using a pre-trained model of a three-dimensional convolutional neural network on the dataset to extract video features, with multiple consecutive frames as a feature extraction unit, two adjacent feature extraction units overlapping, and each feature extraction unit outputting features from the fc6 layer of the pre-trained model, with the video features labeled as V′.

[0024] Furthermore, the text feature reconstruction based on graph convolutional neural networks yields global text features with key semantic information, including:

[0025] Action features are obtained by max pooling the global text features. and entity features Where, N a and N e These are the number of verb phrases and the number of noun phrases, respectively. Features representing action words in the text. Features that represent noun phrases in the text;

[0026] A graph is constructed to represent the relationships between different components in the text. Events, actions, and entities are treated as nodes in the graph, and the relationships between them are represented as edges. The relationships between events, actions, and entities are expressed as follows: Where d i Indicates the total number of events, verbs, and noun phrases;

[0027] A graph convolutional neural network is applied to the constructed graph. The features of each node are updated by aggregating the features of its neighboring nodes. The output of the Lth layer of the graph convolutional neural network is Q. i " (L) It is the desired global text feature Q″ containing semantic information.

[0028] Furthermore, in the graph convolutional neural network, the initial node features Q″ are... i ={Q″ g ,Q″ a ,…,Q″ e Node features Q after semantic characterization i "(0) As the input to the graph convolutional neural network, the calculation formula is as follows:

[0029] Q i " (0) =Q i "⊙W r r ij

[0030] In the formula, r ij Indicates the semantic roles between node i and node j, ⊙ represents element-wise multiplication, and W r These are the parameters that the network learns during the training process.

[0031] Furthermore, in graph convolutional neural networks, residual structures are used between layers to allow the model to have more information from previous layers. The calculation formula for each layer is as follows:

[0032]

[0033] Q i " (l) N is the feature of the i-th node in the l-th layer graph convolutional neural network. i W represents the neighboring nodes of the i-th node. t (l+1) Q is a learnable parameter in the l-th layer of the graph convolutional neural network. j " (l) This represents the features of neighboring nodes in the l-th layer graph convolutional neural network, excluding the i-th node, where j is a neighboring node, and β ij Let be the attention coefficient between node i and its neighbor node j.

[0034] Furthermore, the video feature reconstruction based on the attention mechanism, to obtain the video feature V″ guided by text features, includes:

[0035] Received video features and global text features As input, where N v d represents the total number of video segments. q and d v This refers to the feature dimension, where global average pooling is used to compress the input video features and global text features separately:

[0036] Compression is used to initially fuse video and text features, thereby forming a global joint feature.

[0037] The excitation signal is obtained based on the generated global joint features. Then, the input video features V′ are recalibrated using a gating mechanism to obtain the video features V″ guided by the text features:

[0038] E V′ =W e Z+b e

[0039] V″=2×sigmoid(E V′ )⊙V′

[0040] In the formula, b e This represents the bias parameters in the network, and ⊙ represents the element-wise multiplication.

[0041] Furthermore, global text features and video features are fused. The start and end times of video localization are output through a two-dimensional convolutional neural network, fusing features from both modalities. Then, the normalized multimodal features are input into the two-dimensional convolutional neural network to predict the start time τ. s and termination time τ e :

[0042] τ s ,τ e =2Dconv(norm(V″⊙Q″))

[0043] The temporal intersection-over-union ratio (IoU) is calculated by comparing the predicted start and end times with the ground truth values ​​of each candidate segment, using the letter o. i The supervision label y is then calculated using the following formula. i :

[0044]

[0045] The formula for calculating the final loss function is as follows:

[0046]

[0047] Among them, t min and t max This is the threshold value set, where C represents the total number of candidate segments, and p... i Output score for the i-th candidate segment.

[0048] Compared with the prior art, the present invention has at least the following beneficial effects:

[0049] (1) Text feature reconstruction based on graph convolutional neural network can effectively establish the relationship between the components of the text, thereby inferring the words in the text that play a decisive and key role in video localization, which helps to distinguish similar segments in the video.

[0050] (2) Attention-based video feature reconstruction can enhance text-related video features and suppress text-irrelevant video features under the guidance of text, thereby making the localization more accurate. Attached Figure Description

[0051] Figure 1 This is a flowchart of a text-video temporal localization method based on feature reconstruction provided in an embodiment of the present invention.

[0052] Figure 2 This is a diagram illustrating the text and video content mentioned in the background technology.

[0053] Figure 3 This is a structural diagram of the entire model.

[0054] Figure 4 This is a comparison chart of the predicted results and the actual results obtained by the method of this invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] Please see Figure 1 This invention discloses a text-video temporal localization method based on feature reconstruction, comprising the following steps:

[0057] Step 1: Receive text description and video data.

[0058] Step 2: Extract text features from the text description.

[0059] Step 2 includes the following sub-steps:

[0060] Step 2.1: Encode each word qi in the text Q into an M-dimensional word vector q. i ′, i represents the i-th word in text Q;

[0061] In some embodiments of the present invention, a pre-trained model of the Glove algorithm is used for encoding; in other embodiments, other models such as Word2Vec may also be used.

[0062] In some embodiments of the present invention, a 300-dimensional word vector q is obtained through encoding. i In other embodiments, it can also be any of 50-dimensional, 100-dimensional, 200-dimensional, etc.

[0063] Step 2.2: Convert the text vectors processed by word embedding technology As input, N qThe number of words in the text is used as the basis for calculating the hidden layer states. Then, a neural network is applied to generate the corresponding hidden layer states as preliminary text features with contextual awareness. In some embodiments of this invention, the neural network is a Bidirectional Long Short-Term Memory (Bi-LSTM) network. In other embodiments, any one of a Long Short-Term Memory (LSTM) network, a Gated Recurrent Unit (GRU), or a Bidirectional Gated Recurrent Unit (Bi-GRU) network can be used. When a Bi-LSTM network is used to generate the corresponding hidden layer states, the calculation formula is as follows:

[0064]

[0065] W q , and All are learnable parameters, h i This represents the hidden state of the i-th word. This represents the hidden layer state obtained from the forward computation. This represents the hidden layer state obtained by reverse computation.

[0066] Step 2.3: Input the preliminary text features into the attention mechanism to generate global text features Q″. g The calculation formula is as follows:

[0067]

[0068] W g For learnable parameters, α g,i N represents the weight coefficient of words in the text. q α represents the number of words in the text. g,i Let h represent the proportion of the i-th word in the current text, i.e., the weight coefficient. j This represents the hidden layer state of all words except the i-th word.

[0069] Step 3: Extract video features from the video data.

[0070] For video features, a pre-trained model of a 3D convolutional neural network (C3D) on the Sports1M dataset is used for video feature extraction. 64 consecutive frames are used as a feature extraction unit, and two adjacent feature extraction units overlap by 80%. Each feature extraction unit outputs the features of the fc6 layer in the pre-trained model with a dimension of 4096. The video features are labeled as V′.

[0071] Step 4: Reconstruct text features based on graph convolutional neural networks to obtain global text features with key semantic information.

[0072] Step 4 includes the following sub-steps:

[0073] Step 4.1: Obtain action features by performing max pooling on the global text features. and entity features Where, N a and N e These are the number of verb phrases and the number of noun phrases, respectively. Features representing action words in the text. Features that represent noun phrases (i.e. entities in the following text) in the text.

[0074] Step 4.2: The relationship between events, actions, and entities is represented as follows: Where d i This represents the total number of events, verbs, and noun phrases. A graph is constructed to represent the relationships between different components of the text, with events, actions, and entities as nodes and their relationships as edges.

[0075] Step 4.3: To better learn text features, a graph convolutional neural network is applied to the constructed graph to update the features of each node by aggregating the features of neighboring nodes. The initial node features Q″ are then used... i ={Q″ g ,Q″ a ,…,Q″ e Node features Q after semantic characterization i " (0) As the input to the graph convolutional neural network, the calculation formula is as follows:

[0076] Q i " (0) =Q i "⊙W r r ij

[0077] Where, r ij Indicates the semantic roles between node i and node j, ⊙ represents element-wise multiplication, and W r These are the parameters that the network learns during the training process.

[0078] Step 4.4: Utilize the residual structure between layers to allow the model to incorporate more information from previous layers. The calculation formula for each layer is shown below:

[0079]

[0080] Q i " (l) N is the feature of the i-th node in the l-th layer graph convolutional neural network. i W represents the neighboring nodes of the i-th node. t (l+1) Q is a learnable parameter in the l-th layer of the graph convolutional neural network. j" (l) This represents the features of neighboring nodes in the l-th layer graph convolutional neural network, excluding the i-th node, where j is a neighboring node, and β ij Let be the attention coefficient between node i and its neighbor node j.

[0081] In the formula, the attention coefficient β between node i and its neighbor node j ij It can be calculated using the following formula:

[0082]

[0083] in, and These are learnable weights. f represents the correlation between node i and its neighbor node j. i l , It has been changed to Q in the formula. i " (l) and Q j " (l) Let and represent the features of the i-th node and the features of its neighboring node j in the l-th layer graph convolutional neural network, respectively. T represents the transpose, and D represents the dimension of the node features.

[0084] The output Q of the Lth layer graph convolutional neural network i " (L) It is the desired global text feature Q″ containing semantic information.

[0085] Step 5: Reconstruct video features based on the attention mechanism to obtain video features V″ guided by text features.

[0086] Received video features and global text features As input, where N v d represents the total number of video segments. q and d v The feature dimension is then used to calibrate the input video features by learning and utilizing global multimodal information through a guided activation network. The guided activation network includes two processes: compression and activation.

[0087] Compression refers to compressing global information features. The input video features V′ are composed of independent segment features generated by a pre-trained convolutional neural network, so each segment cannot utilize the contextual semantic information of other segments. To alleviate this phenomenon, global average pooling is first used to compress the input video features and text features separately, and the calculation process is as follows.

[0088]

[0089]

[0090] S V′ (d v ) represents the compressed video features, d v For video features, nv This indicates which dimension is in the dimensions of the video features, i.e., the component of the dimension. S represents the summation of the values ​​of all dimensions of the video features. Q″ (d q ) represents the compressed text features, d q n represents the dimension of the text features. q This refers to the dimension that represents the text feature, i.e., the component of the dimension. This represents the summation of values ​​across all dimensions of the text features.

[0091] To determine the relationship and correlation between text and video features, compression is used to initially fuse the video and text features, thereby forming a global joint feature set. The calculation formula is as follows:

[0092] Z = W S [S V′ ,S Q″ ]+b

[0093] Among them W S For learnable weight parameters, d z =(d v +d q ) / r, where r is the compression ratio, b represents the bias parameter in the network, and S V′ S Q″ For the above S V′ (d v ), S Q″ (d q The abbreviation of ) [S V′ ,S Q″ ] is the concatenation operation between compressed video features and text features, W S [S V′ ,S Q″ ] represents the learnable parameters W in the network. S Features of splicing [S] V′ ,S Q″ Matrix multiplication.

[0094] Excitation refers to recalibrating video features. The excitation signal is obtained from the globally joint features generated by the above operations. Then, the input video features V′ are recalibrated using a gating mechanism to obtain the video features V″ guided by the text features:

[0095] E V′=W e Z+b e

[0096] V″=2×sigmoid(E V′ )⊙V′

[0097] In the formula, b e This represents the bias parameters in the network.

[0098] Step 6: Fuse global text features and video features, and output the start and end time results of video localization through a two-dimensional convolutional neural network.

[0099] Once the video and text features are ready, the features from the two modalities are fused, and then the normalized multimodal features are input into a two-dimensional convolutional neural network to predict the start time τ. s and termination time τ e :

[0100] τ s ,τ e =2Dconv(norm(V″⊙Q″))

[0101] The temporal intersection-of-unity (IoU) is calculated by comparing the predicted start and end times with the ground truth values ​​of each candidate segment, using the letter o. i The supervision label y is then calculated using the following formula. i , where t min and t max It is the set threshold.

[0102]

[0103] The final loss function is calculated as follows, where C represents the total number of candidate segments, and p i Output score for the i-th candidate segment.

[0104]

[0105] In some embodiments of the present invention, such as Figure 3 As shown, the positioning results obtained by the method of the present invention are basically consistent with the actual results, which demonstrates the effectiveness of the method of the present invention.

[0106] The results of the method of this invention and the existing methods are shown in Table 1. It can be seen that the results obtained by the method of this invention are obviously superior to those of other existing models.

[0107] Table 1 Comparison of results between existing models and the method of this invention on the TACOS dataset.

[0108]

[0109] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A feature reconstruction based text-video temporal localization method, characterized in that, The method comprises the following steps: receiving text description and video data; extracting text features from the text description to obtain global text features; Extracting video features from video data ; The global text feature with key semantic information is obtained based on a graph convolutional neural network ; The video feature is reconstructed based on an attention mechanism to obtain a video feature guided by a text feature ; global text features and video features under text feature guidance fusion, outputting start and end time results of video localization by a two-dimensional convolutional neural network the step of extracting text features from the text description comprises: Each word in the text is encoded as an M-dimensional word vector , , represents the i-th word in the text , ​ text vectors that have been subjected to word embedding techniques as input, for the number of words in the text, and then applying a neural network to generate corresponding hidden layer states as context-aware preliminary text features; inputting the preliminary text features into an attention mechanism to generate global text feature representations : wherein, are learnable parameters, are weight coefficients of words in the text, denotes the hidden layer state of the word; denotes the hidden layer state of the word, ≠ ; the step of reconstructing the text features based on the graph convolutional neural network to obtain global text features with key semantic information comprises: An action feature is obtained by maximum pooling on global text features and entity features wherein, and are the number of verb phrases and noun phrases respectively, is a feature representing action words in the text, is a feature representing noun phrases in the text; constructing a graph to represent relationships between different components in the text, events, actions and entities as nodes of the graph, relationships between them as edges of the graph, relationships between events, actions and entities represented as wherein denotes the total number of events, verbs and noun phrases; Applying the graph convolutional neural network on the constructed graph, updating the feature of each node by aggregating the features of neighbor nodes, and Layer graph convolutional neural network output result is the required global text feature with semantic information obtained .

2. The feature reconstruction based text-video temporal localization method according to claim 1, wherein, the neural network is any one of a bidirectional long short-term memory network Bi-LSTM, a long short-term memory network LSTM, a gated recurrent unit GRU and a bidirectional gated recurrent unit Bi-GRU.

3. The feature reconstruction based text-video temporal localization method of claim 2, wherein, when the neural network is a bidirectional long short-term memory network Bi-LSTM, the corresponding hidden layer state is generated as a context-aware preliminary text feature by the following formula: wherein , and are learnable parameters, denotes the hidden layer state of the th word, denotes the hidden layer state computed by the forward pass, denotes the hidden layer state computed by the backward pass.

4. The feature reconstruction based text-video temporal localization method of claim 1, wherein, The video feature extraction from the video data comprises: using a pre-trained model of a three-dimensional convolutional neural network on a data set for video feature extraction, continuous multiple frames as a feature extraction unit, adjacent two feature extraction units overlap, each feature extraction unit outputs the features of the fc6 layer in the pre-trained model, and the video feature is marked as .

5. The feature reconstruction based text-video temporal localization method of claim 1, wherein, In the graph convolutional neural network, the initial node feature The node feature after role semanticization As the input of the graph convolutional neural network, the calculation formula is as follows: In the formula, represents the semantic role between the node and the node , e represents the multiplication of corresponding elements, is the parameter learned by the network itself during the training process.

6. The feature reconstruction based text-video temporal localization method of claim 1, wherein, in the graph convolutional neural network, residual structures are used between layers to make the model have more information of previous layers, and the calculation formula of each layer is as follows: wherein is the i-th layer graph convolutional neural network, feature of the i-th node in the i-th layer graph convolutional neural network, is a learnable parameter in the i-th l layer graph convolutional neural network, denotes the neighbor node feature of the i-th l layer graph convolutional neural network except for the i-th node, j is the neighbor node, is the attention coefficient between the node i and its neighbor node j.

7. The feature reconstruction based text-video temporal localization method of claim 1, wherein, The video feature reconstruction based on the attention mechanism obtains the video feature guided by the text feature , comprising: received video features and global text features as input, where denotes the total number of video segments, and is the dimension of the features, global average pooling is used to compress the input video features and global text features, respectively: The video features and the text features are preliminarily fused using a compression operation, so as to form global joint features ; According to the generated global joint features, an excitation signal is obtained Then, the input video features are recalibrated through a gating mechanism To obtain video features guided by text features : In the formula, denotes a bias parameter in the network, e denotes multiplication of corresponding elements.

8. The feature reconstruction based text-video temporal localization method according to any one of claims 1-7, characterized in that, The global text features and the video features are fused, and the start and end time results of video positioning are output by a two-dimensional convolutional neural network, the features of the two modalities are fused, and then the normalized multi-modal features are input into a two-dimensional convolutional neural network to predict the start time and the end time ​ The intersection over union on the calculated timing of the true value of each candidate segment is represented by the letter , and then the supervision label is calculated by the following formula : Final loss function The formula for the calculation of the final loss function is shown below: wherein, and is a set threshold value, denotes the total number of candidate segments, denotes the output score of the th candidate segment.

Images