A text-video retrieval method based on fine-grained cross-modal alignment model
Through the combination of graph autoencoder and pre-trained model, fine-grained features of video and text are extracted for cross-modal alignment, which solves the problem of difficult video and text interaction in the prior art, and improves text in complex scenarios- Video retrieval performance.
Patent Information
- Application Number
- CN202111312220.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-08
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2041-11-08
AI Technical Summary
Existing cross-modal text-video search technology is difficult to capture fine-grained interactions between video and text, resulting in poor retrieval performance in complex scenarios, especially dissatisfied with the search results of multiple coherent sub-actions/sub-event clips. .
Using a fine-grained cross-modal alignment model based on graph autoencoder, the semantic features of video and text are extracted through video embedding feature learning, text embedding feature learning and joint embedding learning, and the semantic features of video and text are extracted using the pre-trained BERT model and I3D model, and the semantic features of video and text are extracted through the graph. The convolutional encoder and the inner product decoder perform cross-modal alignment, optimizing the loss function to enhance feature representation.
It realizes better cross-modal retrieval performance, better utilizes fine-grained objects and action information in the video, and improves the retrieval accuracy of complex semantic content.
Smart Images

Figure CN114048350B_ABST
Abstract
Description
Technical Field
[0001] The present invention mainly relates to the technical field of information retrieval, and specifically to a text-video retrieval method based on a fine-grained cross-modal alignment model. Background Art
[0002] In recent years, with the popularity of social media platforms (Facebook, Twitter) and video sharing platforms (YouTube, TikTok), multimedia data (images, text, audio, video) on the Internet has grown exponentially. People are in urgent need of a technology that can efficiently retrieve data of different modalities based on the query content.
[0003] Cross-modal text-video retrieval technology aims to retrieve videos (texts) that are most similar to the semantic content of the user's query text (video). The main problem faced by this technology is that the heterogeneity between the two different modalities of text and video hinders the alignment between different modalities.
[0004] The general pattern of existing methods of this technology is to learn a common learning space, and directly measure the distance between the global or local text and video features learned by the model in the common learning space as the similarity between text and video. The higher the similarity, the more relevant the content of the video and the query text, and vice versa. However, these methods only roughly capture the correspondence between different modalities and cannot capture the fine-grained interaction between video and text. Accurate retrieval can be achieved in simple retrieval scenarios that only contain a single action / event, but the retrieval effect in more realistic and complex scenarios is unsatisfactory.
[0005] In order to better capture this fine-grained alignment relationship, retrieval methods based on cross-modal interactions have recently been studied in this technical field. Most of these methods are based on various attention mechanisms to align the semantic content between video and text. Although these methods can discover fine-grained correspondences between different modalities, thereby improving performance, due to the natural heterogeneity differences between video and text, existing attention-based methods may not be able to capture the video sub-action / sub-event segments corresponding to the query text well. In addition, existing research has largely ignored the fine-grained interactions between video sub-action / sub-event segments and sentence phrases in cross-modal learning. As a result, its retrieval performance is often unsatisfactory in real-world scenarios involving multiple coherent sub-action / sub-event segments. Summary of the invention
[0006] In the context of retrieving complex semantic content, videos contain a series of constituent objects with complex interrelationships, and the objects in the video often show different degrees of importance in their corresponding query texts. Therefore, it is critical to model fine-grained video features based on the semantic content of the query text. Compared with the query text, videos usually contain richer and more intuitive semantic content. Because videos contain more meaningless semantic content than the corresponding text, simply using the model to extract frame-level features of each frame and then aggregating the frame-level features into video features will lead to serious information redundancy.
[0007] The present invention mainly provides a text-video retrieval method based on a fine-grained cross-modal alignment model, which is used to mine and enhance meaningful semantic information between video and text modalities to solve the redundant and useless semantic information in the prior art, thereby achieving the purpose of better cross-modal retrieval performance.
[0008] To achieve the above object, the technical solution adopted by the present invention is:
[0009] A text-video retrieval method based on a fine-grained cross-modal alignment model, characterized in that the text-video retrieval method comprises the following steps:
[0010] (1) Video embedding feature learning: Video semantic unit features are extracted at multiple different granularity levels through the video embedding feature learning module;
[0011] (2) Text embedding feature learning: The text embedding feature learning module uses the pre-trained natural language processing model BERT to extract a set of phrase-level text semantic features from the query text;
[0012] (3) Joint embedding learning: The graph autoencoder-based link prediction strategy is integrated into this model through the joint embedding learning module. The key video semantic unit features are aligned with the phrase-level text semantic features using the loss function, and the corresponding text and video features are optimized.
[0013] Furthermore, the video embedding feature learning includes the following steps:
[0014] 1) First, the video is preprocessed. Given a video V, a video frame sequence is obtained by uniformly sampling from the video V at a pre-specified sampling frequency, represented as {v1,v2,…,v N}, where the subscript N represents the length of the video frame in the sequence; the pre-trained I3D model is used to extract the frame-level feature representation of the video. Specifically, the video sub-segment consisting of the first four consecutive frames and the last four consecutive frames of a video frame is input into the I3D model, and the final feature output is used as the feature representation of the current video frame; finally, the video frame sequence {v1,v2,…,vN} is processed into a video clip feature representation, expressed as where f t The dimension of the tth frame is d v The one-dimensional feature vector of
[0015] 2) Then the video feature vectors are clustered. The K-means algorithm is used to cluster the original video frames, where K is the maximum number of sub-actions / sub-events that may appear in the video. Specifically, the Calinski-Harbasz score is used to evaluate the clusters generated by the video features. The higher the score, the better the clustering. Each video feature is divided according to the time sequence. Each video frame feature f t Assign to cluster One of them; sorting the clusters using the timestamps of the video frames to obtain sorted clusters for subsequent decoding;
[0016] 3) In order to further express the characteristics of the cluster C i Convert to vectors of the same dimension for cross-modal alignment; use average pooling for each cluster feature expression, and regard the pooled result as the cluster feature Next, a linear mapping, the specific expression is: in is the embedding feature of each cluster of the video, is a learnable transformation matrix, is the bias term, and the final embedded feature representation of the video clip is the set of all semantic unit representations
[0017] Furthermore, the text embedding feature learning includes the following steps: given a text containing M words {w1, w2, ..., w M} query text Q, first use the existing StanfordCoreNLP toolkit to decompose the query text Q into a string of phrases In order to for each phrase s j The contextual relationship between the words in the sentence is modeled, and this series of phrases is input into a pre-trained BERT language representation model. The 768-dimensional hidden state of the last layer is taken to represent the information of the entire phrase, which is recorded as Then a gating module is used to represent the phrase Convert to embedded features The final embedding feature representation of the query text is the collection of all phrase features
[0018] Furthermore, the step three specifically includes the following steps:
[0019] First, define an undirected bipartite graph as G = {V, E, X}, where V = {x1, x2, …, x n} represents the node set of all visual and textual semantic units, E is the set of link weights between nodes, which can be represented by an adjacency matrix A = [a ij ]∈R n×n To indicate that X is the feature matrix of all nodes, that is represents a cascade operation, n represents the number of nodes, d * Represents the dimension of X. In general link prediction problems, the weights of links in the graph are determined by the input data. In contrast, the weights of the graph are initialized based on the semantic similarity between node features. It is used to update the feature expression of nodes and obtain the link weights of the graph G = {V, E, X}. The graph autoencoder consists of a two-layer graph convolutional encoder and an inner product decoder. The encoder can align visual and text semantic units and obtain relationship-enhanced video and text features for text-video retrieval.
[0020] Furthermore, the specific operation method of the graph convolution encoder is as follows:
[0021] The graph convolutional network takes a graph as input, performs calculations on it, and returns the updated features of each object node as output; for the bipartite graph G, a nonlinear transformation operation is used to map the input feature X into an interactive space; the edge relationship of the graph is constructed through the inner product similarity: A = φ(X)φ(X) T ⊙M'; The graph convolutional network is composed of several layers of networks with the same structure stacked together. A single graph convolutional layer is defined as: The expression of stacking multi-layer graph convolutional network is: The final output of the network is Z 1:L =[Z 1 ,…,Z L ]; Output of the last layer of graph convolutional network As a node representation.
[0022] Furthermore, the specific operation method of the inner product decoder is as follows:
[0023] The inner product decoder is used to reconstruct the adjacency matrix and dynamically mine valuable links between visual and textual semantic units; the representation generated by the encoder already contains rich content and structural information, and a simple inner product decoder can be used to predict the link weights between visual and textual semantic units by reconstructing the adjacency matrix; the reconstructed adjacency matrix It can be expressed as:
[0024] Furthermore, it also includes: reconstructing the loss function, using the cross entropy loss function to measure the approximate error of the autoencoder; by measuring The difference between and A is used to minimize the reconstruction loss, and the formula is:
[0025]
[0026] Finally, the updated adjacency matrix is adopted To obtain the video and text feature representation after relationship enhancement
[0027] Furthermore, the joint embedding learning includes the following steps:
[0028] Aligned video and text feature representations generated by cross-modal alignment method A self-attention pooling operation is used on top of it to obtain the final video and text embedding representation, which consists of two fully connected layers, a tanh activation function and The softmax activation function is performed on the above to obtain the pooled video and text features; the generated video features and text features The expression is as follows:
[0029] A ranking-based ternary loss function is used to optimize the performance of joint embedding learning. Since video-to-text retrieval and text-to-video retrieval are used, the input of the loss function consists of two triplets, namely (T, O, O_) and (O, T, T_). The first element of the triplet is a video (O) or a text query (T), followed by a positive sample and a negative sample from different modalities. The formula is:
[0030] The overall loss function used to train the model is a ranking-based ternary loss function And the reconstruction loss function The sum of is as follows:
[0031] Compared with the prior art, the present invention has the following beneficial effects:
[0032] From the perspective of fine-grained interaction modeling between videos and text sentences, we use graph autoencoders to model the fine-grained interaction between text and video modalities in cross-modal text-video retrieval, so as to better utilize the fine-grained object and action information in the video and achieve better cross-modal retrieval performance.
[0033] The present invention will be explained in detail below with reference to the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 It is a schematic diagram of the model operation flow of the present invention; DETAILED DESCRIPTION
[0035] In order to enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.
[0036] The present invention provides a text-video retrieval method based on a fine-grained cross-modal alignment model, which interactively models the fragment part in the video and the phrase part in the text sentence to achieve cross-modal fine-grained alignment, thereby improving the performance of cross-modal text-video retrieval.
[0037] Based on Graph Auto-Encoder (GAE), the present invention transforms the interaction modeling of video clip units and short phrases in text into a link prediction problem to obtain the explicit relationship between video and text and enhance the aligned feature representation to achieve fine-grained cross-modal alignment.
[0038] like Figure 1 As shown, the cross-modal alignment model proposed by the text-video retrieval method includes: a video embedding feature learning module, which is used to extract video semantic unit features at multiple different granularity levels; a text embedding feature learning module, which uses a pre-trained natural language processing model BERT to extract a set of phrase-level text semantic features from the query text; a joint embedding learning module, which integrates the link prediction strategy based on the graph autoencoder into this model, uses the loss function to align the key video semantic unit features with the phrase-level text semantic features, and optimizes the corresponding text and video features.
[0039] The text-video retrieval method of the present invention specifically comprises the following steps:
[0040] (1) Video embedding feature learning: Video semantic unit features are extracted at multiple different granularity levels through the video embedding feature learning module;
[0041] (2) Text embedding feature learning: The text embedding feature learning module uses the pre-trained natural language processing model BERT to extract a set of phrase-level text semantic features from the query text;
[0042] (3) Joint embedding learning: The graph autoencoder-based link prediction strategy is integrated into this model through the joint embedding learning module. The key video semantic unit features are aligned with the phrase-level text semantic features using the loss function, and the corresponding text and video features are optimized.
[0043] In the present invention, the video embedding feature learning includes the following steps:
[0044] 1) First, the video is preprocessed. Given a video V, a video frame sequence is obtained by uniformly sampling from the video V at a pre-specified sampling frequency, represented as {v1,v2,…,v N}, where the subscript N represents the length of the video frame in the sequence; the pre-trained I3D model is used to extract the frame-level feature representation of the video. Specifically, the video sub-segment consisting of the first four consecutive frames and the last four consecutive frames of a video frame is input into the I3D model, and the final feature output is used as the feature representation of the current video frame; finally, the video frame sequence {v1,v2,…,v N} is processed into a video clip feature representation, expressed as where f t The dimension of the tth frame is d v The one-dimensional feature vector of
[0045] 2) Then the video feature vectors are clustered. The K-means algorithm is used to cluster the original video frames, where K is the maximum number of sub-actions / sub-events that may appear in the video. Specifically, the Calinski-Harbasz score is used to evaluate the clusters generated by the video features. The higher the score, the better the clustering. Each video feature is divided according to the time sequence. Each video frame feature f t Assign to cluster One of them; sorting the clusters using the timestamps of the video frames to obtain sorted clusters for subsequent decoding;
[0046] 3) In order to further express the characteristics of the cluster C i Convert to vectors of the same dimension for cross-modal alignment; use average pooling for each cluster feature expression, and regard the pooled result as the cluster feature Next, a linear mapping, the specific expression is: in is the embedding feature of each cluster of the video, is a learnable transformation matrix, is the bias term, and the final embedded feature representation of the video clip is the set of all semantic unit representations
[0047] In the present invention, the text embedding feature learning includes the following steps: given a text containing M words {w1, w2, ..., w M} query text Q, first use the existing StanfordCoreNLP toolkit to decompose the query text Q into a string of phrases In order to for each phrase s j The contextual relationship between the words in the sentence is modeled, and this series of phrases is input into a pre-trained BERT language representation model. The 768-dimensional hidden state of the last layer is taken to represent the information of the entire phrase, which is recorded as Then a gating module is used to represent the phrase Convert to embedded features The final embedding feature representation of the query text is the collection of all phrase features
[0048] In the present invention, video embedding learning and text embedding learning generate two sets, forming many-to-many pairs of visual and textual semantic units; in order to achieve adaptive mining of valuable information and suppress useless information, cross-modal interaction modeling is regarded as a link prediction problem; specifically, an undirected bipartite graph based on a graph autoencoder is designed for the alignment of many-to-many visual and textual semantic units; the graph helps to mine the explicit relationship between visual and textual semantic units, and further accurately captures cross-modal semantic alignment.
[0049] Using graph autoencoders for cross-modal alignment includes the following steps:
[0050] First, define an undirected bipartite graph as G = {V, E, X}, where V = {x1, x2, …, x n} represents the node set of all visual and textual semantic units, E is the set of link weights between nodes, which can be represented by an adjacency matrix A = [a ij ]∈R n×n To indicate that X is the feature matrix of all nodes, that is represents a cascade operation, n represents the number of nodes, d * Represents the dimension of X. In general link prediction problems, the weights of links in the graph are determined by the input data. In contrast, the weights of the graph are initialized based on the semantic similarity between node features. It is used to update the feature expression of nodes and obtain the link weights of the graph G = {V, E, X}. The graph autoencoder consists of a two-layer graph convolutional encoder and an inner product decoder. The encoder can align visual and text semantic units and obtain relationship-enhanced video and text features for text-video retrieval.
[0051] The specific operation process of the graph convolution encoder and inner product decoder is as follows:
[0052] 1) The graph convolutional encoder aims to convert the original video and text features into relationally enhanced video and text features by constructing a bipartite graph structure; the graph convolutional network (GCN) takes the graph as input, performs calculations on it, and returns the updated features of each object node as output; for the bipartite graph G, a nonlinear transformation operation is used to map the input feature X to an interactive space; in order to dynamically mine the associated phrases of visual semantic units (or the associated visual semantic units of phrases), the edge relationship of the graph is constructed by the inner product similarity: A = φ(X)φ(X) T ⊙M';
[0053] Where A represents the initialized adjacency matrix, φ represents the nonlinear transformation operation used to learn the edge link weights connecting heterogeneous nodes, and M'∈{0,1} n×n is the mask matrix used to construct the bipartite graph, and ⊙ is the matrix element-by-element multiplication operation. A graph convolutional network can be composed of several layers of networks with the same structure stacked together; a single graph convolutional layer can be defined as:
[0054]
[0055] in is the adjacency matrix with the ring edges added, is the corresponding diagonal matrix, whose elements is the learnable weight matrix of each layer, σ is a nonlinear activation function, is the feature matrix output by each layer.
[0056] In order to extract multi-layer graph feature matrices, multi-layer graph convolutional networks are stacked, as shown in the expression:
[0057]
[0058] in is the output of the l-th layer graph convolutional network, is the dimension of the output feature of the lth layer, The parameter matrix will The dimensions are mapped to The final output of the network is Z 1:L =[Z 1 ,…,Z L ], where the number of layers L of the graph convolutional network is set to 2. Finally, the output of the last layer of the graph convolutional network is used As a node representation.
[0059] 2) The inner product decoder is used to reconstruct the adjacency matrix and dynamically mine valuable links between visual and textual semantic units; the representation generated by the encoder already contains rich content and structural information, and a simple inner product decoder can be used to predict the link weights between visual and textual semantic units by reconstructing the adjacency matrix; the reconstructed adjacency matrix It can be expressed as:
[0060]
[0061] in is the adjacency matrix after bipartite graph reconstruction, M'∈{0,1} n×n is the mask matrix used to construct the bipartite graph, and ⊙ is the element-wise matrix multiplication operation.
[0062] 3) Reconstruction loss function: In order to effectively represent the enhanced video and text features Z L , it is necessary to ensure that the reconstructed adjacency matrix is consistent with the initialized adjacency matrix A; therefore, the cross entropy loss function is used to measure the approximation error of the autoencoder; this method measures The difference between and A is used to minimize the reconstruction loss, and the formula is:
[0063]
[0064] In summary, Algorithm 1 summarizes the entire process of the graph autoencoder; the updated adjacency matrix is finally used To obtain the video and text feature representation after relationship enhancement
[0065] The algorithm of text-video retrieval based on the cross-modal fine-grained alignment model is as follows:
[0066]
[0067] In this paper, the joint embedding learning of video and text features aims to compare the similarity of their features.
[0068] The joint embedding learning includes the following steps:
[0069] Aligned video and text feature representations generated by cross-modal alignment method A self-attention pooling operation is used on top of it to obtain the final video and text embedding representation, which consists of two fully connected layers, a tanh activation function and The softmax activation function is performed on the above to obtain the pooled video and text features; the generated video features and text features It is expressed as follows:
[0070] Where K is the number of visual semantic units, J is the number of phrases, and ⊙ is an element-by-element matrix multiplication operation. On this basis, a ranking-based ternary loss function is used to optimize the performance of joint embedding learning. Since video-to-text retrieval and text-to-video retrieval are used, the input of the loss function consists of two triplets, namely (T, O, O_) and (O, T, T_). The first element of the triplet is a video (O) or a text query (T), followed by a positive sample and a negative sample from different modalities. The formula is:
[0071] Where cos(·,·) represents the cosine similarity calculation, δ is a real value greater than 0 and less than 1; the overall loss function used to train the model is the sort-based ternary loss function And the reconstruction loss function The sum of is as follows: where λ is a trade-off parameter.
[0072] In the actual testing phase of the present invention, the model of the present invention is used to calculate the feature vectors of the video and text to be queried in the joint embedding space respectively; based on the feature vector, the corresponding features of another modality with similar semantics can be retrieved.
[0073] 1. Experimental setup
[0074] Experimental tests were conducted on three benchmark datasets: MSR-VTT, YouCook2, and VATEX to evaluate the performance of the model proposed in this invention.
[0075] The MSR-VTT dataset contains 10,000 video clips, each of which is matched with 20 different description texts. Tests are performed on three different splits: 1) Full test set contains 2,990 video clips; 2) 1k-Miech test set contains 1,000 video clips; 3) 1k-Yu test set contains 1,000 video clips.
[0076] The YouCook2 dataset contains 2,000 cooking instruction videos, which are divided into 14,000 video clips. Each video clip has a corresponding description text, of which 3,350 video clips are used for testing.
[0077] The VATEX dataset is a multilingual dataset that contains 25,991 video clips for training, 3,000 video clips for verification, and 6,000 video clips for testing. Each video clip has 10 corresponding English description texts and 10 Chinese description texts.
[0078] 2. Evaluation indicators
[0079] The experiment used the median rank (MedR) and the recall rate at topK (R@K) as evaluation indicators of the model retrieval accuracy. The median rank measures the median of the ranking of positive samples in the samples returned by the retrieval. The lower the value, the higher the accuracy of the model. The topK recall rate measures the proportion of positive samples in the top K samples in the samples returned by the retrieval. The higher the value, the higher the accuracy of the model.
[0080] 3. Experimental results
[0081] Table 1. Comparison of retrieval results on the MSR-VTT dataset
[0082]
[0083]
[0084] Table 2 Comparison of search results on YouCook2 dataset
[0085]
[0086]
[0087] Table 3 Comparison of search results on VATEX data and
[0088]
[0089] It can be seen from the experimental results that the model of the present invention exceeds the retrieval performance of most methods on the three datasets, which shows that the model proposed in the present invention is effective in fine-grained alignment and interaction modeling between video semantic units and sentence phrases, thereby achieving better cross-modal text-video retrieval performance.
[0090] Finally, it should be noted that the above examples are only used to illustrate the present invention and are not intended to limit the technical solutions described in the present invention. Therefore, although this specification has described the present invention in detail with reference to the above examples, a person skilled in the art should understand that the present invention can still be modified or replaced by equivalents. All technical solutions and improvements that do not depart from the spirit and scope of the invention should be included in the scope of the claims of the present invention.
Claims
1. A text-video retrieval method based on a fine-grained cross-modal alignment model, characterized in that: The text-video retrieval method comprises the following steps: (1) Video embedding feature learning: Video semantic unit features are extracted at multiple different granularity levels through the video embedding feature learning module; (2) Text embedding feature learning: The text embedding feature learning module uses the pre-trained natural language processing model BERT to extract a set of phrase-level text semantic features from the query text; (3) Joint embedding learning: The graph autoencoder-based link prediction strategy is integrated into this model through the joint embedding learning module. The key video semantic unit features are aligned with the phrase-level text semantic features using the loss function, and the corresponding text and video features are optimized. The step three specifically includes the following steps: First, define an undirected bipartite graph as G = {V, E, X}, where V = {x1, x2, ..., x n } represents the node set of all visual and textual semantic units, E is the set of link weights between nodes, which can be represented by an adjacency matrix λ = [a ij ]∈R n×n To indicate that X is the feature matrix of all nodes, that is represents a cascade operation, n represents the number of nodes, d * Represents the dimension of X. In general link prediction problems, the weights of links in the graph are determined by the input data. In contrast, the weights of the graph are initialized based on the semantic similarity between node features. It is used to update the feature expression of nodes and obtain the link weights of the graph G = {V, E, X}. The graph autoencoder consists of a two-layer graph convolutional encoder and an inner product decoder. The encoder can align visual and text semantic units and obtain relationship-enhanced video and text features for text-video retrieval. The specific operation method of the graph convolution encoder is as follows: The graph convolutional network takes a graph as input, performs calculations on it, and returns the updated features of each object node as output; for the bipartite graph G, a nonlinear transformation operation is used to map the input feature X into an interactive space; the edge relationship of the graph is constructed through the inner product similarity: A = φ(X)φ(X) T ⊙M′; The graph convolutional network is composed of several layers of networks with the same structure stacked together. A single graph convolutional layer is defined as: The expression of stacking multi-layer graph convolutional network is: The final output of the network is Z 1:L =[Z 1 , ..., Z L ]; Output of the last layer of graph convolutional network As a node representation; The specific operation method of the inner product decoder is as follows: The inner product decoder is used to reconstruct the adjacency matrix and dynamically mine valuable links between visual and textual semantic units; the representation generated by the encoder already contains rich content and structural information, and a simple inner product decoder can be used to predict the link weights between visual and textual semantic units by reconstructing the adjacency matrix; the reconstructed adjacency matrix It can be expressed as:
2. The text-video retrieval method according to claim 1, characterized in that: The video embedding feature learning comprises the following steps: 1) First, the video is preprocessed. Given a video V, a video frame sequence is obtained by uniformly sampling from the video V at a pre-specified sampling frequency, represented as {v1, v2, ..., v N }, where the subscript N represents the length of the video frame in the sequence; the pre-trained I3D model is used to extract the frame-level feature representation of the video. Specifically, the video sub-segment consisting of the first four consecutive frames and the last four consecutive frames of a video frame is input into the I3D model, and the final feature output is used as the feature representation of the current video frame; finally, the video frame sequence {v1, v2, …, v N } is processed into a video clip feature representation, expressed as where f t The dimension of the tth frame is d v The one-dimensional feature vector of 2) Then the video feature vectors are clustered. The K-means algorithm is used to cluster the original video frames, where K is the maximum number of sub-actions / sub-events that may appear in the video. Specifically, the Calinski-Harbasz score is used to evaluate the clusters generated by the video features. The higher the score, the better the clustering. Each video feature is divided according to the time sequence. Each video frame feature f t Assign to cluster One of them; sorting the clusters using the timestamps of the video frames to obtain sorted clusters for subsequent decoding; 3) In order to further express the characteristics of the cluster C i Convert to vectors of the same dimension for cross-modal alignment; use average pooling for each cluster feature expression, and regard the pooled result as the cluster feature Next, a linear mapping, the specific expression is: in is the embedding feature of each cluster of the video, is a learnable transformation matrix, is the bias term, and the final embedded feature representation of the video clip is the set of all semantic unit representations 3. The text-video retrieval method according to claim 1, characterized in that: The text embedding feature learning comprises the following steps: given a text embedding feature containing M words {w1, w2, ..., w M } query text Q, first use the existing StanfordCoreNLP toolkit to decompose the query text Q into a string of phrases In order to for each phrase s j The contextual relationship between the words in the sentence is modeled, and this series of phrases is input into a pre-trained BERT language representation model. The 768-dimensional hidden state of the last layer is taken to represent the information of the entire phrase, which is recorded as Then a gating module is used to represent the phrase Convert to embedded features The final embedding feature representation of the query text is the collection of all phrase features 4. The text-video retrieval method according to claim 1, characterized in that: It also includes: reconstruction loss function, using cross entropy loss function to measure the approximate error of the autoencoder; by measuring The difference between and A is used to minimize the reconstruction loss, and the formula is: Finally, the updated adjacency matrix is adopted To obtain the video and text feature representation after relationship enhancement 5. The text-video retrieval method according to claim 4, characterized in that: The joint embedding learning includes the following steps: Aligned video and text feature representations generated by cross-modal alignment method A self-attention pooling operation is used on top of it to obtain the final video and text embedding representation, which consists of two fully connected layers, a tanh activation function and The softmax activation function is performed on the above to obtain the pooled video and text features; the generated video features and text features The expression is as follows: A ranking-based triple loss function is used to optimize the performance of joint embedding learning. Since video-to-text retrieval and text-to-video retrieval are used, the input of the loss function consists of two triplets, namely (T, O, O - ) and (O, T, T - ), the first element of the triple is a video (O) or a text query (T), followed by a positive sample and a negative sample from different modalities, the formula is: The overall loss function used to train the model is a ranking-based ternary loss function And the reconstruction loss function The sum of is as follows:
Citation Information
Cited By
Video retrieval method based on video-text double alignment and ETVA model
CN121210716A
