A cross-modal video retrieval method, system and storage medium based on multi-head self-attention mechanism
Video and text features are extracted through multi-headed self-attention mechanism and bidirectional GRU/CNN, and mapped them to the shared space using affine transformation and batch regular layers, solving the feature mapping and semantic similarity preservation in cross-modal video retrieval. The problem has achieved high accuracy retrieval effect.
Patent Information
- Application Number
- CN202010718382.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-07-23
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-07-23
AI Technical Summary
Existing cross-modal video retrieval methods based on deep learning have NP problems when mapping original data features to shared spaces, it is difficult to maintain semantic similarity between data, and it is difficult to effectively model the heterogeneous multimodal data. relation.
Using a multi-head self-attention mechanism combined with average pooling and bidirectional GRU/CNN, multi-level features of video and text are extracted, and mapped to the shared space through affine transformation and batch regular layers, using triple sorting loss function Training to improve the representation of the model.
Effectively maintain the similarity of data in the original space, improve the accuracy of cross-modal video retrieval, and generate high-quality vector representations to improve retrieval performance.
Smart Images

Figure CN112241468B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of video technology, and in particular to a cross-modal video retrieval method, system and storage medium based on a multi-head self-attention mechanism. Background Art
[0002] With the explosive growth of multimedia data, traditional single-modal retrieval can no longer meet people's retrieval needs in the multimedia field. Users are eager to use one modal data as a query object to retrieve another modal data content with similar semantics, such as using images to retrieve text, text to retrieve images or videos, etc., that is, cross-modal retrieval.
[0003] Cross-modal retrieval requires processing data from different modalities at the same time. These data have certain similarities in content, but their underlying features are heterogeneous, making it difficult to directly calculate their similarity, that is, there is a "semantic gap problem". The method of mapping different modal data to a common subspace is widely used in cross-modal retrieval because it can build a bridge over the "semantic gap". In this common subspace, the distance between two points reflects the similarity of the two modal data before their mapping.
[0004] The performance of cross-modal video retrieval mainly depends on the quality of the learned shared space. It is usually assumed that the quality of the shared space lies in whether it can effectively retain the semantic information implicit in the original data of different modalities. For example, if the semantics between two different modal data are similar, then their distance in the shared space is small. If the semantics between two different modal data are not similar, then their distance in the shared space is large. In order to improve the quality of the shared space representing data, a deep learning-based method is usually used to generate a mapping function from multimodal data to the shared space position, and when optimizing the model, it is required that the vector represented in the shared space must satisfy a specific statistical distribution law with the original features.
[0005] There are several main problems in cross-modal video retrieval based on deep learning: 1) Mapping the feature space of original data to the shared space is an NP-hard problem; 2) How to maintain the semantic similarity between data during feature mapping; 3) How to model the relationship between heterogeneous multimodal data. Summary of the invention
[0006] In order to solve the problems in the prior art, the present invention provides a cross-modal video retrieval method based on a multi-head self-attention mechanism.
[0007] The present invention provides a cross-modal video retrieval method based on a multi-head self-attention mechanism, comprising the following steps:
[0008] Video encoding step: For a given video modality data, video modality feature extraction is performed, using the feature vector {v1,v2...v n} to describe the video, v t Represents the features of the tth frame of the video, n represents the number of frames of the video, and calculates the multi-level features of the video represents the global feature encoding of the video, represents the video context feature encoding, Represents the local feature encoding of the video;
[0009] Text encoding step: Given a text s of length m, use one-hot encoding to represent each word in the text s, generating a one-hot encoding vector sequence {w1,w2,.....,w m}, where w t Represents the vector of the t-th word, and calculates the multi-level features of the text φ(s) = [f s (1) ,f s (2) ,f s (3) ], f s (1) is the global feature encoding of the text, f s (2) is the text context feature encoding, f s (3) It is the local feature encoding of the text;
[0010] Joint embedding step: The multi-level features φ(v) of the encoded video and the multi-level features φ(s) of the text are mapped into a shared space through affine transformation to obtain the video feature vectors f(v) and f(s) in the shared space, thereby generating a vector representation.
[0011] As a further improvement of the present invention, in the video encoding step, for a given video modality data, a frame is extracted every 0.5 seconds, and for each frame, a deep residual network of ResNet-152 version is used to extract video modality features, and the network parameters are initialized using the parameters of the deep residual network pre-trained on the ImageNet data set; the penultimate layer of the deep residual network is selected as the encoding of the image data, with a dimension of 2018, and then the feature vector {v1,v2...v n} to describe the video, here v t represents the features of the tth frame of the video, and n represents the number of frames in the video.
[0012] As a further improvement of the present invention, in the video encoding step, a multi-head attention mechanism and an average pooling method are used to calculate the video global feature encoding. use To represent the encoding result of average pooling, v t Represents the features of the tth frame of the video. The encoding of the video is obtained by the weighted average of each frame in the video, that is,
[0013] In the video encoding step, the video context feature encoding
[0014] In the video encoding step, the local feature encoding of the video
[0015] As a further improvement of the present invention, in the text encoding step, the text global feature encoding f s (1) It is obtained by averaging all the one-hot encoding vectors in the sequence and concatenating the features obtained by the self-attention mechanism of the one-hot encoding. The text encoding based on the bidirectional GRU is represented as f s (2) , encode the Bi-GRU-CNN-based text into f s (3) .
[0016] As a further improvement of the present invention, in the joint embedding step, the multi-level features φ(v) of the video and the multi-level features φ(s) of the text obtained after encoding are mapped into a shared space by using a full connection followed by a batch regularization layer. Parameter W v and W s They are the parameters of the full connection on the video side and the text side, and their respective parameters are b v and b s . .
[0017] As a further improvement of the present invention, the cross-modal video retrieval method further includes a training step, in which the video feature vectors f(v) and f(s) are input into a triple-based ranking loss function for training.
[0018] The present invention also provides a cross-modal video retrieval system based on a multi-head self-attention mechanism, comprising:
[0019] Video encoding module: used to extract video modality features for given video modality data, using the feature vector {v1,v2...v n} to describe the video, v tRepresents the features of the tth frame of the video, n represents the number of frames of the video, and calculates the multi-level features of the video represents the global feature encoding of the video, represents the video context feature encoding, Represents the local feature encoding of the video;
[0020] Text encoding module: Given a text s of length m, use one-hot encoding to represent each word in the text s, and generate a one-hot encoding vector sequence {w1,w2,.....,w m}, where w t Represents the vector of the t-th word, and calculates the multi-level features of the text φ(s) = [f s (1) ,f s (2) ,f s (3) ], f s (1) is the global feature encoding of the text, f s (2) is the text context feature encoding, f s (3) It is the local feature encoding of the text;
[0021] Joint embedding module: used to map the encoded multi-level features φ(v) of the video and the multi-level features φ(s) of the text into a shared space through affine transformation, and obtain the video feature vectors f(v) and f(s) in the shared space, thereby generating a vector representation.
[0022] As a further improvement of the present invention, in the video encoding module, for given video modality data, one frame is extracted every 0.5 seconds, and for each frame, a deep residual network of ResNet-152 version is used to extract video modality features, and the network parameters are initialized using the parameters of the deep residual network pre-trained on the ImageNet data set; the penultimate layer of the deep residual network is selected as the encoding of the image data, with a dimension of 2018, and then the feature vector {v1,v2...v n} to describe the video, here v t represents the features of the tth frame of the video, and n represents the number of frames in the video;
[0023] In the video encoding module, a multi-head attention mechanism and average pooling are combined to calculate the video global feature encoding. use To represent the encoding result of average pooling, v tRepresents the features of the tth frame of the video. The encoding of the video is obtained by the weighted average of each frame in the video, that is,
[0024] In the video encoding module, the video context feature encoding
[0025] In the video encoding module, the video local feature encoding
[0026] As a further improvement of the present invention, in the text encoding module, the text global feature encoding f s (1) It is obtained by averaging all the one-hot encoding vectors in the sequence and concatenating the features obtained by the self-attention mechanism of the one-hot encoding. The text encoding based on the bidirectional GRU is represented as f s (2) , encode the Bi-GRU-CNN-based text into f s (3) ;
[0027] In the joint embedding module, the multi-level features φ(v) of the video and the multi-level features φ(s) of the text obtained after encoding are mapped into a shared space by using a full connection followed by a batch regularization layer. Parameter W v and W s are the parameters of the full connection on the video side and the text side, and their respective parameters are b v and b s ;
[0028] The cross-modal video retrieval system also includes a training module. In the training step, the video feature vectors f(v) and f(s) are input into a triple-based ranking loss function for training.
[0029] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the cross-modal video retrieval method described in the present invention when called by a processor.
[0030] The beneficial effects of the present invention are as follows: experiments have shown that the present invention can effectively maintain the similarity of data in the original space and improve the accuracy of retrieval. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 This is a cross-modal video retrieval framework diagram based on a multi-head attention mechanism of the present invention. DETAILED DESCRIPTION
[0032] The present invention discloses a cross-modal video retrieval method based on a multi-head self-attention mechanism. The present invention mainly aims at the problem of how to fully mine the semantic information inside the multi-modal data and generate efficient vectors. Through the form of supervised training, the semantic information in the training multi-modal data is fully utilized for training. At the same time, the multi-head self-attention mechanism is introduced to capture the subtle interactions inside the video and text, and the key information of the multi-modal data is selectively focused on to enhance the representation ability of the model, better mine the data semantics, and ensure the consistency of the distance of the data in the original space and in the shared subspace. When training the model, a supervised machine learning method is used, and a sorting loss function based on triples is adopted. The order of positive samples in each batch is introduced to better correct the sorting error. For two different modalities, we use two networks with similar structures to extract features. The features are mainly divided into three parts: global features, time features, and local features. The three features are connected in series, and then the learned features are mapped to the shared space using a joint embedding module, so that high-quality vector representations can be generated. Considering the principle that the distance between approximate sample point pairs in the shared space and the original space is consistent, after the approximate sample point pairs in the original space are mapped to the shared space, the vector distance is as small as possible. Experiments have shown that the present invention can effectively maintain the similarity of data in the original space and improve the accuracy of retrieval. The following is a detailed description:
[0033] 1. Related concepts and problem definitions:
[0034] Assume that the training set O = {o i =[x i ,y i ,c i ], i = 1, 2, ..., n}, where the video modality feature set is The text modality feature set is The class label set is If {x i ,y i} belongs to the jth category, then c ji =1, otherwise c ji = 0. Sample pair The semantic correspondence between video modality and text modality is given. Let the test set be T = {o t =[x t ,y t ],t=1,...,m}, where and are two different modal samples. The goal of the cross-modal retrieval problem is to learn a cross-modal similarity measure sim(□) for a given query word x q ∈X, return the most similar sample of another modality: The □ in sim(□) represents the difference between video x and text y, and sim(□) represents the Manhattan distance between video and text.
[0035] 2. Framework Description
[0036] The present invention utilizes global features, temporal features and local features in extracting multimodal data features. In extracting global features of multimodal data, a multi-head self-attention mechanism is used to automatically learn the important parts of multimodal data and filter out the unimportant parts; at the same time, an average pooling method is used to capture repeated features in the data. In extracting contextual semantic features of multimodal data, a bidirectional neural network (GRU) is used to extract multimodal data time series. In capturing local features of multimodal data, a one-dimensional convolutional neural network (CNN) with multiple different convolution kernels is used to capture the interaction between adjacent features in multimodal data. The present invention fully extracts data features at multiple levels in multimodal data and mines data semantic information. Then, a joint embedding module is used to map the learned features into a shared space, and a high-quality vector representation can be generated.
[0037] 2.1 Model Overview
[0038] The processing method for video and sentence (text) proposed in the present invention is similar, which is a symmetrical network, that is, the video encoding layer and the sentence encoding layer have similar structures. For a given video V and sentence (text) S, assuming that after the video encoding layer and the sentence encoding layer, we encode the video and sentence into φ(v) and φ(s) respectively, the model framework of the present invention is as follows Figure 1 As shown in the figure, each modality data is encoded through several layers, and the encoding results are combined to describe the two modalities in a coarse-to-fine manner. This method allows us to design a similar encoding network to process two different data modalities. Then, the features of the two modal data are fused through the joint embedding module, and finally trained through the improved triple-based ranking loss function.
[0039] 2.2 Video encoding module (video encoding steps)
[0040] For the given video modality data, we extract a frame every 0.5 seconds. For each frame, we use the Deep Residual Network to extract features.
[0041] In this paper, the residual network of ResNet-152 version is used to extract video modality features, and the network parameters are initialized using the parameters of the network pre-trained on the ImageNet dataset. The present invention selects the second to last layer of the network (i.e., excluding the last SoftMax classification layer) as the encoding of the image data, with a dimension of 2018. Then, we can use the feature vector {v1,v2...v n} to describe the video, here v t represents the features of the tth frame of the video, and n represents the number of frames in the video.
[0042] 2.2.1 Video Global Feature Coding
[0043] The present invention adopts a combination of multi-head attention mechanism and average pooling to calculate the global encoding of the video.
[0044] Average pooling is widely used in video cross-modal retrieval, which by definition means pooling to capture visual modalities that appear repeatedly in the video content. These modalities are often global. We use To represent the encoding result of this average pooling:
[0045]
[0046] The principle of the attention mechanism is to calculate weights for different feature representations and obtain the final feature representation by weighted summation, so as to reflect the importance of different frames in the video. In the present invention, the head self-attention mechanism adopted by the present invention is implemented by two single-layer multi-layer perceptrons (MLPs) with a bias term of 0 and a SoftMax layer:
[0047] α=softmax(w2 tanh(w1φ(v)))
[0048] The final value obtained by the multi-head self-attention mechanism is the weight of each frame. The size of the weight indicates the selection tendency of the hidden layer vector. It can automatically learn the important parts of the video and filter out the unimportant parts. Finally, the encoding of the video is obtained by the weighted average of each frame in the video:
[0049]
[0050] By combining the recurring visual features in the video content with the important features in the video, we can get the global feature encoding of the video:
[0051]
[0052] 2.2.2 Video context feature encoding
[0053] Bidirectional GRU (Bi-GRU) is effective in extracting features from data with temporal order. We assume that such a network is also effective in modeling temporal information in videos. We use a bidirectional Bi-GRU, which has fewer parameters than LSTM and therefore requires less training data. A Bi-GRU consists of two separate GRU layers, a forward GRU layer and a reverse GRU. The forward GRU inputs the features of each frame of video in the normal order, while the reverse GRU inputs them in the reverse order. and represents the implicit state at the corresponding time step t=1..n. The implicit state can be expressed by the following formula:
[0054]
[0055]
[0056] Here and Represents the forward and backward GRU respectively, through and To pass the state at time t. and We can get the output of GRU at time t. Based on experience, the hidden vector size in the forward GRU and backward GRU is set to 512. Therefore, h t The size of is 1024. Combining all the outputs, we can get a feature representation queue H = {h1,h2,....,h n}, the dimension is 1024*n. Through average pooling, we can get the video context feature encoding as follows:
[0057]
[0058] 2.2.3 Video Local Feature Coding
[0059] The previous layer gives the same weight to the output of the Bi-GRU at each step. To enhance the local modality data that helps distinguish the nuances of the video, we build a convolutional network on top of the Bi-GRU. In particular, we use a one-dimensional CNN originally developed for sentence classification.
[0060] The input of our CNN is the feature list H generated by the Bi-GRU module in the previous section. k,r represents a one-dimensional convolution module, which contains r = 512 filters of size k. Pass H through Conv1d k,rIt can generate n*k features. Use RELU activation function for nonlinear transformation to enhance the model's fitting ability. Because the number of frames n in each video is different, we use maxpooling to convert the feature list H into a vector c with a fixed length of r. k We use the following formula to express it:
[0061] c k =max pooling(ReLU(Conv1d k,r (H)))
[0062] A convolution kernel with k=2 can be used to find the interaction between two adjacent rows in H. Similarly, k represents the interaction between k adjacent video frames. We use convolution kernels k=2, 3, 4, 5 to generate feature representations in multiple ranges. Their outputs are concatenated to form the Bi-GRU-CNN based encoding.
[0063] f v (3) =[c2,c3,c4,c5]
[0064] because It is obtained sequentially at different levels through a specific coding strategy. We believe that the three coding results are complementary and it is reasonable to have a certain redundancy. Therefore, the outputs of the three layers are connected in series to obtain a multi-level encoding of the input video.
[0065]
[0066] In fact, although this connection operation is simple, it is a common functional combination.
[0067] 2.3. Text encoding module (text encoding step)
[0068] The above encoding network is slightly modified and is also applicable to text. Given a sentence (text) s of length m, we use one-hot encoding to represent each word. Therefore, we can generate a one-hot encoding vector sequence {w1,w2,.....,w m}, where w t Represents the vector of the tth word. Global encoding f s (1) It is obtained by averaging all the one-hot encoded vectors in the sequence and concatenating the features obtained by the self-attention mechanism of the one-hot encoding.
[0069] For biGRU-based encoding, each word is first multiplied with a word embedding matrix to convert it into a dense vector. We initialize the matrix using a word2vec model trained on English tags containing 30 million Flickr images. The rest is basically the same as the video part. We denote the bidirectional GRU-based sentence encoding as f s (2) Similarly, we encode the sentence based on Bi-GRU-CNN as f s (3) Here, we use three 1-d convolutional blocks with k=2, 3, and 4. Concatenate all the encoding results in the text encoding layer to get the multi-level encoding of the sentence:
[0070] φ(s)=[f s (1) ,f s (2) ,f s (3) ]
[0071] There is no direct relationship between φ(s) and φ(v), and they are not directly compared. In the video text similarity calculation, the vector needs to be projected into a common space and then a learning algorithm is given.
[0072] 2.4. Joint Embedding Module (Joint Embedding Step)
[0073] For the video features φ(v) and text features φ(s) obtained after encoding, they are mapped to a shared space through affine transformation. The usual practice is to use a fully connected layer for mapping. After some theoretical research, the present invention adopts a batch normalization layer after the full connection because it works better. Through the following transformation, we can get the video feature vectors f(v) and f(s) in the shared space:
[0074] f(v)=BN(W v φ(v)+b v )
[0075] f(s)=BN(W s φ(s)+b s )
[0076] Here the parameter W v and W s They are the parameters of the full connection on the video side and the text side, and their respective parameters are b v and b s .
[0077] 3. Overall goal and optimization algorithm (training steps)
[0078] The sorting-based triplet loss function is as follows:
[0079] L(v,s;θ)=L(r v )[max(0,α-S θ (v,s)+S θ (v,s - ))]
[0080] +L(r s )[max(0,α-S θ (v,s)+S θ (v - ,s))]
[0081] Where α is the distance constant, and s - and v - They represent a sentence negative sample relative to the video v and a video negative sample relative to the sentence v respectively. These two negative samples are not randomly selected, but are selected based on the most similar non-sample pair in each batch.
[0082] L(.) is a weighting function of different ranks. For video embedding v,r v is the rank of all sentences compared with v. Similarly, for a text embedding sr s is the rank of the matching video embedding v among all compared videos. The loss is a weight-based penalty based on the relative ranking of the correct matches in the retrieval results. If a positive match is ranked high in the list, then L(.) will assign a small amount of weight to the loss without causing too much damage to the loss. However, if a positive match is not ranked high, L(.) will assign a much larger weight to the loss, which will eventually push the positive matching pairs to the top.
[0083] The entire algorithm process is described as follows:
[0084]
[0085] In the actual testing phase, the above model is used to calculate the vector representation of multimodal content features in the latent space. Based on the vector representation, another modal data with similar semantics can be retrieved.
[0086] 1. Experimental setup
[0087] We conducted experiments on two benchmark datasets: MSR-VTT and MSVD and MPIIMD to evaluate the performance of the proposed framework.
[0088] MSR-VTT is a large-scale video description dataset. This dataset contains 10,000 video clips. The dataset is divided into 6513 training videos, 2990 test videos, and 497 validation set videos, each with 20 sentences of description. This is one of the largest video captioning datasets in terms of the number of sentences and vocabulary.
[0089] The MSVD dataset contains 1970 Youtube clips, each with an annotation of about 40 sentences. We only use English descriptions. For fair comparison, we use the same split as most methods, 1200 training videos, 100 validation videos, and 670 test videos.
[0090] The MPIIMD dataset contains 54,000 sentences and 72 high-definition movies. We split the dataset according to the official method and use ResNetXt-101 for feature extraction.
[0091] The present invention is compared with the following three existing cross-modal video retrieval methods:
[0092] DeViSE: A deep visual-semantic embedding model uses a latent factor model composed of different modal features to perform joint matrix decomposition to learn the hinge loss sorting problem.
[0093] VSE++: Visual-Semantic Embeddings is a supervised cross-modal retrieval algorithm that encodes the video layer and the text layer to obtain a joint embedding layer. It is the first method to use hard negative sample encoding in cross-modal retrieval.
[0094] W2VV: Word2vis is a deep neural network architecture based on multi-scale sentence vectorization and multi-layer perceptron learning to predict deep visual features of text input, which has been successfully applied to image (video) caption retrieval and video caption generation.
[0095] WlqV rmil : An improved version of W2VV. The original W2VV uses the MSE loss function, and this algorithm uses the MSE loss function based on ranking loss.
[0096] This experiment uses the median rank (MedR) and the recall rate at top K (R@K) as accuracy evaluation indicators. The median rank refers to the median of the ordinal numbers of the retrieved samples corresponding to all the retrieval samples in the retrieval results. The lower the value, the higher the accuracy of the model. The top K recall rate, for example, the top5 recall rate in the V2S task, refers to the ratio of the text corresponding to the video to appear in the top 5 retrieval results when using the video to retrieve the text. The higher the value, the higher the accuracy of the model.
[0097] 2. Experimental results
[0098]
[0099] Table 1. Results of text retrieval video on the MSRVTT dataset
[0100]
[0101] Table 2. Results of video text retrieval on the MSRVTT dataset
[0102]
[0103] Table 3 mAP results of video text retrieval on MSVD and MPIIMD datasets
[0104] It can also be seen from the experimental results that the retrieval performance of the present invention is higher than that of most methods on the three data sets of MSRVTT, MSVD and MPIIMD, which shows that the method proposed in the present invention can better extract the features of multimodal data and make the semantic representation of multimodal data better.
[0105] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the scope of protection of the present invention.
Claims
1. A cross-modal video retrieval method based on multi-head self-attention mechanism, characterized in that: The steps include: Video encoding step: For a given video modality data, video modality feature extraction is performed, using the feature vector {v1,v2...v n } to describe the video, v t Represents the features of the tth frame of the video, n represents the number of frames of the video, and calculates the multi-level features of the video represents the global feature encoding of the video, represents the video context feature encoding, Represents the local feature encoding of the video; Text encoding step: Given a text s of length m, use one-hot encoding to represent each word in the text s, generating a one-hot encoding vector sequence {w1,w2,.....,w m }, where w t Represents the vector of the tth word, and calculates the multi-level features of the text φ(s) = [f s (1) ,f s (2) ,f s (3) ], f s (1) is the global feature encoding of the text, f s (2) is the text context feature encoding, f s (3) It is the local feature encoding of the text; Joint embedding step: The multi-level features φ(v) of the video and the multi-level features φ(s) of the text obtained after encoding are mapped into a shared space through affine transformation to obtain the video feature vectors f(v) and f(s) in the shared space, thereby generating a vector representation; In the video encoding step, a multi-head attention mechanism and average pooling are combined to calculate the video global feature encoding. use To represent the encoding result of average pooling, v t Represents the features of the tth frame of the video. The encoding of the video is obtained by the weighted average of each frame in the video, that is, where α = softmax(ω2tanh(ω1φ(v) T ))ω2 and ω1 are the parameters of the fully connected layer, α t is the weight of the tth frame; By combining the recurring visual features in the video content and the important features in the video, the global feature encoding of the video is obtained: In the video encoding step, the video context feature encoding n is the number of video frames, h t is the interaction value of the video context to the tth frame; In the video encoding step, the local feature encoding of the video Among them, c t Indicates the influence of t frames before and after a frame on the frame; In the text encoding step, the text global feature encoding f s (1) It is obtained by averaging all the one-hot encoding vectors in the sequence and concatenating the features obtained by the self-attention mechanism of the one-hot encoding. The text encoding based on the bidirectional GRU is represented as f s (2) , encode the Bi-GRU-CNN-based text into f s (3) ; In the video encoding step, for a given video modality data, a frame is extracted every 0.5 seconds. For each frame, a deep residual network of ResNet-152 version is used to extract video modality features, and the network parameters are initialized using the parameters of the deep residual network pre-trained on the ImageNet dataset; the penultimate layer of the deep residual network is selected as the encoding of the image data, with a dimension of 2018. Subsequently, the feature vector {v1,v2...v n } to describe the video, here v t represents the features of the tth frame of the video, and n represents the number of frames in the video; In the joint embedding step, the multi-level features φ(v) of the video and the multi-level features φ(s) of the text obtained after encoding are mapped into a shared space by using a full connection followed by a batch normalization layer. Parameter W v and W s BN represents the Batch Normalization layer, The parameters of the full connection of the video side and the text side are b v and b s , the cross-modal video retrieval method also includes a training step, in which the video feature vectors f(v) and f(s) are input into a triple-based ranking loss function for training; Triple-based ranking loss function: L(v,s;θ)=L(rv)[max(0,α-Sθ(v,s)+Sθ(v,s-))] +L(rs)[max(0,α-Sθ(v,s)+Sθ(v-,s))] where α is the distance constant, s - and v - They represent a sentence negative sample relative to video v and a video negative sample relative to sentence v respectively. These two negative samples are not randomly selected, but are selected based on the most similar non-sample pair in each batch; L(.) is a weighted function of different ranks. For video embedding v,r v is the rank of all sentences compared with v; for a text embedding s, r s is the rank of the matching video embedding v among all compared videos; The loss is a weight-based penalty based on the relative ranking of the correct matches in the retrieval results; if a positive match is ranked high in the list, then L(.) will assign a small amount of weight to the loss without causing too much damage to the loss; however, if a positive match is not ranked high, L(.) will assign a much larger weight to the loss, which will eventually push the positive match pairs to the top.
2. A cross-modal video retrieval system based on multi-head self-attention mechanism, characterized in that: include: Video encoding module: used to extract video modality features for given video modality data, using the feature vector {v1,v2...v n } to describe the video, v t Represents the features of the tth frame of the video, n represents the number of frames of the video, and calculates the multi-level features of the video represents the global feature encoding of the video, represents the video context feature encoding, Represents the local feature encoding of the video; Text encoding module: Given a text s of length m, use one-hot encoding to represent each word in the text s, and generate a one-hot encoding vector sequence {w1,w2,.....,w m }, where w t Represents the vector of the tth word, and calculates the multi-level features of the text φ(s) = [f s (1) ,f s (2) ,f s (3) ], f s (1) is the global feature encoding of the text, f s (2) is the text context feature encoding, f s (3) It is the local feature encoding of the text; Joint embedding module: used to map the multi-level features φ(v) of the encoded video and the multi-level features φ(s) of the text into a shared space through affine transformation, and obtain the video feature vectors f(v) and f(s) in the shared space, thereby generating a vector representation; In the video encoding module, a multi-head attention mechanism and average pooling are combined to calculate the video global feature encoding. use To represent the encoding result of average pooling, v t Represents the features of the tth frame of the video. The encoding of the video is obtained by the weighted average of each frame in the video, that is, where α = softmax(ω2tanh(ω1φ(v) T ))ω2 and ω1 are the parameters of the fully connected layer, α t is the weight of the tth frame; By combining the recurring visual features in the video content and the important features in the video, the global feature encoding of the video is obtained: In the video encoding module, the video context feature encoding n is the number of video frames, h t is the interaction value of the video context to the tth frame; In the video encoding module, the video local feature encoding Among them, c2, c3, c4, and c5 represent the influence of the t frames before and after a certain frame on the frame; In the text encoding module, the text global feature encoding f s (1) It is obtained by averaging all the one-hot encoding vectors in the sequence and concatenating the features obtained by the self-attention mechanism of the one-hot encoding. The text encoding based on the bidirectional GRU is represented as f s (2) , encode the Bi-GRU-CNN-based text into f s (3) ; In the video encoding module, for a given video modality data, a frame is extracted every 0.5 seconds. For each frame, a deep residual network of ResNet-152 version is used to extract video modality features, and the network parameters are initialized using the parameters of the deep residual network pre-trained on the ImageNet dataset; the penultimate layer of the deep residual network is selected as the encoding of the image data, with a dimension of 2018. Subsequently, the feature vector {v1,v2...v n } to describe the video, here v t represents the features of the tth frame of the video, and n represents the number of frames in the video; In the joint embedding module, the multi-level features φ(v) of the video and the multi-level features φ(s) of the text obtained after encoding are mapped into a shared space by using a full connection followed by a batch regularization layer. BN represents the Batch Normalization layer, parameter W v and W s They are the parameters of the full connection on the video side and the text side, and the parameters on the video side and the text side are b v and b s ; The cross-modal video retrieval system also includes a training module, in which the video feature vectors f(v) and f(s) are input into a triple-based ranking loss function for training; Triple-based ranking loss function: L(v,s;θ)=L(rv)[max(0,α-Sθ(v,s)+Sθ(v,s - ))] +L(rs)[max(0,α-Sθ(v,s)+Sθ(v - ,s))] where α is the distance constant, s - and v - They represent a sentence negative sample relative to video v and a video negative sample relative to sentence v respectively. These two negative samples are not randomly selected, but are selected based on the most similar non-sample pair in each batch; L(.) is a weighted function of different ranks. For video embedding v,r v is the rank of all sentences compared with v; for a text embedding s, r s is the rank of the matching video embedding v among all compared videos; The loss is a weight-based penalty based on the relative ranking of the correct matches in the retrieval results; if a positive match is ranked high in the list, then L(.) will assign a small amount of weight to the loss without causing too much damage to the loss; however, if a positive match is not ranked high, L(.) will assign a much larger weight to the loss, which will eventually push the positive match pairs to the top.
3. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the cross-modal video retrieval method according to claim 1 when called by a processor.
