A video text cross-modal retrieval method based on pre-clustering guidance

Abstract

The application relates to video text cross-modal retrieval and relates to a video text cross-modal retrieval method based on pre-clustering guidance. When an unsupervised video text cross-modal retrieval model is trained on an unlabeled data set, the problem of weak supervision information caused by unreasonable sample pair construction exists. 1) Extracting video and text original features of the training data set; 2) splicing time-aligned normalized original features as unified features, setting a group of threshold values and performing DBSCAN clustering multiple times to generate multiple groups of clustering clusters; 3) calculating the mean value of the video and text features of each clustering cluster as the clustering center; 4) sampling a batch of training samples, for each group of clustering centers, finding the nearest center to construct a pseudo label for the video and text, and constructing a video text positive sample pair; 5) projecting the original features into a common space, calculating the loss of the positive sample corresponding to each group of centers, and updating the projection model through backward propagation; 6) repeating steps 4) to 5) until a predetermined number of training rounds is reached.

Description

Technical Field

[0001] This invention relates to cross-modal retrieval of video text, and more particularly to a cross-modal retrieval method for video text based on pre-clustering guidance. Background Technology

[0002] With the growth of massive amounts of video data on the internet, cross-modal video-text retrieval technology has developed rapidly. This task uses video or text as the query, retrieves text or video from a database, and sorts the results by semantic similarity as the output. This task utilizes a projection model to project the features of video and text—two different modalities—into the same common space to obtain embedded representations, thus making video and text features directly comparable. The goal is to reduce the distance between semantically related video and text features and increase the distance between unrelated video and text features. The model is trained on large-scale video datasets. Besides video-text retrieval, it can also be used for various downstream tasks that utilize feature similarity calculations, such as ActionStep Localization and Video Question Answering (VideoQA).

[0003] Training a cross-modal video-text retrieval model on irregular, unlabeled data requires mining effective cross-modal supervision information. In large-scale unlabeled video datasets, each video is accompanied by text description captions automatically generated by the source. Based on start and end times, the video is divided into video segments corresponding to the descriptions. These time-aligned basic sample pairs are used as positive pairs for training, but there is no guarantee that these sample pairs are semantically related, thus potentially lacking effective supervision information and limiting the model's training performance.

[0004] Existing works usually divide the video based on the start and end time of the text description of the video, and other methods enhance the contrast loss by clustering the embedding in the common space to improve the model training effect. The method based on the start and end time divides the video based on the start and end time of the text description, and takes the start and end time of the text description as the standard for constructing positive samples, such as HowTo100M(Miech A, Zhukov D, Alayrac JB, et al. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips[C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2019: 2630-2640.), Everything at Once(Shvetsova N, Chen B, Rouditchenko A, et al. Everything at Once-Multi-Modal Fusion Transformer for Video Retrieval[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022: 20020-20029.) and other works simply take the text description and video clip in the start and end time period as positive sample pairs, while other pairs composed of different time samples are regarded as negative sample pairs. On this basis, MIL-NCE(Miech A, Alayrac J B, Smaira L, et al. End-to-end learning of visual representations from uncurated instructional videos[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2020: 9879-9889.), VideoCLIP(Xu H, Ghosh G, Huang P Y, et al.VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding [C] / / Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021: 6787-6800.) and other works further consider temporally adjacent or overlapping samples to expand the positive sample pairs. These methods can obtain semantic-related supervision information to some extent, but on the one hand, they may cause false matching, i.e. semantic-unrelated positive sample pairs or semantic-related negative sample pairs, and on the other hand, they ignore the possible semantic-related samples composed of samples from different videos, thus lacking the diversity of samples. To further enhance the supervision information, the method MCN (Chen B, Rouditchenko A, Duarte K, et al. Multimodal clustering networks for self-supervised learning from unlabeled videos [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021: 8012-8021.) projects the video and text original features into a common space, first calculates the mean of a pair of video and text embeddings aligned in start and end time as the fusion feature of the two, and then performs K-Means clustering on the fusion feature. Finally, the cluster centers are taken as the reference targets for contrastive learning, and the video and text embeddings are pulled closer to the corresponding centers as an additional constraint part in the loss function. However, due to the differences between modalities, the feature representation ability of the projection model is poor at the beginning of training, and it cannot produce accurate feature representation, which affects the accuracy of clustering and makes the clustering unable to achieve the expected guiding effect, thus having a negative impact on the pre-training process. Therefore, how to fully utilize the video and text features to mine effective cross-modal supervision information is crucial to training the video-text cross-modal retrieval model. SUMMARY

[0005] The present application aims at the problem of insufficient cross-modal supervision information required by the current video-text cross-modal retrieval model, and provides a video-text cross-modal retrieval method based on pre-clustering guidance.

[0006] The present application comprises the following steps:

[0007] 1) A feature extractor extracts video and text original features of a training data set;

[0008] 2) The normalized original features of the time-aligned are spliced as unified features, a set of thresholds are set for multiple DBSCAN clustering, and multiple clustering clusters are generated;

[0009] 3) The video and text features of each clustering cluster are calculated respectively to obtain the mean as the clustering center;

[0010] 4) A batch of training samples are sampled, for each clustering center, the nearest center is found to construct a pseudo label, and a video-text positive sample pair is constructed based on the pseudo label;

[0011] 5) The original features are projected into a common space, the loss is calculated based on the positive samples corresponding to the multiple pre-clustering centers, and the projection model is updated by backward propagation;

[0012] 6) Repeat steps 4) to 5) until a predetermined number of training rounds is reached.

[0013] In step 2), the multiple DBSCAN clustering based on the set of thresholds is a pre-clustering based on the original features, and the specific steps are as follows:

[0014] ① The video and text original features are subjected to L2 normalization and splicing (concatenation) to represent the unified features of a pair of start and end time-aligned video and text;

[0015] ② Given a threshold δ, DBSCAN clustering is performed on the unified features, and the similarity is equivalent to the average of the video similarity and the text similarity, so that the unified features can be compared.

[0016] In step 4), the video-text positive sample pair is constructed based on the pre-clustering center, and the specific steps include:

[0017] ① Based on the similarity of the video and text original features and the video and text clustering center, the most similar center is selected to assign a pseudo label;

[0018] ② Directly construct positive sample pairs: videos and texts with the same pseudo-label as positive sample pairs;

[0019] ③ Indirectly construct positive sample pairs: each video assigns its start-end time aligned text to other videos with the same pseudo-label, and each text assigns its start-end time aligned video to texts with the same pseudo-label.

[0020] In step 5), the loss is calculated based on the positive samples corresponding to the pre-clustering centers in each group, and the specific steps are as follows:

[0021] ① For a group of clustering centers determined by a threshold, after constructing positive samples in step 4), each video and text has its corresponding positive sample set. Based on the commonly used InfoNCE loss in contrast learning, the one-to-one positive sample based on cross-modal contrast learning is expanded to one-to-many positive sample determined by the positive sample set.

[0022] ② Each group of clustering centers corresponds to a loss, and all losses are weighted and summed according to the set weight as the final contrast loss;

[0023] ③ The loss function backpropagation calculates the gradient and updates the model parameters.

[0024] The present application gives a group of different clustering thresholds, generates a group of clustering centers by pre-clustering the original features of videos and texts, to generate a group of clustering centers with different granularities, respectively calculates the contrast loss and then combines them by weighting, decides the effect of pulling or pushing a pair of positive samples in a similar voting manner under the guidance of multiple centers, and performs backpropagation training to project the original features of videos and texts to the projection model in the common space. The present application solves the problem of insufficient cross-modal supervision information required by the current video-text cross-modal retrieval model, and pre-clusters the original features of videos and texts without relying on the projection model that projects the original features to the common space, obtains reliable clustering centers to mine more effective cross-modal supervision information. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is the overall flowchart of the present application. The flow from extracting original features to calculating loss is described.

[0026] Figure 2 is the schematic diagram of the core idea of the present application. The overall flow from pre-clustering based on original features to constructing positive samples is described, wherein the dashed box represents the original feature, and the box without border represents the common space embedding.

[0027] Figure 3 is the method schematic diagram of constructing positive samples in the present application. DETAILED DESCRIPTION

[0028] The specific embodiments of the present application and related symbols are specifically described below in conjunction with the drawings.

[0029] The overall method flow of the present application is shown in Figure 1 .

[0030] 1. Method and specific implementation description

[0031] (1) Problem description and symbol description. The data set D contains video clips and automatically generated text subtitles by the video collection website Each pair of start and end time aligned video text pairs is determined by fixed start and end time segmentation. The video feature extractor is extracting video raw features The text feature extractor is extracting text raw features * indicates that the feature extractor is fixed and not updated in the pre-training process, corresponding to the "lock" symbol in Figure 1 . The goal of the present application is to train a projection model composed of a video module and a text module , respectively projecting the video and text raw features into a common space, i.e. obtaining video embedding and text embedding and when the video v and the text t are semantically related, the cosine similarity is high, otherwise it is low.

[0032] (2) Raw feature extraction. When pre-clustering and pre-training the projection model, raw feature extraction is required. The video feature extractor includes two parts: 2D feature extractor and 3D feature extractor. The 2D feature extractor uses a ResNet-152 model pre-trained on the ImageNet dataset, extracting 1 feature per second; the 3D feature extractor uses a ResNeXt-101 model pre-trained on the Kinetics dataset, extracting 1.5 features per second. The 2D feature extractor and the 3D feature extractor output N 2d ×2048, N 3d ×2048 features (N 2d <N 3d). For pre-clustering and positive sample construction process, temporal max-pooling is performed on 2D and 3D features respectively to obtain 2048-dimensional 2D feature vectors and 2048-dimensional 3D feature vectors, and through splicing, 4096-dimensional feature vectors (i.e. video original features) are formed and L2 normalization is performed; for model pre-training, the pre-trained projection model is based on the performance optimal model Everything at Once (composed of multi-head attention and linear layer) of CVPR 2022 (Shvetsova N, Chen B, Rouditchenko A, et al. Everything at Once-Multi-Modal Fusion Transformer for Video Retrieval [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022: 20020-20029.), and the speech module therein is removed for video text pre-training, and the training settings remain consistent with the past related work. The input of the projection model is processed, and the 2D feature of N 2d ×2048 is upsampled by nearest neighbor interpolation to obtain 2D feature of N 3d ×2048, and is spliced with 3D feature to obtain video feature of N 3d ×4096 as the model input. The text feature extractor uses the Word2Vec model pre-trained on the Google News dataset and the “all-MiniLM-L6-v2” model pre-trained by the SentenceTransformer open source framework. For pre-clustering and positive sample construction process, the “all-MiniLM-L6-v2” model is used to extract 384-dimensional embedding as text feature to obtain feature representation convenient for clustering calculation; for model pre-training, the Word2Vec model is used to extract 20×300-dimensional word vectors for each text, and zero-padding is performed for sentences with less than 20 words as the input of the projection model.

[0033] (3) Raw feature pre-clustering. The present application follows the setting of "fixed pre-trained feature extractor + trainable projection model" adopted by most related works, in which the pre-trained feature extractor of a specific modality can produce features of the corresponding raw modality (video or text) with excellent representation ability to represent the semantic information of the sample, and will not change during the training process, so the raw features can be extracted once and pre-clustered to produce clustering centers stored locally for use in model training. When pre-clustering, a unified feature is constructed based on the start and end time alignment relationship, i.e. for a pair of simultaneously appearing video segments v i and text subtitles t i , the video raw features v and text raw features t are extracted by the video feature extractor and text feature extractor respectively and L2 normalized. The video raw features v i and text raw features t i are concatenated to construct a unified feature [vt] i = [v i , t i ]. In order to obtain more accurate supervision information, a set of different similarity thresholds is set for pre-clustering, and DBSCAN algorithm is used to cluster the unified features under a given distance threshold δ. The similarity between the normalized raw features is equivalent to the average of the video similarity and the text similarity, i.e.

[0034]

[0035] For threshold δ, DBSCAN produces clusters where k is the number of clusters produced based on the threshold clustering. The mean of the video features and text features in each cluster is calculated, i.e. the video center v and text center t of the ith cluster are obtained. For the given training data, the clustering centers are fixed and stored locally, and are used to construct positive sample pairs with more optimal supervision information during the training process. The core idea of pre-clustering based on raw features is shown in Figure 2 .

[0036] Specifically, due to the difference in modalities, the raw features of two different modalities cannot be directly calculated with each other, therefore, by normalizing and concatenating the raw features of the time-aligned video-text pair [v i , t i] into a computable form as a unified feature to calculate cosine similarity, and the inner product of the unified feature vector is equivalent to the average of the video and text similarity in value, so that the features representing a video and text sample can be clustered. The clustering algorithm uses DBSCAN, and a set of clustering centers with a certain granularity is generated by setting a threshold δ. After clustering, the average of the video and text features in the cluster is taken as the video and text modal center of the cluster, respectively, and the mean is used to avoid semantic misalignment in the video and text pairs. i ,t i ] and the sum of the products of the corresponding elements of [v j ,t j ], so that the similarity of the unified feature is numerically equivalent to the average of the video and text similarity, so that the features representing a video and text sample can be clustered. The clustering algorithm uses DBSCAN, and a set of clustering centers with a certain granularity is generated by setting a threshold δ. After clustering, the average of the video and text features in the cluster is taken as the video and text modal center of the cluster, respectively, and the mean is used to avoid semantic misalignment in the video and text pairs.

[0037] (4) Supervised sample construction based on pre-clustering guidance. In the training process, the construction of positive sample pairs is supervised by the pre-clustering result. Given a start and end time aligned training video segment v j and text subtitle t j , and extract the original features v j , t j , based on the pre-clustering center of the data set, based on the principle of nearest distance, assign pseudo-labels to the samples, video center label text center label Based on the pseudo-labels, positive sample pairs are constructed in two ways: ① video and text with the same pseudo-label are constructed as positive sample pairs; ② video sample, and start and end time aligned text of the same pseudo-label video, are constructed as positive sample pairs. The same applies to text samples.

[0038] Specifically, in the training process, the construction of positive sample pairs is supervised by the pre-clustering result. After sampling a batch of training samples, first construct positive samples based on the pre-clustering center. Take the cosine similarity as the measure, and assign the corresponding modal center sequence number with the highest similarity (i.e. the nearest distance) to each video and text as the pseudo-label. As shown in Figure 3 , positive samples are constructed in two ways. ① Direct construction: if video v1 and text t m both have pseudo-label i, then v1 and t m are considered as positive samples; ② Indirect construction: if video v1 and video v o both have pseudo-label i, then v1 and t o are considered as positive samples, and t1 and v o are considered as positive samples. Through these two ways, compared with the basic time alignment relationship, the semantic related positive samples are greatly expanded.

[0039] (5) Multi-granularity center weighted contrastive loss. Both K-Means clustering with specified cluster number and DBSCAN clustering based on similarity threshold involve the specification of hyperparameters. If only a specific parameter is manually set, the optimal effect cannot be achieved. Therefore, in order to accurately mine cross-modal supervision information, a set of different thresholds are used to perform DBSCAN clustering in pre-clustering. Under each threshold, different granularity of clustering centers are generated according to the sample distance. As the threshold increases, the sample distinction becomes stricter, thereby generating clusters with finer granularity, and vice versa. Given the threshold Δ = {δ1, δ2, …, δM} (where M is the number of thresholds), a plurality of pre-clustering centers will be generated M Given the threshold δm m , the corresponding pre-clustering centers are assigned labels according to the method in (2). For a sample v i , when the threshold is δm m , the positive sample text index set constructed based on pre-clustering guidance is Similarly, for a text t i , the positive sample video index set is The cross-modal contrastive loss is based on the InfoNCE loss function, where the positive samples are replaced by the constructed positive sample set. The loss is divided into two parts: video-to-text (v2t) and text-to-video (t2v). Taking the mth pre-clustering center as an example, they are defined as:

[0040]

[0041] where N b is the number of samples in a batch (i.e., batch size), and are the normalized common space embeddings, is a similarity measure function with a temperature coefficient τ. Then the loss function corresponding to the mth pre-clustering center is After weighting with the corresponding weight, the final loss function is:

[0042]

[0043] where α m is the weight of the loss constructed by the mth pre-clustering center. Different granularity of clustering centers have different definitions for the same pair of video and text samples, and have different effects of pulling or pushing. Therefore, the plurality of losses are weighted to form a voting-like form to jointly determine whether the sample pair is a true positive pair (true positive pairs) from different perspectives. Through the loss function for backpropagation, the gradient is calculated to update the projection model. ​

[0044] The purpose of using multiple pre-cluster centers with different granularities is to provide a broader perspective and obtain more accurate positive sample pairs. Specifically, at low thresholds, most semantically similar samples will cluster in the same cluster, forming coarse-grained clusters; at high thresholds, the sample distinctions are more stringent, the clusters are more dispersed, forming fine-grained clusters, which are relatively more accurate when constructing positive samples. Therefore, a relatively larger weight is given to the loss corresponding to a high threshold, and a smaller weight is given to a lower threshold, so that when multiple losses jointly determine whether samples are brought closer or pushed further apart, the effect of the high threshold is considered relatively more.

[0045] In practice, excessively high thresholds can generate a large number of clusters, which increases computational overhead. Therefore, through the implementation examples, six thresholds were ultimately selected for clustering, Δ = {0.35, 0.4, 0.45, 0.5, 0.55, 0.6}, with weights of {0.1, 0.1, 0.15, 0.15, 0.25, 0.25}.

[0046] 2. Model Input

[0047] During model training, such as Figure 1 As shown, video clips and text descriptions are both used as input to the model. Since the feature extractor is fixed, the original features can be extracted in advance and stored locally, and then loaded directly during training. During model inference, the trained model can extract and project video and text modal features separately, depending on the retrieval task. If text-to-video retrieval is performed, a query text is input each time for feature extraction and projection into a common space. The video features to be retrieved are pre-extracted as a database. The same applies to video-to-text retrieval.

[0048] 3. Model Training Process

[0049] (1) Loss Function. The loss function for training the model is the multi-granularity center weighted contrastive loss obtained by weighting multiple sets of pre-cluster centers, based on the InfoNCE loss function commonly used in contrastive learning. In cross-modal contrastive learning, the one-to-one positive sample pairs with start and end times aligned in the InfoNCE molecule are replaced with one-to-many positive sample pairs formed by the set of positive samples determined by the pre-cluster centers, minimizing the distance between all possible positive sample pairs, and vice versa.

[0050] (2) Training parameters. The parameter settings are consistent with the Everything at Once settings. The Adam optimizer is used to update the model parameters using gradients. The learning rate is set to 0.00005, and exponential learning rate decay is performed with a sparsity of 0.9.

[0051] (3) Model training. Since the feature extractor is fixed, to avoid the additional computational overhead caused by repeated extraction during the training process, the original un-pooled features are saved in advance. During training, according to the requirements of pre-clustering and model input, the original features are processed according to the operation described in the "original feature extraction". After the original features are projected by the trainable projection model, the loss function is calculated according to the multiple sets of positive samples determined by the pre-clustering center, and the projection model parameters are updated by back propagation after weighted combination.

[0052] 4. Model evaluation:

[0053] The commonly used indicators for retrieval, recall at K (Recall@K, K = 1, 5, 10) and median ranking (MedR), are used to evaluate the performance of the model. Recall@K evaluates the proportion of paired samples in the top K samples in all retrieval results, and the higher the better. MedR evaluates the median of the ranking position of all paired samples in the retrieval sequence, and the lower the better.

[0054] 5. Model deployment process

[0055] For the trained model, it is usually used for text-to-video retrieval, that is, the database contains a large number of videos (stored in the form of common space embedding), the input query text is used for feature extraction and projection, and then the obtained embedding is used for similarity calculation with the embedding in the database to achieve retrieval. When used in actual engineering, if there is a large amount of data, in order to further improve the retrieval efficiency, Faiss (Johnson J, Douze M, Jégou H. Billion-scale similarity search with gpus [J]. IEEE Transactions on Big Data, 2019, 7(3):535-547.) can be used to construct high-efficiency index to achieve the purpose of fast retrieval.

Claims

1.A video-text cross-modal retrieval method based on pre-clustering guidance, characterized in that The method comprises the following steps: 1) a feature extractor extracts video and text original features of a training data set; 2) normalized original features after time alignment are spliced as unified features, a set of threshold values are set for multiple DBSCAN clustering, and multiple clustering groups are generated; The setting of a set of threshold values for multiple DBSCAN clustering is a pre-clustering based on original features, and the specific steps are as follows: ① Video and text original features are subjected to L2 normalization and splicing to represent unified features of a pair of start and end time aligned video and text; ② Given a threshold value δ, the unified features are subjected to DBSCAN clustering, and the similarity is equivalent to the average of video similarity and text similarity, so that the unified features can be compared; 3) The video and text features of each clustering group are respectively calculated as clustering centers; 4) A batch of training samples are sampled, for each clustering center, the nearest center is found to construct a pseudo label, and a video and text positive sample pair is constructed based on the pseudo label; The construction of the video and text positive sample pair is based on the positive sample pair constructed by the pre-clustering center, and the specific steps include: ① Based on the similarity of video and text original features and video and text clustering centers, the most similar center is selected to assign a pseudo label; ② Directly construct a positive sample pair: video and text with the same pseudo label are used as a positive sample pair; ③ Indirectly construct a positive sample pair: each video assigns its start and end time aligned text to other videos with the same pseudo label, and each text assigns its start and end time aligned video to the text expected to have the same pseudo label; 5) Project the original features into a common space, calculate the loss based on the positive samples corresponding to the multiple pre-clustering centers, and combine the loss by weighting, and update the projection model by back propagation; The specific steps of calculating the loss based on the positive samples corresponding to the multiple pre-clustering centers are as follows: ① For a set of threshold-determined clustering centers, after constructing the positive samples in step 4), each video and text has its corresponding positive sample set, based on the commonly used InfoNCE loss in contrast learning, the one-to-one positive sample based on cross-modal contrast learning is expanded to one-to-many positive sample determined by the positive sample set; ② Each clustering center corresponds to a loss, and according to the set weight, the sum of all losses is weighted to obtain the final contrast loss; ③ The loss function is back propagated to calculate the gradient, and the model parameters are updated; 6) Repeat steps 4) to 5) until a predetermined number of training rounds is reached.

Images