Multimodal transfer learning framework for video corpus segment retrieval

Abstract

A multi-modal transfer learning framework for video corpus segment retrieval, the learning framework is trained by a segmented pre-training transformer using image-text pairs, video-text pairs and a VCMR dataset to transfer the knowledge of ITR and VR to VCMR, so that better performance can be achieved on a relatively small VCMR dataset, comprising: a retrieval module, the retrieval module comprises a text encoder, a visual encoder and a video retrieval module; the text encoder adopts a segmented pre-training transformer to generate a query representation, and encodes the query representation to generate a text representation; the visual encoder is used to generate a video representation by adopting a visual transformer; the video retrieval module is used to retrieve video segments; a segment positioning module, the segment positioning module introduces L1 loss and IoU loss to directly predict the time span in a fine-grained manner.

Description

Technical fields:

[0001] This invention relates to a multimodal transfer learning framework for video corpus segment retrieval. Background technology:

[0002] Deep learning has facilitated the development of cross-modal retrieval between visual and textual cues. With the help of large-scale pre-training techniques, significant progress has been made in image-text retrieval tasks, allowing for a focus on more challenging cross-modal tasks—video corpus fragment retrieval, which includes three representative subtasks: video retrieval (VR), single video fragment retrieval (SVMR), and video corpus fragment retrieval (VCMR). VR aims to retrieve the corresponding video from a video corpus based on a given text query, while SVMR retrieves a short segment from a given video.

[0003] To better match real-world applications, such as query-based video surveillance, search, and navigation in video corpora, more and more researchers are exploring Video Corpus Segment Retrieval (VCMR), which aims to retrieve and locate short segments from large video corpora.

[0004] Existing VCMR methods utilize offline knowledge from frozen high-level pre-trained models based on classification frameworks, namely RoBERTa for language and ResNet152 and I3D for video retrieval. Due to deficiencies in the retrieval content and the pre-trained models, the extracted video and text features are of poor quality, affecting the prediction of video segments and failing to adapt well to VCMR tasks, thus failing to meet the practical needs of VCMR. Furthermore, the prediction training models based on classification frameworks are coarse-grained, which affects the accuracy of segment retrieval, failing to meet the required precision. Consequently, it is difficult to predict the accurate time step of the target segment, limiting segment retrieval to text-based retrieval within a single video and preventing the handling of multiple videos. Summary of the Invention:

[0005] This invention provides a multimodal transfer learning framework for video corpus segment retrieval. Its rationally designed structure and fine-grained regression prediction improve the accuracy of model prediction and retrieval, accurately predicting the precise time step of the target segment. It also provides a segmented pre-trained transformer to transfer ITR and VR knowledge to VCMR, improving the quality of extracted video and text features. Employing image and text data, video data, and a staged pre-trained model, it achieves better performance on a relatively small VCMR dataset. This framework can be applied to tasks such as image retrieval, video retrieval, video segment localization, and video library segment localization. Its general-purpose nature further enhances retrieval performance and addresses problems existing in current technologies.

[0006] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0007] A multimodal transfer learning framework for video corpus segment retrieval, wherein the learning framework is trained by a segmented pre-trained transformer using image-text pairs, video-text pairs, and VCMR datasets to transfer knowledge from ITR and VR to VCMR, enabling it to achieve better performance on a relatively small VCMR dataset, including:

[0008] The retrieval module includes a text encoder, a visual encoder, and a video retrieval module. The text encoder uses a segmented pre-trained transformer to generate a query representation and encodes the query representation to generate a text representation. The visual encoder uses a visual transformer to generate a video representation. The video retrieval module is used to retrieve video segments.

[0009] The segment localization module introduces L1 loss and IoU loss to directly predict the time span in a fine-grained manner.

[0010] The segmented pre-training transformer includes three stages: ITR, VR, and VCMR. Through staged pre-training, the knowledge of ITR and VR is transferred to VCMR. The retraining of ITR enhances the model's visual-language alignment ability, and the pre-training of VR enhances the retrieval ability of the video retrieval module.

[0011] The segment localization module directly learns the segment position using L1 loss and IoU loss.

[0012] In the video retrieval module, query q is compared with each candidate video in the video corpus. The similarity between them, i.e.

[0013]

[0014] A segmented pre-trained transformer is used to calculate the intra-batch visual text similarity of randomly sampled video text. The intra-batch query for visual similarity is:

[0015]

[0016] Contrast loss is defined as cross-entropy

[0017]

[0018] Where H(·) represents the cross-entropy, y q2v and y v2q These represent the similarity from a hot query to the visual representation and the similarity from the visual representation to the query, respectively.

[0019] In the segment localization module, the query representation q and the video representation V are first fused by concatenating the query representation with each frame representation in V:

[0020]

[0021] Where [·, ·] denotes a chain operation, and f(·) represents the summation of each term of the variable.

[0022] Using a multilayer perceptron with a nonlinear activation function to predict time spans:

[0023]

[0024] Where MLP(·) represents a multilayer perceptron with a nonlinear activation function, and T contains the start and end times of the predicted segment (T0). s and T e );

[0025] The instantaneous position loss, consisting of L1 loss and IoU loss, is defined as:

[0026]

[0027] in Indicates the actual time span;

[0028] IoU loss is defined as follows:

[0029]

[0030] The definitions of Inter and Union are as follows:

[0031] Inter = max(τ) e T e )-max(τ s T s ),

[0032] Union=(τ e -τ s )+(T e -T s )-Inter.

[0033] The text encoder uses a segmented pre-trained transformer based on a self-attention mechanism to produce text representations.

[0034] This invention employs the aforementioned structure, transferring knowledge from ITR and VR to VCMR to achieve better performance on a relatively small VCMR dataset. It trains a segmented pre-trained transformer using staged learning for retrieval and localization across image-text pairs, video-text pairs, and the VCMR dataset. A text encoder generates query and text representations, while a visual encoder uses a visual transformer to generate video representations. The three stages of the segmented pre-trained transformer enhance the model's visual-language alignment and retrieval capabilities, offering advantages of simplicity, practicality, accuracy, and efficiency. Attached image description:

[0035] Figure 1 This is a schematic diagram of the structure of the present invention.

[0036] Figure 2 This is a schematic diagram of the retrieval module of the present invention. Detailed implementation method:

[0037] To clearly illustrate the technical features of this solution, the invention will be described in detail below through specific implementation methods and in conjunction with the accompanying drawings.

[0038] like Figure 1-2 The multimodal transfer learning framework for video corpus segment retrieval, shown in the figure, is trained by a segmented pre-trained transformer using image-text pairs, video-text pairs, and VCMR datasets to transfer knowledge from ITR and VR to VCMR, enabling it to achieve better performance on a relatively small VCMR dataset. The framework includes:

[0039] The retrieval module includes a text encoder, a visual encoder, and a video retrieval module. The text encoder uses a segmented pre-trained transformer to generate a query representation and encodes the query representation to generate a text representation. The visual encoder uses a visual transformer to generate a video representation. The video retrieval module is used to retrieve video segments.

[0040] The segment localization module introduces L1 loss and IoU loss to directly predict the time span in a fine-grained manner.

[0041] The segmented pre-training transformer includes three stages: ITR, VR, and VCMR. Through staged pre-training, the knowledge of ITR and VR is transferred to VCMR. The retraining of ITR enhances the model's visual-language alignment ability, and the pre-training of VR enhances the retrieval ability of the video retrieval module.

[0042] The segment localization module directly learns the segment position using L1 loss and IoU loss.

[0043] In the video retrieval module, query q is compared with each candidate video in the video corpus. The similarity between them, i.e.

[0044]

[0045] A segmented pre-trained transformer is used to calculate the intra-batch visual text similarity of randomly sampled video text. The intra-batch query for visual similarity is:

[0046]

[0047] Contrast loss is defined as cross-entropy

[0048]

[0049] Where H(·) represents the cross-entropy, y q2v and y v2q These represent the similarity from a hot query to the visual representation and the similarity from the visual representation to the query, respectively.

[0050] In the segment localization module, the query representation q and the video representation V are first fused by concatenating the query representation with each frame representation in V:

[0051]

[0052] Where [·, ·] represents a concatenation operation, and f() represents summing each term of the variable.

[0053] Using a multilayer perceptron with a nonlinear activation function to predict time spans:

[0054]

[0055] Where MLP(·) represents a multilayer perceptron with a nonlinear activation function, and T contains the start and end times of the predicted segment (T0). s and T e );

[0056] The instantaneous position loss, consisting of L1 loss and IoU loss, is defined as:

[0057]

[0058] in Indicates the actual time span;

[0059] IoU loss is defined as follows:

[0060]

[0061] The definitions of Inter and Union are as follows:

[0062] Inter = max(τ) e T e )-max(τ s T s ),

[0063] Union=(τ e -τ s )+(T e -T s )-Inter.

[0064] The text encoder uses a segmented pre-trained transformer based on a self-attention mechanism to produce text representations.

[0065] The working principle of the multimodal transfer learning framework for video corpus segment retrieval in this embodiment of the invention is as follows: using fine-grained regression prediction can improve the accuracy of model prediction and retrieval, accurately predict the accurate time step of the target segment, and provide a segmented pre-trained transformer to transfer knowledge of ITR and VR to VCMR, improving the quality of extracted video and text features. By using image and text data, video data, and a staged pre-trained model, it can achieve better performance on a relatively small VCMR dataset. It can be applied to tasks such as image retrieval, video retrieval, video segment localization, and video library segment localization. The general-purpose framework can further improve retrieval performance.

[0066] In existing technologies, a cross-modal temporal regression locator model is proposed to jointly model text queries and video segments, outputting the boundaries of candidate segments and their alignment scores to complete single video segment retrieval. However, it can only retrieve segments based on text within a single video and cannot process multiple videos. Furthermore, the quality of the extracted features is poor, resulting in low prediction accuracy for video segments.

[0067] The existing technology also proposes the ReLoCLNet model, which mainly extracts features through a transformer and then adapts them to various downstream tasks through a simple structure; it uses frozen pre-trained models to extract features, which are not well adapted to VCMR tasks and cannot be used to train the model with data from related tasks; it uses classification-based fragment retrieval, and this coarse-grained retrieval affects the accuracy of fragment retrieval by the model.

[0068] The overall solution mainly includes a retrieval module, which comprises a text encoder, a visual encoder, and a video retrieval module. The text encoder uses a segmented pre-trained transformer to generate a query representation and encodes the query representation to generate a text representation. The visual encoder is used to generate a video representation using a visual transformer. The video retrieval module is used to retrieve video segments. The segment localization module introduces L1 loss and IoU loss to directly predict the time span in a fine-grained manner.

[0069] The learning framework in this application is trained by a segmented pre-trained transformer using image-text pairs, video-text pairs, and VCMR datasets to transfer knowledge from ITR and VR to VCMR, enabling it to achieve better performance on a relatively small VCMR dataset.

[0070] Since existing VCMR datasets contain only about 10k-20k pairs of videos, queries, and time spans, which is an order of magnitude smaller than typical parallel datasets used for image text retrieval and video text retrieval, data becomes a bottleneck for training effective VCMR models, forcing these models to use coarse-grained classification for localization. To alleviate this problem, it is necessary to pre-train models stepwise from image text retrieval datasets and video text retrieval datasets and transfer the learned knowledge to VCMR tasks.

[0071] For segmented pre-trained transformers, large-scale image and text and video text corpora are used to improve video corpus segment retrieval. Specifically, it includes three stages: ITR, VR, and VCMR. Through staged pre-training, the knowledge of ITR and VR is transferred to VCMR. The retraining of ITR enhances the model's visual-language alignment ability, and the pre-training of VR enhances the retrieval ability of the video retrieval module.

[0072] To verify the effectiveness of the learning framework, as shown in the figure below, experiments were conducted on three datasets: ANetCaps, DiDeMo, and TVR. The framework outperformed all previous models on all three benchmarks across all metrics and reached state-of-the-art performance.

[0073]

[0074]

[0075]

[0076] Within the overall structure, various formulas are used to perform corresponding calculations in order to improve the practicality and accuracy of the learning framework.

[0077] It is worth noting that the general-purpose learning framework of this application has good performance, uses fine-grained regression for prediction, improves the upper limit of model prediction accuracy, and can be widely applied to various models; at the same time, it uses image and video data and pre-trains in stages, enabling the model to acquire good capabilities from data in other fields and improve the performance of this task.

[0078] In summary, the multimodal transfer learning framework for video corpus segment retrieval in this embodiment of the invention uses fine-grained regression prediction, which can improve the accuracy of model prediction and retrieval, accurately predict the precise time step of the target segment, and provide a segmented pre-trained transformer to transfer ITR and VR knowledge to VCMR, improving the quality of extracted video and text features. By using image and text data, video data, and a staged pre-trained model, it can achieve better performance on a relatively small VCMR dataset. It can be applied to tasks such as image retrieval, video retrieval, video segment localization, and video library segment localization. The general-purpose framework can further improve retrieval performance.

[0079] The above specific embodiments should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, any alternative improvements or modifications made to the embodiments of the present invention shall fall within the scope of protection of the present invention.

[0080] Any aspects of this invention not described in detail are well-known to those skilled in the art.

Claims

1. A multi-modal transfer learning framework for video corpus segment retrieval, characterized in that, The learning framework is trained using a segmented pre-trained transformer with image-text pairs, video-text pairs, and VCMR datasets to transfer knowledge from ITR and VR to VCMR, enabling better performance on a relatively small VCMR dataset, including: The retrieval module includes a text encoder, a visual encoder, and a video retrieval module. The text encoder uses a segmented pre-trained transformer to generate a query representation and encodes the query representation to generate a text representation. The visual encoder uses a visual transformer to generate a video representation. The video retrieval module is used to retrieve video segments. The segment localization module introduces L1 loss and IoU loss to directly predict the time span in a fine-grained manner; In the video retrieval module, the similarity between the query q and each candidate video in the video corpus is computed, i.e. sim(q, v) ； A segmented pre-trained transformer is used to calculate the intra-batch visual text similarity of randomly sampled video text. The intra-batch query for visual similarity is: ； Contrast loss is defined as cross-entropy ； Where H(·) represents the cross-entropy, y q2v and y v2q These represent the visual similarity from a hot query and the visual similarity from the query, respectively. In the segment localization module, the query representation q and the video representation V are first fused by concatenating the query representation with each frame representation in V: ； Where [·,·] represents a chain operation, and f() represents summing over each term of the variable; Using a multilayer perceptron with a nonlinear activation function to predict time spans: ； where MLP() represents a multi-layer perceptron with a non-linear activation function, T contains the start and end times of the predicted segment T s and T e .

2. The multi-modal transfer learning framework for video corpus segment retrieval according to claim 1, wherein: The segmented pre-training transformer includes three stages: ITR, VR, and VCMR. Through staged pre-training, the knowledge of ITR and VR is transferred to VCMR. The retraining of ITR enhances the model's visual-language alignment ability, and the pre-training of VR enhances the retrieval ability of the video retrieval module. 3.The multi-modal transfer learning framework for video corpus segment retrieval of claim 1, wherein: The segment localization module directly learns the segment position using L1 loss and IoU loss.

4. The multi-modal transfer learning framework for video corpus segment retrieval according to claim 1, wherein: The text encoder uses a segmented pre-trained transformer based on a self-attention mechanism to produce text representations.

Images