Generating captions for videos using multiple cross-modal teachers

Abstract

Automatic caption generation pipelines and methods for automatically annotating video data with speech captions that can be obtained using automatic speech recognition (ASR). Automatic caption generation pipelines with multi-modal data inputs expand the dataset of high-quality video-caption pairs. Automatic caption generation pipelines generate video-caption pairs by building and using large video-linguistic datasets and utilizing automatic caption generation methods that leverage multi-modal inputs such as textual video descriptions, speech captions, and individual video frames.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Application Serial No. 18 / 422,721, filed January 25, 2024, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] This topic relates to the automatic captioning generation of large amounts of visual data (such as videos) with textual descriptions. Background Technology

[0004] The quality of data and annotations sets the upper limit for the quality of downstream models. While a large number of text corpora and image-text pairs exist, high-quality video-text data remains difficult to collect. Attached Figure Description

[0005] The accompanying drawings are not necessarily drawn to scale, and the same figures may depict similar parts in different views. The drawings show some non-limiting examples, in which:

[0006] Figure 1 Includes illustrations depicting examples of caption generation using large-scale datasets and example datasets based on this disclosure;

[0007] Figure 2 This is a flowchart describing the steps of an automatic caption generation method using a pipeline;

[0008] Figure 3 It is a chart that includes a list of eight high-performing teacher models;

[0009] Figure 4 This is a diagram depicting a student caption generation model that includes visual and text branches utilizing multimodal inputs, including both visual and textual elements.

[0010] Figure 5 It is a diagram depicting the predictions of caption generation with and without text input;

[0011] Figure 6 This is a diagram depicting a video sample generated as an example;

[0012] Figure 7 It is a chart depicting the distribution of video lengths after video segmentation; and

[0013] Figure 8 This is a block diagram of a machine in which instructions for performing any one or more of the methods described herein can be executed. Detailed Implementation

[0014] This disclosure includes examples of an automatic captioning generation pipeline and methods for automatically annotating video data with subtitles, which can be obtained using automatic speech recognition (ASR). The automatic captioning generation pipeline with multimodal data input expands the dataset for high-quality video caption pairs. The automatic captioning generation pipeline generates video-caption pairs by building and using large video-language datasets and automatic captioning generation methods that leverage multimodal inputs such as textual video descriptions, captions, and individual video frames.

[0015] A large number of high-resolution videos can be curate from publicly available datasets, where the videos are segmented into semantically consistent video segments, and then multiple cross-modal teacher models are applied to obtain captions for each video segment. The retrieval model is fine-tuned on a relatively small subset of video segments, where the best caption for each video segment is manually selected, and then the retrieval model is applied to its entire dataset to select the best caption as annotation.

[0016] In the example described herein, 70 million (70 million, 70M) video pairs with high-quality text captions were generated, commonly referred to in this disclosure as the Panda-70M dataset. The value of the dataset is demonstrated on three downstream tasks: video captioning generation, video and text retrieval, and text-driven video generation. Models trained on the data scored significantly higher on most metrics across all tasks.

[0017] Additional objects, advantages, and novel features of the example will be set forth in part in the description which follows, and will in part become apparent to those skilled in the art upon examination of the following description and drawings, or may be learned by production or operation of the example. The objects and advantages of this subject matter may be realized and obtained by means of the methods, tools, and combinations particularly pointed out in the appended claims.

[0018] In the following detailed description, numerous specific details are illustrated by way of example to provide a comprehensive understanding of the teachings. However, those skilled in the art will understand that these teachings can be practiced without these details. In other instances, well-known methods, processes, components, and circuits have already been described at a relatively high level without detail to avoid unnecessarily obscuring aspects of these teachings.

[0019] As used herein, the term "coupled" refers to any logical, optical, physical, or electrical connection, link, etc., through which a signal or light generated or provided by one system element is transmitted to another coupled element. Unless otherwise stated, coupled elements or devices are not necessarily directly connected to each other and may be spaced apart by intermediate components, elements, or communication media that can modify, manipulate, or carry light or signals.

[0020] Now, let’s refer in detail to the example shown in the attached figures and discuss it below.

[0021] Large-scale multimodal learning typically requires massive amounts of data and heavy computational demands. Most breakthroughs are achieved through large-scale computing infrastructure, large-scale models, and massive amounts of data. Powerful text-to-image and image-to-text models exist due to these three components. Scaling model size or computational requirements is challenging and expensive; however, these challenges can often be addressed within a limited amount of engineering time. Scaling data is relatively more challenging because humans need to spend time analyzing each sample.

[0022] The computing community has amassed datasets containing millions, even billions, of representative image-text pairs. In contrast, video-text pairs are much harder to obtain. First, annotating videos is more time-consuming because annotators need to watch the entire video before tagging it. Second, videos typically contain multiple scenes or events stitched together and include content that changes over time. Finally, the metadata itself (such as dialogue captions, video descriptions, or narration) tends to be too broad and not properly aligned temporally, and fails to accurately describe the video. For example, ASR annotated several datasets in the 100 million (100M) size (such as HD-VILA-100M and HowTo100M). However, these datasets (such as...) Figure 1 The HD-VILA-100M dataset (shown at position 100) typically includes dialogue subtitles that do not include the main content and actions presented in the video. This limits the value of such datasets in multimodal training. Table 1 summarizes some publicly available datasets. Some have lower resolution, some use ASR for subtitle generation, some contain data from limited domains, some are small-scale, and some provide short subtitles.

[0023]

[0024] Table 1

[0025] This disclosure includes examples of caption generation pipelines that automatically annotate video data with dialogue captions to create large-scale datasets generated by ASR. The automatic caption generation pipelines expand the dataset of high-quality video-caption pairs through multimodal data input. In the examples, the Panda-70M dataset is generated. Some samples from the Panda-70M dataset are... Figure 1 As shown at point 102 in the document. The Panda-70M dataset comprises high-resolution videos from open (public) domain sources with rich captions, averaging 13.2 words per caption. While manually annotating 70M videos is very expensive and time-consuming, the examples in this disclosure utilize automatic annotation. An important insight is that videos often contain information from multiple modalities that can aid automatic captioning utilities. This includes the video's title, description, dialogue captions, individual still frames, and the video itself. The value of this data cannot be fully maximized when only one modality is used. In contrast, this disclosure utilizes different combinations of multimodal data as input to multiple cross-modal teachers. By generating captions for subsets of videos using different models and then presenting the results to humans for evaluation, it can be seen that no single model is able to generate good captions for more than 35% of the videos. However, by jointly collecting all captions from different models, it can be observed that 88.8% of the videos can be annotated with at least one good caption.

[0026] Figure 2 This is a flowchart 201 depicting the steps of an example method for constructing an automatic captioning pipeline 200 for the Panda-70M dataset by selecting 3.8M high-resolution videos 202 from the HD-VILA-100M dataset and processing them using the following three steps. First, a semantically aware video segmentation algorithm 204 segments the long video 202 into semantically consistent segments 206, striking a balance between semantic coherence and the duration of the video segments. A total of 70M semantically consistent video segments are obtained. Second, a series of cross-modal teacher models 208, including an image captioning generation model and an image / video visual question answering (VQA) model (such as video descriptions and dialogue captions) with additional text input 210, are used to predict several candidate captions for the video segments. Finally, a dataset of 100,000 (100K) videos is collected, where human annotators select the best captions for each video. This dataset is then used to fine-tune a fine-grained video-to-text (V2T) retrieval model 212 and select precise captions for annotating the entire dataset.

[0027] Running multiple teacher models 208 on all data can be relatively expensive and time-consuming. To achieve efficient video annotation at scale, in one example, the system uses a trained student captioning generation model to extract knowledge from the teacher models 208. The student model employs a two-branch architecture that accepts both visual and textual input to leverage multimodal information for the captioning process.

[0028] Extensive experiments demonstrate that pre-training on the Panda-70M dataset facilitates several downstream tasks, including video captioning generation, video and text retrieval, and text-driven video generation. Training the student model in a knowledge extraction manner helps develop a powerful student model, as shown in Table 4 (below), which outperforms any single teacher model by more than 7.7%, with additional text inputs (such as video descriptions and dialogue captions) further enhancing performance.

[0029] The desired video samples in the video captioning generation dataset should possess two somewhat contradictory characteristics. On the one hand, the videos should be semantically consistent so that the captions accurately express their semantic content without ambiguity. On the other hand, the videos cannot be too short or fragmented to contain meaningful motion content, which is beneficial for downstream tasks such as text-to-video generation. To achieve these two goals, in some examples, the system uses a two-stage semantically aware segmentation algorithm 204 to cut long videos 202 into semantically coherent segments 206. In the first stage, the long video 202 is segmented based on cut scene detection, as semantics typically change when a new scene begins. In the second stage, if the first stage incorrectly segmented adjacent segments, the adjacent segments are stitched together to ensure that the final video is not too short. For this purpose, ImageBind, available from Meta Inc. in Menlo Park, California, can be used to select video frame embeddings, and adjacent embedded segments are merged if the last frame of the previous segment and the first frame of the next segment are close. Additional processing steps include: 1) long videos without any cut scenes; 2) videos with fade-in or fade-out transitions, which are generally not detected as cut scenes; and 3) removing redundant segments to increase the diversity of the dataset.

[0030] In the first stage (Stage 1), segmentation is performed based on cut-scene detection. In the example, a tool for detecting shot changes in a video is used to segment the long video 202 into video segments 206. This tool automatically segments the video into individual segments (e.g., PySceneDetect). A utility such as ContentDetector compares the content differences between adjacent frames to a set threshold / score, and if the threshold / score is exceeded, a scene cut is triggered. cutscene_threshold is set to 25, and min_scene_len is set to 15 frames. A two-step post-processing algorithm handles 1) long videos 206 with transitions, such as fade-in and fade-out effects, which cannot be detected as cut-scenes, or 2) unedited shots that do not contain any cut-scenes.

[0031] In the first step, the maximum length of video segment 206 is set to 5 seconds. If video segment 206 is longer than the maximum length, the first 5 seconds are recursively cut into new video segments 206 until the remaining video meets the condition. Video segments 206 should be semantically consistent. Therefore, ImageBind features are extracted at the start and end frames, and video segment 206 is deleted if two frames are perceptually different. Specifically, given n video segments C, let frame C be... A and C B Extracting features f(C) A ) and f(C B ), Frame C A and C B This is the number of video clip 206. and The frame. If it satisfies If the transition effect is not present, then video segment 206 is retained. Therefore, video segment 206 is excluded from the transition effect, or has significantly different semantics at the beginning and end.

[0032] In the second stage (Stage 2), splicing is performed based on semantic similarity. To avoid segmenting the long video 202 as in Stage 1, adjacent video segments 206 are spliced ​​together if they are semantically similar. Formally, given two adjacent video segments C1 and C2 in sequence, if they satisfy... They are then concatenated into video clip 206.

[0033] Post-processing is performed through the following steps to stabilize the quality and diversity of video clip 206. First, video clips 206 that are shorter than 2 seconds or contain only slight motion (i.e., ...) are processed. Videos longer than 60 seconds were excluded. Only the first 60 seconds were retained. Next, each video segment 206 was represented by the average ImageBind feature extracted from Stage 1, and only video segments 206 that were semantically different from previous video segments 206 (i.e., Euclidean distance > 0.3) were retained to increase the diversity of the video samples. Finally, the first and last 10% of video segments 206 were pruned as the beginning and end, which typically contained unstable camera movements or transition effects. Using the segmentation algorithm, 3,790,459 long videos 202 were segmented into 70,817,315 video segments 206, with an average segment duration of 8.477 seconds. Figure 7 The image shows a distribution chart of video lengths.

[0034] To quantitatively validate the semantic consistency of video clip 206, maximum running LPIPS was used, which highlights the most significant perceptual changes within video clip 206. Formally, given a video clip 206 of n seconds in length, video frames are sampled twice per second, and keyframes are represented as {f1, ..., f...} n The maximum running LPIPS is shown in Formula 1 below:

[0035] (1)

[0036] Where LPIPS(·, ·) represents the perceptual similarity between two images. As shown in Table 2, compared with segmentation based on dialogue caption sentence alignment, the segmentation achieves better semantic consistency while maintaining a longer video length than ordinary cut scene detection.

[0037]

[0038] Table 2

[0039] The videos in the HD-VILA-100M dataset contain rich multimodal information, which is beneficial for video captioning generation. Therefore, in addition to the video itself, textual information, such as video titles, descriptions, dialogue subtitles, and still images, such as individual video frames, can also be used for captioning generation. Based on this observation, several captioning generation models can be utilized using inputs from different modalities.

[0040] An automatic captioning pipeline 200 employs multiple cross-modal teacher models 208. In the example, a large pool comprising 31 captioning generation teacher models 208 is used. Since inferring all teacher models 208 across 70M video clips is prohibitively costly, a short list of eight well-performing teacher models 208 was constructed based on user research. This list is shown in... Figure 3On the y-axis. Teacher model 208 consists of five base models with different pre-trained weights and input information. The five base models shown in Table 3 include Video LLaMA (Video VQA), VideoChat (Video VQA), VideoChat Text (a natural language model that first textualizes the video content and summarizes it using a large language model (LLM), BLIP-2 (Image Caption Generation), and MiniGPT-4 (Image VQA). To achieve video caption generation through teacher model 208 with different modalities, different caption generation processes were developed for each modality. For example, for the VQA model, in addition to visual data, input prompts are provided, including additional textual information such as video descriptions and dialogue captions, and teacher model 208 is required to summarize all multimodal inputs into a single sentence.

[0041]

[0042] Table 3

[0043] The caption generation process for each teacher model 208 outlines the inference details for each base model. Three suitable base models are Video-LlaMA, VideoChat, and VideoChat Text. These models will be described sequentially below.

[0044] Video-LlaMA: For all teacher models 208, the system uses a visual branch instead of an audio branch and uses Vicuna-7B as the LLM. Two official weights are used, including pre-trained weights trained on 2.5M video-text pairs and LLaVA-CC3M, and fine-tuned weights further fine-tuned on instruction-tuning data.

[0045] VideoChat: The system uses Vicuna-7B as the LLM, and the rest of the settings follow the official codebase.

[0046] VideoChat Text: This model is an algorithm based on natural language processing (NLP). It first textualizes the video content into video tags using a Swing Transformer, then textualizes dense subtitles using GRiT, and finally textualizes general subtitles using a T5 language model. The original codebase uses ChatGPT-4 as a chatbot to implement VQA. LLaMA can be used for large-scale subtitle generation.

[0047] Teacher models 208, using different modalities, performed well on various types of videos. For example, video teacher models 208 performed better on videos with complex dynamics due to the additional module that processes temporal information. On the other hand, image teacher models 208 accurately generated captions for videos with rare and uncommon objects because they were trained using a large dataset of image-text pairs. Finally, VQA teacher models 208 had an advantage for visually difficult-to-understand videos because they could take on additional text input. This was supported by numerical evaluation, in which a user study was conducted, asking participants to select the best captions from eight teacher model candidates. Figure 3 The selection rate of each teacher model was plotted. The results show that the best captions were generated by various teacher models 208. On the other hand, the highest selection rate of a single teacher BLIP-2 with opt6.7b was only 17.85%, which means that the single teacher model 208 has limited caption generation capabilities across various videos.

[0048] Given multiple candidate captions for a video, the caption best aligned with the video content is used. Commonly available retrieval methods often fail to select the optimal result. One reason is that these general models are trained using a contrastive learning objective. They are required to distinguish one sample from completely unrelated samples. In contrast, in this disclosure, all candidate captions are highly correlated with the video samples, and the teacher model 208 is required to discern subtle differences within each caption to achieve optimal performance.

[0049] Retrieval Model 212 was customized for "fine-grained" retrieval scenarios, collecting a subset of 100K videos for which human annotators selected captions containing the most accurate and detailed information about the main content of the videos. This subset was split to obtain 98K training samples and 2K test samples. Fine-tuning was performed on this dataset using Unmasked Teacher (UMT), and hard negative mining was implemented on the contrastive loss, where the seven captions not selected by the annotators constituted the hard negative samples and were assigned greater training weights.

[0050] Quantitative evaluations of the retrieval performance of UMT were conducted on the test set with and without fine-tuning. Experiments show that the fine-tuned UMT achieves an R@1 accuracy of 35.90%, significantly outperforming the pre-trained UMT, which has an R@1 accuracy of 21.82%. It should be noted that another human-consistent annotation method only achieved an R@1 accuracy of 48%, because retrieval becomes subjective when more than one equally well-performing caption exists. Figure 3 As shown, the finely tuned UMT can select subtitles to distribute in a manner similar to human-selected subtitles.

[0051] While the automatic captioning pipeline 200 can generate accurate captions, its heavy computational requirements may hinder its ability to scale to larger datasets. For example, it might require 8+1 different models to annotate a single video segment. According to this disclosure, a student captioning generation model is trained on the Panda-70M dataset to extract knowledge from multiple teacher models 208.

[0052] like Figure 4 As shown, the student caption generation model 400 includes a visual branch 402 and a text branch 404 utilizing multimodal inputs including visual and textual data. For the visual branch 402, Video-LLaMA is used to select LLM-compatible video embeddings. For the text branch 404, the text embeddings selected by the text encoder 406 are directly input into the LLM 408. However, this can lead to two problems: first, the text prompts with video descriptions and dialogue captions may be too long, affecting the decisions of the LLM 408 and increasing the computational burden; second, the information from the descriptions and dialogue captions may be noisy and not necessarily aligned with the video content. To address these issues, the example system includes a text query transformer (Q-former) 410, which selects text embeddings with a fixed length and better bridges the video and text embeddings. The text query transformer 410 has the same architecture as the query transformer in BLIP-2. During the training of the student caption generation model, gradient propagation from the text branch 404 to the visual branch 402 is blocked, and the visual encoder 412 and the video query transformer & linear layer 414 based on the video input are trained.

[0053] Some samples from the Panda-70M dataset, such as Figure 1 As shown in Figure 102. To quantitatively evaluate the effectiveness of the Panda-70M dataset, the pre-training performance was tested on three downstream applications: video captioning generation, video and text retrieval, and video generation.

[0054] To evaluate the performance of the student caption generation model 400 in video caption generation, Video-LLaMA and the visual branch 402 were used as the base models. Two pre-trained weights were compared: the official weights, which were jointly trained on 2.5M video-text pairs and 595K image-text pairs; and weights trained from scratch on the Panda-2M dataset. The Panda-2M dataset is a randomly sampled subset of the Panda-70M dataset and shares the same number of training samples as the official weights. The student caption generation model 400 was also trained separately on the full Panda-70M dataset for the video branch 402 and the text branch 404 for better caption generation performance. For all models, the same backbone was used, including Vicuna-7B as LLM 408, ViT as the visual encoder 412, a video query transformer, and a linear projection layer from MiniGPT-4. For the Panda-2M pre-training, only video and caption data were used for a fair comparison without using other text information. In order to train the student caption generation model 400, in addition to the video, metadata and dialogue captions were also randomly input into the student caption generation model 400.

[0055] Zero-shot video caption generation was tested on two benchmarks: MSR VTT and MSVD. For MSR-VTT, it consists of 10,000 videos, each with 20 manually annotated captions, and reports results for 2,990 test segments. For MSVD, it consists of 1,970 videos with a total of 80,000 descriptions, and reports the numbering of 670 test videos. Note that no training or validation videos were used in the downstream dataset. To quantitatively evaluate the quality of the output captions, common protocols were followed and BLEU-4, ROGUE-L, METEOR, and CIDEr were reported. All metrics were calculated using the pycocoevalcap package. BERTScore was also calculated to evaluate the contextual similarity of each symbol in the ground truth and predicted captions. The results are shown in Table 4. For fair comparison, no additional textual information was input to the student model 400 during inference on the downstream dataset. Figure 5 Video samples from the test set of the Panda-70M dataset and several captions predicted from different teacher models 208 are shown at position 500 for qualitative comparison.

[0056]

[0057] Table 4

[0058] As shown in Table 4, Video-LLaMA with Panda-2M pre-trained weights achieves significantly better performance compared to the official weights. Numerically, these pre-trained weights show improvements of 18.5% and 19.6% in B-4 compared to MSR-VTT and MSVD, respectively. Figure 5 As shown, the subtitles from the original Video-LLaMA contain irrelevant and generic information, such as dates and locations. In contrast, the predictions based on this disclosure are better aligned with the video content.

[0059] To evaluate the performance of Student Model 400, a user study was conducted in which participants were asked to select the best caption from ten candidates for each video. Ten captions were predicted from eight Teacher Models 208 and two Student Models with different inputs (with and without text). The preference ratios for each model are reported, and the R@1 accuracy of the fine-tuned UMT (i.e., all teachers) is shown in Table 5. Video samples were collected from a test set not seen during the training of the Student Models and UMT. As shown in Table 5, Student Model 400 outperforms any single model and achieves comparable performance to All Teacher Models 208.

[0060]

[0061] Table 5

[0062] Student Model 400 supports cross-modal input to leverage additional textual information. The numerical evaluation in Table 5 supports this, with Student Model 400 having both video and text inputs showing a 3% higher preference ratio than its counterpart with only video input. Qualitatively, Figure 5 The image shows caption generation predictions with and without text input. While predictions with pure video input can include portions of the video, such as "cactus," the corresponding model with both video and text input can more comprehensively include keywords from the video title, description, and dialogue captions, such as "succulents" and "different species."

[0063] UMT was used as the base model for evaluating video and text retrieval performance. The standard approach uses 3M images and 2.5M videos from CC3M as the pre-training dataset. Therefore, for a fair comparison, a subset of Panda-5M was randomly sampled, sharing the same number of training samples as the standard pre-training dataset. The same backbone consisting of ViT-L / 16 and BERTlarge was used for both datasets. Official weights for the standard dataset were used, and the model was trained from scratch for the Panda-5M dataset.

[0064] Zero-shot retrieval was tested on three benchmarks: MSR-VTT, DiDeMo, and MSVD. For MSR-VTT, evaluation was performed on 1,000 test segments following a common protocol. For DiDeMo, it contained 10,000 Flickr videos with a total of 40,000 dense captions. As with previous standards, segment-to-video retrieval was evaluated by concatenating all sentence descriptions of a video into a single query. Results for the 1,000 test sets are reported. For MSVD, results for 670 test videos are reported. Standard metrics were used, and the R@1, R@5, and R@10 accuracies for both text-to-video and video-to-text retrieval are reported in Table 6.

[0065]

[0066] Table 6

[0067] Table 6 shows that pre-training on the Panda-5M dataset outperforms the official weights on most datasets. Notably, pre-training brings a 7.0% improvement in R@1 for MSR-VTT. Although pre-training performs poorly in video-to-text retrieval on MSVD, Table 6 highlights that MSVD is an early-built dataset, and some of the videos have low resolution (…). This results in a domain gap with the dataset.

[0068] To evaluate the effectiveness of text-to-video generation, AnimateDiff was used as the base model, and the weights trained on the official dataset (trained on 2.5M text-video pairs) and those trained on the Panda-2M dataset (a 2.5M subset of Panda-70M) were compared. The official codebase was followed, and Stable Diffusion v1.5 (SD) was used as the base text-to-image (T2I) generator. During training, the T2I module was fixed, and the motion modeling module was trained. For each training video, 16 frames were sampled with a stride of 4, and then resized and cropped to the center. Resolution.

[0069] To evaluate the model, a zero-shot evaluation was performed on the UCF101 and MSR-VTT following the evaluation protocol. Specifically, 16 frames of video were used. Resolution generation. For UCF101, text prompts were generated for each class, and 10,000 videos were generated that shared the same class distribution as the original dataset. The Frechet Video Distance (FVD) was computed on the I3D embeddings. For MSR-VTT, video samples were generated for each of the 59,800 test prompts, and CLIP similarity (CLIPSim) was computed. These figures are reported in Table 7. The generated video samples... Figure 6 The result is shown at position 600. For visualization, the official codebase was followed, and SD T2I was replaced with a personalized Dreambooth weight, TUSUN. Note the test hints and video samples with official weights from AnimtateDiff. Figure 6 (The top row in the text) comes from the AnimateDiff project page.

[0070]

[0071] Table 7

[0072] Compared to the official weights, pre-training on the Panda-2M dataset demonstrates state-of-the-art performance on both metrics. As highlighted, pre-training resulted in a 77.4 reduction in FVD on UCF101 and outperforms existing models pre-trained on datasets within the 10M range in terms of FVD. In terms of quality, pre-training on the dataset leads to the generation of watermark-free videos with more meaningful motion and realistic appearance.

[0073] Figure 8This is an illustration of machine 800, in which instructions 810 (e.g., software, programs, applications, applets, applications, or other executable code) can be executed to cause machine 800 to perform any or more methods discussed herein. For example, instructions 810 can cause machine 800 to perform any or more methods described herein, including automatic caption generation pipeline 200. Instructions 810 transform a general, unprogrammed machine 800 into a specific machine 800 programmed to perform the functions described and shown in the manner described. Machine 800 can operate as a standalone device or be coupled (e.g., networked) to other machines. In a networked deployment, machine 800 can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 800 may include, but is not limited to, server computers, client computers, personal computers (PCs), tablets, laptops, netbooks, set-top boxes (STBs), personal digital assistants (PDAs), entertainment media systems, cellular phones, smartphones, mobile devices, wearable devices (e.g., smartwatches), smart home devices (e.g., smart appliances), other smart devices, network devices, network routers, network switches, bridges, or any machine capable of sequentially or otherwise executing instructions 810 that specify the actions to be taken by machine 800. Furthermore, although only a single machine 800 is shown, the term "machine" should also be considered as a collection of machines that individually or jointly execute instructions 810 to perform any one or more methods discussed herein. In some examples, machine 800 may also include both client and server systems, where certain operations of a particular method or algorithm are performed on the server side, while certain operations of a particular method or algorithm are performed on the client side.

[0074] Machine 800 may include processor 804, memory 806, and input / output (I / O) components 802, which may be configured to communicate with each other via bus 840. In the example, processor 804 (e.g., a Central Processing Unit (CPU), a Reduced Instruction Set Computing (RISC) processor, a Complex Instruction Set Computing (CISC) processor, a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Radio-Frequency Integrated Circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, processors 808 and 812 that execute instruction 810. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") capable of executing instructions simultaneously. Although Figure 8 Multiple processors 804 are shown, but machine 800 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0075] Memory 806 includes main memory 814, static memory 816, and memory cells 818, all of which are accessible to processor 804 via bus 840. Main memory 806, static memory 816, and memory cells 818 store instructions 810 for any one or more of the methods or functions described herein. During execution of instructions 810 by machine 800, instructions 810 may also reside wholly or partially in main memory 814, static memory 816, machine-readable medium 820 within memory cell 818, at least one of processor 804 (e.g., within processor cache), or any suitable combination thereof.

[0076] I / O component 802 may include a wide variety of components for receiving input, providing output, generating output, sending information, exchanging information, capturing measurements, etc. The specific I / O component 802 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It should be understood that I / O component 802 may include... Figure 8 Many other components are not shown. In various examples, I / O component 802 may include user output component 826 and user input component 828. User output component 826 may include visual components (e.g., displays such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistive mechanisms), other signal generators, etc. User input component 828 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input, photoelectric keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens or other haptic input components that provide position and force for touch or touch gestures), audio input components (e.g., microphones), etc.

[0077] In a further example, I / O component 802 may include biometric component 830, motion component 832, environmental component 834, or position component 836, as well as various other components. For example, biometric component 830 includes components for detecting facial expressions (e.g., hand gestures, facial expressions, vocal expressions, body posture, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brainwaves), and identifying a person (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition). Motion component 832 includes accelerometer components (e.g., accelerometers), gravity sensor components, and rotation sensor components (e.g., gyroscopes).

[0078] Environmental components 834 include, for example, one or more cameras (with still image / photograph and video capabilities), lighting sensor components (e.g., photometers), temperature sensor components (e.g., one or more thermometers for detecting ambient temperature), humidity sensor components, pressure sensor components (e.g., barometers), sound sensor components (e.g., one or more microphones for detecting background noise), proximity sensor components (e.g., infrared sensors for detecting nearby objects), gas sensors (e.g., gas detection sensors for detecting hazardous gas concentrations to ensure safety or measuring pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment.

[0079] The position component 836 includes a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure and from which altitude can be determined), an orientation sensor component (e.g., a magnetometer), and the like.

[0080] Communication can be implemented using a wide variety of technologies. I / O component 802 also includes communication component 838, operable to couple machine 800 to network 822 or device 824 via appropriate coupling or connection. For example, communication component 838 may include a network interface component or another suitable device interfaced with network 822. In further examples, communication component 838 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Bluetooth components, etc. ® Components (e.g., Bluetooth) ® Low power consumption, Wi-Fi ® Components and other communication components to provide communication via other modes. Device 824 can be another machine or any of various peripheral devices (e.g., a peripheral device coupled via USB).

[0081] Furthermore, communication component 838 can detect identifiers or include components that can be used to detect identifiers. For example, communication component 838 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional barcodes (such as Universal Product Code (UPC) barcodes), multi-dimensional barcodes (such as Quick Response (QR) codes), Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D barcodes, and other optical codes) or an acoustic detection component (e.g., a microphone for identifying tagged audio signals). Additionally, various information can be derived via communication component 838, such as location via Internet Protocol (IP) geolocation, location via Wi-Fi® signal triangulation, location by detecting NFC beacon signals that can indicate a specific location, etc.

[0082] Various memories (e.g., main memory 814, static memory 816, and memory of processor 804) and storage units 818 may store one or more sets of instructions and data structures (e.g., software) embodied or used by any one or more methods or functions described herein. When executed by processor 804, these instructions (e.g., instruction 810) cause various operations to implement the disclosed examples, including automatic caption generation pipeline 200.

[0083] Instruction 810 can be sent or received via network 822, using a transmission medium, via a network interface device (e.g., a network interface component included in communication component 838), and using any of several known transmission protocols (e.g., Hypertext Transfer Protocol, HTTP). Similarly, instruction 810 can be sent or received using a transmission medium via coupling to device 824 (e.g., peer-to-peer coupling).

[0084] It will be understood that the terms and expressions used herein have the general meaning consistent with their respective fields of investigation and research, unless otherwise specified herein. Relational terms such as "first" and "second" may be used merely to distinguish one entity or action from another, without necessarily requiring or implying any actual such relationship or order between these entities or actions. The terms "comprises," "comprising," "includes," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes or comprises a list of elements or steps may include not only those elements or steps but also other elements or steps not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element beginning with "a" or "an" does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes that element.

[0085] Unless otherwise stated, all measurements, values, ratings, positions, amplitudes, dimensions, and other specifications set forth in this specification (including the following claims) are approximate, not precise. These quantities are intended to have a reasonable range consistent with the functions they relate to and the conventions of the field to which they belong. For example, unless expressly stated otherwise, parameter values ​​or similar values ​​may differ from the quantities by up to ±10%.

[0086] Furthermore, in the foregoing detailed description, various features have been grouped together in various examples for the purpose of simplifying this disclosure. This method of disclosure should not be construed as reflecting an intention to claim more features than are expressly recited in each claim. Rather, as reflected in the following claims, the subject matter to be protected lies in fewer than all features of any single disclosed example. Therefore, the following claims are thus incorporated into the detailed description, each claim existing independently as a separately claimed subject matter.

[0087] While what is considered the best pattern and other examples have been described above, it should be understood that various modifications can be made, and the subject matter disclosed herein can be implemented in various forms and examples, and can be applied to many applications, only a few of which are described herein. The following claims are intended to claim protection for any and all modifications and variations that fall within the true scope of this concept.

Claims

1. A pipeline configured to automatically annotate video data with dialogue subtitles, the pipeline being configured to: Organize multiple high-resolution videos from publicly available datasets; The video is segmented into semantically consistent video segments; as well as Multiple cross-modal teacher models are applied to obtain captions for each video segment in the video clip.

2. The assembly line according to claim 1, wherein, The pipeline is configured to use automatic speech recognition (ASR) to automatically annotate the video data with dialogue subtitles.

3. The assembly line according to claim 1, wherein, The teacher model has different pre-training weights and is configured to receive different input information.

4. The assembly line according to claim 1, wherein, The pipeline is configured to use a retrieval model to fine-tune a subset of the video clips, wherein the accurate captions for each video clip can be manually selected.

5. The assembly line according to claim 4, wherein, The subset is configured to be split to obtain training samples and test samples.

6. The assembly line according to claim 4, wherein, Following the fine-tuning of the subset, the pipeline is configured to use the retrieval model across the entire dataset to select accurate captions as annotations.

7. The assembly line according to claim 1, wherein, The pipeline is configured to use multimodal data as input to the teacher model and to expand the dataset into high-quality video-caption pairs.

8. The assembly line according to claim 7, wherein, The multimodal data includes text video descriptions, dialogue subtitles, and individual video frames.

9. The assembly line according to claim 8, wherein, The pipeline is configured to enable the student caption generation model to learn on the dataset in order to extract knowledge from the teacher model.

10. The assembly line according to claim 9, wherein, The student caption generation model includes a visual branch and a text branch, which are configured to utilize the multimodal input, wherein gradient propagation from the text branch to the visual branch is blocked during training of the student caption generation model.

11. The assembly line according to claim 10, wherein, The visual branch is configured to select video embeddings compatible with Large Language Models (LLM), and the text branch includes a text encoder and a text query transformer, which are configured to select text embeddings of a fixed length to bridge the video embeddings and the text embeddings.

12. A method for automatically annotating video data with dialogue subtitles using a pipeline, the method comprising the following steps: Organize multiple high-resolution videos from publicly available datasets; The video is segmented into semantically consistent video segments; as well as Multiple cross-modal teacher models are applied to obtain captions for each of the video segments.

13. The method according to claim 12, wherein, The pipeline uses Automatic Speech Recognition (ASR) to automatically annotate the video data with dialogue subtitles.

14. The method according to claim 12, wherein, The teacher model has different pre-training weights and receives different input information.

15. The method according to claim 12, wherein, The pipeline uses a retrieval model to fine-tune a subset of the video clips, wherein the accurate captions for each video clip can be manually selected.

16. The method according to claim 15, wherein, The subset is split to obtain training samples and test samples.

17. The method according to claim 15, wherein, Following the fine-tuning of the subset, the pipeline uses the retrieval model across the entire dataset to select accurate captions as annotations.

18. The method according to claim 12, wherein, The pipeline uses multimodal data as input to the teacher model and expands the dataset into high-quality video-caption pairs.

19. The method according to claim 18, wherein, The multimodal data includes text video descriptions, dialogue subtitles, and individual video frames.

20. A non-transitory computer-readable medium storing program code, said program code being operable at execution time to cause a pipeline to perform the following steps: Organize multiple high-resolution videos from publicly available datasets; The video is segmented into semantically consistent video segments; as well as Multiple cross-modal teacher models are applied to obtain captions for each of the video segments.

Images