Pre-training approaches for video representation models

Abstract

Provided is a computer-implemented method for training a machine-learned video representation model, some example implementations of which can be referred to as "VideoPrism." This model can be used for a variety of video understanding tasks, such as action recognition, spatiotemporal localization, video-text retrieval, video captioning, and video question answering. The model can be trained using a hybrid dataset containing high-quality video-caption pairs, videos with noisy parallel text, and / or training examples containing only video.

Description

PRE-TRAINING APPROACHES FOR VIDEO REPRESENTATION MODELSRELATED APPLICATIONS

[0001] This application claims priority to and the benefit of United States Provisional Patent Application Number 63 / 548,779, filed February 1, 2024. United States Provisional Patent Application Number 63 / 548,779 is hereby incorporated by reference in its entirety.FIELD

[0002] The present disclosure relates generally to machine learning. More particularly, the present disclosure relates to systems and methods for pre-training a video representation model.BACKGROUND

[0003] Videos are a rich source of information, capturing complex visual and auditory narratives that span across a multitude of domains. The ability to automatically understand and reason about video content is a highly sought-after capability in artificial intelligence, with wide-ranging applications from content recommendation systems to autonomous vehicle navigation and scientific discovery.

[0004] A significant problem encountered in prior approaches to video understanding is the excessive consumption of computational resources. This is due, in part, to the high dimensionality of video data, which encompasses spatial, temporal, and often multimodal elements. Each frame of a video may be considered as a high-resolution image, and a typical video contains tens or hundreds of such frames per second. Analyzing this data to extract meaningful information requires substantial computational power and storage, particularly when employing deep learning models that necessitate large-scale datasets for training.

[0005] Moreover, the traditional paradigm of training or fine-tuning models end-to-end for each specific task or dataset exacerbates the consumption of computational resources. This approach necessitates repeated learning processes for each new task or dataset, leading to a duplication of effort and an inefficient use of computational resources. As the size and complexity of models grow to accommodate the intricacies of video data, the computational burden becomes increasingly unsustainable.

[0006] In addition to computational inefficiencies, another problem that arises is the lack of generalizability. Models trained on specific datasets may perform well on tasks for which they were designed but often fail to transfer this performance to other tasks or datasets,particularly those from different domains, particularly in scientific fields where the visual data may differ significantly from the content found in standard training datasets.SUMMARY

[0007] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or can be learned from the description, or can be learned through practice of the embodiments.

[0008] One example aspect of the present disclosure is directed to a computer- implemented method for pre-training a machine-learned video representation model. The method includes obtaining, by a computing system comprising one or more computing devices, a training dataset comprising a plurality of training examples, wherein each of the plurality of training examples comprises a respective video, and wherein at least some of the plurality of training examples further comprise respective textual content associated with the respective video. The method includes performing, by the computing system, a first training stage on the machine-learned video representation model, wherein the first training stage comprises training the machine-learned video representation model with a video-text contrastive loss for at least some of the training examples that comprise both video and textual content. The method includes, subsequent to the first training stage, performing, by the computing system, a second training stage on the machine-learned video representation model, wherein the second training stage comprises training the machine-learned video representation model with a masked video modeling loss for at least some of the training examples that comprise video content. The method includes, subsequent to the second training stage, providing, by the computing system, the machine-learned video representation model as an output.

[0009] Another example aspect of the present disclosure is directed to a computer- implemented method for training video representation models. The method can be performed for each of one or more training examples that comprise a respective video. The method includes generating, by the computing system, a plurality of image tokens for a plurality of patches of a plurality of image frames of the respective video. The method includes processing, by the computing system, the plurality of image tokens with a teacher video representation model to generate a plurality of target video representation tokens. The method includes designating, by the computing system, at least some of the plurality of image tokens as masked tokens and at least some of the plurality of image tokens as unmasked tokens. The method includes processing, by the computing system, only the unmasked tokens with astudent video representation model to generate a plurality of partial video representation tokens. The method includes shuffling, by the computing system, the plurality of partial video representation tokens. The method includes, after shuffling, by the computing system, the plurality of partial video representation tokens, processing, by the computing system, the plurality of partial video representation tokens with a token-wise decoder model to predict one or more remaining video representation tokens. The method includes modifying, by the computing system, one or more values of one or more parameters of at least the student video representation model based on the masked video modeling loss that compares the one or more remaining video representation tokens with at least some of the plurality of target video representation tokens.

[0010] Other aspects of the present disclosure are directed to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0011] These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the related principles.BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Detailed discussion of embodiments directed to one of ordinary skill in the art is set forth in the specification, which makes reference to the appended figures, in which:

[0013] Figure 1 illustrates a graphical diagram of an example contrastive training approach according to example embodiments of the present disclosure.

[0014] Figure 2 illustrates a graphical diagram of an example masked video modeling approach according to example embodiments of the present disclosure.

[0015] Figure 3 A depicts a block diagram of an example computing system according to example embodiments of the present disclosure.

[0016] Figure 3B depicts a block diagram of an example computing device according to example embodiments of the present disclosure.

[0017] Figure 3C depicts a block diagram of an example computing device according to example embodiments of the present disclosure.

[0018] Reference numerals that are repeated across plural figures are intended to identify the same features in various implementations.DETAILED DESCRIPTION

[0019] Example aspects of the present disclosure are directed to improved techniques for training a general -purpose video representation model that can be applied to a number of different downstream video understanding or reasoning tasks. In particular, prior techniques for video understanding often required training or fine-tuning models for specific datasets or tasks, which is resource-intensive and impractical with rapidly growing model sizes. Additionally, it was unclear if these systems could generalize to data from other domains, such as those captured for scientific experiments. The present disclosure addresses these limitations by providing a general -purpose video representation model that can be easily adapted to various video understanding or reasoning tasks across domains.

[0020] One aspect of the present disclosure is directed to a two-stage training strategy that combines contrastive and masked modeling. In the first training stage, the model is trained with a video-text contrastive loss using training examples that include both video and textual content (e.g., captioned videos or videos with some parallel text). In the second training stage, the model is trained with a masked video modeling loss using training examples that comprise video content. This two-stage strategy enables the model to demonstrate improved performance on both motion-focused and appearance-focused tasks.

[0021] Another aspect of the present disclosure is directed to specific approaches for improving masked video modeling. As one example, the training system can perform token shuffling before prediction of the masked tokens in the masked video modeling approach. Shuffling of the tokens can help to prevent the model from creating shortcuts that would reduce the value of the training signal. As another example, the training system can apply both a global distillation loss and also a local or token-wise distillation loss. Applying both of these losses can enable the video representation model to learn both representations that capture information relating both localized motion and also global appearance.

[0022] Once trained, the video representation model can be applied to a wide spectrum of video-centric understanding tasks with capability and generalizability. These tasks can include, as example, general video-only understanding, zero-shot video-text retrieval, zeroshot video captioning and question answering, and computer vision for science. Thus, the present disclosure provides a powerful tool for video understanding and reasoning tasks, overcoming the limitations of prior techniques and offering improved performance and generalizability across a wide range of tasks and domains. In some instances, because the video representation model is generating a latent representation or embedding of a video, the video representation model can be referred to as a “video encoder” or a “visual encoder”.

[0023] More particularly, one aspect of the present disclosure is directed to a computer- implemented method for training a machine-learned video representation model, some example implementations of which can be referred to as “VideoPrism.” This model can be used for a variety of video understanding tasks, such as action recognition, spatiotemporal localization, video-text retrieval, video captioning, and video question answering. The model can be trained using a hybrid dataset containing high-quality video-caption pairs, videos with noisy parallel text, and / or training examples containing only video.

[0024] In some implementations, the video representation model can be trained in two stages. The first stage includes training the model with a video-text contrastive loss. This can be done by processing the video and the associated textual content of a training example with the video representation model and a text encoder model respectively. A contrastive loss can then be applied between the video embedding generated by the model and the text embedding generated by the text encoder model. This first stage can result in a partially trained version of the video representation model.

[0025] The second stage of training can include training the model with a masked video modeling loss. In this stage, the training system can generate a set of image tokens for patches of image frames within a video. These tokens are then processed by a “teacher” instantiation of the partially trained model to generate a set of target video representation tokens. Further, some of the image tokens can be designated as masked tokens and others as unmasked tokens. The unmasked tokens are processed by a “student” instantiation of the model to generate a set of partial video representation tokens.

[0026] In order to improve the performance of the second stage, the present disclosure introduces a novel token shuffling scheme. The partial video representation tokens generated by the student model are randomly shuffled before being processed by a token-wise decoder model. This token-wise decoder model is then used to predict the remaining video representation tokens. The values of the parameters of the student model can then be modified based on a masked video modeling loss that compares the predicted remaining tokens with the target tokens.

[0027] In addition to token shuffling, the present disclosure also introduces a global distillation loss in addition or alternatively to the token-wise masked modeling (distillation) loss described above. In this case, in addition to the plurality of target tokens, the video can be processed by the teacher model to generate a global target video representation. For the student, the partial video representation tokens can be processed by a global decoder model to generate a predicted global video representation. The values of the parameters of the studentmodel can then be modified based on a global distillation loss that compares the global target video representation with the predicted global video representation.

[0028] The systems and methods of the present disclosure provide a number of technical solutions to a number of technical problems. As one example, a technical problem addressed by the present disclosure is the resource-intensive nature of training video understanding models for specific datasets or tasks, as well as the challenge of generalizing these models to data from other domains. Traditional methods often require end-to-end training or fine-tuning for each new task, which is computationally expensive and impractical with the increasing size of models. Furthermore, models trained on specific datasets may not perform well on tasks including different types of data, such as those in scientific research.

[0029] The technical solutions provided by the present disclosure include a two-stage training strategy that combines video-text contrastive learning and masked autoencoding. This strategy enables the video representation model to learn rich visual semantics from language supervision and to capture both appearance and motion semantics from videos. The first stage aligns the video encoder with a text encoder using video-text pairs, while the second stage focuses on video-only data to refine the understanding of motion content. This approach reduces the computational burden by leveraging a pre-trained model that can be adapted to various tasks without the need for task-specific retraining.

[0030] The technical effects of the disclosed methods include improved performance on both motion-focused and appearance-focused video understanding tasks, as well as the ability to generalize across different domains. The use of a hybrid dataset that includes high-quality video-caption pairs and videos with noisy parallel text contributes to the robustness and versatility of the trained model.

[0031] Further technical solutions include specific improvements to masked video modeling, such as token shuffling and global-local distillation. Token shuffling prevents the model from taking shortcuts during training, which increases the effectiveness of the training process. Global-local distillation allows the model to learn from both the global context of the video and the local details, enhancing its performance on downstream tasks and balancing performance with respect to appearance- and motion-based tasks.

[0032] Once the model has been trained, it can be adapted to perform a variety of video understanding tasks. For example, it can be used for classification and localization, which includes identifying the specific locations within a video where certain actions or objects are present. It can also be used for video-text retrieval, which includes searching for videos that match a given text query.

[0033] The model can also be used for video captioning, which includes generating a textual description of the content of a video. This can be particularly useful for providing accessibility features for users who are unable to watch the video. Additionally, the model can be used for video question answering, which includes generating answers to questions about the content of a video.

[0034] Finally, the model can be used for various scientific applications. For example, it can be used in behavioral neuroscience to analyze videos of animal behavior, in ethology to study animal communication, in cognitive science to analyze human behavior, and in ecology to analyze videos of natural environments. These diverse applications demonstrate the broad potential of the video representation model described in the present disclosure.

[0035] More particularly, the video representation model described herein can be trained to be general-purpose and adaptable to various downstream tasks. The adaptability of the model to downstream tasks typically can include a process known as transfer learning, where a model trained on a large and diverse dataset (pre-training) is then fine-tuned on a smaller, task-specific dataset (fine-tuning). This process allows the model to leverage the rich representations it has learned during pre-training and apply them to the specific requirements of the downstream task.

[0036] The video representation model can be adapted to downstream tasks in a number of different ways. In some cases, one or more additional layers can be added on top of the pre-trained model. For example, a multi-head attentional pooling (MAP) layer can be added on top of the pretrained models for classification tasks. During fine-tuning, this parameters of the additional layer(s) can be learned from scratch, while the pre-trained layers may either be frozen or fine-tuned with a lower learning rate.

[0037] As another example, adapter layers (e.g., small neural networks) can be inserted between the layers of a pre-trained model. During fine-tuning, only the parameters of these adapter layers are updated, while the original pre-trained parameters remain frozen. This approach allows for efficient transfer learning by reducing the number of parameters that need to be trained for the downstream task.

[0038] As another example, depending on the similarity of the downstream task to the pre-training task, the entire model or parts of it can be fine-tuned. This includes continuing the training process on the downstream dataset, often with a reduced learning rate to avoid overfitting. Fine-tuning can be applied to all layers or selectively to only the topmost layers that are more task-specific.

[0039] For some tasks, the pre-trained model can be used as a fixed feature extractor. Here, the output of one of the intermediate layers or the final layer can be used as a feature vector, which is then fed into a separate classifier or regressor tailored to the downstream task.

[0040] As another example, for multitask learning, different task-specific heads (output layers) can be added to the pre-trained model. Each head is responsible for a different task, such as video captioning, question answering, or object detection. These heads can be trained simultaneously or sequentially, leveraging shared representations learned by the model.

[0041] Alternatively or additionally, the video representation model can be connected to a language model (e.g. “LLM”), for example, with a connecting layer in the middle (e.g., a one-layer Perceiver Resampler).

[0042] With reference now to the Figures, example embodiments of the present disclosure will be discussed in further detail.

[0043] Referring now to Figure 1, a graphical diagram is presented illustrating an example contrastive training approach in accordance with example embodiments of the present disclosure. The diagram depicts the interaction between video content 12 and textual content 14 with their respective processing models, culminating in the application of a contrastive loss function 24 to align the resulting embeddings.

[0044] The video content 12 can be processed by a video representation model 16, which can be configured to extract features and representations pertinent to the understanding of video data. The video representation model 16 can process the video content 12 to produce an intermediate representation, which is then passed through a pooler 17. The pooler 17 can be designed to aggregate the features and produce a video embedding 20 that encapsulates the visual semantics captured from the video content 12.

[0045] The textual content 14 can be processed (e.g., in parallel) by a text encoder model 18. The text encoder model 18 can be a machine learning model specialized in understanding and encoding language-based information. The text encoder model 18 can transform the textual content 14 into a text embedding 22, which is a vector representation that captures the linguistic and semantic nuances of the associated text.

[0046] The video embedding 20 and the text embedding 22 are then evaluated with a contrastive loss function 24. The contrastive loss function 24 can be employed to minimize the distance between the video and text embeddings that are correctly paired while maximizing the distance between those that are incorrectly paired. This loss function 24 facilitates the alignment of the video representation model 16 with the text encoder model 18.The loss function 24 can increase the likelihood that the video embeddings 20 are semantically consistent with their corresponding text embeddings 22.

[0047] The training approach depicted in Figure 1 can be implemented in various configurations. For instance, the video representation model 16 and the text encoder model 18 can be instantiated with different architectures, such as convolutional neural networks, recurrent neural networks, or transformers, depending on the specific requirements of the training task. Additionally, the pooler 17 can employ various pooling strategies, such as average pooling, max pooling, or attention-based pooling, to effectively condense the video features into a robust video embedding 20.

[0048] Furthermore, the contrastive loss function 24 can be adapted to include regularization terms, margin-based constraints, or other modifications to refine the training process and enhance the learning of correlated video and text representations. The flexibility of this contrastive training approach enables it to be tailored to diverse video understanding applications, ranging from action recognition to video captioning and beyond.

[0049] Referring now to Figure 2, an example masked video modeling approach is illustrated in accordance with example embodiments of the present disclosure. The process begins with a video 202, which contains a sequence of image frames that capture both the appearance and motion characteristics inherent in the video content. The video 202 is then subjected to a two-fold processing pathway aimed at enhancing the video representation model’s ability to encode detailed and contextually relevant information from the video content.

[0050] On one pathway, the video 202 is processed by a teacher video representation model 204, which has been pretrained to capture rich video semantics. The teacher model 204 processes the video 202 to generate a plurality of target video representation tokens 206, which serve as a high-level encoding of the video’s content, capturing both spatial and temporal features.

[0051] The video with masked image patches 208 is processed (e.g., in parallel to the other pathway) by a student video representation model 210. The student model 210 is tasked with generating partial video representation tokens 212, which are a subset of the video representation that correspond to image patches that were not masked. Masked placeholders can be added, as shown at 213, simulating an incomplete understanding of the video content. A token shuffling operation 214 is then applied to the partial video representation tokens and the masked placeholders 213, resulting in shuffled partial video representation tokens 215. This shuffling operation can help to ensure that the student model 210 learns to predict thevideo content without relying on the sequential order of tokens, thereby enhancing its ability to generalize and understand videos.

[0052] The shuffled partial video representation tokens 215 are then processed by a token-wise decoder 216, which attempts to predict the remaining video representation tokens 218 that correspond to image patches that were initially masked. Specifically, in some implementations, decoder 216 attempts to predict tokens 218 that correspond to all image patches from the video (both masked and unmasked). Some or all of this prediction can be compared against the target video representation tokens 206 using a masked video modeling loss 220, which guides the learning process of the student model 210 by penalizing inaccuracies in the predicted tokens 218 as compared to the target tokens 206. In some implementations, the masked video modeling loss 220 can be referred to as a token-wise distillation loss or a local distillation loss.

[0053] In some implementations, in parallel to the token-wise processing, the teacher model 204 also generates a global target video representation 221, which is an aggregated encoding of the entire video content. In parallel, a global decoder 222 can be applied to the partial video representation tokens 212 to produce a predicted global video representation 224. The predicted global video representation 224 is compared to the global target video representation 221 using a global distillation loss 226. This loss function can increase the likelihood that the student model 210 not only learns to predict local features accurately but also maintains a consistent understanding of the video’s global context.

[0054] Thus, in the masked video modeling approach described in Figure 2, loss functions 220 and 226 can be used to update or refine the parameters of the student video representation model 210 to enhance its video understanding capabilities. The masked video modeling loss 220 can quantify the discrepancy between the predicted remaining video representation tokens 218 and the target video representation tokens 206, which are generated by the teacher model 204. Similarly, the global distillation loss 226 can measure the difference between the predicted global video representation 224 and the global target video representation 221. During the training process, one or both of these loss functions can be backpropagated through the student model 210, which means that the gradients of the loss with respect to the model’s parameters are computed and used to update the parameter values. This b ackpropagation can be performed iteratively, with the goal of minimizing the loss values by adjusting the parameters in a direction that reduces the prediction errors. As a result, the student model 210 learns to generate more accurate video representations, bothlocally for individual masked tokens and globally for the overall context of the video, thereby improving its performance on downstream video understanding tasks.

[0055] The approach depicted in Figure 2 illustrates a training methodology that leverages the strengths of both local and global feature learning to produce a video representation model capable of sophisticated video understanding tasks. This methodology can be implemented with various modifications, such as: varying the proportion of masked tokens; employing different architectures for the teacher and student models or initializing the same model with the teacher being frozen; and / or adjusting the loss functions to prioritize certain aspects of the video content. The flexibility and robustness of this approach enable the resulting representation model to be applied across a wide array of downstream video understanding tasks and domains, including but not limited to action recognition, video captioning, and scientific analysis.

[0056] One example model architecture is as follows: An example VideoPrism model architecture stems from the standard Vision Transformer (ViT), with a factorized design in space and time following ViViT. However, some example implementations remove the global average pooling layer of ViViT immediately after the spatial encoder so that the spatiotemporal dimensions remain in the output token sequence, facilitating the downstream tasks that require fine-grained features (e.g., spatiotemporal action localization).

[0057] Example training algorithms include two stages. A first stage can include videotext contrastive training, examples of which are as follows: In the first stage, some example implementations conduct contrastive learning to align a video encoder with a text encoder using all the video-text pairs. Some example implementations minimize a symmetric crossentropy loss over the similarity scores of all video-text pairs in a mini-batch, initialize the spatial encoding modules using the image model of CoCa, and include WebLI (about IB images with alt-text) to the pretraining. The video encoder’s features can be aggregated through a multi-head attention pooler (MAP) before the loss computation. This stage allows the video encoder to learn rich visual semantics from language supervision, and the resulting model supplies semantic video embeddings for the second-stage training.

[0058] A second stage can include masked video modeling, examples of which are as follows: Training solely on vision-text data as in Stage 1 presents challenges: text descriptions can be noisy, and they often capture appearance more than motion. To address this, an example proposed second-stage training focuses on learning both appearance and motion information from video-only data. Building upon the success of masked autoencoding for motion understanding, some example implementations adapt this approach for the secondstage, while ensuring that the model retains the semantic knowledge acquired in the first stage.

[0059] In this stage, some example implementations continue to train the video encoder on video-only data using improved masked video modeling. These improvements can optionally include (1) a novel token shuffling scheme to prevent decoding shortcuts and (2) global and token-wise distillation losses to effectively leverage the knowledge acquired in the first stage. The second-stage (student) model learns to predict the first-stage (teacher) model’s embeddings of all tokens based on a masked video. The encoder-decoder Transformers can be decoupled.

[0060] Some implementations perform token shuffling, examples of which are as follows: As some example implementations effectively initialize the second-stage model from the first stage, one issue is that the model may create a shortcut for the decoder to copy and paste the unmasked tokens while predicting only the masked ones, making it an easier task to solve than predicting all tokens. To address this issue, some example implementations randomly shuffle the token sequence output by the encoder before feeding it to the decoder, and the decoder adds positional embeddings to this sequence after the shuffling. Note that example implementations of this shuffling operation avoid the copy-and-paste shortcut of unmasked tokens that the decoder can potentially explore. One can also view it akin to Jigsaw puzzles which the decoder tries to solve for the unmasked tokens while it predicts the masked ones.

[0061] Some implementations can perform global -local distillation, examples of which are as follows: Unlike the masked distillation for images, it was observed that, in some settings, an example second-stage model underperforms the first-stage teacher on appearance-heavy tasks when only the masked modeling loss is utilized, probably attributing to catastrophic forgetting in the two-stage pretraining. To mitigate this issue, some example implementations add an additional loss to let the second-stage model distill the global embedding of the full intact video from the first-stage teacher using the visible tokens. Hence, the second-stage training loss combines the token-wise masked video modeling and global distillation.

[0062] Figure 3 A depicts a block diagram of an example computing system 100 according to example embodiments of the present disclosure. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 that are communicatively coupled over a network 180.

[0063] The user computing device 102 can be any type of computing device, such as, for example, a personal computing device (e.g., laptop or desktop), a mobile computing device (e.g., smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0064] The user computing device 102 includes one or more processors 112 and a memory 114. The one or more processors 112 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 114 can include one or more non -transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 can store data 116 and instructions 118 which are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0065] In some implementations, the user computing device 102 can store or include one or more machine-learned models 120. For example, the machine-learned models 120 can be or can otherwise include various machine-learned models such as neural networks (e.g., deep neural networks) or other types of machine-learned models, including non-linear models and / or linear models. Neural networks can include feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks or other forms of neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine-learned models can include multi-headed self-attention models (e.g., transformer models). Example machine-learned models 120 are discussed with reference to Figures 1-2.

[0066] In some implementations, the one or more machine-learned models 120 can be received from the server computing system 130 over network 180, stored in the user computing device memory 114, and then used or otherwise implemented by the one or more processors 112. In some implementations, the user computing device 102 can implement multiple parallel instances of a single machine-learned model 120 (e.g., to perform parallel video representation across multiple instances of videos).

[0067] Additionally or alternatively, one or more machine-learned models 140 can be included in or otherwise stored and implemented by the server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the machine-learned models 140 can be implemented by the server computing system 140 as a portion of a web service (e.g., a video representation, understanding, or analysis service). Thus, one or more models 120 can be stored and implemented at the usercomputing device 102 and / or one or more models 140 can be stored and implemented at the server computing system 130.

[0068] The user computing device 102 can also include one or more user input components 122 that receives user input. For example, the user input component 122 can be a touch-sensitive component (e.g., a touch-sensitive display screen or a touch pad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component can serve to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0069] The server computing system 130 includes one or more processors 132 and a memory 134. The one or more processors 132 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 134 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 can store data 136 and instructions 138 which are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0070] In some implementations, the server computing system 130 includes or is otherwise implemented by one or more server computing devices. In instances in which the server computing system 130 includes plural server computing devices, such server computing devices can operate according to sequential computing architectures, parallel computing architectures, or some combination thereof.

[0071] As described above, the server computing system 130 can store or otherwise include one or more machine-learned models 140. For example, the models 140 can be or can otherwise include various machine-learned models. Example machine-learned models include neural networks or other multi-layer non-linear models. Example neural networks include feed forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine-learned models can include multi-headed self-attention models (e.g., transformer models). Example models 140 are discussed with reference to Figures 1-2.

[0072] The user computing device 102 and / or the server computing system 130 can train the models 120 and / or 140 via interaction with the training computing system 150 that iscommunicatively coupled over the network 180. The training computing system 150 can be separate from the server computing system 130 or can be a portion of the server computing system 130.

[0073] The training computing system 150 includes one or more processors 152 and a memory 154. The one or more processors 152 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 154 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 154 can store data 156 and instructions 158 which are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0074] The training computing system 150 can include a model trainer 160 that trains the machine-learned models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130 using various training or learning techniques, such as, for example, backwards propagation of errors. For example, a loss function can be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on a gradient of the loss function). Various loss functions can be used such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques can be used to iteratively update the parameters over a number of training iterations.

[0075] In some implementations, performing backwards propagation of errors can include performing truncated backpropagation through time. The model trainer 160 can perform a number of generalization techniques (e.g., weight decays, dropouts, etc.) to improve the generalization capability of the models being trained.

[0076] In particular, the model trainer 160 can train the machine-learned models 120 and / or 140 based on a set of training data 162. The training data 162 can include, for example, videos paired with text and / or videos that are not paired with text.

[0077] In some implementations, if the user has provided consent, the training examples can be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 can be trained by the training computing system 150 on user-specific data received from the user computing device 102. In some instances, this process can be referred to as personalizing the model.

[0078] The model trainer 160 includes computer logic utilized to provide desired functionality. The model trainer 160 can be implemented in hardware, firmware, and / or software controlling a general purpose processor. For example, in some implementations, the model trainer 160 includes program files stored on a storage device, loaded into a memory and executed by one or more processors. In other implementations, the model trainer 160 includes one or more sets of computer-executable instructions that are stored in a tangible computer-readable storage medium such as RAM, hard disk, or optical or magnetic media.

[0079] The network 180 can be any type of communications network, such as a local area network (e.g., intranet), wide area network (e.g., Internet), or some combination thereof and can include any number of wired or wireless links. In general, communication over the network 180 can be carried via any type of wired and / or wireless connection, using a wide variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0080] Figure 3 A illustrates one example computing system that can be used to implement the present disclosure. Other computing systems can be used as well. For example, in some implementations, the user computing device 102 can include the model trainer 160 and the training dataset 162. In such implementations, the models 120 can be both trained and used locally at the user computing device 102. In some of such implementations, the user computing device 102 can implement the model trainer 160 to personalize the models 120 based on user-specific data.

[0081] Figure 3B depicts a block diagram of an example computing device 10 that performs according to example embodiments of the present disclosure. The computing device 10 can be a user computing device or a server computing device.

[0082] The computing device 10 includes a number of applications (e.g., applications 1 through N). Each application contains its own machine learning library and machine-learned model(s). For example, each application can include a machine-learned model. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.

[0083] As illustrated in Figure 3B, each application can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0084] Figure 3C depicts a block diagram of an example computing device 50 that performs according to example embodiments of the present disclosure. The computing device 50 can be a user computing device or a server computing device.

[0085] The computing device 50 includes a number of applications (e.g., applications 1 through N). Each application is in communication with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and model(s) stored therein) using an API (e.g., a common API across all applications).

[0086] The central intelligence layer includes a number of machine-learned models. For example, as illustrated in Figure 3C, a respective machine-learned model can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine-learned model. For example, in some implementations, the central intelligence layer can provide a single model for all of the applications. In some implementations, the central intelligence layer is included within or otherwise implemented by an operating system of the computing device 50.

[0087] The central intelligence layer can communicate with a central device data layer. The central device data layer can be a centralized repository of data for the computing device 50. As illustrated in Figure 3C, the central device data layer can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0088] The technology discussed herein makes reference to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a great variety of possible configurations, combinations, and divisions of tasks and functionality between and among components. For instance, processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0089] While the present subject matter has been described in detail with respect to various specific example embodiments thereof, each example is provided by way ofexplanation, not limitation of the disclosure. Those skilled in the art, upon attaining an understanding of the foregoing, can readily produce alterations to, variations of, and equivalents to such embodiments. Accordingly, the subject disclosure does not preclude inclusion of such modifications, variations and / or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art. For instance, features illustrated or described as part of one embodiment can be used with another embodiment to yield a still further embodiment. Thus, it is intended that the present disclosure cover such alterations, variations, and equivalents.

Claims

WHAT IS CLAIMED IS:

1. A computer-implemented method for pre-training a machine-learned video representation model, the method comprising: obtaining, by a computing system comprising one or more computing devices, a training dataset comprising a plurality of training examples, wherein each of the plurality of training examples comprises a respective video, and wherein at least some of the plurality of training examples further comprise respective textual content associated with the respective video; performing, by the computing system, a first training stage on the machine-learned video representation model, wherein the first training stage comprises training the machine- learned video representation model with a video-text contrastive loss for at least some of the training examples that comprise both video and textual content; subsequent to the first training stage, performing, by the computing system, a second training stage on the machine-learned video representation model, wherein the second training stage comprises training the machine-learned video representation model with a masked video modeling loss for at least some of the training examples that comprise video content; and subsequent to the second training stage, providing, by the computing system, the machine-learned video representation model as an output.

2. The computer-implemented method of claim 1, wherein performing the first training stage comprises, for each of one or more of the training examples that comprise both video and textual content: processing, by the computing system, the respective video of the training example with the machine-learned video representation model to generate a video embedding; processing, by the computing system, the respective textual content of the training example with a text encoder model to generate a text embedding; and applying the contrastive loss between the video embedding generated by the machine- learned video representation model for the respective video and the text embedding generated by the text encoder model for the respective textual content;wherein the first training stage results in a partially-trained version of the machine- learned video representation model.

3. The computer-implemented method of claim 2, wherein performing the second training stage comprises, for each of one or more of the training examples that comprise video content: generating, by the computing system, a plurality of image tokens for a plurality of patches of a plurality of image frames of the respective video; processing, by the computing system, the plurality of image tokens with a teacher instantiation of the partially-trained version of the machine-learned video representation model to generate a plurality of target video representation tokens; designating, by the computing system, at least some of the plurality of image tokens as masked tokens and at least some of the plurality of image tokens as unmasked tokens; processing, by the computing system, only the unmasked tokens with a student instantiation of the partially-trained version of the machine-learned video representation model to generate a plurality of partial video representation tokens; processing, by the computing system, the plurality of partial video representation tokens with a token-wise decoder model to predict one or more remaining video representation tokens; and modifying, by the computing system, one or more values of one or more parameters of at least the student instantiation of the partially-trained version of the machine-learned video representation model based on the masked video modeling loss that compares the one or more remaining video representation tokens with at least some of the plurality of target video representation tokens.

4. The computer-implemented method of claim 3, further comprising, prior to processing, by the computing system, the plurality of partial video representation tokens with the decoder model: shuffling, by the computing system, the plurality of partial video representation tokens.

5. The computer-implemented method of claim 3 or 4, wherein performing the second training stage comprises, for each of one or more of the training examples that comprise video content: processing, by the computing system, the respective video with the teacher instantiation of the partially-trained version of the machine-learned video representation model to generate a global target video representation; processing, by the computing system, the plurality of partial video representation tokens with a global decoder model to generate a predicted global video representation; and modifying, by the computing system, one or more values of one or more parameters of at least the student instantiation of the partially-trained version of the machine-learned video representation model based on a global distillation loss that compares the global target video representation with the predicted global video representation.

6. The computer-implemented method of any preceding claim, further comprising: adapting, by the computing system, the machine-learned video representation model to perform a video understanding task, wherein the video understanding task comprises: classification and localization, video-text retrieval, video captioning, or video question answering.

7. A computer-implemented method for training video representation models, the method comprising, for each of one or more training examples that comprise a respective video: generating, by the computing system, a plurality of image tokens for a plurality of patches of a plurality of image frames of the respective video; processing, by the computing system, the plurality of image tokens with a teacher video representation model to generate a plurality of target video representation tokens; designating, by the computing system, at least some of the plurality of image tokens as masked tokens and at least some of the plurality of image tokens as unmasked tokens; processing, by the computing system, only the unmasked tokens with a student video representation model to generate a plurality of partial video representation tokens; shuffling, by the computing system, the plurality of partial video representation tokens.after shuffling, by the computing system, the plurality of partial video representation tokens, processing, by the computing system, the plurality of partial video representation tokens with a token-wise decoder model to predict one or more remaining video representation tokens; and modifying, by the computing system, one or more values of one or more parameters of at least the student video representation model based on the masked video modeling loss that compares the one or more remaining video representation tokens with at least some of the plurality of target video representation tokens.

8. The computer-implemented method of claim 7, further comprising, for each of the one or more of the training examples: processing, by the computing system, the respective video with the teacher video representation model to generate a global target video representation; processing, by the computing system, the plurality of partial video representation tokens with a global decoder model to generate a predicted global video representation; and modifying, by the computing system, one or more values of one or more parameters of at least the student video representation model based on a global distillation loss that compares the global target video representation with the predicted global video representation.

9. The computer-implemented method of claim 7 or claim 8, wherein the teacher video representation model and the student video representation model comprise two separate instantiations of a partially-trained video representation model.

10. The computer-implemented method of claim 9, wherein the partially-trained video representation model has been trained using a video-text contrastive loss.

11. The computer-implemented method of any of claims 7-10, further comprising: adapting, by the computing system, the student video representation model to perform a video understanding task, wherein the video understanding task comprises: classification and localization, video-text retrieval, video captioning, or video question answering.

12. A computing system comprising a machine-learned video representation model that has been trained according to the method of any of claims 1-11.

13. One or more non -transitory computer-readable media that collectively store a machine-learned video representation model that has been trained according to the method of any of claims 1-11.

Images