Deep image tokens for masked machine learning targets

Abstract

Techniques are described herein for processing video data. For example, a computing device can project, via a projection network, the plurality of patches to generate features representing the plurality of patches. The computing device can generate a first similarity score based on a projection of the features representing the plurality of patches onto features of a prototype. The computing device can generate a second similarity score based on a projection of predicted masked target projection tokens output from a decoder onto the features of the prototype. The computing device can predict, via a clustering engine, a first cluster assignment based on the second similarity score. The computing device can predict, via the clustering engine, a second cluster assignment based on the first similarity score. The computing device can determine, via the clustering engine, a loss based on the first cluster assignment and the second cluster assignment.

Description

PATENT Qualcomm Ref. No.2402956WO DEEP IMAGE TOKENS FOR MASKED MACHINE LEARNING TARGETS TECHNICAL FIELD

[0001] The present disclosure generally relates to processing image data using one or more machine learning models (e.g., neural networks). For example, aspects of the present disclosure include systems and techniques for using deep image tokens for video masked machine learning (e.g., autoencoder) targets. BACKGROUND

[0002] Machine learning models (e.g., deep neural networks, such as large language models (LLMs), convolutional neural networks, transformers, diffusion models, etc.) are trained to provide an inference or prediction based on input data. For example, deep neural networks (e.g., LLMs, etc.) can be pre-trained on large datasets to generalize to a wide range of tasks. Applications of deep neural networks include optical flow estimation, text summarization, text generation, sentiment analysis, content creation such as performing generative operations, chatbots, virtual assistants, and conversational artificial intelligence, named entity recognition, speech recognition and synthesis, image annotation, text-to-speech synthesis, spell correction, machine translation, recommendation systems, fraud detection, accomplishing tasks and code generation. SUMMARY

[0003] Systems and techniques are described herein for using deep image tokens for video masked machine learning (e.g., autoencoder) targets.

[0004] In some aspects, an apparatus to process video data is provided. The apparatus includes one or more memories configured to store a plurality of patches associated with a video and one or more processors coupled to the one or more memories and configured to: project, via a projection network, the plurality of patches to generate features representing the plurality of patches; generate a first similarity score based on a projection of the features representing the plurality of patches onto features of a prototype; generate a second similarity score based on a projection of predicted masked target projection tokens output from a decoder onto the features of the prototype; predict, via a clustering engine, a first cluster assignment based on the second similarity score; predict, via thePATENT Qualcomm Ref. No.2402956WO clustering engine, a second cluster assignment based on the first similarity score; and determine, via the clustering engine, a loss based on the first cluster assignment and the second cluster assignment.

[0005] In some aspects, a method for processing video data is provided. The method includes: projecting, via a projection network, a plurality of patches associated with a video to generate features representing the plurality of patches; generating a first similarity score based on a projection of the features representing the plurality of patches onto features of a prototype; generating a second similarity score based on a projection of predicted masked target projection tokens output from a decoder onto the features of the prototype; predicting, via a clustering engine, a first cluster assignment based on the second similarity score; predicting, via the clustering engine, a second cluster assignment based on the first similarity score; and determining, via the clustering engine, a loss based on the first cluster assignment and the second cluster assignment.

[0006] In some aspects, an apparatus to process video is provided. The apparatus includes: means for projecting, via a projection network, a plurality of patches associated with a video to generate features representing the plurality of patches; means for generating a first similarity score based on a projection of the features representing the plurality of patches onto features of a prototype; means for generating a second similarity score based on a projection of predicted masked target projection tokens output from a decoder onto the features of the prototype; means for predicting, via a clustering engine, a first cluster assignment based on the second similarity score; means for predicting, via the clustering engine, a second cluster assignment based on the first similarity score; and means for determining, via the clustering engine, a loss based on the first cluster assignment and the second cluster assignment.

[0007] In some aspects, a non-transitory computer-readable medium is provided having stored thereon instructions which, when executed by one or more processors, cause the one or more processors to train a neural network by being configured to: project, via a projection network, a plurality of patches associated with a video to generate features representing the plurality of patches; generate a first similarity score based on a projection of the features representing the plurality of patches onto features of a prototype; generate a second similarity score based on a projection of predicted masked target projection tokens output from a decoder onto the features of the prototype; predict, via a clustering engine, a first cluster assignment based on the second similarity score; predict, via the clustering engine, a second cluster assignment based on the first similarity score; andPATENT Qualcomm Ref. No.2402956WO determine, via the clustering engine, a loss based on the first cluster assignment and the second cluster assignment.

[0008] In some aspects, one or more of apparatuses described herein include a mobile device (e.g., a mobile telephone or so-called “smart phone” or other mobile device), a wireless communication device, a vehicle or a computing device, system, or component of the vehicle or an autonomous driving vehicle, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, an extended reality (XR) or a mixed reality (MR) device), a wearable device, a personal computer, a laptop computer, a server computer, a camera, or other device, devices used for image / video editing and image / video generation and editing. In some aspects, the one or more processors include an image signal processor (ISP). In some aspects, each apparatus includes a camera or multiple cameras for capturing one or more images. In some aspects, each apparatus includes an image sensor that captures the image data. In some aspects, each apparatus includes a display for displaying the image, one or more notifications (e.g., associated with processing of the image), and / or other displayable data.

[0009] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0010] The foregoing, together with other features and aspects, will become more apparent upon referring to the following specification, claims, and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Illustrative aspects of the present application are described in detail below with reference to the following figures:

[0012] FIG.1A is a diagram illustrating an architecture for using clustering (e.g., online clustering), in accordance with some aspects of this disclosure;

[0013] FIG. 1B is a diagram illustrates an alternate architecture for using clustering (e.g., online clustering), in accordance with some aspects of this disclosure;

[0014] FIG. 2 illustrates an example process for using a transformer-based neural ordinary differential equation, according to some aspects of this disclosure;PATENT Qualcomm Ref. No.2402956WO

[0015] FIG. 3 is a block diagram illustrating an example of a deep learning network, in accordance with some aspects of this disclosure; and

[0016] FIG.4 is a diagram illustrating an example system architecture for implementing certain aspects described herein, in accordance with some aspects of this disclosure. DETAILED DESCRIPTION

[0017] Certain aspects of this disclosure are provided below. Some of these aspects may be applied independently and some of them may be applied in combination as would be apparent to those of skill in the art. In the following description, for the purposes of explanation, specific details are set forth in order to provide a thorough understanding of aspects of the application. However, it will be apparent that various aspects may be practiced without these specific details. The figures and description are not intended to be restrictive.

[0018] The ensuing description provides example aspects only, and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the ensuing description of the example aspects will provide those skilled in the art with an enabling description for implementing an example aspect. It should be understood that various changes may be made in the function and arrangement of elements without departing from the scope of the application as set forth in the appended claims.

[0019] Machine learning systems (e.g., deep neural network systems or models, such as large language models (LLMs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), transformers, diffusion models, etc.) can be used to perform a variety of tasks such as, for example and without limitation, analyzing optical flow, generative modeling such as text-to- image generation and text-to-video generation, computer code generation, text generation, speech recognition, natural language processing tasks, detection and / or recognition (e.g., scene or object detection and / or recognition, face detection and / or recognition, speech recognition, etc.), depth estimation, pose estimation, image reconstruction, classification, three-dimensional (3D) modeling, dense regression tasks, data compression and / or decompression, and image processing, among other tasks. Moreover, machine learning models can be versatile and can achieve high quality results in a variety of tasks. The principles disclosed herein will be described in the context of optical flow but can be applied in other areas or tasks as well.PATENT Qualcomm Ref. No.2402956WO

[0020] A video masked autoencoder (VideoMAE) is one example of a machine learning system (e.g., a neural network). The VideoMAE includes models that are trained using a self-supervised pretraining approach, which attempts to initialize the parameters (e.g., weights) of a given model with general features that can be used in one or more down-stream tasks. The ability to re-use the learned features in down-stream tasks gives such a parameter initialization a superior computational efficiency and performance compared to the situation where the weights are initialized randomly and optimized for each task separately. In VideoMAE, the general features are learned by training a model that receives a masked video and predicts the masked locations in the red-green-blue (RGB) space. The loss function used is L2, which is a known mean square error (MSE) loss function that quantifies the magnitude of the error between a machine learning algorithm prediction and an actual output by taking the average of the squared difference between the predictions and the target values. However, while using the L2loss can guide the network to make the predictions similar to ground truth targets, it makes the predictions blurred as well. The blurring happens due to minimizing the average distance of the prediction and target vectors instead of per-element distance.

[0021] In the video domain, as opposed to the image domain, motion across video frames provides semantically-relevant information in addition to the information in the frames themselves. Such motion can be modeled by object movements occurring between consecutive frames. The movement may be subtle and not as dominant as spatial information. Therefore, the better a model is able to leverage such motion information, the better the quality of the learned features. Some models (e.g., the VideoMAE model) use the L2loss as an objective function. However, such models do not attempt to precisely reconstruct motion clues existing at each corresponding region (referred to herein as a “tube”) of a number of frames across time, as the L2 loss minimizes the average reconstruction error and motion clues are often subtle are thus may not be statistically significant in the average reconstruction error. As a result, a solution utilizing such a model may not be optimal for motion modeling where motion sensitive tasks exist.

[0022] Systems and techniques are described herein for using deep image tokens for video masked machine learning (e.g., autoencoder) targets. In some aspects, the deep image tokens can be used to train a machine learning model (e.g., a neural network). In some case, the systems and techniques can use masked features from a plurality of patches associated with frames of a videoPATENT Qualcomm Ref. No.2402956WO to generate a first similarity score relative to a prototype. The systems and techniques can also use an output from a decoder to generate a second similarity score relative to the prototype. The systems and techniques can then predict (e.g., in a symmetric manner) a first clustering assignment from the second similarity score and a second clustering assignment from the second similarity score to obtain a loss. The loss can be used to train the machine learning model (e.g., the neural network).

[0023] In one illustrative example, a video is transformed into a set of equal cubic tubes. Each tube can be made up of a corresponding (e.g., commonly-located) region (or patch) in each frame of a plurality of frames of the video. Some of the regions or patches in each frame are randomly masked and the others are passed to a neural network encoder. The pixel data (or features) associated with the masked regions (or patches) of the frames are passed to a projection network to create projected feature tokens. A decoder receives the output from the neural network encoder and predicts feature tokens. The decoder’s predicted feature tokens and the projected feature tokens can be made similar using a clustering loss (e.g., an online clustering loss) and a set of prototypes.

[0024] The systems and techniques can improve the generation of a loss value used to train the machine learning system. As mentioned above, L2loss is one of the factors that can lead to suboptimal modelling of motion clues in videos. As the loss is applied in the pixel space, the model is required to reconstruct a large amount of low-level information, which is irrelevant to the modelling of the motion clues and are more focused on the spatial aspects. The systems and techniques described herein can improve machine learning model (e.g., neural network model) training by providing a loss value that is more aligned with motion cues.

[0025] FIG.1A illustrates a self-supervised video clustering model 100. First, an input such as video 101 (which can be a downsampled video clip with a series of frames over time) is transformed into sets of patches 102 having a set of equal cubic tubes. Patches from the sets of patches 102 are randomly masked and other patches are unmasked. The unmasked patches are referred to as unmasked tokens 104. The unmasked tokens 104 are input to an encoder 106 or encoder network for processing. The encoder 106 can generate features 108 (e.g., unmasked latent feature tokens or embeddings) that represent the unmasked tokens 104. The features 108 can be combined with learnable features, embeddings, or tokens (which can be referred to as placeholderPATENT Qualcomm Ref. No.2402956WO features or embeddings) that represent the masked patches from the sets of patches 102 to generate output 110. A tube 103 is a series of patches associated for example with a patch 105 at a same location or region of a frame over time or a series of images as is shown in FIG.1A. One tube 103 is shown but each patch can be part of a tube 103 which defines or organizes the patch 105 and corresponding patches across the various images. Some of the patches can be masked where the various set of masked patches can be defined as masked tubes. Masked tubes (XPatch) can be passed to a projection head or projection network P 116 to create projected feature tokens (XPrj).

[0026] A decoder 112 receives the output 110 and predicts projection feature tokens or x∅114 represented as xϕ. The decoder’s predictions (e.g., the projection feature tokens or x∅114) are then made similar using a clustering engine 122 and a set of prototypes 118 using a first similarity component 120 and a second similarity component 124. The clustering engine 122 can be configured to cluster (e.g., group) the features output by the decoder 112 (e.g., the projection feature tokens or x∅114) and features from the projection head P 116 (e.g., projected feature tokens XPrj) based on high-level commonalities among the patches from which the features are predicted. In some cases, the clustering engine 122 is an online clustering engine (e.g., the clustering engine 122 operates in an online manner, such as at an inference stage of the model 100). The clusters (corresponding to common patterns in the patches) can be stored as prototypes 118. For instance, after clustering is performed by the clustering engine 122, the prototypes 118 can be updated. In some cases, the prototypes 118 include the most common patterns that are observed. In one aspect, a first similarity score is obtained via the first similarity component 120 and a second score is obtained from the second similarity component 124. The first similarity score can be used by the clustering engine 122 (or the swapped prediction loss module 123 of FIG.1B) to predict a first clustering assignment and the second similarity score can be used by the clustering engine 122 to predict a second clustering assignment. The clustering assignments relate to the commonalities of features across patches.

[0027] Vision transformers in general operate on a sequence of patches of fixed size ^^ ൈ ^^. Fora color image I of spatial size ^^ ൈ ^^, one obtains ^^ ൌ ுൈ^ଶ^మpatches of size ^^ (assume for simplicity that H and W are multiples of P). Each patch is first embedded in a d-dimensional latent space via a trained linear projection layer (e.g., the projection network P 116). An additional, learned vector called the “class token”, CLS (which stands for classification), is adjoined to thePATENT Qualcomm Ref. No.2402956WO patch embeddings, yielding a transformer input in ^^^ேା^^ௗ. In some aspects, one does not have to use a CLS token in the pretraining pipeline. An example of a vision transformer is found in Dosovitskiy, Alexey, et al. "An image is worth 16x16 words: Transformers for image recognition at scale," arXiv:2010.11929 (2020), incorporated herein by reference.

[0028] An example algorithm that can be used to solve an optimal assignment problem implements an iterative approximation. For example, the Sinkhorn-Knopp algorithm can be used as described in Cuturi, Marco. "Sinkhorn distances: Lightspeed computation of optimal transport," Advances in neural information processing systems, 26 (2013), incorporated herein by reference. In some aspects of this disclosure, one can look after finding the optimal assignment of spatial patches to a set of prototypes 118. The main benefit of the Sinkhorn-Knopp algorithm is keeping the entropy of assignment between the patches and prototypes to a given minimum threshold, which avoids getting into trivial solutions and mode collapse.

[0029] As noted above, L2 loss in VideoMAE is one of the factors that can lead to suboptimal modelling of motion clues in videos. Furthermore, as the loss is applied in the pixel space the model is required to reconstruct a lot of low-level information, which are irrelevant to the modelling of the motion clues and are more focused on the spatial aspects. For instance, suppose a car is moving from A to B, it shouldn’t matter for the model what the color of the car is; yet the VideoMAE loss function explicitly forces the model to reconstruct all the information, including the color. The inclusion of color in the analysis of the loss makes it even harder for the model to attend to subtle temporal differences, reflecting motion, compared to spatial information that are usually more dominant.

[0030] In the realm of standard masked video modeling, the primary objective is the precise reconstruction of masked pixel values. Specifically, a video V or the video 101, composed of frames {x1,...,xT }, is segmented into space-time tubes t with a significant proportion (approximately 90%) being masked due to the inherent redundancy in video data. An asymmetric encoder-decoder architecture is trained to accurately reconstruct the pixels of the masked space- time tubes tMleveraging the unmasked tubes tN and the positional information of the masked patches via a loss function as Eq.1: ^ ே ^^^(1)PATENT Qualcomm Ref. No.2402956WO

[0031] where ^^ represents the number of masked space-time tubes, ^^^denotes the pixels of the ^^-th space-time tube, and ^^^^^ே^^represents the reconstructed pixels for this tube given the set of all non-masked tubes ^^ே. The loss function of Eq.1 encourages strictly predicting the pixel values of the input video or the video 101. Thus, mainly low-level features are captured in the (trained) model or the video model ^^ (see the video model ^^ 156 in FIG.1B) as evidenced by low linear probing performance. Yet, video reasoning necessitates an understanding of how scenes and objects interact with one another and transform across time. To achieve such a higher-level semantic understanding, the systems and techniques described herein can first exchange the target space in which the reconstruction loss is applied, to a deeper feature space.

[0032] To address the issue of what data the traditional loss function represents, there are several changes to the existing framework that are disclosed herein. First, the system projects the target from the pixel space to the latent space or the more meaningful feature space by the projection head or projection network P 116. Next, the approach includes applying a clustering method (e.g., an online clustering method), which can be performed by the clustering engine 122. As noted previously, the clustering can include group features output by the decoder 112 and features from the projection head P 116 based on high-level commonalities among the patches from which the features are predicted. The clusters (corresponding to common patterns in the patches) can be stored as prototypes 118. Non-limiting examples of online clustering can include the SwAV (Swapped-Contrastive Learning of Visual Representations) approach described in Ziegler, Adrian, and Yuki M. Asano. “Self-Supervised Learning of Object Parts for Semantic Segmentation,” Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (2022), incorporated herein by reference. The SwAV approach applies in the context of self-supervised learning for visual representations. The approach is designed to learn effective visual representations from unlabeled data. To make the predictions and corresponding targets similar by contrasting cluster assignments.

[0033] Mathematically, suppose the input or the video 101 is ^^ the, ^^^^^௧^^is the ^^th output of a function Patchify(x) (e.g., such as a patchify component 152 shown in FIG.1B) that receives the video 101 and returns equal cubic tubes across time, such as tube 103. A functionMask(^^^^௧^^, ^^^^^^^^^^^ (such as the masking component 154 of FIG. 1B) receives all the tubes andmasks one or more patches of the set of patches 102 based on a given ratio. For example, forPATENT Qualcomm Ref. No.2402956WO images / video with lots of sky patches that are just blue, many of those patches can be masked, with remaining patches with more complex pixel data being unmasked. The function Prj head(^^^^^௧^^^ is performed by the projection head or projection network P 116 that receives each tube and projects the tube to a target projection tokens. The encoder 106 can be an encoder^∙^ that receives unmasked patches and encodes them as features 108 (e.g., unmasked latent features or embeddings). The decoder 112 can be a decoder^∙^ that receives the features 108 (e.g., unmasked latent features, embeddings, or tokens) and appends learnable masked tokens (e.g., the placeholder features or embeddings) to generate output 110. The decoder^∙^can predict the masked target projection tokens or x∅114 or ^^^, and a set of learnable prototypes ^^ or the set of prototypes 118. A clustering loss (e.g., an online clustering loss) can then be defined from the following steps:

[0034] 1. ^^^^௧^^ ൌ Patchify^^^^

[0035] 2. ^^^^^ ൌ Prj_head^^^^^௧^^^

[0036] 3. ^ ^^ ൌ Encoder^Decoder^^^^^௧^^^^

[0037] 4. Sim^^^_^ൌ Normalize^^^^^^^்Normalize^^^^

[0038] 5. Sim௬^_^ൌ Normalize^^^^^்Normalize^^^^

[0039] 6. Loss = Online_Clustering(Sim^^^_^, Sim௬^_^)

[0040] Thedefined as: Online_Clustering(Sim^^^_^ , Sim௬^_^) ൌ Cross_Entropy^Sim^^^_^ , Sinkhorn(Sim௬^_^^)+ Cross_Entropy^Sim௬^_^, Sinkhorn(Sim^^^_^^)

[0041] where the reference to Sinkhorn is the abbreviation of Sinkhorn-Knopp. The Sinkorn algorithm is one example algorithm that can be used. Other algorithms are contemplated as well. Experiments show that the proposed method achieves considerably better performance on large- scale datasets which supports the applicability of the disclosed method across different tasks.

[0042] FIG. 1B illustrates an alternate infrastructure for the self-supervised video clustering approach 150. Many of the elements in FIG. 1B are also shown in FIG. 1A but reconfigured to illustrate other aspects of the approach. A video model ϕ 156 (which can be an encoder-decoder video model) is shown to include the encoder 106 and decoder 112 and respective inputs and outputs similar to FIG.1A.PATENT Qualcomm Ref. No.2402956WO

[0043] An overview of the self-supervised video clustering approach 150 is as follows. A video 101 is embedded with the projection network ^^ 116 leading to features such as x^162. The video model ^^ 156 predicts feature embeddings or xம114 of the masked space-time tubes. Both embeddings are projected (via a first similarity component 164 and a second similarity component 166) onto the learnable prototypes or set of prototypes 118 representing cluster centroids. Cluster assignments are created with an adapted Sinkhorn algorithm (or other algorithm) enforcing equipartition across all prototypes as performed by a swapped prediction loss module123. The pseudo labels ^^థ, ^^^ (see a first algorithm 170 and a second algorithm 178 in FIG. 1B,which can be the Sinkhorn algorithm) are then used as targets for the predictive task ℒ^ாwith which the networks are optimized.

[0044] Rather than directly predicting pixel values, the approach is to change the target space to a deep feature space by introducing the projection network P 116 that can be jointly optimized. The projection network P 116 embeds space-time tubes T from the video 101 into first features 158 which, after masking 160 (or inverse masking), are represented as deep features xP= P(t) or xP162 representing as shown in the lower stream in FIG. 1B. Specifically, the video 101,composed of frames ^^^^, ... , ^^்^, is segmented into a set of space-time tubes T (see tube 103 inFIG. 1A) with a significant proportion (in some aspects, the proportion can be approximately 90%) being masked via a masking component 154 motivated by the inherent redundancy in video data. Thus, the target space has been replaced, and Eq.1 changes to Eq.2: ^^ி ൌ ^ே ^‖^^ ^^^ െ ^^థ^^ ฮ2(2) with the prediction xϕleads to a trivial solution as both the parameters of the projection network P and a video model ϕ 156 have been jointly optimized. To address the trivial solution, this disclosure introduces a self- supervised clustering approach utilizing an algorithm, such as, for example, the Sinkhorn algorithm, for optimal transport. The feature space cam be constrained to be partitioned into a limited number of clusters. The approach can be interpreted as a bottleneck with enforced high entropy.

[0045] The approach can be accomplished in an online manner by mapping the features ^^ ofspace-time tubes to a set of learnable prototypes ^^ ൌ ^^^^, … , ^^^^, representing cluster centroids.PATENT Qualcomm Ref. No.2402956WOFor example, a mapping ^^^^ℝ^^^ା of embeddings ^^ ൌ ^^^^, … , ^^^^ to prototypes ^^ can bedefined. An equipartition constraint can be introduced on ^^ to enforce that all prototypes are equally used, mitigating the trivial solution of all data points collapsing onto a single prototype. By that, similar and nearby space-time tubes are assigned to the similar prototypes, infusing semantic spatial and temporal meaning into the feature space. The assignment can be formulated as an entropy-regularized optimal transport problem for all space-time tubes over a mini-batch ℬ min 1 (3) ^^ 〈^^,െlog ^^〉 ^λKL^^^||^^^^^^,1 with ^^ ൌ ^^^, ^^ ൌ(4) ^ ^^. ^^ ^^

[0046] Here ^^ controls the entropy regularisation, ^^, ^^, are the marginals for the prototypes, and^^ ൌ ^^^ x ^^் represents all samples ^^^ in the mini-batch ^^ for all space-time tubes ^^். TheSinkhorn algorithm can be used to solve Eq. 4 which can be done extremely quickly on thegraphical processing unit (GPU) and yields soft pseudo-labels ^^, such that argmax ^^^^ ൌ ^^.Pseudo-labels can be generated for the embedding such as x^162 of the projection network P 116 which then can be used as targets for the video model ^^ 156 and vice versa. A goal of ^^ and ^^ is to predict each other’s cluster assignments ^^ based on the projection of the features onto the shared prototypes ^^ or the set of prototypes 118. The predictive task can be formulated via cross-entropy between the pseudo label ^^ and the probability distribution achieved by applyingsoftmax to the scores x^ ൌ x^^^ the dot products between x and all ^^ prototypes in ^^, whichmay be expressed as follows: ℒ^ா^^^^,^^^ ൌ െ^ ^^^^^ log^^^^^(5)(6)

[0047] In the above Eq.all space-time tubes ^^ in a mini-batch ^^ can be written as a symmetric prediction task:PATENT Qualcomm Ref. No.2402956WO ^ 1^ா^^^^,^^థ൯ ^ ℒ^ா൫^^^థ(7) ℒൌ^^ℒ ൫^,^^^൯൧ ^^^ ^ ^with thevideo model ^^ 156 one that despite the dual loss, the disclosed approach circumvents the need for any augmentation to create different views in conjunction or the need for a momentum encoder. The approach allows one to not be constrained by the architecture choice for the projection network ^^ 116 and ablate or remove different choices.

[0048] In FIG. 1B, a swapped prediction loss module 123 can correspond or perform similar functions to the clustering engine 122 of FIG. 1A. The approach involves mapping the learned embeddings of space-time tubes to a set of prototypes 118 such as a set of learnable prototypes C = {c1,...,cK}, representing cluster centroids and which are represented as a set of prototypes 118. The mapping of embeddings or xϕ114 to the set of prototypes 118 as codes Q = [q1,...,qB] can be performed by a first similarity component 164 which aims to enhance the correspondence between the embeddings and the centroids, which may be expressed a follows: ^ ^^^∈^^^^^^^^^^^^^்^^்^^^ ^ ^^^^^^^^ ^8^

[0049] where H denotes theinfluencing the mapping’s ^^entropy results in a uniform distribution of all samples across all prototypes, leading to a loss of distinctiveness. A first score 168 is obtained from the mapping of the feature embeddings or xϕ114 to the set of prototypes 118. The first score 168 can be provided to a first algorithm 170 (e.g., an example of which can be the Sinkhorn algorithm as described above). A second score 176 is obtained from the mapping of xP162 to the set of prototypes 118. The second score can be provided to a second algorithm 178 (e.g., the Sinkhorn algorithm as described above). The approach includes maintaining a low ^ value. Furthermore, the approach also ensures an even distribution across clusters by requiring Q to satisfy the constraints of the transportation polytope as defined in Yuki M. Asano et al, Self-Labelling Via Simultaneous Clustering and Representation Learning, arXiv:1911.05371v3, 19 February 2020, incorporated herein by reference. Due to the constrained number of prototypes K, both similar patches and patches originating from the same object are allocated to identical prototypes, infusing semantic significance into the feature space.PATENT Qualcomm Ref. No.2402956WO

[0050] The prototypes for a specific video are depicted in Fig.1B. The disclosed scenario sets the stage for a task defined by contrasting cluster assignments between the embedding produced by the projection network P 116 (e.g., xP162) and the embeddings derived from xϕ114. The systems and techniques can predict the codes ^^^^^^ℎ^^^^^^, t, the cluster assignment of xP162, based on the projection of xϕ114 onto the set of prototypes 118 (e.g., the set of prototypes C), and vice versa. The process is performed by the swapped prediction loss module 123 with a first loss equation 172 and a second loss equation 174 operating on the first score 168, the second score 176, and a first output from the first algorithm 170 and a second output from the second algorithm 178 as shown in FIG.1B. The predictive task can be formulated via cross-entropy between the code q and the probability distribution achieved by applying softmax (which is a function that takes a vector of real numbers and transforms them into a probability distribution) to the dot products between x and all K prototypes in C, which may be expressed as follows: ℓ൫^^^, ^^థ൯ ൌ െ∑ థ^ ^^^^^^ ^^^^^^ ^^^^^^9^

[0051] where τ represents thetube t is conceptualized as a ^^^ா൫^^^,^^థ൯ ൌ ℓ൫^^^,^^థ൯ ^ ℓ൫^^థ,^^^൯ ^11^

[0052] The loss, being ofdense space-time features, solely to entire images. Furthermore, by leveraging a masked video modeling approach, the disclosed method alleviates the need for augmentations to generate different views.

[0053] FIG.2 is a flowchart illustrating an example process 200 to train a neural network. In one example, the process 200 can be performed by a system or apparatus that can include one or more of the projection network P 116, an encoder-decoder video model such as video model ϕ 156, a neural network 300, the clustering engine 122, the swapped prediction loss module 123, a transformer, an encoder 106, a decoder 112, a projection network P 116, a computing system 400, or any subset therefore or a combination thereof. For instance, a computing device with the computing device architecture of the computing system 400 shown in FIG.4 can implement thePATENT Qualcomm Ref. No.2402956WO operations of FIG.2 and / or the components and / or operations described herein with respect to any of FIGs.1A, 1B, 2 and / or 3.

[0054] At operation 202, the system (e.g., one or more of the projection network P 116, an encoder-decoder video model such as video model ϕ 156, a neural network 300, the clustering engine 122, the swapped prediction loss module 123, a transformer, an encoder 106, a decoder 112, a projection network P 116, a computing system 400, or any subset therefore or a combination thereof) can be configured to, and can, project, via a projection network P 116, a plurality of patches (e.g., the set of patches 102 in FIG.1A) to generate features representing the plurality of patches. In some aspects, patches in the plurality of patches are each masked. The plurality of patches can be in a pixel space and the features representing the plurality of patches are in a feature space.

[0055] In some aspects, the projection network P 118 can include a multilayer perceptron or some other type of neural network.

[0056] At operation 204, the system (e.g., one or more of the projection network P 116, an encoder-decoder video model such as video model ϕ 156, a neural network 300, the clustering engine 122, the swapped prediction loss module 123, a transformer, an encoder 106, a decoder 112, a projection network P 116, a computing system 400, or any subset therefore or a combination thereof) can be configured to, and can, generate a first similarity score based on a projection of the features representing the plurality of patches onto features of a prototype. In some aspects, the prototype can be a learnable prototype.

[0057] At operation 206, the system (e.g., one or more of the projection network P 116, an encoder-decoder video model such as video model ϕ 156, a neural network 300, the clustering engine 122, the swapped prediction loss modulea transformer, an encoder 106, a decoder 112, a projection network P 116, a computing system 400, or any subset therefore or a combination thereof) can be configured to, and can, generate a second similarity score based on a projection of predicted masked target projection tokens output from a decoder onto the features of the prototype.

[0058] At operation 208, the system (e.g., one or more of the projection network P 116, an encoder-decoder video model such as video model ϕ 156, a neural network 300, the clustering engine 122, the swapped prediction loss module 123, a transformer, an encoder 106, a decoder 112, a projection network P 116, a computing system 400, or any subset therefore or a combinationPATENT Qualcomm Ref. No.2402956WO thereof) can be configured to, and can, predict, via a clustering engine (e.g., an online clustering engine), a first cluster assignment based on the second similarity score. In some aspects, the online clustering engine 122 applies a Sinkhorn-Knopp algorithm.

[0059] At operation 210, the system (e.g., one or more of the projection network P 116, an encoder-decoder video model such as video model ϕ 156, a neural network 300, the clustering engine 122, the swapped prediction loss module 123, a transformer, an encoder 106, a decoder 112, a projection network P 116, a computing system 400, or any subset therefore or a combination thereof) can be configured to, and can, predict, via the clustering engine, a second cluster assignment based on the first similarity score. In some aspects, the one or more processors coupled to the one or more memories are configured to predict the first cluster assignment and predict the second cluster assignment in a symmetric manner without a chronological order. In some aspects, the first cluster assignment and the second cluster assignment relate to common patterns amongst the plurality of patches.

[0060] At operation 212, the system (e.g., one or more of the projection network P 116, an encoder-decoder video model such as video model ϕ 156, a neural network 300, the clustering engine 122, the swapped prediction loss module 123, a transformer, an encoder 106, a decoder 112, a projection network P 116, a computing system 400, or any subset therefore or a combination thereof) can be configured to, and can, determine, via the clustering engine 122, a loss based on the first cluster assignment and the second cluster assignment.

[0061] Furthermore, the process 200 can include the system (e.g., one or more of the projection network P 116, an encoder-decoder video model such as video model ϕ 156, a neural network 300, the clustering engine 122, the swapped prediction loss module 123, a transformer, an encoder 106, a decoder 112, a projection network P 116, a computing system 400, or any subset therefore or a combination thereof) can be configured to, and can: optimize, based on the loss, first weights associated with the projection network P 116 and second weights associated with the decoder 112.

[0062] In some aspects, an apparatus (e.g., one or more of the projection network P 116, an encoder-decoder video model such as video model ϕ 156, a neural network 300, the clustering engine 122, the swapped prediction loss module 123, a transformer, an encoder 106, a decoder 112, a projection network P 116, a computing system 400, or any subset therefore or a combination thereof) to train a neural network can include one or more memories (e.g., memory 415, ROMPATENT Qualcomm Ref. No.2402956WO 420, RAM 425, cache 412 or combination thereof); and one or more processor (e.g., processor 410) coupled to the one or more memories and configured to: project, via a projection network, the plurality of patches to generate features representing the plurality of patches; generate a first similarity score based on a projection of the features representing the plurality of patches onto features of a prototype; generate a second similarity score based on a projection of predicted masked target projection tokens output from a decoder onto the features of the prototype; predict, via a clustering engine (e.g., an online clustering engine), a first cluster assignment based on the second similarity score; predict, via the clustering engine, a second cluster assignment based on the first similarity score; and determine, via the clustering engine, a loss based on the first cluster assignment and the second cluster assignment.

[0063] In some aspects, an apparatus (e.g., one or more of the projection network P 116, an encoder-decoder video model such as video model ϕ 156, a neural network 300, the clustering engine 122, the swapped prediction loss module 123, a transformer, an encoder 106, a decoder 112, a projection network P 116, a computing system 400, or any subset therefore or a combination thereof) is disclosed to train a neural network, the apparatus including one or more: means for projecting, via a projection network, the plurality of patches to generate features representing the plurality of patches; means for generating a first similarity score based on a projection of the features representing the plurality of patches onto features of a prototype; means for generating a second similarity score based on a projection of predicted masked target projection tokens output from a decoder onto the features of the prototype; means for predicting, via a clustering engine (e.g., an online clustering engine), a first cluster assignment based on the second similarity score; means for predicting, via the clustering engine, a second cluster assignment based on the first similarity score; and means for determining, via the clustering engine, a loss based on the first cluster assignment and the second cluster assignment.

[0064] In some aspects, a computer-readable device (e.g., memory 415, ROM 420, RAM 425, cache 412 or combination thereof) stores instructions which, when executed by one or more processors, cause the one or more processors to be configured to: project, via a projection network, the plurality of patches to generate features representing the plurality of patches; generate a first similarity score based on a projection of the features representing the plurality of patches onto features of a prototype; generate a second similarity score based on a projection of predictedPATENT Qualcomm Ref. No.2402956WO masked target projection tokens output from a decoder onto the features of the prototype; predict, via a clustering engine (e.g., an online clustering engine), a first cluster assignment based on the second similarity score; predict, via the clustering engine, a second cluster assignment based on the first similarity score; and determine, via the clustering engine, a loss based on the first cluster assignment and the second cluster assignment.

[0065] As described herein, the neural network 300 of FIG.3 may be implemented using a neural network or multiple neural networks. FIG. 3 is an illustrative example of a deep learning neural network such as neural network 300 that can be used by the neural network 300 of FIG.3. An input layer 320 includes input data. In one illustrative example, the input layer 320 can include data representing the pixels of an input video frame. The neural network 300 includes multiple hidden layers 322a, 322b, through 322n. The hidden layers 322a, 322b, through 322n include “n” number of hidden layers, where “n” is an integer greater than or equal to one. The number of hidden layers can be made to include as many layers as needed for the given application. The neural network 300 further includes an output layer 324 that provides an output resulting from the processing performed by the hidden layers 322a, 322b, through 322n. In one illustrative example, the output layer 324 can provide a classification for an object in an input video frame. The classification can include a class identifying the type of object (e.g., a person, a dog, a cat, or other object).

[0066] The neural network 300 is a multi-layer neural network of interconnected nodes. A node 326 can represent a piece of information. Information associated with the nodes is shared among the different layers and each layer retains information as information is processed. In some cases, the neural network 300 can include a feed-forward network, in which case there are no feedback connections where outputs of the network are fed back into itself. In some cases, the neural network 300 can include a recurrent neural network, which can have loops that allow information to be carried across nodes while reading in input.

[0067] Information can be exchanged between nodes through node-to-node interconnections between the various layers. Nodes of the input layer 320 can activate a set of nodes in the first hidden layer 322a. For example, as shown, each of the input nodes of the input layer 320 is connected to each of the nodes of the first hidden layer 322a. The nodes of the hidden layers 322a, 322b, through 322n can transform the information of each input node by applying activation functions to the information. The information derived from the transformation can then be passedPATENT Qualcomm Ref. No.2402956WO to and can activate the nodes of the next hidden layer 322b, which can perform their own designated functions. Example functions include convolutional, up-sampling, data transformation, and / or any other suitable functions. The output of the hidden layer 322b can then activate nodes of the next hidden layer, and so on. The output of the last hidden layer 322n can activate one or more nodes of the output layer 324, at which an output is provided. In some cases, while nodes (e.g., node 326) in the neural network 300 are shown as having multiple output lines, a node has a single output and all lines shown as being output from a node represent the same output value.

[0068] In some cases, each node or interconnection between nodes can have a weight that is a set of parameters derived from the training of the neural network 300. Once the neural network 300 is trained, it can be referred to as a trained neural network, which can be used to classify one or more objects. For example, an interconnection between nodes can represent a piece of information learned about the interconnected nodes. The interconnection can have a tunable numeric weight that can be tuned (e.g., based on a training dataset), allowing the neural network 300 to be adaptive to inputs and able to learn as more and more data is processed.

[0069] The neural network 300 is pre-trained to process the features from the data in the input layer 320 using the different hidden layers 322a, 322b, through 322n in order to provide the output through the output layer 324. In an example in which the neural network 300 is used to identify objects in images, the neural network 300 can be trained using training data that includes both images and labels. For instance, training images can be input into the network, with each training image having a label indicating the classes of the one or more objects in each image (basically, indicating to the network what the objects are and what features they have). In one illustrative example, a training image can include an image of a number 2, in which case the label for the image can be [0010000000].

[0070] In some cases, the neural network 300 can adjust the weights of the nodes using a training process called backpropagation. Backpropagation can include a forward pass, a loss function, a backward pass, and a weight update. The forward pass, loss function, backward pass, and parameter update is performed for one training iteration. The process can be repeated for a certain number of iterations for each set of training images until the neural network 300 is trained well enough so that the weights of the layers are accurately tuned.PATENT Qualcomm Ref. No.2402956WO

[0071] For the example of identifying objects in images, the forward pass can include passing a training image through the neural network 300. The weights are initially randomized before the neural network 300 is trained. The image can include, for example, an array of numbers representing the pixels of the image. Each number in the array can include a value from 0 to 255 describing the pixel intensity at that position in the array. In one example, the array can include a 28 x 28 x 3 array of numbers with 28 rows and 28 columns of pixels and 3 color components (such as red, green, and blue, or luma and two chroma components, or the like).

[0072] For a first training iteration for the neural network 300, the output will likely include values that do not give preference to any particular class due to the weights being randomly selected at initialization. For example, if the output is a vector with probabilities that the object includes different classes, the probability value for each of the different classes may be equal or at least very similar (e.g., for ten possible classes, each class may have a probability value of 0.1). With the initial weights, the neural network 300 is unable to determine low level features and thus cannot make an accurate determination of what the classification of the object might be. A loss function can be used to analyze error in the output. Any suitable loss function definition can be used. One example of a loss function includes a mean squared error (MSE). The MSE is definedas ^^௧^௧^^ ൌ ∑^ ଶ^^^^^^^^^^^^^ െ ^^^^^^^^^^^^^ଶ, which calculates the sum of one-half times a ground truthoutput (e.g., the actual answer) minus the predicted output (e.g., the predicted answer) squared. The loss can be set to be equal to the value of ^^௧^௧^^.

[0073] The loss (or error) will be high for the first training images since the actual values will be much different than the predicted output. A goal of training is to minimize the amount of loss so that the predicted output is the same as the training label. The neural network 300 can perform a backward pass by determining which inputs (weights) most contributed to the loss of the network, and can adjust the weights so that the loss decreases and is eventually minimized.

[0074] A derivative of the loss with respect to the weights (denoted as dL / dW, where W are the weights at a particular layer) can be computed to determine the weights that contributed most to the loss of the network. After the derivative is computed, a weight update can be performed by updating all the weights of the filters. For example, the weights can be updated so that they changein the opposite direction of the gradient. The weight update can be denoted as ^^ ൌ ^^ௗ^ ^െ ^^ௗ^, where w denotes a weight, widenotes the initial weight, and η denotes a learningPATENT Qualcomm Ref. No.2402956WO rate can be set to any suitable value, with a high learning rate including larger weight updates and a lower value indicating smaller weight updates.

[0075] In some cases, the neural network 300 can be trained using self-supervised learning.

[0076] The neural network 300 can include any suitable deep network. One example includes a convolutional neural network (CNN), which includes an input layer and an output layer, with multiple hidden layers between the input and out layers. An example of a CNN is described below with respect to FIG.10. The hidden layers of a CNN include a series of convolutional, nonlinear, pooling (for downsampling), and fully connected layers. The neural network 300 can include any other deep network other than a CNN, such as an autoencoder, a deep belief nets (DBNs), a Recurrent Neural Networks (RNNs), among others.

[0077] The proposed solution is on improving optical flow estimation, which is an important and standard task used in many practical use cases, such as automobile uses, XR, drone use, and camera uses. The proposed model can be used for video processing, temporal information aggregation, tracking, etc. One alternative to the solution disclosed above is to use the RNN / GRU- based updates in optical flow networks. Such an approach might not be as efficient compared to the proposed solution, in terms of both modeling and accuracy.

[0078] FIG.4 is a diagram illustrating an example of a system for implementing certain aspects of the present disclosure. In particular, FIG. 4 illustrates an example of computing system 400, which can be for example any computing device making up a computing system, a camera system, or any component thereof in which the components of the system are in communication with each other using connection 405. Connection 405 can be a physical connection using a bus, or a direct connection into processor 410, such as in a chipset architecture. Connection 405 can also be a virtual connection, networked connection, or logical connection.

[0079] In some examples, computing system 400 is a distributed system in which the functions described in this disclosure can be distributed within a datacenter, multiple data centers, a peer network, etc. In some examples, one or more of the described system components represents many such components each performing some or all of the function for which the component is described. In some examples, the components can be physical or virtual devices.

[0080] Example system 400 includes at least one processing unit (CPU or processor) 410 and connection 405 that couples various system components including system memory 415, such asPATENT Qualcomm Ref. No.2402956WO read-only memory (ROM) 420 and random access memory (RAM) 425 to processor 410. Computing system 400 can include a cache 411 of high-speed memory connected directly with, in close proximity to, or integrated as part of processor 410.

[0081] Processor 410 can include any general purpose processor and a hardware service or software service, such as services 432, 434, and 436 stored in storage device 430, configured to control processor 410 as well as a special-purpose processor where software instructions are incorporated into the actual processor design. Processor 410 may essentially be a completely self- contained computing system, containing multiple cores or processors, a bus, memory controller, cache, etc. A multi-core processor may be symmetric or asymmetric.

[0082] To enable user interaction, computing system 400 includes an input device 445, which can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, keyboard, mouse, motion input, speech, etc. Computing system 400 can also include output device 435, which can be one or more of a number of output mechanisms. In some instances, multimodal systems can enable a user to provide multiple types of input / output to communicate with computing system 400. Computing system 400 can include communications interface 440, which can generally govern and manage the user input and system output.

[0083] The communication interface may perform or facilitate receipt and / or transmission wired or wireless communications using wired and / or wireless transceivers, including those making use of an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, an Apple® Lightning® port / plug, an Ethernet port / plug, a fiber optic port / plug, a proprietary wired port / plug, a BLUETOOTH® wireless signal transfer, a BLUETOOTH® low energy (BLE) wireless signal transfer, an IBEACON® wireless signal transfer, a radio-frequency identification (RFID) wireless signal transfer, near-field communications (NFC) wireless signal transfer, dedicated short range communication (DSRC) wireless signal transfer, 802.11 Wi-Fi wireless signal transfer, wireless local area network (WLAN) signal transfer, Visible Light Communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), Infrared (IR) communication wireless signal transfer, Public Switched Telephone Network (PSTN) signal transfer, Integrated Services Digital Network (ISDN) signal transfer, 3G / 4G / 5G / LTE cellular data network wireless signal transfer, ad- hoc network signal transfer, radio wave signal transfer, microwave signal transfer, infrared signalPATENT Qualcomm Ref. No.2402956WO transfer, visible light signal transfer, ultraviolet light signal transfer, wireless signal transfer along the electromagnetic spectrum, or some combination thereof.

[0084] The communications interface 440 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers that are used to determine a location of the computing system 400 based on receipt of one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the US- based Global Positioning System (GPS), the Russia-based Global Navigation Satellite System (GLONASS), the China-based BeiDou Navigation Satellite System (BDS), and the Europe-based Galileo GNSS. There is no restriction on operating on any particular hardware arrangement, and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed.

[0085] Storage device 430 can be a non-volatile and / or non-transitory and / or computer-readable memory device and can be a hard disk or other types of computer readable media which can store data that are accessible by a computer, such as magnetic cassettes, flash memory cards, solid state memory devices, digital versatile disks, cartridges, a floppy disk, a flexible disk, a hard disk, magnetic tape, a magnetic strip / stripe, any other magnetic storage medium, flash memory, memristor memory, any other solid-state memory, a compact disc read only memory (CD-ROM) optical disc, a rewritable compact disc (CD) optical disc, digital video disk (DVD) optical disc, a blu-ray disc (BDD) optical disc, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a Memory Stick® card, a smartcard chip, a EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash EPROM (FLASHEPROM), cache memory (L1 / L2 / L3 / L4 / L5 / L#), resistive random-access memory (RRAM / ReRAM), phase change memory (PCM), spin transfer torque RAM (STT-RAM), another memory chip or cartridge, and / or a combination thereof.

[0086] The storage device 430 can include software services, servers, services, etc., that when the code that defines such software is executed by the processor 410, it causes the system to perform a function. In some examples, a hardware service that performs a particular function canPATENT Qualcomm Ref. No.2402956WO include the software component stored in a computer-readable medium in connection with the necessary hardware components, such as processor 410, connection 405, output device 435, etc., to carry out the function. The term “computer-readable medium” includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other mediums capable of storing, containing, or carrying instruction(s) and / or data. A computer-readable medium may include a non-transitory medium in which data can be stored and that does not include carrier waves and / or transitory electronic signals propagating wirelessly or over wired connections. Examples of a non-transitory medium may include, but are not limited to, a magnetic disk or tape, optical storage media such as compact disk (CD) or digital versatile disk (DVD), flash memory, memory or memory devices. A computer-readable medium may have stored thereon code and / or machine-executable instructions that may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, or the like.

[0087] In some aspects the computer-readable storage devices, mediums, and memories can include a cable or wireless signal containing a bit stream and the like. However, when mentioned, non-transitory computer-readable storage media expressly exclude media such as energy, carrier signals, electromagnetic waves, and signals per se.

[0088] Specific details are provided in the description above to provide a thorough understanding of the aspects and examples provided herein. However, it will be understood by one of ordinary skill in the art that the aspects may be practiced without these specific details. For clarity of explanation, in some instances the present technology may be presented as including individual functional blocks comprising devices, device components, steps or routines in a method embodied in software, or combinations of hardware and software. Additional components may be used other than those shown in the figures and / or described herein. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order not to obscure the aspects in unnecessary detail. In other instances, well-known circuits,PATENT Qualcomm Ref. No.2402956WO processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the aspects.

[0089] Individual aspects may be described above as a process or method which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be re- arranged. A process is terminated when its operations are completed, but could have additional steps not included in a figure. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination can correspond to a return of the function to the calling function or the main function.

[0090] Processes and methods according to the above-described examples can be implemented using computer-executable instructions that are stored or otherwise available from computer- readable media. Such instructions can include, for example, instructions and data which cause or otherwise configure a general purpose computer, special purpose computer, or a processing device to perform a certain function or group of functions. Portions of computer resources used can be accessible over a network. The computer executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, firmware, source code. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during methods according to described examples include magnetic or optical disks, flash memory, USB devices provided with non-volatile memory, networked storage devices, and so on.

[0091] Devices implementing processes and methods according to these disclosures can include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and can take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks (e.g., a computer-program product) may be stored in a computer-readable or machine- readable medium. A processor(s) may perform the necessary tasks. Typical examples of form factors include laptops, smart phones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rackmount devices, standalone devices, and so on. Functionality described herein also can be embodied in peripherals or add-in cards. SuchPATENT Qualcomm Ref. No.2402956WO functionality can also be implemented on a circuit board among different chips or different processes executing in a single device, by way of further example.

[0092] The instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functions described in the disclosure.

[0093] In the foregoing description, aspects of the application are described with reference to specific aspects thereof, but those skilled in the art will recognize that the application is not limited thereto. Thus, while illustrative aspects of the application have been described in detail herein, it is to be understood that the inventive concepts may be otherwise variously embodied and employed, and that the appended claims are intended to be construed to include such variations, except as limited by the prior art. Various features and aspects of the above-described application may be used individually or jointly. Further, aspects can be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. The specification and drawings are, accordingly, to be regarded as illustrative rather than restrictive. For the purposes of illustration, methods were described in a particular order. It should be appreciated that in alternate aspects, the methods may be performed in a different order than that described.

[0094] One of ordinary skill will appreciate that the less than (“<”) and greater than (“>”) symbols or terminology used herein can be replaced with less than or equal to (“^”) and greater than or equal to (“^”) symbols, respectively, without departing from the scope of this description.

[0095] Where components are described as being “configured to” perform certain operations, such configuration can be accomplished, for example, by designing electronic circuits or other hardware to perform the operation, by programming programmable electronic circuits (e.g., microprocessors, or other suitable electronic circuits) to perform the operation, or any combination thereof.

[0096] The phrase “coupled to” refers to any component that is physically connected to another component either directly or indirectly, and / or any component that is in communication with another component (e.g., connected to the other component over a wired or wireless connection, and / or other suitable communication interface) either directly or indirectly.PATENT Qualcomm Ref. No.2402956WO

[0097] Claim language or other language reciting “at least one of” a set and / or “one or more” of a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, claim language reciting “at least one of A and B” or “at least one of A or B” means A, B, or A and B. In another example, claim language reciting “at least one of A, B, and C” or “at least one of A, B, or C” means A, B, C, or A and B, or A and C, or B and C, A and B and C, or any duplicate information or data (e.g., A and A, B and B, C and C, A and A and B, and so on), or any other ordering, duplication, or combination of A, B, and C. The language “at least one of” a set and / or “one or more” of a set does not limit the set to the items listed in the set. For example, claim language reciting “at least one of A and B” or “at least one of A or B” may mean A, B, or A and B, and may additionally include items not listed in the set of A and B. The phrases “at least one” and “one or more” are used interchangeably herein.

[0098] Claim language or other language reciting “one or more processors configured to,” “one or more processors being configured to,” “one or more processors configured to,” “one or more processors being configured to,” or the like indicates that one processor or multiple processors (in any combination) can perform the associated operation(s). For example, claim language reciting “one or more processors configured to: X, Y, and Z” means a single processor can be used to perform operations X, Y, and Z; or that multiple processors are each tasked with a certain subset of operations X, Y, and Z such that together the multiple processors perform X, Y, and Z; or that a group of multiple processors work together to perform operations X, Y, and Z. In another example, claim language reciting “one or more processors configured to: X, Y, and Z” can mean that any single processor may only perform at least a subset of operations X, Y, and Z.

[0099] Where reference is made to one or more elements performing functions (e.g., steps of a method), one element may perform all functions, or more than one element may collectively perform the functions. When more than one element collectively performs the functions, each function need not be performed by each of those elements (e.g., different functions may be performed by different elements) and / or each function need not be performed in whole by only one element (e.g., different elements may perform different sub-functions of a function). Similarly, where reference is made to one or more elements configured to cause another element (e.g., an apparatus) to perform functions, one element may be configured to cause the other element toPATENT Qualcomm Ref. No.2402956WO perform all functions, or more than one element may collectively be configured to cause the other element to perform the functions.

[0100] Where reference is made to an entity (e.g., any entity or device described herein) performing functions or being configured to perform functions (e.g., steps of a method), the entity may be configured to cause one or more elements (individually or collectively) to perform the functions. The one or more components of the entity may include one or more memories, one or more processors, at least one communication interface, another component configured to perform one or more (or all) of the functions, and / or any combination thereof. Where reference to the entity performing functions, the entity may be configured to cause one component to perform all functions, or to cause more than one component to collectively perform the functions. When the entity is configured to cause more than one component to collectively perform the functions, each function need not be performed by each of those components (e.g., different functions may be performed by different components) and / or each function need not be performed in whole by only one component (e.g., different components may perform different sub-functions of a function).

[0101] The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the examples disclosed herein may be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate the interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0102] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices such as general purposes computers, wireless communication device handsets, or integrated circuit devices having multiple uses including application in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, then the techniques may be realizedPATENT Qualcomm Ref. No.2402956WO at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, performs one or more of the methods, algorithms, and / or operations described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise memory or data storage media, such as random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non- volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), FLASH memory, magnetic or optical data storage media, and the like. The techniques additionally, or alternatively, may be realized at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as propagated signals or waves.

[0103] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, an application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general purpose processor may be a microprocessor; but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Accordingly, the term “processor,” as used herein may refer to any of the foregoing structure, any combination of the foregoing structure, or any other structure or apparatus suitable for implementation of the techniques described herein.

[0104] Illustrative aspects of the present disclosure include:

[0105] Aspect 1. An apparatus to train a neural network, the apparatus comprising: one or more memories configured to store a plurality of patches associated with a video; and one or more processors coupled to the one or more memories and configured to: project, via a projection network, the plurality of patches to generate features representing the plurality of patches; generate a first similarity score based on a projection of the features representing the plurality of patchesPATENT Qualcomm Ref. No.2402956WO onto features of a prototype; generate a second similarity score based on a projection of predicted masked target projection tokens output from a decoder onto the features of the prototype; predict, via a clustering engine, a first cluster assignment based on the second similarity score; predict, via the clustering engine, a second cluster assignment based on the first similarity score; and determine, via the clustering engine, a loss based on the first cluster assignment and the second cluster assignment.

[0106] Aspect 2. The apparatus of Aspect 1, wherein patches in the plurality of patches are each masked.

[0107] Aspect 3. The apparatus of any of Aspects 1 or 2, wherein the plurality of patches is in a pixel space and the features representing the plurality of patches are in a feature space.

[0108] Aspect 4. The apparatus of any of Aspects 1 to 3, wherein the one or more processors coupled to the one or more memories are configured to: predict the first cluster assignment and predict the second cluster assignment in a symmetric manner without a chronological order.

[0109] Aspect 5. The apparatus of any of Aspects 1 to 4, wherein the prototype comprises a learnable prototype.

[0110] Aspect 6. The apparatus of any of Aspects 1 to 5, wherein the clustering engine applies a Sinkhorn-Knopp algorithm.

[0111] Aspect 7. The apparatus of any of Aspects 1 to 6, wherein the projection network comprises a multilayer perceptron.

[0112] Aspect 8. The apparatus of any of Aspects 1 to 7, wherein the first cluster assignment and the second cluster assignment relate to common patterns amongst the plurality of patches.

[0113] Aspect 9. The apparatus of any of Aspects 1 to 8, wherein the one or more processors coupled to the one or more memories are configured to: optimize, based on the loss, first weights associated with the projection network and second weights associated with the decoder.

[0114] Aspect 10. The apparatus of any of Aspects 1 to 9, further comprising one or more cameras configured to capture the video.

[0115] Aspect 11. A method to train a neural network, the method comprising: projecting, via a projection network, a plurality of patches associated with a video to generate features representing the plurality of patches; generating a first similarity score based on a projection of the featuresPATENT Qualcomm Ref. No.2402956WO representing the plurality of patches onto features of a prototype; generating a second similarity score based on a projection of predicted masked target projection tokens output from a decoder onto the features of the prototype; predicting, via a clustering engine, a first cluster assignment based on the second similarity score; predicting, via the clustering engine, a second cluster assignment based on the first similarity score; and determining, via the clustering engine, a loss based on the first cluster assignment and the second cluster assignment.

[0116] Aspect 12. The method of Aspect 10, wherein patches in the plurality of patches are each masked.

[0117] Aspect 13. The method of any of Aspects 11 or 12, wherein the plurality of patches is in a pixel space and the features representing the plurality of patches are in a feature space.

[0118] Aspect 14. The method of any of Aspects 11 to 13, further comprising: predicting the first cluster assignment and predict the second cluster assignment in a symmetric manner without a chronological order.

[0119] Aspect 15. The method of any of Aspects 11 to 14, wherein the prototype comprises a learnable prototype.

[0120] Aspect 16. The method of any of Aspects 11 to 15, wherein the clustering engine applies a Sinkhorn-Knopp algorithm.

[0121] Aspect 17. The method of any of Aspects 11 to 16, wherein the projection network comprises a multilayer perceptron.

[0122] Aspect 18. The method of any of Aspects 11 to 17, wherein the first cluster assignment and the second cluster assignment relate to common patterns amongst the plurality of patches.

[0123] Aspect 19. The method of any of Aspects 11 to 18, further comprising: optimizing, based on the loss, first weights associated with the projection network and second weights associated with the decoder.

[0124] Aspect 20. An apparatus to train a neural network, the apparatus comprising: means for projecting, via a projection network, a plurality of patches associated with a video to generate features representing the plurality of patches; means for generating a first similarity score based on a projection of the features representing the plurality of patches onto features of a prototype; means for generating a second similarity score based on a projection of predicted masked targetPATENT Qualcomm Ref. No.2402956WO projection tokens output from a decoder onto the features of the prototype; means for predicting, via a clustering engine, a first cluster assignment based on the second similarity score; means for predicting, via the clustering engine, a second cluster assignment based on the first similarity score; and means for determining, via the clustering engine, a loss based on the first cluster assignment and the second cluster assignment.

[0125] Aspect 21. The apparatus of Aspect 20, further comprising one or more means for performing operations according to any of Aspects 12 to 19.

[0126] Aspect 22. A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to: project, via a projection network, a plurality of patches associated with a video to generate features representing the plurality of patches; generate a first similarity score based on a projection of the features representing the plurality of patches onto features of a prototype; generate a second similarity score based on a projection of predicted masked target projection tokens output from a decoder onto the features of the prototype; predict, via a clustering engine, a first cluster assignment based on the second similarity score; predict, via the clustering engine, a second cluster assignment based on the first similarity score; and determine, via the clustering engine, a loss based on the first cluster assignment and the second cluster assignment.

[0127] Aspect 23. The non-transitory computer-readable medium of Aspect 22, further instructions that, when executed by the one or more processors, cause the one or more processors to perform operations according to any of Aspects 12 to 19.

Claims

PATENT Qualcomm Ref. No.2402956WO CLAIMS WHAT IS CLAIMED IS:

1. An apparatus to process video data, the apparatus comprising: one or more memories configured to store a plurality of patches associated with a video; and one or more processors coupled to the one or more memories and configured to: project, via a projection network, the plurality of patches to generate features representing the plurality of patches; generate a first similarity score based on a projection of the features representing the plurality of patches onto features of a prototype; generate a second similarity score based on a projection of predicted masked target projection tokens output from a decoder onto the features of the prototype; predict, via a clustering engine, a first cluster assignment based on the second similarity score; predict, via the clustering engine, a second cluster assignment based on the first similarity score; and determine, via the clustering engine, a loss based on the first cluster assignment and the second cluster assignment.

2. The apparatus of claim 1, wherein patches in the plurality of patches are each masked.

3. The apparatus of claim 1, wherein the plurality of patches is in a pixel space and the features representing the plurality of patches are in a feature space.

4. The apparatus of claim 1, wherein the one or more processors coupled to the one or more memories are configured to: predict the first cluster assignment and predict the second cluster assignment in a symmetric manner without a chronological order.

5. The apparatus of claim 1, wherein the prototype comprises a learnable prototype.PATENT Qualcomm Ref. No.2402956WO 6. The apparatus of claim 1, wherein the clustering engine applies a Sinkhorn-Knopp algorithm.

7. The apparatus of claim 1, wherein the projection network comprises a multilayer perceptron.

8. The apparatus of claim 1, wherein the first cluster assignment and the second cluster assignment relate to common patterns amongst the plurality of patches.

9. The apparatus of claim 1, wherein the one or more processors coupled to the one or more memories are configured to: optimize, based on the loss, first weights associated with the projection network and second weights associated with the decoder.

10. The apparatus of claim 1, further comprising one or more cameras configured to capture the video.

11. A method to train a neural network, the method comprising: projecting, via a projection network, a plurality of patches associated with a video to generate features representing the plurality of patches; generating a first similarity score based on a projection of the features representing the plurality of patches onto features of a prototype; generating a second similarity score based on a projection of predicted masked target projection tokens output from a decoder onto the features of the prototype; predicting, via a clustering engine, a first cluster assignment based on the second similarity score; predicting, via the clustering engine, a second cluster assignment based on the first similarity score; and determining, via the clustering engine, a loss based on the first cluster assignment and the second cluster assignment.

12. The method of claim 11, wherein patches in the plurality of patches are each masked.PATENT Qualcomm Ref. No.2402956WO 13. The method of claim 11, wherein the plurality of patches is in a pixel space and the features representing the plurality of patches are in a feature space.

14. The method of claim 11, further comprising: predicting the first cluster assignment and predict the second cluster assignment in a symmetric manner without a chronological order.

15. The method of claim 11, wherein the clustering engine applies a Sinkhorn-Knopp algorithm.

16. The method of claim 11, wherein the projection network comprises a multilayer perceptron.

17. The method of claim 11, wherein the first cluster assignment and the second cluster assignment relate to common patterns amongst the plurality of patches.

18. The method of claim 11, further comprising: optimizing, based on the loss, first weights associated with the projection network and second weights associated with the decoder.

19. A non-transitory computer-readable medium having stored thereof instructions which, when executed by one or more processors, cause the one or more processors to train a neural network by being configured to: project, via a projection network, a plurality of patches associated with a video to generate features representing the plurality of patches; generate a first similarity score based on a projection of the features representing the plurality of patches onto features of a prototype; generate a second similarity score based on a projection of predicted masked target projection tokens output from a decoder onto the features of the prototype;PATENT Qualcomm Ref. No.2402956WO predict, via a clustering engine, a first cluster assignment based on the second similarity score; predict, via the clustering engine, a second cluster assignment based on the first similarity score; and determine, via the clustering engine, a loss based on the first cluster assignment and the second cluster assignment.

20. The non-transitory computer-readable medium of claim 19, wherein patches in the plurality of patches are each masked.

Images