Hybrid video models with attention and recurrence

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing videos using neural networks. In particular, the neural network has a hybrid architecture that includes both recurrent and self-attention layers.

Description

[0001] ATTORNEY DOCKET NO. 45288-0537WO1

[0002] HYBRID VIDEO MODELS WITH ATTENTION AND RECURRENCE

[0003] CROSS-REFERENCE TO RELATED APPLICATION

[0004] This application claims priority to U.S. Provisional Application No. 63 / 702,147, filed on October 1, 2024. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application.

[0005] BACKGROUND

[0006] This specification relates to processing videos using machine learning models.

[0007] As one example, neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to another layer in the network, e.g., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of w eights.

[0008] SUMMARY

[0009] This specification describes a system implemented as computer programs on one or more computers in one or more locations that processes video sequences using a “hybrid” neural network implementing both attention and recurrence to perform one or more machine learning tasks on the video sequences.

[0010] Particularly, this specification introduces a hybrid neural network architecture that interleaves temporal (e.g., gated) recurrent blocks with spatial attention blocks, resulting in a flexible model that can address sparse and dense tasks, trained in either supervised or selfsupervised regimes. The hybrid neural network can obtain state-of-the-art or competitive results on multiple machine learning tasks, while using significantly less memory and compute cycles than conventional video Transformer models, e.g., Video Vision Transformers (ViViTs).

[0011] According to a first aspect a method performed by one or more computers is provided, the method comprising: receiving a video sequence comprising a respective video frame at each of a plurality of time steps; and processing the video sequence to generate a respective predicted output for each time step in a subset of the plurality of time steps, comprising, for each time step: receiving the video frame at the time step; processing the video frame at the time step to generate an input sequence for the time step comprising a respective input embedding at each of a plurality of input positions; and processing the input sequence for the ATTORNEY DOCKET NO. 45288-0537WO1 time step, using a neural network, to generate a network output for the time step; wherein the neural network comprises: a recurrent block comprising a recurrent layer configured to, for each time step and each input position of the input sequence for the time step: receive a layer input comprising: (i) a recurrent state for a preceding time step and the input position, and (ii) a layer input embedding for the input position; and process the layer input to generate a layer output comprising: (i) a recurrent state for the time step and the input position, and (ii) a layer output embedding for the input position; an attention block comprising an attention layer configured to, for each time step: receive a layer input sequence comprising a respective layer input embedding for each input position of the input sequence for the time step; and apply an attention mechanism over the layer input sequence to generate a layer output sequence comprising a respective layer output embedding for each input position of the input sequence for the time step; and an output prediction head configured to, for each time step in the subset: receive the network output for the time step; and process the network output to generate the predicted output for the time step.

[0012] In some implementations of the method, the subset of time steps is a proper subset.

[0013] In some implementations of the method, for each time step, the network output for the time step comprises a respective output embedding at each of one or more input positions of the input sequence for the time step.

[0014] In some implementations of the method, the method further comprises, for each time step, adding a respective input embedding of each of one or more auxiliary tokens to the input sequence for the time step, and for each time step, the network output for the time step comprises a respective output embedding of each auxiliary token.

[0015] In some implementations of the method, the attention layer is a global attention layer, and for each time step, the global attention layer applies a global attention mechanism that, for each input position of the input sequence for the time step, attends over all of the input positions preceding or equal to the input position.

[0016] In some implementations of the method, the global attention mechanism is a dense attention mechanism.

[0017] In some implementations of the method, the attention layer is a local attention layer, and for each time step, the local attention layer applies a local attention mechanism that, for each input position of the input sequence for the time step, attends only over a local subset of the input positions that are within a local window of the input position.

[0018] In some implementations of the method, for each time step, the layer input and output embeddings of the attention layer each have a plurality of dimensions, and the attention ATTORNEY DOCKET NO. 45288-0537WO1 mechanism applies positional encoding to each of the plurality of dimensions of the layer input and output embeddings.

[0019] In some implementations of the method, the positional encoding is a relative positional encoding or a Rotan ■ Positional Embedding (RoPE).

[0020] In some implementations of the method, the recurrent layer is a linear recurrent layer, and for each time step and each input position of the input sequence for the time step, the recurrent state for the time step and the input position is linear in the recurrent state for the preceding time step and the input position.

[0021] In some implementations of the method, the recurrent block further comprises a onedimensional convolutional layer immediately preceding the recurrent layer, and the onedimensional convolutional layer is configured to, for each time step: receive a layer input sequence comprising a respective layer input embedding for each input position of the input sequence for the time step; and apply a convolution operation over the layer input sequence to generate a layer output sequence comprising a respective layer output embedding for each input position of the input sequence for the time step.

[0022] In some implementations of the method, the recurrent block further comprises a linear layer immediately preceding the one-dimensional convolutional layer, and the linear layer is configured to, for each time step: receive a layer input sequence comprising a respective layer input embedding for each input position of the input sequence for the time step; and apply a linear transformation over the layer input sequence to generate a layer output sequence comprising a respective layer output embedding for each input position of the input sequence for the time step.

[0023] In some implementations of the method, the recurrent block is a gated recurrent block comprising: a first channel comprising a feedforward layer configured to, for each time step: receive a layer input sequence comprising a respective layer input embedding for each input position of the input sequence at the time step: and apply an activation function over the layer input sequence to generate a layer output sequence comprising a respective layer output embedding for each input position of the input sequence for the time step; a second, parallel channel comprising: (i) the linear layer, (ii) the one-dimensional convolutional layer, and (iii) the recunent layer; and a multiplicative gate proceeding the first and second channels.

[0024] In some implementations of the method, the feedforward layer is a Rectified Linear Unit (ReLU) or a Gaussian error Linear Unit (GeLU).

[0025] In some implementations of the method, the first channel of the gated recurrent block further compnses a linear layer immediately preceding the feedforward layer. ATTORNEY DOCKET NO. 45288-0537WO1

[0026] In some implementations of the method, for each time step and each input position of the input sequence for the time step, the recunent state for the time step and the input position is the layer output embedding for the input position.

[0027] In some implementations of the method, the attention block further comprises a multilayer perceptron proceeding the attention layer.

[0028] In some implementations of the method, the attention block further comprises: a first skip connection between: (i) a first input preceding the attention layer, and (ii) a first additive gate proceeding the attention layer; and a second skip connection between: (i) a second input preceding the multilayer perceptron, and (ii) a second additive gate proceeding the multilayer perceptron.

[0029] In some implementations of the method, for each time step, processing the video frame at the time step to generate the input sequence for the time step comprises: dividing the video frame at the time step into a plurality of image patches for the time step; and processing the plurality7of image patches for the time step, using an encoder, to generate the input sequence for the time step.

[0030] In some implementations of the method, for each time step, the input sequence for the time step comprises a respective input embedding of each of the plurality of images patches for the time step.

[0031] According to a second aspect, a method performed by one or more computers is provided. The method includes receiving a video sequence comprising a respective video frame at each of a plurality of frame positions; and processing the video sequence to generate a video processing output for the video sequence, comprising: for each frame position: receiving the video frame at the frame position; processing the video frame at the frame position to generate an input sequence for the frame position comprising a respective input embedding at each of a plurality of input positions; and processing the input sequence for each frame position, using a neural network, to generate the video processing output for the video sequence, wherein the neural network comprises: a recurrent block comprising a recurrent layer configured to, for each frame position and each input position of the input sequence for the frame position: receive a layer input comprising: (i) a recurrent state for a preceding frame position and the input position, and (ii) a layer input embedding for the input position; and process the layer input to generate a layer output comprising: (i) a recurrent state for the frame position and the input position, and (ii) a layer output embedding for the input position; and an attention block comprising an attention layer configured to, for each frame position: receive a layer input sequence comprising a respective layer input embedding for each input position of the input ATTORNEY DOCKET NO. 45288-0537WO1 sequence for the frame position; and apply an attention mechanism over the layer input sequence to generate a layer output sequence comprising a respective layer output embedding for each input position of the input sequence for the frame position.

[0032] In some implementations of the method, the video processing output comprises a respective predicted output for each of one or more video frames in the video sequence.

[0033] In a third aspect, a system is provided. The system includes one or more computers; and one or more storage devices communicatively coupled to the one or more computers. The one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the method of the first or second aspects in any of their abovementioned implementations.

[0034] In a fourth aspect, one or more non-transitory storage media are provided. The one or more non-transitory storage media store instructions that, when executed by one or more computers, cause the one or more computers to perform the method of the first or second aspects in any of their abovementioned implementations.

[0035] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0036] Video processing and understanding is a challenging task as it involves low-level scene understanding (e.g., how7objects move) and high-level reasoning (e.g., causal relations between events) over a signal that is high-dimensional, can be noisy, and contains high correlations and redundancies in both spatial and temporal dimensions. Efficient video modelling requires high- capacity models that can represent the sheer diversity and richness of real-world videos, while having a reasonable speed and memory footprint at both training and inference time. Convolutional neural networks have been a successful family of models for video, but their scaling capabilities (in both data and parameters) are limited due to their inductive biases (e.g., locality, invariance). Recurrent neural networks have some desirable properties for video modelling (e.g., constant inference time per video frame independent of the length of the video), but they are slow to train due to their sequential nature and have difficulties learning over long, complex sequences. Transformers have emerged as a powerful family of models for all modalities, with impressive scaling capabilities and parallelization options for faster training. However, they have significant memory footprint due to the quadratic complexify of the self-attention operation, making them suitable only for short video modelling at modest resolution.

[0037] This specification introduces a hybrid neural network architecture for video processing and understanding that significantly improves on existing architecture. Particularly, the hybrid ATTORNEY DOCKET NO. 45288-0537WO1 neural network described herein alternates temporal (e.g., gated) recurrent blocks with spatial attention blocks to ensure an efficient flow of information, while allowing the model to discover and learn long-range dependencies in both space and time. The hybrid neural network uses spatial image patches, but the temporal extent of the image patches is not hard-wired like in existing video Transformer models. Instead, the recurrent blocks can learn in a continuous fashion what information to keep or drop from the image patch embeddings received at each time step. The parameters of the recurrent blocks can be shared over space, preventing the number of parameters from exploding as the resolution of the video sequences increases.

[0038] In some implementations of the hybrid neural network, the recurrent blocks include linear recurrent layers. Due to the linear state equation in a linear recurrent layer, the computation can be parallelized at training time using linear scan operations and the learning dynamics are simpler but still stable. The resulting hybrid neural network is highly flexible and can address various video understanding tasks, both sparse (e.g., video classification) and dense (e.g., point tracking), trained in a supervised or self-supervised manner, e.g., using masked auto-encoding.

[0039] As an example of the performance of the described techniques, a number of experiments were performed for assessing the performance of the hybrid neural network. In the experiments, a causal setup was used that respects the arrow of time, so the model was suitable for any downstream applications, e.g.. robotics. In addition, the hybrid neural network was significantly more efficient in both memory footprint and FLOPs compared to video Transformer models that were used as a benchmark comparison. Despite this, the described techniques significantly outperform the video Transformer models on various benchmark video understanding tasks.

[0040] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

[0041] BRIEF DESCRIPTION OF THE DRAWINGS

[0042] FIG. 1 is a diagram of an example video processing system.

[0043] FIG. 2A shows an example architecture of the hybrid neural netw ork.

[0044] FIG. 2B shows an example of the operations performed by the hybrid neural netw ork.

[0045] FIG. 3 is a flow diagram of an example process for processing a video sequence.

[0046] FIG. 4 shows an example architecture of the recurrent block. ATTORNEY DOCKET NO. 45288-0537WO1

[0047] Like reference numbers and designations in the various drawings indicate like elements.

[0048] DETAILED DESCRIPTION

[0049] FIG. 1 is a diagram of an example video processing system 100. The video processing system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

[0050] The system 100 processes video sequences 102 using a "hybrid" neural network 110 implementing both attention and recurrence to perform one or more machine learning tasks on the video sequences 102.

[0051] More specifically, the hybrid neural network 102 has a hybrid architecture that interleaves temporal (e.g., gated) recurrent blocks 120 with spatial attention blocks 130, resulting in a flexible model that can address sparse and dense tasks, and that can be trained in either supervised or self-supervised regimes. The hybrid neural network 110 can obtain state-of-the-art or competitive results on multiple machine learning tasks, while using significantly less memory and compute cycles than conventional video Transformer models, e.g., Video Vision Transformers (ViViTs).

[0052] The hybrid neural network 110 can be configured, e.g., through training, to perform any kind of machine learning task on a video sequence 102. That is, the hybrid neural network 1 10 can be configured to receive any kind of video sequence 102 and process the video sequence 102 to generate any kind of video processing output 112 based on the video sequence 102.

[0053] For example, the video processing output 112 can include a predicted output for the (entire) video sequence 102, a respective predicted output for each time step in a subset of the time steps (e.g., for the video frame at the time step), or a predicted output for the subset of time steps (e.g., for a segment of the video sequence). In some cases, the hybrid neural network 110 generates a predicted output for each video frame in the video sequence, e.g., in a streaming fashion.

[0054] A predicted output can be any of multiple types of predicted outputs relating to any of multiple types of machine learning tasks. The predicted output can include a score, a classification output, a regression output, or a text output. Examples of predicted outputs include: (i) an image classification output classifying a video or a video frame into one of multiple object categories, (ii) an object detection output specifying coordinates of one or ATTORNEY DOCKET NO. 45288-0537WO1 more bounding boxes in a video frame that are predicted to encompass objects, (iii) a segmentation output classifying each pixel in a video frame into one of multiple categories, (iv) an object tracking output specifying a trajectory of one or more objects that are predicted to be in movement across multiple video frames, (v) an action detection output specifying a start or end frame (or start or end time step) of each of one or more actions, e.g., including an action class label for the action, (vi) a pose estimation output specifying keypoints of one or more joint positions of a person in the video frame, (vii) a sequence of generated video frames representing a continuation of the video sequence, (viii) a text output describing a video or a video frame, (ix) an anomaly detection output specifying a score for a level of anomaly in a video frame, (x) a flow prediction output representing a direction and magnitude of pixel movement in a video frame, or (xi) a depth estimation output including a depth value for each pixel in a video frame.

[0055] In general, the video sequence 102 includes a respective video frame 104 at each of multiple time steps.

[0056] The system divides the video frame 104 at each time step into multiple image patches and then encodes the image patches into an input sequence 106 including a respective input embedding of each image patch of the video frame at the time step.

[0057] Following this, the system 100 processes the input sequence 106 at each time step, using the hybrid neural network 110, to generate a network output 114 including one or more output embeddings at the time step.

[0058] For example, the input sequence 106 can also include an auxiliary' token and the output embeddings can include an output embedding of the auxiliary (e.g., class) token. As another example, the output embeddings can include respective output embeddings for each of the input positions in the input sequence 106.

[0059] In general, the hybrid neural network 110 includes a recurrent block 120, an attention block 130, and an output prediction head 140.

[0060] The recurrent block 120 applies a recurrence operation to the input sequences temporally via a recurrent layer, e.g., over input sequences at different time steps. In other words, the recurrent block 120 applies a recurrence operation over a respective “temporal tube” for each input position in the input sequence 106 that includes only embeddings at the input position in the input sequence 106 and not any embeddings at any other input positions.

[0061] The attention block 130 applies a self-attention operation spatially to the input sequences via an attention layer, e.g., the attention layer applies self-attention over different input positions of an input sequence within a given time step. ATTORNEY DOCKET NO. 45288-0537WO1

[0062] The output prediction head 140 can then provide a predicted output 116 for a video frame at a time step by processing the network output 114 at the time step.

[0063] Unlike conventional attention or recurrent neural networks that each utilize only attention or recurrence operations, the neural network 1 10 described in this specification utilizes both attention and recurrence. Hence, the neural network 110 may be referred to as a “hybrid neural network’' 110.

[0064] A hybrid neural network 110 that uses a recurrent block 120 for temporal processing and an attention block 130 for spatially processing can significantly speed up inference times and reduce memory costs compared to conventional attention-based video processing neural networks, e.g., Video Transformers, while simultaneously having equal or better inference accuracy (and training loss) for the same model size. For example, in some implementations, the hybrid neural network 110 described herein can be employed for streaming applications that involve generating predictions at the frame rate of a given video sequence, e.g., a live video sequence captured by the camera of a robot. In general, this increased speed and accuracy, particularly for long video sequences in a causal setup, is due to leveraging recurrence over time where self-attention would be too computationally expensive to model. Particularly, the standard self-attention mechanism scales quadratically O(T2) with respect to the video sequence length T, measuring the number of video frames (or time steps) in the video sequence. In contrast, recurrence scales linearly 0(7") with the video sequence length, therefore providing a more computationally efficient model for learning on long video sequences.

[0065] Moreover, in some implementations, the recurrent block 120 includes a linear recurrent layer for implementing the recurrence operation. In this case, the recurrence relation is linear in the recurrent state which allows the linear recurrent layer to be parallelized during training, e g., using parallel scans. Linear recurrence can also mitigate (or eliminate) the exploding (and vanishing) gradient problem in conventional recurrent neural networks, which has limited their use in modem applications involving long-range sequence processing, e.g., natural language processing (NLP). Particularly, the hybrid neural network 110 described in this specification can be scaled to sizes typical of large language models (LLMs), e.g., having 14 billion or more parameters, and can efficiently process and understand long-range video sequences including 2048 or more video frames, e.g., 4096 or more video frames, 8192 or more video frames, 16384 or more video frames, 32768 or video frames, 65536 or more video frames. ATTORNEY DOCKET NO. 45288-0537WO1

[0066] As described above, the attention block 130 includes an attention layer. The attention layer is configured to, for each time step, receive a layer input sequence including a respective layer input embedding for each input position of the input sequence 106 for the time step and apply an attention mechanism over the layer input sequence to generate a layer output sequence including a respective layer output embedding for each input position of the input sequence for the time step. In general, the layer output embeddings are updated versions of the layer input embeddings. These are subsequently forwarded, as input, to the next layer block in the sequence to be updated further.

[0067] The attention layer can be a global attention layer or a local attention layer.

[0068] A global attention layer applies a global attention mechanism that, for each input position, attends over all of the input positions in the sequence. The global attention mechanisms applied by the global attention layer can be dense attention mechanisms or sparse attention mechanisms.

[0069] A local attention layer, on the other hand, applies a local attention mechanism that, for each input position, attends only over a local subset of input positions that are within a local window of the input position. That is, unlike the global attention mechanisms, the local attention mechanism does not attend to any position that is outside of the local window of the input position. The local window contains a fixed number of input positions. The fixed number of input positions is generally much smaller than the total number of input positions in the input sequence and is referred to as the size of the local window.

[0070] The attention layer can also use an attention mechanism that applies a positional encoding to each of the input positions. “Positional encoding” refers to modifying the operations applied by the attention layer for a given input position based on the absolute or relative position of the input position within the input sequence. For example, the positional encoding can be a learned positional encoding for each position, a Rotary Positional Embedding (RoPE) or a different type of positional encoding, e.g., a relative positional encoding or an Attention w ith Linear Biases (ALiBi) positional encoding.

[0071] Generally, to apply the self-attention operation, an attention mechanism uses one or more attention heads. Each attention head generates a set of queries, a set of keys, and a set of values, and then applies any of a variety of variants of query-key-value (QKV) attention, e g., a dot product attention function or a scaled dot product attention function, using the queries, keys, and values to generate an output. Each query, key, value can be a vector that includes one or more vector elements. ATTORNEY DOCKET NO. 45288-0537WO1

[0072] When there are multiple attention heads, the attention block then combines the outputs of the multiple attention heads, e.g., by concatenating the outputs and, optionally, processing the concatenated outputs through a linear layer.

[0073] In some cases, the attention mechanism uses multi-query attention, where each attention head shares a common set of keys and values but does not share queries.

[0074] For local attention mechanisms, for each position, the positions that are used to generate the queries, keys, and values for the position are defined by the local window size for the local attention mechanism, i.e., non-zero attention weights for a given position are computed only for positions that are within the local window of the given position.

[0075] As described above, the recurrent block 120 includes a recurrent layer. The recurrent layer is configured to, for each time step and each input position of the input sequence for the time step: receive a layer input including: (i) a recurrent state for a preceding time step and the input position, and (ii) a layer input embedding for the input position. The recurrent layer then processes the layer input to generate a layer output including: (i) a recurrent state for the time step and the input position, and (ii) a layer output embedding for the input position.

[0076] In some implementations, the recurrent block can be a gated recurrent block that includes: (i) a first channel, (ii) a second, parallel channel, and (iii) a multiplicative gate proceeding the first and second channels. The multiplicative gate can combine the outputs of the first and second channels by performing an elementwise multiplication operation between their respective outputs. The first channel can include a linear layer followed by a feedforward layer configured to: receive a layer input sequence; and apply an activation function over the layer input sequence to generate a layer output sequence. For example, the feedforward layer can be a Rectified Linear Unit (ReLU) or a Gaussian error Linear Unit (GeLU). The second channel can include (i) a linear layer, (ii) a one-dimensional convolutional layer immediately proceeding the linear layer, and (iii) the recurrent layer immediately proceeding the one-dimensional convolutional layer.

[0077] An example of this gated architecture is described below with reference to FIG. 4.

[0078] While FIG. 1 shows only a single recurrent block 120 and a single attention block 130 for ease of illustration, generally the hybrid neural network 110 will include multiple recurrent blocks 120 and multiple attention blocks 130, e.g., interleaved with one another. An example of this is described below with reference to FIG. 2A.

[0079] FIG. 2A shows an example 200 of the architecture of the hybrid neural network 110. In the example 2 A. the hybrid neural network 110 includes a sequence of combined blocks 210, each of which includes a recurrent block 120 and an attention block 130. More ATTORNEY DOCKET NO. 45288-0537WO1 specifically, in the example 200, each combined block 210 includes a recurrent block 120 followed bv an attention block 130.

[0080] As shown in FIG. 2 A, in the example 200, the recurrent block 120 is implemented as a gated linear recurrent unit (GLRU), i.e., a gated recurrent block that has a linear recurrent layer (also referred to as a linear recurrent unit (LRU)).

[0081] The recurrent block 120 performs temporal mixing of the embeddings in the input sequence, i.e., transfonns the input to the block 120 for each input position conditioned on inputs at the same input position in preceding video frames and not conditioned on any of the other inputs at other input positions within the same video frame.

[0082] The attention block 130 includes an attention layer 220 that performs self-attention across the inputs to the attention layer 220 at the input positions and multi-layer perceptron (MLP) 230 that processes each input to the MLP 230 at each input position independently. Thus, the attention layer 220 performs spatial mixing to modify each input at each input positions conditioned on the other inputs at the other input positions while the MLP 230 performs channel mixing by updating the “‘channels,” i.e.. dimensions, of each input independently.

[0083] The recurrent block 120, the attention block 130, or both can also include additional operations other than those shown in FIG. 2A. For example, the attention block 130 can also include one or both of: a) a first skip connection between: (i) a first input preceding the attention layer, and (ii) a first additive gate proceeding the attention layer; or b) a second skip connection between: (i) a second input preceding the multilayer perceptron, and (ii) a second additive gate proceeding the multilayer perceptron. That is, the first skip connection applies an additive gate, i.e., adds, the input to the attention layer and the output of the attention layer while the second skip connection applies an additive gate, i.e., adds, the input to the MLP and the output of the MLP. Optionally, the recurrent block 120 can also include a similar skip connection between (i) a first input preceding the recurrent block, and (ii) a first additive gate proceeding the attention block.

[0084] The operations performed by the recurrent and attention blocks 120 and 130 will be described in more detail below.

[0085] FIG. 2B shows an example 250 of the operation of the hybrid neural network 1 10. In the example 250, the hybrid neural network 110 includes A “combined” blocks that each include a recurrent block 120 and an attention block 130, i.e., N interleaved sets of recurrent block 120 and attention blocks 130. That is, the hybrid neural network 110 includes N ATTORNEY DOCKET NO. 45288-0537WO1 instances of a recurrent block 120, each followed by a respective instance of an attention block 130.

[0086] In the example 250, the hybrid neural network 110 is processing a current video frame xtfrom a sequence of video frames that each have a respective time index t.

[0087] As shown in FIG. 2B, the system divides the current video frame into patches, e.g., by partitioning the video frame into non-overlapping patches of a fixed size. In the example of FIG. 2B, each video frame is divided into nine patches.

[0088] The system then encodes the image patches into an input sequence that includes a respective input embedding of each image patch of the video frame at the time step, e.g., by applying a linear projection to each of the image patches. The linear projection can be a learned projection that is learned during the training of the hybrid neural network 110.

[0089] In the example of FIG. 2B, the input sequence also includes a class embedding that does not correspond to any of the image patches. For example, the class embedding can be a fixed embedding or an embedding that is learned during the training of the hybrid neural network 110.

[0090] In other examples, the input sequence can include more than one auxiliary embedding or can include only the input embeddings for the image patches.

[0091] As described above, in some implementations, the system also applies a positional embedding to the outputs of the linear proj ection as part of the encoding.

[0092] The hybrid neural network 110 then processes the input sequence through the N combined blocks. Within a given combined block, the recurrent block 120 updates the embedding at each of the input positions independently from each other, conditioned on embeddings at the same input position in earlier frames in the video sequence. The attention block 130 then updates each of the embeddings, i.e., generated as output by the recurrent block 120, conditioned on the other embeddings for the other input positions in the sequence.

[0093] After the input sequence has been processed through the N combined blocks, the hybrid neural network 110 can generate a predicted output j’ffor the current video frame xtfrom the output embeddings generated as output by the last combined block, e.g., by processing a network output that is generated from one or more of the output embeddings using an output prediction head.

[0094] In the example of FIG. 2B, the hybrid neural network processes the output embedding for the class token using the output prediction head to generate the predicted output.

[0095] In other examples, the hybrid neural network can process, e.g., a pooled embedding generated by pooling the output embeddings for the positions in the input sequence or can ATTORNEY DOCKET NO. 45288-0537WO1 process each output embedding for each position in the input sequence, using the output prediction head to generate the predicted output.

[0096] FIG. 3 is a flow diagram of an example process 300 for processing a video sequence using the hybrid neural network. For convenience, the process 300 will be described as being perfonned by a system of one or more computers located in one or more locations. For example, a video processing system, e.g., the video processing system 100 depicted in FIG. 1, appropriately programmed in accordance with this specification, can perform the process 300.

[0097] The system receives a video sequence that has a respective video frame at each of a plurality of time steps (step 302).

[0098] The system processes the video sequence to generate a respective predicted output for each time step in a subset of the plurality of time steps (step 304). For example, the system can generate a respective predicted output for each video frame in the sequence or the subset can be a proper subset, so that the system generates a predicted output for less than all of the video frames in the sequence, e.g., only for the last video frame, only for every "-th video frame, and so on.

[0099] Generally, as part of the processing the system performs steps 306-310 for each time step in the video sequence.

[0100] The system receives the video frame at the time step (step 306).

[0101] The system processes the video frame at the time step to generate an input sequence for the time step (step 308).

[0102] The input sequence includes a respective input embedding at each of a plurality of input positions.

[0103] For example, to generate the input sequence, the system can divide the video frame at the time step into a plurality of image patches, e.g., by dividing the video frame into multiple non-overlapping regions that each have the same size or by using another appropriate image partitioning scheme.

[0104] The system can then process the plurality of image patches for the time step, using an encoder, to generate the input sequence for the time step. For example, the encoder can be a linear projection or a multi-layer perceptron or other appropriate learned transformation that maps each image patch, i.e., the intensity values of the pixels of the image patch, to a respective input embedding. Thus, the input sequence includes a respective input embedding for each of the patches of the video frame. The encoder can be learned jointly with the ATTORNEY DOCKET NO. 45288-0537WO1 training of the hybrid neural network or can be a pre-trained encoder, e.g., that has been trained jointly with another image processing neural network.

[0105] In some implementations, the input sequence also includes one or more auxiliary embeddings. That is, the system can add a respective input embedding of each of one or more auxiliary tokens to the input sequence for the time step, e.g., before or after the sequence of input embeddings that represent image patches of the video frame.

[0106] The system processes the input sequence for the time step, using the hybrid neural network, to generate a network output for the time step (step 310).

[0107] Generally, the network output includes a respective output embedding at each of one or more input positions of the input sequence for the time step. For example, the network output can include a respective output embedding for all of the input positions in the input sequence. As another example, the network output can include a respective output embedding for the auxiliary token(s) in the input sequence.

[0108] As described above, the hybrid neural network generally includes one or more recurrent blocks interleaved with one or more attention blocks. That is, the hybrid neural network includes a sequence of blocks, with one or more of the blocks being attention blocks and one or more of the blocks being recurrent blocks. A ‘'block,” as used in this specification, is a collection of one or more neural network layers.

[0109] As a particular example, the hy brid neural network can have an architecture made up of a sequence of multiple combined blocks, with each combined block including (i) a recurrent block followed by an attention block or (ii) an attention block followed by a recurrent block.

[0110] The neural network can optionally also include other types of blocks, e.g., blocks that perform feed-forward operations, blocks that perform normalization, skip connection blocks, and so on.

[0111] The recurrent block includes a recurrent layer that is configured to, for each time step and each input position of the input sequence for the time step, receive a layer input that includes: (i) a recurrent state for a preceding time step and the input position, and (ii) a layer input embedding for the input position.

[0112] The recurrent layer processes the layer input to generate a layer output that includes: (i) a recurrent state for the time step and the input position, and (ii) a layer output embedding for the input position. The layer output embedding is provided as output to the next component of the neural network, e.g., as output of the recurrent block or as input to another ATTORNEY DOCKET NO. 45288-0537WO1 component of the recurrent block. The recurrent state is maintained for use in processing the layer input embedding for the input position at the next time step.

[0113] That is, the recurrent layer block maintains a respective recurrent state for each input position in the input sequence and updates the recurrent state for each input position using the layer input embeddings for the input positions (and not any layer input embeddings from any other input positions).

[0114] As described above, in some implementations, the recurrent layer is a linear recurrent layer. In these implementations, the recurrent state for the time step and the input position is linear in the recurrent state for the preceding time step and the input position. That is, the recurrent layer generates the updated recurrent state for a given input position without applying any non-linear operations to the recurrent state for the preceding time step. In other words, each transformation applied to the recunent state for the preceding time step is a linear transformation, i.e., satisfies the properties of homogeneity and additivity.

[0115] The recurrent layer can generally have any of a variety of recurrent layer architectures. As one example, when the recurrent layer is a linear recunent layer, the recurrent layer can have a gated architecture as follows: where xtis the observation at time step ?, i.e., the input to the recunent layer for any given input position, ht-i is the recurrent state for the given input position from the preceding time step t - 1, htis the recurrent state for the given input position for the time step t, o(X) is a vector containing the eigenvalues of the recunent weight, c is a constant value, the learnable parameters X are passed through a sigmoid function o to ensure they are between 0 and 1, rtand it are the gates of the recurrent layer which are dependent only on the observations xt in order not to prevent parallelization of the computation. The input gate allows the recurrent layer to ignore certain observations, while the recurrent gate interpolates the value of the eigenvalue to be between o(X) and 1. At a high level, when the recurrent gate pushes to be 1, the recurrent layer ignores the current observation, preserving its state intact.

[0116] In the above example (and in some other example architectures), the recurrent state for the given input position for the time step t is used as the layer output embedding for the input position. That is, for each time step and each input position of the input sequence for ATTORNEY DOCKET NO. 45288-0537WO1 the time step, the recurrent state for the time step and the input position is the layer output embedding for the input position.

[0117] The layer input embedding for the input position that is received by the recurrent layer depends on the architecture of the recurrent block but generally is or is derived from (i) if the recurrent block is not the first block in the sequence of blocks, the output of the preceding block in the sequence for the input position or (ii) if the recurrent block is the first block in the sequence of blocks, the embedding in the input sequence for the input position.

[0118] As one example, the recurrent block can have a gated architecture, i.e., is a gated recurrent block. In this example, the recurrent block has multiple channels.

[0119] An example of this architecture is described below with reference to FIG. 4.

[0120] If the time step is in the subset, the system can then process the network output for the time step to generate the predicted output for the time step (step 312).

[0121] That is, the system can process the netw ork output using the output prediction head. The output prediction head is configured to, for each time step in the subset, receive the network output for the time step and process the network output to generate the predicted output for the time step.

[0122] The output prediction head can generally have any appropriate architecture. For example, the output prediction head can be a single linear or non-linear neural network layer, can be a multi-layer perceptron (MLP), or can include one or more self-attention neural network layers.

[0123] FIG. 4 shows an example 400 of the architecture of the recurrent block 120. In the example 400, the recurrent block is a gated recurrent block and the recurrent layer in the recurrent block is a linear recurrent unit (LRU) 402.

[0124] Thus, the recurrent block 120 includes a first channel 410 and a second channel 420. As shown in FIG. 4, the recurrent block can optionally include, prior to the channels 410 and 420, a normalization operation 430, e.g., RMSNorm, that applies a normalization to the embeddings for the input positions that are received as input by the recurrent block 120.

[0125] The first channel 410 includes a linear layer 440 followed by a feedforward layer 450 (also referred to as an activation function layer) that applies an activation function to the outputs of the linear layer 440. In other words, the first channel 410 includes the linear layer 440 immediately proceeding the feedforward layer 450. In the example 400, the activation function applied by the feedforward layer 450 is the Gaussian error linear unit (GeLU) activation function. More generally, however, the feedforward layer 450 can apply any appropriate type of activation, e.g., GeLU, ReLI, sigmoid, and so on. ATTORNEY DOCKET NO. 45288-0537WO1

[0126] The second channel 420 includes a linear layer 460, a one-dimensional convolutional layer 470, and the recurrent layer, i.e., the LRU 402.

[0127] The linear layer 460 applies a learned linear transformation to each input to the linear layer.

[0128] In other words, the linear layer 460 receives a layer input sequence that includes a respective layer input embedding for each input position of the input sequence for the time step and applies a linear transformation over the layer input sequence to generate a layer output sequence that includes a respective layer output embedding for each input position of the input sequence for the time step. Because the linear transformation is applied to each layer input embedding independently, each respective layer output embedding depends only on the corresponding layer input embedding.

[0129] The one-dimensional convolutional layer 470 applies a ‘‘temporal’’ ID convolution to each input to the convolutional layer 470.

[0130] In other words, the convolutional layer 470 receive a layer input sequence that includes a respective layer input embedding for each input position of the input sequence for the time step and applies a convolution operation, i.e., a ID convolution operation, over the layer input sequence to generate a layer output sequence that includes a respective layer output embedding for each input position of the input sequence for the time step. Because the convolution is a ID convolution, the convolution is applied to each layer input embedding independently, and the respective layer output embedding depends only on the corresponding layer input embedding.

[0131] The LRU 402 then performs the operations described above to update the respective hidden state for each of the input positions using the output of the ID convolutional layer 470 for each of the input positions.

[0132] As can be seen from FIG. 4, the block 120 includes a multiplicative gate 480 proceeding the first and second channels 410 and 420. That is, for each input position, the block element- ise multiplies the output of the first channel 410 for the input position and the output of the second channel 420 for the input position to generate the output of the multiplicative gate for the input position. In some implementations, the output of the multiplicative gate is the output of the block 120. In other implementations, like the one shown in FIG. 5, the block 120 includes a linear layer 490 that that processes the output of the multiplicative gate 480 to generate the output of the block 120. ATTORNEY DOCKET NO. 45288-0537WO1

[0133] Prior to using the hybrid neural network 110 to perform inference on new videos, the system 100 or a different training system trains the hybrid neural network 110 on training data.

[0134] Because of the architecture of the hybrid neural network 110, the neural network can effectively be trained using supervised or self-supervised regimes or on a regime that first trains the neural network 110 through self-supervised learning and then fine-tunes the hybrid neural network 110.

[0135] As a particular example, given the factorised nature of the proposed architecture and the redundancy present in the video signal, the training system can apply masked autoencoding to enable self-supervised pre-training, i.e., pre-trains the neural network on a masked auto-encoding task in which random patches of input video frames are masked, and network outputs generated by the neural network for the un-masked portions are used to reconstruct the masked portions. For example, the system can use a masking strategy where a 2D random mask is generated and repeated for all the frames in the video, leading to masking entire temporal tubes. In this example, the output head is not used during the pre-training

[0136] The system can then fine-tune the hybrid neural network with the output head, e.g., by training only the output head or by training both the output head and the pre-trained portion of the hybrid neural network, on training data for a particular video prediction task. For example, the task can be one of the tasks described above. The loss function for this training can be any appropriate loss function for the corresponding video prediction task. Examples include cross-entropy losses for classification tasks, mean squared error or intersection over union losses for regression tasks, and so on.

[0137] In this specification, the term "configured" is used in relation to computing systems and environments, as well as computer program components. A computing system or environment is considered "configured" to perform specific operations or actions when it possesses the necessary software, firmware, hardware, or a combination thereof, enabling it to carry’ out those operations or actions during operation. For instance, configuring a system might involve installing a software library with specific algorithms, updating firmware with new instructions for handling data, or adding a hardware component for enhanced processing capabilities. Similarly, one or more computer programs are "configured" to perform particular operations or actions when they contain instructions that, upon execution by a computing device or hardware, cause the device to perform those intended operations or actions. ATTORNEY DOCKET NO. 45288-0537WO1

[0138] The embodiments and functional operations described in this specification can be implemented in various forms, including digital electronic circuitry, software, firmware, computer hardware (encompassing the disclosed structures and their structural equivalents), or any combination thereof. The subject matter can be realized as one or more computer programs, essentially modules of computer program instructions encoded on a tangible non- transitory storage medium for execution by or to control the operation of a computing device or hardware. The storage medium can be a storage device such as a hard drive or solid-state drive (SSD), a storage medium, a random or serial access memory device, or a combination of these. Additionally or alternatively, the program instructions can be encoded on a transmitted signal, such as a machine-generated electrical, optical, or electromagnetic signal, designed to carry infonnation for transmission to a receiving device or system for execution by a computing device or hardware. Furthermore, implementations may leverage emerging technologies like quantum computing or neuromorphic computing for specific applications, and may be deployed in distributed or cloud-based environments where components reside on different machines or within a cloud infrastructure.

[0139] The term "computing device or hardware" refers to the physical components involved in data processing and encompasses all types of devices and machines used for this purpose. Examples include processors or processing units, computers, multiple processors or computers working together, graphics processing units (GPUs), tensor processing units (TPUs), and specialized processing hardware such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). In addition to hardware, a computing device or hardware may also include code that creates an execution environment for computer programs. This code can take the form of processor firmware, a protocol stack, a database management system, an operating system, or a combination of these elements. Embodiments may particularly benefit from utilizing the parallel processing capabilities of GPUs, in a General-Purpose computing on Graphics Processing Units (GPGPU) context, where code specifically designed for GPU execution, often called kernels or shaders, is employed. Similarly. TPUs excel at running optimized tensor operations crucial for many machine learning algorithms. By leveraging these accelerators and their specialized programming models, the system can achieve significant speedups and efficiency gains for tasks involving artificial intelligence and machine learning, particularly in areas such as computer vision, natural language processing, and robotics.

[0140] A computer program, also referred to as software, an application, a module, a script, code, or simply a program, can be written in any programming language, including compiled ATTORNEY DOCKET NO. 45288-0537WO1 or interpreted languages, and declarative or procedural languages. It can be deployed in various fonns, such as a standalone program, a module, a component, a subroutine, or any other unit suitable for use within a computing environment. A program may or may not correspond to a single file in a file system and can be stored in various ways. This includes being embedded within a file containing other programs or data (e.g., scripts within a markup language document), residing in a dedicated file, or distributed across multiple coordinated files (e.g., files storing modules, subprograms, or code segments). A computer program can be executed on a single computer or across multiple computers, whether located at a single site or distributed across multiple sites and interconnected through a data communication network. The specific implementation of the computer programs may involve a combination of traditional programming languages and specialized languages or libraries designed for GPGPU programming or TPU utilization, depending on the chosen hardware platform and desired performance characteristics.

[0141] In this specification, the term "engine" broadly refers to a software-based system, subsystem, or process designed to perform one or more specific functions. An engine is typically implemented as one or more software modules or components installed on one or more computers, which can be located at a single site or distributed across multiple locations. In some instances, one or more dedicated computers may be used for a particular engine, while in other cases, multiple engines may operate concurrently on the same one or more computers. Examples of engine functions within the context of Al and machine learning could include data pre-processing and cleaning, feature engineering and extraction, model training and optimization, inference and prediction generation, and post-processing of results. The specific design and implementation of engines will depend on the overall architecture and the distribution of computational tasks across various hardware components, including CPUs, GPUs, TPUs, and other specialized processors.

[0142] The processes and logic flows described in this specification can be executed by one or more programmable computers running one or more computer programs to perform functions by operating on input data and generating output. Additionally, graphics processing units (GPUs) and tensor processing units (TPUs) can be utilized to enable concurrent execution of aspects of these processes and logic flows, significantly accelerating performance. This approach offers significant advantages for computationally intensive tasks often found in Al and machine learning applications, such as matrix multiplications, convolutions, and other operations that exhibit a high degree of parallelism. By leveraging the parallel processing capabilities of GPUs and TPUs, significant speedups and efficiency ATTORNEY DOCKET NO. 45288-0537WO1 gains compared to relying solely on CPUs can be achieved. Alternatively or in combination with programmable computers and specialized processors, these processes and logic flows can also be implemented using specialized processing hardware, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs), for even greater performance or energy' efficiency in specific use cases.

[0143] Computers capable of executing a computer program can be based on general-purpose microprocessors, special-purpose microprocessors, or a combination of both. They can also utilize any other type of central processing unit (CPU). Additionally, graphics processing units (GPUs), tensor processing units (TPUs), and other machine learning accelerators can be employed to enhance performance, particularly for tasks involving artificial intelligence and machine learning. These accelerators often work in conjunction with CPUs, handling specialized computations while the CPU manages overall system operations and other tasks. Typically, a CPU receives instructions and data from read-only memory (ROM), random access memory (RAM), or both. The elements of a computer include a CPU for executing instructions and one or more memory devices for storing instructions and data. The specific configuration of processing units and memory will depend on factors like the complexity of the Al model, the volume of data being processed, and the desired performance and latency requirements. Embodiments can be implemented on a wide range of computing platforms, from small embedded devices with limited resources to large-scale data center systems with high-performance computing capabilities. The system may include storage devices like hard drives, SSDs, or flash memory for persistent data storage.

[0144] Computer-readable media suitable for storing computer program instructions and data encompass all forms of non-volatile memory', media, and memory' devices. Examples include semiconductor memory devices such as read-only memory (ROM), solid-state drives (SSDs), and flash memory devices; hard disk drives (HDDs); optical media; and optical discs such as CDs, DVDs, and Blu-ray discs. The specific type of computer-readable media used will depend on factors such as the size of the data, access speed requirements, cost considerations, and the desired level of portability or permanence.

[0145] To facilitate user interaction, embodiments of the subject matter described in this specification can be implemented on a computing device equipped with a display device, such as a liquid cry stal display (LCD) or an organic light-emitting diode (OLED) display, for presenting information to the user. Input can be provided by the user through various means, including a keyboard), touchscreens, voice commands, gesture recognition, or other input modalities depending on the specific device and application. Additional input methods can ATTORNEY DOCKET NO. 45288-0537WO1 include acoustic, speech, or tactile input, while feedback to the user can take the form of visual, auditory’, or tactile feedback. Furthermore, computers can interact with users by exchanging documents with a user's device or application. This can involve sending web content or data in response to requests or sending and receiving text messages or other forms of messages through mobile devices or messaging platforms. The selection of input and output modalities will depend on the specific application and the desired form of user interaction.

[0146] Machine learning models can be implemented and deployed using machine learning frameworks, such as TensorFlow or JAX. These frameworks offer comprehensive tools and libraries that facilitate the development, training, and deployment of machine learning models.

[0147] Embodiments of the subject matter described in this specification can be implemented within a computing system comprising one or more components, depending on the specific application and requirements. These may include a back-end component, such as a back-end server or cloud-based infrastructure; an optional middleware component, such as a middleware server or application programming interface (API), to facilitate communication and data exchange; and a front-end component, such as a client device with a user interface, a web browser, or an app, through which a user can interact with the implemented subject matter. For instance, the described functionality could be implemented solely on a client device (e.g., for on-device machine learning) or deployed as a combination of front-end and back-end components for more complex applications. These components, when present, can be interconnected using any form or medium of digital data communication, such as a communication network like a local area network (LAN) or a wide area network (WAN) including the Internet. The specific system architecture and choice of components will depend on factors such as the scale of the application, the need for real-time processing, data security' requirements, and the desired user experience.

[0148] The computing system can include clients and servers that may be geographically- separated and interact through a communication network. The specific type of network, such as a local area network (LAN), a wide area network (WAN), or the Internet, will depend on the reach and scale of the application. The client-server relationship is established through computer programs running on the respective computers and designed to communicate with each other using appropriate protocols. These protocols may include HTTP, TCP / IP, or other specialized protocols depending on the nature of the data being exchanged and the security requirements of the system. In certain embodiments, a server transmits data or instructions to ATTORNEY DOCKET NO. 45288-0537WO1 a user's device, such as a computer, smartphone, or tablet, acting as a client. The client device can then process the received information, display results to the user, and potentially send data or feedback back to the sen- er for further processing or storage. This allows for dynamic interactions between the user and the system, enabling a wide range of applications and functionalities.

[0149] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0150] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0151] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0152] What is claimed is:

Claims

ATTORNEY DOCKET NO. 45288-0537WO1CLAIMS1. A method performed by one or more computers, the method comprising: receiving a video sequence comprising a respective video frame at each of a plurality of time steps; and processing the video sequence to generate a respective predicted output for each time step in a subset of the plurality of time steps, comprising, for each time step: receiving the video frame at the time step; processing the video frame at the time step to generate an input sequence for the time step comprising a respective input embedding at each of a plurality of input positions; and processing the input sequence for the time step, using a neural network, to generate a network output for the time step; wherein the neural network comprises: a recurrent block comprising a recurrent layer configured to, for each time step and each input position of the input sequence for the time step: receive a layer input comprising: (i) a recurrent state for a preceding time step and the input position, and (ii) a layer input embedding for the input position; and process the layer input to generate a layer output comprising: (i) a recurrent state for the time step and the input position, and (ii) a layer output embedding for the input position; an attention block comprising an attention layer configured to, for each time step: receive a layer input sequence comprising a respective layer input embedding for each input position of the input sequence for the time step; and apply an attention mechanism over the layer input sequence to generate a layer output sequence comprising a respective layer output embedding for each input position of the input sequence for the time step; and an output prediction head configured to, for each time step in the subset: receive the network output for the time step; and process the network output to generate the predicted output for the time step.ATTORNEY DOCKET NO. 45288-0537WO12. The method of claim 1, wherein the subset of the plurality of time steps is a proper subset.

3. The method of any preceding claim, wherein for each time step, the network output for the time step comprises a respective output embedding at each of one or more input positions of the input sequence for the time step.

4. The method of claim 3, wherein: the method further comprises, for each time step, adding a respective input embedding of each of one or more auxiliary tokens to the input sequence for the time step, and for each time step, the network output for the time step comprises a respective output embedding of each auxiliary token.

5. The method of any preceding claim, wherein the attention layer is a global attention layer, and for each time step, the global attention layer applies a global attention mechanism that, for each input position of the input sequence for the time step, attends over all of the input positions preceding or equal to the input position.

6. The method of claim 5, wherein the global attention mechanism is a dense attention mechanism.

7. The method of any of claims 1 -4, wherein the attention layer is a local attention layer, and for each time step, the local attention layer applies a local attention mechanism that, for each input position of the input sequence for the time step, attends only over a local subset of the input positions that are within a local window of the input position.

8. The method of any preceding claim, wherein for each time step, the layer input and output embeddings of the attention layer each have a plurality of dimensions, and the attention mechanism applies positional encoding to each of the plurality of dimensions of the layer input and output embeddings.

9. The method of claim 8, wherein the positional encoding is a relative positional encoding or a Rotary Positional Embedding (RoPE).ATTORNEY DOCKET NO. 45288-0537WO110. The method of any preceding claim, wherein the recurrent layer is a linear recurrent layer, and for each time step and each input position of the input sequence for the time step, the recunent state for the time step and the input position is linear in the recurrent state for the preceding time step and the input position.

11. The method of claim any preceding claim, wherein: the recunent block further comprises a one-dimensional convolutional layer immediately preceding the recurrent layer, and the one-dimensional convolutional layer is configured to, for each time step: receive a layer input sequence comprising a respective layer input embedding for each input position of the input sequence for the time step; and apply a convolution operation over the layer input sequence to generate a layer output sequence comprising a respective layer output embedding for each input position of the input sequence for the time step.

12. The method of claim 11, wherein: the recurrent block further comprises a linear layer immediately preceding the onedimensional convolutional layer, and the linear layer is configured to, for each time step: receive a layer input sequence comprising a respective layer input embedding for each input position of the input sequence for the time step; and apply a linear transformation over the layer input sequence to generate a layer output sequence comprising a respective layer output embedding for each input position of the input sequence for the time step.

13. The method of claim 12, wherein the recurrent block is a gated recurrent block comprising: a first channel comprising a feedforward layer configured to, for each time step: receive a layer input sequence comprising a respective layer input embedding for each input position of the input sequence at the time step; and apply an activation function over the layer input sequence to generate a layer output sequence comprising a respective layer output embedding for each input position of the input sequence for the time step;ATTORNEY DOCKET NO. 45288-0537WO1 a second, parallel channel comprising: (i) the linear layer, (ii) the one-dimensional convolutional layer, and (iii) the recurrent layer; and a multiplicative gate proceeding the first and second channels.

14. The method of claim 13, wherein the feedforward layer is a Rectified Linear Unit (ReLU) or a Gaussian error Linear Unit (GeLU).

15. The method of any of claims 13-14, wherein the first channel of the gated recurrent block further comprises a linear layer immediately preceding the feedforward layer.

16. The method of any preceding claim, wherein for each time step and each input position of the input sequence for the time step, the recunent state for the time step and the input position is the layer output embedding for the input position.

17. The method of any preceding claim, wherein the attention block further comprises a multilayer perceptron proceeding the attention layer.

18. The method of claim 17, wherein the attention block further comprises: a first skip connection betw een: (i) a first input preceding the attention layer, and (ii) a first additive gate proceeding the attention layer; and a second skip connection between: (i) a second input preceding the multilayer perceptron, and (ii) a second additive gate proceeding the multilayer perceptron.

19. The method of any preceding claim, wherein for each time step, processing the video frame at the time step to generate the input sequence for the time step comprises: dividing the video frame at the time step into a plurality of image patches for the time step; and processing the plurality of image patches for the time step, using an encoder, to generate the input sequence for the time step.

20. The method of claim 19, wherein for each time step, the input sequence for the time step comprises a respective input embedding of each of the plurality' of images patches for the time step.ATTORNEY DOCKET NO. 45288-0537WO121. A method performed by one or more computers, the method comprising: receiving a video sequence comprising a respective video frame at each of a plurality of frame positions; and processing the video sequence to generate a video processing output for the video sequence, comprising: for each frame position: receiving the video frame at the frame position; processing the video frame at the frame position to generate an input sequence for the frame position comprising a respective input embedding at each of a plurality of input positions; and processing the input sequence for each frame position, using a neural network, to generate the video processing output for the video sequence, wherein the neural network comprises: a recurrent block comprising a recurrent layer configured to, for each frame position and each input position of the input sequence for the frame position: receive a layer input comprising: (i) a recurrent state for a preceding frame position and the input position, and (ii) a layer input embedding for the input position; and process the layer input to generate a layer output comprising: (i) a recurrent state for the frame position and the input position, and (ii) a layer output embedding for the input position; and an attention block comprising an attention layer configured to, for each frame position: receive a layer input sequence comprising a respective layer input embedding for each input position of the input sequence for the frame position; and apply an attention mechanism over the layer input sequence to generate a layer output sequence comprising a respective layer output embedding for each input position of the input sequence for the frame position.

22. The method of claim 21, wherein the video processing output comprises a respective predicted output for each of one or more video frames in the video sequence.

23. A system, comprising:ATTORNEY DOCKET NO. 45288-0537WO1 one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the method of any of claims 1- 22.

24. One or more non-transitory storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any of claims 1-22.

Images