System and method for forming a consistent scene model from a video

Abstract

A system is proposed for generating a video representation of a video item composed of a plurality of frame representations corresponding to one (or more) of the frames of the video item. The video representation generation system includes a predictor network and a corrector network, defined by respective sets of multiple variable numerical parameters. For successive current frames of a sequence of frames, the predictor network receives a frame representation of the preceding frame, and from it obtains a preliminary representation of the current frame. The preliminary representation is then corrected by a corrector network conditioned on an embedding of the current frame, to generate a frame representation of the current frame.

Description

Attorney Docket No.: 45288-0470W01 SYSTEM AND METHOD FOR FORMING A CONSISTENT SCENE MODEL FROM A VIDEO BACKGROUND

[0001] This specification relates to processing data using machine learning models.

[0002] Machine learning models receive an input and generate an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate the output based on the received input and on values of the parameters of the model.

[0003] Some machine learning models are deep models that employ multiple layers of models to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a non-linear transformation to a received input to generate an output. SUMMARY

[0004] This specification generally describes how a system implemented as computer programs in one or more computers in one or more locations can use an adaptive system (“video representation generation system”) defined by a plurality of variable numerical parameters, to form respective representations of an ordered sequence of frames included in a video item (“frame representations”). The video item itself comprises, or consists of, a plurality of frames having an order. Each frame (or “image frame”) is a dataset including (or consisting of) one or more respective intensity values (e.g. three numerical values respectively defining RBG intensities) for each of an array (e.g. a two-dimensional array) of pixels. The frame representations may be considered as a model of a scene depicted in the video item.

[0005] The specification further describes how a system implemented as computer programs in in one or more computers in one or more locations can perform a method to train a video representation generation system for generating frame representations for a video item (e.g. a compressed representation, having a smaller number of bits than the video item), and a method to train a further adaptive system (a “task performance system”) to perform a video processing task using the frame representations for the video item. The video processing task may, for example, comprise recognizing and / or tracking content depicted in the video item.Attorney Docket No.: 45288-0470W01

[0006] Note that the representation generation system and / or task performance system may optionally not use all the frames of the video item. Instead, they may operate on an ordered sequence of frames which is a proper subset of the frames of the video item, i.e. selected frames from the video item. The ordered sequence of frames is generally in the same order as the selected frames have in the video item itself. For example, the ordered sequence of frames may be frames of the video item which are spaced apart pairwise by a certain number of frames of the video item which are not included in the ordered sequence of frames.

[0007] In general terms, a first aspect of the disclosure proposes that the frame generation network includes a predictor network and a corrector network, defined by respective sets of multiple variable numerical parameters. For successive “current” frames of the sequence of frames, the predictor network receives a frame representation of the preceding frame (i.e. the frame – if any - in the ordered sequence which precedes the current frame), and from it obtains (generates) a preliminary representation of the current frame. The preliminary representation is then corrected by a corrector network conditioned on an embedding of the current frame, to generate the frame representation of the current frame (i.e. the frame representation which will be processed by the predictor network in the next iteration, when the next frame of the ordered sequence is the current frame).

[0008] The predictor network and corrector network may be considered together as forming a recurrent model. The frame representation may be considered as a hidden state of the recurrent model, which is updated, for each successive frame of the ordered sequence, using that frame, to form a new frame representation corresponding to that frame.

[0009] The frame representation generated by the corrector network forms the input to the predictor network to generate the preliminary representation for the next frame of the ordered sequence of frames (i.e. the frame of the ordered sequence following the current frame). In this way, a frame representation is generated for each frame of the ordered sequence. The set of frame representations for the ordered sequence of frames may be referred to as a “video representation” of the video item.

[0010] When the current frame is the first frame of the video item, the received frame representation which is received by the predictor network is not based on a preceding frame (since there is no preceding frame), and may be formed in any way, e.g. each component may be selected independently from a distribution, or there may be a default frame representation (e.g. with all values set to a default value).

[0011] A specific expression of this concept is a computer-implemented method of processing a video item comprising an ordered sequence of a plurality of successive frames toAttorney Docket No.: 45288-0470W01 generate a representation of the video item, the method comprising, for successive current frames of the video item, receiving, by a predictor network, a frame representation which, except when the current frame is the first frame of the video item, is a frame representation of the preceding frame of the ordered sequence; processing, by the predictor network, the received frame representation to obtain a preliminary representation of the current frame of the video item; and processing, by a corrector network conditioned on an embedding of the current frame of the video item, the preliminary representation of the current frame of the sequence to generate a frame representation for the current frame

[0012] During the training of frame generation network, the preliminary representation of the current network is processed by a decoder network (also defined by a corresponding set of multiple variable numerical parameters) to generate a “decoded representation” which is indicative of a prediction of the current frame. A cost term is obtained which is a measure of a distance (difference) between the prediction of the embedding of the current frame indicated by the decoded representation and the actual current frame. Then, in a modification step, the corresponding numerical parameters of one or more of the predictor network, the corrector network and the decoder network are updated to reduce a loss function which includes the cost term for the current frame.

[0013] A specific expression of this concept is a computer-implemented method of training a video representation generation system comprising a predictor network, a decoder network and a corrector network, using a training database of one or more video items (“training video items”), each video item comprising a corresponding plurality of successive frames. The method comprises repeatedly: (i) obtaining a cost term for each current frame of an ordered sequence of frames included in one of the video items by: receiving a frame representation which (unless the current frame is the first frame of the ordered sequence) is based on a preceding frame of the ordered sequence, processing the received frame representation by a predictor network to obtain a preliminary representation of a current frame of the sequence, processing the preliminary representation of the current frame by a decoder network to obtain a decoded representation indicative of a prediction for the current frame (e.g. the decoded representation may be a prediction of at least a portion of the current frame (at theAttorney Docket No.: 45288-0470W01 same resolution, or a different resolution), though in principle it might instead by a prediction for an embedding of at least a portion of the current frame); updating the preliminary representation of the current frame by a corrector network conditioned on the current frame, to generate a frame representation for the current frame; and forming a cost term for the current frame indicative of a difference between current frame and the prediction for the current frame; and (ii) modifying at least one of the predictor network, corrector network and decoder network to reduce a loss function including the corresponding cost terms for the ordered sequence of frames.

[0014] The method may be performed multiple times (i.e. for iterations) and all of the predictor network, corrector network and decoder network are updated in multiple (e.g. corresponding) ones of the iterations; thus all of the predictor network corrector network and decoder network are iteratively trained in the method.

[0015] This process is carried out multiple times (i.e. iteratively), so that gradually, over multiple modification steps, the predictor network, corrector network and decoder network are trained. Once the frame generation network has been trained, the decoder network may be discarded.

[0016] The predictor network may use at least one attention neural network layer which applies an attention mechanism. This may be a self-attention mechanism in which the key vector(s), value vector(s) and query vector(s) are all based on the frame representation for the previous frame. The attention neural network layers may collectively generate an attention output based on at least one key vector, at least one corresponding value vector and at least one corresponding query vector which are all based on the frame representation for the previous frame. The predictor network may optionally add the attention output to received frame representation.

[0017] The corrector network too may use at least one attention neural network layer, which applies an attention mechanism. This may be a cross-attention mechanism in which the key vector(s) and value vector(s) may be based on embedding of the current frame, and the query vector(s) are based on the preliminary representation for the current frame. The attention neural network layers may collectively generate an attention output based on at least one such key vector, at least one such corresponding value vector and at least one such corresponding query vector. The corrector network may optionally add the attention output to preliminary representation for the current frame.Attorney Docket No.: 45288-0470W01

[0018] The embedding of the current frame may be generated in various ways. For example, as noted above, it may be generated using a trained embedding network. The embedding network may be pre-trained, e.g. for a different image processing task, or it may optionally be trained jointly with the predictor network, the corrector network and the decoder network (e.g. using the same loss function). By “jointly” is meant that updates to the embedding network are substantially simultaneous with, or interleaved with, updates to the predictor network, the corrector network and the decoder network. It may operate on low-level features of the frames (e.g. edges in the image, or intensity values for the pixels), and typically does not employ metadata for the frame. The embedding network may for example be a convolutional neural network (CNN) which generates a feature map based on the pixelated intensity values of the frame. The embedding of the current frame is one in which portions of the frame correspond to respective portions of the embedding, and a spatial relationship between different ones of the portions of the frame applies also between the respective portions of the embedding. That is, the embedding generally preserves spatial information in the frame. The embedding of the current frame is generally defined as one or more values for each of an array (e.g. a two-dimensional array) of points (“locations”). The number of locations is typically less, in each dimension, than the size of the pixel array of the frame in the corresponding direction, so that the embedding is a sub-sampling of the current frame.

[0019] The decoder network may use at least one attention neural network layer, which applies an attention mechanism. This may be a cross-attention mechanism in which the key vector(s), value vector(s) may be based on the preliminary representation of the current frame, and the query vector(s) correspond to spatial locations in the current frame. For example, each query vector may be based on a Fourier positional embedding of an array of points having correspondence to locations in the current frame (though optionally offset from them by a random amount and / or optionally reduced in number in each dimension, i.e. subsampled). The Fourier positional embedding may optionally be flattened to form a matrix based on which the query vectors are selected.

[0020] The cost term for a current frame may be based on any distance metric (e.g. Euclidean distance, Manhattan distance) which is a measure of the discrepancy between the predicted current frame and the actual current frame.

[0021] As noted above, there are a plurality of modification steps based on corresponding loss functions for each modification step. Optionally, a modification step, in which, as noted above, at least one of the predictor network, corrector network and decoder network isAttorney Docket No.: 45288-0470W01 modified, may be performed for each respective current frame, e.g. based on a loss function which includes only a cost term for the current frame.

[0022] Alternatively, a loss function may be formed, and used in the corresponding modification step, using a plurality of cost terms, generated for different respective current frames. For example, the loss function may include a respective cost term for each of the frames of the ordered sequence of frames, optionally omitting the cost term for one or more of the first frames of the ordered sequence. Additionally, the loss function may include a respective cost term for frames of multiple training video items (e.g. multiple ones of a training database of training video items).

[0023] In some cases, each modification step may include modifying all of the predictor network, corrector network and decoder network. In an alternative, in different ones of the modification steps, different corresponding one(s) of the predictor network, corrector network and decoder network may be modified. In this way, all of the predictor network, corrector network and decoder network are iteratively trained jointly (i.e. with the updates to different ones of the networks being substantially simultaneous or interleaved) based on the loss function(s).

[0024] Optionally, any modification step may include modifying the embedding network which produces the embedding of the current frame (e.g. a CNN, as mentioned above), also to reduce the loss function for the modification step.

[0025] The modification step may be performed by any known algorithm for reducing a cost function, e.g. by back propagation.

[0026] Following the training of the predictor network and the corrector network (and optionally when the decoder network has been discarded), the predictor network, the corrector network, and embedding network which produce the embedding of current frames used by the corrector network, may be used together as the video representation generation system, to generate a video representation of a video item. The video representation comprises, or consists of, frame representations of an ordered sequence of frames included in the video item. It has been found experimentally that the video representations have the emergent property of emphasizing objects which are depicted in multiple frames of the video item. Elements of the frame representation typically have no constant mapping to respective portions of the pixel array of the frame, so the frame representation is “off the grid”.

[0027] A task performance system may be trained to use a video representation produced by processing a video item using the trained video representation generation system, to perform a “downstream” video processing task on the video representation. Note that in a variation,Attorney Docket No.: 45288-0470W01 the task performance system could be trained jointly with the video representation generation system.

[0028] In some implementations, during the training of the task performance system, the representation generation system is used to generate video representations of training video items in a training database, and the task performance system is trained (e.g. by supervised learning, based on desired network outputs) to employ the video representations to perform the video processing task. Alternatively, the video representations may be generated and stored in a second database, so that the task performance system may then be trained using video representations extracted from the second database, i.e. the representation generation system is not required further during the training of the task performance system.

[0029] Once the task performance system has been trained, it may be used in combination with the video representation generation system as a “video processing system” for processing an input video item to perform the video processing task on the input video item.

[0030] The task performance system has an architecture which depends upon the video processing task.

[0031] In the case that the video processing task comprises the tracking of the movement of elements (e.g. objects) within the images, a “first kind” of task performance system may employ a dataset which is recursively updated. Specifically, an “initial” dataset may be defined representing elements of the frames which are to be tracked over the video item (e.g. points on objects depicted in multiple frames of the video item, or bounding boxes of objects depicted in multiple frames of the video item). The initial dataset may specify the positions of these points in a first frame of the ordered sequence of video frames included in the video item. Optionally, the positions may be encoded by positional encoding. The result may optionally be encoded further by a trained adaptive system (“encoding network”, e.g. a multilayer perceptron) to produce a set of latent values (tokens) describing the position of the elements in the first frame of the ordered sequence.

[0032] For each later current frame of the ordered sequence, the task performance system may update the dataset for the preceding frame based on the frame representation (and optionally the preliminary representation, or in principle the preliminary representation alone) of the current frame, to generate an updated dataset for the current frame. Unless the current frame is the last frame of the ordered sequence of frames, the updated dataset is passed on for use in generating an updated dataset for the next frame of the ordered sequence. Thus, the task performance system may successively update the dataset for each successive current frame of the ordered sequence of frames. The task performance system may be thought of asAttorney Docket No.: 45288-0470W01 comprising a recurrent network in which the dataset corresponds to the hidden state of the recurrent network.

[0033] The task performance system may include a decoder network (“task decoder network”), defined by a set of multiple variable parameters, which produces an output from the updated dataset of the current frame. This output is the result of processing the video processing task for the current frame. For example, the task decoder network may include a first adaptive system (e.g. a multi-layer perceptron MLP) to obtain coordinates indicating a position in the current frame of an element depicted in one or more frames of the video item, and / or a second adaptive system (e.g. a multi-layer perceptron MLP) to obtain an output indicating whether the element is visible in the current frame. A network output for the whole video item may be assembled by collecting the respective outputs of the task decoder network for one or more of (e.g. each of) the ordered sequence of frames included in the video item.

[0034] Optionally, the task performance system may perform the updating of the dataset using a corrector network (“a task corrector network”) defined by a set of multiple variable parameters.

[0035] In some implementations, for each current frame of the ordered sequence except the first frame, the dataset for the preceding frame is received by the task corrector network. The task corrector network is conditioned on the frame representation (and optionally also the preliminary representation). The task corrector network may, for example, comprise one or more attention neural network layers which apply a cross-attention mechanism, in which key vector(s) and value vector(s) are generated based on the frame representation (and optionally the preliminary representation), and the query vectors are generated based on the received dataset.

[0036] The task performance system may further include a task predictor network. The task predictor network may receive an output of the task corrector network (i.e. the updated dataset) and perform a modification to it, to generate to dataset which, except if the current frame is the last frame of the ordered sequence, is processed by the task corrector network for the next frame of the ordered sequence.

[0037] The concept of a task performance system which employs an ordered sequence of frame representations from a video item (e.g. frame representations generated by a predictor network and corrector network as discussed above), and generates a network output which is the result of performing the task on the video item of the task, by recurrently updating a dataset, where each update is performed when the task performance system (e.g. a taskAttorney Docket No.: 45288-0470W01 corrector network of the task performance system) is conditioned on corresponding one of the sequence of frame representations, constitutes a second, independent aspect of the disclosure

[0038] Note that for some video processing tasks, the task performance system may have a different architecture. For example, a “second kind” of task performance system may contain a task decoder network which receives data based on the frame representation for (only) the current frame, and a spatial query which indicates one or more locations in the current frame. For example, the spatial query may be similar to the query vector received by the decoder network during the training of the representation generation system. For example, the spatial query may be based on a Fourier positional embedding of an array of points having correspondence to locations in the embedding of the current frame (though optionally offset from them by a random amount and / or optionally reduced in number in each dimension, i.e. subsampled). The Fourier positional embedding may optionally be flattened to form a matrix based on which the query vectors are selected. The task decoder network may be implemented using one or more attention layers in which the query vector is based on the spatial query, and the key vector and value vectors are based on the frame representation (and optionally preliminary representation) for a current frame.

[0039] The task performance system may be trained, e.g. by a supervised learning technique. In the case of a task performance system (of the first kind) with a task corrector network, this comprises training at least the task corrector network, and optionally also the encoding network (if present), though that may alternatively have been pre-trained in some way, and the task predictor network (if present). In the case of a task performance system of the second kind, described in the preceding paragraph, it may involve training at least one key matrix, query matrix and value matrix of each attention layer. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Fig.1 shows an example video representation generation system for generating a representation of a video item.

[0041] Fig.2 shows steps of an exemplary method processing a video item comprising an ordered sequence of a plurality of successive frames to generate a representation of the video item.

[0042] Fig.3 shows a method for training a video representation generation system.

[0043] Fig.4 shows a decoder network of the video representation generation system.Attorney Docket No.: 45288-0470W01

[0044] Fig.5 shows a method for training a video representation generation system.

[0045] Fig.6 shows a video processing system for performing a video processing task on a video item to obtain information about the video item.

[0046] Fig.7 shows a method for performing a video processing task on a video item to obtain information about the video item.

[0047] Fig.8 shows a first possible form for a task performance system.

[0048] Fig.9 shows a second possible form for a task performance system.

[0049] Fig.10 shows experimental results from a comparison of an example of the present disclosure and five other video representation generation systems.

[0050] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0051] Fig. 1 shows a video representation generation neural network system 100 (more concisely, a video representation generation system) for generating a representation (“video representation”) of a video item. The video item comprises an ordered sequence of a plurality of video frames, e.g. each frame being one or more intensity values for each of an array (e.g. a two-dimensional array) of pixels. The video representation is composed of a plurality of frame representations, each corresponding to one (or more) of the frames of the video item.

[0052] The number of frames in the ordered sequence of the video item is denoted T. Individual ones of the frames of the (of the ordered sequence of frames) correspond to values of the integer index t, so that for convenience a given frame may be referred to as e.g. frame i. The T frames of the ordered sequence are each an array of intensity values denoted ^^^௧^்௧ୀ^. Each ^^௧∈ℝுൈ^ൈௗ, where an ^^ ൈ ^^ array of pixels, and each pixel is a number d of(e.g. d=3). video item may, for example, be one which is being captured during the video representation generation process, i.e. later frames of the video item may be captured while the frame representation of earlier frames of the video item are being generated. Alternatively, the video item may be one captured (or otherwise obtained) before the video representation generation process is performed.

[0054] Note that the terms “later” and “earlier” refer here to the order of the ordered sequence, which is not necessarily the same order in which actions depicted in the video occur. That is,Attorney Docket No.: 45288-0470W01 if the video item is depicting a scene in which events happen over a period of time, the order of the ordered sequence might be such that later events in the period of time are later in the ordered sequence, but in principle, particularly if the video item is one obtained before the video representation generation process is performed, the order of the events could be opposite to this, i.e. the “later” frames of the ordered sequence may depict events which occurred earlier than events depicted by earlier frames of the ordered sequence.

[0055] The video representation generation system 100 comprises a predictor (neural) network 101 operative to receive a frame representation of a given frame of the ordered sequence of frames, and to predict from the given frame of the ordered sequence data describing the next frame of the ordered sequence of frame (a “preliminary representation of the next frame of the ordered sequence of frame, which is a preliminary version of the frame representation for the frame).

[0056] For simplicity in this explanation, it will be assumed that the frame representations each relate to a single frame of the video item, and that the ordered sequence of frames constitute all the frames of the video item in the ordered sequence. However, in variations within the scope of the present disclosure, the video item might contain other frames in addition to the ordered sequence of frames; for example, each pair of consecutive frames in the ordered sequence of frames might be spaced apart in the video item by a certain number M of other frames which are not part of the ordered sequence. This possibility may reduce the computational burden of generating and employing the video representation. Alternatively, in other variations, the video item might be partitioned into a plurality of groups of frames, where each group contains plural frames, and a frame representation might be generated for each group. In this case, the ordered sequence of frames may be identified with the first frame of each group. This possibility too is not considered here further.

[0057] The frame representation for frame t is denoted ^^௧^where ^^^ ^ൈ^௧ ∈ ℝ is a two-dimensional array of latent values (tokens), where K and D are integers.

[0058] Thus, at a given time the predictor network may receive the frame representation for a given frame (say the frame representation ^^௧^ି^of the frame t-1), and process it to generate a preliminary representation of the video item immediately following the given frame, e.g. frame t, which here is called the “current frame”. The preliminary representation of the set of T frames is denoted ^^^^் ௧^௧ୀ^ where ^^^௧ ∈ ℝ^ൈ^ is a two-dimensional array of latent values. Note that thelatent e.g. higher, dimensionality, and even in the case thatit is a two- is, in general, no spatial relationship between the array ofAttorney Docket No.: 45288-0470W01 latent values and the pixel array (e.g. no pre-determined relationship, although such a relationship may possibly be emergent in a given instance).

[0059] The predictor network 101 may be implemented as a transformer (denoted ^^) which applies an attention mechanism to its input ^^௧^ି^to generate a ℝ^ൈ^attention output, and adds the attention output to ^^^to generate the pr^௧ି^eliminary representation ^^௧of the current frame. The transformer ^^ may use ^^௧^ି^as all of the key vector k , query vector q and value vector v. Thus, ^^^௧ ൌ ^^ ^௧ି^ ^ ^^^^^^^^^ ൌ ^^௧^ି^). (1) The role of the ^^^௧for the frame representation of the current frame (i.e. ^^௧^) based on the previous frames which are encoded in ^^௧^ି^. This is done without employing the current frame ^^௧, or any data derived from it. Note that ^^^^may be initialized in any way, e.g. set to random Gaussian noise (e.g. with zero mean and standard deviation 10-4). It is not learned. This choice of random initialization is made because it breaks the symmetry among the latent values (tokens) ^^^^. It also reduces “specialization” of tokens and maintains permutation symmetry.

[0060] The video representation generation system 100 further comprises an embedding (neural) network 102. The embedding network 102 is operative to receive a frame of the video item (e.g. frame t) and, from it, generate an encoding (“embedding”) of the frame. The embedding of the current frame t (that is, the embedding of the set of pixelated intensity values ^^௧^ is denoted ^^௧. In an implementation, the embedding network 102 may comprise a convolutional neural network ℇ with an added Fourier positional embedding to its output. This processes the set of pixelated intensity values ^^௧, and the result is linearly projected to producethe embedding ^^௧ ൌ ℇ^^^௧^ ∈ ℝுᇱൈ^ᇱൈ^, where H’ and W’ are integers which may differ fromH and W due to

[0061] The video further comprises a corrector (neural) network 103. The corrector network receives both the embedding ^^௧of the current frame t, and the preliminary representation ^^^௧ of the current frame t. The corrector network processes them together. That is, the corrector network 103 is “conditioned on” the embedding ^^௧of the current frame t while it processes the preliminary representation ^^^௧ of the current frame t. The result of this processing is a frame representation ^^^for the frame

[0062] The corrector network 103 may be as a cross-attention transformer, denoted ^^. It may use a query vector which is (or is based on) the preliminary representation ^^^௧ of the current frame, and define the key and value vectors based on ^^௧. The output of theAttorney Docket No.: 45288-0470W01 cross-attention mechanism is added to the preliminary representation ^^^௧ of the current frame t. Thus, ^^^௧ ൌ ^^ ^௧ ^ ^^^^^^^ ൌ ^^௧ , ^^ ൌ ^^௧^ି^). (2)

[0063] Turning to Fig. 2, aan example of the present disclosure. The method 200 may be implemented as computer programs in one or more computers in one or more locations. For example, the video representation generation system 100 of Fig.1 can perform the method 200. The method 200 is performed for each current frame t of a video item which is an ordered sequence of T frames.

[0064] In step 201, a predictor network (such as the predictor network 101) receives a frame representation ^^௧^ି^of the preceding frame t-1 of the ordered sequence.

[0065] In step 202, the predictor network, processes the received frame representation ^^௧^ି^to obtain a preliminary representation ^^^௧ of the current frame of the video item.

[0066] In step 203, the corrector network (such as the corrector network 103), conditioned on an embedding ^^௧of the current frame t (i.e. the pixelated intensity values ^^௧), processes the preliminary representation ^^^௧ of the current frame of the video item, to generate a frame representation ^^௧^of the current frame. That is, the corrector network updates ^^^௧ to ^^௧^based on the embedding ^^௧of the current frame t.

[0067] Turning to Fig. 3 a system is shown for training the video representation generation system 100 of Fig. 1. The training is done using a training system 300 comprising a training engine 301. The training engine 301 has access to a training database 302 which comprises one of more training video items, which are each an ordered sequence of frames. The training system 300 further comprises a decoder (neural) network 303. The decoder network 303 is depicted in more detail in Fig.4.

[0068] Note that each of the predictor network 101, the embedding network 102, the corrector network 103 and the decoder network 303 are defined by a respective set of a plurality of numerical values. The training engine 301 is operative to update one or more (e.g. all) of these sets of numerical values during the training. The training may comprise the training engine 301 iteratively updating all of these sets of numerical values (with any one or more of the sets of numerical values being updated in each iteration, but all of the sets of numerical values being updated in at least some of the iteration). Note that in a variation, one or more of the predictor network 101, the embedding network 102, the corrector network 103 and the decoder network 303 (e.g. the embedding network 102) may not be iteratively updated. That is, its respective setAttorney Docket No.: 45288-0470W01 of numerical values may take predetermined values (e.g. ones learnt in a previous training process).

[0069] The training engine 301 selects a training video item from the training database 302. The training engine successively passes the frames successively to the embedding network 102. The training engine may also receive the corresponding output generated from each frame by the embedding network 102.

[0070] The representation generation system generates a series of frame representations based on the selected training video item in the manner explained above with reference to Fig.1 and Fig.2.

[0071] The preliminary representations^^^^௧^generated from the training video item are successively passed to the decoder network 303. For each current frame t, the decoder network 303 processes the current prediction ^^^௧ to decode it into a decoded representation indicative of a prediction of the pixelated intensity values ^^௧of the current frame t. The decoded representation may be an image (e.g. an RGB image) which is a prediction, of any selected resolution, of at least a portion of the current frame.

[0072] As depicted in Fig. 4, the decoder network 303 may be implemented as a transformer 401 which implements a cross-attention mechanism, where the query vectors are embedded pixel co-ordinates ^^ in an array of pixels 402 having the selected resolution. The pixels of the array of pixels 402 have a mapping to the array of pixels of each frame of the video item. The transformer 401 uses a key vector and a value vector based on the preliminary representation ^^^௧. A respective query vector may be successively generated for each of one or more positions ^^ in the array of pixels 402. The transformer 401 processes the query vector and the preliminary representation ^^^௧ ^^o generate a respective output, which is used as the corresponding pixel of the decoded representation (decoded image). Note that the decoder network 303 of Fig. 4 is particularly suitable because the latent values (tokens) of the preliminary representation ^^^௧ are “off the grid” (i.e. they have no fixed (e.g. predetermined) correspondence to the spatial image grid of the frames of the video item). The decoded imagefor the current frame t is denoted ^^^௧. Thus,^^^௧ ൌ ^^^^^^^ ൌ ^^ ^௧ , ^^ ൌ ^^) (3)for different successiveNote that, to reduce the computational demands of the process, a respective query vector may not be generated for all positions (pixels) ^^. Rather, a respective component of ^^^௧may be generated for aAttorney Docket No.: 45288-0470W01 plurality of positions in the array of pixels 402 which is a (proper) subset of all pixels in the array of pixels 402.

[0073] If the resolution of the decoded image is lower than that of the corresponding currentframe ^^௧, then ^^^௧ approximates a downsampled version of ^^௧.

[0074] For each frame, a corresponding cost term Lt is defined as a function of ^^^௧ and ^^௧, andindicative of a difference between the current frame ^^௧and the prediction for the current frame indicated by the decoded representation ^^^௧. The difference may, for example, be evaluated as an L2 loss (sum of squared differences), i.e. the cost term is given by ^^௧ ൌ ^^ଶ^ ^^^௧, ^^௧^. (4)Note that thisresolution of ^^^௧and ^^௧, and if ^^^௧is not evaluated for all pixel positions ^^, ^^௧is evaluated only taking into accountin the sum the pixel positions ^^ at which ^^^௧ was evaluated.

[0075] This process is repeated for multiple frames of the training video item, to generate a respective cost term Ltfor each of those frames. For the first frame of the training video item, the frame representation ^^^^(i.e. the frame representation which the predictor network 101 uses to evaluate ^^^^), is set randomly, e.g. to random Gaussian noise as described above. A loss function is then defined as the sum of the corresponding cost terms Lt for each of the frames of the training video item.

[0076] An update is then made to one or more of the respective sets of numerical parameters defining the predictor network 101, the embedding network 102, the corrector network 103 and the decoder network 303. The update is such as to reduce the loss function.

[0077] Turning to Fig. 5 a flow chart is shown of a method 500 which is an example of a method according to the present disclosure, for training a video representation generation system, such as the video representation generation system 100 of Fig.1. The method 500 may be implemented as computer programs in one or more computers in one or more locations. For example, the training system 300 of Fig. 3 can perform the method 500. The method 500 is performed for each current frame t of a selected training video item which is an ordered sequence of T frames. Note that the value of T need not be the same as the value of T used in the method 200 of Fig..2.

[0078] In step 501, a predictor network (such as the predictor network 101) receives a frame representation ^^௧^ି^of the preceding frame t-1 of the ordered sequence of frames comprised by the selected vide item.Attorney Docket No.: 45288-0470W01

[0079] In step 502, the predictor network processes the received frame representation ^^௧^ି^to obtain a preliminary representation ^^^௧ of the current frame of the selected trainingitem.

[0080] In step 503, a decoder network (such as the decoder network 303) processes the preliminary representation ^^^௧ of the current frame of the selected training video item to generate a decoded representation indicative of a prediction for the current frame ^^௧. For example, the decoded representation may be a decoded image ^^^௧which is a prediction (possibly at a different resolution) of at least a portion of the current frame ^^௧.

[0081] In step 504, the corrector network (such as the corrector network 103), conditioned on an embedding ^^௧of the current frame t (i.e. the pixelated intensity values ^^௧), processes the preliminary representation ^^^௧ of the current frame of the selected training video item, to generate a frame representation ^^௧^of the current frame. That is, the corrector network updates ^^^௧ to ^^௧^based on the embedding ^^௧of the current frame t of the selected training video item.

[0082] In step 505, a corresponding cost term Ltis defined as a function of the decodedrepresentation (e.g. ^^^௧) and the current frame ^^௧. The cost term is indicative of a differencebetween the current frame ^^௧and the prediction for the current frame indicated by the decoded representation. The difference may, for example, be evaluated as an L2 loss (sum of squared differences), i.e. the cost term is given by Eqn. (4).

[0083] In step 506, a modification is made to at least one of the predictor network, the corrector network, the decoder network (and optionally also the embedding network). The modification is such as to reduce the loss function comprising the cost term Lt. Optionally, the loss function may contain further loss terms obtained from different respective current frames of the selected training video item (or from current frames of different selected training video items). That is, the set of steps 501-505 may be performed on multiple occasions, using different frames of the selected training video item as the respective current frame, to generate respective cost terms, and the loss function may comprise a sum of those cost terms. Optionally, for a first one of these occasions, the frame representation employed in step 501 (denoted ^^^^) may be set without reference to the training video item, e.g. randomly, such as Gaussian noise as explained above. Then set of steps 501-505 are performed multiple times, with each performance using consecutive corresponding frames of the selected training video item as the current frame, to generate respective cost terms, which are summed to form (at least one term of) the loss function.

[0084] The set of steps 501-506 may be performed multiple times, each being a respective iteration of a training process for the video representation generation system. The process stopsAttorney Docket No.: 45288-0470W01 when a termination criterion is reached, e.g. that the amount by which the update reduces the loss function is below a threshold, or that a maximum number of iterations has been reached.

[0085] Turning to Fig.6, an exemplary video processing system 600 is shown for performing a video processing task on a video item to obtain information about the video item.

[0086] The video processing system 600 of Fig. 6 is identical the video representation generation system 100 of Fig. 1, except that it further comprises a task performance system 601 which is configured to receive the frame representations and / or the preliminary representations of the video item generated by the video representation generation system 100. The task performance system 601 is configured to generate a network output which is the result of performing a video processing task on the video item. Two exemplary realizations of the task performance system 601 are explained below with reference to Figs.8 and 9. Optionally, the task performance system 601 may also receive the preliminary representations of the video item; this is indicated in Fig.6 by a dashed line.

[0087] Turning to Fig. 7, a flow chart is given of a method 700 which is an example of the present disclosure. The method 700 may be implemented as computer programs in one or more computers in one or more locations. For example, the system 600 of Fig. 6 can perform the method 700.

[0088] In step 701 a video item is processed, such as by the method 200 of Fig.2, to generate a video representation comprising a plurality of frame representations for respective ones of the ordered frames of the video representation.

[0089] In step 702, one or more (or all) of the frame representations generated in step 701 are processed by a task performance system (such as the task performance system 601) to produce a network output indicative of (e.g. which is) the result of performing a video processing task on the video item.

[0090] Optionally, the step 702 may be performed in parallel with step 701, e.g. such that upon a frame representation being generated in step 701 corresponding to a first frame of the video item, the frame representation is processed in step 702 while step 701 is being performed to generate a frame representation of a later frame (e.g. the next frame, immediately following the first frame) of the video item.

[0091] Fig. 8 shows a first possible realization of the task performance system 601. The task performance system has a structure similar to that of the decoder network 303 of Fig. 4. As depicted in Fig.8, the task performance system 601 may be implemented as a transformer 801. The transformer implements a cross-attention mechanism. Multiple successive query vectors are generated based on corresponding embedded pixel co-ordinates ^^ in an array of pixels 802Attorney Docket No.: 45288-0470W01 having a selected resolution (which may be the same or different from the resolution of the pixel array of the frames of the video item), and the key vector and value vectors are based on the preliminary representation ^^^௧ and / or the frame representation ^^௧^of a current frame of the video item.

[0092] A respective query vector may be successively generated for each of one or more positions ^^ in the array of pixels 802 having a mapping to the array of pixels of each frame of the video item. The transformer 401 processes each query vector, and the key vector and the value vector based on the preliminary representation ^^^௧ and / or frame representation ^^௧^, to generate a respective output, which is used as the corresponding pixel of a network output in the form of an image. Note that, like the decoder network 303, the task performance system 601, is particularly suitable because the latent values (tokens) of the preliminary representation ^^^௧ are “off the grid” (i.e. they have no direct correspondence to the spatial image grid of the frames of the video item.

[0093] The video processing task may be to generate information about specified positions in a (current) frame t of the video item. The network output may be used to produce a grid-based readout, such as a depth associated with the current frame t of the video item (other examples of obtaining information about a single frame of the video item are given below). That is, individual spatial locations ^^ are queried, using their (x,y) locations, in a transformer which (solely) uses cross-attention. Optionally, the pixel array 802 is a subsampled spatial grid with a random offset to reduce or avoid any risk of overfitting on a particular subset of pixels.

[0094] Turning to Fig. 9, an alternative form of the task performance system 601 is shown. This form of task performance system uses a recurrent readout architecture, in which a dataset is successively updated based on each of the frame representations of a video item.

[0095] An input to the task performance system 601 is a set of one or more queries 901, denoted ^^^. Each query specifies a respective portion of a pixel array corresponding to (e.g. being equal to, or being a lower resolution version of) the pixel array of each frame of the video item. The portion(s) of the pixel array specified by the query / queries 901 may be points (pixels) of the pixel array, and / or multi-pixel areas of the pixel array (e.g. boxes of the pixel array). Thenumber of queries is denoted by N, and ^^ ேൈ^^ ∈ ℝ ^, where ^^^ is the number of dimensions ofthe data used to specify the position corresponding to a single query, e.g. two digits if the query specifies a respective portion of the pixel array which is a point, and 8 digits if the query specifies a respective portion of the pixel array which is a rectangular box (i.e. the coordinates of the four corners of the box).Attorney Docket No.: 45288-0470W01

[0096] The queries ^^^are used to indicate corresponding positions in the pixel array of the first frame of the videoThe video processing task may be to generate, for each later frame t of the video item, a corresponding dataset ^^௧. Each element of each ^^௧corresponds to a respective element (position) of ^^^, and indicates the respective position in frame t of an objects depicted at the corresponding position of ^^^. The task is to predict ^^^௧^ for t=2,…,T.

[0097] The query ^^^901 is embedded by an embedding unit 902 (implemented as a neuralnetwork), to form a latent encoding ^^^ ∈ ℝேൈ^^903 (readout “latents”).

[0098] In each of a series ofcorresponding to respective current frames t of the video item, a task corrector (neural) network 904, conditioned on the corresponding frame representation ^^^(and / or th^௧ e preliminary representation ^^௧), applies a cross-attention mechanism to the corresponding current version ^^௧of the latent encoding (corresponding to the current frame t), to produce an updated (“corrected”) version ^^^௧of the latent encoding (a set of readout “latents”). For example, for the case t=1, ^^^^is labelled 905 in Fig.9.

[0099] The task corrector network 904 may be implemented as a transformer 904. In each iteration, the transformer 904 forms a query vector based on ^^௧(in fact, ^^௧may be used as the query vector), and forms a key vector and value vector based on the preliminary representation ^^^௧and / or frame representation ^^௧^of a current frame of the video item.

[0100] A task predictor (neural) network 908 then uses the corrected latent encoding ^^^௧to predict ^^௧ା^(e.g. ^^ଶ, labelled 909), which is a current predicted state for the next frame. The task predictor network 908 may be implemented (solely) as a multi-layer perceptron (MLP) defined by a respective set of values, i.e. without any self-attention between readout latents.

[0101] Although Fig.9 only full depicts the iteration for the frame t=1, it is to be understood that the process may continue until a corresponding corrected version of the latents has been generated for all the frames of the video item. Each iteration corresponds to a respective frame of the video item, and can be considered a respective readout layer. In each iteration except the first, the corrector network 904 processes a current predicted state ^^௧(latent encoding) corresponding to a current frame t (e.g. ^^௧is latent encoding 909 when the current frame is t=2) generated by the task predictor network 908 in the previous iteration. The corrector network 904, when implemented as a transformer, may do this solely cross-attending the current predicted state into the inputs of the corrector network (the preliminary representation ^^^௧and / or frame representation ^^௧^of the current frame). This is done without self-attention between the readout layers. For example, the corrector network 904 does not process the preliminary representation ^^^௧ି^and / or frame representation ^^௧^ି^of the preceding frame of theAttorney Docket No.: 45288-0470W01 video item; nor does the task corrector network 904 process ^^௧ି^or ^^^௧ି^, though the task corrector network 904 does receive and process ^^௧which the task predictor neural network 908 generated from ^^^௧ି^.

[0102] Each corrected version of the readout latents ^^^௧is processed by a task decoder (neural) network to form the network output. In Fig. 9, for example, the task decoder network is implemented as two neural networks 906, 907, which respectively form data of a different type, e.g. co-ordinates, visibility, etc., for each of the queries. The output of the task decoder network 906, 907 forms the network output of the system 600. The neural networks 906, 907 may optionally be implemented by MLPs, defined by respective sets of values.

[0103] Returning to Fig. 6, the training of the task performance system 601 (e.g. the task performance systems 601 depicted in Figs.8 and 9) is generally performed after the training of the video representation generation system (e.g. system 100 of Fig. 1) which it comprises. It uses a training database of training video items (e.g. the training database 302).

[0104] A training video item is selected from the training database, and processed by the video representation generation item, to generate inputs to the task performance system 601 (e.g. the preliminary representation ^^^௧ and / or frame representation ^^௧^corresponding to each frame of the training video item). The preliminary representations ^^^௧ and / or frame representations ^^௧^for the training video item, are then processed by the task performance system 601, to generate a network output (e.g. a network output comprising a component for each frame of the training video item).

[0105] A task loss function is then defined which is a measure of a difference between the network output and a desired network output for the training video item. An update is then made to numerical parameters defining the task performance system 601, to reduce the task loss function. Multiple iterations of this process may be performed, to gradually train the task performance system 601. The process terminates when a termination criterion is reached, e.g. that the amount by which the update reduces the task loss function is below a threshold, or that a maximum number of iterations has been reached.

[0106] For example, if the video processing task is to find a depth value for positions in frames of a video item (e.g. using a task performance system 601 as depicted in Fig. 8), the desired network output may be a “ground truth” depth value for corresponding positions in frames of the training video item. The task loss function may indicate a measure (e.g. a Euclidean distance) of the difference between the portion of the network output for the frame of the training video, and the ground truth depth values for that frame (e.g. summed over one or moreAttorney Docket No.: 45288-0470W01 positions in one or more frames of the training video item). The update may be to change numerical parameters defining the transformer 801 (e.g. query, key and value matrices of the transformer 801).

[0107] Alternatively, if the video processing task is to track, or monitor the visibility of, elements depicted in a video item (e.g. using a task performance system as depicted in Fig.9), the desired network output may be a “ground truth” position / visibility value in each frame of the training video item (except the first frame) for objects which, in the first frame of the video item, are at a set of positions specified by one or more queries (e.g. queries 901). In the case of tracking positions, the task loss function may indicate a measure (e.g. a Euclidean distance) of a difference between the positions in each frame specified by the MLP 906 and the ground truth positions. Alternatively or additionally, in the case of tracking visibility, the task loss function may indicate an absolute difference between a visibility, specified by the MLP 907, in each frame of the training video item (except the first frame) of an object which, in the first frame of the video item, is at a corresponding one of the position(s) specified by of the query ^^^, and the corresponding ground truth visibility value. In either case, the task loss function may be summed over one or more frames of the training video item. The update may be to change numerical parameters defining any one or more of respective sets of numerical parameters defining the operation of the embedding unit 902, the task corrector network 904 (e.g. when the task corrector network is a transformer 904, numerical parameters defining query, key and value matrices of the transformer 904), the task predictor network 908 and / or the task decoder network 906, 907. In different iterations of the training procedure, the update may update different corresponding ones of these units, or each unit may be updated in each iteration.

[0108] We now turn to a discussion of the video processing tasks which the video processing system can be trained to perform using the disclosed methods.

[0109] As noted above, some of those tasks may comprise tracking elements depicted in multiple ones of the ordered sequence of frames. For example, the elements may be specific points on corresponding objects depicted in multiple frames of the ordered sequence of frames. Alternatively, the elements may be bounding boxes for objects in multiple frames of the ordered sequence of frames. The task processing system may be of a first kind, such as the task processing system depicted in Fig.9. The initial dataset may include a plurality of portions for each of the elements which is to be tracked. The task corrector network, task decoder network and (if present) task predictor network preferably treat these portions separately, e.g. the task corrector network and (if present) task predictor network process the portions of the datasetAttorney Docket No.: 45288-0470W01 separately, to generate corresponding respective portions of the update datasets they produce. Thus, information about multiple ones of the elements is not mixed up.

[0110] Another possibility is for the video processing task to comprise identifying a corresponding depth distance, i.e. distance from the camera which captured the frame, for each of one or more groups of pixels in a plurality of frames in the video item. These pixels represent an object depicted in the frame and which is at a certain distance from the camera. A task processing system of a second kind (such as the one depicted in Fig. 8) may be used in this case.

[0111] Another possibility is for the video processing task to comprise recognizing the content of a received video. This may be treated as a classification task, that is to generate, based on a video representation corresponding to a video item, one or more labels indicative of content of the video item. In one example, the labels may indicate whether the video depicts (shows) an object or animal in one or a plurality of predetermined categories (e.g. the category “dogs”, or the category “humans”), or a real-world event in one of a plurality of determined categories (e.g. a car crash). Thus, using the disclosed method, a task performance system (e.g. of the second kind) is produced which is able to generate labels of this kind. One use of a video processing system including the task performance system would be to scan a database of videos to generate metadata based on the labels and describing the content of the videos. Another use of the video processing system would be to scan a database of videos to identify videos of a certain type (e.g. ones in which an object or animal in one of the categories appears), such as to display them or to send a message identifying them, e.g. to a user.

[0112] The process of training the task processing system may for example be performed in a supervised manner, based on labels indicating the content of the corresponding video items stored in the training database. The labels may be supplied to the training system together with the video items which are stored in the first training database. The training algorithm may be any known algorithm used in the field of supervised learning, e.g. to minimize a loss function which characterizes discrepancies, when the adaptive system receives a video representation of a video item in the training database, between labels it generates and the corresponding labels associated with the received video representation.

[0113] Optionally, the labels which the adaptive system is trained to generate for a given received data item may relate not to the video item as a whole, but to specific portions of the video items, such as (proper) subsets of the frames of the video item (e.g. such that the label indicates which of the frames depicts an object or animal of a given category and / or an event of a given category). In this case, the labels relate to subsets of the frames of the video items inAttorney Docket No.: 45288-0470W01 the training database, e.g. indicating that those (and only those) frames depict objects, animals or events in one of the defined categories.

[0114] Furthermore, the specific portions of the video items for which the adaptive system is trained to generate labels, may be areas (i.e. groups of pixels) in frames of a video item. For example, the labels may indicate that a specific portion (sub-area) of one or more specific frames depicts an object or animal in a given category or an event in a given category. Thus, thus label generates a segmentation within frames of a received video item. In this case, the labels associated with the video items in the training database relate to specific portions of frames of the video items in the database, e.g. indicating that those (and only those) specific portions of the frames of the video items depict objects, animals or events in one of the defined categories.

[0115] An alternative video processing task which the adaptive system can be trained to perform is to generate data indicating whether a certain frame (an “index image frame”), or another frame meeting a similarity criterion with respect to the index image frame, is present in at least a portion of a video item. For example, the index image frame may be a frame which the adaptive system receives at a current time, and the video processing task may be to generate data indicating whether an identical frame (or one meeting a similarity criterion with respect to the index image frame) was present in a portion of the video item, such as an earlier portion of the video item (i.e. the portion of the video item before the index image frame).

[0116] Another possible example of a video processing task is to reconstruct a video item from a compressed representation of the video item.

[0117] Another possible example of a video processing task is prediction of data describing future frames of the video item. For example, the video processing system may process in order the frame representations for an ordered sequence of frames included in the video items, and for each current one of these frames, the video processing task may comprise predicting data about one or more later frames of the sequence (i.e. after the current frame). This task may be performed using the frame representation for the current frame, and optionally the frame representations for respective frames earlier in the ordered sequence than the current frame, but not using the frame representation for frames after the current frame in the ordered sequence. In other words, the video processing task is to predict, given (only) frames included in the video item up to a certain current frame in the video item, data about one or more subsequent frames in the video item. For example, the data may be a prediction of the subsequent frame(s), or a sampled version of the subsequent frame(s), or data describing object depicted in the subsequent frame(s).Attorney Docket No.: 45288-0470W01

[0118] In a further possibility, the video processing task is an agent control task. In this case, the video item comprises observations of successive states of a real-world environment and the output of the video processing system defines physical actions to be performed by the agent in response to the observations (e.g. obtained from sensors capturing the observations from the real world), to perform a task in the environment. The agent can be a mechanical agent in the real-world environment, e.g. a real-world robot interacting with the environment to accomplish a manipulation task, or an autonomous or semi-autonomous land or air or water vehicle navigating through the environment to perform a navigation task. The agent may move in the real-world environment, e.g. translationally (i.e. changing its location in the environment) and / or altering its configuration. The video items in the training database may comprise videos of the task being correctly performed. The actions may comprise control inputs to control a physical behavior of the mechanical agent e.g. in the case of a robot, torques for the joints of the robot or higher-level control commands.

[0119] The possibility of the video processing task comprising controlling an agent may be combined with the concept mentioned above of the video processing task comprising predicting data about future frames of the video item, i.e. such that the commands to control the agent are generated based on the data describing frames of the video item subsequent to the current frame. For example, a camera may be arranged to collect a series of images of a real- world environment where an agent acts (the series of images being the video item), the representation generation system may generate a corresponding frame representation for one or more of the images, and the video processing system may use these frame representations (e.g. as they are generated), and optionally together with data based on frame representations of other frames captured earlier, to generate data describing future evolution of the environment, and this in turn may be used to generate control actions for the agent. For example, the data may indicate that an object depicted in the environment is about to have a certain behavior (e.g. is about to fall over), and the control data may control the agent to act according to this predicted behavior (e.g. position itself to arrest the fall).

[0120] Turning to Fig.10, experimental data is presented from an example implementation of the present techniques. Video representations were generated using an example of the present system shown as “MooG” (“Moving Off-the-Grid”). It was implemented with 35M parameters, for the embedding network, corrector network and predictor network combined.

[0121] Two baselines for the experiment are “Grid”, a simple auto-encoder with high capacity (and no corrector or predictor), and a recurrent on-the-grid baseline, in which a corrector implements a cross-attention between an output of an encoder (i.e. on-the-grid latents) andAttorney Docket No.: 45288-0470W01 corrected latents from a previous step. These systems are “on-the-grid” in the sense that corresponding latents of the frame representation of each frame of a video item have an unchanging correspondence to corresponding portions of the pixel array of each frame of the video item.

[0122] Two other baselines used were DINOv1 using a ViT / B / 16 checkpoint (M. Caron, et al., “Emerging properties in self-supervised vision transformers”, in Proceedings of the IEEE / CVF international conference on computer vision, pp.9650-9660, 2021), and DINOv2 using a Vit- B / 14 checkpoint (M. Oquab, et al., “DINOv2: Learning robust visual features without supervision”, Transactions on Machine Learning Research, 2024).

[0123] Three other baselines used were VideoMAEv2 (L. Wang, et al., “Videomae v2: Scaling video masked autoencoders with dual masking”, in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp 14549-14560, 2023) for three different model sizes (S=20M parameters, B=80M parameters, G=1000M parameters).

[0124] Point tracking (“Points” in Fig. 10) was evaluated by average Jaccard (AJ, where a higher value is better, as shown by the corresponding arrow in Fig. 10). Monocular depth estimation (“Depth” in Fig. 10) was evaluated by absolute relative error (AbsRel). Object tracking (“Boxes” in Fig. 10) evaluated by intersection over union (IoU). Experiments for all three of these were performed using the annotations in the Kubric MOVi-E database (K. Greff et al., “Kubrik: A scalable dataset generator”, in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2022). An experiment for point tracking was performed using the annotations in the Davis database (J. Pont-Tuset, et al., “The 2017 Davis challenge on video object segmentation”, 2017), and an experiment for object tracking was performed using the annotations in the Waymo database (Pei Sun, et al., “Scalability in perception for autonomous driving: WAYMO open dataset”, in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition”, pp 2446-2454, 2020).

[0125] As can be seen, MooG gave the best point tracking for the MOVi-E dataset, and box tracking for the MOVie-E dataset and Waymo database. In depth estimation, and in point tracking using the Davis database, MooG consistently outperformed all the baselines except VMAEv2 (G), but this has about 30 times more parameters than MooG. A better comparison is the VMAEv2 (B) model, which MooG outperformed.

[0126] The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages.

[0127] First, as demonstrated experimentally, the video representations generated in the present technique exhibit the emergent property that portions of the frame representationsAttorney Docket No.: 45288-0470W01 “attach” to surface elements of a scene, and track the movement of those elements over time. For example, in the case of a video item collected by a camera mounted on a vehicle, the frame representations generated after the first few frames of the video item include portions for each of various relatively moving objects in the scene, and so allow information to be obtained about those objects which could not be obtained from a single frame.

[0128] Secondly, the method of generating video representations is very general, so that it does not need to be engineered with a specific application (e.g. a specific downstream video processing task) in mind. Experimentally, it has been demonstrated that it can be used for a wide variety of downstream video processing tasks.

[0129] Thirdly, it has been found that even though the training can be carried out with a video item training database in which each video item is a quite short ordered sequences of frames (e.g. of length 6 to 10 frames), the trained networks can be used for processing much longer sequences (e.g. including hundreds, or thousands of frames) with no added memory or computation requirements and no stability issues.

[0130] Together, these factors mean that using the presently disclosed methods an adaptive system may be trained, within the capacities of present day computer systems, to perform a wide variety of video processing tasks on videos which are 100MBs or larger (e.g. many minutes or even many hours of video).

[0131] Furthermore, the video representations may have reduced size (e.g. number of bits) compared to the video items, so that they may be stored and / or transmitted (e.g. for performance of a subsequent video processing task, including one which may not even have been defined when the video representation is produced, and for which no task performance system has yet been trained) with a low computational burden.

[0132] As noted above, various ones of the networks may employ an attention mechanism. That is, each of the networks may comprise one or more attention neural network layers (processing layer) which each apply an attention mechanism. The one or more attention neural network layers may for example be a stack (sequence) of stacked attention layers which (except for the last layer of the stack) pass their output to be the input to the following attention layer of the stack. Each attention neural network layer may employ any sort of attention mechanism. Generally, an attention mechanism maps a query and a set of key-value pairs to an output, where the query, keys, and values are all vectors. The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function e.g. a dot product or scaled dot product, of the query with the corresponding key. InAttorney Docket No.: 45288-0470W01 implementations, this may be followed by one or more feed-forward neural network layers to generate the attention layer output.

[0133] In general, an attention mechanism determines a relationship between two sequences. A “self”-attention mechanism is configured to relate different positions in the same sequence to determine a transformed version of the sequence as an output. For example, the attention layer input may comprise a vector for each element of the input sequence. These vectors provide an input to the self-attention mechanism and are used by the self-attention mechanism to determine a new representation of the same sequence for the attention layer output, which similarly comprises a vector for each element of the input sequence. An output of the self- attention mechanism may be used as the attention layer output, or it may be processed by one or more feed-forward layers to provide the attention layer output.

[0134] In some implementations the attention mechanism is a self-attention mechanism configured to apply each of a query transformation e.g. defined by a matrix ^^ொ, a key transformation e.g. defined by a matrix ^^^, and a value transformation e.g. defined by a matrix ^^^, to the attention layer input for each element denoted by a vector x of the input sequence X (where X is a matrix in which each row is one of the elements x of the sequence; note that the number of rows of X may be limited to a value N+1, so that the output of the attention mechanism for a given input x may only be based on that input x and the N preceding inputs)to derive a respective query matrix (formed of query vectors) ^^ ൌ ^^^^ொ, a respective keymatrix (formed of key vectors) ^^ ൌ ^^^^^, and a respective value matrix (formed of valuevectors) ^^ ൌ ^^^^^, which are used determine an attended sequence for the output. Forexample the attention mechanism may be a dot product attention mechanism applied by applying each query vector to each key vector to determine respective weights for each value vector, then combining the value vectors using the respective weights to determine the attention layer output for each element of the input sequence. The attention layer output may be scaled by a scaling factor e.g. by the square root of the dimensions of the queries and keys, to implement scaled dot product attention. Thus, for example, an output of the attentionmechanism may be determined as ^^^^^^^^^^^^^^ ^^^^^^√ௗ ^V where d is a dimension of the key (and the query and commonly value) vector. A summation over the value vectors included in V isassumed here, weighted by the respective values ^^^^^^^^^^^^^^ ^^^^^^√ௗ ^.

[0135] An attention mechanism may also be a cross-attention mechanism (rather than a self- attention mechanism), i.e. each attention layer input x is applied to another data matrix YAttorney Docket No.: 45288-0470W01 having rows composed of corresponding vector elements y, the key matrix and value matrix are multiplied by Y rather than X, to give the corresponding matrix of key vectors and value vectors.

[0136] In another implementation the attention mechanism be comprise an “additive attention” mechanism that computes the compatibility function using a feed-forward network with a hidden layer. As previously mentioned, output of the attention mechanism may be further processed by one or more fully-connected, feed forward neural network layers.

[0137] The attention mechanism may implement multi-head attention. That is, it may apply multiple different attention mechanisms in parallel. The outputs of these may then be combined, e.g. concatenated, with a learned linear transformation applied to reduce to the original dimensionality if necessary.

[0138] Examples of suitable architectures for the transformer neural network subsystem are described in Ashish Vaswani et al., “Attention is all you need”, Advances in Neural Information Processing Systems, pp. 5998–6008, 2017; arXiv:1810.04805 Devlin et al. (BERT); and arXiv:1901.02860 Dai et al. (Transformer-XL).In this specification, the term "configured" is used in relation to computing systems and environments, as well as computer program components. A computing system or environment is considered "configured" to perform specific operations or actions when it possesses the necessary software, firmware, hardware, or a combination thereof, enabling it to carry out those operations or actions during operation. For instance, configuring a system might involve installing a software library with specific algorithms, updating firmware with new instructions for handling data, or adding a hardware component for enhanced processing capabilities. Similarly, one or more computer programs are "configured" to perform particular operations or actions when they contain instructions that, upon execution by a computing device or hardware, cause the device to perform those intended operations or actions.

[0139] The embodiments and functional operations described in this specification can be implemented in various forms, including digital electronic circuitry, software, firmware, computer hardware (encompassing the disclosed structures and their structural equivalents), or any combination thereof. The subject matter can be realized as one or more computer programs, essentially modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by or to control the operation of a computing device or hardware. The storage medium can be a storage device such as a hard drive or solid-state drive (SSD), a storage medium, a random or serial access memory device, or a combination of these. Additionally or alternatively, the program instructions can be encoded on a transmitted signal,Attorney Docket No.: 45288-0470W01 such as a machine-generated electrical, optical, or electromagnetic signal, designed to carry information for transmission to a receiving device or system for execution by a computing device or hardware. Furthermore, implementations may leverage emerging technologies like quantum computing or neuromorphic computing for specific applications, and may be deployed in distributed or cloud-based environments where components reside on different machines or within a cloud infrastructure.

[0140] The term "computing device or hardware" refers to the physical components involved in data processing and encompasses all types of devices and machines used for this purpose. Examples include processors or processing units,, graphics processing units (GPUs), tensor processing units (TPUs), and specialized processing hardware such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). In addition to hardware, a computing device or hardware may also include code that creates an execution environment for computer programs. This code can take the form of processor firmware, a protocol stack, a database management system, an operating system, or a combination of these elements. Embodiments may particularly benefit from utilizing the parallel processing capabilities of GPUs, in a General-Purpose computing on Graphics Processing Units (GPGPU) context, where code specifically designed for GPU execution, often called kernels or shaders, is employed. Similarly, TPUs excel at running optimized tensor operations crucial for many machine learning algorithms. By leveraging these accelerators and their specialized programming models, the system can achieve significant speedups and efficiency gains for tasks involving artificial intelligence and machine learning, particularly in areas such as computer vision, natural language processing, and robotics.

[0141] A computer program, also referred to as software, an application, a module, a script, code, or simply a program, can be written in any programming language, including compiled or interpreted languages, and declarative or procedural languages. It can be deployed in various forms, such as a standalone program, a module, a component, a subroutine, or any other unit suitable for use within a computing environment. A program may or may not correspond to a single file in a file system and can be stored in various ways. This includes being embedded within a file containing other programs or data (e.g., scripts within a markup language document), residing in a dedicated file, or distributed across multiple coordinated files (e.g., files storing modules, subprograms, or code segments). A computer program can be executed on a single computer or across multiple computers, whether located at a single site or distributed across multiple sites and interconnected through a data communication network. The specific implementation of the computer programs may involve a combinationAttorney Docket No.: 45288-0470W01 of traditional programming languages and specialized languages or libraries designed for GPGPU programming or TPU utilization, depending on the chosen hardware platform and desired performance characteristics.

[0142] In this specification, the term "engine" broadly refers to a software-based system, subsystem, or process designed to perform one or more specific functions. An engine is typically implemented as one or more software modules or components installed on one or more computers, which can be located at a single site or distributed across multiple locations. In some instances, one or more dedicated computers may be used for a particular engine, while in other cases, multiple engines may operate concurrently on the same one or more computers. Examples of engine functions within the context of AI and machine learning could include data pre-processing and cleaning, feature engineering and extraction, model training and optimization, inference and prediction generation, and post-processing of results. The specific design and implementation of engines will depend on the overall architecture and the distribution of computational tasks across various hardware components, including CPUs, GPUs, TPUs, and other specialized processors.

[0143] The processes and logic flows described in this specification can be executed by one or more programmable computers running one or more computer programs to perform functions by operating on input data and generating output. Graphics processing units (GPUs) and tensor processing units (TPUs) can be utilized to enable concurrent execution of aspects of these processes and logic flows, significantly accelerating performance. This approach offers significant advantages for computationally intensive tasks often found in AI and machine learning applications, such as matrix multiplications, convolutions, and other operations that exhibit a high degree of parallelism. By leveraging the parallel processing capabilities of GPUs and TPUs, significant speedups and efficiency gains compared to relying solely on CPUs can be achieved. These processes and logic flows can be implemented using specialized processing hardware, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs), for even greater performance or energy efficiency in specific use cases.

[0144] Computers capable of executing a computer program can be based on general-purpose microprocessors, special-purpose microprocessors, or a combination of both. They can utilize any type of central processing unit (CPU) graphics processing units (GPUs), tensor processing units (TPUs), and other machine learning accelerators. GPUs, TPUs, and other machine learning accelerators may be employed to enhance performance, particularly for tasks involving artificial intelligence and machine learning. These accelerators may work in conjunction with CPUs, handling specialized computations while the CPU manages overallAttorney Docket No.: 45288-0470W01 system operations and other tasks. The specific configuration of processing units and memory will depend on factors like the complexity of the AI model, the volume of data being processed, and the desired performance and latency requirements. Embodiments can be implemented on a wide range of computing platforms, from small embedded devices with limited resources to large-scale data center systems with high-performance computing capabilities. The system may include storage devices like hard drives, SSDs, or flash memory for persistent data storage.

[0145] Computer-readable media suitable for storing computer program instructions and data encompass all forms of non-volatile memory, media, and memory devices. Examples include semiconductor memory devices such as read-only memory (ROM), solid-state drives (SSDs), and flash memory devices; hard disk drives (HDDs); optical media; and optical discs such as CDs, DVDs, and Blu-ray discs. The specific type of computer-readable media used will depend on factors such as the size of the data, access speed requirements, cost considerations, and the desired level of portability or permanence.

[0146] To facilitate user interaction, embodiments of the subject matter described in this specification can be implemented on a computing device equipped with a display device, such as a liquid crystal display (LCD) or an organic light-emitting diode (OLED) display, for presenting information to the user. Input can be provided by the user through various means, including a keyboard), touchscreens, voice commands, gesture recognition, or other input modalities depending on the specific device and application. Additional input methods can include acoustic, speech, or tactile input, while feedback to the user can take the form of visual, auditory, or tactile feedback. Furthermore, computers can interact with users by exchanging documents with a user's device or application. This can involve sending web content or data in response to requests or sending and receiving text messages or other forms of messages through mobile devices or messaging platforms. The selection of input and output modalities will depend on the specific application and the desired form of user interaction.

[0147] Machine learning models can be implemented and deployed using machine learning frameworks, such as TensorFlow or JAX. These frameworks offer comprehensive tools and libraries that facilitate the development, training, and deployment of machine learning models.

[0148] Embodiments of the subject matter described in this specification can be implemented within a computing system comprising one or more components, depending on the specific application and requirements. These may include a back-end component, such as a back-end server or cloud-based infrastructure; an optional middleware component, such as a middleware server or application programming interface (API), to facilitate communication and dataAttorney Docket No.: 45288-0470W01 exchange; and a front-end component, such as a client device with a user interface, a web browser, or an app, through which a user can interact with the implemented subject matter. For instance, the described functionality could be implemented solely on a client device (e.g., for on-device machine learning) or deployed as a combination of front-end and back-end components for more complex applications. These components, when present, can be interconnected using any form or medium of digital data communication, such as a communication network like a local area network (LAN) or a wide area network (WAN) including the Internet. The specific system architecture and choice of components will depend on factors such as the scale of the application, the need for real-time processing, data security requirements, and the desired user experience.

[0149] The computing system can include clients and servers that may be geographically separated and interact through a communication network. The specific type of network, such as a local area network (LAN), a wide area network (WAN), or the Internet, will depend on the reach and scale of the application. The client-server relationship is established through computer programs running on the respective computers and designed to communicate with each other using appropriate protocols. These protocols may include HTTP, TCP / IP, or other specialized protocols depending on the nature of the data being exchanged and the security requirements of the system. In certain embodiments, a server transmits data or instructions to a user's device, such as a computer, smartphone, or tablet, acting as a client. The client device can then process the received information, display results to the user, and potentially send data or feedback back to the server for further processing or storage. This allows for dynamic interactions between the user and the system, enabling a wide range of applications and functionalities.

[0150] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.Attorney Docket No.: 45288-0470W01

[0151] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0152] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0153] What is claimed is:

Claims

Attorney Docket No.: 45288-0470W01 CLAIMS 1. A computer-implemented method of processing a video item comprising an ordered sequence of a plurality of successive frames to generate a representation of the video item, the method comprising, for successive current frames of the video item, receiving, by a predictor network, a frame representation which, except when the current frame is the first frame of the video item, is a frame representation of the preceding frame of the ordered sequence, processing, by the predictor network, the received frame representation to obtain a preliminary representation of the current frame of the video item; and processing, by a corrector network conditioned on an embedding of the current frame of the video item, the preliminary representation of the current frame of the sequence to generate a frame representation for the current frame.

2. A computer-implemented method of training a representation generation system comprising a predictor network and a corrector network, using one or more video items, each video item comprising a corresponding plurality of successive frames, the method comprising repeatedly: obtaining a cost term for each current frame of an ordered sequence of frames included in one of the video items by: receiving, by the predictor network, a frame representation which, except when the current frame is the first frame of the video item, is based on a preceding frame of the ordered sequence, processing the received frame representation by the predictor network to obtain a preliminary representation of a current frame of the sequence, processing the preliminary representation of the current frame by a decoder network to obtain a decoded representation indicative of a prediction for the current frame; updating the preliminary representation of the current frame by the corrector network conditioned on an embedding of the current frame, to generate a frame representation for the current frame; and forming a cost term for the current frame indicative of a difference between the current frame and prediction for the current frame indicated by the decoded representation; andAttorney Docket No.: 45288-0470W01 modifying at least one of the predictor network, the corrector network and the decoder network to reduce a loss function including the cost term for at least one frame of the sequence of frames.

3. The computer-implemented method of claim 2 in which the decoded representation is a prediction the current frame at a plurality of positions in the current frame which are collectively a subset of the current frame, and the cost term is evaluated over as a difference between the decoded representation and the current frame evaluated at the plurality of positions.

4. The computer-implemented method of any preceding claim, in which the predictor network is configured to apply an attention mechanism to the received frame representation to generate an attention output.

5. The computer-implemented method of claim 4, in which the predictor network obtains the preliminary representation for the current frame as a sum of the received frame representation and the output of the attention mechanism.

6. The computer-implemented method of any preceding claim, in which the corrector network is configured to apply a cross-attention mechanism to the preliminary representation of the current frame, and an embedding of the current frame, to generate an attention output.

7. The computer-implemented method of claim 6, in which the corrector network obtains the frame representation for the current frame as a sum of the preliminary representation and the attention output of the cross-attention mechanism.

8. The computer-implemented method of any preceding claim further including obtaining the embedding of the current frame, the obtaining of the embedding comprising spatially sub-sampling the current frame, the embedding of the current frame having one or more pixel values for each location of a two-dimensional array of locations corresponding to respective locations in the sub-sampled current frame.

9. The computer-implemented method of claim 2, or any of claims 3-8 when dependent on claim 2, in which the decoder network is configured to apply a cross-attention mechanismAttorney Docket No.: 45288-0470W01 to the preliminary representation based on queries corresponding to locations in the embedding of the current frame to generate the decoded representation.

10. A computer-implemented method of processing a video item comprising a plurality of successive frames, to generate a network output indicating a result of performing a video processing task on the video item, the method comprising: processing an ordered sequence of frames included in the video item using a predictor network and a corrector network obtained by the method of claim 2, or any of claims 3-9 when dependent on claim 2, to obtain a representation of the video item comprising a frame representation of each of the ordered sequence of frames; and processing the representation of the video item using a trained task performance system to obtain the network output.

11. The computer-implemented method of claim 8 in which processing the representation of the video item comprises, for each current frame of an ordered sequence of frames included the video item: (i) receiving a dataset for the current frame which, unless the current frame is the first frame of the ordered sequence, is based on a preceding frame of the sequence, (iii) updating the received dataset of the current frame using a task corrector network of the task performance system conditioned at least on the frame representation of the current frame, to generate a updated dataset for the current frame of the sequence, and (iv) processing the updated dataset representation using a task decoder network of the task performance system to generate to obtain an output for the current frame; and generating the network output based on the outputs for the ordered sequence of current frames.

12. A computer-implemented method of processing a video item comprising a plurality of successive frames, to generate a network output indicating a result of performing a video processing task on the video item, the method comprising: processing a video item comprising ordered sequence of a plurality of frames by the method of claim 1, or any of claims 4-8 when dependent on claim 1, to obtain a representation of the video item comprising a frame representation of each of the sequence of frames; andAttorney Docket No.: 45288-0470W01 processing the at least one of the preliminary representation and the frame representation corresponding to each frame of the video item using a trained task performance system to obtain a network output.

13. The computer-implemented method of claim 12 in which processing the preliminary representation and the frame representation corresponding to each frame of the video item comprises, for each current frame of an ordered sequence of frames included the video item: (i) receiving a dataset for the current frame which, unless the current frame is the first frame of the ordered sequence, is based on a preceding frame of the sequence, (iii) updating the received dataset of the current frame using a task corrector network of the task performance system conditioned at least on the preliminary representation and / or frame representation of the current frame, to generate an updated dataset for the current frame of the sequence, and (iv) processing the updated dataset representation using a task decoder network of the task performance system to generate to obtain a network output for the current frame.

14. The computer-implemented method of claim 11 or 12 in which the task performance system further includes a task predictor network configured to receive the updated dataset for each current frame and to modify the updated dataset to form a dataset which, unless the current frame is the last frame of the ordered sequence, is the dataset received for the next frame of the ordered sequence.

15. A computer-implemented method of training a task performance system to perform a video processing task, the method comprising: processing at least one video item by the method of claim 10 or claim 11 to generate a network output, and modifying the task performance system to reduce a measure of a difference between the network output and a desired network output for the video item.

16. The computer-implemented method of any of claims 10 to 15 in which the video processing task comprises identifying a corresponding position in each of a plurality of frames of a video item of one or more elements depicted in the video item.Attorney Docket No.: 45288-0470W01 17. The computer-implemented method of claim 16 when dependent upon claim 11 or 12, in which the dataset comprises respective portions for each of the elements, and the task corrector network processes the portions of the preliminary updated dataset separately to generate respective portions of the corrected updated dataset.

18. The computer-implemented method of claim 17 when dependent upon claim 14, in which the task predictor network processes the portions separately to generate respective portions of the dataset which, unless the current frame is the last frame of the ordered sequence, is received for the next frame of the ordered sequence.

19. The computer-implemented method of any of claims 17 to 18, in which the elements comprises points on objects depicted in the video item.

20. The computer-implemented method of any of claims 17 to 19, in which the elements comprise bounding boxes for one or more objects depicted in the video item.

21. The computer-implemented method of any of claims 10 to 20 in which the video processing task comprises identifying a corresponding depth distance for each of one or more groups of pixels in a plurality of frames in the video item.

22. The computer-implemented method of any of claims 10 to 21, in which the video processing task comprises generating, based on a compressed representation corresponding to a video item, one or more labels indicative of content of the video item.

23. The computer-implemented method of claim 22 in which the labels comprise labels which are associated with one or more frames which are a sub-set of the frames of the video item, and / or labels which are associated with a sub-set of the pixels in one or more of the frames of the video item.

24. The computer-implemented method of any of claims 10 to 23, in which the video processing task comprises receiving an index image frame, generating output data indicative of whether the index image frame, or another frame meeting a similarity criterion with respect to the index image frame, is present in at least a portion of a video item.Attorney Docket No.: 45288-0470W01 25. The computer-implemented method of any of claims 10 to 24, in which the video processing task comprises reconstructing a video item from a compressed representation of the video item.

26. The computer-implemented method of any of claims 10 to 25, in which the video processing task comprises predicting, based on one or more of the frame representations for frames of the ordered sequence which not later in the ordered sequence than a current frame of the ordered sequence of frames, data describing one or more frames of the video item subsequent to the current frame of the ordered sequence of frames.

27. The computer-implemented method of any of claims 10 to 26, in which the video processing task comprises generating commands to control an agent to perform a task in an environment depicted in the video item.

28. The computer-implemented method of claim 27 when dependent upon claim 26, in which the commands to control the agent are generated based on the data describing one or more frames of the video item subsequent to the current frame of the ordered sequence of frames.

29. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform the method of any preceding claim.

30. One or more computer storage media storing instructions that when executed by one or more computers cause the one or more computers to implement the system of claim 29.

Images