Training a media item encoder

Abstract

A method is proposed for training a media item encoder neural network. The media item encoder neural network is configured to receive a media item and to generate an output comprising feature data which is an encoding of the media item. The method comprises jointly training the media item encoder neural network and a decoder neural network. The decoder neural network is arranged to receive feature data output by the media item encoder neural network, and to generate text tokens based on the received feature data. The training is based on one or more training examples which each comprise a media item and a corresponding text token string. The corresponding text token strings of one or more of the training examples comprise first text tokens describing an object present in the corresponding media item, and second text tokens defining a position of the object in the media item.

Description

Attorney Docket No.45288-0442WO1 TRAINING A MEDIA ITEM ENCODER CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Application No. 63 / 562,630, filed on March 7, 2024. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application. BACKGROUND

[0002] This specification relates to processing data using machine learning models.

[0003] Machine learning models receive an input and generate an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate the output based on the received input and on values of the parameters of the model.

[0004] Some machine learning models are deep models that employ multiple layers of models to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a non- linear transformation to a received input to generate an output. SUMMARY

[0005] This specification describes a system, implemented as computer programs on one or more computers in one or more locations, and a method to train a media item encoder neural network (“media item encoder”) as part of an adaptive system which also incorporates a decoder neural network (“decoder”). The media item encoder may be configured to process a media item to generate feature data. The decoder is configured to generate text (i.e., a string of text tokens) based on feature data generated by the encoder, and also based on text tokens received by the decoder.

[0006] The media item may comprise at least one image (e.g., a still image, or a moving image which is a sequence of still images; that is one or more intensity values for each pixel of one or more multi-dimensional, e.g., two-dimensional, pixel arrays). The image may be one captured from the real-world by a camera device or other imaging device. In this case, the media item encoder is a visual encoder. Alternatively, the media item encoder may be an audio encoder, and the media item may be a sound signal (i.e., comprising amplitude values in the time or frequency domain, and describing sound during a time period). The media item encoder neuralAttorney Docket No.45288-0442WO1 network may also be configured to encode media items which include both at least one image and at least one sound signal. Alternatively or additionally, the media item may comprise the output of a sensor of any type which collects measurements of the real world periodically, e.g. a temperature sensor or humidity sensor.

[0007] In general terms, the disclosure proposes that the encoder and decoder are trained together (i.e., jointly), based on training examples including media items (e.g., images / sound signals) and corresponding text token strings. The text token strings comprise first text tokens describing an object in the corresponding media item. The text token strings further comprise second text tokens defining a position (“location”) of the object in the corresponding media item.

[0008] In the case that the media item comprises at least one image, and the encoder is a visual encoder, an object “in” the image means an object depicted in at least a part of the image. The “position” may specify a (proper) subset of the pixels in the pixel array(s) of the image which depict the object. For example, the position may be a bounding box which is a shape (e.g., a rectangle) which is a subset of the image such that pixels of the image outside the bounding box do not depict the object, and at least some of the pixels within the bounding box do.

[0009] In the case that image is a moving image, the bounding box may be defined in at least one frame of the moving image, but not necessarily in all frames of the moving image. Note that for a frame which does not depict the object, the second text token string may not define a bounding box. The bounding box may be different in different frames, e.g., if the object is in a different position in different ones of the frames.

[0010] In the case, that the media item comprises at least one sound signal, and the encoder is an audio encoder, the “position” may specify a sub-section of the time period spanned by the sound signal during which an object (e.g., a certain sound, such as the voice of a particular individual or a noise produced by a certain device) is present.

[0011] In this way, the media item encoder (e.g., visual encoder) learns to generate a location-aware output. The trained adaptive system may itself be deployed for an inference application. Alternatively, the trained media item encoder may be subsequently employed in an adaptive system which is trained to perform another computing task (“media item processing task”), such as a location-aware inference task, e.g., a location-aware image inference task which requires determination of locations in input images.Attorney Docket No.45288-0442WO1

[0012] The text tokens of the text token string may be selected from a vocabulary. The text tokens of the vocabulary may comprise letters, words and / or portions of words. The first text tokens may be natural language describing the object depicted in the corresponding media item. The second text tokens may comprise text tokens of natural language, such as English (i.e., describing a position of the object according to the grammar of a natural language, such as English). Alternatively, the second text tokens may comprise numeric values, defining the position of the object numerically, e.g., by defining a bounding box of the object in the corresponding image, or defining a time sub-period within a sound signal).

[0013] For example, in the case of a media item which is a still image the text token string may read “A puffin standing in the front [0.0, 1.0, 0.3, 0.4]”, where “a puffin standing” are first text tokens describing an object. “In the front [0.0, 0.1, 0.3, 0.4]” is composed of second text tokens, including second text tokens (“in the front”) in natural language, and second text tokens which are numeric values (“0.0, 0.1, 0.3, 0.4”) referencing a non-verbal identification of the position of the object, e.g., in a format identified by label characters (the square brackets) of the text token string, which may be neither numerical nor in natural language. The values 0.0 and 0.1 may specify the position in the image of one corner of a rectangular bounding box with horizontal and vertical sides, and the values 0.3 and 0.4 may specify the position in the image of the opposite corner.

[0014] In the case of a media item which is a moving image (i.e. a video), the text tokens string may read “A puffin appears at the end of the video in the foreground [90, 100, 0.1, 0.1, 0.3, 0.4]”, where the second text tokens are “at the end of the video in the foreground [90, 100, 0.1, 0.1, 0.3, 0.4]”. The second text tokens include numerical values [.9, 1.0, 0.1, 0.1, 0.3, 0.4] which specify a range of the duration of the video (i.e. the period from 90% to 100% of the length of the video) in which the puffin is in the foreground, and which specify the foreground position numerically (i.e. in a bounding box with corners at locations (0.1, 0.1) and (0.3, 0.4)).

[0015] In the case of a media item which is a sound signal, the text token string may read “a dog barks near the end of the sound signal”, where “a dog barks” are first text tokens, and the second text tokens are the words “near the end of the sound signal”. The second tokens may alternatively or additionally including numerical tokens, e.g. [0.9, 1.0], specifying the sub- period from 90% to 100% of the time period of the sound signal.Attorney Docket No.45288-0442WO1

[0016] The training examples may be provided in at least one training database. Optionally, some or all of the training examples may include first and second text tokens for multiple objects in the media item. For example, the text token string may read “A puffin standing in front and another puffin sleeping in the back”. Here “a puffin standing” and “a puffin sleeping” are first text tokens, and “in front” and “in the back” are second text tokens.

[0017] The training may be performed by iterative (repeated) updates, performed by a training engine, to numerical parameters defining the encoder and numerical parameters defining the decoder. Joint training of the encoder and decoder means that, in the training, successive updates to the corresponding numerical parameters defining the encoder and decoder are either substantially simultaneous or else interleaved.

[0018] The decoder may be configured to generate an output comprising (or consisting of) text tokens selected from a vocabulary of text tokens which is the same or different from the text token vocabulary of the text token string. The decoder may be configured to generate a distribution (e.g., a probability distribution) over each selection from the vocabulary. Optionally, it may include a unit for selecting one of the tokens from the vocabulary based on the distribution, e.g., as the token for which the distribution is maximal, or by treating the distribution as a probability distribution and selecting a token from the vocabulary randomly according to the probability distribution.

[0019] The decoder may be an auto-regressive decoder. That is, for each token of an output of the decoder, the decoder generates data defining a token, or a defining a distribution over the options for the token, based on corresponding data it has previously generated for any previous token(s) of the output. For example, the decoder may generate, based on any previously generated tokens of the output, a distribution for a current text token over the vocabulary. A current text token of the output may be selected based on the distribution. As described below, the distribution may be used to define an update of the training process, e.g., without necessarily using it to make a selection of a current text token.

[0020] The decoder may be one employing an attention mechanism, such as a transformer. It may include multiple processing layers arranged in a sequence, in which each processing layer except the first receives the output of the preceding layer. The output of the last processing layer may be the output of the decoder.Attorney Docket No.45288-0442WO1

[0021] For the decoder to generate a given “current” token of an textual output which responds to a training example, the first layer receives (i) media item features (e.g., visual features) generated by the encoder from the media item of the training example, and (ii) optionally earlier tokens of the textual output, e.g. token(s) of the textual output which the decoder has previously generated. As discussed below, the first layer may also receive data (a “task prefix”) which indicates a class of the training example. Each processing layer may apply a cross-attention mechanism to the input it receives, to generate an output of the processing layer. The output of the last processing layer may be an output of the decoder.

[0022] The training may be based on the adaptive system predicting, given the image and an initial portion of the text token string, a later part of the text token string, e.g., the token of the text token string immediately following the initial portion, or the rest of the text token string after the initial portion. More particularly, a given update of the training (i.e., an update to either or both of the parameters defined the encoder or the parameters defining the decoder) may be based on a success function which is a function of output(s) of the decoder, which in turn depends on the parameters, e.g., collectively denoted θ, of the encoder and decoder. The success function may be a sum over a plurality of the training examples (e.g., a training database of training examples), of a calculated measure (“success term”) of a part of the output of the decoder and a portion of the text token string of the training example. For example, the success term may be a measure of the similarity (according to a similarity metric) between an output of the decoder (based on the output of the encoder upon processing the media item of the training example, and based on an initial portion of the corresponding text token string) and a later portion of the text token string (e.g., the next token of the string, or a plurality of the next tokens, or the rest of the string). The success function may be calculated as a sum of the success terms over the training examples, and / or, for each training example, a sum over the possible choices of the initial portion of the text token string. A given update may be calculated by approximating the value of the success function for a “batch” (i.e., a subset) of the training examples of the training database.

[0023] As noted above, the decoder model may be one which generates a distribution, e.g., a probability distribution, over the possibilities for the next text token of the string. In this case, the success function may include a success term which is a function of the value, under the distribution, of the token of the text token string immediately following the initial portion.Attorney Docket No.45288-0442WO1 For example, the media item (e.g., image) for a given one of the training examples may be denoted as x, and the text token string for the training example may be denoted as {yj}, for j=1,…n, where n is the number of text tokens in the text token string of the training example. Labelling the current text token of the output of the decoder an integer index i, which is in the range 1 to n (that is, the “current text token”, i.e. a text token which the decoder generates by processing the feature data generated by the encoder using the media item of the training example, and an initial portion of the training example which consists of i-1 characters), the one or more tokens of the initial portion of the text token string may be denoted y<i , and the success term may be based on P(yi|y<i, x), e.g., this may be the probability of the decoder selecting, given the corresponding media item x and the initial portion of the text token string, text token yias the text token which follows the initial portion of the text token string.

[0024] The success term may be a function of the probability P(yi|y<i, x), such as its logarithm.

[0025] Optionally, the success function may be calculated as a sum of a respective success term over each possible value of i, or over a range of possible values of i.

[0026] Furthermore, the success function may be summed over a plurality of the training examples (the batch).

[0027] In other words, a given update may be a change to θ which increases a sum, over the training examples (or a batch of one or more thereof), and over one or more (e.g., all) of the values of i, of log P(yi|y<i, x).

[0028] Note that the success function may also include a term for the case of predicting y1 given just the media item x (i.e., predicting the first term of the text token string from just x), i.e., log P(y1|x).

[0029] The training examples used in the training process may not all have the same format. Instead, they may be examples of a corresponding one of a plurality of respective training tasks (i.e., tasks for the adaptive system comprising the encoder and decoder) associated with corresponding training example formats. In an implementation, there are multiple training examples for each of the corresponding training tasks. Thus, each training task amounts to predicting the text token string of any one of the corresponding training examples, given the corresponding media item (e.g., image); and in particular, to predicting later portionsAttorney Docket No.45288-0442WO1 of the text token string given an initial portion of the text token string of the corresponding training example and the corresponding image.

[0030] For a given training example, the input to the decoder may further comprise data identifying the corresponding training task. This may be a separate input to the decoder. Alternatively, in an implementation, the data may be a “task prefix” included as one or more text tokens of the input to the decoder, e.g., an initial portion of the text token string. The text tokens of the task prefix may not be used in the calculation of the success function.

[0031] The training tasks may comprise a task in which the first text tokens precede (at least some of) the second text tokens. Thus, the task is a “referring task”, which includes (e.g., at later stages of the predicting the text token string) predicting the second text tokens (e.g., as numerical values), given the first text tokens and the media item x.

[0032] The training tasks may further comprise a task in which (at least some of) the second text tokens precede the first text tokens. Thus, the text is a “grounded caption task” which includes (e.g., at later stages of the predicting the text token string) predicting the first text tokens as a “caption”, given the second text tokens (e.g., as numerical values) and the media item x.

[0033] The training tasks may include predicting text token strings which are entirely in natural language, e.g., according to the grammar of a natural language. In this case, in particular, the text token string may describe multiple objects and their corresponding positions, e.g., in relation to another of the objects in an image or a foreground / background of the image.

[0034] The training examples may be collected from a repository of documents containing text and associated media items (e.g., images), e.g., the internet. Objects in the media item (e.g. image) may be detected and recognized automatically. Text for each object is also obtained, so that an object-text tuple is formed. Localization data for each detected object (e.g., a bounding box) may also be obtained automatically. This may provide dense regional annotations. A confidence test may optionally be applied to determine whether a confidence in the accuracy of the identification of the object, and / or in the accuracy of the localization data is met. The object identification, the localization data and / or the corresponding text may then be used to define the text token string of a training example. For example, a corresponding training example may be defined for one or more of (e.g., each of) the training tasks.Attorney Docket No.45288-0442WO1

[0035] At least one of the encoder and decoder, and in an implementation both, and indeed the adaptive system as a whole, may include no parameters which were derived from a previous training process (such as one in which at least a non-zero proportion of the training examples included both first text tokens describing an object depicted in the corresponding image and second text tokens defining a position (“location”) in the corresponding image of the object). In other words, in an implementation of the present technique, the training may be “from scratch”, rather than fine-tuning of either or both of the models.

[0036] An adaptive system comprising the encoder and decoder trained as explained above, may be deployed (i.e., used) as a neural network model for performing a media item processing task (e.g., an image processing task) which is one of the training tasks (e.g., a different training task from those described above).

[0037] Alternatively, at least the trained encoder may be used as part of a neural network model to perform a different media item processing task (a “downstream task”), e.g., an image processing task. To perform the media item processing task, the neural network model may receive data comprising a media item. The media item is processed by the trained encoder to generate feature data (an embedding), and further data (e.g., a series of text tokens) defining a token string may be received and processed by a decoder (e.g., a language decoder) of the neural network model. The decoder of the neural network model may also process the embedding of the media item generated by the trained encoder. The output of the neural network model may be the output of the decoder, or an output which is generated based on the output of the decoder by an output unit (e.g., if the output of the decoder is a distribution over a vocabulary of tokens, the output unit may select one of the tokens based on the distribution).

[0038] In either case, the neural network model may operate as a “multi-modal machine learning model”, since in the media item processing task it may process input data having two modalities: the media item (which is generally not composed of text tokens), and optionally the token string. The token string may be defined by a user. Similarly, the output may comprise one or both of a media item and text tokens, e.g., an image annotated by location data such as bounding boxes and text (e.g., for each bounding box).

[0039] At least part of the neural network model may be trained in a “second” training procedure to perform the downstream processing task. Specifically, the part of the neural network model which is trained may include any one or more of (i) the media item encoder, (ii)Attorney Docket No.45288-0442WO1 the decoder and / or (iii) an adapter module of the neural network model (i.e., a module of the neural network model which is addition to the media item encoder neural network and the decoder). The adapter module may for example comprise one or more neural network layers which act as an interface between the media item encoder and the decoder (i.e., process an output of the media item encoder to generate an input of the decoder), and / or one or more neural network layers which process respective outputs of the media item encoder and the decoder, to generate an output of the neural network model.

[0040] The second training procedure may be carried out after the first procedure, and may be performed by the same training engine or a different one. It may be based on “second” training examples of the downstream processing task. The second training procedure may iteratively modify parameters of the neural network model to increase a measure of similarity between desired output data of the neural network model defined by the second training example, and data actually output by the neural network model upon receiving input data defined by the second training example (e.g., a media item and optionally a token string).

[0041] Optionally, the media item encoder may be frozen during the second training procedure. Alternatively, it may be iteratively updated as part of the second training procedure, based on hyper-parameters which are selected to keep the learning rate low (e.g., minimal learning rate and weight decay) to avoid forgetting.

[0042] The downstream task may comprise any one or more of: (i) media item classification (CLS), i.e., assigning an input media item (e.g., image) to one (or more) of set of classes based on content of the media item. (ii) captioning (generating a caption, such as a sequence of tokens selected from a vocabulary) describing the content of the media item; (iii) question answering (e.g., visual question answering (VQA)) of generating an answer which is an appropriate response to a question about the media item defined by the token string received by the decoder of the neural network model; (iv) a segmentation task of identifying a (proper) subset of the media item having specific properties; and (v) an agent control task of generating control data for an agent in an environment (e.g., an electro-mechanical agent such as a robot) based on a media item describing theAttorney Docket No.45288-0442WO1 environment (e.g., received sensor data, such as an image of an environment captured by a camera) and a token string defining an action the robot is instructed to perform.

[0043] In another option, the decoder of the neural network model may be a pre-trained decoder. The second training procedure may comprise fine-tuning the decoder and / or training an adapter module described above, for the downstream task. The second training procedure, using the second training examples, is based on outputs of the pre-trained media item encoder, upon receiving the media items of the second training examples.

[0044] According to a third aspect, there are provided one or more computer storage media, e.g., tangible computer storage media or other computer program products, storing instructions that when executed by one or more computers cause the one or more computers to perform the operations of the respective methods of the first and second aspects.

[0045] According to a fourth aspect, there is provided system comprising one or more computers and one or more storage devices communicatively coupled to the one or more computers. The one or more storage devices may store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of the respective methods of the first and second aspects.

[0046] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] FIG. 1 shows schematically a neural network model comprising a media item encoder network and a decoder network.

[0048] FIG. 2 shows a training system for the neural network model.

[0049] FIG. 3 is composed of FIG. 3(a) which shows an example media item, and FIGS. 3(b)-(d) which show text token strings of three respective training examples which comprise the media item and the respective text token string, and illustrate respective training tasks.

[0050] FIG. 4 is a flow diagram of a method of jointly training a media item encoder and a decoder.Attorney Docket No.45288-0442WO1

[0051] FIG. 5 is a flow diagram of a method of processing a media item to generate an output.

[0052] FIG. 6 is a flow diagram of a method of forming and training a neural network model.

[0053] FIG.7 shows experimental results comparing performance metrics of an example of the present disclosure and three known methods for a COCO (common objects in context) detection task.

[0054] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0055] Fig. 1 shows a neural network model 100 comprising a media item encoder neural network 101 (for conciseness, referred to also as a “media item encoder”) and a decoder neural network 102 (“decoder”).

[0056] The media item encoder 101 is configured to receive a media item 103, and to generate from it an encoding 104 (or “embedding”) of the media item 103 which is referred to as feature data. The encoding 104 comprises information relating to the content of the media item 103, though not necessarily enough information to reconstruct it accurately.

[0057] The media item 103 may be an image, e.g., a still image, or a moving image which is a sequence of still images; that is one or more intensity values for each pixel of one or more multi-dimensional, e.g., two-dimensional, pixel arrays. The image 103 may be one captured from the real world by a camera or other imaging device. The media item encoder 101 may be a visual encoder, which is configured to process pixel-level intensity data in the image 103 to generate the encoding. The feature data 104 encodes content (“objects”) depicted by the image.

[0058] Alternatively, the media item encoder 101 may be an audio encoder, and the media item may be a sound signal (i.e., comprising amplitude values in the time or frequency domain, and describing sound during a period of time), such as one captured from the real world using a microphone. The media item encoder neural network may also be configured to encode media items which include both an image and sound signal. In another possibility, theAttorney Docket No.45288-0442WO1 media item may be the output of a sensor of any type which is configured to sense the real world.

[0059] The media item encoder 101 may comprise a sequence (stack) of processing layers. The first layer receives the media item 103 (or optionally, an embedding of the media item 103 produced by a fixed input unit of the media item encoder 101). Each layer but the first receives the output of the preceding layer in the sequence. The output of the last processing layer may be the output 104 of the encoder. Each processing layer performs an operation defined by corresponding numerical parameters. In particular, at least one of the processing layers may employ an attention mechanism. These processing layer(s) may be implemented as a transformer. In the case that the encoder is a visual encoder, the encoder may include an input unit which divides a received image into image patches. It may comprise a Vision Transformer (ViT) (Dosovitskiy A., et al., “An image is worth 16x16 Words; Transformers For Image Recognition at Scale”, ICLR 2021), such as the Vit-L / 14 design (a ViT model in the “Large” variant, with a 14x14 input patch size).

[0060] The decoder 102 may be configured to receive the feature data 104 output by the media item encoder 101. The decoder 102 is also configured to receive a text token string 105, i.e. a string of text tokens selected from a vocabulary. The text tokens of the vocabulary may comprise letters, words and / or portions of words. At least one of the text tokens in the vocabulary may be a token which indicates a termination of string.

[0061] The neural network model 101 is configured to operate in a sequence of time steps. In each given time step (except optionally a first time step), the decoder network 102 is configured to generate at least one “next text token” 106 which is a token which forms a continuation of the text token string 105, conditioned on the feature data 104, i.e., conditioned on the media item 103.

[0062] At a next time step, the text token string 105 is supplemented by the next text token generated at the previous time step (i.e. the next text token 106 is placed at the end of the text token string 105, as the last text token of the string), and the decoder 102 is used to process the feature data 104 and the updated text token string 104, to generate a new next text token 106. Thus, the decoder 102 is “auto-regressive”. That is, in a sequence of time steps, the decoder 102 generates, at each time step, a corresponding text token 106, based on the feature data 104, on any text tokens of the text token string 105 at the first time step of the sequence ofAttorney Docket No.45288-0442WO1 time steps, and on the text tokens the decoder 102 has generated at any previous time steps of the sequence.

[0063] Optionally, in the first time step of the sequence of time steps, the text token string 105 may be “empty” (e.g. it comprises no text tokens, or contains a default text token). In this case, the text token 106 generated by the decoder neural network may be the first text token of the string. The first text token may be selected based entirely on the feature data 104. Alternatively, in the first time step, the text token string 105 may alternatively comprise one or more text tokens.

[0064] The process may terminate when a termination criterion is reached. For example, the termination criterion may be that the next text token 106 is one which indicates a termination of the text token string. Alternatively, the operation of the decoder 102 may terminate when another termination criterion is met, e.g., that the number of text tokens 106 which have been generated reaches a threshold. Denoting the media item 103 by x, the sequence of text tokens successively generated by the decoder 102 may be denoted y={yj}, for j=1,…n, where n is the number of text tokens 106 generated by the decoder 102 before the termination criterion is met.

[0065] The decoder 102 may optionally include an input encoder for generating an embedding of the text token string 105. It may alternatively or additionally employ a structure comprising a stack of one or more layers in which each layer operates as a transformer, as described in Ashish Vaswani et al., “Attention is all you need”, Advances in Neural Information Processing Systems, pp. 5998–6008, 2017; arXiv:1810.04805 Devlin et al. (BERT); and arXiv:1901.02860 Dai et al. (Transformer-XL), summarized below. A first of the layers applies a cross-attention mechanism to the feature data 104 and the text token string 105 (or an embedding of it formed by the input encoder, if any). Each subsequent one of the layers of the decoder may apply a self-attention mechanism to the output of the preceding layer, or a cross-attention mechanism between the feature data 104 and the output of the preceding layer.

[0066] The decoder 102 may be configured to generate a distribution (“token distribution”) over the possibilities for the next text token 106 of the string (i.e., a corresponding numerical value for each token of the vocabulary), and the decoder 102 may include an output unit which uses the distribution to select a choice for the next token 106. For example, the next token may be chosen as the token for which the corresponding numericalAttorney Docket No.45288-0442WO1 value of the distribution is highest, or the distribution may be a probability distribution and the output unit of the decoder may select the next text token 106 with a probability which is proportional to the corresponding value of the distribution. Note that a special case of this is when the output unit is omitted, but in which the decoder 102 is configured, in a given time step, to output a distribution which indicates a single text token of the vocabulary, e.g. by generating a “one hot” vector having respective components for the text tokens of the vocabulary, and which is 1 for one component and zero for all the other components.

[0067] Referring to FIG. 2, a system 200 is shown for training the neural network model 100 of FIG. 1. The system of FIG. 2 is an example of the present disclosure. It may be implemented as computer programs on one or more computers in one or more locations.

[0068] In addition to the neural network model 100, the training system 200 includes a training engine 201. The training engine is operative to control the media item 103, and the text token string 105. The training engine 201 receives the next text token 106 or the token distribution generated by the decoder neural network 102 based on the media item 103 and the text token string 105.

[0069] The training engine 201 has access to multiple training databases 201a, 201b, 201c, which contain examples (“training examples”) of respective training tasks of different respective classes. Each training example comprises a media item and a corresponding text token string. FIG. 3(a) shows a media item 300 (an image (picture) depicting two puffins) which could be the media item for three training examples which are respectively included in training databases 201a, 201b and 201c.

[0070] In each of a plurality of iterations, the training engine 201 is operative to select a training example from one of the databases 201a, 201b, 201c. The training engine causes the media item of the selected training example to be the media item 103 which the media item encoder 101 receives, and causes an initial portion of the corresponding text token string of the selected training example to be the text token string 105 input to the decoder 102. The initial portion of the corresponding text token string is referred to as a “text token string portion”.

[0071] The encoder 101 processes the media item of the selected training example to generate feature data 104. The decoder neural network 102 processes the feature data 104 and the text token string portion to generate a next text token 106, or a token distribution. The training engine then modifies corresponding numerical parameters defining the media itemAttorney Docket No.45288-0442WO1 encoder 101 and / or the decoder 102, based on the next text token 106 or token distribution. This is to increase the likelihood that the next text token 106 is the text token of the text token string of the selected training example which follows the text token string portion. For example, in the case that the decoder 102 generates a token distribution, the modification may be such to as to increase the numerical value of the distribution for the text token of the text token string of the selected training example which follows the text token string portion, and / or to decrease the numerical value of the distribution for all other text tokens of the vocabulary.

[0072] FIG. 3(b) shows the corresponding text token string for a training example in training database 201a. This training example also comprises the media item 300. The text token string of the training example comprises, e.g. at the start, text tokens (a “task prefix”) identifying a training task corresponding to the training database 201a. The task prefix may be the same for all training examples of the training database 201a. Specifically, in this example, the task prefix 301 for a training example in the training database 201a may be “Cap:” indicating that the corresponding task is to produce a caption for the corresponding media item of the training example. The remainder of the text token string of the training example is a caption which is the result of performing the training task (“A picture of a puffin standing on a cliff edge and another puffin curled up in the back”). Thus, the task prefix 301 is in addition to the caption in natural language.

[0073] The task prefix, and optionally one or more of the first tokens of the remainder of the text token string of the training example, are included in the initial portion of the text token string of the training example (i.e., the text token string portion) which the training engine 201 uses as text token string 105. Thus, the modification made by the training engine 201 to the numerical parameters defining the media item encoder 101 and / or the decoder 102, increases the likelihood that, upon the neural network model 100 receiving a media item 103 which is the media item of the training example of the database 201a, and the decoder neural network 102 receiving a text token string 105 which is the initial portion of the text token string of the training example, the neural network model generates, as the next text token 106, the first token of the remainder of the text token string of the training example. For example, if the initial portion of the text token string is “Cap: A picture of a puffin st”, the modification is such as to increase the likelihood that the next token 106 is “a” (i.e., the letter of the word “standing” after “st”). Note that this training task is predicting text token strings which are entirely inAttorney Docket No.45288-0442WO1 natural language, e.g., according to the grammar of a natural language. The text token string of each training example may describe multiple objects and their corresponding positions, e.g., in relation to another of the objects in an image or a foreground / background of the image.

[0074] FIG. 3(c) shows the corresponding text token string for a training example in training database 201b, i.e., a training example comprising the media item 300. This text token string comprises, e.g., at the start, a task prefix 302 “ARef:” indicating that the corresponding task is an “automatic referring” task, to produce a caption for the corresponding media item of the training example which indicates an object in the image and then numerical data defining a bounding box for the object. The remainder of the text token string is a possible solution to this task, given the image 300. It consists of “first text tokens” 303 which collectively form words which indicate content depicted in the image (“A puffin standing on a cliff edge”) and “second text tokens” 304 which are numerical data (“[20, 480,150,200]”) defining the bounding box for the content. In this case, the first text tokens precede the second text tokens 304. Thus the task prefix 302 is in addition to the first text tokens 303 describing an object present in the corresponding media item, and second text tokens 304 defining a position of the object in the media item numerically. The second text tokens 304 define the position of the object by a plurality of numbers indicating a bounding box of the object within the media item. The second text tokens may use the same format for all the training examples in the training database 201b. Each number may be an integer selected from a corresponding range of integers, where each range labels the rows, or labels the columns of the images.

[0075] The task prefix 302, and optionally one or more of the first tokens of the remainder of the text token string of the training example, are included in the initial portion of the text token string of the training example (i.e., the text token string portion) which the training engine 201 uses as a text token string 105. Thus, the modification made by the training engine 201 to the numerical parameters defining the media item encoder 101 and / or the decoder 102, increases the likelihood that, upon the neural network model 100 receiving a media item 103 which is the media item of the training example of the database 201b, and the decoder neural network 102 receiving a text token string 105 which is the initial portion of the text token string of the training example, the neural network model generates, as the next text token 106, the first token of the remainder of the text token string of the training example. For example, if the initial portion of the text token string is “ARef: A puffin standing on a cliff edgeAttorney Docket No.45288-0442WO1 ” (i.e. the task prefix and all the first text tokens), the modification is such as to increase the likelihood that the next token 106 is “[” (i.e. the first of the second text tokens). Similarly, if the initial portion of the text token string is “ARef: A puffin standing on a cliff edge [20,4”, the modification is such as to increase the likelihood that the next token 106 is “8” (i.e., the second digit of 480).

[0076] Thus, in the case of a training example selected from the database 201b, the modification made by the training engine 201 to the numerical parameters defining media item encoder 101 and / or the decoder 102, trains the neural network model to perform the automatic referring task, of generating a caption of first tokens describing content in the media item, followed by second tokens describing a location (bounding box) in the media item of that content.

[0077] FIG. 3(d) shows the corresponding text token string for a training example in training database 201c, i.e., a training example comprising the media item 300. This text token string comprises, e.g. at the start, a task prefix 305 “GCap:” indicating that the corresponding task is an “grounded captioning” task, to produce a caption for the corresponding media item of the training example which includes numerical data defining a location (bounding box) for an object depicted in the media item and then text which describes an object depicted at that that location. The remainder of the text token string is a possible solution to this task, given the image 300. It consists of “second text tokens” 306 which are numerical data (“[400, 110,460,40]”) 304 defining the bounding box, and “first text tokens” 307 which collectively form words which indicate content depicted in the defined bounding pox of the image (“Another puffin curled up in the back”). In this case, the second text tokens 306 precede the first text tokens 307. The second text tokens 306 may be in the same format at the second text tokens 304 of the training example from the database 201b depicted in Fig. 3(c).

[0078] The task prefix 305 (“GCap”), and optionally one or more of the first tokens of the remainder of the text token string of the training example, are included in the initial portion of the text token string of the training example (i.e. the text token string portion) which the training engine 201 uses as a text token string 105. Thus, the modification made by the training engine 201 to the numerical parameters defining the media item encoder 101 and / or the decoder 102, increases the likelihood that, upon the neural network model 100 receiving a media item 103 which is the media item of the training example of the database 201b, and the decoderAttorney Docket No.45288-0442WO1 neural network 102 receiving a text token string 105 which is the initial portion of the text token string of the training example, the neural network model generates, as the next text token 106, the first token of the remainder of the text token string of the training example. For example, if the initial portion of the text token string is “GCap: [400,110,460,40] ” (i.e. the task prefix 305 and all the second text tokens 306), the modification is such as to increase the likelihood that the next token 106 is “A” (i.e. the first of the first text tokens). Similarly, if the initial portion of the text token string is “GCap: [400,110,460,40] Another pu”, the modification is such as to increase the likelihood that the next token 106 is “f” (i.e., the third character of the word “puffin”).

[0079] Thus, in the case of a training example selected from the database 201c, the modification made by the training engine 201 to the numerical parameters defining media item encoder 101 and / or the decoder 102, trains the neural network model to perform the grounded captioning task, of generating a caption of second tokens describing a location (bounding box) in the media item, followed by first tokens describing content in that location of the media item.

[0080] The text tokens string of each training example may also include, e.g., as a final token, a token indicating a termination of the text token string. In one case, though not shown in Fig. 3, a period symbol (e.g., “.”) may function as the termination token. Alternatively, another character may be used.

[0081] The training examples in databases 201a, 201b, 201c, may be collected from a repository of documents containing text and associated media items (e.g., images), e.g., the internet.

[0082] Turning to FIG. 4, a method 400 is shown which is an example of the present disclosure. The method can be carried out by the training system 200 of Fig. 2, to train the neural network model 100 of Fig. 1. It may be implemented as computer programs on one or more computers in one or more locations. The method 400 comprises a set of steps (401-405) which are carried out repeatedly until at least one termination criterion is met, e.g., that the set of steps has been carried out a certain number of times, and / or that, in one of the set of steps, a magnitude of the update made to the neural network model 100 (as measured by a metric of the update amount) is below a threshold.

[0083] In a first step the training engine 201 selects a training example from one of the databases 201a, 201b, 201c.Attorney Docket No.45288-0442WO1

[0084] The training engine 201 uses the text token string of the selected training example to define (step 402) an initial portion of the text token string. This may be any number of text tokens of the text token string, starting from the first text token of the text token string. The number of text tokens included may be selected at random, e.g., as a number in the range from zero to the total number of text tokens in the text token string of the selected training example. Alternatively, the initial portion of the text token string may always comprise the text prefix of the text token string of the selected training example, i.e., it is a number of characters which is at least the number of characters of the task prefix, and which is a (e.g. randomly chosen) number of characters from the remainder of the text token string, e.g. as a number in the range from zero to the total number of text tokens in the remainder of the text token string of the selected training example. Alternatively, the initial portion of the text token string may always comprise all of the first text tokens (in the case of a training example from database 201a) or all of the second text tokens (in the case of a training example from database 201b).

[0085] The training engine 201 transmits the media item of the selected training example to be an input of the media item encoder 101, which processes it to generate feature data 104. The decoder 102 receives the feature data 104 output by media item encoder (step 403).

[0086] The decoder 102 generates an output (step 404) by processing the feature data output by the encoder 101 and the initial portion of the text token string defined in step 402. This output may be data specifying a single next text token, or a distribution over the possible choices for the next text token, i.e., the distribution which can be used by the output unit of the decoder 102 to select the next text token as described above. Since, as described above, the first possibility can be considered a specific case of the second possibility (e.g., if the distribution has a specific value (e.g., non-zero value) for only one of the text tokens of the vocabulary, thereby indicating that that text token is the next text token), only the second possibility is considered below.

[0087] In step 405, the training engine 201 updates the numerical parameters (defining the operation) of the encoder 101 and / or the numerical parameters (defining the operation) of the decoder 102. Both may be updated in each performance of the set of steps of 401-405, or, in each performance of the set of steps 401-405 the numerical parameters for a selected one of the encoder 101 and decoder 102 may be updated, such that updates to the numerical parameters ofAttorney Docket No.45288-0442WO1 the encoder 101 are interleaved with updates to the numerical parameters defining the operation of the decoder 102.

[0088] The numerical parameters defining the encoder 101 and the numerical parameters defining the decoder 102 may be collectively denoted θ. In step 405 the update to θ is based on a success function which is a function of the output of the decoder in step 404, and in particular its similarity to the text token of the text token string of the training example selected in step 401 which follows the initial portion of the text token string of the training example.

[0089] Although steps 401-404 have been described in the case that the single training example is used, it is to be understood that that steps 401-404 may be performed for a plurality (“batch”) of more than one of the training examples. Alternatively or additionally, for any given training example, steps 402-404 may be performed for different choices of the initial portion of the training example. Thus, the update carried out in step 405 may be performed using the respective results of performing steps 401-404 for each of plurality of training examples, and / or for the respective results of performing steps 402-404 for multiple choices of the initial portion of the training example chosen in step 401.

[0090] The success function may be a sum, over one or more selected ones of the training examples, and, for each training example, over one or more selected choices of the initial portion of the text token string of the training example, of a respective calculated measure (“success term”) of the corresponding output of the decoder 102 in step 404 and the corresponding text token of the text token string of the training example which follows the selected initial portion of the text token string. The success function may be calculated as a sum of respective success term for multiple ones of the training examples, and / or, for each training example, a sum over multiple possible choices of the initial portion of the text token string. A given update may be calculated by approximating the value of the success function for a “batch” (i.e., a subset) of the training examples of the training database.

[0091] The media item for a given one of the training examples may be denoted as x. The corresponding text token string for the training example may be denoted as {yj}, for j=1,…n, where n is the number of text tokens in the text token string of the training example. For a certain performance of the steps 402 and 404, the initial portion of the text token string of the training example defined in step 402 consists of a number of text tokens denoted by i-1,Attorney Docket No.45288-0442WO1 where i is an integer index in the range 1 to n. The initial portion of the text token string is denoted y<i, and the text token of the text token string of the training example which follows the initial portion is denoted yi. The value for this text token yi in the distribution over the text token output by the decoder 102 upon processing x and y<i,is denoted P(yi|y<i, x). For example, as described above, this may be the probability that the token yi would be selected by the decoder, given the corresponding media item x and the initial portion of the text token string, as the text token which should follow the initial portion of the text token string. The success term is a function of the probability P(yi|y<i, x), such as its logarithm.

[0092] As noted, for a given selected training example, steps 402 and 404 may be carried out for multiple choices of the initial portion of the text token string of the selected training example. Optionally, the success function may be calculated as a sum of a respective success term over each possible value of i, e.g., subject to a limit such that the initial portion of the text token string includes at least the task prefix or other data as mentioned above.

[0093] In other words, in step 405 the update to θ may be one which increases a sum, over a batch of one or more of the training examples, and, for each training example of the batch, over one or more choices of the value of i (e.g., all possible choices, or any possible choices subject to a limit, such as i being longer than the task prefix), of log P(yi|y<i, x).

[0094] Following the training in method 100, the neural network model 100 may be deployed to perform a media item processing task (e.g., an image processing task), one comprising any one or more of the training tasks exemplified by the training examples in training databases 201a, 201b, 201c.

[0095] FIG. 5, a method 500 is shown which is an example of the present disclosure. The method can be carried out by the trained neural network model 100 of Fig. 1. It may be implemented as computer programs on one or more computers in one or more locations.

[0096] In step 501 a media item (e.g., an image) is received.

[0097] In step 502, the media item is processed by the encoder 101 trained as explained above to generate feature data.

[0098] In step 503, the output of the encoder 101 is processed to perform a media item processing task on the media item.

[0099] For example, the media item processing task may be to perform one of the training tasks using the decoder trained in the method 400 of Fig. 4, using the neural networkAttorney Docket No.45288-0442WO1 model 100. Alternatively, the media item processing task may be a different task (a “downstream task”), performed using a “new” neural network model which has the same overall form as the neural network 100 of Fig. 1, and which incorporates the media item encoder trained using the method 400 of Fig. 4, but in which the decoder 102 is replaced with a different (“new”) decoder. A method of forming a “new” neural network model of this kind is explained below with reference to FIG. 6. [000100] In either case, in step 503 in each of a sequence of multiple time steps, the feature data is input to the decoder (e.g. the decoder 102). In the first time step, the decoder may also receive a text prompt, which may be empty or may be one of the task prefixes (i.e., specifying one of the training tasks), and the decoder selects a text token. In each subsequent time step, the text token generated in the last time step is added to the end of the prompt, and input to the decoder in addition to the feature data 104. Thus, the decoder auto-regressively generates a response to the media item. The process may terminate when the decoder 102 generates a token of a predefined type. [000101] This is not the only way in which step 503 can be implemented, however. For example, the media item encoder 101 trained by the method 400 may be included in a “new” neural network model having a different geometry; for example, one which does not include a decoder for the feature data. For example, the “new” neural network model may include a text encoder neural network model (“text encoder”) which does not receive the feature data. An output unit may compare the feature data generated by the media item encoder 101 upon processing a media item, to the output of the text processing model. For example, such a neural network model may perform contrastive learning, e.g., such that a text having a similarity to a given media item can be identified by the output of the text processing model upon receiving the text being similar (according to a similarity metric) to feature data generated by the trained media item encoder upon receiving the media item. The text processing model of the “new” neural network model can be trained based on training examples, with the media item encoder optionally being frozen during this process. [000102] The use of the two “proxy” training tasks ARef and GCap enhances the inference flexibility of the trained media item encoder 101, and thus of the trained neural network model 100 or a “new” neural network model incorporating the trained media item encoder neural network 101. This allows for varied input configurations. For instance, in theAttorney Docket No.45288-0442WO1 case of the trained neural network model 100, a user can input a single task prefix (e.g., “ARef:”) as a prompt to cause the model to identity and describe an area of interest along with its location. Alternatively, by inputting both the task and a conditional input (e.g., “ARef: a black and white cat: “), the trained neural network model 100 can be directed to focus solely on predicting the location of specified content. This flexibility allows customized responses to various enquiries, highlighting the model’s adaptability to meet specific user needs. [000103] Turning to Fig. 6, a method 600 is shown which is another example of the present disclosure. In method 600 a “new” neural network model is formed which incorporates at least the media item encoder trained by a method such as method 400 of Fig. 4. The “new” neural network model is trained to perform a media item processing task (a “downstream task”), such as an image processing task, which may be different from any of the training tasks. Method 600 may be implemented as computer programs on one or more computers in one or more locations. [000104] In step 601, a media item encoder is obtained which has been trained by joint training of a media item encoder and a decoder, such as the method 400 explained above with reference to FIG. 4. [000105] In step 602, a “new” neural network is formed. The neural network may have the same structure depicted in FIG. 1. If so, the media item encoder 101 is the encoder trained by the method 400 explained above, but the decoder may either by the same decoder 102 used in the joint training, or it may be a new decoder, such as a decoder having a different architecture. [000106] Alternatively, the new neural network may have a different structure, according to any known media item processing system. For example, the new neural network model may include, in addition to the trained media item encoder 101 and a decoder (which may be the decoder 102), an adapter module, such as one which processes the feature data 104 generated by the media item encoder 101 and / or the text token string 105, before they is processed by the decoder. In another example, the new neural network model may include the trained media item encoder 101 and a text encoder which does not receive the output of the trained encoder. [000107] In step 603, the “new” neural network model formed in step 602 is trained on the downstream task; this is referred to as a “second training procedure”. [000108] The second training procedure may be performed using “second” training examples of the downstream task. The second training procedure may iteratively modifyAttorney Docket No.45288-0442WO1 parameters of the “new” neural network model to increase a measure of similarity between desired output data of the neural network model defined by at least one second training example, and data output by the “new” neural network model upon receiving input data defined by the second training example (e.g., a media item and optionally a token string). [000109] For example, in the case that the “new” neural network model has the structure depicted in FIG. 1, then to perform the media item processing task, the trained media item encoder 101 of the “new” neural network model may receive a media item, and generate feature data from it, and the decoder may process the feature data, and also auto-regressively text token data, to generate new text tokens. The output of the “new” neural network model may be the output of the decoder, or the output of the “new” neural network model may be an output which is generated based on the output of the decoder by an output unit (e.g., if the output of the decoder is a distribution over a vocabulary of tokens, the output unit may select one of the tokens based on the distribution). [000110] As in the method 400, the second training examples may be examples of the downstream task which each include (at least one) media item and a corresponding text token string. The second training procedure may include modifying the decoder, such that upon the decoder processing a media item of a training example to generate feature data, and the decoder processing the feature data and a portion of the corresponding text token string of the training example, the decoder produces an output which indicates the next text token of the corresponding text string token (i.e. the text token of the text token string which follows the portion of the text token string). [000111] Alternatively, as noted, the “new” neural network model may have a different structure from that of FIG. 1. For example, it may include the media item encoder and a text encoder (rather than a text decoder). This form of neural network model may be trained by contrastive training. That is, the text encoder may be trained, upon receiving the text token string of a second training example (e.g. all at once), to generate an embedding of the text token string of the second training example which is similar (according to a similarity metric) to feature data produced by the trained media item encoder from the media item of the training example. [000112] In this case, the downstream task may be to identify which of a set of captions is associated with a given media item. The task may be performed by the media item encoderAttorney Docket No.45288-0442WO1 generating feature data from the media item, and the text encoder generating a respective embedding of each of the set of captions. One of the set of captions may be selected for which the corresponding embedding produced by the text encoder is most similar to the feature data produced by the media item embedder. [000113] In either case (i.e. whether the “new” neural network model has the same structure as in FIG. 1 or otherwise), the “new” neural network model may operate as a “multi- modal machine learning model”, since in the media item processing task it may process input data having two modalities: the media item (which is generally not composed of text tokens), and optionally the token string. At least part (e.g., an initial part) of the token string may be defined by a user. Similarly, the output may comprise one or both of a media item and text tokens, e.g., an image annotated by location data such as bounding boxes and text (e.g., for each bounding box). Some example multimodal machine learning models with which the techniques described herein may be used include: Flamingo (Alayrac et al. arXiv:2204.14198); ALIGN (Jia et al., arXiv:2102.05918); PaLI (Chen et al. arXiv:2209.06794); and PaLI-X (Chen et al. arXiv:2305.18565). [000114] The part of the “new” neural network model which is trained in the second training procedure may include any one or more of (i) the media item encoder, (ii) the decoder and / or (iii) an adapter module of the neural network model (i.e., a module of the neural network model which is addition to the media item encoder neural network and the decoder). The adapter module may for example comprise one or more neural network layers which act as an interface between the media item encoder and the decoder (i.e., process an output of the media item encoder to generate an input of the decoder), and / or one or more neural network layers which process respective outputs of the media item encoder and the decoder, to generate an output of the neural network model. [000115] The downstream task may comprise any one or more of: (i) media item classification (CLS), i.e., assigning an input media item (e.g., image) to one (or more) of set of classes based on content of the media item. Note that “content” here means content defined by pixelated intensity values in the case that the media item is an image, or defined by amplitude values in the case that the media item is a sound. The assignment to class(es), or the definition of the classes, may be at least partially defined by the received token string.Attorney Docket No.45288-0442WO1 (ii) captioning (generating a caption, such as a sequence of tokens selected from a vocabulary) describing the content of the media item; (iii) question answering (e.g., visual question answering (VQA)) of generating an answer which is an appropriate response to a question about the media item defined by the token string received by the decoder of the neural network model; (iv) a segmentation task of identifying a (proper) subset of the media item having specific properties, such as properties defined by the token string (e.g., for a media item which is an image, “a penguin in the foreground”, “cancerous material”; or for a media item which is a sound signal, “a child crying after a dog has barked”); and (v) an agent control task of generating control data for an agent in an environment (e.g., an electro-mechanical agent such as a robot) based on a media item describing the environment (e.g., received sensor data, such as an image of an environment captured by a camera) and a token string defining an action the robot is instructed to perform. [000116] During the second training procedure, the decoder of the neural network model may be trained from scratch. This may be done, particularly in the case that the media item encoder is a visual encoder, using the “PaLI-3” training strategy outlined in Chen, X., et al., “PaLI-3 vision language models: Smaller, faster, stronger” (2023), https: / / arxiv.org / pdf / 2310.09199.pdf). It may use a visual encoder pre-trained as described above in place of the visual encoder mentioned there. [000117] Alternatively, the decoder of the neural network model may be the decoder 102 which was jointly trained with the media item encoder 101, e.g., by the method 400 of FIG. 400, and which is further trained (fine-tuned) in the second training procedure, to perform the downstream processing task. [000118] In another option, the decoder of the neural network model may be a pre-trained decoder. The second training procedure may comprise fine-tuning the decoder and / or training an adapter module described above, for the downstream task. The second training procedure, using the second training examples, is based on outputs of the pre-trained media item encoder, upon receiving the media items of the second training examples. The second training procedure may use the PaLI-3 training strategy mentioned above, and indeed the decoder of the neural network model may the same decoder used in the PaLI-3 paper.Attorney Docket No.45288-0442WO1 [000119] Note that because the method 600 uses a media item encoder 101 which has been trained using the method 400 of Fig. 4, the feature data output by the media item encoder upon processing a media item, encodes positional information, e.g., to a higher degree than in a known media item encoder. [000120] We now turn to experimental results of examples of the present disclosure, using a neural network model as described above. The experiments were performed using, as the media item encoder 101, a Vit-L / 14 design with 24 transformer blocks handing input patches of size 14. The decoder 102 is a Transformer-L model consisting of 12 transformer decoder blocks. In total the neural network model 100 comprised approximately 600M parameters. The first training procedure (method 400) was performed using 9 billion image-text examples. The optimizer which performed step 405 was the Scaling-Vit AdaFactor variant of Xiaohua Zhai et al, “Scaling vision transformers”, in CVPR 2022, using a batch size of 8192, with the learning rate and decay factor adjusted to 10-3and 10-4respectively. During this procedure the images of the training examples were uniformly resized to a resolution of 224x224 pixels, and the corresponding text token strings of the training examples were formed using a vocabulary consisting of 32,000 tokens using a sentence piece model. Bounding box coordinates were up to 500 integral numbers, which were converted into text tokens to form the text token strings of the training databases 201b and 201c. The fraction of examples used by the method 400 which are selected from the training database 201a (i.e., the “vanilla image captioning task”) was set to 50%. [000121] In the second training procedure (step 603 of method 600), the downstream tasks chosen were: “referring expression comprehension” (REC), such as described in S. Kazemzadeh, et al., “ReferItGame: Referring to objects in photographs of natural scenes); “referring expression segmentation” (RES) such as described in Ronghang Hu, et al., “Segmentation from natural language expressions”, in ECCV 2016); and object detection (OD; such as described in Shaoqing Ren, et al., “Faster r-cnn: Towards real-time object detection with region proposal networks”, in NeurIPS, 2015. In some experiments, the trained decoder model 102 was fine-tuned for these tasks. [000122] First experiments were performed of referring expression comprehension. These were performed using the comprehension benchmarks RefCOCO, RefCOCO+ and RefCOCOg disclosed at Licheng Yu, et al, “Modeling context in referring expressions”, in ECCV, 2016.Attorney Docket No.45288-0442WO1 The results are shown in Table 1. The examples of the present disclosure are indicated here as LocCaL and LocCaG. LocCaL has 600M trainable numerical parameters, and LocCaG has 2.8 billion trainable numerical parameters.Table 1 [000123] It will be seen that the examples of the present disclosure were consistently more successful than benchmark techniques. The benchmark techniques referred to in Table 2 are PixelLLM (Jiarui Xu, et al., “Pixel Aligned language models”, CoRR, arXiv:2312.09237, 2023), UniTAB (Zhengyuan Yang, et al., “Unitab: Unifying text and box outputs for grounded vision-language modeling”, in ECCV, 2016), OFAL (Peng Wang, et al., “OFA: unifying architectures, tasks and modalities through a simple sequence-to-sequence learning framework”, ICML 2022), UNINEXTL (Bin Yan, et al., “Universal instance perception as object discovery and retrieval), ONE-PEACE1.5B (Peng Wang, et al., “One-peace: Exploring one general representation model toward unlimited modalities”, CoRR, 2020), Shikra13B(Keqin Chen, et al., “Shikra: Unleashing multimodal 11m’s referential dialogue magic”, CoRR 2023), Ferret13B (Hoaxuan You, et al., “Ferret: Refer and Ground anything anywhere at anyAttorney Docket No.45288-0442WO1 granularity”, ICLR, 2024), UNITERL(Yen-Chun Chen, et al., “UNITER: universal image-text representation learning”, ECCV, 2020), VILLAL (Zhe Gan, et al., “Large scale adversarial training for vision-and-language representation learning”, CoRR 2020), MDETR (Aishwarya Kamath, et al., “Mdetr-modulated detection for end-to-end multi-modal understanding”, ICCV, 2021) and RefTR (Li Muchen et al., “Referring transformer: A one-step approach to multi-task visual grounding”, in NeurIPS 2021). [000124] Turning to the downstream task of referring expression segmentation, the results are shown in Table 2. In this downstream task, the second training examples include the task prefix “Mask:” followed by indexes which contain data (“segmentation tokens”) specifying the shape and position of a bounding box identified during the REC task. This experiment used a pre-trained VQ-VAE (86) to convert semantic marks into tokens. As shown in Table 2, the examples of the present disclosure (“LocCA”) outperformed benchmark neural network models substantially. The benchmarks used in this case were CLIP (A. Radford, et al., “Learning transferrable visual models from natural language supervision”, in ICML, 2021), a CLIP-style contrastively pre-trained dual-encoder model referred to as CLIP*, and captioning-aware encoder-decoder models Cap and CapPa (M. Tschannen, et al., “Image captioners are scalable vision learners too”, NeurIPS, 2023). The examples provided competitive results even compared to the state-of-the-art PaLI-3 model, though with considerably fewer parameters (0.6B versus 5B).Attorney Docket No.45288-0442WO1 [000125] Turning to the downstream task of object detection, the results are shown in Fig. 7. The COCO dataset (Xinlei Chen, et al., “Microsoft COCO captions: Data collection and evaluation server”, CoRR, 2015) was used, following the methodology of A. Susano Pinto, et al., “Tuning computer vision models with task rewards”, in ICML, 2023. This models the task with an encoder-decoder model that outputs sequences of bounding boxes. The second training procedure was in two phases: a first phase in which the neural network model maximizes the log-likelihood of generating ground truth sequences of boxes; and a second, reinforcement phase in which the neural network model is trained to tune the model for a reward related to the mAP (mean average precision) metric. Fig. 7 shows results for mAP and AR (average recall) both with and without reinforcement tuning (reinforcement learning, “RL”). The examples of the present method significantly outperformed the image-text trained baselines. [000126] As noted above, layers of the encoder and / or decoder of the neural network model of Fig. 1, or the new decoder (if any) used in step 602 of the method 600 of FIG. 6 to form the “new” neural network model, may employ an attention mechanism applied by an attention neural network layer (processing layer). This may be any sort of attention mechanism. Generally, an attention mechanism maps a query and a set of key-value pairs to an output, where the query, keys, and values are all vectors. The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function, e.g., a dot product or scaled dot product, of the query with the corresponding key. [000127] In implementations, the attention mechanism is configured to apply a self- attention mechanism over the attention layer input; this may be followed by one or more feed- forward neural network layers to generate the attention layer output. In general, an attention mechanism determines a relationship between two sequences; a self-attention mechanism is configured to relate different positions in the same sequence to determine a transformed version of the sequence as an output. For example, the attention layer input may comprise a vector for each element of the input sequence. These vectors provide an input to the self-attention mechanism and are used by the self-attention mechanism to determine a new representation of the same sequence for the attention layer output, which similarly comprises a vector for each element of the input sequence. An output of the self-attention mechanism may be used as the attention layer output, or it may be processed by one or more feed-forward layers to provide the attention layer output.Attorney Docket No.45288-0442WO1 [000128] In some implementations the attention mechanism is a self-attention mechanism configured to apply each of a query transformation, e.g., defined by a matrix ^^, a key transformation, e.g., defined by a matrix ^^, and a value transformation, e.g., defined by a matrix ^^, to the attention layer input for each element denoted by a vector x of the input sequence X (where X is a matrix in which each row is one of the elements x of the sequence; note that the number of rows of X may be limited to a value N+1, so that the output of the attention mechanism for a given input x may only be based on that input x and the N precedinginputs) to derive a respective query matrix (formed of query vectors) ^ = ^^^, a respectivekey matrix (formed of key vectors) ^ = ^^^, and a respective value matrix (formed of valuevectors) ^ = ^^^, which are used determine an attended sequence for the output. Forexample, the attention mechanism may be a dot product attention mechanism applied by applying each query vector to each key vector to determine respective weights for each value vector, then combining the value vectors using the respective weights to determine the attention layer output for each element of the input sequence. The attention layer output may be scaled by a scaling factor, e.g., by the square root of the dimensions of the queries and keys, to implement scaled dot product attention. Thus, for example, an output of the attention ^mechanism may be determined as ^^^^^^^ ^^^√^^V where d is a dimension of the key (and the query and commonly value) vector. A summation over the value vectors included in V is ^^^assumed here, weighted by the respective values ^^^^^^^ ^ √^^. [000129] An attention mechanism may also be amechanism (rather than a self-attention mechanism), i.e., each attention layer input x is applied to another data matrix Y having rows composed of corresponding vector elements y, the key matrix and value matrix are multiplied by Y rather than X, to give the corresponding matrix of key vectors and value vectors. [000130] In another implementation the attention mechanism may comprise an “additive attention” mechanism that computes the compatibility function using a feed-forward network with a hidden layer. As previously mentioned, output of the attention mechanism may be further processed by one or more fully-connected, feed forward neural network layers. [000131] The attention mechanism may implement multi-head attention. That is, it may apply multiple different attention mechanisms in parallel. The outputs of these may then beAttorney Docket No.45288-0442WO1 combined, e.g., concatenated, with a learned linear transformation applied to reduce to the original dimensionality if necessary. [000132] Examples of suitable architectures for an attention mechanism are described in Ashish Vaswani et al., “Attention is all you need”, Advances in Neural Information Processing Systems, pp. 5998–6008, 2017; arXiv:1810.04805 Devlin et al. (BERT); and arXiv:1901.02860 Dai et al. (Transformer-XL). [000133] There is now a discussion of applications of the neural network model. In other words, the following are possible choices for the “media item processing task” discussed above, which may be a downstream task or one of the training tasks. [000134] As one example the media item processing task may comprise an object or action detection task. A media item may comprise an image, video, or audio item containing one or more objects or actions. The token string may describe or otherwise label the object(s) or action(s) and (for an image or video) may include text giving bounding box coordinates for the object(s) or action(s). The neural network model output may comprise or represent text that describes or otherwise labels detected object(s) or action(s) in the media item, and may (for an image or video) include bounding-box coordinates for the detected object(s) or action(s), e.g., “102090100 cat 2030100100 dog”. [000135] As another example the media item processing task may comprise a classification task, e.g., an object or action classification task. A media item may comprise an image, video, or audio item containing one or more objects or actions and a sequence of text. The token string may describe or otherwise classify the object(s) or action(s). The output of the neural network model may comprise data, e.g., text that classifies the object(s) or action(s) in the second modality input into one of a plurality of classes. [000136] As another example, the media item processing task may comprise an image, video, or audio question-answering task. A media item may comprise an image, video, or audio item and a token string defining a question. The output of the neural network model may comprise data, e.g., text, that answers a question about the media item. This may be used, e.g., to answer questions about visual plots and charts or about sounds. [000137] As another example the media item processing task may comprise a character or word recognition task, e.g., an OCR (optical character recognition) task. A media item mayAttorney Docket No.45288-0442WO1 comprise an image or video. The output of the neural network model may comprise text that represents characters or words in the media item. [000138] In another example, the media item comprises a sound signal (i.e., an audio data item comprising a representation of a digitized audio waveform, e.g., a speech waveform, e.g., captured from the real-world using a microphone). Such a representation may comprise samples representing digitized amplitude values of the waveform or, e.g., a time-frequency domain representation of the waveform such as a STFT (Short-Term Fourier Transform) or MFCC (Mel-Frequency Cepstral Coefficient) representation. In one application, a user may generate a token string (e.g., by typing, or by speaking into a microphone to generate a signal which is transmitted to a speech-to-text converter) which is “Car horn”. Upon processing a sound signal and the token string, the neural network model may output label data indicating the location(s) of any car horn sounds in the sound signal. [000139] The media item may represent the waveform of any signal, e.g., a signal from a sensor, e.g., a sensor sensing a characteristic of the real world, e.g., an object in the real world. The signal may be a sound signal, measured by a microphone. In one application, a user may generate a token string which is, e.g., “Unstable”. Upon processing a sensor signal and the token string, the neural network model may output label data indicating the location(s) of any unstable portions of the sensor signal. Note that the system may be effective in this task even if none of the examples used to train the adaptive system related to an unstable sensor signal. [000140] In another example, the media item may be an image, such a still, two- dimensional image. For example, in one application, a user may generate a token string which is “Cat”. Upon processing an image and the token string, the neural network model may output label data indicating the location(s) of any cats in the image. [000141] In one application, the user may generate a token string which is, for example, “flying vehicle”, and the neural network model may output data which is bounding boxes showing the location(s) of any flying vehicle in the image. Note that the neural network model may be effective in this task even if the examples used to train the adaptive system did not include images of flying vehicles. [000142] In another application, the image may be a medical image, such as an X-ray image, CAT scan or MRI image. The user may generate a token string which is a medical analysis of the medical image, for example, “cancerous matter”, and the trained adaptiveAttorney Docket No.45288-0442WO1 system may output segmentation data indicating the locations of cancerous matter in the medical image. [000143] In another example, the media item may be a video (i.e., a sequence of image frames), such as one captured from the real world using a camera. The user may generate a token string which is, for example, “skiing”, and the neural network model may output data which indicates frames in the video, and optionally locations in the frames, where skiing activity is carried out. [000144] In another example, the media item includes an image, video, or audio item (such as ones captured from the real-world by camera(s) or microphone(s)), and the token string might specify “Generate a caption”, “Generate a description”, “Answer the following question: [about the image, video, or audio item]”, or “Detect a person”. Where the neural network model is used for an agent control task a prompt may define “Take the knife out of the drawer”, or “Q: What action should the robot take to take the knife out of the drawer?”. [000145] Such a prompt may give one or more examples of a task to be performed. For example, it may read “Black cat. Red dog. What is in the top left corner of the image”. In this case, if the image depicts an animal in the top left corner, the neural network model may be more likely to generate text that follows the format of the examples (e.g., “orange budgerigar”) than if the examples had not been provided. [000146] The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages. [000147] The media item encoder (e.g. vision encoder) is trained in the present method to generate feature data from a media item which preserves information in the media item which is useful in media item processing tasks which require “localization”, i.e., understanding the location of content in the media item. The feature data encodes this information as feature data in a format which is suitable for linking fine-grained regional visual elements with appropriate textual descriptions. Experiments show that the examples of the present disclosure outperform known models significantly on downstream media item processing tasks which require localization, while maintaining comparable performance on “holistic” downstream tasks which do not.Attorney Docket No.45288-0442WO1 [000148] The training of the media item encoder may be achieved using training data which can be obtained and annotated automatically at low expense. In other words, successful training of the visual encoder is not reliant on expensive hand-annotated training data. [000149] This specification uses the term “configured” in connection with computing systems and environments, as well as computer program components. A computing system or environment is considered "configured" to perform specific operations or actions when it possesses the necessary software, firmware, hardware, or a combination thereof, enabling it to carry out those operations or actions during operation. For instance, configuring a system might involve installing a software library with specific algorithms, updating firmware with new instructions for handling data, or adding a hardware component for enhanced processing capabilities. Similarly, one or more computer programs are "configured" to perform particular operations or actions when they contain instructions that, upon execution by a computing device or hardware, cause the device to perform those intended operations or actions. [000150] The embodiments and functional operations described in this specification can be implemented in various forms, including digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. Additionally or alternatively, the program instructions can be encoded on a transmitted signal, such as a machine-generated electrical, optical, or electromagnetic signal, designed to carry information for transmission to a receiving device or system for execution by a computing device or hardware. Furthermore, implementations may leverage emerging technologies like quantum computing or neuromorphic computing for specific applications, and may be deployed inAttorney Docket No.45288-0442WO1 distributed or cloud-based environments where components reside on different machines or within a cloud infrastructure. [000151] The terms “data processing apparatus” or “computing device or hardware” refer to the physical components involved in data processing and encompass all types of devices and machines used for this purpose. Examples include processors or processing units, computers, multiple processors or computers working together, graphics processing units (GPUs), tensor processing units (TPUs), and specialized processing hardware such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). In addition to hardware, a computing device or hardware may also include code that creates an execution environment for computer programs. This code can take the form of processor firmware, a protocol stack, a database management system, an operating system, or a combination of these elements. Embodiments may particularly benefit from utilizing the parallel processing capabilities of GPUs, in a General-Purpose computing on Graphics Processing Units (GPGPU) context, where code specifically designed for GPU execution, often called kernels or shaders, is employed. Similarly, TPUs excel at running optimized tensor operations crucial for many machine learning algorithms. By leveraging these accelerators and their specialized programming models, the system can achieve significant speedups and efficiency gains for tasks involving artificial intelligence and machine learning, particularly in areas such as computer vision, natural language processing, and robotics. [000152] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may or may not correspond to a single file in a file system and can be stored in various ways. This includes being embedded within a file containing other programs or data (e.g., scripts within a markup language document), residing in a dedicated file, or distributed across multiple coordinated files (e.g., files storing modules, subprograms, or code segments). A computer program can be executed on a single computer or across multiple computers, whether located at a single site or distributed across multiple sites and interconnected through a data communication network. The specific implementation of theAttorney Docket No.45288-0442WO1 computer programs may involve a combination of traditional programming languages and specialized languages or libraries designed for GPGPU programming or TPU utilization, depending on the chosen hardware platform and desired performance characteristics. [000153] In this specification, the term “database” is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. Thus, for example, the index database can include multiple collections of data, each of which may be organized and accessed differently. [000154] In this specification, the term "engine" broadly refers to a software-based system, subsystem, or process designed to perform one or more specific functions. An engine is typically implemented as one or more software modules or components installed on one or more computers, which can be located at a single site or distributed across multiple locations. In some instances, one or more dedicated computers may be used for a particular engine, while in other cases, multiple engines may operate concurrently on the same one or more computers. Examples of engine functions within the context of AI and machine learning could include data pre-processing and cleaning, feature engineering and extraction, model training and optimization, inference and prediction generation, and post-processing of results. The specific design and implementation of engines will depend on the overall architecture and the distribution of computational tasks across various hardware components, including CPUs, GPUs, TPUs, and other specialized processors. [000155] The processes and logic flows described in this specification can be executed by one or more programmable computers running one or more computer programs to perform functions by operating on input data and generating output. Additionally, graphics processing units (GPUs) and tensor processing units (TPUs) can be utilized to enable concurrent execution of aspects of these processes and logic flows, significantly accelerating performance. This approach offers significant advantages for computationally intensive tasks often found in AI and machine learning applications, such as matrix multiplications, convolutions, and other operations that exhibit a high degree of parallelism. By leveraging the parallel processing capabilities of GPUs and TPUs, significant speedups and efficiency gains compared to relying solely on CPUs can be achieved. Alternatively or in combination with programmable computers and specialized processors, these processes and logic flows can also be implementedAttorney Docket No.45288-0442WO1 using specialized processing hardware, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs), for even greater performance or energy efficiency in specific use cases. [000156] The processes and logic flows described in this specification can be executed by one or more programmable computers running one or more computer programs to perform functions by operating on input data and generating output. Additionally, graphics processing units (GPUs) and tensor processing units (TPUs) can be utilized to enable concurrent execution of aspects of these processes and logic flows, significantly accelerating performance. This approach offers significant advantages for computationally intensive tasks often found in AI and machine learning applications, such as matrix multiplications, convolutions, and other operations that exhibit a high degree of parallelism. By leveraging the parallel processing capabilities of GPUs and TPUs, significant speedups and efficiency gains compared to relying solely on CPUs can be achieved. Alternatively or in combination with programmable computers and specialized processors, these processes and logic flows can also be implemented using specialized processing hardware, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs), for even greater performance or energy efficiency in specific use cases. [000157] Computer-readable media suitable for storing computer program instructions and data encompass all forms of non-volatile memory, media, and memory devices. Examples include semiconductor memory devices such as read-only memory (ROM), solid-state drives (SSDs), and flash memory devices; hard disk drives (HDDs); optical media; and optical discs such as CDs, DVDs, and Blu-ray discs. The specific type of computer-readable media used will depend on factors such as the size of the data, access speed requirements, cost considerations, and the desired level of portability or permanence. [000158] To facilitate user interaction, embodiments of the subject matter described in this specification can be implemented on a computing device equipped with a display device, such as a liquid crystal display (LCD) or an organic light-emitting diode (OLED) display, for presenting information to the user. Input can be provided by the user through various means, including a keyboard), touchscreens, voice commands, gesture recognition, or other input modalities depending on the specific device and application. Additional input methods can include acoustic, speech, or tactile input, while feedback to the user can take the form of visual, auditory, or tactileAttorney Docket No.45288-0442WO1 feedback. Furthermore, computers can interact with users by exchanging documents with a user's device or application. This can involve sending web content or data in response to requests or sending and receiving text messages or other forms of messages through mobile devices or messaging platforms. The selection of input and output modalities will depend on the specific application and the desired form of user interaction. [000159] Machine learning models can be implemented and deployed using machine learning frameworks, such as TensorFlow or JAX. These frameworks offer comprehensive tools and libraries that facilitate the development, training, and deployment of machine learning models. [000160] Embodiments of the subject matter described in this specification can be implemented within a computing system comprising one or more components, depending on the specific application and requirements. These may include a back-end component, such as a back- end server or cloud-based infrastructure; an optional middleware component, such as a middleware server or application programming interface (API), to facilitate communication and data exchange; and a front-end component, such as a client device with a user interface, a web browser, or an app, through which a user can interact with the implemented subject matter. For instance, the described functionality could be implemented solely on a client device (e.g., for on- device machine learning) or deployed as a combination of front-end and back-end components for more complex applications. These components, when present, can be interconnected using any form or medium of digital data communication, such as a communication network like a local area network (LAN) or a wide area network (WAN) including the Internet. The specific system architecture and choice of components will depend on factors such as the scale of the application, the need for real-time processing, data security requirements, and the desired user experience. [000161] The computing system can include clients and servers that may be geographically separated and interact through a communication network. The specific type of network, such as a local area network (LAN), a wide area network (WAN), or the Internet, will depend on the reach and scale of the application. The client-server relationship is established through computer programs running on the respective computers and designed to communicate with each other using appropriate protocols. These protocols may include HTTP, TCP / IP, or other specialized protocols depending on the nature of the data being exchanged and the security requirements ofAttorney Docket No.45288-0442WO1 the system. In certain embodiments, a server transmits data or instructions to a user's device, such as a computer, smartphone, or tablet, acting as a client. The client device can then process the received information, display results to the user, and potentially send data or feedback back to the server for further processing or storage. This allows for dynamic interactions between the user and the system, enabling a wide range of applications and functionalities. [000162] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination. [000163] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. [000164] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

Attorney Docket No.45288-0442WO1 CLAIMS 1. A method of training a media item encoder neural network, the media item encoder neural network being configured to receive a media item and to generate an output comprising feature data which is an encoding of the media item, the method comprising: jointly training the media item encoder neural network and a decoder neural network, the decoder neural network being arranged to receive feature data output by the media item encoder neural network, and to generate text tokens based on the received feature data, the training being based on one or more training examples which each comprise a media item and a corresponding text token string, the corresponding text token strings of one or more of the training examples comprising first text tokens describing an object present in the corresponding media item, and second text tokens defining a position of the object in the media item.

2. The method of claim 1 in which the decoder neural network is an auto-regressive decoder neural network, and the joint training comprises repeatedly updating a plurality of first numerical parameters defining the operation of the media item encoder neural network and a plurality of second numerical parameters defining the operation of the decoder neural network, the updating being to increase the value of a success function which comprises a sum, over one or more of the training examples, of a measure of a similarity between: (a) an output of the decoder neural network upon receiving as an input (i) the output of the media item encoder neural network upon receiving the corresponding media item, and (ii) an initial portion of the corresponding text token string; and (b) a portion of the corresponding text token string following the initial portion.

3. The method of claim 1 in which the training examples are each in a corresponding one of a plurality of classes corresponding to respective training tasks, the training examples of each class exemplifying the corresponding training task.

4. The method of claim 3 wherein said output of the decoder neural network for a given training example is based on data received by the decoder neural network and indicative of the corresponding training task.Attorney Docket No.45288-0442WO1 5. The method of claim 4 when dependent on claim 2, in which the data indicative of the corresponding training task is included in the initial portion of the text token string.

6. The method of any of claims 3 to 5, in which the training tasks comprise a task of generating a text token string in which the first text tokens precede the second text tokens.

7. The method of any of claim 3-6, in which the training tasks comprise a task of generating a text token string in which the second text tokens precede the first text tokens.

8. The method of any of claims 3-7, in which the training tasks comprise a task of generating a text token string in which the second text tokens define the position of the object numerically.

9. The method of any preceding claim, in which the media item comprises a sound signal.

10. The method of any preceding claim, in which the media item comprises an image.

11. The method of claim 10 in which the second text tokens define the position of the object as a bounding box within the image, the bounding box containing the object.

12. The method of 10 or claim 11 when dependent upon claim 3, in which the training tasks comprise a task in which the first text tokens describe a plurality of objects in the image, and the second text tokens define a relative position of the plurality of objects in a natural language.

13. The method of any of claims 10 to 12, in which the training examples are obtained by obtaining corresponding text-image pairs, generating respective localization data from each of the images, and generating the corresponding text token string of each training example based on the respective localization data.Attorney Docket No.45288-0442WO1 14. The method of any preceding claim, in which at least one of the media encoder neural network and the decoder neural network does not include a component which has been trained prior to the joint training.

15. A method of forming a neural network model for performing a media item processing task on a media item, the method comprising: obtaining a media item encoder neural network trained by the method of any of claims 1 to 14, and forming the neural network model by providing a decoder arranged to receive an output of the trained media item encoder neural network, wherein the media item encoder is configured to process a media item to generate feature data as the output of the trained media item encoder neural network, and the decoder is configured to process the feature data to generate a result of performing the media item processing task on the media item.

16. The method of claim 15, in which the decoder is a decoder which has previously been trained to perform a training task other than the media item processing task.

17. The method of claim 15 or claim 16 comprising training the neural network model based on second training examples comprising respective media items, the second training examples exemplifying the media item processing task.

18. The method of claim 17 when dependent upon claim 16, in which the training of the neural network model based on the second training examples is performed by training an adapter module of the neural network model which is different from the media item encoder neural network and the decoder.

19. The method of claim 17, in which the decoder has not previously been trained, and the decoder is trained as part of the training of the neural network model based on the second training examples.Attorney Docket No.45288-0442WO1 20. The method of claim 17, in which the decoder is the trained decoder neural network, the training of the neural network model comprising further training the decoder neural network based on the second training examples.

21. A method of processing a media item, the method comprising: receiving a media item; and using a media item encoder neural network obtained using the method of any of claims 1- 14 to generate from the media item an output comprising feature data.

22. A method of processing a media item, the method comprising: receiving a media item; and using a neural network model formed using the method of any of claims 15-20 to perform the media item processing task on an input comprising the media item.

23. A method of processing a media item to generate an output, the method comprising: receiving a media item, processing the media item using a media item encoder neural network, to generate an output comprising feature data, and processing the feature data to perform a media item processing task on the media item; the media item encoder neural network having been jointly trained with a decoder neural network, the joint training comprising: training the media item encoder to process a media item to generate feature data, and training the decoder neural network to generate a text token string based on the feature data, the text token string comprising first text tokens describing an object present the media item and second text tokens defining a position of the object in the media item.

24. The method of claim 23, in which decoder neural network has been trained to generate a text token string in which the first text tokens precede the second text tokens.

25. The method of claim 23 or 24, in which decoder neural network has been trained to generate a text token string in which the second text tokens precede the first text tokens.Attorney Docket No.45288-0442WO1 26. The method of any of claims 23-25, in which the second text tokens define the position of the object numerically.

27. The method of any of claims 15-26 in which the media item comprises a sound signal.

28. The method of any of claims 15-27, in which the media item comprises at least one image.

29. One or more computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the operations of the respective method of any one of claims 1-28.

30. A system comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of the respective method of any one of claims 1-28.

Images