Processing multimodal input using language models

Abstract

A method, system, and apparatus, including a computer program encoded on a computer storage medium, for processing a multi-modal input using a language model. In particular, the input includes an image, which is encoded by an image encoder neural network to generate a sequence of image embeddings representative of the image. The sequence of image embeddings is provided as at least a portion of an input sequence to be processed by the language model neural network.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority from Greek National Patent Application No. GR20210100374, filed June 8, 2021. The disclosure of the prior application is considered part of the disclosure of this application and is incorporated by reference.

[0002] This specification relates to processing inputs, including images, using neural networks. [Background technology]

[0003] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to the next layer in the network, e.g., the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values ​​of a respective set of parameters. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu, "Exploring the limits of transfer learning with a unified text-to-text transformer", arXiv preprint arXiv:1910.10683, 2019. [Non-patent document 2] Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le, “Towards a human-like open-domain chatbot,” CoRR, abs / 2001.09977, 2020. [Non-patent document 3] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell et al., "Language models are few-shot learners", arXiv preprint arXiv:2005.14165, 2020. [Non-patent document 4] Vaswani et al., "Attention is all you need," 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA Summary of the Invention [Means for solving the problem]

[0005] This specification describes a system, implemented as a computer program on one or more computers at one or more locations, that performs a multimodal task using an image encoder neural network and a language model neural network, where the language model neural network is pre-trained, e.g., not trained during the training of the image encoder neural network.

[0006] In one aspect, one of the methods includes the steps of obtaining an input image, processing the input image using an image encoder neural network trained to process the input image to generate an image sequence of image embedding vectors that characterize the input image, generating an input sequence of input embedding vectors including the image sequence of image embedding vectors, and processing the input sequence of input embedding vectors including the image sequence of image embedding vectors using a trained language model neural network to generate as output an output sequence of text tokens that represent text responses to the input sequence of input embedding vectors, wherein each text token in the output sequence is selected from a vocabulary of text tokens, and the trained language model neural network has been trained to perform a language modeling task on a training input sequence of text embedding vectors that represent text tokens from the vocabulary of text tokens, and each image embedding vector in the image sequence of image embedding vectors has the same dimensionality as the text embedding vector that represents a text token from the vocabulary of text tokens.

[0007] Thus, an image encoder neural network is used to represent an image as a sequence of embeddings, so that a pre-trained language model, prompted with an input containing this sequence (also called a "prefix"), generates an appropriate output for the corresponding multimodal task.

[0008] In some implementations, the image encoder neural network is trained by backpropagating gradients of a loss function with respect to parameters of the image encoder neural network through the trained language model neural network, the loss function being a loss function for a multimodal task that requires using the trained language model neural network to generate training output text sequences that include multiple text tokens by processing training inputs that include a sequence of training image embedding vectors generated by the image encoder neural network from training images. This allows the image encoder to learn to generate embeddings that can be effectively used to perform the task, even though the trained language model neural network was trained to process embeddings that represent text, and the embeddings generated by the image encoder represent images.

[0009] In some of these implementations, the multimodal task is an image captioning task, and the training output text sequences are at least a portion of text captions for the training images.

[0010] In some implementations, the trained language model neural network is not trained with a loss function for multimodal tasks during the training of the image encoder neural network; training of the image encoder neural network involves backpropagating gradients of the loss function through the trained language model neural network and into the image encoder neural network without adjusting parameters of the language model neural network. That is, the language model neural network is "frozen" during the training of the image encoder neural network and does not need to be fine-tuned after the image encoder neural network is trained to ensure high-quality performance on any of a variety of multimodal tasks. The resulting combined model retains all the capabilities of large-scale language models, but is also capable of processing text and image inputs in any arbitrary sequence to perform a wide variety of multimodal tasks.

[0011] In some implementations, the loss function measures the likelihood of each assigned to a text token in the ground truth text output for the multimodal task by the score distribution over the text tokens in the vocabulary of text tokens generated by the trained language model neural network as a result of processing the training input.

[0012] In some implementations, the trained language model neural network is trained on a language modeling task prior to training the image encoder neural network on a loss function for a multimodal task.

[0013] In some implementations, the language model neural network is an autoregressive neural network that autoregressively generates an output sequence of text tokens by (i) generating each particular text token in the output sequence conditional on a current input sequence that includes an input sequence of input embedding vectors followed by the text embedding vectors of any text tokens that precede the particular text token in the output sequence.

[0014] In some implementations, the trained language model neural network is an autoregressive transformer-based neural network that includes multiple layers, each applying a self-attention operation. Thus, the described approach leverages the "knowledge" encoded in large, pre-trained transformer-based language models to generate combined models capable of performing a wide variety of multimodal tasks in both zero-shot and few-shot settings.

[0015] In some implementations, the trained language model neural network is configured with multiple layers to apply relative position encoding to the input sequence prior to processing the input sequence.

[0016] In some implementations, the image encoder neural network includes a convolutional neural network.

[0017] In some implementations, the convolutional neural network is configured to process input images to generate output tensors that characterize the input images, and the image encoder neural network is configured to map the output tensors to an image sequence of image embedding vectors.

[0018] In some implementations, the output sequence of text tokens represents at least a portion of a predictive text caption for the input image.

[0019] In some implementations, the input sequence of input embedding vectors also includes (i) one or more sequences of text embedding vectors, each representing a respective text sequence, (ii) one or more other sequences of image embedding vectors, each characterizing another input image, or (iii) both.

[0020] In some implementations, the input sequence of input embedding vectors includes an image sequence of image embedding vectors and a sequence of text embedding vectors that represent questions about the input images, and the output sequence of text tokens represents predicted text answers to the questions about the input images.

[0021] In some implementations, the input sequence of input embedding vectors includes (i) one or more other sequences of image embedding vectors and (ii) for each other sequence of image embedding vectors, a respective sequence of text embedding vectors that represents a classification for another image characterized by the other sequence of image embedding vectors, and the output sequence of text tokens represents a predicted classification for the input image.

[0022] In some implementations, the input sequence of input embedding vectors includes (i) one or more other sequences of image embedding vectors, each characterizing a respective additional input image, and (ii) for each other sequence of image embedding vectors, a respective sequence of text embedding vectors that represents an answer to a question when asked about the additional image characterized by the other sequence of image embedding vectors, and the output sequence of text tokens represents a predicted answer to the question when asked about the input image.

[0023] In some implementations, the input sequence of input embedding vectors includes (i) one or more other sequences of image embedding vectors, each characterizing a respective additional input image; (ii) a respective sequence of text embedding vectors, for each other sequence of image embedding vectors, providing a particular type of information about the additional image characterized by the other sequence of image embedding vectors; and (iii) a partial sequence of text embeddings, for each image sequence of image embedding vectors, representing a text prompt for providing a particular type of information about the input image, and the output sequence of text tokens represents a predicted completion of the text prompt that provides a particular type of information about the input image.

[0024] In some implementations, the input sequence of input embedding vectors includes a first sequence of text embedding vectors representing a text sequence that describes a task to be performed by the text response represented by the output sequence.

[0025] The subject matter described herein can be implemented in particular embodiments to realize one or more of the following advantages.

[0026] When trained at sufficient scale, language models—e.g., transformer-based language models that generate outputs via autoregression with self-attention—exhibit a remarkable ability to learn novel language tasks after being prompted with just a few examples. The described technique transfers this short-shot learning ability to a multimodal setting by using a visual encoder to represent images as a sequence of continuous embeddings, such that a pre-trained, frozen language model prompted with a "prefix" containing this sequence generates appropriate outputs for the corresponding multimodal task. The resulting system is a multimodal short-shot learner with a remarkable ability to learn a variety of novel tasks, given examples represented as sequences of any number of interleaved image and text embeddings. As specific examples, the resulting system can rapidly "learn" vocabulary for new objects and novel visual categories, perform visual question answering with only a small number of examples, and utilize external knowledge without any additional training or fine-tuning after the image encoder has been trained. The resulting system can therefore effectively perform multimodal tasks even if (i) the language model was only trained on a language modeling task that operates only on text data, and (ii) the image encoder neural network was trained while performing an entirely different multimodal task.

[0027] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0028] [Figure 1] FIG. 1 is a diagram of an exemplary multimodal system. [Figure 2] FIG. 1 illustrates training of an image encoder neural network. [Figure 3] 1 is a flow diagram of an example process for performing a multimodal task. [Figure 4] FIG. 1 illustrates an example of an image encoder neural network and a language model neural network used to perform a visual question answering task. [Figure 5] FIG. 1 illustrates an example of an image encoder neural network and a language model neural network used to perform a k-shot external knowledge visual question answering task. [Figure 6] FIG. 1 illustrates an example of an image encoder neural network and a language model neural network used to perform a k-shot image classification task. DETAILED DESCRIPTION OF THE INVENTION

[0029] Like reference numbers and designations in the various drawings indicate like elements.

[0030] This specification describes a system, implemented as a computer program on one or more computers at one or more locations, that performs one or more multimodal tasks.

[0031] 1 is a diagram of an exemplary multimodal system 100. Multimodal system 100 is an example of a system in which the systems, components, and techniques described below are implemented, which may be implemented as a computer program on one or more computers at one or more locations.

[0032] The multimodal system 100 is configured to perform one or more multimodal tasks using an image encoder neural network 110 and a language model neural network 120 .

[0033] A multimodal task, as used herein, is a task that requires processing a network input 102 that includes images 104 (e.g., real-world images captured by a camera) to produce a network output 152 that includes a sequence of text tokens 154. In some cases, the network input 102 may also include text, one or more other images, or both.

[0034] Examples of multimodal tasks are described in more detail below.

[0035] The image encoder neural network 110 is a neural network configured to receive an image and to process the image, for example, by processing intensity values ​​of pixels of the image, to generate an image sequence of image embedding vectors that characterize the image.

[0036] An "embedding vector," as used herein, is a vector of numeric values, e.g., floating point or other type of numeric values, having a predetermined number of dimensions, e.g., having a predetermined number of values.

[0037] The image encoder neural network 110 can generally have any suitable architecture that maps an image to a sequence of embeddings, which can have a predetermined number of embeddings, for example, 1, 2, 4, or 8.

[0038] For example, the image encoder neural network 110 may include a convolutional neural network.

[0039] As a specific example, a convolutional neural network may be configured to process an image to generate an output tensor that characterizes the input image, and the image encoder neural network 110 may be configured to map the output tensor to an image sequence of image embedding vectors. That is, the neural network 110 may include (i) a convolutional neural network and (ii) an additional component that maps the output tensor to the image sequence. For example, the additional component may apply a linear transformation to linearly map the output tensor to D*n channels and then reshape the result as a sequence of n embeddings, each with dimensionality D.

[0040] As another example, the image encoder neural network 110 may be a vision transformer neural network. A vision transformer is a neural network that takes multiple image patches of an image, generates a respective embedding for each image patch, and then updates each embedding using one or more self-attention layer blocks (e.g., ordered in a sequence such that the output of each self-attention layer block except the final self-attention layer block is the input of the subsequent self-attention layer block in the sequence). In this example, the sequence of embeddings may be updated embeddings for some or all of the patches generated by the final self-attention layer block.

[0041] The language model neural network 120 is a neural network configured to receive as input a sequence of input embedding vectors, and to process the input to produce as output an output sequence of text tokens.

[0042] Each input embedding vector has the same dimensionality as the image embedding vector generated by the image encoder neural network 110 .

[0043] As will be discussed in more detail below, during training of the language model neural network 120, the input embedding vectors represent text tokens, and after training and while performing multimodal tasks, at least some of the input embedding vectors are image embedding vectors.

[0044] More specifically, system 100 may also include a text embedder, which is an embedding function that independently converts each token in the vocabulary of tokens into a respective text embedding having the same dimensionality as the input embedding. The text embeddings may be fixed, e.g., predetermined, or may be learned during training of language model neural network 120.

[0045] The tokens in the vocabulary may be any suitable text tokens, such as, for example, words, word fragments, punctuation marks, etc., that represent elements of text in one or more natural languages, and optionally, numbers and other text symbols found in a corpus of text. For example, the system may tokenize a given sequence of words by applying a tokenizer, e.g., a SentencePiece tokenizer or another tokenizer, to split the sequence into tokens from the vocabulary.

[0046] The language model neural network 120 may be an autoregressive neural network that autoregressively generates an output sequence of text tokens by generating each particular text token in the output sequence conditional on a current input sequence that includes (i) an input sequence of input embedding vectors, followed by (ii) the text embedding vectors (generated by the text embedder) of any text tokens that precede the particular text token in the output sequence.

[0047] More specifically, to generate the particular text token, neural network 120 may process the current input sequence to generate a score distribution, e.g., a probability distribution, that assigns a respective score, e.g., a respective probability, to each token in a vocabulary of text tokens. Neural network 120 may then use the score distribution to select a text token from the vocabulary as the particular text token. For example, neural network 120 may greedily select the highest-scoring token or may sample the token from the distribution using, for example, kernel sampling or another sampling technique.

[0048] As a specific example, the trained language model neural network 120 may be an autoregressive transformer-based neural network that includes multiple layers, each applying a self-attention operation. The neural network 120 may have any of a variety of transformer-based neural network architectures. Examples of such architectures are Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu, "Exploring the limits of transfer learning with a unified text-to-text transformer," arXiv preprint arXiv:1910.10683, 2019; Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le, "Towards a human-like open-domain chatbot," CoRR, abs / 2001.09977, 2020; and Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, and Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell et al., "Language models are few-shot learners," arXiv preprint arXiv:2005.14165, 2020.

[0049] Optionally, language model neural network 120 can apply relative positional encoding to the input sequence (or the current input sequence) prior to processing the input sequence (or the current input sequence) using multiple layers. Applying relative positional encoding to the input sequence refers to modifying each embedding in the input sequence based on the embedding's position in the input sequence to inject information about the embedding's position, for example, prior to applying any self-attention operations. For example, neural network 120 can combine, e.g., add or concatenate, each input embedding with a positional embedding that corresponds to the input embedding's position in the input sequence.

[0050] Prior to using image encoder neural network 110 and language model neural network 120 to perform one or more multimodal tasks, neural networks 110 and 120 are trained, for example, by system 100, by one or more other systems, or partly by system 100 and partly by one or more other systems.

[0051] Specifically, system 100 or one of the other systems pre-trains language model neural network 120 on a language modeling task, e.g., a task that seeks to predict the next token that follows the current sequence in training data, given a current sequence of text tokens. As a specific example, language model neural network 120 may be pre-trained with a maximum likelihood function on text, e.g., a large dataset of publicly available text from the Internet or another text corpus.

[0052] Therefore, during this pre-training, the input sequences of the neural network 120 only contain text embeddings of text tokens from the vocabulary.

[0053] Because the text used for pre-training does not need to be labeled, and because large amounts of unlabeled text are readily available, for example, from the Internet or in other large-scale text corpora, language model neural network 120 may be pre-trained on a large set of training data, and the trained language model neural network 120 can thus encode large amounts of text-based knowledge.

[0054] After training language model neural network 120, system 100 or another system trains image encoder neural network 110 with training data for the multimodal task while language model neural network 120 remains "frozen," meaning that during this training, the parameters of image encoder neural network 110 are adjusted, but the parameters of language model neural network 120 remain fixed.

[0055] This training of the image encoder neural network 110 is described in more detail below with reference to FIG.

[0056] After training, to perform a multimodal task, the system 100 receives an input 102 including an input image 104 and processes the input image 104 using an image encoder neural network 110 .

[0057] As described above, the image encoder neural network 110 is trained to process the input image 104 and generate an image sequence of image embedding vectors 112 that characterize the input image 104 .

[0058] If the input 102 also includes one or more other input images, the system 100 also processes each other input image using the image encoder neural network 110 to generate a respective sequence of image embedding vectors for each other input image.

[0059] The system 100 then generates an input sequence of input embedding vectors 114 that includes the image sequence of image embedding vectors 112 .

[0060] Specifically, depending on the multimodal task that system 100 is performing, the input sequence may include only the image sequence of image embedding vectors 112, or it may include (i) one or more sequences of text embedding vectors, each representing a respective text sequence, (ii) one or more other sequences of image embedding vectors, each characterizing a different input image, or (iii) both. That is, each input embedding vector 114 has the same dimensionality as the image embedding vector 112, but for some tasks, the embedding vector 114 includes other text or image embeddings in addition to the image embedding vector 112.

[0061] That is, even if the image encoder neural network 110 is trained only for a single multimodal task, the system 100 can perform multiple different multimodal tasks after training by appropriately generating input sequences.

[0062] The system 100 uses the trained language model neural network 120 to process an input sequence of input embedding vectors 114, including an image sequence of image embedding vectors 112, to generate, as at least a portion of the network output 152, an output sequence of text tokens 154 representing text responses to the input sequence of input embedding vectors 114.

[0063] Specifically, as described above, each text token 154 in the output sequence is selected from a vocabulary of text tokens, and the trained language model neural network 120 is trained to perform a language modeling task on a training input sequence of text embedding vectors representing text tokens from the vocabulary of text tokens. By doing so, the system 100 effectively conditions the language model 120 with a representation of the input images 104, such that each image embedding vector 112 in the image sequence of image embedding vectors 112 has the same dimensionality as the text embedding vector representing a text token from the vocabulary of text tokens.

[0064] Thus, even if the language model neural network 110 is trained only on text input sequences, at inference time the system 100 uses the language model neural network 110 to effectively perform multimodal tasks.

[0065] FIG. 2 illustrates the training of the image encoder neural network 110.

[0066] As described above, after the language model neural network 120 is pre-trained on a language modeling task, the image encoder neural network 110 is trained on a multimodal task.

[0067] The training data for a multimodal task includes multiple training examples, each of which includes (i) a training input for the task, including one or more images, and (ii) ground truth text for the training input, e.g., text to be generated by performing the multimodal task on the training input.

[0068] For example, the multimodal task may be an image captioning task that requires generating text captions for input images, where each training input may include a single image and the ground truth text for the training input may be (at least a portion of) the ground truth text caption for the single image.

[0069] Specifically, the multimodal task calls for generating a training output text sequence comprising a plurality of text tokens by processing a training input 204, which includes a sequence of training image embedding vectors 206 generated by the image encoder neural network 110 from training images 202, using the trained language model neural network 120. When the multimodal task is an image captioning task, the training output sequence may be (at least a portion of) a text caption for the training image 202.

[0070] To train the image encoder neural network 110 on a multimodal task, the system backpropagates gradients of the loss function for the multimodal task with respect to the parameters of the image encoder neural network 110 through the trained language model neural network 120. The parameters of the image encoder neural network 110 include weights and, in some cases, biases of the layers of the image encoder neural network 110, such as kernels of any convolutional layers in the network 110, weight matrices of any fully connected layers in the network 110, etc.

[0071] Specifically, in each of multiple training iterations, the system may sample training examples for the training data and use the sampled training examples to compute gradients of a loss function for the multimodal task with respect to the parameters of the image encoder neural network 110, for example, by backpropagating through the trained language model neural network 120. The system may then update the parameters of the image encoder neural network 110 by applying an optimizer to the computed gradients to generate updates, and then adding or subtracting the updates from the current values ​​of the parameters.

[0072] As a specific example, a loss function for a multimodal task may measure the likelihood of each assigned to a text token in the ground truth text output for the multimodal task by the score distribution over the text tokens in the vocabulary of text tokens generated by the trained language model neural network 120 as a result of processing the training input.

[0073] For example, when the language model neural network 120 is an autoregressive neural network as described above, the loss function may be a negative log-likelihood loss, which measures, for each particular token in the ground truth text output for a given training input, the negative logarithm of the probability assigned to the particular token by the score distribution generated by the neural network 120 by processing an input sequence that includes embeddings of (i) the given training input and (ii) any tokens that precede the particular token in the ground truth text output.

[0074] FIG. 2 shows an example of training an image encoder neural network (“visual encoder”) 110 with training input for an image captioning task, including (i) a training image 202 and (ii) an opening portion 204 of a caption for the training image 202.

[0075] As shown in FIG. 2, the system utilizes (i) a text embedder 210 that receives text tokens and generates a respective embedding for each of the text tokens, and (ii) a language model neural network 120, which in the example of FIG. 2 is a transformer-based neural network, e.g., a neural network including a set of one or more self-attention layers that apply (causally masked) self-attention, as described above and in more detail below.

[0076] In the example of FIG. 2, the training image 202 is an image of a red boat, the opening part of the caption 204 is "A small red boat," and the task asks the neural network 120 to generate an output that completes the caption, e.g., to produce the completion 206 "on the water."

[0077] As shown in FIG. 2, the system processes training images 202 using image encoder 110 to generate a sequence of image embeddings that includes two image embeddings 222 .

[0078] The system then processes the training input sequence, which includes embeddings 222 and 224 followed by the text embedding of opening portion 204, and evaluates a loss function that is a combination, e.g., the sum, of the respective losses for each of the tokens in closing 206, e.g., the respective losses for token "on" 232, token "the" 234, and token "water" 236.

[0079] The loss for a given token may be, for example, the negative logarithm of the probability assigned to the given token by the score distribution generated by neural network 120 by processing an input sequence that includes embeddings of (i) the given training input and (ii) any token that precedes the given token in the ground truth text output.

[0080] The system then backpropagates the loss through language model neural network 120 while keeping language model neural network 120 (and text embedder 210) frozen to compute the respective gradients for each of image embeddings 222 and 224. The system can then backpropagate the gradients through image encoder neural network 110 to determine the gradients with respect to the parameters of neural network 110.

[0081] Thus, as shown in FIG. 2, the trained language model neural network 120 is not trained during the training of the image encoder neural network 110 (nor is the text embedder 210) with a loss function for the multimodal task, and the system backpropagates the gradient of the loss function through the trained language model neural network 120 and into the image encoder neural network 110 without adjusting the parameters of the language model neural network 120.

[0082] Although the example in Figure 2 shows processing for a single training example, in practice the system may process multiple training examples in parallel in a given training step, and the overall gradient calculated for the training step may be the sum of the respective gradients for each of the training examples.

[0083] 2 as words, the tokens may be any suitable text tokens, e.g., words, word fragments, punctuation, etc. For example, the system may tokenize a given sequence of words by applying a tokenizer, e.g., a SentencePiece tokenizer or another tokenizer, to split the sequence into tokens from a vocabulary.

[0084] 3 is a flow diagram of an exemplary process 300 for performing a multimodal task. For convenience, process 300 is described as being performed by a system of one or more computers at one or more locations. For example, a multimodal system suitably programmed in accordance with this specification, such as multimodal system 100 shown in FIG. 1, can perform process 300.

[0085] The system acquires an input image (step 302).

[0086] The system processes the input image using an image encoder neural network that has been trained to process the input image to generate an image sequence of image embedding vectors that characterize the input image (step 304).

[0087] The system generates an input sequence of input embedding vectors, which includes an image sequence of image embedding vectors (step 306).

[0088] The way in which the system generates the input sequence depends on the multimodal task it is performing.

[0089] Generally, for at least some tasks, in addition to the image sequence of image embedding vectors, the input sequence can also include one or more "prompt" sequences of text embedding vectors, which describe the task to be performed by a text response in the output sequence, e.g., represent a text sequence that provides the language model neural network with information about what task should be performed using the image embedding vectors.

[0090] For example, for an image captioning task or other multimodal task operating on a single image, the input sequence may include (i) only an image sequence of image embedding vectors, or (ii) an image sequence of image embedding vectors followed by or preceded by a "prompt" sequence of text embedding vectors representing a text sequence describing the task to be performed by the text response represented by the output sequence. In the example of an image captioning task, the prompt sequence could represent, for example, "Caption this" or "Caption:".

[0091] As another example, for an image caption completion task, visual question answering task, or other multimodal task operating on a single image and its associated text, the input sequence may include (i) only an image sequence of image embedding vectors and text embeddings of text sequences associated with the images, or (ii) an image sequence of image embedding vectors and text embeddings of text sequences associated with the images followed by, preceded by, or both, one or more "prompt" sequences of text embedding vectors representing text sequences describing the task to be performed by the text response represented by the output sequence. In the example of a visual question answering task, the text embeddings may be preceded by a first prompt sequence representing "Q:" or "Question" and may be followed by a second prompt sequence representing, for example, "A:" or "Answer:".

[0092] Other examples of multimodal tasks and input sequences for those tasks are described below with reference to FIGS.

[0093] The system processes the input sequence of input embedding vectors using the trained language model neural network to generate as output an output sequence of text tokens that represent text responses to the input sequence of input embedding vectors (step 308).

[0094] FIG. 4 shows an example of an image encoder neural network 110 and a language model neural network 120 used to perform a visual question answering task.

[0095] In visual question answering tasks, the input sequence of input embedding vectors includes an image sequence of image embedding vectors for the input images and a sequence of text embedding vectors representing questions about the input images, and the output sequence of text tokens represents predicted text answers to questions about the input images.

[0096] In the example of Figure 4, the system receives an image of a car and a question about the image ("what color is the car?"). The system then generates an input sequence that includes an image sequence of image embedding vectors for the received image and a sequence of text embedding vectors that represent questions about the input image. In the example of Figure 4, the input sequence also includes two prompt sequences that provide information to the neural network 120 about what task is being performed, so that, for example, the overall sequence of text embedding vectors includes an embedding of the text "Question: What color is the car? Answer:".

[0097] The system then processes the input sequence to generate an output sequence of text tokens that represent a predictive text answer ("Blue") to a question about the input image. In the example of Figure 4, the output sequence also includes a designated end-of-sequence ("EOS") token that can be removed before the output is provided to the user.

[0098] Figure 5 shows an example of an image encoder neural network 110 and a language model neural network 120 used to perform a k-shot external knowledge visual question answering task. In the example of Figure 5, k is equal to 1, but more generally, k may be equal to any positive integer, e.g., any integer greater than or equal to 1.

[0099] In this task, the input sequence of input embedding vectors includes, in addition to the embedding of the input image, (i) k other sequences of image embedding vectors, each characterizing a respective additional input image, and (ii) for each other sequence of image embedding vectors, a respective sequence of text embedding vectors that represents the answer to a question when asked about the additional image characterized by the other sequence of image embedding vectors.

[0100] 5, the input sequence includes an embedding of an image of an airplane and an embedding of a text sequence representing a question and answer about the image, e.g., "Q: Who invented this? A: The Wright brothers." The input sequence also includes an input image, e.g., an image of a mobile phone, and a text sequence representing a question about the input image, "Q: Who invented this? A:."

[0101] The system processes the input sequence and produces an output sequence of text tokens that represent the predicted answer to the question when asked about the input image. In the example in Figure 5, the answer is "Steve Jobs."

[0102] In the example of FIG. 5, the output sequence also includes a designated end-of-sequence ("EOS") token that can be removed before the output is given to the user.

[0103] Alternatively, rather than presenting the task as a question and answer, the system can present the contextual text sequence as a statement, e.g., "This was invented by the Wright brothers," and the text sequence representing the question can be presented as an incomplete statement, e.g., "This was invented by," that needs to be completed by the output of the neural network.

[0104] The task in FIG. 5 is called an “external knowledge” task because the answer to the question is not shown in the image and the task therefore requires the language model neural network 120 to use information encoded as a result of pre-training to determine the correct answer.

[0105] Figure 6 shows an example of an image encoder neural network 110 and a language model neural network 120 used to perform a k-shot image classification task. In the example of Figure 6, k is equal to 2, but more generally, k may be equal to any positive integer, e.g., any integer greater than or equal to 1.

[0106] In this task, the input sequence of input embedding vectors includes, in addition to the embedding of the input image, (i) k other sequences of image embedding vectors, each characterizing a respective additional input image, and (ii) for each other sequence of image embedding vectors, a respective sequence of text embedding vectors, representing the text classification ("text label") for the other image.

[0107] 6, the input sequence includes a first image of a first fruit and a label embedding for the first image ("This is a dax.") and a second image of a second fruit and a label embedding for the second image ("This is a blicket."). The input sequence also includes an input image, e.g., an image of a third fruit, and a prompt text sequence for the classification task: "Question: What is this? Answer:."

[0108] The system processes the input sequence and produces an output sequence of text tokens that represent the predicted classification for the input image. In the example of Figure 6, the answer is "This is a dax."

[0109] In the example of FIG. 6, the output sequence also includes a designated end-of-sequence ("EOS") token that can be removed before the output is given to the user.

[0110] More generally, the classification task in the example of FIG. 6 may be framed as a task that asks the neural network to provide a particular type of information about one or more additional images and to provide the same type of information about the input image.

[0111] That is, for these types of tasks, the input sequence of input embedding vectors includes (i) one or more other sequences of image embedding vectors, each characterizing a respective additional input image, (ii) a respective sequence of text embedding vectors, which, for each other sequence of image embedding vectors, provides a particular type of information about the additional image characterized by the other sequence of image embedding vectors, and (iii) a partial sequence of text embeddings, for the image sequence of image embedding vectors, representing text prompts for providing the particular type of information about the input images. The output sequence then represents the predicted completion of the text prompts that provide the particular type of information about the input images.

[0112] While Figures 4-6 show examples of multimodal tasks, these are merely examples of various types of few-shot or zero-shot multimodal tasks that the system can perform without having to retrain the image encoder or language model neural network with training data for these types of tasks.

[0113] The multimodal system 100 of FIG. 1 has several technical applications. For example, the image 104 of the input 102 may be an image of the real world captured by a camera, and the text data 154 may be text describing the real world. Specifically, the text 154 may answer a question about the real world posed by the text contained in the input 102. Thus, one technical application of the multimodal system 100 is its use by a visually impaired user to obtain information about the image 104. For example, the multimodal system 100 may be part of a computer system that obtains the image 104 (e.g., using the computer system's camera). The user can provide the text of the input 102 (e.g., by controlling a data input device or by speaking tokens representing commands, e.g., strings of letters or phonemes, that are converted into text by the computer system). The multimodal system 100 can then generate the output text 154, e.g., as a string of tokens that is converted into an audio signal broadcast to the user by a speaker device. A useful feature of this computer system is its flexibility, as the user may freely define the information that the query processing system should extract from the images 104 .

[0114] Optionally, the computer system may be configured to allow a user to generate a series of multiple inputs 102, each containing the same image 104 but with different text. For example, the text of a first input 102 may be "What is shown in the picture?" If the query processing system generates a corresponding output text 154 (e.g., "A bus") in response to the first input 102, a person may be able to define a second input 102 containing the same image 104 but with new text (e.g., "What destination is on the front of the bus?").

[0115] Optionally, each input 102 in the sequence of inputs 102, except for the first input 102 in the sequence, may include input text and output text 154 from one or more of the previous inputs 102 in the sequence. In this way, a person may engage in an explicit discussion with the multimodal system 100 about the images 104 (typically contained in all of the inputs 102 in the sequence), and at each stage, the multimodal system 100 generates output text 154 that is a reasonable continuation of the earlier part of the discussion.

[0116] Another example technical application of the multimodal system 100 is a captioning system for images 104. The input 102 text may define an image processing task such as "provide a caption that explains what is happening," in which case the output text 154 may include a caption that describes the event or action shown in the image 104.

[0117] In another technical application, image 104 may include text, and the computer system may include an OCR (optical character recognition) system for extracting text from image 104 to generate at least a portion of any text in input 102. For example, if image 104 is an image of a scene including signs with instructions (e.g., the image may be an image of a road including road signs with text), OCR may generate the text for input 102 based on the instructions in the image. In this case, output text 154 may indicate the importance of the instructions given the rest of the scene (e.g., for image 104 being an image of an empty parking lot and a sign indicating that parking is available at this time, output text 154 may be "Please park here").

[0118] In another technical application, the input 102 may instruct the multimodal system 100 to generate code executable by a processor of a computer system as output text 154. For example, the input 102 may include the text "Generate HTML code that, when executed, generates a video image of the object shown in the image."

[0119] A self-attention layer, as mentioned above, is a neural network layer that includes an attention mechanism that operates on the self-attention layer inputs (or inputs derived from the layer inputs) to generate the self-attention layer outputs. The self-attention mechanism can be causally masked such that any given position in the input sequence does not pay attention to (e.g., do not use data from) any positions after that given position in the input sequence. There are many different possible attention mechanisms.Some examples of self-attention layers including attention mechanisms are Vaswani et al., "Attention is all you need," 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, California, USA; Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu, "Exploring the limits of transfer learning with a unified text-to-text transformer," arXiv preprint arXiv:1910.10683, 2019; Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le, "Towards a human-like open-domain chatbot," CoRR, abs / 2001.09977, 2020, and Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell et al., "Language models are few-shot learners," arXiv preprint arXiv:2005.14165, 2020.

[0120] In general, an attention mechanism maps a query and a set of key-value pairs to an output, where the query, key, and value are all vectors. The output is computed as a weighted sum of the values, and the weight assigned to each value is calculated by a compatibility function, e.g., dot product or scaled dot product, of the query and the corresponding key.

[0121] Generally, a self-attention mechanism is configured to associate different positions in the same sequence to determine a transformed version of the sequence as an output. For example, the attention layer input may include a vector for each element of the input sequence. These vectors provide input to the self-attention mechanism, which is used by the self-attention mechanism to determine a new representation of the same sequence for the attention layer output, the output of which also includes a vector for each element of the input sequence. The output of the self-attention mechanism may be used as the attention layer output or may be processed by one or more of a feedforward layer, a skip connection, or a normalization operation to provide the attention layer output.

[0122] In some implementations, the attention mechanism may be implemented using, for example, a matrix W Q Query transformation defined by, for example, matrix W K and, for example, the matrix W V Each of the value transformations defined by Q , a key matrix K=XW containing a respective key for each vector in the input sequence K , and a value matrix V=XW containing the respective values ​​for each vector in the input sequence VThe attention layer is configured to derive a self-attention layer output for each element of the input sequence, which is used to determine an attention sequence for output. For example, the attention mechanism may be a dot-product attention mechanism applied by applying each query vector to each key vector to determine a respective weight for each value vector, and then combining the value vectors using the respective weights to determine a self-attention layer output for each element of the input sequence. The self-attention layer output may be scaled by a scaling factor, e.g., by the square root of the dimensions of the query and key, to implement scaled dot-product attention. Thus, for example, the output of the attention mechanism may be

[0123]

number

[0124] where d is the dimension of the key (and value) vector. In another implementation, the attention mechanism may include an "additive attention" mechanism that computes the compatibility function using a feed-forward network with hidden layers. The output of the attention mechanism may be further processed by one or more fully connected, feed-forward neural network layers.

[0125] The attention mechanism may implement multi-head attention, i.e., apply multiple different attention mechanisms in parallel, and the outputs of these mechanisms may then be combined, e.g., concatenated, with a learned linear transformation that is applied to reduce the original dimensionality if necessary.

[0126] This specification uses the term "configured" in the context of systems and computer program components. A system of one or more computers configured to perform a particular operation or action means that the system has installed software, firmware, hardware, or a combination thereof that, when run, causes the system to perform the operation or action. One or more computer programs configured to perform a particular operation or action means that one or more programs contain instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0127] Embodiments of the subject matter and functional operations described herein can be implemented in digital electronic circuitry, or in tangibly embodied computer software or firmware, or in computer hardware including the structures disclosed herein and their structural equivalents, or in a combination of one or more of these. Embodiments of the subject matter described herein can be implemented as one or more computer programs, e.g., as one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or control the operation of a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to an appropriate receiver apparatus for execution by a data processing apparatus.

[0128] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus may be or further include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, an apparatus may optionally include code that creates an execution environment for a computer program, e.g., code comprising processor firmware, a protocol stack, a database management system, an operating system, or any combination of one or more of these.

[0129] A computer program may also be called or written as a program, software, software application, app, module, software module, script, or code, and may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in portions of files that hold other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple cooperating files, e.g., files that store one or more modules, subprograms, or portions of code. A computer program may be deployed to be executed on one computer located at one site or on multiple computers, or may be distributed across multiple sites and interconnected by a data communications network.

[0130] The term "database" is used broadly herein to refer to any collection of data, although the data need not be structured in any particular way, or even at all, and can be stored on storage devices in one or more locations. Thus, for example, an index database may contain multiple collections of data, each of which can be organized and accessed in different ways.

[0131] Similarly, the term "engine" is used broadly herein to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components and installed on one or more computers at one or more locations. In some cases, one or more computers will be dedicated to a particular engine, and in other cases, multiple engines may be installed and running on the same computer or multiple computers.

[0132] The processes and logic flows described herein may be implemented by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be implemented by special purpose logic circuitry, e.g., FPGAs, or ASICs, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0133] A computer suitable for running a computer program may be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit will receive instructions and data from a read-only memory, a random-access memory, or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by, or incorporated in, special-purpose logic circuitry. Generally, a computer also includes one or more mass storage devices, e.g., magnetic, magneto-optical, or optical disks, for storing data, or is operatively coupled to the mass storage devices to receive data from, transfer data to, or both. However, a computer need not have such devices. Moreover, a computer can be incorporated in another device, e.g., a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0134] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0135] To provide for user interaction, embodiments of the subject matter described herein can be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, as well as a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide user interaction; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. Furthermore, a computer can interact with a user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from the web browser. A computer can also interact with a user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving a reply message in return from the user.

[0136] A data processing apparatus for implementing machine learning models may also include special purpose hardware accelerator units, for example for handling the common and numerically intensive parts of the machine learning training or production, i.e., inference, workload.

[0137] Machine learning models can be implemented and deployed using machine learning frameworks, for example the TensorFlow framework.

[0138] Embodiments of the subject matter described herein can be implemented in a computing system that includes back-end components, e.g., as data servers, or includes middleware components, e.g., application servers, or includes front-end components, e.g., a client computer having a graphical user interface or web browser or app through which a user may interact with an implementation of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), e.g., the Internet.

[0139] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., HTML pages, to a user device, e.g., for purposes of displaying the data to and receiving user input from a user interacting with the device acting as a client. Data generated at the user device, e.g., a result of user interaction, may be received from the device at the server.

[0140] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Also, some features described herein in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described above as working in some combination and even initially claimed as such, one or more features from a claim can, in some cases, be deleted from that combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0141] Similarly, although operations may be illustrated in the figures or recited in the claims in a particular order, this should not be understood as requiring such operations to be performed in the particular order illustrated, or sequentially, or that all of the illustrated operations be performed to achieve desired results. In some situations, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products.

[0142] Specific embodiments of the present subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. By way of example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0143] 100 Multimodal Systems, Systems 110 Image Encoder Neural Network, Neural Network, Visual Encoder, Image Encoder 120 Language Model Neural Networks, Neural Networks, Language Models 210 Text Embedder

Claims

1. 1. A method implemented by one or more computers, comprising: obtaining an input image; processing the input image using an image encoder neural network that has been trained to process the input image to generate a sequence of image embedding vectors that characterize the input image; generating an input sequence of input embedding vectors comprising the sequence of image embedding vectors; processing an input sequence of said input embedding vectors, including said sequence of image embedding vectors, using the trained language model neural network to generate as output an output sequence of text tokens representing text responses to said input sequence of input embedding vectors; each text token in the output sequence is selected from a vocabulary of text tokens; the trained language model neural network is trained to perform a language modeling task on a training input sequence of text embedding vectors representing text tokens from the vocabulary of text tokens; each image embedding vector in the sequence of image embedding vectors has the same dimensionality as the text embedding vector representing the text token from the vocabulary of text tokens; The language model neural network is an autoregressive neural network that autoregressively generates an output sequence of text tokens by generating each particular text token in the output sequence conditioned on a current input sequence that includes: an input sequence of input embedding vectors followed by the text embedding vector of any text token preceding each particular text token in the output sequence. method.

2. the image encoder neural network is trained by backpropagating gradients of a loss function with respect to parameters of the image encoder neural network through the trained language model neural network; 2. The method of claim 1 , wherein the loss function is a loss function for a multimodal task that requires generating, using the trained language model neural network, training output text sequences comprising a plurality of text tokens by processing training inputs comprising a sequence of training image embedding vectors generated by the image encoder neural network from training images.

3. The method of claim 2 , wherein the multimodal task is an image captioning task and the training output text sequences are at least a portion of text captions for the training images.

4. the trained language model neural network is not trained during the training of the image encoder neural network on the loss function for the multimodal task; 3. The method of claim 2, wherein the training of the image encoder neural network comprises backpropagating the gradient of the loss function through the trained language model neural network and into the image encoder neural network without adjusting parameters of the language model neural network.

5. 3. The method of claim 2, wherein the loss function measures the likelihood of each assigned to a text token in the ground truth text output for the multimodal task by a score distribution over the text tokens in the vocabulary of text tokens produced by the trained language model neural network as a result of processing the training input.

6. 3. The method of claim 2, wherein the trained language model neural network is trained on the language modeling task prior to the training of the image encoder neural network on the loss function for the multimodal task.

7. 10. The method of claim 1, wherein the trained language model neural network is an autoregressive transformer-based neural network including multiple layers, each layer applying a self-attention operation.

8. 8. The method of claim 7, wherein the trained language model neural network is configured to apply relative position encoding to the input sequence prior to processing the input sequence with the plurality of layers.

9. The method of claim 1 , wherein the image encoder neural network comprises a convolutional neural network.

10. 10. The method of claim 9, wherein the convolutional neural network is configured to process the input image to generate an output tensor that characterizes the input image, and the image encoder neural network is configured to map the output tensor to the sequence of image embedding vectors.

11. The method of claim 1 , wherein the output sequence of text tokens represents at least a portion of a predictive text caption for the input image.

12. 2. The method of claim 1, wherein the input sequence of input embedding vectors also includes (i) one or more sequences of text embedding vectors, each representing a respective text sequence, (ii) one or more other sequences of image embedding vectors, each characterizing another input image, or (iii) both.

13. 13. The method of claim 12, wherein the input sequence of input embedding vectors includes a sequence of the image embedding vectors and a sequence of text embedding vectors representing questions about the input images, and the output sequence of text tokens represents predicted text answers to the questions about the input images.

14. 13. The method of claim 12, wherein the input sequence of input embedding vectors includes (i) one or more other sequences of image embedding vectors and (ii) for each other sequence of image embedding vectors, a respective sequence of text embedding vectors representing a classification for the other image characterized by the other sequence of image embedding vectors, and the output sequence of text tokens represents a predicted classification for the input image.

15. 13. The method of claim 12, wherein the input sequence of input embedding vectors includes (i) one or more other sequences of image embedding vectors, each characterizing a respective additional input image, and (ii) for each other sequence of image embedding vectors, a respective sequence of text embedding vectors representing an answer to a question when asked about the additional input image characterized by the other sequence of image embedding vectors, and the output sequence of text tokens represents a predicted answer to the question when asked about the input image.

16. 13. The method of claim 12, wherein the input sequence of input embedding vectors includes: (i) one or more other sequences of image embedding vectors, each characterizing a respective additional input image; (ii) a respective sequence of text embedding vectors, for each other sequence of image embedding vectors, providing a particular type of information about the additional input image characterized by the other sequence of image embedding vectors; and (iii) a partial sequence of text embeddings, for the sequence of image embedding vectors, representing a text prompt for providing the particular type of information about the input image; and wherein the output sequence of text tokens represents a predicted completion of the text prompt that provides the particular type of information about the input image.

17. 13. The method of claim 12, wherein the input sequence of input embedding vectors includes a first sequence of text embedding vectors representing a text sequence that describes a task to be performed by the text response represented by the output sequence.

18. one or more computers; and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the respective operations of any one of claims 1 to 17.

19. 18. One or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform each of the operations of the method of any one of claims 1 to 17.

Images