Processing multimodal input using language models

Abstract

We will perform a multimodal task using a neural network. [Solution] A method, system, and apparatus comprising a computer program encoded in a computer storage medium for processing multimodal input using a language model. Specifically, the input includes an image, which is encoded by an image encoder neural network to generate a sequence of image embeddings representing the image. The sequence of image embeddings is given as at least a portion of the input sequence processed by the language model neural network.

Description

Technical Field

[0001] Cross - reference to related applications This application claims the priority of Greek national patent application No. GR20210100374, filed on June 8, 2021. The disclosure of the prior application is regarded as part of the disclosure of this application and is incorporated herein by reference.

[0002] This specification relates to the processing of inputs including images using neural networks.

Background Art

[0005] This specification describes a system implemented as a computer program on one or more computers in one or more locations that performs a multimodal task using an image encoder neural network and a language modeling neural network. The language modeling neural network is pre-trained and, for example, is not trained during the training of the image encoder neural network.

[0006] In one embodiment, one method includes the steps of: acquiring an input image; processing the input image using an image encoder neural network trained to process the input image to generate an image sequence of image embedding vectors that characterize the input image; generating an input sequence of input embedding vectors, which includes an image sequence of image embedding vectors; and processing the input sequence of input embedding vectors, which includes an image sequence of image embedding vectors, using a trained language model neural network to generate an output sequence of text tokens representing a text response to the input sequence of input embedding vectors, wherein each text token in the output sequence is selected from a vocabulary of text tokens, and the trained language model neural network is trained to perform a language modeling task on a training input sequence of text embedding vectors that represent text tokens from a vocabulary of text tokens, and each image embedding vector in the image sequence of image embedding vectors has the same number of dimensions as a text embedding vector representing a text token from a vocabulary of text tokens.

[0007] Therefore, an image encoder neural network is used to represent images as embedded sequences, thereby allowing a pre-trained language model, prompted by an input (also called a "prefix") containing this sequence, to generate appropriate outputs for the corresponding multimodal task.

[0008] In some implementations, the image encoder neural network is trained by backpropagating the gradient of a loss function with respect to the parameters of the image encoder neural network through a trained language model neural network, where the loss function is a loss function for a multimodal task that asks the trained language model neural network to generate a trained output text sequence containing multiple text tokens by processing a training input containing a sequence of training image embedding vectors generated by the image encoder neural network from a training image. This allows the image encoder to learn to generate embeddings that can be effectively used to perform the task, even though the trained language model neural network is trained to process embeddings that represent text, and the embeddings generated by the image encoder represent images.

[0009] In some of these implementations, the multimodal task is an image captioning task, and the training output text sequence is at least a portion of the text captions for the training images.

[0010] In some implementations, the trained language model neural network is not trained during the training of the image encoder neural network with respect to the loss function for multimodal tasks. Training the image encoder neural network involves backpropagating the gradient of the loss function through the trained language model neural network and into the image encoder neural network, without tuning the parameters of the language model neural network. In other words, the language model neural network is "frozen" during the training of the image encoder neural network and does not need to be fine-tuned after the image encoder neural network is trained to ensure high-quality performance on any of the various multimodal tasks. The resulting combinatorial model retains all the capabilities of the large language model but can also process text and image inputs in any arbitrary sequence for performing a wide variety of multimodal tasks.

[0011] In some implementations, the loss function measures the likelihood of each text token being assigned to a text token in a ground truth text output for a multimodal task, by the score distribution across text tokens in a vocabulary of text tokens generated by a trained language model neural network as a result of processing the training input.

[0012] In some implementations, the trained language model neural network is trained on a language modeling task prior to training the image encoder neural network with a loss function for multimodal tasks.

[0013] In some implementations, a language model neural network is an autoregressive neural network that autoregressively generates an output sequence of text tokens by generating each specific text token in the output sequence, given an input sequence of input embedding vectors, where (i) the current input sequence includes an input sequence of input embedding vectors, and each specific text token in the output sequence is followed by a text embedding vector of any text token preceding a particular text token in the output sequence.

[0014] In some implementations, the trained language model neural network is an autoregressive transformer-based neural network containing multiple layers, each applying self-attentional behavior. Therefore, the described method leverages the "knowledge" encoded in a large, pre-trained transformer-based language model to generate a combinatorial model capable of performing a wide variety of multimodal tasks in both zero-shot and few-shot settings.

[0015] In some implementations, the trained language model neural network is configured to use multiple layers to apply relative position coding to the input sequence before processing it.

[0016] In some implementations, image encoder neural networks include convolutional neural networks.

[0017] In some implementations, a convolutional neural network is configured to process an input image and generate an output tensor that characterizes the input image, while an image encoder neural network is configured to map the output tensor to an image sequence of image embedding vectors.

[0018] In some implementations, the output sequence of text tokens represents at least a portion of the predicted text caption for the input image.

[0019] In some implementations, the input sequence of input embedding vectors includes (i) one or more sequences of text embedding vectors, each representing a respective text sequence, (ii) one or more other sequences of image embedding vectors, each characterizing a different input image, or (iii) both.

[0020] In some implementations, the input sequence of input embedding vectors includes a sequence of image embedding vectors for an image sequence and a sequence of text embedding vectors representing questions about the input image, and the output sequence of text tokens represents a predicted text answer to the questions about the input image.

[0021] In some implementations, the input sequence of input embedding vectors includes (i) one or more other sequences of image embedding vectors and (ii) for each other sequence of image embedding vectors, a respective sequence of text embedding vectors representing a classification for another image characterized by another sequence of image embedding vectors, and the output sequence of text tokens represents a predicted classification for the input image.

[0022] In some implementations, the input sequence of input embedding vectors includes (i) one or more other sequences of image embedding vectors, each characterizing a respective additional input image, and (ii) for each other sequence of image embedding vectors, a respective sequence of text embedding vectors representing an answer to a question asked about an additional image characterized by another sequence of image embedding vectors, and the output sequence of text tokens represents a predicted answer to the question when asked about the input image.

[0023] In some implementations, the input sequence of input embedding vectors includes (i) one or more other sequences of image embedding vectors, each characterizing a respective additional input image, (ii) for each other sequence of image embedding vectors, a respective sequence of text embedding vectors that provides a particular type of information about an additional image characterized by another sequence of image embedding vectors, and (iii) a partial sequence of text embeddings representing a text prompt for providing a particular type of information about the input image for the image sequence of image embedding vectors, and the output sequence of text tokens represents a predicted completion of a text prompt that provides a particular type of information about the input image.

[0024] In some implementations, the input sequence of input embedding vectors includes a first sequence of text embedding vectors that represents a text sequence that describes a task to be performed by a text response represented by the output sequence.

[0025] The subject matter described in this specification may be implemented in particular implementations to realize one or more of the following advantages.

[0026] When trained on a sufficient scale, language models, such as transformer-based language models that generate outputs by autoregression using self-attention, exhibit a remarkable ability to learn new language tasks after being prompted with only a few examples. The technique described moves this fusion learning ability to a multimodal setting by using a visual encoder to represent images as sequences of continuous embeddings, thereby allowing a pre-trained frozen language model, prompted with “prefixes” containing this sequence, to generate appropriate outputs for the corresponding multimodal tasks. The resulting system is a multimodal fusion learner with a remarkable ability to learn a variety of new tasks, given any number of examples represented as sequences of interleaved image and text embeddings. Specifically, the resulting system can rapidly “learn” words about new objects and novel visual categories, answer visual questions with only a few examples, and utilize external knowledge without any additional training or fine-tuning after the image encoder has been trained. Therefore, the resulting system can effectively perform multimodal tasks even if (i) the language model is trained only on language modeling tasks that act only on text data, and (ii) the image encoder neural network is trained while performing entirely different multimodal tasks.

[0027] Details of one or more embodiments of the subject matter of this specification are described in the accompanying drawings and the following description. Other features, aspects and advantages of the subject matter will become apparent from the description, drawings and claims. [Brief explanation of the drawing]

[0028] [Figure 1] This is a diagram illustrating an exemplary multimodal system. [Figure 2] This figure shows the training of an image encoder neural network. [Figure 3] This is an illustrative process flowchart for performing a multimodal task. [Figure 4] This figure shows examples of image encoder neural networks and language model neural networks used to perform visual question answering tasks. [Figure 5] This figure shows examples of image encoder neural networks and language model neural networks used to perform k-shot external knowledge visual question answering tasks. [Figure 6] This figure shows examples of image encoder neural networks and language model neural networks used to perform k-shot image classification tasks. [Modes for carrying out the invention]

[0029] The same reference number and name in various drawings refer to the same element.

[0030] This specification describes a system implemented as a computer program on one or more computers located in one or more locations, which performs one or more multimodal tasks.

[0031] Figure 1 shows an exemplary multimodal system 100. The multimodal system 100 is an example of a system that can be implemented as a computer program on one or more computers in one or more locations, in which the systems, components, and techniques described below are implemented.

[0032] The multimodal system 100 is configured to perform one or more multimodal tasks using an image encoder neural network 110 and a language model neural network 120.

[0033] A multimodal task, as used herein, is a task that requires processing a network input 102, which includes an image 104 (for example, a real-world image captured by a camera), to generate a network output 152, which includes a sequence of text tokens 154. In some cases, the network input 102 may also include text, one or more other images, or both.

[0034] An example of a multimodal task will be explained in more detail below.

[0035] The image encoder neural network 110 is a neural network configured to receive an image and process the image, for example, by processing the intensity values ​​of the pixels in the image, in order to generate an image sequence of image embedding vectors that characterize the image.

[0036] As used herein, "embedded vector" refers to a vector of numbers, such as floating-point or other types of numbers, having a predetermined number of dimensions, for example, a predetermined number of values.

[0037] The image encoder neural network 110 can generally have any suitable architecture for mapping images to a sequence of embeddings. The sequence of embeddings can have a predetermined number, for example, 1, 2, 4, or 8 embeddings.

[0038] For example, the image encoder neural network 110 may include a convolutional neural network.

[0039] As a concrete example, a convolutional neural network may be configured to process an image and generate an output tensor that characterizes the input image, and an image encoder neural network 110 may be configured to map the output tensor to an image sequence of image embedding vectors. In other words, the neural network 110 may include (i) a convolutional neural network and (ii) an additional component that maps the output tensor to an image sequence. For example, the additional component may apply a linear transformation to linearly map the output tensor to D*n channels, and then reshape the result as a sequence of n embeddings, each with dimension D.

[0040] As another example, the image encoder neural network 110 may be a vision transformer neural network. A vision transformer is a neural network that takes multiple image patches of an image, generates an embedding for each image patch, and then updates each embedding using one or more self-aware layer blocks (for example, arranged in a sequence such that the output of each self-aware layer block other than the final self-aware layer block becomes the input to a subsequent self-aware layer block in the sequence). In this example, the sequence of embeddings may be updated embeddings for some or all of the patches generated by the final self-aware layer block.

[0041] The language model neural network 120 is a neural network configured to receive a sequence of input embedding vectors as input and to process the input to generate an output sequence of text tokens as output.

[0042] Each input embedding vector has the same number of dimensions as the image embedding vector generated by the image encoder neural network 110.

[0043] As will be discussed in more detail later, during training of the language model neural network 120, the input embedding vectors represent text tokens, and after training and while performing multimodal tasks, at least some of the input embedding vectors are image embedding vectors.

[0044] More specifically, system 100 may also include a text embedder, which is an embedding function that independently transforms each token in a token vocabulary into a text embedding having the same number of dimensions as the input embedding. The text embeddings may be fixed, for example, predetermined, or learned during training of the language model neural network 120.

[0045] Tokens in a vocabulary may be any appropriate text token, such as a word, word fragment, or punctuation mark, that represent one or more elements of text in natural language and, optionally, the number and other text symbols found in a corpus of text. For example, a system can tokenize a given sequence of words by applying a tokenizer, such as a SentencePiece tokenizer or another tokenizer, to split the sequence from a vocabulary into tokens.

[0046] The language model neural network 120 may be an autoregressive neural network that autoregressively generates an output sequence of text tokens by generating each specific text token in the output sequence, given the current input sequence which includes (i) an input sequence of input embedding vectors and (ii) a text embedding vector (generated by a text embedder) of any text token that precedes a particular text token in the output sequence.

[0047] More specifically, to generate a particular text token, the neural network 120 can process the current input sequence to generate a score distribution, e.g., a probability distribution, which assigns a score, e.g., a probability, to each token in the vocabulary of text tokens. The neural network 120 can then use the score distribution to select a text token from the vocabulary as the particular text token. For example, the neural network 120 could greedily select the token with the highest score, or it could sample tokens from the distribution using, for example, nuclear sampling or another sampling technique.

[0048] As a concrete example, the trained language model neural network 120 may be an autoregressive transformer-based neural network containing multiple layers, each applying self-attentional behavior. The neural network 120 may have any of the various transformer-based neural network architectures. Examples of such architectures include Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu, "Exploring the limits of transfer learning with a unified text-to-text transformer," arXiv preprint arXiv:1910.10683, 2019; Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le, "Towards a human-like open-domain chatbot," CoRR, abs / 2001.09977, 2020; and Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Including works by Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, and Amanda Askell, "Language models are few-shot learners," arXiv preprint arXiv:2005.14165, 2020.

[0049] Optionally, the language model neural network 120 may apply relative position coding to the input sequence (or current input sequence) before processing the input sequence (or current input sequence) using multiple layers. Applying relative position coding to the input sequence means modifying each embedding in the input sequence based on the embedding's position in the input sequence in order to inject information about the embedding's position, for example, before applying any self-attentional actions. For example, the neural network 120 may combine, for example, append or concatenate each input embedding with a position embedding that corresponds to the position of the input embedding in the input sequence.

[0050] Prior to using the image encoder neural network 110 and the language model neural network 120 to perform one or more multimodal tasks, the neural networks 110 and 120 are trained, for example, by system 100, by one or more other systems, or partially by system 100 and partially by one or more other systems.

[0051] Specifically, the language model neural network 120 is pre-trained on a language modeling task in which system 100 or one of the other systems is asked to predict the next token following the current sequence in the training data, given the current sequence of text tokens. For example, the language model neural network 120 may be pre-trained on a maximum likelihood function on a large dataset of publicly available text, such as from the internet or another text corpus.

[0052] Therefore, during this pre-training, the input sequence for the neural network 120 consists only of text embeddings of text tokens from the vocabulary.

[0053] Since the text used for pre-training does not need to be labeled, and because large amounts of unlabeled text are readily available, for example, from the internet or in other large-scale text corpora, the language model neural network 120 may be pre-trained on a large set of training data, and the trained language model neural network 120 can therefore encode large amounts of text-based knowledge.

[0054] After training the language model neural network 120, system 100 or another system trains the image encoder neural network 110 with training data for a multimodal task, while keeping the language model neural network 120 "frozen." In other words, during this training, the parameters of the image encoder neural network 110 are adjusted, while the parameters of the language model neural network 120 remain fixed.

[0055] The training of the image encoder neural network 110 is described in more detail below with reference to Figure 2.

[0056] After training, in order to perform a multimodal task, system 100 acquires input 102 containing input image 104 and processes input image 104 using image encoder neural network 110.

[0057] As described above, the image encoder neural network 110 is trained to process the input image 104 and generate an image sequence of image embedding vectors 112 that characterize the input image 104.

[0058] If input 102 also includes one or more other input images, system 100 uses the image encoder neural network 110 to process each of the other input images as well, generating a corresponding sequence of image embedding vectors for each of the other input images.

[0059] The system 100 then generates an input sequence for the input embedding vector 114, which includes an image sequence for the image embedding vector 112.

[0060] Specifically, depending on the multimodal task being performed by system 100, the input sequence may include only the image sequence of the image embedding vector 112, or it may include (i) one or more sequences of text embedding vectors, each representing a different text sequence, (ii) one or more other sequences of image embedding vectors, each characterizing a different input image, or (iii) both. In other words, each input embedding vector 114 has the same number of dimensions as the image embedding vector 112, but for some tasks, the embedding vector 114 includes other text or image embeddings in addition to the image embedding vector 112.

[0061] In other words, even if the image encoder neural network 110 is trained for only one multimodal task, the system 100 can perform multiple different multimodal tasks after training by appropriately generating input sequences.

[0062] The system 100 uses a trained language model neural network 120 to process the input sequence of the input embedding vector 114, which includes the image sequence of the image embedding vector 112, and generates an output sequence of text tokens 154, which represent the text response to the input sequence of the input embedding vector 114, as at least a portion of the network output 152.

[0063] Specifically, as described above, each text token 154 in the output sequence is selected from the text token vocabulary, and the trained language model neural network 120 is trained to perform a language modeling task on a training input sequence of text embedding vectors representing text tokens from the text token vocabulary. In doing so, the system 100 effectively conditions the language model 120 with the representation of the input image 104, so that each image embedding vector 112 in the image sequence of image embedding vectors 112 has the same number of dimensions as the text embedding vector representing a text token from the text token vocabulary.

[0064] Therefore, even if the language model neural network 110 is trained only on text input sequences, during inference, the system 100 uses the language model neural network 110 to effectively perform multimodal tasks.

[0065] Figure 2 shows the training of the image encoder neural network 110.

[0066] As described above, after the language model neural network 120 is pre-trained on a language modeling task, the image encoder neural network 110 is trained on a multimodal task.

[0067] The training data for the multimodal task includes multiple training examples, each of which includes (i) a training input for the task, which includes one or more images, and (ii) ground truth text for the training input, such as text to be generated by performing the multimodal task on the training input.

[0068] For example, a multimodal task may be an image captioning task that asks to generate text captions for input images, where each training input may contain a single image, and the ground truth text for the training input may be (at least a portion of) a ground truth text caption for a single image.

[0069] Specifically, the multimodal task asks the programmer to generate a training output text sequence containing multiple text tokens by processing a training input 204, which includes a sequence of training image embedding vectors 206 generated by an image encoder neural network 110 from a training image 202, using a trained language model neural network 120. When the multimodal task is an image captioning task, the training output sequence may be a text caption (or at least a portion thereof) for the training image 202.

[0070] To train the image encoder neural network 110 on a multimodal task, the system backpropagates the gradient of the loss function for the multimodal task with respect to the parameters of the image encoder neural network 110 through the trained language model neural network 120. The parameters of the image encoder neural network 110 include weights and, in some cases, biases of the layers of the image encoder neural network 110, such as the kernel of any convolutional layer in the network 110, the weight matrix of any fully connected layer in the network 110, etc.

[0071] Specifically, the system can sample training examples for training data in each of multiple training iterations, and use the sampled training examples to compute the gradient of the loss function for the image encoder neural network 110 with respect to the parameters of the image encoder neural network 110 with respect to the parameters of the image encoder neural network 110 by, for example, backpropagating through the trained language model neural network 120. The system can then update the parameters of the image encoder neural network 110 by applying an optimizer to the computed gradient to generate updates, and then by adding the updates to or subtracting the updates from the current values ​​of the parameters.

[0072] As a concrete example, the loss function for a multimodal task can measure the likelihood of each text token being assigned to a text token in the ground truth text output for a multimodal task by the score distribution across text tokens in the vocabulary of text tokens generated by the trained language model neural network 120 as a result of processing the training input.

[0073] For example, if the language model neural network 120 is an autoregressive neural network as described above, the loss function may be a negative log-likelihood loss that measures the negative log of the probability assigned to each particular token in the ground truth text output for a given training input by the score distribution generated by the neural network 120 by processing an input sequence that includes embeddings of (i) the given training input and (ii) any tokens preceding the particular token in the ground truth text output.

[0074] Figure 2 shows an example of training an image encoder neural network ("visual encoder") 110 with a training input for an image captioning task, which includes (i) a training image 202 and (ii) the beginning portion 204 of a caption for the training image 202.

[0075] As shown in Figure 2, the system utilizes (i) a text embedder 210 that receives text tokens and generates an embedding for each of the text tokens, and (ii) a language model neural network 120, the network 120 being a transformer-based neural network, for example, a neural network that includes a set of one or more self-attention layers that apply (causally masked) self-attention, as described above and in more detail below.

[0076] In the example in Figure 2, the training image 202 is an image of a red boat, and the beginning of the caption 204 is "A small red boat". The task asks the neural network 120 to produce an output that completes the caption, for example, to produce the completion 206 "on the water".

[0077] As shown in Figure 2, the system processes the training image 202 using the image encoder 110 to generate a sequence of image embeddings containing two image embeddings 222.

[0078] The system then processes the training input sequence, which includes embeddings 222 and 224, followed by the introductory text embedding 204, to evaluate a loss function, which is a combination of the respective losses for each token in the conclusion 206, for example, the sum of the respective losses for tokens "on" 232, "the" 234, and "water" 236.

[0079] The loss for a given token could be, for example, the negative logarithm of the probability assigned to the given token by the score distribution generated by the neural network 120 by processing an input sequence that includes (i) a given training input and (ii) embeddings of any tokens preceding the given token in the ground truth text output.

[0080] The system then backpropagates the loss through the language model neural network 120 (and text embedder 210), keeping the language model neural network 120 frozen, in order to calculate the respective gradients for each of the image embeds 222 and 224. The system then backpropagates the gradient through the image encoder neural network 110 so that the gradient can be determined with respect to the parameters of the neural network 110.

[0081] Therefore, as shown in Figure 2, the trained language model neural network 120 is not trained during the training of the image encoder neural network 110 (the text embedder 210 is also not trained) in the loss function for the multimodal task, and the system backpropagates the gradient of the loss function through the trained language model neural network 120 and into the image encoder neural network 110 without tuning the parameters of the language model neural network 120.

[0082] The example in Figure 2 shows the processing for a single training example, but in reality, the system can process multiple training examples in parallel during a given training step, and the overall gradient calculated for the training step may be the sum of the individual gradients for each training example.

[0083] As mentioned above, tokens are shown as words in Figure 2, but tokens can be any appropriate text token, such as words, word fragments, or punctuation marks. For example, the system can tokenize a given sequence of words by applying a tokenizer, such as the SentencePiece tokenizer or another tokenizer, to split the sequence from vocabulary into tokens.

[0084] Figure 3 is a flowchart of an exemplary process 300 for performing a multimodal task. For convenience, the process 300 is described as being performed by a system consisting of one or more computers located in one or more locations. For example, a multimodal system appropriately programmed according to this specification, such as the multimodal system 100 shown in Figure 1, can perform the process 300.

[0085] The system acquires the input image (step 302).

[0086] The system processes the input image using an image encoder neural network trained to process the input image to generate an image sequence of image embedding vectors that characterize the input image (step 304).

[0087] The system generates an input sequence of input embedding vectors, which includes an image sequence of image embedding vectors (step 306).

[0088] The way the system generates the input sequence depends on the multimodal task the system is performing.

[0089] Generally, for at least some tasks, the input sequence may include, in addition to the image sequence of image embedding vectors, one or more "prompt" sequences of text embedding vectors. The prompt sequence of text embedding vectors represents a text sequence that describes the task to be performed by the text response of the output sequence, for example, a text sequence that provides information to a language model neural network about what task should be performed using the image embedding vectors.

[0090] For example, for an image captioning task or other multimodal tasks acting on a single image, the input sequence may include an image sequence of image embedding vectors followed by, or preceded by, a “prompt” sequence of text embedding vectors, which represents (i) an image sequence of image embedding vectors only, or (ii) a text sequence describing the task to be performed by a text response represented by the output sequence. In the example of the image captioning task, the prompt sequence could represent, for example, “Caption this” or “Caption:”.

[0091] As another example, for an image caption completion task, a visual question answering task, or other multimodal tasks acting on a single image and text associated with that image, the input sequence may include (i) an image sequence of image embedding vectors and text embedding only of an image-associated text sequence, or (ii) an image sequence of image embedding vectors and text embedding of an image-associated text sequence, followed by, preceded by, or both of, one or more “prompt” sequences of text embedding vectors representing a text sequence describing the task to be performed by the text response represented by the output sequence. In the example of a visual question answering task, the text embedding may be preceded by a first prompt sequence representing “Q:” or “Question,” followed by a second prompt sequence representing, for example, “A:” or “Answer:.”

[0092] Examples of other multimodal tasks and their input sequences are described below with reference to Figures 4-6.

[0093] The system uses a trained language model neural network to process the input sequence of input embedding vectors and generates an output sequence of text tokens as output, which represents the text response to the input sequence of input embedding vectors (step 308).

[0094] Figure 4 shows examples of an image encoder neural network 110 and a language model neural network 120 used to perform a visual question answering task.

[0095] In the visual question answering task, the input sequence of input embedding vectors includes an image sequence of image embedding vectors for the input image and a sequence of text embedding vectors representing a question about the input image, and the output sequence of text tokens represents a predicted text answer to the question about the input image.

[0096] In the example in Figure 4, the system receives an image of a car and a question about the image ("What color is the car?"). The system then generates an input sequence that includes an image sequence of image embedding vectors for the received image and a sequence of text embedding vectors representing the question about the input image. In the example in Figure 4, the input sequence also includes two prompt sequences that give the neural network 120 information about what task is being performed, so that, for example, the overall sequence of text embedding vectors includes the embedding of the text "Question: What color is the car? Answer:".

[0097] The system then processes the input sequence to generate an output sequence of text tokens representing the predicted text answer ("Blue") to a question about the input image. In the example in Figure 4, the output sequence also includes a specified End of Sequence ("EOS") token that can be deleted before the output is given to the user.

[0098] Figure 5 shows examples of an image encoder neural network 110 and a language model neural network 120 used to perform a k-shot external knowledge visual question answering task. In the example in Figure 5, k is equal to 1, but more generally, k can be equal to any positive integer, for example, any integer greater than or equal to 1.

[0099] In this task, the input sequence of the input embedding vectors includes, in addition to the embedding of the input image, (i) k other sequences of image embedding vectors, each characterizing one of the additional input images, and (ii) for each other sequence of image embedding vectors, a sequence of text embedding vectors representing the answer to a question about the additional image characterized by the other sequence of image embedding vectors.

[0100] In the example in Figure 5, the input sequence includes an embedded image of an airplane and an embedded text sequence representing a question and answer about the image, for example, "Q: Who invented this? A: The Wright brothers." The input sequence also includes an input image, for example, an image of a mobile phone, and a text sequence representing a question about the input image, "Q: Who invented this? A:".

[0101] The system processes the input sequence to generate an output sequence of text tokens that represent predicted answers to questions about the input image. In the example in Figure 5, the answer is "Steve Jobs".

[0102] In the example in Figure 5, the output sequence also includes a specified End of Sequence ("EOS") token that can be deleted before the output is provided to the user.

[0103] Alternatively, instead of presenting tasks as questions and answers, the system could present a contextual text sequence as a statement, for example, "This was invented by the Wright brothers," and a text sequence representing a question could be presented as an incomplete statement, for example, "This was invented by," which would need to be completed by the output of a neural network.

[0104] The task in Figure 5 is called an "external knowledge" task because the answer to the question is not shown in the image, and the task therefore asks the language model neural network 120 to use encoded information as a result of pre-training to determine the correct answer.

[0105] Figure 6 shows examples of an image encoder neural network 110 and a language model neural network 120 used to perform a k-shot image classification task. In the example in Figure 6, k is equal to 2, but more generally, k can be any positive integer, for example, any integer greater than or equal to 1.

[0106] In this task, the input sequence of input embedding vectors includes, in addition to the embedding of the input image, (i) k other sequences of image embedding vectors, each characterizing one of the additional input images, and (ii) for each other sequence of image embedding vectors, a sequence of text embedding vectors representing a text classification ("text label") for the other image.

[0107] In the example in Figure 6, the input sequence includes a first image of a first fruit and an embedded label for that first image ("This is a dax."), and a second image of a second fruit and an embedded label for that second image ("This is a blicket."). The input sequence also includes an input image, for example, an image of a third fruit, and a prompt text sequence for a classification task, "Question: What is this? Answer:".

[0108] The system processes the input sequence to generate an output sequence of text tokens that represent a predicted classification of the input image. In the example in Figure 6, the answer is "This is a dax."

[0109] In the example in Figure 6, the output sequence also includes a specified End of Sequence ("EOS") token that can be deleted before the output is provided to the user.

[0110] More generally, the classification task in the example in Figure 6 may be framed as a task that asks a neural network to provide a specific type of information about one or more additional images and to provide the same type of information about the input images.

[0111] In other words, for this type of task, the input sequence of the input embedding vectors includes (i) one or more other sequences of image embedding vectors, each characterizing each additional input image; (ii) each sequence of text embedding vectors, providing a specific type of information about the additional image characterized by the other sequences of image embedding vectors; and (iii) a partial sequence of text embeddings, representing a text prompt for providing a specific type of information about the input image to the image sequence of image embedding vectors. The output sequence then represents the predicted completion of the text prompt providing a specific type of information about the input image.

[0112] Figures 4–6 show examples of multimodal tasks, but these are just examples of various types of fu-shot or zero-shot multimodal tasks that the system can perform without needing to retrain the image encoder or language model neural network with training data for these types of tasks.

[0113] The multimodal system 100 in Figure 1 has several technical applications. For example, the image 104 of input 102 may be an image of the real world captured by a camera, and the text data 154 may be text describing the real world. Specifically, the text 154 may answer questions about the real world presented by the text contained in input 102. Thus, one technical application of the multimodal system 100 is that it is used by a visually impaired user to obtain information about image 104. For example, the multimodal system 100 may be part of a computer system that acquires image 104 (for example, using a camera in the computer system). The user can supply the text of input 102 (for example, by controlling a data input device or by speaking commands, e.g., tokens representing strings or phonemes, which are converted into text by the computer system). The multimodal system 100 can then generate output text 154 as a sequence of tokens which are converted into an audio signal broadcast to the user by a speaker device, for example. A useful feature of this computer system is its flexibility, as users are free to define the information that the query processing system should extract from image 104.

[0114] Optionally, the computer system may be configured to allow the user to generate a series of inputs 102, each containing the same image 104 but with different text. For example, the text of the first input 102 may be "What is shown in the picture?". If the query processing system has generated a corresponding output text 154 (e.g., "A bus") in response to the first input 102, the user may define a second input 102 containing the same image 104 but with new text (e.g., "What destination is on the front of the bus?").

[0115] Optionally, each input 102 in a sequence of inputs 102 may contain input text and output text 154 from one or more of the previous inputs 102 in the sequence, with the exception of the first input 102 in the sequence. In this way, a person may engage in a clear discussion with a multimodal system 100 about an image 104 (usually contained in all of the inputs 102 in the sequence), and at each stage, the multimodal system 100 generates output text 154 which is a reasonable continuation of the initial part of the discussion.

[0116] Another technical application example of the multimodal system 100 is a captioning system for image 104. The text of input 102 may define an image processing task such as "provide a caption that explains what is happening," in which case the output text 154 may include a caption that describes the event or action shown in image 104.

[0117] In another technical application, image 104 may contain text, and the computer system may include an OCR (Optical Character Recognition) system for extracting text from image 104 to generate at least a portion of some text in input 102. For example, if image 104 is an image of a scene containing a sign with a command (for example, the image may be an image of a road containing a road sign with text), the OCR may generate text for input 102 based on the command in the image. In this case, the output text 154 may indicate the importance of the command, given the rest of the scene (for example, if image 104 is an image of an empty parking lot and a sign indicating that parking is possible at this point, the output text 154 may be "Please park here").

[0118] In another technical application, input 102 may instruct the multimodal system 100 to generate code executable by the computer system's processor as output text 154. For example, input 102 may contain the text, "Generate HTML code that, when executed, will produce a video image of the objects shown in the image."

[0119] A self-attention layer, as mentioned above, is a neural network layer that contains an attention mechanism that acts on the self-attention layer input (or input derived from the layer input) to generate the self-attention layer output. The self-attention mechanism can causally mask any given position in the input sequence such that no attention is paid to any positions in the input sequence that follow that given position (e.g., data from there is not used). There are many different possible attention mechanisms.Several examples of self-attention layers, including attention mechanisms, are presented in Vaswani et al., "Attention is all you need," 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, California, USA; Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu, "Exploring the limits of transfer learning with a unified text-to-text transformer," arXiv preprint arXiv:1910.10683, 2019; Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le, "Towards a human-like open-domain chatbot," CoRR. This is described in abs / 2001.09977, 2020, and in Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, and Amanda Askell, "Language models are few-shot learners," arXiv preprint arXiv:2005.14165, 2020.

[0120] Generally, attention mechanisms map queries and sets of key-value pairs to an output, where the query, key, and value are all vectors. The output is computed as a weighted sum of values, with the weights assigned to each value calculated by a compatibility function between the query and the corresponding key, such as a dot product or a scaled dot product.

[0121] Generally, a self-attention mechanism is configured to associate different positions within the same sequence to determine a transformed version of the sequence as its output. For example, the attention layer input may contain a vector for each element of the input sequence. These vectors are input to the self-attention mechanism, which uses the self-attention mechanism to determine a new representation of the same sequence for the attention layer output, which similarly contains a vector for each element of the input sequence. The output of the self-attention mechanism may be used as the attention layer output, or it may be processed by one or more of the following: a feedforward layer, a skip connection, or a normalization operation, to give an attention layer output.

[0122] In some implementations, the attention mechanism is, for example, matrix W Q A query transformation defined by, for example, matrix W K Key transformations defined by, and for example, matrix W V Each of the value transformations defined by is applied to the attention layer input, which is the input data X to the attention layer, to form a query matrix Q = XW containing the respective queries for each vector in the input sequence. Q , a key matrix K=XW containing the respective keys for each vector in the input sequence K , and a value matrix V=XW containing the respective values ​​for each vector in the input sequence. VThe system is configured to derive the following, which are used to determine the attention sequence for the output. For example, the attention mechanism may be a dot product attention mechanism applied by applying each query vector to each key vector to determine the respective weights for each value vector, and then combining the value vectors using the respective weights to determine the self-attention layer output for each element of the input sequence. The self-attention layer output may be scaled by a scaling factor, for example, by the square root of the dimensions of the query and key, in order to implement scaled dot product attention. Thus, for example, the output of the attention mechanism is

[0123]

number

[0124] This can be determined as follows, where d is the dimension of the key (and value) vector. In another implementation, the attention mechanism may include an "additive attention" mechanism that computes the compatibility function using a feedforward network with hidden layers. The output of the attention mechanism may be further processed by one or more fully connected, feedforward neural network layers.

[0125] The attention mechanism may implement multi-head attention, meaning that multiple different attention mechanisms may be applied in parallel. The outputs of these mechanisms may then be combined, for example, concatenated, with learned linear transformations that are applied to reduce the original dimensionality if necessary.

[0126] This specification uses the term “configured” in relation to system and computer program components. A system consisting of one or more computers being configured to perform a particular operation or action means that the system has installed software, firmware, hardware, or a combination thereof that causes the system to perform the operation or action when it is running. A computer program being configured to perform a particular operation or action means that the program contains instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0127] The subject matter and functional operating embodiments described herein can be implemented in digital electronic circuit mechanisms, or in tangibly implemented computer software or firmware, or in computer hardware including the structures disclosed herein and their structural equivalents, or in combination of one or more thereof. Embodiments of the subject matter described herein can be implemented as one or more computer programs, for example, as one or more modules of computer program instructions encoded on a tangible non-temporary storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage board, a random or serial access memory device, or a combination of one or more thereof. Alternatively or additionally, the program instructions may be encoded on artificially generated propagating signals, such as mechanically generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0128] The term "data processing device" refers to data processing hardware and encompasses all kinds of devices, machines, and equipment for processing data, including, for example, a programmable processor, a single computer, or multiple processors or computers. A device may be a special-purpose logic circuit mechanism, such as an FPGA (Field-Programmable Gate Array) or ASIC (Application-Specific Integrated Circuit), or may further include these. Optionally, in addition to hardware, a device may include code that creates an execution environment for computer programs, such as processor firmware, a protocol stack, a database management system, an operating system, or code constituting one or more of these, or a combination thereof.

[0129] Computer programs may be called, or written as, programs, software, software applications, apps, modules, software modules, scripts, or code, and may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A program may, but may not, correspond to a file in a file system. A program may be stored in a file portion that holds one or more scripts stored in other programs or data, such as a markup language document, in a single file dedicated to the program in question, or in multiple collaborative files, such as files that store one or more modules, subprograms, or parts of code. Computer programs may be deployed to run on one computer or on multiple computers located in one place, or they may be distributed across multiple locations and interconnected by data communication networks.

[0130] In this specification, the term “database” is used broadly to refer to any collection of data, but data does not need to be structured in any particular way, or does not need to be structured at all, and can be stored on one or more storage devices in one or more locations. Thus, for example, an index database may contain multiple collections of data, each of which can be organized and accessed in different ways.

[0131] Similarly, in this specification, the term “engine” is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components and installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine, while in other cases, multiple engines may be installed and running on the same computer or on multiple computers.

[0132] The processes and logic flows described herein may be implemented by one or more programmable computers running one or more computer programs to perform their functions by acting on input data and generating outputs. The processes and logic flows may be implemented by special-purpose logic circuit mechanisms, such as FPGAs or ASICs, or by a combination of special-purpose logic circuit mechanisms and one or more programmed computers.

[0133] A computer suitable for running computer programs may be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit will receive instructions and data from read-only memory or random-access memory, or both. Essential elements of a computer are a central processing unit for executing or running instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory can be complemented by or incorporated into special-purpose logic circuit elements. Generally, a computer may include one or more mass storage devices for storing data, such as magnetic, magneto-optical disks, or optical disks, or be operably coupled to mass storage devices to receive data from or transfer data, or both. However, a computer is not required to have such devices. Moreover, a computer can be incorporated into another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, such as a Universal Serial Bus (USB) flash drive, to name just a few.

[0134] Computer-readable media suitable for storing computer program instructions and data include, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks, and all forms of non-volatile memory, media, and memory devices.

[0135] To provide user interaction, embodiments of the subject matter described herein can be implemented on a computer having a display device such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor for displaying information to the user, as well as a keyboard and a pointing device such as a mouse or trackball to which the user can provide input to the computer. User interaction can also be provided using other types of devices, for example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or haptic feedback, and input from the user can be received in any form, including acoustic, voice, or haptic input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from a web browser. The computer can also interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving response messages from the user in return.

[0136] Data processing devices for implementing machine learning models may include, for example, special-purpose hardware accelerator units for processing machine learning training or production, i.e., inference, and the common and numerical computation parts of the workload.

[0137] Machine learning models can be implemented and deployed using machine learning frameworks, such as the TensorFlow framework.

[0138] Embodiments of the subject matter described herein can be implemented in a computing system that includes, for example, a data server as a backend component, or a middleware component, for example, an application server, or a client computer having a frontend component, for example, a graphical user interface or web browser or application that allows a user to interact with the implementation of the subject matter described herein, or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected by digital data communications of any form or medium, such as communication networks. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0139] A computing system may include a client and a server. The client and server are generally separate from each other and typically interact through a communication network. The client-server relationship arises from computer programs running on each computer and having a client-server relationship with each other. In some embodiments, the server transmits data, such as an HTML page, to a user device, for example, to display data to a user interacting with a device acting as a client, and to receive user input from the user. Data generated on the user device, such as the results of user interaction, may be received from the device to the server.

[0140] This specification includes many specific implementation details, but these should not be construed as limiting the scope of any invention or claim, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Furthermore, some features described herein in the context of separate embodiments can be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented separately or in any suitable partial combination in multiple embodiments. Moreover, features are described above as working in combination, and may even be initially claimed as such, but one or more features from a claim may, in some cases, be removed from that combination, and the claimed combination may be a partial combination or a variation of a partial combination.

[0141] Similarly, while operations are shown in the drawings and described in the claims in a specific order, this should not be understood as requiring such operations to be performed in a specific illustrated order or sequentially, or requiring all described operations to be performed to achieve a desired result. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged in multiple software products.

[0142] This document describes specific embodiments of the subject matter. Other embodiments are within the scope of the following claims. For example, the actions described in the claims may be performed in a different order and still achieve the desired results. As an example, the process shown in the accompanying drawings does not necessarily require a specific illustrated order or sequence to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0143] 100 multimodal systems, systems 110 Image encoders, neural networks, visual encoders, image encoders 120 Language Models Neural Networks, Neural Networks, Language Models 210 Text Embedder

Claims

1. A method carried out by one or more computers, Steps to obtain the input image, The steps include: processing the input image using an image encoder neural network trained to process the input image to generate an image sequence of image embedding vectors that characterize the input image; The steps include generating an input sequence of input embedding vectors, which includes an image sequence of the aforementioned image embedding vectors, A step of using a trained language model neural network to process the input sequence of the input embedding vector, which includes the image sequence of the image embedding vector, and to generate an output sequence of text tokens as an output, which represents a text response to the input sequence of the input embedding vector, Each text token in the output sequence is selected from the vocabulary of text tokens, The trained language model neural network is trained to perform a language modeling task on a training input sequence of text embedding vectors representing text tokens from the vocabulary of text tokens. A method comprising steps, wherein each image embedding vector in the image sequence of the image embedding vectors has the same number of dimensions as the text embedding vector representing the text token from the vocabulary of text tokens.

2. The image encoder neural network is trained by backpropagating the gradient of the loss function with respect to the parameters of the image encoder neural network through the trained language model neural network. The method according to claim 1, wherein the loss function is a loss function for a multimodal task which seeks to generate a training output text sequence containing a plurality of text tokens by processing a training input which includes a sequence of training image embedding vectors generated by the image encoder neural network from a training image using the trained language model neural network.

3. The method according to claim 2, wherein the multimodal task is an image captioning task, and the training output text sequence is at least a portion of the text captions for the training images.

4. The trained language model neural network is not trained during the training of the image encoder neural network in the loss function for the multimodal task. The method according to claim 2 or 3, wherein the training of the image encoder neural network includes backpropagating the gradient of the loss function through the trained language model neural network and into the image encoder neural network without adjusting the parameters of the language model neural network.

5. The method according to any one of claims 2 to 4, wherein the loss function measures the likelihood of each text token in the ground truth text output for the multimodal task being assigned by the score distribution across the text tokens in the vocabulary of text tokens generated by the trained language model neural network as a result of processing the training input.

6. The method according to any one of claims 2 to 5, wherein the trained language model neural network is trained on the language modeling task prior to the training of the image encoder neural network with the loss function for the multimodal task.

7. The method according to any one of claims 1 to 6, wherein the language model neural network is an autoregressive neural network that autoregressively generates an output sequence of text tokens by generating each of the particular text tokens in the output sequence, given a current input sequence including an input sequence of the input embedding vectors, where each particular text token in the output sequence is followed by a text embedding vector of any text token preceding any text token in the output sequence.

8. The method according to claim 7, wherein the trained language model neural network is an autoregressive transformer-based neural network comprising multiple layers, each applying a self-attentional behavior.

9. The method according to claim 8, wherein the trained language model neural network is configured to apply relative position coding to the input sequence prior to processing the input sequence using the plurality of layers.

10. The method according to any one of claims 1 to 9, wherein the image encoder neural network includes a convolutional neural network.

11. The method according to claim 10, wherein the convolutional neural network is configured to process the input image to generate an output tensor characterizing the input image, and the image encoder neural network is configured to map the output tensor to an image sequence of image embedding vectors.

12. The method according to any one of claims 1 to 11, wherein the output sequence of text tokens represents at least a portion of a predicted text caption for the input image.

13. The method according to any one of claims 1 to 12, wherein the input sequence of the input embedding vectors includes (i) one or more sequences of text embedding vectors, each representing a different text sequence; (ii) one or more other sequences of image embedding vectors, each characterizing a different input image; or (iii) both.

14. The method according to claim 13, wherein the input sequence of the input embedding vector includes an image sequence of the image embedding vector and a sequence of text embedding vectors representing a question about the input image, and the output sequence of text tokens represents a predicted text answer to the question about the input image.

15. The method according to claim 13, wherein the input sequence of the input embedding vectors includes (i) one or more other sequences of image embedding vectors, and (ii) for each other sequence of image embedding vectors, each sequence of text embedding vectors representing a classification for the other image characterized by the other sequence of image embedding vectors, and the output sequence of text tokens represents a predictive classification for the input image.

16. The method according to claim 13, wherein the input sequence of the input embedding vectors includes (i) one or more other sequences of image embedding vectors, each characterizing each additional input image, and (ii) for each other sequence of image embedding vectors, a sequence of text embedding vectors representing an answer to a question asked about the additional image characterized by the other sequence of image embedding vectors, and the output sequence of text tokens represents a predicted answer to the question asked about the input image.

17. The method according to claim 13, wherein the input sequence of the input embedding vectors includes (i) one or more other sequences of image embedding vectors, each characterizing each additional input image; (ii) each sequence of text embedding vectors, which provides for each other sequence of image embedding vectors a specific type of information about the additional image characterized by the other sequences of image embedding vectors; and (iii) a partial sequence of text embeddings, which represents a text prompt for providing the specific type of information about the input image to the image sequence of the image embedding vectors, and the output sequence of text tokens represents the predicted completion of the text prompt for providing the specific type of information about the input image.

18. The method according to any one of claims 13 to 17, wherein the input sequence of the input embedding vector includes a first sequence of text embedding vectors representing a text sequence describing a task to be performed by the text response represented by the output sequence.

19. One or more computers, A system comprising: one or more storage devices that, when executed by the one or more computers, store instructions causing the one or more computers to perform each of the operations described in any one of claims 1 to 18.

20. A computer-readable storage medium storing instructions, wherein, when executed by one or more computers, the instructions cause one or more computers to perform each of the operations according to any one of claims 1 to 18.

Images