A language model for processing multimodal query input

Abstract

A query processing system is described that receives a query input that includes an input token string and also includes at least one data item having a second, different modality, and generates a corresponding output token string.

Description

[Technical field]

[0001] This specification relates to neural networks configured to process multi-modal query inputs (e.g., a mixture of text and sound / images) and generate outputs that are responses to the query inputs. [Background technology]

[0002] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values ​​of a respective set of parameters.

[0003] Language models are known that utilize neural networks to receive an input token string, which is a sequence of tokens selected from a token vocabulary (e.g., a fragment of text consisting of letters selected from an alphabet (e.g., a natural language alphabet), a fragment of text consisting of subwords or word pieces selected from a corresponding vocabulary, a fragment of text consisting of phonemes selected from a corresponding vocabulary, etc.), and generate an output token string (another sequence of tokens selected from the token vocabulary) that is a perceptible response to the input token string, e.g., a plausible continuation of the input token string or an answer to a question posed by the input token string. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] "Language models are few-shot learners", Brown et al., Conference on Neural Information Processing Systems, 2020. [Non-Patent Document 2] "Attention is all you need," Vaswani et al., 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA [Non-Patent Document 3] "High-Performance Large-Scale Image Recognition Without Normalization," A. Brock et al., 2021 [Non-Patent Document 4] “Learning Transferable Visual Models From Natural Language Supervision”, A. Radford et al., 2021 [Non-Patent Document 5] "Training compute-optimal large language models", J. Hoffmann et al., 2022 [Non-Patent Document 6] "Perceiver: General Perception with iterative attention," A. Jaegle et al., 2021 Summary of the Invention [Means for solving the problem]

[0005] The present disclosure describes a system ("query processing system") implemented as a computer program in one or more computers at one or more locations that receives a query input including an input token string and also including at least one data item having a second, different modality.

[0006] For example, the data items may be images. Each data item may be a still image, e.g., the data item may be a pixel value (e.g., red-green-blue (RGB) value) for each pixel of a pixel array. Alternatively, one or more of the data items may be a video image, e.g., the data may be a pixel value for each pixel of a respective array for each of a number of frames. The images may be captured by imaging the real world with a still or video camera.

[0007] In another possibility, the data item may be an audio signal, which is audio data representing values ​​of an audio waveform at each of a number of times, for example sound captured by a microphone during a period of time.

[0008] In a further possibility, the data items may be video images with respective accompanying soundtracks.

[0009] The query processing system is operable to generate an output token string based on a query input. The input token string and the output token string are respective sequences of tokens selected from a (single) token vocabulary (e.g., both strings may be strings of characters from a Roman alphabet, strings of subwords or wordpieces from a wordpiece vocabulary, etc.). The token vocabulary may be a token vocabulary of a natural language (e.g., Roman letters for English, or Roman letters plus accented Roman letters for French).

[0010] In general, the present disclosure proposes a query processing system to extract data items from a query input and input the data items to a modality network, which is configured to generate from the data items one or more corresponding compressed representations of each data item. The query processing system uses the input token string to generate a prompt input for a data item token processing model. The data item token processing model comprises a stack of processing layers including a token processing layer and a gated mutual attention layer interleaved with the token processing layers. The prompt input may be provided to a first processing layer of the stack. The gated mutual attention layers each receive at least one of the compressed representations and perform a gating operation based on the received compressed representation. The output token string is an output of the data item token processing model.

[0011] The output token string is a perceptible response to the query input. In particular, the input token string may at least partially define a question about the content of at least one of the data items ("subject data items") in the query input, and the output string may be a (correct) response to the question. The question thus defines a data item processing task to be performed on the subject data item.

[0012] A query processing system may be considered as a generalization of a classifier neural network. A classifier neural network typically receives a data item and determines to which of a number of predefined classes the data item belongs (e.g., if the data item is an image showing an object, the classifier may output data indicating to which of a predefined set of classes the object belongs). In contrast, in a query processing system, the input token string of the query input determines the processing tasks that the query processing system should perform on the subject data items to generate the output token string. In other words, the input token string defines the transformations (e.g., in natural language) that are applied to the subject data items to generate the appropriate output token string. From another perspective, the input token string defines the information that the query processing system should extract from the subject data items, and the output token string encodes that information.

[0013] The input token string may explicitly include a question (a data item processing task). For example, if the subject data item is a still image or video, the input token string might be "What activity is being shown?"

[0014] Alternatively or additionally, the input token string may implicitly define a question, for example by giving one or more examples of data item processing tasks. That is, a query input may include one or more "task example" portions in addition to a subject data item. Each task example portion includes a data item and a respective section of the input token string that is the answer to the question (i.e., the result of applying the question to the data item) when the question is asked about the contents of the data item in the example portion.

[0015] For example, each task example part may be an image and a statement about the subject of the image. One possible task example part would be a picture of a chinchilla and a section of the input token string that is the text "This is a chinchilla. It is mainly found in Chile." This task example part implicitly defines the question as "What animal is shown in the image. In what country is that animal usually found?" Note that the question not only yields information about the content of the image, but also information that is not present in the image but is associated with the content (i.e. the country in which the animal shown in the subject data item is usually found).

[0016] Thus, the query processing system identifies a data item processing task which, when applied to a data item in the task example portion, produces a section of the input token string in the task example portion, and the system applies the processing task to the subject data item to generate an output token string. If there are multiple task example portions in the query input, the query processing system identifies a data item processing task which, when applied to each data item in the task example portion, produces a respective section of the input token string in each of the task example portions, and the system applies the processing task to the subject data item to generate an output token string.

[0017] A query input with no task example parts (e.g., a query input that only contains a subject data item and an input token string that explicitly defines a question) may be called a "zero-shot" query input. A query input that contains n example parts (in addition to a subject data item and, optionally, a part of the input token string that contributes to the definition of the question, such as an explicit statement of the question) is called an "n-shot" query input. A "few-shot" query input is one where n is small, e.g., one or two. For example, a query input may consist of the text "Name the artist", then a data item that is a picture of a painting by Picasso, then the word "Picasso", then a subject data item that is a picture of a painting by Salvador Dali. This is a "one-shot" query input with one task example part (a picture of a painting by Picasso and the word "Picasso"). The input token string of the query input is the text "Name the artist Picasso". The text contains a word ("Picasso") that is a section of the input token string that is part of the task example portion.

[0018] If the data items are video images, the query processing system may operate as a video processing system configured to analyze a sequence of video frames to detect objects in the video frames and provide information about the detected objects in response to a question defined by an input token string. The questions may include, for example, a request for a prediction of a future event or state regarding one or more of the objects (e.g., "Will objects X and Y collide?"), or a request for conditional or counterfactual information regarding one or more of the objects (e.g., "What events would [not] occur if object X were modified, moved, or not present?"), or a request for analysis of the video frames to determine a property or characteristic of one or more of the objects (e.g., "How many objects of type Z are moving?").

[0019] The response to the question given as the output token string may, for example, be in the form of a yes / no answer, or may define a probability distribution over a set of possible answers, or the response may define the location of the object. Implementations of the system are flexible in the types of questions that can be defined, and a single implementation of the system can handle multiple different types of questions. By way of example only, the system can be used to predict whether two objects will collide, or how a collision can be avoided. The response to the query can be used in many ways by a human or computer system. For example, the analysis may be useful in itself, or may be used to provide a warning and / or to control the movement of one or more of the objects.

[0020] The query processing system has several other technical applications. For example, it may be used by a person (e.g., a visually impaired person if the data item is an image (still or video) or a hearing impaired person if the data item is an audio signal) to obtain information about a subject data item. For example, the query processing system may be part of a computer system that obtains the subject data item (e.g., in the form of a still or moving image using a camera of the computer system). The person can provide an input token string (e.g., by controlling a data input device or by speaking a command, e.g., a string of letters or tokens representing phonemes, which are converted into an input token string). The query processing system can then generate an output token string, e.g., as a sequence of tokens that are converted into an audio signal broadcast to the person by a speaker device. A useful feature of this computer system is its flexibility, since the person may freely define the information that the query processing system should extract from the subject data item.

[0021] Optionally, the computer system may be configured to allow a person to generate a sequence of multiple query inputs, each including the same subject data item but with different input token strings. For example, if the subject data item is a picture, the input token string of a first query input may be "What does the picture show?". If the query processing system generates a corresponding output token string (e.g., "bus") in response to the first query input, the person may be able to define a second query input including the same subject data item and with a new input token string (e.g., "What destination is displayed on the front of the bus?").

[0022] Optionally, each query entry in the sequence may include the input token strings and output token strings from one or more of the previous query entries in the sequence, except for the first query entry. In this way, a person may engage in an explicit discussion with the query processing system about the subject data items (which will typically be contained in all the query entries in the sequence), and at each stage the query processing system generates an output token string that is a rational continuation of the earlier part of the discussion.

[0023] Another technical application of the query processing system is a captioning system for data items that are video items. An input token string may define a video item processing task as "provide captions for a video that explain what is happening," in which case the output token string may include captions that describe events or actions taking place in the video. If the data item is a video item with a soundtrack, the captions may include a transcription of the soundtrack.

[0024] In another application, the query processing system may generate output data that categorizes a data item into one or more of a plurality of categories, for example, by defining a score for each of the plurality of possible categories for the data item. As another particular application, where the data items include images (still or video) or sounds (audio data describing an audio waveform), the data item processing task (question) may be to provide output data that defines (e.g., as a score) whether an image or video or audio waveform is described by the text contained in the input token string.

[0025] In another technical application, the data item may include a still or video image containing text, and the computer system may include an OCR (Optical Character Recognition) system for extracting the text from the image to generate at least a portion of the input token string. For example, if the data item is an image of a scene containing a sign with instructions (e.g., the image may be an image of a road containing a road sign containing text), the OCR may generate the input token string based on the instructions in the image. In this case, the output token image may indicate the importance of the instructions given the rest of the scene (e.g., for a data item that is an image of an empty parking lot and a sign indicating that parking is available at this time, the output token string may be "Please park here"). In another example, the data item may be an image (e.g., a meme) containing text extracted by the OCR to form the input token string, and the output token string may indicate (e.g., as a score) whether the combination of the image and the text is offensive (e.g., inappropriate, disruptive, or profane).

[0026] In another technical application, the query input may instruct the query processing system to generate code executable by a processor of a computer system as an output token string. For example, the query input may read: “[Image of a dog] generate HTML code that, when executed, generates a video image of the animal exercising” (where the square brackets mark a data item in the query input, i.e., an image of a dog).

[0027] Note that a stack of token processing layers may be operable on its own (i.e., when arranged as a stack in the same order, without an interleaved gated mutual attention layer) to perform token string processing operations that receive an input token string and generate therefrom an output token string. For example, the output token string may be an answer to a question defined by the input token string, or a (perceptible) continuation of the sequence of tokens defined by the input token string, e.g., a continuation of a topic begun by the input token string.

[0028] The token processing layer of the data item token processing model may be a layer generated by known language model training methods (e.g., Brown et al., “Language models are few-shot learners”, Conference on Neural Information Processing Systems, 2020). The term “training” is used here to mean that the parameters of the token processing layer are iteratively adapted, for example, based on a training base of examples illustrating the tasks that the token processing layer should perform. After training of the token processing layer, the data item token processing model may be formed by interleaving a gated mutual attention layer with the trained token processing layer. The gated mutual attention layer may then be trained, for example, jointly with the modality network, based on a training database. Here, training refers to iteratively modifying the variable parameters that define the gated mutual attention layer and the variable parameters that define the modality network (which may be variable parameters that define a compressed representation generation system that is a component of the modality network, as described below). The training is based on a training database of “multi-modal” training examples, as described below. During this training, the token processing layer is preferably “frozen”, i.e., largely unchanged. This means that the amount of data required in the training database is small, for example much smaller than the training database previously used to train the token processing layer.

[0029] Optionally, the data items in a given query input may have different respective modalities, for example, one of the data items may be an audio signal and another may be a (still or moving) image. Optionally, a different modality network may be provided for each respective modality, and the data items in the query input having each respective modality are sent as input to the respective modality network. Each modality network produces a compressed representation of the corresponding received data item, which are sent to some of the gated mutual attention layers. For simplicity, the following description considers the case where data items of one modality are present in the query, and only one modality network suitable for the data items of that modality.

[0030] It should be noted that a data item consisting of a video image with an associated soundtrack may in some implementations be treated as having a single modality and therefore processed as a whole by a single modality network. Alternatively, in other implementations, the video image of each data item may be split from the soundtrack, thereby generating two respective data items of different respective modalities, which are processed by different respective modality networks of the query processing system.

[0031] The (or each) modality network may utilize a pre-trained encoder network that has been pre-trained (e.g., according to known methods) to generate an encoded data item upon receiving the data item. The encoded data item may have a smaller number of components than the data item itself. The encoder network may, for example, comprise one or more stacked convolutional layers, and the encoded data item may be a feature map.

[0032] The modality network may comprise an encoder network and a compressed representation generation system arranged to receive encoded data items and generate an output. The output of the modality network is (or, more generally, is based on) the output of the compressed representation generation system. Using a pre-trained encoder network as part of the modality network and changing it little (or not at all) during training of the compressed representation generation system means that the compressed representation generation system may have fewer parameters than if the encoder network were not present and the compressed representation generation system instead received raw (unencoded) data items of training examples. This parameter reduction means that the training database previously jointly trained the modality network (i.e., the compressed representation generation system part of the modality network), the gated mutual attention layer does not require the same number of training examples, and the training process requires fewer computational resources.

[0033] As mentioned above, the condensed representation system and the gated mutual attention layer may be jointly trained using the training database in an iterative process, where the term "joint training" means that iterative changes to the condensed representation generation system are interleaved with, or near-simultaneously with, corresponding iterative changes to the gated mutual attention layer.

[0034] The training database consists of training examples ("multimodal" training examples), each of which includes at least one data item and a token string. The token string is a string of tokens from the same token vocabulary as the input token string. During training, one of the training examples may be selected repeatedly. The data items in the training examples are input to the modality network to generate corresponding compressed representations, which are used by the gated mutual attention layer. The token strings of the training examples are split into a first part (representing the first half of the token string) and a second part (representing the second half of the token string). The first part of the token string part is used to generate a prompt input for the data item token processing model. Thus, based on the training examples, the data item token processing model generates an output token string from the compressed representation and the prompt input. The training and gated mutual attention layers of the modality network are then modified to increase the statistical correlation between the output token strings and the second part of the token strings of the training examples.

[0035] For example, if the token string of the training example is denoted as y and the data item of the training example is denoted as x, then the correction is

[0036]

number

[0037] where L is the number of characters in the token string of the training examples, and y l denotes the lth character of the token string, and y <l denotes the l-1th previous character in the token string, and x ≦l denotes the data item preceding the lth character of the token string, and p(y l |y <l ,x ≦l ) is a token processing network that processes data items<l Based on the prompt input, l The output token string begins with and the data item x ≦l Let us denote the probability of generating a compressed representation based on the modality network from

[0038] In practice, preferably, each training iteration is performed using multiple training examples. For example, the training examples in the training database may be partitioned into groups of training examples in the training database ("datasets"). There may be an integer number M of datasets. The mth dataset, m=1,...,M, is denoted by D m This dataset has a set of training examples, λ, which may be the same for all batches, but alternatively may be different for different batches (e.g., lower for batches of training examples taken from sources with lower correlation between text and images). m A corresponding scalar is assigned that weights B. If all datasets have the same quality, it may not be worthwhile to partition the training examples (e.g., M may be set to 1). At each training iteration, B m A respective batch of training examples may be randomly selected from each of the M datasets, and updates are made to the variable parameters of the compressed representation generation system and the gated mutual attention layer to reduce a loss function indicating that the data item token processing model failed to predict the succession of tokens in the batch of training examples.

[0039] For example, the loss function may be the expected negative log-likelihood of a second portion of a token string of training examples given a first portion, averaged over all possible positions l, where the token string of training examples can be divided as follows:

[0040]

number

[0041] In the above equation, E denotes the expectation value. The gradients obtained from equation (2) over all M datasets can be calculated to perform an update step to the variable parameters of both the gated mutual attention layer and the compressed representation generation system.

[0042] The compressed representation generating system may comprise one or more resampler layers arranged, for example, as a stack of resampler layers. Each resampler layer receives the output of the encoder network (e.g., following a transformation to the output of the encoder layer, such as a flattening operation) or the previous layer of the stack. Each of the resampler layers may perform a transformer operation, i.e., each layer includes one or more transformer blocks or self-attention layers. A transformer block typically includes an attention or self-attention neural network layer followed by a feedforward neural network. An attention, or self-attention, neural network layer is a neural network layer that includes an attention, or self-attention, operation (operating over the attention layer inputs to generate the attention layer output). The self-attention operation may be masked such that any given position in the input sequence does not pay attention over any positions after that given position in the input sequence. There are many different possible (self-)attention operations. Some examples of transformer blocks that include attention operations are described in Vaswani et al., "Attention is all you need," 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.

[0043] In general, an attention operation maps a query and a set of key-value pairs to an output, where the query, key, and value are all vectors. The output is computed as a weighted sum of the values, and the weight assigned to each value is calculated by a compatibility function of the query and the corresponding key, e.g., the dot product or scaled dot product.

[0044] In some implementations, the attention operation is configured to apply a self-attention operation to the attention layer input, which may be followed by one or more feed-forward neural network layers to generate the attention layer output. In general, the attention operation determines the relationship between two sequences, and the self-attention operation is configured to relate different positions in the same sequence to determine a transformed version of the sequence as the output. For example, the attention layer input may include a vector for each element of the input sequence. These vectors provide input to the self-attention operation, which is used by the self-attention operation to determine a new representation of the same sequence for the attention layer output, which also includes a vector for each element of the input sequence. The output of the self-attention operation may be used as the attention layer output, or may be processed by one or more feed-forward layers to provide the attention layer output.

[0045] In some implementations, the attention operation may be, for example, Q For example, the query transformation defined by the key matrix W K and, for example, the value matrix W V Each of the value transformations defined by Q , key vector K=XW K , and the value vector V=XW V The attention layer output may be scaled by a scaling factor, e.g., by the square root of the dimensions of the query and key, to implement a scaled dot-product attention. Thus, for example, the output of the attention operation may be

[0046]

number

[0047] where d is the dimension of the key (and value) vector. In another implementation, the attention operation may include an "additive attention" mechanism that uses a feed-forward network with hidden layers to compute the compatibility function. As mentioned above, the output of the attention operation may be further processed by one or more fully connected feed-forward neural network layers.

[0048] The attention computation may implement multi-headed attention, i.e., multiple different attention computations may be applied in parallel. The outputs of these mechanisms may then be combined, e.g., concatenated, with learned linear transformations that are applied to reduce the original dimensionality if necessary.

[0049] Optionally, the first resampler layer of the stack of layers may further receive at least one latent vector (a set of input latent values). The components of the latent vector may be among the variable parameters of the compressed representation system that are trained in the training operation. The latent vector may have a number of components that is smaller (e.g., much smaller) than the number of components of the (e.g., flattened) output of the encoder network. Each resampler layer is trained to generate an output, which also has a number of components that is much smaller than the number of components of the output of the encoder network, for example, may have the same number of components as the layer query vector. Other resampler layers (if any) of the stack receive the output of the preceding resampler layer in the stack.

[0050] Each resampler layer may receive (i) the (e.g., flattened) output of the encoder network and (ii) a “latent input”, which is the output of the previous resampler layer in the stack, or, in the case of the first resampler layer in the stack, a latent vector. Q , W K, and W V may be used to generate (for each of one or more heads) a key vector, a value vector and a query vector from different corresponding ones of data inputs (i) and (ii), as described above (in principle any combination of data inputs (i) and (ii) may be used, with at least one of the key vector, value vector and query vector being based on (i.e. derived from data including) each of data inputs (i) and (ii), and preferably at least one of the key vector, value vector and query vector being based on both data inputs (i) and (ii)). For example, the key vector and value vector may be generated by inputting data input (i) (or the concatenation of data input (i) and data input (ii)) into the matrix W K , and W V and the query vector may be generated by multiplying the data input (ii) with the matrix W Q The output of the resampler layer may then be generated based on the key vector, the value vector, and the query vector, as described above, e.g., by multiplying

[0051]

number

[0052] where d is the number of elements in the key vector. Q , W K , and W V may be among the variable parameters of the compressed representation generation system that are trained in a training process.

[0053] The output of the stack of resampler layers may be passed to an adaptive system, which may be a single-layer or multi-layer perceptron, which generates a compressed representation that is the output of the compressed representation generation system. Indeed, optionally, an adaptive system, such as a single-layer or multi-layer perceptron, may be provided within each of the resampler layers to receive the output of the corresponding attention operation of the resampler layer. In either case, the perceptron may be a substantially fully connected ("dense") network. The perceptron has variable parameters that are among the variable parameters of the compressed representation generation system that are trained in the training process.

[0054] The use of a modality network including one or more resampler layers, as described above, to generate a compressed representation of a data item is an independent aspect of this disclosure.

[0055] Another independent aspect of the disclosure is the use of such a resampler layer in a system for generating an output token string based on a query input including an input token string and one or more data items, where the input token string and the output token string are strings of tokens selected from a token vocabulary, and the data items are modalities other than the tokens selected from the token vocabulary. The method includes generating one or more condensed representations of each data item by processing the data items with a modality network, generating a prompt input including the input token string of the query input, and inputting the prompt input and the condensed representations of each data item to a data item token processing model configured to generate an output token string based on the prompt input and the condensed representations of each data item.

[0056] Moving to gated mutual attention layers, each gated mutual attention layer may propagate the data it receives from the preceding processing layer of the data item token network (or the prompt input, if the gated mutual attention layer is the first processing layer of the data item token network (although in general the first layer of the data item token network is one of the token processing layers)) plus a modification (interaction term) that has a magnitude that depends exactly on the value of the gating parameter (gating value), i.e. the magnitude of the interaction term increases with increasing values ​​of the gating parameter. The interaction term is added component-wise to the input of the gated mutual attention layer. For example, the output of the gated mutual attention layer, upon receiving a (vector) input y from the preceding processing layer, may be y+tanh(αx), where α is the gating parameter (scalar), x is the (vector) output of an adaptive system (described below) that receives y, and tanh is a component-wise tanh operation applied separately to each component of αx.

[0057] When the value of the gating parameter is low, the influence of the data items in the query input is correspondingly low. At the beginning of training, the value of the gating parameter may be chosen to be low (e.g., 0.01) so that the gated mutual attention layer does not interfere too much with the operation of the trained token processing layer of the data item token processing network. During training, as the gated mutual attention layer gradually learns to cooperate with the token processing layer, the value of the gating parameter is gradually increased.

[0058] "Masking" may be applied to at least one (or all) of the gated mutual attention layers, meaning that rather than the adaptive system of gated mutual attention layers generating interaction terms based on compressed representations of all data items in the query input and on data derived from all the prompt inputs, the interaction terms are generated based on compressed representations of only a (true) subset of the data items in the query input and on data based only on a (true) subset of the prompt inputs.

[0059] For example, we assume that the query input may include (or consist of) multiple (non-overlapping) parts, each of which includes exactly one of the data items, and also includes multiple of the tokens of the input token string. For example, each part may be one of the input data items and the tokens of the input token string after that data item up to the next data item, or up to the end of the input token string if that data item is the final data item of the query input. For example, some of the parts may be the aforementioned "task example" parts, and / or one of the parts may include the subject data item. For example, the query input may be "[image of a cat] This is a cat [Image of a dog] This". A first part of the query input is "[Image of a cat] This is a cat" (this part is the task example part, where [Image of a cat] describes an image of a cat), and a second part of the query input is "[Image of a dog] This" (where [Image of a dog] is the subject data item, i.e., an image of a dog). For each portion, the modality network generates at least one respective compressed representation of the corresponding data item, as described above. For masking, one or more of the gated mutual attention layers may be associated with each of the portions. For a given portion, the one or more associated gated mutual attention layers generate interaction terms based only on the compressed representations of the data items in the relevant portion (i.e., without using compressed representations of other data items of the query input) and without utilizing data generated based on tokens of the input token string other than within the relevant portion (i.e., using only tokens included in the portion of the input token string that is within the relevant portion).

[0060] Note that masking can only be applied to interaction terms, so it does not prevent information about the entirety of the prompt input from flowing through the data item token processing model, and therefore all token processing layers have access to this information.

[0061] The adaptive system of the gated mutual attention layer may comprise a cross transformer operation and / or processing by a perceptron (single or multi-layer). For example, the output of the cross transformer operation may be the input to a perceptron, which outputs a value denoted as x above.

[0062] The cross transformer operation is performed by computing a set matrix W for each gated mutual attention layer (for each of one or more heads). Q , W K , and W V Based on the matrix W Q , W K , and W V The components of,,and the variable parameters (if any) of the perceptron of the,gated mutual attention layer are (or at least are among) the,variable parameters of the gated mutual attention layer that are trained,in the training process.

[0063] The cross transformer operation may receive (i) one or more of the compressed representations of the data items in the query input, and (ii) the output of a preceding processing layer, or, if the gated mutual attention layer is the first processing layer in the stack, a prompt input. The cross transformer operation may receive a matrix W Q , W K , and W V may be used to generate (for each of one or more heads) a key vector, a value vector, and a query vector from different ones of the data inputs (i) and (ii), as described above. For example, the key vector and the value vector may be generated by dividing the data input (i) by the matrix W K , and W V and the query vector may be generated by multiplying the data input (ii) with the matrix W Q The output of the mutual transformer operation may then be multiplied based on the key vector, the value vector, and the query vector, as described above, e.g.,

[0064]

number

[0065] where d is the number of components in the key vector.

[0066] As described above, the prompt input includes the input token string. It also includes a "marker" (one or more tokens, e.g., tokens in the same vocabulary as the input token string) that indicate the presence and, optionally, location of a data item within the query input. For example, if the query input is "[image of a cat] This is a cat [Image of a dog] This", where "[image of a cat]" is a data item that is an image of a cat and "[Image of a dog]" is a data item that is an image of a dog, then the prompt input might be " This is a cat marker " or " " (i.e., seven ASCII characters, where ASCII is the token vocabulary of the input token string in this case) is a marker that indicates a data item.

[0067] A technical effect of the present disclosure is to provide a query processing system that has been experimentally demonstrated to provide an accurate manner of processing multi-modal query inputs (i.e., consisting of both tokens and data items of separate, different modalities). Accuracy has been demonstrated to be higher compared to known computer systems designed for specific multi-modal query processing tasks, even for zero-shot query inputs, and even higher for few-shot query inputs.

[0068] Furthermore, because embodiments of the present invention can utilize existing token processing layers and encoder networks, the amount of training data and computational resources required to train them is significantly reduced: the token processing layer (which may contain billions of variable parameters) may be trained based on a known text database, the encoder network may be trained based on a known image database, and only a relatively small training database of multi-modal training examples is needed to train the gated mutual attention layer and modality network.

[0069] Examples of the invention will now be described, by way of illustration only, with reference to the following drawings: [Brief description of the drawings]

[0070] [Figure 1(a)] FIG. 2 illustrates an encoder network. [Figure 1(b)] FIG. 2 illustrates a token processing model. [Diagram 2] FIG. 1(a) illustrates a query processing system comprising a modality network including the encoder network shown in FIG. 1(a) and a data item token processing model including the token processing model shown in FIG. 1(b). [Diagram 3] FIG. 1 illustrates a first modality network. [Figure 4] FIG. 2 illustrates a second modality network. [Diagram 5] FIG. 3 illustrates a pair of layers of a data item token processing model for the query processing system of FIG. 2. [Figure 6] FIG. 2 illustrates a query input to a query processing system and an output token string generated by the query processing system from the query input. [Figure 7] FIG. 2 illustrates a training example used to train the query processing system. [Figure 8]FIG. 1 illustrates a training system for training a query processing system based on training examples. [Figure 9] FIG. 2 illustrates a method for generating an output token string based on a query input. [Figure 10] FIG. 1 illustrates a method for training a query processing system. [Figure 11] FIG. 2 illustrates a process for interacting with a query processing system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0071] The same reference numbers refer to the same items in the various drawings.

[0072] This specification describes a query processing system operable to receive a query input including at least an input token string (a sequence of tokens selected from a vocabulary) and at least one data item having a different modality than the input token string. The data item typically does not include tokens selected from a vocabulary. The query processing system is an example of a neural network system that can be implemented by one or more computer systems at one or more locations.

[0073] The data items may include one or more images (e.g., still or video images captured by a camera) and / or audio data representing values ​​of an audio waveform at each of a number of times, e.g., sound captured by a microphone during a period of time. The query processing system includes a portion (the "modality network") that includes an encoder network for encoding the data items (e.g., as a set of features) and a compressed representation generation system for generating a compressed representation of the data items from the encoded data items. For example, in the case of data items including images, the encoder network may be configured to apply at least one two-dimensional convolution layer to each image. In the case of data items including audio data, the encoder network may be configured to apply a one-dimensional convolution to the audio data.

[0074] The encoder network may be pre-trained in a different neural network system. For example, FIG. 1(a) shows a neural network system including an encoder network 11 and an optional output layer 12. The encoder network 11 receives data items and generates an output therefrom. The optional output layer 12 (if present) receives the output of the encoder network 11 and generates an output of the neural network system. The encoder network 11 may be trained with a database of data items and corresponding desired outputs, such that when a data item is input to the encoder network 11, the encoder network 11 and the output layer 12 (if present) generate the corresponding desired output. During the training process, the output layer 12 (if present) may also be optionally trained. If the output layer 12 is present, the encoder network may be trained to extract features of the data items it receives, and the output layer 12 processes those features to generate the desired output (encoded data items). Following training, the output layer 12 (if present) may be discarded. The encoded data items may have fewer components than the data items from which they are generated. The encoder network 11 may comprise one or more stacked convolutional layers, and the encoded data items may be (e.g., two-dimensional) feature maps.

[0075] In the experiments reported below, the encoder network11 utilized was the “f6” model taken from the paper “High-Performance Large-Scale Image Recognition Without Normalization” by A. Brock et al., 2021. The encoder network11 was pre-trained using a contrastive objective on a dataset of image-text pairs using the binomial contrastive loss from “Learning Transferable Visual Models From Natural Language Supervision” by A. Radford et al., 2021. The output of the final stage, i.e. the 2D spatial grid of features, was “flattened” to form a 1D sequence.

[0076] The query processing system further includes a data item token processing model comprising token processing layers taken from the token processing model. The token processing model may be pre-trained. For example, FIG. 1(b) shows a suitable token processing model comprising an integer number n of token processing layers 131, 132, ... 13n arranged in a stack (sequence). The first token processing layer receives an input token string. Each token processing layer receives the output of the preceding token processing layer, except for the first one. The output of the nth token processing layer is the output token string of the token processing model. The token processing layers may constitute a "language model" (e.g., a "large language model") trained on data, e.g., a large database of natural language data, such that when an input token string (a sequence of tokens from a vocabulary) is input to the first token processing layer, the output token string is an appropriate response. For example, if the input token string is a question, the output token string may be an appropriate answer. In the experiments reported below, the token processing model was the Chinchilla model from "Training compute-optimal large language models," J. Hoffmann et al., 2022. Alternatively, the token processing layer may be a layer generated by another known language model training method (e.g., Brown et al., "Language models are few-shot learners," Conference on Neural Information Processing Systems, 2020).

[0077] Turning to Figure 2, there is shown the structure of a query processing system 200. The query processing system 200 is configured to receive as input a query input 201 that includes one or more data items and an input token string. In one case shown in Figure 2, the query input 201 may include two data items 203, 205, although it should be understood that there may be any number of data items.

[0078] The query processing system 200 extracts two data items 203, 205 from the query input 201 and processes each data item using a modality network. As shown in Figure 2, the query processing system includes two modality networks 207, 209, but alternatively, if all data items have the same modality, there may be only a single modality network that receives the data items 203, 205 and generates a corresponding output from each.

[0079] It should be noted that if there are multiple data items in the query input 201 and they have different corresponding modalities, a modality network may be provided for each corresponding modality, with each modality network configured to receive the data items of the corresponding modality (e.g., sequentially, if there are multiple data items of the corresponding modality in the query input). For example, modality network 207 may be configured to receive a data item of the query input that is a single image (e.g., a still image), and modality network 209 may be configured to receive a data item of the query input that is audio data. In another example, modality network 207 may be configured to receive a data item of the query input that is a single image (e.g., a still image), and modality network 209 may be configured to receive a data item that is a video (i.e., a sequence of images), optionally with a soundtrack of audio data. There may be any desired number of modality networks for each corresponding data item modality that the query processing system 200 is configured to process.

[0080] Each modality network 207, 209 is configured to generate, from each received data item, one or more corresponding compressed representations of the data item. Each modality network may include a pre-trained encoder network 11 (e.g., trained in the system of FIG. 1(a)) and a compressed representation generation system in the form of a respective resampler 210, 211. Each resampler 210, 211 may be, for example, a "perceiver" as described in "Perceiver: General Perception with iterative attention", A. Jaegle et al., 2021. Alternatively, the resamplers 210, 211 may be as described below with reference to FIGS. 3 and 4.

[0081] The resamplers 210, 211 take as input a variable number of image, video or audio features produced from a data item by a corresponding encoder network 11, and produce a compressed representation of the data item having a fixed number of outputs (eg visual outputs).

[0082] 3 shows the structure of a possible modality network 207 when data items 203 are images, including respective "perceptual" resamplers 210. Note that modality network 209 has the same structure when data items 205 are images (indeed, as mentioned above, when data items 203, 205 have the same modality, a single network may serve the role of both modality networks 207, 209).

[0083] The resampler 210 receives the output of the encoder 11 (the encoded version of the data item 203) and converts it to f The perceptual resampler also receives a predefined number of latent vectors (i.e., a set of latent values), each denoted as X, which are learned (trained) during training of the modality network 207 described below.

[0084] The resampler 210 also includes one or more "resampler layers" 300. For simplicity, the resampler 210 is shown in Figure 3 as having a single resampler layer 300. The compressed representation generation system may alternatively comprise multiple resampler layers, each having the structure shown as 300 in Figure 3. The resampler layers 300 may be arranged as a stack of resampler layers, each having the same shape as the resampler layer 300 shown in Figure 3. Each resampler layer 300 (except the first) receives the output of the previous resampler layer in the stack.

[0085] The resampler layer 300 (or, if there are multiple resampler layers 300, each resampler layer in the stack) receives the output of the corresponding encoder network 11 following a transformation to the output of the encoder network 11, such as, for example, a flattening operation (i.e., converting, for example, a two-dimensional array to a 1-D sequence).

[0086] Each of the resampler layers 300 may implement a transformer operation as defined above, i.e. each layer includes one or more transformer blocks or self-attention layers. To this end, each layer comprises a unit 301, which receives the flattened data X f , and alternately, use different ones of the latent vectors X to generate the key vector K(X f , X) and a value vector V(X f, X). This process uses the key matrix and value matrix as described above. A unit 302 uses the latent vector X and the value matrix to generate a query vector Q(X). An attention unit 303 combines K, V and Q by the attention operation described above. The result is added to the latent vector X by summation unit 305. The result is passed to an adaptive unit such as a multi-layer feed-forward (FFW) perceptron 307. The output of the FFW perceptron 307 is added to the output of summation unit 305 by a further summation unit 311. The result is an output vector for each of the latent vectors X, providing different respective compressed representations of the data items. The output of each resampler layer 300 may have the same dimensionality as the dimensionality of the latent variable X. This dimensionality may be much lower than the dimensionality of the encoder network 11.

[0087] Figure 4 shows a variation of the structure of Figure 3 where the input data item 203 is a number of video frames. The data item 203 is split into individual frames 41, 42, 43. Each of these frames is in turn encoded by the encoder network 11 to generate a corresponding encoded frame. Respective data 44, 45, 46 is added to each of the encoded frames to specify the position of the frame within the data item 203. The resulting data is input to a resampler 210, which may have the same structure as described above with reference to Figure 3. It should be noted that although the input to the resampler 210 differs from that in Figure 3, the output for a given latent vector X is the same number of output tokens.

[0088] The operation of the perceptual resampler can be represented by the following pseudocode: Let x_f represent the output of the corresponding encoder network 11. This is an array with dimensionality [T,S,d], where T is an integer variable representing the number of times (i.e. the number of encoder networks 11, or the number of times a single encoder network is used), S is an integer representing the number of positions in the space, and d is a further integer indicating the number of feature values ​​for each position. Let x represent the R learned latent variables and is therefore an array with dimensionality [R, d]. The number of layers in the resampler is denoted as num_layers. First, the time padding is added and the result is flattened: x_f=x_f+time_embeddings x_f=flatten(x_f) This results in an array of dimension [T*S,d] Then, for each integer value i in the range 1 to num_layers: x=x+attention_i(q=x, kv=concatenation([x_f,x])) x = x + ffw_i(x). Here, attention_i represents the attention unit 303 performed by the i-th resampler layer 300 based on the corresponding key, value and query matrix, and ffw_i represents the operation performed by the corresponding perceptron 307 of the i-th resampler layer 300.

[0089] Thus, resamplers 210 map a variable-sized grid of spatial-visual features (spatio-temporal features in the case of FIG. 4) to a fixed number of output tokens, independent of the input image resolution of the number of input video frames. Each resampler 210 has a set of learned latent vectors as queries, and the keys and values ​​are the concatenation of the visual features and the learned latent vectors.

[0090] Returning to Figure 2, the query processing system 200 further includes a data item token processing model 220. This comprises a stack of layers that are the token processing layers 131, 132, ..., 13n of the token processing model of Figure 1(b) interleaved by j gated mutual attention layers 231, 232, ..., 23j. The number of gated mutual attention layers j may be equal to the number of token processing layers n, although in variants n and j may differ from each other, e.g., such that there may be any number of token processing layers between any given pair of gated mutual attention layers.

[0091] The output from the resamplers 210, 211 is used as the control input to the gated mutual attention layer. Optionally, the same output from the resamplers 210, 211 can be used as the control input for all the gated layers 231, 232, ... 23j. This has been implemented in some successful experiments, where 64 latent vectors were used, and the corresponding 64 vectors output by the six resampler layers and the perceptual resamplers 210, 211 were used together as the control input for all the gated mutual attention layers. However, this is not the only possibility. For example, there may be corresponding different (learned) latent vectors to generate the respective control inputs for the corresponding ones of the gated mutual attention layers.

[0092] The input token string is used as the prompt input for data item token processing model 220 and is provided to the first processing layer in the stack, which may be first gated mutual attention layer 231. Data passes through data item token processing model 220 (upwards in FIG. 2) producing an output token string, which is the output of data item token processing model 220.

[0093] Each of the gated mutual attention layers 231, 232, ..., 23j receives at least one of the compressed representations from the modality networks 207, 209 and performs a gating operation based on the received compressed representations.

[0094] Figure 5 shows the structure of two successive layers 523, 513 of the data item token processing model 220 of Figure 2. Layer 523 is a gated mutual attention layer that is one of layers 231, 232, ..., 23j. Layer 513 is one of the token processing layers 131, 132, ..., 13n.

[0095] The gated mutual attention layer 523 receives visual inputs 51, 52 (compressed representations) from each of the modality networks 207, 209. The network further receives a vector that is a linguistic input 53, which may be denoted as y. If the gated mutual attention layer 523 is the first layer in the stack, the linguistic input 53 is typically the input token string (prompt input), or a portion thereof. Otherwise, the linguistic input 53 to the gated mutual attention layer 523 is the output from a preceding layer of the data item token processing model 220.

[0096] The gated mutual attention layer 523 applies an attention function 501 as described above, where the visual inputs 51, 52 may define key and value vectors, and a query vector is defined based on the language input 53. The output of the attention function 501 is added to the language input 53 by a summing unit 502. The result is then input to an FFW network 503, such as a multi-layer perceptron. The output of the FFW network 503 is a vector, which may be denoted x. This is input to a tanh gating unit 504, which applies a component-wise operation where each component is multiplied by a gating parameter α, and a tanh function is applied to the result. The output of the FFW network 504 is added to its input by a summing unit 505. This produces the output of the gated mutual attention layer 523, which is y+tanh(αx) can be written as:

[0097] The token processing layer 513 includes a self-attention layer 506, as in some known systems. The attention layer operates on the key vectors, value vectors, and query vectors generated from the inputs to the token processing layer 513. The output of the self-attention layer 506 is added to its input by summation unit 507, and the result is fed to a feedforward (e.g., multi-layer perceptron) layer 508. The output of the FFW network 508 is added to the input of the FFW network 508 by summation unit 509. This results in the output of the token processing layer 513.

[0098] Note that when the gating parameter α is close to zero, the output of the gated mutual attention layer 523 is very close to the linguistic input 53, so that the query processing system 200 performs almost the same function as the trained token processing model shown in FIG. 1(b). However, when in use (i.e., after the query processing system 200 has been trained), the value of the gating parameter is significantly higher than zero, so that the compressed representations (e.g., visual inputs) 51, 52 influence the output of the token processing layer 513.

[0099] A typical operation of the trained query processing system 200 is shown in Figure 6. The query input 61 includes two data items 203, 205 in the form of still images, of a cat and a dog, respectively. As explained above, the data items 203, 205 are input to respective modality networks 207, 209, or input consecutively to the same modality network to produce corresponding compressed representations of the data items 203, 205. When the data items 203, 205 are in the form of still images, the query processing system 200 uses a modality network of the form shown in Figure 3. If the data items were videos, the modality network could be of the form shown in Figure 4.

[0100] The words of the query input 61 are used to generate an input token string for the query input of the query processing system 200. Optionally, the query input may include one or more "markers" for each data item 203, 205 of the query input 61, for example, to indicate the presence of related data items. For example, the input token string may include This is a very cute dog. This is the ASCII text. Optionally, additional markers can be provided. For example, a marker (e.g., "end of chunk") <eoc>) can be added prior to any image and at the end of the query input. The markers can be formed of strings of tokens from the vocabulary or tokens from outside the vocabulary.

[0101] The query processing system 200 processes the input token string by passing it in sequence through layers 231, 131, 232, 132,....23j, 13n, where the gated layers 231, 232,....23j are controlled based on visual input generated by the modality networks 207, 209 based on the two still images of the corresponding data items 203, 205, i.e., the query input 61. In this manner, the query processing system 200 generates an output token string, such as the output token string 62 of FIG. 6. The output token string is a string of ASCII characters that reads "a very serious cat", which is a logical and grammatical continuation of the query input 61. The first image 203 and the associated text "This is a very cute dog" constitute an "example task". The output token string is the result of performing the task exemplified by the example task on the second data item 205 (the "subject data item").

[0102] Methods for training a query processing system such as the query processing system 200 of FIG. 2 are now discussed. The methods may be implemented by one or more computer systems at one or more locations. As explained above, the encoder network 11 and the token processing layers 131, 132, ..., 13n may be pre-trained as part of the system of FIG. 1(a) and FIG. 1(b). The encoder network 11 and the token processing layers 131, 132, ..., 13n are not further trained during the training of the other layers of the query processing system 200. This means that the amount of data required in the training database is small, e.g., much smaller than the training database previously used to train the token processing layers and / or the encoder network.

[0103] Training is based on a training database of "multimodal" training examples. FIG. 7 shows a training example 71, which optionally includes one or more task examples. A single task example 701 is shown, which includes a data item (a picture of a dog) and text about the data item ("This is a picture of my dog"). Different ones of the training examples may include a different number of task examples or may not include any task examples. Training example 71 further includes a portion 702, which is a data item (a picture of a cat) and text about the data item ("This is a picture of my cat"). If a training example includes a task example, the text of portion 702 has the same semantic relationship to the data items of portion 702 as the text of (each of) the training examples has to the data items of that training example.

[0104] The modality networks 207, 209 and the gated mutual attention layers 231, 232, ... 23j may be trained jointly in the system shown in Figure 8. The system utilizes a training database 801 consisting of training examples, such as those shown in Figure 7, each of which includes at least one data item and a token string (a "multi-modal" training example).

[0105] During training, a series of training iterations are performed. In each iteration, one or more of the training examples may be selected. In a typical implementation, multiple training examples are selected and treated as a batch, but for simplicity, the case where a single training example is selected is discussed here. A splitter unit 802 receives the selected training examples and splits the token string of the selected training examples into a first part, which is the first half of the token string, and a second part, which is the second half of the token string. The data items of the selected training examples and the first part of the token string of the selected training examples together constitute a query input for the query processing system 200. The data items are input to the modality networks 207, 209 of the query processing system 200, and the first part of the token string is used to generate a prompt input for the data item token processing model 220 of the query processing system 200. Thus, based on the selected training examples, the data item token processing model 220 generates an output token string from the compressed representation generated by the modality network and from the prompt input.

[0106] A mismatch unit 803 receives the second portion of the token string of the selected training example from the splitter unit and the output token string, and calculates a measure of mismatch (difference between them).

[0107] An update unit 804 then generates updates to the query processing system 200, i.e., to the modality network and the gated mutual attention layer, that increase the statistical correlation between the output token string and the second portion of the token string of the training example. The process may be repeated for different selections of positions in the token string of the selected training example where the splitter unit 802 splits the token string into the first and second portions, and the updates generated by the update unit 804 may be such as to reduce the inconsistency for more than one of these selections, e.g., the average for the selection results.

[0108] For example, if the token string of the training example is denoted as y and the data item of the training example is denoted as x, then the correction is

[0109]

number

[0110] where L is the number of characters in the token string of the training examples, and y l denotes the lth character of the token string, and y <l denotes the l-1th previous character in the token string, and x ≦l denotes the data item preceding the lth character of the token string, and p(y l |y <l ,x ≦l ) is a token processing network that processes data items <l Based on the prompt input, l The output token string begins with and the data item x ≦l Let us denote the probability of generating a compressed representation based on the modality network from

[0111] Indeed, as mentioned above, preferably each training iteration is performed using multiple training examples. For example, the training examples in the training database may be partitioned into groups of training examples in the training database ("datasets"). There may be an integer number M of datasets. The mth dataset, m=1,...,M, is denoted by D m This dataset has a set of training examples, λ, which may be the same for all batches, but alternatively may be different for different batches (e.g., lower for batches of training examples taken from sources with lower correlation between text and images). m A corresponding scalar is assigned that weights B. If all datasets have the same quality, it may not be worthwhile to partition the training examples (e.g., M may be set to 1). At each training iteration, B m A respective batch of training examples may be randomly selected from each of the M datasets, and updates are made to the variable parameters of the compressed representation generation system and the gated mutual attention layer to reduce a loss function indicating that the data item token processing model failed to predict the succession of tokens in the batch of training examples.

[0112] For example, the loss function may be the expected negative log-likelihood of a second portion of a token string of training examples given a first portion, averaged over all possible positions l, where the token string of training examples can be divided as follows:

[0113]

number

[0114] In the above equation, E denotes the expectation value. The gradient of the loss function may be calculated over all M datasets to perform an update step on the variable parameters of both the gated mutual attention layer and the modality network.

[0115] Initially, during the training procedure, the value of the gating parameter α may be chosen to be low (e.g., 0.01), so that the influence of data items on the output token string is correspondingly low. Thus, at the start of training, the gated mutual attention layer does not interfere much with the operations of the trained token processing layers 231, 232,...23j of the data item token processing network 220. During training, as the gated mutual attention layer gradually learns to cooperate with the token processing layers, the value of the gating parameter α is gradually increased, for example, to α=1.

[0116] Preferably, during both training and operation, "masking" is applied to at least one (or all) of the gated mutual attention layers 231, 232, ... 23j. This means that the gated mutual attention layers do not generate interaction terms based on compressed representations of all data items of the query input and on data derived from all the prompt inputs, but rather the interaction terms are generated based on compressed representations of only a (true) subset of the data items in the query input and only a (true) subset of the prompt inputs. For example, the query input may include (or consist of) multiple (non-overlapping) parts, each including exactly one of the data items, and may also include multiple of the tokens of the input token string. For example, some of the parts may be the aforementioned "task example" parts, and / or one of the parts may include the subject data item. For each part, the modality network generates a respective compressed representation of at least one of the corresponding data items, as described above. In case of masking, one or more of the gated mutual attention layers may be associated with each of the parts. For a given portion, one or more associated gated mutual attention layers generate interaction terms based only on the compressed representations of the data items in the relevant portion (i.e., without using compressed representations of other data items in the query input), and without utilizing data generated based on tokens of the input token string other than within the relevant portion (i.e., using only tokens contained in parts of the input token string that are within the relevant portion). Note that because masking is preferably applied only to the interaction terms, it does not prevent information about the entirety of the prompt input from flowing through the data item token processing model, and therefore all token processing layers have access to this information.

[0117] Turning to Figure 9, an example method 900 of the present disclosure is shown. The method may be performed by one or more computers at one or more locations. The method 900 is performed by a query processing system, such as the query processing system 200 of Figure 2. The method is performed on a query input having one or more data items and an input token string.

[0118] In step 901, each data item in a query input (such as query input 61 in FIG. 6) is input to a corresponding modality network to generate one or more condensed representations of each data item. If the data items have different modalities, there is typically at least one modality network for data items of each modality. For a given data item modality, there may be one or more modality networks used for data items of that modality.

[0119] In step 902, the input token string is used to generate a prompt input, which may include inserting at least one marker for at least one data item of the query input into the prompt input.

[0120] In step 903, the prompt input is processed using a data item token processing model, such as data item token processing model 220 of Figure 2, which also receives the condensed representation generated in step 901 to control a gating layer of the data item token processing model. The output of the data item processing model is an output token string, such as output token string 62 of Figure 6.

[0121] Turning to Fig. 10, an example method 1000 of the present disclosure is shown. The method may be performed by one or more computers at one or more locations. The method 1000 generates a query processing system, such as the query processing system 200 of Fig. 2.

[0122] In step 1001, a token processing layer of a trained language model, such as the model shown in FIG. 1(b), is used to form a data item token processing model, such as data item token processing model 220 of FIG. 2, by interleaving token processing layers 131, 132, 13n (trained in a system such as that of FIG. 1(b)) with untrained gated mutual attention layers 231, 232, 23j.

[0123] In step 1002, a query processing system, such as query processing system 200, is formed using the data item token processing model formed in step 1001 and a modality network, such as modality networks 207, 209. The modality network utilizes a trained encoder network 11 and (e.g., untrained) resamplers 210, 211.

[0124] In step 1003, the resamplers 210, 211 and the gated layers 231, 232, ... 23j are jointly trained, for example, by the system shown in Figure 8. Training the resamplers 210, 211 and the gated layers 231, 232, ... 23j includes training corresponding key, value and query matrices. Training the resampler 210 further includes training a latent vector.

[0125] Experiments have been conducted using the query processing system 200 of Fig. 2 to demonstrate its ability to perform diverse and challenging tasks. For example, Fig. 11 shows a dialogue between a human user and an implementation of the system. The query processing system was based on a large-scale language model, i.e., the token processing system shown in Fig. 1(b), which was the Chinchilla model of J. Hoffmann et al., implemented with 80 billion variable parameters.

[0126] First, the user generated a query input that included a data item 111 (a still image) and an input token string 112 (the words "What city is this?").

[0127] The query processing system processed the query input to generate an output token string 113 (the words "This is a New York street sign").

[0128] The user then generated a new query input including data item 111, input token string 112, output token string 113, and new input token string 114. The new input token string consists of the words "Which street corner?"

[0129] The query processing system processed the new query input and generated a new output token string 115 (the words "Madison Avenue East at 42nd Street").

[0130] Thus, a dialogue is practiced between a human user and the query processing system, in which the human user obtains successive information based on the data items of the initial query input. This dialogue can continue in a series of additional steps, in which the user concatenates, at each step, the data items 111, the input token strings and corresponding generated output strings of any previous steps, and a new token input string to generate a new query input, which is input to the query processing system.

[0131] A system of one or more computers being configured to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when in operation, causes the system to perform the operation or action. A system of one or more computer programs being configured to perform a particular operation or action means that one or more programs contain instructions that, when executed by a data processing apparatus, cause the apparatus to perform the operation or action.

[0132] The subject matter and functional operations described herein may be implemented in digital electronic circuitry, or in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed herein and their structural equivalents, or in a combination of one or more of them. The subject matter described herein may be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by or control of the operation of a data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to an appropriate receiver device for execution by the data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. However, the computer storage medium is not a propagated signal.

[0133] The term "stack" of layers refers to a sequence of layers, each of which receives a data input and produces a data output. Each layer other than the first receives as at least part of its input at least a portion of the output of the preceding layer in the sequence. Thus, data flows through the stack from the first layer in the sequence to the last layer, and the output of the stack of layers comprises the output of the last layer in the sequence.

[0134] The term "data processing apparatus" encompasses any kind of apparatus, device, and machine for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus may include special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). In addition to hardware, an apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting a processor firmware, a protocol stack, a database management system, an operating system, or any combination of one or more of these.

[0135] A computer program (which may be called or known as a program, software, software application, module, software module, script, or code) may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in file portions that hold other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple cooperating files, e.g., files that store one or more modules, subprograms, or code portions. A computer program may be deployed to be executed on one computer located at one site or on multiple computers, or distributed across multiple sites and interconnected by a communication network.

[0136] As used herein, an "engine," or a "software engine," refers to a software-implemented input / output system that provides an output that is distinct from an input. An engine may be a coded block of functionality, such as a library, a platform, a software development kit ("SDK"), or an object. Each engine may be implemented on any suitable type of computing device, for example, a server, a mobile phone, a tablet computer, a notebook computer, a music player, an e-book reader, a laptop or desktop computer, a PDA, a smartphone, or other fixed or portable device that includes one or more processors and a computer-readable medium. Furthermore, two or more of the engines may be implemented on the same computing device or on different computing devices.

[0137] The processes and logic flows described herein may be implemented by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be implemented by, and apparatus implemented as, special purpose logic circuitry, e.g., FPGAs (field programmable gate arrays) and ASICs (application specific integrated circuits). For example, the processes and logic flows may be implemented by, and apparatus implemented as, a graphics processing unit (GPU).

[0138] A computer suitable for executing a computer program may include, and may be based, for example, on a general-purpose or special-purpose microprocessor or both, or any other type of central processing unit. In general, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for implementing or executing instructions, and one or more memory devices for storing instructions and data. In general, a computer also includes one or more mass storage devices, for example, magnetic, magneto-optical, or optical disks, for storing data, or is operatively coupled to a mass storage device to receive data from or transfer data to the mass storage device, or both. However, a computer need not have such devices. Moreover, a computer may be incorporated in another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, for example, a universal serial bus (USB) flash drive, to name just a few examples.

[0139] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0140] To enable interaction with a user, embodiments of the subject matter described herein can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, e.g., a mouse or trackball, for allowing the user to provide input to the computer. Other types of devices can also be used to provide interaction with the user, e.g., feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from devices used by the user, e.g., by sending web pages to a web browser on the user's client device in response to a request received from the web browser.

[0141] Embodiments of the subject matter described herein can be implemented in a computing system that includes back-end components, e.g., as a data server, or includes middleware components, e.g., an application server, or includes front-end components, e.g., a client computer having a graphical user interface or a web browser by which a user can interact with an implementation of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network ("LAN") and a wide area network ("WAN"), e.g., the Internet.

[0142] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0143] Although this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to certain embodiments of a particular invention. Also, some features described in the context of separate embodiments can be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as working in some combinations and even initially claimed as such, one or more features from the claims may be deleted from the combination in some cases, and the claimed combination may be directed to a subcombination or a variation of the subcombination.

[0144] Similarly, although operations are shown in the figures in a particular order, this should not be understood as requiring that such operations be performed in the particular order or sequential order shown, or that all of the operations shown be performed, to achieve desirable results. In some situations, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products.

[0145] Specific embodiments of the present subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequence, to achieve desirable results. In some implementations, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0146] 11 Encoder network, encoder 12 Output layer 51 Visual Input 52 Visual Input 53 Language Input 61 Query Input 62 Output Token String 111 Data Items 112 Input Token String 113 Output Token String 114 Input Token String 115 Output Token String 131 Token Processing Layer 132 Token Processing Layer 13n Token Processing Layer 200 Query Processing System 201 Query Input 203 Data items 205 Data Items 207 Modality Network 209 Modality Network 210 Resampler, Perception Resampler 211 Resampler 220 Data Item Token Processing Model, Data Item Processing Model 231 Gated mutual attention layer, gated layer, layer 232 Gated mutual attention layer, gated layer, layer 23j Gated mutual attention layer, gated layer, layer 300 Resampler Layer 301 units 302 units 303 Attention Unit 305 Addition Unit 307 Multi-layer Feedforward (FFW) Perceptron, Perceptron 311 Addition Unit 501 Attention Function 502 Addition Unit 503 FFW Network 504 tanh gating unit 505 Addition Unit 506 Self-attention layer 507 Addition Unit 508 Feedforward Layer 509 Addition Unit 513 Token Processing Layer, Layer 523 Gated Mutual Attention Layer, Layer 801 Training Database 802 Splitter Unit 803 Mismatched Units 804 Update Unit< / eoc>

Claims

1. 1. A computer-implemented method for generating an output token string based on a query input including an input token string and one or more data items, wherein the input token string and the output token string are strings of tokens selected from a token vocabulary, and the data items are modalities other than the tokens selected from the token vocabulary, the method comprising: inputting each data item of the query input to a modality network trained to generate one or more condensed representations of each data item upon receiving the data items of the modality; generating a prompt input comprising the input token string of the query input; inputting the prompt input to a data item token processing model having a plurality of processing layers arranged in a stack, the output token string being an output of the data item token processing model, the processing layers including a plurality of token processing layers and a plurality of gated mutual attention layers, each gated mutual attention layer being arranged to receive at least one of the compressed representations, the token processing layers being interleaved with the gated mutual attention layers; A method comprising:

2. 2. The computer-implemented method of claim 1, wherein the token processing layers are operable to jointly provide a token string processing model for receiving an input token string and for generating a corresponding output token string in the absence of the gated mutual attention layer.

3. generating an output token string based on the query input; A set of steps, namely: forming a new query input based on the query input and the output token string; and generating a new output token string based on the new query input; at least once performing a set of 3. The computer-implemented method of claim 1 or claim 2, comprising:

4. 1. A computer-implemented method for training a query processing system for generating an output token string based on a query input comprising an input token string and one or more data items, the input token string and the output token string being strings of tokens selected from a token vocabulary, the data items being modalities other than the tokens selected from the token vocabulary; The method utilizes a token processing model comprising a stack of token processing layers configured to receive an input token string and to generate a corresponding output token string and a database of training examples, each training example including at least one data item and at least one token string; The method comprises: forming a data item token processing model by interleaving the token processing layer with a gated mutual attention layer, the data item token processing model being configured to generate an output token string upon receiving a prompt input, the prompt input being a token string; forming the query processing system, the query processing system comprising: (a) a modality network configured to receive the data items of the query input and generate one or more condensed representations of each data item; and (b) the data item token processing model, the data item token processing model configured to receive a prompt input including the input token string of the query input, and each gated mutual attention layer is arranged to receive at least one of the compressed representations. and Using the training database, the modality network, and training the plurality of gated mutual attention layers; 4. A computer-implemented method comprising:

5. 5. The computer-implemented method of claim 4, wherein when the encoder of the modality network receives the at least one data item of any of the training examples and the data item token processing model receives a prompt input including a first portion of the token string of the training example, the training trains the query processing system to generate an output of the query processing system that is statistically positively correlated with a subsequent portion of the token string of the training example.

6. The modality network includes: an encoder configured to encode a data item received by the encoder to generate an encoded data item; a condensed representation generation system arranged to receive the encoded data items and generate an output, the output of the modality network being based on the output of the condensed representation generation system; 6. The computer-implemented method of claim 1 , comprising:

7. The computer-implemented method of claim 6 when dependent on claim 4, wherein the encoder is trained to encode data items received by the encoder to generate encoded data items, and wherein the training of the modality network and the gated mutual attention layers comprises training the compressed representation generation system without further training the encoder.

8. said condensed representation generation system comprising a stack of one or more resampler layers; Each resampler layer is adapted to perform an attention operation utilising a key vector, a value vector and a query vector, a subset of the key vectors, value vectors and query vectors being based on the encoded data items and the remainder of the key vectors, value vectors and query vectors being based on either the output of a previous one of the resampler layers or, in the case of the first resampler layer of the stack, a set of input potential values; 8. The computer-implemented method of claim 6 or claim 7, wherein the output of the modality network is based on the output of a final resampler layer of the stack of resampler layers.

9. 9. The computer-implemented method of claim 8, wherein the key vector and value vector of each resampler layer are based on the encoded data items and potential inputs, which are either the outputs of a previous one of the resampler layers or, in the case of a first resampler layer in the stack, the set of input potential values, and the query vector is based on the potential inputs.

10. 10. The computer-implemented method of claim 8 or 9, wherein each resampler layer further comprises a perceptron arranged to receive the output of the attention operation and to generate an output, the output of the modality network being based on the output of the perceptron of a final resampler layer of the stack.

11. 11. The computer-implemented method of claim 1, wherein the prompt input further comprises one or more corresponding marker items for each data item in the query input, the one or more marker items indicating a presence of the data item in the query input.

12. 12. The computer-implemented method of claim 11, wherein a position of each marker item in the prompt input indicates a position of a corresponding data item in the query input.

13. Each gated mutual attention layer outputs a first input that is the output of the preceding processing layer in the stack of processing layers, or, if the gated mutual attention layer is the first processing layer in the stack of processing layers, the prompt input; an interaction term based at least in part on the output of the condensed representation generation system received by the gated mutual attention layer and the first input to the gated mutual attention layer; 13. The computer-implemented method of claim 1, wherein the component-wise sum of

14. 14. The computer-implemented method of claim 13 when dependent on claim 4, wherein the interaction term has a magnitude that depends strictly on the value of a gating parameter, and the training comprises gradually increasing a learning parameter.

15. If the query input includes multiple parts, each part including one of said data items, then for each part: generating a condensed representation of at least one of the corresponding data items by the modality network; at least one of the gated mutual attention layers generating the interaction terms based solely on the compressed representation of the corresponding data item and without utilizing data generated based on tokens of the input token string other than within the portion; 15. A computer-implemented method according to claim 13 or claim 14, comprising:

16. 16. The computer-implemented method of claim 1, wherein each gated mutual attention layer includes a mutual attention layer that utilizes key vectors, value vectors, and query vectors, a subset of the key vectors, value vectors, and query vectors based on the at least one compressed representation received by the gated mutual attention layer, and the remainder of the key vectors, value vectors, and query vectors based on the output of the previous processing layer in the stack of processing layers or, if the gated mutual attention layer is a first processing layer in the stack of processing layers, based on the prompt input.

17. 17. The computer-implemented method of claim 16, wherein the key vector and value vector of each gated mutual attention layer are obtained based on the at least one compressed representation received by the gated mutual attention layer, and the query vector of each gated mutual attention layer is based on the output of the previous processing layer in the stack of processing layers or, if the gated mutual attention layer is a first processing layer in the stack of processing layers, based on the prompt input.

18. 18. The computer-implemented method of claim 16 or 17, wherein the gated mutual attention layer further comprises a perceptron that receives the output of the mutual attention layer, and the output of the gated mutual attention layer is based on an output of the perceptron.

19. 1. A computer-implemented method for generating a condensed representation of a data item having modality, the method comprising processing the data item using a modality network, the modality network comprising: an encoder configured to encode the data item to generate an encoded data item; a condensed representation generating system arranged to receive the encoded data items and to generate an output; Equipped with said condensed representation generation system comprising a stack of one or more resampler layers; Each resampler layer is configured to perform an attention operation utilising a key vector, a value vector and a query vector, each of which is based on at least one of the encoded data items and latent inputs, the latent inputs being either an output of a previous one of the resampler layers or, in the case of a first resampler layer in the stack, a set of input latent values, and at least one of the key vector, value vector and query vector is based on both the encoded data items and the latent inputs; 11. A computer-implemented method, wherein the output of the modality network is based on the output of a final resampler layer of the stack of resampler layers.

20. 20. The computer-implemented method of claim 19, wherein the key vectors and value vectors of each resampler layer are based on the encoded data items and the potential inputs, and the query vectors of each resampler layer are based on the potential inputs.

21. 21. The computer-implemented method of claim 19 or claim 20, wherein each resampler layer is configured to apply the latent input to the output of the attention operation.

22. 22. The computer-implemented method of claim 19, wherein each resampler layer further comprises a perceptron arranged to receive the output of the attention operation and to generate an output, the output of the modality network being based on the output of the perceptron of a final resampler layer of the resampler layer stack.

23. 1. A computer-implemented method for generating an output token string based on a query input including an input token string and one or more data items, wherein the input token string and the output token string are strings of tokens selected from a token vocabulary, and the data items are modalities other than the tokens selected from the token vocabulary, the method comprising: (a) generating one or more condensed representations of each data item by processing the data items with a modality network, the modality network comprising: an encoder configured to encode the data item to generate an encoded data item; a condensed representation generating system arranged to receive the encoded data items and to generate an output; Equipped with said condensed representation generation system comprising a stack of one or more resampler layers; each resampler layer is configured to perform an attention operation utilizing a key vector, a value vector, and a query vector, the key vector, the value vector, and the query vector each being based on at least one of the encoded data items and latent inputs, the latent inputs being either an output of a previous one of the resampler layers or, in the case of a first resampler layer in the stack, a set of input latent values, the key vector, the value vector, and the query vector each being based on both the encoded data items and the latent inputs, and the output of the modality network being based on an output of a final resampler layer in the stack of resampler layers; Steps and (b) generating a prompt input comprising the input token string of the query input; (c) inputting the prompt input and the compressed representation of each data item into a data item token processing model configured to generate the output token string based on the prompt input and the compressed representation of each data item; 4. A computer-implemented method comprising:

24. 24. The computer-implemented method of claim 1, wherein the output token string is a response to a query about the content of a subject data item that is one of the data items in the query input, the query being defined based on the input token string.

25. 25. The computer-implemented method of claim 24, wherein the query input includes, in addition to the subject data item, one or more task example portions, each including a respective data item and a respective section of the input token string, and for each task example portion, the respective section of the input token string is a response to the query when the query is about the content of the respective data item.

26. 26. The computer-implemented method of claim 24 or 25, wherein the query input is a question and the response to the query is an answer to the question.

27. 27. The computer-implemented method of claim 1, wherein the token vocabulary comprises symbols of a natural language writing system.

28. 28. The computer-implemented method of claim 1, wherein the modalities include at least one of: (i) still or video images; or (ii) audio signals.

29. 29. A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform a method according to any one of claims 1 to 28.

30. One or more computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to implement the method of any one of claims 1 to 28.

Images