LANGUAGE MODEL FOR PROCESSING A MULTI-MODE QUERY INPUT

Abstract

A query processing system is described that receives a query input comprising an input token string and at least one data element having a second, different modality, and generates a corresponding output token string.

Description

GENERAL STATE OF THE ART

[0001] This specification relates to a neural network configured to process a multi-mode query input (e.g., a mixture of text and sound / image(s)) to produce an output that is a response to the query input.

[0002] Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output from a received input. Some neural networks contain one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network—the next hidden layer or output layer. Each layer of the network generates an output from a received input according to the current values ​​of a respective set of parameters.

[0003] Language models are known that employ neural networks that, upon receiving an input token string, which is a sequence of tokens selected from a token vocabulary (e.g., a piece of text consisting of letters selected from an alphabet (e.g., a natural language alphabet), a piece of text consisting of subwords or word fragments selected from a corresponding vocabulary, a piece of text consisting of phonemes selected from a corresponding vocabulary, and so on), generate an output token string (another sequence of tokens selected from the token vocabulary) that is a meaningful answer to the input token string, e.g., a plausible continuation of the input token string or an answer to a question posed by the input token string. SUMMARY

[0004] The present disclosure describes a system (a "query processing system") implemented as computer programs in one or more computers at one or more locations that receives a query input comprising an input token string and also at least one data element having a second, different modality.

[0005] For example, the data element(s) may be an image (multiple images). Each data element may be a still image—e.g., the data elements may be pixel values ​​(e.g., red-green-blue (RGB) values) for each pixel of a pixel array. Alternatively, one or more of the data elements may be video images—e.g., the data for each of multiple frames may be pixel values ​​for each pixel of a respective array. The image(s) may be captured by imaging the real world, by a still-image or video camera.

[0006] Another possibility is that the data element(s) is / are audio signals. A sound signal is audio data representing values ​​of an audio waveform at each of a plurality of points in time, such as the sound picked up by a microphone over a period of time.

[0007] Another possibility is that the data elements are video images with a respective audio track.

[0008] The query processing system is operable to generate an output token string based on the query input. The input token string and the output token string are each sequences of tokens selected from a (single) token vocabulary (e.g., both sequences may be strings of letters from the Roman alphabet, strings of partial words or word fragments from a partial word vocabulary, and so on). The token vocabulary may be the token vocabulary of a natural language (e.g., Latin letters in the case of English, or Latin letters plus accented Latin letters in the case of French).

[0009] In general, the disclosure proposes that the query processing system extract the data item(s) from the query input and input them to a modality network configured to generate one or more corresponding compressed representations of each data item. The query processing system uses the input token string to generate a prompt input for a data item token processing model. The data item token processing model comprises a stack of processing layers including token processing layers and controlled cross-attention layers interleaved with the token processing layers. The prompt input may be fed to the first processing layer of the stack.The cross-attention layers each receive at least one of the compressed representations and perform a gating operation based on the received compressed representations. The output token string is the output of the data item token processing model.

[0010] The output token string is a meaningful response to the query input. In particular, the input token string may at least partially define a question about the content of at least one of the data items in the query input (the "data item in question"), and the output string may be a (correct) answer to the question. Thus, the question defines a data item processing task to be performed on the data item in question.

[0011] The query processing system can be thought of as a generalization of a classification neural network. A classification neural network typically receives data items and determines which of a predetermined plurality of classes the data item belongs to (e.g., if the data item is an image depicting an object, the classifier may output data indicating which predetermined set of classes the object belongs to). In contrast, in the query processing system, the input token string of the query input determines the processing task the query processing system should perform on the data item in question to generate an output token string. In other words, the input token string (e.g., in natural language) defines a transformation that is applied to the data item in question to generate an appropriate output token string.Viewed from another point of view, the input token string defines information that the query processing system should extract from the data element in question and the output token string encodes this information.

[0012] The input token string can explicitly include the question (data element processing task). For example, if the data element in question is an image or a video, the input token string can be "What activity is shown?"

[0013] Alternatively or additionally, the input token string may define the question implicitly, e.g., by specifying one or more examples of the data item processing task. That is, the query input may include one or more "task example" sections in addition to the data item in question. Each task example section contains a data item and the respective portion of the input token string that represents the answer to the question when the question is asked about the content of the data item in the example section (i.e., the result of applying the question to that data item).

[0014] For example, each task example section might consist of an image and a statement related to the subject matter of the image. A possible task example section would be a photograph of a chinchilla and a section of the input token string containing the text "This is a chinchilla. They are mainly found in Chile." This task example section implicitly defines the question as "What animal is shown in the image? In which country is it commonly found?" Note that the question elicits not only information about the content of the image, but also information that is not in the image but is associated with the content (e.g., the country in which an animal depicted in the data item in question is commonly found).

[0015] Thus, the query processing system identifies a data item processing task that, when applied to the data item in the task example section, results in the range of the input token string in the task example section, and applies this processing task to the respective data item to generate the output token string. If there are multiple task example sections in the query input, the query processing system identifies a data item processing task that, when applied to the respective data items in the task example sections, results in the respective ranges of the input token string in each of the task example sections, and applies this processing task to the respective data item to generate the output token string.

[0016] A query input with no problem example sections (e.g., a query input that includes only the data item in question and an input token string that explicitly defines the question) can be called a "zero-shot" query input. A query input that includes n number of example sections (in addition to the data item in question and, optionally, some part of the input token string that helps define the question, such as an explicit explanation of the question) is called an "n-shot" query input. A "few-shot" query input is an input where n is low, such as one or two. For example, the query input might consist of the text "Name the artist," then a data item that is an image of a Picasso painting, then the word "Picasso," then a problem data item that is an image of a Salvador Dali painting.This is a "one-shot" query because there is only one sample task section (the image of the Picasso painting and the word "Picasso"). The input token string of the query is the text "Name the artist Picasso." It includes one word ("Picasso"), which is the range of the input token string that is part of the sample task section.

[0017] In the case where the data elements are video images, the query processing system may operate as a video processing system configured to analyze a sequence of video frames to detect objects in the video frames and provide information about the detected objects in response to a question defined by the input token string. The questions may, for example, include a request for a prediction of a future event or state related to one or more of the objects (e.g., "Will objects X and Y collide?"), or a request for conditional or counterfactual information related to one or more of the objects (e.g.,"Which event would [not] occur if object X is modified, moved, or absent?"), or a requirement to analyze the frames of the video to determine a property or characteristic of one or more objects (e.g., "How many objects of type Z are moving?").

[0018] The answer to the question, provided as an output token string, can, for example, be in the form of a yes / no answer or define a probability distribution over a set of possible answers; or the answer can define the location of an object. Implementations of the system are flexible in the types of questions that can be defined, and a single implementation of the system can handle several different types of questions. Using the system, for example, it can be predicted whether or not two objects will collide, or how this can be avoided. The answer to the query can be used by a human or a computer system in many ways. For example, the analysis can be useful in itself, or it can be used to provide a warning and / or control the movement of one or more objects.

[0019] The query processing system has a number of other technical applications. For example, a person (e.g., a visually impaired person in the case where the data elements are (still or video) images, or a hearing-impaired person in the case where the data elements are audio signals) can obtain information about the data element in question. For example, the query processing system can be part of a computer system that obtains the data element in question (e.g., in the form of a still image or a moving image, using a camera of the computer system). The person can provide the input token string (e.g., by controlling a data input device or by speaking a command that is converted into the input token string, e.g., a string of letters or tokens representing phonemes). The query processing system can then generate the output token string, e.g.,as a string of tokens that are converted into a sound signal sent to the person by a speaker device. A useful feature of this computer system is its flexibility, as the person can freely define the information that the query processing system should extract from the data element in question.

[0020] Optionally, the computer system can be configured to allow the individual to generate a series of multiple query inputs, each including the same data item of interest, but with different input token strings. For example, if the data item of interest is an image, the input token string of a first query input may be "What is shown in the image?" If the query processing system generates a corresponding output token string (e.g., "A bus") in response to the first query input, the individual can define a second query input that includes the same data item of interest and a new input token string (e.g., "What destination is on the front of the bus?").

[0021] Optionally, each query input in the series of query inputs, except for the first query input, may include the input token string and the output token string of one or more of the preceding query inputs in the series. This allows the individual to have a coherent discussion with the query processing system about the data item in question (which is typically included in all query inputs in the series), with the query processing system producing, at each stage, an output token string that represents a meaningful continuation of the earlier part of the discussion.

[0022] Another technical application of the query processing system is a closed captioning system for a data item that is a video item. The input token string may define the video item processing task as "providing closed captions for the video that explain what is happening," and in this case, the output token string may include closed captions describing events or actions performed in the video. If the data item is a video item with an audio track, the closed captions may include a transcription of the audio track.

[0023] In another application, the query processing system may generate output data that categorizes the data items into one or more of a plurality of categories, e.g., by defining a score for each category from a plurality of possible categories for the data items. In another particular application, where the data items comprise (still or video images) or sounds (audio data describing an audio waveform), the task (question) of the data item processor may be to provide output data that defines (e.g., as a score) whether the image, video, or audio waveform is described by the text comprised in the input token string.

[0024] In another technical application, the data element is a still or video image that includes text, and the computer system may include an optical character recognition (OCR) system for extracting the text from the image to generate at least part of the input token string. For example, if the data element is an image of a scene that includes a sign with instructions (e.g., the image may be an image of a road that includes a road sign that includes text), the OCR may generate the input token string based on the instructions in the image. In this case, the output token image may indicate the meaning of the instructions given in the rest of the scene (e.g.,In the case of a data item that is an image of an empty parking bay and a sign indicating that parking is currently permitted, the output token string may be "Please park here"). In another example, the data item may be an image (e.g., a meme) that includes text extracted by the OCR to form the input token string, and the output token string may indicate (e.g., as a score) whether the combination of image and text is offensive (e.g., indecent, subversive, or blasphemous).

[0025] In another technical application, the query input may instruct the query processing system to generate executable code as an output token string by a processor of a computer system. For example, the query input might be "[Image of a dog] Generate HTML code that, when executed, generates a video image of this animal moving its body" (here, the square brackets denote the data element in the query input, i.e., an image of a dog).

[0026] Note that the stack of token processing layers can be operational on its own (i.e., when arranged as a stack in the same order, without the nested cross-attention layers) to perform a token string processing operation, which involves receiving an input token string and generating an output token string from it. For example, the input token string can be the answer to a question defined by the input token string or a (meaningful) continuation of the sequence of tokens defined by the input token string, e.g., a continuation of a story started by the input token string.

[0027] The token processing layers of the data item token processing model can be layers generated by a publicly known language model training procedure (e.g., Brown et al., "Language models are few-shot learners," in Conference on Neural Information Processing Systems, 2020). The term "training" here means that the parameters of the token processing layers are adjusted iteratively, e.g., based on a training base of examples that illustrate a task that the token processing layers are intended to perform. After training the token processing layers, the data item token processing model can be formed by interleaving the trained token processing layers with the guided cross-attention layers. The guided cross-attention layers can then be trained, e.g., jointly with the modality network, based on a training database.Here, training refers to the iterative modification of variable parameters defining the controlled cross-attention layers and variable parameters defining the modality network (as described below, these may be variable parameters defining a compressed representation generation system, which is a component of the modality network). Training is based on a training database of "multi-mode" training examples, as described below. During training, the token processing layers are preferably "frozen," meaning they are essentially unchanged. This means that the amount of data required in the training database is small, e.g., much smaller than a training database previously used to train the token processing layers.

[0028] Optionally, the data items in a given query input may have different respective modalities, e.g., one of the data items may be an audio data signal and another a (still or moving) image. Optionally, a different modality network may be provided for each respective modality, and data item(s) in the query input, each having a modality, are passed as inputs to the respective modality network. Each modality network creates compressed representations of the corresponding received data item(s), and these are passed to one of the controlled cross-attention layers. For simplicity, the following only considers the case where data items of one modality are present in the query and there is only one modality network suitable for data items of that modality.

[0029] Note that in some implementations, data items consisting of an image and associated audio may be treated as having a single modality and thus processed as a whole by a single modality network. Alternatively, in other implementations, the video image of each data item may be separated from the audio track, generating two respective data items of different respective modalities, which are processed by different respective modality networks of the query processing system.

[0030] The (or each) modality network may employ a pre-trained encoder network that has been pre-trained (e.g., using a publicly known method) to generate an encoded data item upon receiving a data item. The encoded data item may have fewer components than the data item itself. For example, the encoder network may comprise one or more stacked convolutional layers, and the encoded data item may be a feature map.

[0031] The modality network may comprise the encoder network and a compressed representation generation system arranged to receive the encoded data item and generate an output. The output of the modality network is (or is generally based on) the output of the compressed representation generation system. Using a pre-trained encoder network as part of the modality network that does not change significantly (or even at all) during training of the compressed representation generation system means that the compressed representation generation system may have fewer parameters than if the encoder network were not present and the compressed representation generation system instead received the raw (uncoded) data items of the training examples. This reduction in parameters, in turn, means that the training database required to jointly train the modality network (i.e.the compressed representation generation system section of the modality network) and the guided cross-attention layers, does not require as many training examples and that the training process requires fewer computational resources.

[0032] As mentioned above, the compressed representation system and the guided cross-attention layers can be jointly trained in an iterative process using the training database. The term "joint training" here means that the iterative changes to the compressed representation generation system are interleaved with, or essentially concurrent with, corresponding iterative changes to the guided cross-attention layers.

[0033] The training database consists of training examples, each of which includes at least one data item and one token string (“multi-mode” training examples). The token string is a string of tokens from the same token vocabulary as the input token string. During training, one of the training examples can be repeatedly selected. The data item(s) in the training item are input to the modality network to generate corresponding compressed representations used by the controlled cross-attention layers. The token string of the training example is divided into a first portion (representing an earlier part of the token string) and a second portion (representing a later part of the token string). The first portion of the token string portion is used to generate prompt input for the data item token processing model.Thus, based on the training example, the data item token processing model generates an output token string from the compressed representations and the prompt input. The training of the modality network and the guided cross-attention layers are then modified to increase the statistical correlation between the output token string and the second portion of the token string from the training example.

[0034] For example, if the token string of the training example is denoted by y and the data item(s) of the training item are denoted by x, the modification may be such to increase p(y|x)=∏l=1Lp(yl|y <l,x≤l), where L is the number of characters in the token string of the training example, y l denotes the l-th character of the token string, y <l denotes the l-1 previous characters of the token string, x ≤ldenotes the data element(s) before the l-th character of the token string, and p(y l |y <l , x ≤l ) denotes the probability of the data element token processing network producing an output token string equal to y l begins, based on a prompt that y <l and the compressed representations based on the modality network from the data elements x ≤l be generated.

[0035] In fact, each of the training iterations is preferably performed using multiple training examples. For example, the training examples in the training database can be partitioned into groups ("datasets") of training examples in the training database. There can be an integer number M of datasets. The m-th dataset, where m = 1,..., M = 1, is denoted by D m This is assigned a corresponding scalar weighting λ mwhich can be the same for all batches, but can alternately differ for different batches (e.g., lower for batches of training examples obtained from sources where there is a lower correlation between text and images). If all datasets are of the same quality, partitioning the training examples may have no value (e.g., M can be set to 1). In each training iteration, a corresponding batch of B m Training examples are randomly selected from each of the M datasets, and an update of the variable parameters of the compressed representation generation system and the guided cross-attention layers is performed to reduce a loss function that indicates the failure of the data item token processing model in predicting consecutive tokens in the batches of training examples.

[0036] For example, the loss function can be the expected negative log-likelihood of the second section of the token string of the training examples, given the first section, averaged over all possible positions l where the token string of the training example can be positioned: ∑m=1Mλm⋅E(x,y)~Dm[−∑l=1Llogp(yl|y <l,x≤l)] where E denotes the expected value. The gradients obtained from Eq. (2) over all M datasets can be computed to perform an update step for the variable parameters of both the controlled cross-attention layers and the compressed representation generation system.

[0037] The compressed representation generation system may comprise one or more resampler layers, arranged, for example, as a stack of resampler layers. Each resampler layer receives the output of the encoder network (e.g., after a transformation to the output of the encoder layer, such as a flattening operation) or a previous layer of the stack. Each of the resampler layers may perform a transformer operation, i.e., it includes one or more transformer blocks or self-attention layers. A transformer block typically includes an attention neural network or self-attention neural network layer, followed by a feedforward neural network.An attention or self-attention neural network layer is a neural network layer that incorporates an attention or self-attention operation (which operates on the neural network input to generate the attention layer output). A self-attention operation can be masked, so that all given positions in an input sequence are not visited via all positions after the given position in the sequence. There are many different possible (self-)attention operations. Some examples of transformer blocks that incorporate attention operations are described in Vaswani et al., "Attention is all you need," 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.

[0038] In general, an attention operation maps a query and a set of key-value pairs to an output, where the query, keys, and values ​​are all vectors. The output is computed as a weighted sum of the values, with the weight assigned to each value calculated by a compatibility function, such as a dot product or scaled dot product, of the query with the corresponding key.

[0039] In implementations, the attention operation is configured to apply a self-attention operation to attention layer input; this may be followed by one or more feed-forward neural network layers to generate the attention layer output. In general, an attention operation determines a relationship between two sequences; a self-attention operation is configured to concatenate different positions in the same sequence to determine a transformed version of the sequence as output. For example, the attention layer input may include a vector for each element of the input sequence. These vectors represent input to the self-attention operation and are used by the self-attention operation to determine a new representation of the same sequence for the attention layer output, similarly including a vector for each element of the input sequence.The output of the self-attention operation can be used as the attention layer output or it can be processed by one or more feed-forward layers to provide the attention layer output.

[0040] In some implementations, the attention operation is configured to assign each query transformation, e.g., represented by a query matrix W Q is defined, a key transformation, which is e.g. defined by a key matrix W K and a value transformation, which is defined, for example, by a value matrix W V is defined to the attention layer inputs, which are the input data X for the attention layer, to generate a corresponding query vector Q = XW Q , a key vector K = XW K and a value vector V = XW Vwhich are used to determine a visited sequence for the output. For example, the attention operation can be a dot product attention operation, which is performed by applying each query vector to each key vector to determine the respective weights for each value vector, then combining the value vectors using the respective weights to determine the attention layer output for each element of the input sequence. The attention layer output can be scaled by a scaling factor, e.g., the square root of the dimensions of the queries and keys, to implement scaled dot product attention. Thus, for example, an output of the attention operation can be softmax(QKTd)V where d is a dimension of the key (and value) vector. In another implementation, the attention operation may include an "additive attention" mechanism that computes the compatibility function using a feed-forward network with a hidden layer. As mentioned above, the output of the attention operation may be further processed by one or more fully connected feed-forward neural network layers.

[0041] The attention operation can implement multi-head attention, meaning it can apply multiple different attention operations in parallel. The outputs can then be combined, e.g., combined with a learned linear transformation applied to reduce the original dimensionality if necessary.

[0042] Optionally, the first resampler layer of the stack of layers may additionally receive at least one latent vector (a set of latent input values). The components of the latent vector may be among the variable parameters of the compressed representation system trained in the training operation. The latent vector may have a number of components that is less (e.g., much less) than the number of components of the (e.g., flattened) output of the encoder network. Each resampler layer is trained to produce an output that also has a number of components that is much less than the number of components in the output of the encoder network; e.g., it may have the same number of components as the layer query vector. The other resampler layer(s) of the stack (if any) receive the output of the previous resampler layer in the stack.

[0043] Each resampler layer may receive (i) the (e.g., flattened) output of the encoder network and (ii) a "latent input," which is the output of the preceding resampler layer in the stack, or in the case of the first resampler layer of the stack, the latent vector. The resampler layer may generate (for each of one or more heads) a key vector, a value vector, and a query vector from different corresponding ones of the data inputs (i) and (ii) (in principle, any combination of data inputs (i) and (ii) may be used; at least one of the key vectors, value vectors, and query vectors is based on (i.e., derived from data comprising) each of the data inputs (i) and (ii), and preferably at least one of the key vector, the value vector, and the query vector is based on both data inputs (i) and (ii)), as described above, using the matrices W Q , W Kand W V . For example, the key vector and the value vector can be obtained by multiplying the input data (i) (or a combination of the input data (i) and the input data (ii)) by the matrices W K and W V generated, and the query input vector can be calculated by multiplying the input data (ii) by the matrix W Q Then, the output of the resampler layer is generated as already explained based on the key vector, the value vector and the query vector, e.g. as softmax(QKTd)V. Where d is the number of components of the key vector. The components of the matrices W Q , W K and W V and may be among the variable parameters of the compressed representation generation system that are trained in the training process.

[0044] The output of the stack of resampler layers can be passed to an adaptive system, which is a single- or multi-layer perceptron, that generates the compressed representation, which is the output of the compressed representation generation system. However, an adaptive system, such as a single- or multi-layer perceptron, can be provided in each of the resampler layer(s) to receive the output of the corresponding attention operation of the resampler layer. In both cases, the perceptron can be a substantially fully connected ("dense") network. The perceptron has variable parameters that are among the variable parameters of the compressed representation generation system being trained in the training process.

[0045] The use of a modality network including one or more resampler layers as described above to generate compressed representations of data elements represents an independent aspect of the disclosure.

[0046] Another independent aspect of the disclosure is the use of such resampler layers within a system for generating an output token string based on a query input comprising an input token string and one or more data items, wherein the input token string and the output token string are strings of tokens selected from a token vocabulary, and wherein the data items are of a different modality than tokens selected from the token vocabulary.The method includes generating one or more compressed representations of each data item by processing the data item using the modality network, generating a prompt input comprising the input token string of the query input, and inputting the prompt input and the compressed representation of each data item into a data item token processing model configured to generate the output token string based on the prompt input and the compressed representation of each data item.

[0047] As for the controlled cross-attention layers, each controlled cross-attention layer can forward data received from the preceding processing layer of the data item token network (or, in the case where the controlled cross-attention layer is the first processing layer of the data item token network (although the first layer of the data item token network is usually one of the token processing layers), the prompt input) plus a modification (interaction term) whose size depends positively on the value of a gating parameter (gating value), i.e., the size of the interaction term increases with an increasing value of the gating parameter. The interaction term is added component by component to the input of the controlled cross-attention layer.For example, the output of the guided cross-attention layer upon receiving a (vector) input y from the previous layer may be y + tanh (ax), where α is the gating parameter (a scalar), x is the (vector) output of an adaptive system (as described below) receiving y, and tanh is a component-wise tanh operation applied separately to each component of αx.

[0048] If the value of the gating parameter is low, the influence of the data items in the query inputs is correspondingly small. At the beginning of training, the value of the gating parameter can be chosen low (e.g., 0.01), so that the guided cross-attention layers barely affect the operation of the trained token processing layers of the data item token processing network. During training, as the guided cross-attention layers gradually learn to cooperate with the token processing layers, the value of the gating parameter is gradually increased.

[0049] "Masking" can be applied to at least one (or all) of the directed cross-attention layers. This means that the adaptive system of the directed cross-attention layer does not generate the interaction terms based on the compressed representations of all data elements in the query input and the data derived from the entire prompt input, but rather generates the interaction term based on the compressed representations of only a (correct) subset of the data elements in the query input and the data based on only a (correct) subset of the prompt input.

[0050] For example, assume the query input includes (or consists of) a plurality of (non-overlapping) sections, each of which contains exactly one of the data items and may also contain a plurality of tokens of the input token string. For example, each section may be one of the input data items and the tokens of the input token string after that data item and up to the next data item, or in the case of a section for which that data item is the last data item of the query input, up to the end of the input token string. For example, some of the sections may be "task example" sections that have already been discussed, and / or one of the sections may contain the data item in question. For example, the query input might be "[Picture of a cat] This is a cat [Picture of a dog] This."A first section of the query input is "[Image of a cat] This is a cat" (this section is a task example section; [Image of a cat] denotes an image of a cat), and a second section of the query input is "[Image of a dog] This" (where [Image of a dog] is the data item in question, i.e., an image of a dog). For each section, the modality network generates at least one respective compressed representation of the corresponding data item, as described above. In the case of masking, one or more of the steered cross-attention layers may be associated with each of the sections. For a given section, the one or more associated steered cross-attention layers generate the interaction term based only on the compressed representation of the data item of the associated section (i.e.,without using the compressed representations of other data elements of the query input) and without inserting data generated based on tokens of the input token string that are not within the allocated section (i.e., using only tokens contained in the part of the input token string that is within the allocated section).

[0051] Note that although masking only applies to the interaction term, it does not prevent information about the entire prompt input from flowing through the data element token processing model, so all token processing layers have access to this information.

[0052] The adaptive system of the gated cross-attention layer may include a cross-transformer operation and / or processing by a (single- or multi-layer) perceptron. For example, the input of the cross-transformer operation may be the input to the perceptron, which outputs the value x denoted above.

[0053] The cross-transformer operation is based (for each of one or more heads) on a set of matrices W Q , W K and W V for each controlled cross-attention layer. The components of the matrices W Q , W K and W V and the variable parameters of the perceptron of the directed cross-attention layer (if any), are (or at least are among) the variable parameters of the directed cross-attention layer that are trained in the training process.

[0054] The cross-transformer operation may receive (i) one or more of the compressed representations of the data elements in the query input and (ii) the output of the previous processing layer or, in the case where the controlled cross-attention layer is the first processing layer of the stack, the prompt input. The cross-transformer operation may generate (for each of one or more heads) a key vector, a value vector, and a query vector from different ones of the data inputs (i) and (ii), as described above, using the matrices W Q , W K and W V For example, the key vector and the value vector can be calculated by multiplying the input data (i) by the matrices W K and W V generated, and the query input vector can be calculated by multiplying the input data (ii) by the matrix W QThen, the output of the cross-transformer operation is generated as described above based on the key vector, the value vector, and the query vector, e.g., as softmax(QKd)V, where d is the number of components of the key vector.

[0055] As mentioned earlier, the prompt input includes the input token string. It may also include "markers" (one or more tokens, e.g., tokens contained in the same vocabulary as the input token string) that indicate the existence and, if applicable, the location of the data items within the query input. For example, if the query input is "[Image of a cat] This is a cat [Image of a dog] This," where "[Image of a cat]" is a data item that is an image of a cat, and "[Image of a dog]" is a data item that is an image of a dog, the prompt input may be " This is a cat marking ", where " “ (i.e. these 7 ASCII characters, where ASCII in this case is the token vocabulary of the input token string) is the marker that specifies a data item.

[0056] A technical effect of the present disclosure is to enable the provision of a query processing system that has been experimentally demonstrated to provide a precise way of processing multi-mode query inputs (i.e., those consisting of both tokens and data elements of another, distinct modality). Accuracy has been proven to be high, even compared to known computer systems designed for specialized multi-mode query processing tasks. It is high even for zero-shot query inputs and becomes higher for query inputs with a low shot fraction.

[0057] Since embodiments of the invention can additionally utilize existing token processing layers and encoder networks, the amount of training data required to train them and the required computational resources are significantly reduced. The token processing layers (which may include billions of variable parameters) can be trained based on known text databases, and the encoder network can be trained based on known image databases, and only a relatively small training database of training examples is required to train the gated cross-attention layers and the modality network. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Examples of the invention will now be described, by way of illustration only, with reference to the following drawings. Fig. Figure 1(a) shows an encoder network. Fig. Figure 1(b) shows a token processing model. Fig. Figure 2 shows a query processing system comprising modality networks, including the Fig. 1(a), and a data element token processing model that represents the Fig. 1(b) includes the token processing model shown. Fig. Figure 3 shows a first modality network. Fig. Figure 4 shows a second modality network. Fig. Figure 5 shows a pair of layers of the data element token processing model of the query processing system of Fig. 2. Fig. 6 shows a query input to the query processing system and an output token string generated by the query processing system from the query input. Fig. Figure 7 shows a training example used to train the query processing system. Fig. 8 shows a training system for training a query processing system based on training examples. Fig. 9 shows a method for generating an output token string based on a query input. Fig. 10 shows a method of training a query processing system. Fig. Figure 11 illustrates a process of interacting with a query processing system.

[0059] Similar reference numbers indicate the same elements in different figures. DETAILED DESCRIPTION

[0060] This specification describes a query processing system operable to receive a query input that includes at least one input token string (a series of tokens selected from a vocabulary) and at least one data element that has a different modality than the input token string. The data element typically does not include tokens selected from the vocabulary. The query processing system is an example of a neural network system, which may be implemented by one or more computer systems at one or more locations.

[0061] The data elements may comprise one or more images (e.g., still or video image(s) captured by a camera) and / or audio data representing values ​​of an audio waveform at each of a plurality of points in time, e.g., sound captured by a microphone during a period of time. The query processing system includes a portion (“a modality network”) that includes an encoder network for encoding the data elements (e.g., as a set of features) and a compressed representation generation system for generating a compressed representation of the data element from the encoded data element. For example, in the case of a data element comprising images, the encoder network may be configured to apply at least one 2-dimensional convolutional layer to each image. In the case of a data element comprising audio data, the encoder network may be configured to apply a one-dimensional convolution to the audio data.

[0062] The encoder network can be one that has been pre-trained in another neural network system. For example, Fig. 1(a) shows a neural network system including an encoder network 11 and an optional output layer 12. The encoder network 11 receives a data item and generates an output therefrom. The optional output layer 12 (if present) receives the output of the encoder network 11 and generates an output of the neural network system. The encoder network 11 can be trained using a database of data items and corresponding desired outputs, so that when a data item is input to the encoder network 11, the encoder network 11 and (if present) the output layer 12 generate the corresponding desired output. During the training process, the output layer 12 (if present) can optionally also be trained.In the case of an output layer 12, the encoder network can be trained to extract features of a received data item, and the output layer 12 processes these features to produce the desired output (encoded data item). After training, the output layer 12 (if present) can be discarded. The encoded data item can have a smaller number of components than the data item from which it is generated. The encoder network 11 can comprise one or more stacked convolutional layers, and the encoded data item can be a (e.g., two-dimensional) map.

[0063] In the experiments described below, the "f6" model from the article "High-Performance Large-Scale Image Recognition Without Normalization" by A. Brock et al. (2021) was used as encoder network 11. The encoder network 11 was pre-trained on datasets of image and text pairs using a contrastive objective, using the twofold contrastive loss from "Learning Transferable Visual Models From Natural Language Supervision" by A. Radford et al. (2021). The output of the final stage, a 2D spatial mesh of features, was "flattened" to form a 1D sequence.

[0064] The query processing system further includes a data element token processing model that includes token processing layers from a token processing model. The token processing model may be one that has been pre-trained. For example, Fig. 1(b) shows a suitable token processing model comprising an integer n of token processing layers 131, 132, ... 13n arranged in a stack (sequence). The first token processing layer receives an input token string. Each token processing layer, except the first, receives the output of the previous token processing layer. The output of the nth token processing layer is an output token string of the token processing model. The token processing layers can form a "language model" (e.g., a "large language model") trained on a large database of data, e.g., natural language data, so that when an input token string (sequence of tokens from the vocabulary) is input to the first token processing layer, the output of the token string is an appropriate response. For example, if the input token string is a question, the output token string can be a suitable answer.In the experiments reported below, the Chinchilla model from "Training compute-optimal large language models," J. Hoffmann et al., 2022, was used as the token processing model. Alternatively, the token processing layers can be layers generated by another publicly known language model training method (e.g., Brown et al., "Language models are few-shot learners," in Conference on Neural Information Processing Systems, 2020).

[0065] In Fig. 2 again illustrates the structure of the query processing system 200. The query processing system 200 is configured to receive as input a query input 201 that includes one or more data elements and an input token string. In a Fig. In the case illustrated in Figure 2, the query input 201 may contain two data elements 203, 205. However, it should be understood that there may be any number of data elements.

[0066] The query processing system 200 extracts the two data elements 203, 205 from the query input 201 and processes each element using a modality network. As shown in Fig. 2, the query processing system includes two modality networks 207, 209, but alternately, in the case where all data elements have the same modality, there may be only a single modality network, which in turn receives the data elements 203, 205 and produces a corresponding output from each.

[0067] It should be noted that if multiple data items are present in the query input 201 and if they have different corresponding modalities, a modality network may be provided for each corresponding modality, and each modality network is configured to receive the data item(s) of the corresponding modality (e.g., sequentially if multiple data items of the corresponding modality are present in the query input). For example, the modality network 207 may be configured to receive data item(s) of the query input that are a single image (e.g., a still image), and the modality network 209 may be configured to receive data item(s) of the query input that are audio data. In another example, the modality network 207 may be configured to receive data items of the query input that are a single image (e.g.,a still image), and the modality network 209 may be configured to receive data items that are a video (i.e., a sequence of images), optionally with a soundtrack of audio data. There may be any number of modality networks for each corresponding data item that the query processing system 200 is configured to process.

[0068] Each modality network 207, 209 is configured to generate, from each data element it receives, one or more corresponding compressed representations of the data element. Each modality network may include a pre-trained encoder network 11 (e.g., trained within the system of Fig. 1(a)), and a compressed representation generation system in the form of a respective resampler 210, 211. Each resampler 210, 211 may, for example, be a “recipient” as described in “Perceiver: General Perception with iterative attention”, A. Jaegle et al, 2021. Alternately, the resampler 210, 211 may be as described below with reference to Fig. 3 and Fig. 4 be described.

[0069] The resamplers 210, 211 take as input a variable number of image, video or audio features generated from a data element by the corresponding encoder network 11 and generate a compressed representation of the data element having a fixed number of outputs (e.g., visual outputs).

[0070] Fig. Figure 3 shows the structure of a possible modality network 207 in the case where the data element 203 is an image containing a respective "recipient" resampler 210. Note that the modality network 209 has the same structure when the data element 205 is an image (as already mentioned, when the data elements 203, 205 have the same modality, a single network can perform the role of both modality networks 207, 209).

[0071] The resampler 210 receives the output of the encoder 11 (a coded version of the data element 203) and generates flattened data with X f The recipient resampler also receives a predefined number of latent vectors (i.e., sets of latent values), each denoted by X, which are learned (trained) during the training of the modality network 207 described below.

[0072] The resampler 210 also includes one or more “resampler layers” 300. For simplicity, the resampler 210 is Fig. 3 as having a single resampler layer 300. The compressed representation generation system may alternately comprise a plurality of resampler layers, each of which Fig. 3. The resampler layers 300 may be arranged as a stack of resampler layers, each layer having the same shape as the structure 300 shown in Fig. 3. Each resampler layer 300 (except the first) receives the output of the previous resampler layer of the stack.

[0073] The resampler layer 300 (or, if there are multiple resampler layers 300, each resampler layer of the stack) receives the output of the corresponding encoder network 11, e.g., after a transformation to the output of the encoder network 11, such as a flattening operation (i.e., converting a 2-dimensional array into a 1-D sequence, for example).

[0074] Each of the resampler layers 300 may perform a transformer operation as defined above, i.e., it may include one or more transformer blocks or self-attention layers. For this purpose, it may comprise a unit 301 that processes the flattened data X f and again uses different latent vectors X to generate a key vector K(X f , X) and a value vector V(X f, X). This process uses a key matrix and a value matrix as previously described. A unit 302 uses the latent vector X and a value matrix to generate a query vector Q(X). An attention unit 303 combines K, V, and Q through the attention operation described above. The result is added to the latent vector X by an addition unit 305. The result is passed to an adaptive unit, such as a multi-layer feedforward (FFW) perceptron 307. The output of the FFW perceptron 307 is added to the output of the addition unit 305 by another addition unit 311. The result is an output vector for each of the latent vectors X, providing different respective compressed representations of the data item. The output of each resampler layer 300 may have the same dimensionality as the dimensionality of the latent variable X.This dimensionality can be much smaller than the dimensionality of the encoder network 11.

[0075] Fig. 4 shows a variation of the structure of Fig. 3 in the case where the input data element 203 is a plurality of video frames. The data element 203 is divided into individual frames 41, 42, 43. Each of these frames is sequentially coded by the encoder network 11 to generate a corresponding coded frame. To each of the coded frames, the corresponding data 44, 45, 46 are added, which specify the position of the frame in the data element 203. The resulting data is input to the resampler 210, which may have the same structure as already described with reference to Fig. 3. It should be noted that the input to the resampler 210 differs from that in Fig. 3, the output for a given latent vector X, but the same number of output tokens.

[0076] The operation of a recipient resampler can be represented by the following pseudocode: x_f represents the output of the corresponding encoder network(s). This is an array with dimensionality [T, S, d], where T is an integer variable representing the number of times (i.e., the number of encoder networks or the number of times a single encoder network is used), S is an integer indicating the number of spatial positions, and d is another integer indicating the number of feature values ​​for each position.

[0077] x represents R latent variables, so it is an array with dimensionality [R, d].

[0078] The number of layers of the resampler is denoted by num_layers.

[0079] First, the time embeddings are added and the result is flattened: x_f = x_f + time_embeddings x_f =flatten(x_f) This creates an array of dimension [T*S,d] Then for each integer i in the range 1 to num_layers: x=x+attention_i(q=x, kv=concatenation([x_f,x])) x=x+ffw_i (x).

[0080] Here, attention_i represents the attention unit 303 performed by the i-th resampler layer 300 based on the corresponding key, value and query matrices, and ffw_i represents the operation performed by the corresponding perceptron 307 of the i-th resampler layer 300.

[0081] Thus, the resampler 210 forms a variable-size network of spatial visual features (spatial-temporal features in the case of Fig. 4) to a fixed number of output tokens, regardless of the resolution of the input image or the number of input video frames. Each resampler 210 has a set of learned latent vectors as queries, and the keys and values ​​are a concatenation of the visual features with the learned latent vectors.

[0082] Back to Fig. 2, the query processing system 200 further includes a data element token processing model 220. This comprises a stack of layers, which are the token processing layers 131, 132, ..., 13n of the token processing model of Fig. 1(b), interleaved by a number j of directed cross-attention layers 231, 232, ...., 23j. The number j of directed cross-attention layers may be equal to the number n of token processing layers, but in variations, n and j may differ from each other, so that, for example, any given number of token processing layers may lie between any given pair of directed cross-attention layers.

[0083] The outputs from the resampler layers 210, 211 are used as control inputs for the controlled cross-attention layers. Optionally, the same outputs of the resampler layers 210, 211 can be used as control inputs for all controlled layers 231, 232, ...23j. This has been implemented in some successful experiments in which 64 latent vectors were used, and six resampler layers and the corresponding 64 vectors output by the recipient resamplers 210, 211 were used together as control inputs for all controlled cross-attention layers. However, this is not the only possibility. For example, there can be correspondingly different (learned) latent vectors to generate respective control inputs for corresponding ones of the controlled cross-attention layers.

[0084] The input token string is used as the prompt input for the data element token processing model 220 and is fed to the first processing layer of the stack, which may be the first controlled cross-attention layer 231. The data passes through (in Fig. 2 onwards) the data element token processing model 220 to create a token string that is the output of the data element token processing model 220.

[0085] The cross-attention layers 231, 232, ..., 23j each receive at least one of the compressed representations from the modality networks 207, 209 and perform a gating operation based on the received compressed representations.

[0086] Fig. 6 shows the structure of two successive layers 523, 513 of the data element processing model 220 of Fig. 2. Layer 523 is a controlled cross-attention layer, which is one of layers 231, 232, ..., 23j. Layer 513 is one of the token processing layers 131, 132, ..., 13n.

[0087] The directed cross-attention layer 523 receives the inputs 51, 52 (compressed representations) from the respective modality networks 207, 209. Furthermore, a vector is received, which is a linguistic input 53 and can be denoted by y. If the directed cross-attention layer 523 is the first layer of the stack, the linguistic input 53 is typically the input token string (prompt input) or a portion thereof. Otherwise, the linguistic input 53 to the directed cross-attention layer 523 is the output of a previous layer of the data element token processing model 220.

[0088] The controlled cross-attention layer 523 applies an attention function 501 as previously described, where the vision inputs 51, 52 can define key and value vectors, and a query vector is defined based on the speech input 53. The output of the attention function 501 is added to the speech input 53 by the addition unit 502. The result is then input to an FFW network 503, such as a multi-layer perceptron. The output of the FFW network 503 is a vector that can be denoted by x. This is input to a tanh gating unit 504, which applies a component-wise operation in which each component is multiplied by a gating parameter α and a tanh function is applied to the result. The output of the FFW network 504 is added to its input by the addition unit 505.This produces the output of the steered cross-attention layer 523, which can be written as. y + tanh (αx).

[0089] The token processing layer 513 includes, as in some known systems, a self-attention layer 506. The attention layer operates on key vectors, value vectors, and query vectors generated from the input to the token processing layer 513. The output of the self-attention layer 506 is added to its input by an addition unit 507, and the result is fed to a feedforward layer (e.g., a multi-layer perceptron) 508. The output of the FFW network 508 is added to the input of the FFW network 508 by the addition unit 509. This creates the output of the token processing layer 513.

[0090] Note that when the gating parameter α is close to zero, the output of the controlled cross-attention layer 523 is very close to the speech input 53, so the query processing system 200 performs almost the same function as that in Fig. 1(b). However, in use (i.e., after the query processing system 200 has been trained), the value of the gating parameter is significantly higher than zero, so the compressed representations (e.g., vision inputs) 51, 52 influence the output of the token processing layer 513.

[0091] A typical operation of the trained query processing system 200 is shown in Fig. 6. The query input 61 includes two data elements 203, 205 that are in the form of still images: still images of a cat and a dog, respectively. As previously explained, the data elements 203, 205 are input to respective modality networks 207, 209 or sequentially input to the same modality network to generate corresponding compressed representations of the data elements 203, 205. Since the data elements 203, 205 are in the form of still images, the query processing system 200 uses modality network(s) in the form shown in Fig. 3. If the data elements are videos, the modality network(s) can be configured in the Fig. 4 illustrated form.

[0092] The words of query input 61 are used to generate an input token string for a query input of query processing system 200. Optionally, the query input may contain one or more "markers," e.g., for each data element 203, 205 of query input 61, to indicate the existence of the associated element. For example, the input token string may consist of the following ASCII characters: " This is a very cute dog. This is". Optionally, additional markers can be provided. For example, a marker (e.g. <eoc>, which means "end of chunk"). The markers can be formed from strings of tokens from the vocabulary or from tokens from outside the vocabulary.

[0093] The query processing system 200 processes the input token string by passing it through the layers 231, 131, 232, 132, ...., 23j, 13n, where the controlled layers 231, 232, ...., 23j are controlled based on the vision inputs generated by the modality networks 207, 209 based on the corresponding data elements 203, 205, i.e., the two still images of the query input 61. In this way, the query processing system 200 generates an output token string, such as the output token string 62 from Fig. 6. The output token string is a string of ASCII characters that reads "a very serious cat." This is a logical, grammatical continuation of the query input 61. The first image 203 and the associated text "This is a very cute dog" represent "a task example." The output of tokens is the result of performing the task exemplified by the task example for the second data element 205 (the "data element in question").

[0094] Method for training a query processing system, such as the query processing system 200 of Fig. 2, are now considered. The methods may be implemented by one or more computer systems at one or more locations. As already mentioned, the encoder network 11 and the token processing layers 131, 132, ..., 13n may be implemented as part of the systems of Fig. 1(a) and Fig. 1(b). The encoder network 11 and the token processing layers 131, 132, ....13n are not additionally trained during the training of the other layers of the query processing system 200. This means that the amount of data required in the training database is small, e.g., much smaller than a training database previously used to train the token processing layers and / or the encoder network(s).

[0095] Training is based on a training database of multi-mode training examples. Fig. 7 shows a training example 71. This optionally contains one or more task examples. A task example 701 is shown, which includes a data element (photo of a dog) and text that refers to the data element ("This is a picture of my dog"). Different training examples may include a different number of tasks or no tasks at all. The training example 71 further includes a section 702, which is a data element (photo of a cat) and text that refers to the data element ("This is a picture of my cat"). If the training example includes task examples, the text of section 702 has the same semantic relationship to the data element of section 702 as the text in each of the training examples has to the data element of the training example.

[0096] The modality network(s) 207, 209 and the gated cross-attention layers 231, 232, ... 23j can be used together in the Fig. 8. The system may employ a training database 801 consisting of training examples, such as the one shown in Fig. 7, each of which includes at least one data element and one token string (“multi-mode” training examples).

[0097] During training, a series of training iterations are performed. In each iteration, one or more of the training examples may be selected. In a typical implementation, multiple training examples are selected and treated as a batch, but for simplicity, the case of a single selected training example is considered here. A splitting unit 802 receives the selected training example and splits the token string of the selected training example into a first portion, which is an earlier part of the token string, and a second portion, which is a later part of the token string. The data item(s) of the selected training example and the first portion of the token string of the selected training example together constitute a query input to the query processing system 200.The data item(s) are input to the modality network(s) 207, 209 of the query processing system 200, and the first portion of the token string is used to generate a prompt input for the data item token processing model 220 of the query processing system 200. Thus, based on the selected training example, the data item token processing model 220 generates an output token string from the compressed representations generated by the modality network(s) and the prompt input.

[0098] A discrepancy unit 803 receives the second portion of the token string of the selected training example from the splitting unit and the output token string and calculates a measure of the discrepancy (the difference between them).

[0099] An update unit 804 then generates an update for the query processing system 200—i.e., for the modality network(s) and the controlled cross-attention layer(s)—that increases the statistical correlation between the output token string and the second portion of the token string from the training example. The process may be repeated for different selections of the position in the token string of the selected training example, where the split unit 802 splits the token string into the first and second portions, and the update generated by the update unit 804 may be such that the discrepancy is reduced for several of these selections, e.g., an average across the selections.

[0100] For example, if the token string of the training example is denoted by y and the data item(s) of the training item are denoted by x, the modification may be such to increase p(y|x)=∏l=1Lp(yl|y <l,x≤l), where L is the number of characters in the token string of the training example, y l denotes the l-th character of the token string, y <l denotes the l-1 previous characters of the token string, x ≤l denotes the data element(s) before the l-th character of the token string, and p(y l |y <l , x ≤l ) denotes the probability of the data element token processing network producing an output token string equal to y l begins, based on a prompt that y <l and the compressed representations based on the modality network from the data elements x ≤l be generated.

[0101] Indeed, as stated above, each of the training iterations is preferably performed using multiple training examples. For example, the training examples in the training database can be partitioned into groups ("datasets") of training examples in the training database. There can be an integer number M of datasets. The m-th dataset, where m = 1,..., M = 1, is denoted by D m This is assigned a corresponding scalar weighting λ m which can be the same for all batches, but can alternately differ for different batches (e.g., lower for batches of training examples obtained from sources where there is a lower correlation between text and images). If all datasets are of the same quality, partitioning the training examples may have no value (e.g., M can be set to 1). In each training iteration, a corresponding batch of B m Training examples are randomly selected from each of the M datasets, and an update of the variable parameters of the compressed representation generation system and the guided cross-attention layers is performed to reduce a loss function that indicates the failure of the data item token processing model in predicting consecutive tokens in the batches of training examples.

[0102] For example, the loss function can be the expected negative log-likelihood of the second section of the token string of the training examples, given the first section, averaged over all possible positions l where the token string of the training example can be positioned: ∑m=1Mλm⋅E(x,y)~Dm[−∑l=1Llogp(yl|y <l,x≤l)] where E denotes the expected value. Gradients of the loss function can be computed over all M datasets to perform an update step on the variable parameters of both the controlled cross-attention layer and the modality network(s).

[0103] Initially, during the training procedure, the value of the gating parameter α can be chosen low (e.g., 0.01), so that the influence of the data elements on the output token string is correspondingly small. Thus, at the beginning of training, the controlled cross-attention layers can barely affect the operation of the trained token processing layers 231, 232, ... 23j of the data element token processing network 220. During training, as the controlled cross-attention layers gradually learn to cooperate with the token processing layers, the value of the gating parameter α is gradually increased, e.g., to α=1.

[0104] Preferably, the "masking" is applied to at least one (or all) of the controlled cross-attention layers 231, 232, ...23j during both training and operation. This means that the controlled cross-attention layer does not generate the interaction terms based on the compressed representations of all data elements of the query input and the data derived from the entire prompt input, but rather that the interaction term is generated based on the compressed representations of only a (correct) subset of the data elements in the query input and only a correct subset of the prompt input. For example, the query input may include (or consist of) a plurality of (non-overlapping) sections, each of which may contain exactly one of the data elements and may also contain a plurality of tokens of the input token string.For example, some of the section(s) may be "task example" sections that have already been discussed, and / or one of the sections may include the data item in question. For each section, the modality network generates at least one respective compressed representation of the corresponding data item, as described above. In the case of masking, one or more of the directed cross-attention layers may be associated with each of the sections. For a given section, the one or more associated directed cross-attention layers generate the interaction term based only on the compressed representation of the data item of the associated section (i.e., without using the compressed representations of other data items of the query input) and without employing data generated based on tokens of the input token string that are not within the associated section (i.e.,only using tokens contained in the part of the input token string that lies within the associated section). Note that masking preferably applies only to the interaction term but does not prevent information about the entire prompt input from flowing through the data element token processing model, so that all token processing layers have access to this information.

[0105] In Fig. 9, a method 900 is shown that is an example of the present disclosure. The method may be performed by one or more computers at one or more locations. The method 900 is performed by a query processing system, such as the query processing system 200 of Fig. 2. The procedure is performed on a query input that has one or more data elements and an input token string.

[0106] In step 901, each data element of a query input (such as query input 61 of Fig. 6) input into a corresponding modality network to generate one or more compressed representations of each data item. If the data items have different modalities, then typically at least one modality network exists for the data item(s) of the respective modality. For a given data item, there may be one or more modality networks used for data items of that modality.

[0107] In step 902, the input token string is used to generate a prompt input. This may include inserting at least one marker for at least one data element of the query input into the prompt input.

[0108] In step 903, the prompt input is processed using a data element token processing model, such as the data element token processing model 220 of Fig. 2, which also receives the compressed representations generated in step 901, to control the layers of the data element processing model. The output of the data element processing model is an output token string, such as the output token string 62 in Fig. 6.

[0109] In Fig. 10, a method 1000 is shown that is an example of the present disclosure. The method may be performed by one or more computers at one or more locations. The method 1000 generates a query processing system, such as the query processing system 200 of Fig. 2.

[0110] In step 1001, token processing layers of a trained language model, such as the one in Fig. 1(b) is used to implement a data element token processing model, such as the data element token processing model 220 of Fig. 2, by using the token processing layers 131, 132,13n (trained in a system such as that of Fig. 1(b)) are nested with untrained controlled cross-attention layers 231, 232,23j.

[0111] In step 1002, a query processing system, such as query processing system 200, is formed using the data item token processing model formed in step 1001 and modality network(s), such as modality networks 207, 209. The modality networks employ trained encoder networks 11 and (e.g., untrained) resamplers 210, 211.

[0112] In step 1003, the resamplers 210, 211 and the gated layers 231, 232, ...23j are trained together so that they are Fig. 8. Training the resampler layers 210, 211 and the controlled layers 231, 232, ...23j involves training the corresponding key, value, and query matrices. Training the resampler 210 further involves training the latent vectors.

[0113] The experiments were conducted in the query processing system 200 from Fig. 2 to demonstrate that it is capable of performing diverse and demanding tasks. For example, Fig. 11 a dialogue between a human user and an implementation of the existing system. The query processing system was thus based on a large language model, ie a token processing system such as the one in Fig. 1(b) shown the Chinchilla model by J. Hoffmann et al., which was implemented with 80 billion variable parameters.

[0114] Initially, the user created a query input containing data element 111 (a still image) and an input token string 112 (the words "Which city is this?").

[0115] The query processing system processed the query input to produce an output token string 113 (the words "This is a street sign in New York").

[0116] The user then created a new query input containing data element 111, input token string 112, output token string 113, and a new input token string 114. The new input token string consists of the words "Which street corner is it?"

[0117] The query processing system processed the new query input to produce a new output token string 115 (the words "It is the intersection of Madison Avenue and East 42 nd Street").

[0118] Thus, a dialogue was conducted between the human user and the query processing system, in which the human user successively obtained information based on the data element of the initial query input. This dialogue could be continued in a series of additional steps, with the user, in each step, concatenating the data element 111, the input token strings and the corresponding generated output strings of one or more previous steps, as well as a new input token string to generate a new query input that is input to the query processing system.

[0119] A system comprising one or more computers may be configured to perform specific operations or actions by installing software, firmware, hardware, or a combination thereof that, when used, causes the system to perform the operations or actions. One or more computer programs may be configured to perform specific operations or actions, meaning that the one or more programs include instructions that, when executed by a computing device, cause the device to perform the operations or actions.

[0120] Embodiments of the subject matter and the functional operations described in this specification may be implemented in digital electronic circuits or in physically embodied computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a physical non-transitory program carrier for execution by, or for controlling the operation of, a data processing device.Alternatively or additionally, the program instructions may be encoded on a synthetically generated propagated signal, for example, a machine-generated electrical, optical, or electromagnetic signal generated to encode information for transmission to a suitable receiving device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or a combination of one or more thereof. However, the computer storage medium is not a propagated signal.

[0121] The term "stack" of layers refers to a sequence of layers, each of which receives a data input and produces a data output. Each layer other than the first layer receives as at least part of its input a portion of the output of the previous layer in the sequence. Thus, the data flows through the stack from the first layer to the last layer of the sequence, and the output of the stack of layers includes the output of the last layer of the sequence.

[0122] The term "data processing device" encompasses all types of devices, apparatus, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. The device may include special-purpose logic circuits, such as an FPGA (field-programmable gate array) or ASIC (application-specific integrated circuit). The device may also optionally include hardware code that generates an execution environment for the particular computer program in question, such as code representing processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0123] A computer program (which may also be called or described as a program, software, software application, module, software module, script, or code) may be written in any programming language, including compiled or interpreted languages, or declarative or procedural languages, and it may be provided in any form, including as a stand-alone program or module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a part of a file that contains other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in several coordinated files, such asFiles that store one or more modules, subprograms, or sections of code. A computer program can be deployed to run on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0124] As used in this specification, an "engine" or "software engine" refers to a software-implemented input / output system that provides output that is different from the input. An engine can be a coded block of functionality, such as a library, a platform, a software development kit ("SDK"), or an item. Each engine can be implemented on any suitable type of computing device, such as servers, mobile phones, tablet computers, notebooks, music players, e-book readers, laptop or desktop computers, PDAs, smartphones, or other stationary or portable devices that include one or more processors and computer-readable media. Additionally, two or more of the engines can be implemented on the same computing device or on different computing devices.

[0125] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by processing input data and generating output. The processes and logic flows can also be performed by special-purpose logic circuits, and the device can also be implemented as such, such as an FPGA (field-programmable gate array) or ASIC (application-specific integrated circuit). For example, the processes and logic flows can be performed by a device and can also be implemented as a graphics processing unit (GPU).

[0126] Computers suitable for executing a computer program may, for example, include general-purpose or special-purpose microprocessors, or both, or any other type of central processing unit. Generally, a central processing unit receives instructions and data from read-only memory or random-access memory, or both. The essential elements of a computer are a central processing unit for performing or executing instructions, and one or more storage devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic, magneto-optical, or optical disks, or is operatively coupled to one or more of these devices to receive data from them, transmit data to them, or both. However, a computer is not required to include such devices.In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) memory stick, to name a few.

[0127] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by or integrated with dedicated logic circuitry.

[0128] To provide interaction with a user, the subject matter embodiments described in this specification may be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) with which the user can provide input to the computer. In addition, other types of devices for interacting with a user may be used; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including auditory, voice, or tactile input.Additionally, a computer can interact with a user by sending and receiving documents to and from a device used by the user. For example, it can send web pages to a web browser on a user's client device in response to requests received from the web browser.

[0129] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component, e.g., as a data server; or that includes a middleware component, e.g., an application server; or that includes a front-end component, e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein; or any combination of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0130] The computing system can include clients and servers. A client and a server are generally remote from each other and typically interact via a communications network. The client-server relationship arises because the respective computers run computer programs that are connected to each other in a client-server relationship.

[0131] Although this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention that may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features described in this specification in connection with separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in connection with a single embodiment may also be implemented in multiple embodiments separately or in any suitable combination.Furthermore, although features are described above as operating in certain combinations and are initially claimed as such, in some cases one or more features of a combination may be extracted from that combination, and the claimed combination may relate to a subordinate combination or a modification of a subordinate combination.

[0132] Although the operations are depicted in the drawings in a particular order, this should not be understood to mean that it is required to achieve the desired results by performing these operations in the particular order shown or in the order specified, or that all of the illustrated operations be performed. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system modules and components in the embodiments described above should not be understood to mean that such separation is required in all embodiments. It should also be understood that the described program components and systems may generally be integrated together into a single software product or bundled into multiple software products.

[0133] Certain embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims may be performed in a different order and still achieve the desired results. By way of example, the processes depicted in the accompanying figures do not necessarily require the particular order or sequential order shown to achieve the desired results. In certain implementations, multitasking and parallel processing may be advantageous. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Zitierte Nicht-Patentliteratur

[0000] Brown et al, „Language models are few-shot learners", in Conference on Neural Information Processing Systems, 2020 [0027, 0064] Vaswani et al. „Attention is all you need", 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA

[0037] High-Performance Large-Scale Image Recognition Without Normalization" von A. Brock et al 2021

[0063] Learning Transferable Visual Models From Natural Language Supervision", von A. Radford et al, 2021

[0063] Chinchilla-Modell von „Training compute-optimal large language models", J. Hoffmann et al, 2022

[0064] < / eoc>

Claims

[1] A computer-implemented method for generating an output token string based on a query input comprising an input token string and one or more data items, wherein the input token string and the output token string are strings of tokens selected from a token vocabulary, and wherein the data items are of a modality different from tokens selected from the token vocabulary, the method comprising: Inputting each data item of the query input into a modality network that is trained upon receiving a data item of the modality to generate one or more compressed representations of each data item; generating a prompt input that includes the input token string of the query input; and Inputting the prompt input into a data item token processing model having a plurality of processing layers arranged as a stack, wherein the output token string is an output of the data item token processing model, wherein the processing layers include a plurality of token processing layers and a plurality of controlled cross-attention layers, each controlled cross-attention layer arranged to receive at least one of the compressed representations, wherein the token processing layers are interleaved with the controlled cross-attention layers. [2] The computer-implemented method of claim 1, wherein the token processing layers are operable, in the absence of the gating cross-attention layers, to jointly provide a token string processing model for receiving input token strings and generating corresponding output token strings. [3] A computer-implemented method according to claim 1 or claim 2, comprising: Generating an output token string based on a query input; and performing at least once the set of the following steps: based on the query input and the output token string, forming a new query input; and Generate a new output token string based on the new query input. [4] A computer-implemented method for training a query processing system, the query processing system being for generating an output token string based on a query input comprising an input token string and one or more data items, wherein the input token string and the output token string are strings of tokens selected from a token vocabulary, and wherein the data items are of a modality different from tokens selected from the token vocabulary, wherein the method employs a token processing model comprising a stack of token processing layers, the stack of token processing layers configured to receive input token strings and generate corresponding output token strings, and a database of training examples, each training example comprising at least one data item and at least one token string; the method comprising: Forming a data item token processing model by nesting the token processing layers with controlled cross-attention layers, wherein the data item token processing model is configured to generate an output token string upon receiving a prompt input that is a token string; Forming the query processing system, comprising: (a) a modality network configured to receive the data elements of the query input to generate one or more compressed representations of each data element; and (b) the data element token processing model, wherein the data element token processing model is configured to receive a prompt input comprising the input token string of the query input, and each controlled cross-attention layer is configured to receive at least one of the compressed representations; and Using the training database, training: the modality network and the multitude of controlled cross-attention layers. [5] The computer-implemented method of claim 4, wherein training trains the query processing system when an encoder of the modality network receives the at least one data item of one of the training examples, and wherein the data item token processing model receives a prompt input comprising a first portion of the token string of the training example to generate an output of the query processing system that is positively statistically correlated with a subsequent portion of the token string of the training example. [6] A computer-implemented method according to any preceding claim, wherein the modality network comprises: an encoder configured to encode a data item received by the encoder to generate an encoded data item, and a compressed representation generation system arranged to receive the encoded data item and generate an output, wherein the output of the modality network is based on the output of the compressed representation generation system. [7] A computer-implemented method according to claim 6 when dependent on claim 4, wherein the encoder has been trained to encode a data item received by the encoder to generate an encoded data item, and wherein training the modality network and the plurality of controlled cross-attention layers comprises training the compressed representation generation system without further training the encoder. [8] A computer-implemented method according to claim 6 or claim 7, wherein the compressed representation generation system comprises a stack of one or more resampler layers, wherein each resampler layer is adapted to perform an attention operation employing a key vector, a value vector, and a query vector, wherein a subset of the key vector, the value vector, and the query vector is based on the encoded data item, and the remainder of the key vector, the value vector, and the query vector is based either on an output of the previous one of the resampler layers or, in the case of the first resampler layer of the stack, on a set of latent input values, where the output of the modality network is based on an output of the last resampler layer of the stack of resampler layers. [9] A computer-implemented method according to claim 8, wherein the key vector and the value vector of each resampler layer are based on the encoded data item and a latent input which is either the output of the previous one of the resampler layers or, in the case of the first resampler layer of the stack, the set of latent input values, and the query vector is based on the latent input. [10] A computer-implemented method according to claim 8 or 9, wherein each resampler layer further comprises a perceptron arranged to receive the output of the attention operation and to generate an output, the output of the modality network being based on the output of the perceptron of the last resampler layer of the stack. [11] A computer-implemented method according to any preceding claim, wherein the prompt input further comprises one or more corresponding marker elements for each data element in the query input, the one or more marker elements indicating the presence of the data element in the query input. [12] The computer-implemented method of claim 11, wherein a position of each marker element in the prompt input indicates a position of the corresponding data element in the query input. [13] A computer-implemented method according to any preceding claim, wherein each controlled cross-attention layer produces its output as a component-wise sum of: a first input, which is the output of the previous processing layer in the stack of processing layers or, in the case that the controlled cross-attention layer is the first processing layer of the stack of processing layers, the prompt input, and an interaction term based on the output of the compressed representation generation system received by the directed cross-attention layer and at least a portion of the first input to the directed cross-attention layer. [14] A computer-implemented method according to claim 13 when dependent on claim 4, wherein the interaction term has a magnitude that depends positively on the value of a gating parameter, wherein training comprises incrementally increasing the learning parameter. [15] A computer-implemented method according to claim 13 or claim 14, including, in the case of a query input comprising a plurality of sections, each section comprising one of the data elements, wherein for each section: the modality network generates at least one respective compressed representation of the corresponding data element, and at least one of the controlled cross-attention layers that generates the interaction term based only on the compressed representation of the corresponding data item and without inserting data generated based on tokens of the input token string other than within the section. [16] A computer-implemented method according to any one of the preceding claims, wherein each controlled cross-attention layer comprises a cross-attention layer employing a key vector, a value vector, and a query vector, wherein a subset of the key vector, the value vector, and the query vector is based on the at least one compressed representation received from the controlled cross-attention layer, and wherein the remainder of the key vector, the value vector, and the query vector is based on the output of the previous processing layer in the stack of processing layers, or in the case that the controlled cross-attention layer is the first processing layer of the stack of processing layers, is based on the prompt input. [17] The computer-implemented method of claim 16, wherein the key vector and the value vector of each steered cross-attention layer are obtained based on the at least one compressed representation received by the steered cross-attention layer, and wherein the query vector of each steered cross-attention layer is based on the output of the previous processing layer in the stack of processing layers or, in the case that the steered cross-attention layer is the first processing layer of the stack of processing layers, on the prompt input. [18] The computer-implemented method of claim 16 or 17, wherein the controlled cross-attention layer further comprises a perceptron that receives the output of the cross-attention layer, the output of the controlled cross-attention layer being based on an output of the perceptron. [19] A computer-implemented method for generating a compressed representation of a data item having a modality, the method comprising processing the data item using a modality network, comprising: an encoder configured to encode the data element to generate an encoded data element and a compressed representation generation system arranged to receive the encoded data item and generate an output, wherein the compressed representation generation system comprises a stack of one or more resampler layers, wherein each resampler layer is configured to perform an attention operation employing a key vector, a value vector, and a query vector, wherein the key vector, the value vector, and the query vector are each based on at least one of the encoded data item and a latent input that is either an output of the previous one of the resampler layers or, in the case of the first resampler layer of the stack, a set of latent input values, wherein at least one of the key vector, the value vector, and the query vector is based on both the encoded data item and the latent input, where the output of the modality network is based on an output of the last resampler layer of the stack of resampler layers. [20] The computer-implemented method of claim 19, wherein the key vector and the value vector of each resampler layer are based on the encoded data item and the latent input, and wherein the query vector of each resampler layer is based on the latent input. [21] A computer-implemented method according to claim 19 or claim 20, wherein each resampler layer is configured to add the latent input to the output of the attention operation. [22] A computer-implemented method according to any one of claims 19 to 21, wherein each resampler layer further comprises a perceptron arranged to receive the output of the attention operation and to generate an output, the output of the modality network being based on the output of the perceptron of the last resampler layer of the stack of resampler layers. [23] A computer-implemented method for generating an output token string based on a query input comprising an input token string and one or more data items, wherein the input token string and the output token string are strings of tokens selected from a token vocabulary, and wherein the data items are of a modality different from tokens selected from the token vocabulary, the method comprising: (a) generating one or more compressed representations of each data item by processing the data item using a modality network comprising: an encoder configured to encode the data element to generate an encoded data element and a compressed representation generation system arranged to receive the encoded data item and generate an output, wherein the compressed representation generation system comprises a stack of one or more resampler layers, wherein each resampler layer is configured to perform an attention operation employing a key vector, a value vector, and a query vector, wherein the key vector, the value vector, and the query vector are each based on at least one of the encoded data item and a latent input that is either an output of the previous one of the resampler layers or, in the case of the first resampler layer of the stack, a set of latent input values, wherein at least one of the key vector, the value vector, and the query vector is based on both the encoded data item and the latent input, wherein the output of the modality network is based on an output of the last resampler layer of the stack of resampler layers; (b) generating a prompt input comprising the input token string of the query input; and (c) inputting the prompt input and the compressed representation of each data element into a data element token processing model configured to generate the output token string based on the prompt input and the compressed representation of each data element. [24] A computer-implemented method according to any one of claims 1 to 18 or 23, wherein the input token string is the response to a query about the contents of a respective data item that is one of the data items in the query input, the query being defined based on the input token string. [25] A computer-implemented method according to claim 24, wherein the query input comprises, in addition to the respective data item, one or more task example portions, each including a respective data item and a respective portion of the input token string, and wherein for each task example portion, the respective portion of the input token string is the answer to the query when the query relates to the contents of the respective data item. [26] A computer-implemented method according to claim 24 or 25, wherein the query input is a question and the response to the query is an answer to the question. [27] A computer-implemented method according to any one of claims 1 to 18 or 23-26, wherein the token vocabulary comprises the symbols of a natural language writing system. [28] A computer-implemented method according to any preceding claim, wherein the modality comprises at least one of: (i) still or video images or (ii) audio signals. [29] A system comprising one or more computers and one or more memory devices storing instructions which, when executed by the one or more computers, cause the one or more computers to perform the method of any preceding claim. [30] One or more computer storage media storing instructions which, when executed by one or more computers, cause the one or more computers to implement the method of any one of claims 1 to 28.

Images