Language model for processing multi-modal query input

Through the cooperation of the modal network of the query processing system and the gated cross attention layer, the accurate output of the multi-mode query input is generated, which solves the problem of insufficient processing accuracy of the multi-mode query input in the prior art, and reduces the amount of training data and calculation Resource requirements improve system efficiency and flexibility.

CN119096249APending Publication Date: 2024-12-06GDM HOLDING LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380036379.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-04-28
Filing Date
2023-04-28
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

When existing neural networks process multi-mode query inputs, it is difficult to generate accurate output responses, especially when zero-test or few-test query inputs, the accuracy is insufficient, and the amount of training data and computing resources are required.

Method used

Using a query processing system, the system includes a modal network and a data item word processing model. It generates a compressed representation of data items through a modal network, and uses a gated cross attention layer to cooperate with the pre-trained word processing layer to generate output. Metastring. The system reduces the amount of training data and computing resource requirements through multi-modal training examples.

Benefits of technology

It realizes high accuracy processing of multi-mode query input, especially when query input is excellent, and reduces the amount of training data and computing resources requirements, and improves the efficiency and flexibility of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119096249A_ABST
    Figure CN119096249A_ABST
Patent Text Reader

Abstract

A query processing system is described that receives a query input comprising an input string of lems and at least one data item having a second, different modality, and generates a corresponding output string of lems.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present specification relates to a neural network configured to process a multimodal query input (eg, a mixture of text and sound / image) to generate an output that is a response to the query input. Background Art

[0002] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict outputs from received inputs. Some neural networks include one or more hidden layers in addition to the output layer. The output of each hidden layer is used as the input to the next layer in the network, the next hidden layer or output layer. Each layer of the network generates an output from the received inputs based on the current values ​​of the corresponding set of parameters.

[0003] It is known to employ a language model that, upon receiving an input grammar string, which is a sequence of grammars selected from a grammar vocabulary (e.g., a text piece consisting of letters selected from an alphabet (e.g., a natural language alphabet), a text piece consisting of subwords or word pieces selected from a corresponding vocabulary, a text piece consisting of phonemes selected from a corresponding vocabulary, etc.), generates an output grammar string (another sequence of grammars selected from the grammar vocabulary), which output grammar string is a reasonable response to the input grammar string, e.g., a seemingly reasonable continuation of the input grammar string, or an answer to a question posed by the input grammar string. Summary of the invention

[0004] The present disclosure describes a system ("query processing system") implemented as a computer program in one or more computers in one or more locations that receives a query input comprising an input word-gram string and at least one data item having a second, different modality.

[0005] For example, the data items may be images. Each data item may be a still image, for example, the data item may be a pixel value (e.g., a red-green-blue (RGB) value) for each pixel of a pixel array. Alternatively, one or more of the data items may be a video image, for example, for each of a plurality of frames, the data may be a pixel value for each pixel of a corresponding array. The images may be captured by imaging the real world with a still or video camera.

[0006] In another possibility, the data item may be a sound signal.A sound signal is audio data representing the value of an audio waveform at each of a plurality of times, for example, sound captured by a microphone during a period of time.

[0007] In a further possibility, the data item may be a video image with an accompanying corresponding audio track.

[0008] The query processing system is operable to generate an output word-gram string based on a query input. The input word-gram string and the output word-gram string are each a sequence of word-grams selected from a (single) word-gram vocabulary (e.g., both strings may be strings of letters from a Roman alphabet, strings of subwords or word-grams from a word-gram vocabulary, etc.). The word-gram vocabulary may be a word-gram vocabulary of a natural language (e.g., Roman letters in the case of English, or Roman letters plus Roman letters with accents in the case of French).

[0009] In general, the present disclosure suggests that a query processing system extracts data items from a query input and inputs them into a modal network that is configured to generate one or more corresponding compressed representations of each data item from them. The query processing system uses an input word-meta string to generate a prompt input for a data item word-meta processing model. The data item word-meta processing model includes a stack of processing layers, the stack including a word-meta processing layer and a gated cross-attention layer interleaved with the word-meta processing layer. The prompt input can be supplied to a first processing layer of the stack. The gated cross-attention layers each receive at least one of the compressed representations and perform a gating operation based on the received compressed representations. The output word-meta string is the output of the data item word-meta processing model.

[0010] The output word-element string is a reasonable response to the query input. Specifically, the input word-element string may at least partially define a question about the content of at least one of the data items in the query input (the "subject data item"), and the output string may be a (correct) response to the question. Thus, the question defines a data item processing task to be performed on the subject data item.

[0011] A query processing system can be viewed as a generalization of a classifier neural network. A classifier neural network typically receives a data item and determines to which of a predetermined plurality of classes the data item belongs (e.g., if the data item is an image showing an object, the classifier may output data indicating to which of a predetermined set of classes the object belongs). In contrast, in a query processing system, an input term-gram string input by a query determines the processing tasks that the query processing system should perform on the subject data item to generate an output term-gram string. In other words, the input term-gram string defines (e.g., in a natural language) the transformations that are applied to the subject data item to generate an appropriate output term-gram string. Viewed from another perspective, the input term-gram string defines the information that the query processing system is to extract from the subject data item, and the output term-gram string encodes that information.

[0012] The input word-gram string may explicitly include a question (data item processing task). For example, in the case where the subject data item is a still image or a video, the input word-gram string may be "what activity is shown?".

[0013] Alternatively or additionally, the input word-element string may implicitly define a question, for example, by giving one or more examples of data item processing tasks. That is, in addition to the subject data item, the query input may also include one or more "task example" parts. Each task example part includes a data item and a corresponding section of the input word-element string, which is the answer to the question when a question is asked about the content of the data item in the example section (i.e., the result of applying the question to the data item).

[0014] For example, each task example part can be an image and a sentence related to the subject of the image. One possible task example part would be a photo of a chinchila, and a segment of the input word-word string that is the text "This is a chinchila. They are mainly found in Chile". This task example part implicitly defines the question as "What animal is shown in the image. In what country is it commonly found?" Note that this question leads not only to information about the content of the image, but also to information that is not present in the image but is associated with the content (i.e., the countries where the animal depicted in the subject data item is commonly found).

[0015] Thus, the query processing system identifies a data item processing task that, when applied to a data item in a task example portion, results in a segment of an input word meta string in the task example portion, and applies the processing task to the subject data item to generate an output word meta string. If there are multiple task example portions in the query input, the query processing system identifies a data item processing task that, when applied to a corresponding data item in the task example portion, results in a corresponding segment of an input word meta string in each of the task example portions, and applies the processing task to the subject data item to generate an output word meta string.

[0016] A query input without a task example portion (e.g., a query input that includes only a subject data item and an input word string that explicitly defines the question) may be referred to as a "zero shot" query input. The query input includes (in addition to the subject data item and optionally a portion of the input word string that helps define the question, such as an explicit statement of the question) the number The query input for the sample portion is called " try( -shot)" query input. The "fewshot" query input is A query input when the value is lower, such as 1 or 2. For example, the query input may consist of the text "Name the artist", followed by a data item of a picture of a painting by Picasso, followed by the word "Picasso", followed by a subject data item of a picture of a painting by Salvador Dali. This is a "one-shot" query input because there is one task example portion (a picture of a painting by Picasso and the word "Picasso"). The input term-meta string of the query input is the text "Name the artist Picasso". It includes the word ("Picasso"), which is a segment of the input term-meta string that is a part of the task example portion.

[0017] In the case where the data items are video images, the query processing system can operate as a video processing system that is configured to analyze a sequence of video frames to detect objects in the video frames and provide information related to the detected objects in response to a question defined by an input word-gram string. The question can include, for example, a request for a prediction of a future event or state related to one or more of the objects (e.g., "will objects X and Y collide?"), or a request for conditional or counterfactual information related to one or more of the objects (e.g., "what event would [not] happen if object X is modified, moved or absent?"), or a request for analysis of video frames to determine properties or characteristics of one or more of the objects (e.g., "how many objects of type Z are moving?").

[0018] The response to the question provided as the output word-word string can be, for example, in the form of a yes / no answer, or a probability distribution over a set of possible answers can be defined; or the response can define the position of an object. The implementation of the system is flexible in the types of questions that can be defined, and a single implementation of the system can handle a variety of different types of questions. By way of example only, the system can be used to predict whether two objects will collide, or how a collision can be avoided. The response to the query can be used in many ways by a human or a computer system. For example, the analysis itself may be useful, or it may be used to provide a warning and / or control the movement of one or more of the objects.

[0019] The query processing system has many other technical applications. For example, a person (e.g., a visually impaired person in the case where the data item is a (still or video) image, or a hearing impaired person in the case where the data item is a sound signal) can use it to obtain information about the subject data item. For example, the query processing system can be part of a computer system that obtains the subject data item (e.g., in the form of a still or moving image using a camera of the computer system). A person can supply an input word-element string (e.g., by controlling a data input device, or by speaking a command that is converted into an input word-element string of letters or words, e.g., representing a phoneme). The query processing system can then generate an output word-element string, e.g., as a word-element string that is converted into a sound signal broadcast to the person by a speaker device. A useful feature of the computer system is its flexibility, because a person can freely define the information that the query processing system should extract from the subject data item.

[0020] Optionally, the computer system may be configured to allow a person to generate a series of multiple query inputs, each query input including the same subject data item but with a different input word element string. For example, if the subject data item is an image, the input word element string of the first query input may be "What is shown in the picture?" If the query processing system generates a corresponding output word element string (e.g., "Abus") in response to the first query input, the person may be able to define a second query input including the same subject data item and having a new input word element string (e.g., "What destination is on the front of the bus?").

[0021] Optionally, each query input other than the first query input in the series of query inputs may include input term-gram strings and output term-gram strings from one or more previous query inputs in the series. In this way, a person may engage in a coherent discussion with the query processing system about a subject data item (which is typically included in all query inputs in the series), wherein at each stage the query processing system generates an output term-gram string that is a reasonable continuation of an earlier part of the discussion.

[0022] Another technical application of the query processing system is a subtitle system for data items that are video items. The input word element string may define the video item processing task as "provide subtitles that explain what is happening in the video", and in this case, the output word element string may include subtitles that describe the events or actions performed in the video. If the data item is a video item with an audio track, the subtitles may include a transcription of the audio track.

[0023] In another application, the query processing system may generate output data that classifies the data item into one or more of a plurality of categories, for example by defining a score for each of a plurality of possible categories for the data item. As another specific application, in the case where the data item comprises a (still or video) image or a sound (audio data describing an audio waveform), the data item processing task (problem) may be to provide output data that defines (e.g., as a score) whether the image or video or audio waveform is described by the text included in the input word-element string.

[0024] In another technical application, the data item is a still or video image including text, and the computer system may include an at least partial OCR (optical character recognition) system for extracting text from the image to generate an input word-meta string. For example, if the data item is an image of a scene including a sign with instructions (e.g., the image may be an image of a road including a road sign containing text), the OCR may generate the input word-meta string based on the instructions in the image. In this case, the output word-meta image may indicate the importance of the instruction, given the rest of the scene (e.g., in the case of a data item that is an image of an empty parking space and a sign indicating that parking is allowed at the current time, the output word-meta string may be "Please park here"). In another example, the data item may be an image (e.g., a meme) including text extracted by the OCR to form the input word-meta string, and the output word-meta string may indicate (e.g., as a score) whether the combination of the image and text is offensive (e.g., inappropriate, disruptive, or profane).

[0025] In another technical application, the query input may instruct the query processing system to generate code that can be executed by the processor of the computer system as an output word element string. For example, the query input may read "[Image of a dog]Generate HTML code which when executed generates a video image of this animalmoving its body" (where the square brackets represent the data item in the query input, i.e., the image of a dog).

[0026] Note that a stack of token processing layers can operate on their own (i.e., if they are arranged as a stack in the same order, without interleaved gated cross-attention layers) to perform a token string processing operation that receives an input token string and generates an output token string from it. For example, the output token string can be an answer to a question defined by the input token string, or a (perceptible) continuation of a token sequence defined by the input token string, such as a continuation of a story started by the input token string.

[0027] The word-gram processing layer of the data item word-gram processing model may be a layer generated by a well-known language model training method (e.g., Brown et al, “Language models are few-shot learners”, in Conference on Neural Information Processing Systems, 2020). The term “training” is used herein to indicate that the parameters of the word-gram processing layer are iteratively adapted, for example based on a training basis of examples illustrating the tasks to be performed by the word-gram processing layer. After training the word-gram processing layer, the data item word-gram processing model may be formed by interleaving the trained word-gram processing layer with a gated cross-attention layer. The gated cross-attention layer may then be trained based on a training database, for example jointly with a modal network. Here, training refers to iteratively modifying variable parameters defining the gated cross-attention layer and variable parameters defining the modal network (as described below, this may be variable parameters defining a compressed representation generation system as a component of the modal network). The training is based on a training database of “multimodal” training examples, as described below. During this training, the word-gram processing layer is preferably “frozen”, i.e., substantially unchanged. This means that the amount of data required in the training database is small, for example much smaller than the training database used earlier to train the word-gram processing layer.

[0028] Optionally, the data items in a given query input may have different corresponding modalities, for example, one of the data items may be an audio signal while another may be a (still or moving) image. Optionally, a different modality network may be provided for each corresponding modality, and the data items in the query input having each corresponding modality are transmitted as input to the corresponding modality network. Each modality network produces a compressed representation of the corresponding received data item, and these compressed representations are transmitted to some of the gated cross-attention layers. For simplicity, the following discussion only considers the case where data items of one modality are present in the query, and only considers one modality network suitable for data items of that modality.

[0029] Note that in some embodiments, a data item consisting of a video image with an associated audio track can be considered to have a single modality and thus processed as a whole by a single modality network. Alternatively, in other embodiments, the video image of each data item can be split from the audio track, thereby generating two corresponding data items of different corresponding modalities that are processed by different corresponding modality networks of the query processing system.

[0030] The (or each) modality network may employ a pre-trained encoder network that has been pre-trained (e.g. according to known methods) to generate encoded data items when data items are received. The encoded data items may have a smaller number of components than the data items themselves. The encoder network may, for example, comprise one or more stacked convolutional layers, and the encoded data items may be feature maps.

[0031] The modality network may include an encoder network and a compressed representation generation system arranged to receive encoded data items and generate an output. The output of the modality network is (or more generally, is based on) the output of the compressed representation generation system. Using a pre-trained encoder network as part of the modality network, and not changing it substantially (or even at all) during the training of the compressed representation generation system, means that the compressed representation generation system can have fewer parameters than if the encoder network did not exist and the compressed representation generation system instead received the original (unencoded) data items of the training examples. This reduction in parameters, in turn, means that the training database used to jointly train the modality network (i.e., the compressed representation generation system part of the modality network) and the gated cross-attention layer does not require as many training examples, and the training process requires fewer computational resources.

[0032] As mentioned above, the compressed representation system and the gated crisscross attention layer can be jointly trained using a training database in an iterative process. Here, the term "joint training" means that iterative changes of the compressed representation generation system are interleaved or substantially simultaneous with corresponding iterative changes of the gated crisscross attention layer.

[0033] The training database consists of training examples, each training example comprising at least one data item and a word-meta string ("multimodal" training examples). A word-meta string is a word-meta string from the same word-meta vocabulary as the input word-meta string. During training, repeatedly, one of the training examples can be selected. The data items in the training items are input to the modal network in order to generate a corresponding compressed representation used by a gated cross-attention layer. The word-meta string of the training example is divided into a first part (representing an earlier local part of the word-meta string) and a second part (representing a later local part of the word-meta string). The first part of the word-meta string local part is used to generate a prompt input for the data item word-meta processing model. Therefore, based on the training example, the data item word-meta processing model generates an output word-meta string from the compressed representation and the prompt input. The training of the modal network and the gated cross-attention layer is then modified to increase the statistical correlation between the output word-meta string and the second part of the word-meta string of the training example.

[0034] For example, if the token string of a training example is represented as And the data items of the training items are represented as , then the modification can be such as adding (1) in is the number of characters in the word-meta string of the training example, Represents the first characters, Indicates the beginning of a word string characters, Represents the first characters before the data item, and Representing data items word processing network based on The prompt input and the modal network from the data items The compressed representation generated to generate the output token string starts with probability.

[0035] In practice, each of the training iterations is preferably performed using multiple training examples. For example, the training examples in the training database may be split into groups ("data sets") of training examples in the training database. There may be an integer Dataset. The dataset is represented as ,in . This is assigned the corresponding scalar weight , which can be the same for all batches, but can alternatively be different for different batches (e.g., lower for batches of training examples taken from sources with lower correlations between text and images). If all datasets are of the same quality, then there may be no value in splitting the training examples (e.g., can be set to 1). In each training iteration, In each of the datasets, randomly select the corresponding batch Training examples are provided, and variable parameters of a compressed representation generation system and a gated criss-cross attention layer are updated so as to reduce a loss function that indicates a failure of a data item token processing model to predict consecutive tokens among tokens in a batch of training examples.

[0036] For example, the loss function could be the expected negative log-likelihood of the second part of a word-gram string of a training example given the first part, over all possible positions that could partition the word-gram string of the training example The average of: (2) in represents the expected value. The gradient obtained from equation (2) can be is computed on the dataset in order to perform an update step on the variable parameters of both the gated criss-cross attention layer and the compressed representation generation system.

[0037] The compressed representation generation system may include one or more resampler layers, for example, arranged as a stack of resampler layers. Each resampler layer receives the output of the encoder network (e.g., after a transformation to the output of the encoder layer such as a flattening operation) or the previous layer of the stack. Each resampler layer in the resampler layer can perform a transformer operation, that is, it includes one or more transformer blocks or self-attention layers. The transformer block typically includes an attention or self-attention neural network layer, followed by a feedforward neural network. The attention or self-attention neural network layer is a neural network layer including an attention or self-attention operation (which operates on the attention layer input to generate the attention layer output). The self-attention operation can be masked so that any given position in the input sequence does not pay attention to any position after the given position in the input sequence. There are many different possible (self) attention operations. Some examples of transformer blocks including attention operations are described in Vaswani et al. "Attention is all you need", 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.

[0038] Typically, an attention operation maps a query and a set of key-value pairs to an output, where the query, key, and value are all vectors. The output is computed as a weighted sum of these values, where the weight assigned to each value is computed by a compatibility function of the query and the corresponding key, such as a dot product or a scaled dot product.

[0039] In an embodiment, the attention operation is configured to apply a self-attention operation to the attention layer input; this may be followed by one or more feed-forward neural network layers to generate an attention layer output. Typically, the attention operation determines a relationship between two sequences; the self-attention operation is configured to associate different locations in the same sequence to determine a transformed version of the sequence as an output. For example, the attention layer input may include a vector for each element of the input sequence. These vectors provide input to the self-attention operation and are used by the self-attention operation to determine a new representation of the same sequence for the attention layer output, which similarly includes a vector for each element of the input sequence. The output of the self-attention operation may be used as the attention layer output, or it may be processed by one or more feed-forward layers to provide an attention layer output.

[0040] In some embodiments, the attention operation is configured to convert, for example, The query transformation defined, for example, by the key matrix Define the key transformation and value matrix by example Each of the defined value transformations is applied to the attention layer input, which is the input data to the attention layer , to derive the corresponding query vector for determining the output sequence of interest , key vector Sum value vector . For example, the attention operation can be a dot-product attention operation applied by applying each query vector to each key vector to determine the corresponding weights for each value vector, and then combining the value vectors using the corresponding weights to determine the attention layer output for each element of the input sequence. The attention layer output can be scaled by a scaling factor, such as by the square root of the dimensions of the query and key, to achieve scaled dot-product attention. Thus, for example, the output of the attention operation can be determined as , where d is the dimension of the key (and value) vectors. In another embodiment, the attention operation may include an "additive attention" mechanism that uses a feedforward network with a hidden layer to compute the compatibility function. As mentioned above, the output of the attention operation may be further processed by one or more fully connected feedforward neural network layers.

[0041] The attention operation can implement multi-head attention, i.e. multiple different attention operations can be applied in parallel. If necessary, the outputs of these can then be combined with a learned linear transformation applied, such as concatenation, to reduce to the original dimensionality.

[0042] Optionally, the first resampler layer of the layer stack may additionally receive at least one latent vector (input latent value set). The components of the latent vector may be among the variable parameters of the compressed representation system trained in the training operation. The latent vector may have a number of components that is lower (e.g., much lower) than the number of components of the (e.g., flattened) output of the encoder network. Each resampler layer is trained to generate an output that also has a number of components that is much lower than the number of components of the output of the encoder network, for example, it may have the same number of components as the layer query vector. The other resampler layers of the stack (if any) receive the output of the previous resampler layer in the stack.

[0043] Each resampler layer can receive (i) the (e.g., flattened) output of the encoder network, and (ii) a "latent input" which is the output of the previous resampler layer in the stack, or in the case of the first resampler layer in the stack, a latent vector. The resampler layer can use the matrix , and , a key vector, a value vector and a query vector are generated from different corresponding ones (or more) of the data inputs (i) and (ii) (for each of the one or more headers) (in principle, any combination of data inputs (i) and (ii) can be used; at least one of the key vector, the value vector and the query vector is based on each of the data inputs (i) and (ii) (i.e., derived from the data including each of the data inputs (i) and (ii)), and preferably, at least one of the key vector, the value vector and the query vector is based on both of the data inputs (i) and (ii)). For example, the data input (i) (or the concatenation of the data input (i) and the data input (ii)) can be respectively connected to the matrix and The key vector and value vector are generated by multiplying them together, and the data input (ii) can be combined with the matrix Then, as explained above, the output of the resampler layer is generated based on the key vector, the value vector and the query vector, for example as .here, is the number of components in the key vector. The matrix , and The components of can be variable parameters of the compressed representation generation system trained during the training process.

[0044] The output of the stack of resampler layers may be passed to an adaptive system, such as a single or multi-layer perceptron, which generates a compressed representation as the output of the compressed representation generating system. In fact, optionally, an adaptive system, such as a single or multi-layer perceptron, may be provided within each of the resampler layers to receive the output of the corresponding attention operation of the resampler layer. In either case, the perceptron may be a substantially fully connected ("dense") network. The perceptron has variable parameters among the variable parameters of the compressed representation generating system that are trained during the training process.

[0045] Using a modal network comprising one or more resampler layers as described above to generate a compressed representation of a data item constitutes an independent aspect of the present disclosure.

[0046] Another independent aspect of the present disclosure is to use such a resampler layer within a system for generating an output term-gram string based on a query input comprising an input term-gram string and one or more data items, the input term-gram string and the output term-gram string being term-gram strings selected from a term-gram vocabulary, and the data items having modalities other than the term-grams selected from the term-gram vocabulary. The method includes generating one or more compressed representations of each data item by processing the data items using a modality network, generating a prompt input comprising the input term-gram string of the query input, and inputting the prompt input and the compressed representation of each data item into a data item term-gram processing model configured to generate the output term-gram string based on the prompt input and the compressed representation of each data item.

[0047] Turning to the gated crisscross attention layers, each gated crisscross attention layer may pass the data it receives from the previous processing layer of the data-term-token network (or, in the case where the gated crisscross attention layer is the first processing layer of the data-term-token network (although typically the first layer of the data-term-token network will be one of the term-token processing layers), the prompt input) plus a modification (interaction term) having a magnitude that positively depends on the value of a gating parameter (gate value), i.e., the magnitude of the interaction term increases as the value of the gating parameter increases. The interaction term is added component-wise to the input of the gated crisscross attention layer. For example, upon receiving the (vector) input _{\mathcal {E} from the previous processing layer, the (vector) input _{\mathcal {E}} is a vector of the interaction term. The interaction term is added component-wise to the input of the gated crisscross attention layer. For example, upon receiving the (vector) input _{\mathcal {E}} from the previous processing layer, the (vector) input _{\mathcal {E}} is a vector of the interaction term. When , the output of the gated cross attention layer can be ,in is the gating parameter (scalar), Yes Receive The (vector) output of the adaptive system (described below), and tanh is applied separately Component-wise tanh operation on each component of .

[0048] If the value of the gating parameter is low, the influence of the data item in the query input is correspondingly low. At the beginning of training, the value of the gating parameter can be selected to be low (e.g., 0.01) so that the gated cross-attention layer does not interfere too much with the operation of the trained word processing layer of the data item word processing network. During training, as the gated cross-attention layer gradually learns to cooperate with the word processing layer, the value of the gating parameter is gradually increased.

[0049] "Masking" may be applied to at least one (or all) of the gated criss-cross attention layers. This means that instead of the adaptive system of gated criss-cross attention layers generating interaction terms based on a compressed representation of all data items of the query input and data derived from all prompt inputs, interaction terms are generated based on a compressed representation of only a (true) subset of the data items in the query input and data based on only a (true) subset of the prompt inputs.

[0050] For example, assume that the query input includes (or consists of) multiple (non-overlapping) parts, each part including only one of the data items, and may also include multiple tokens of the input token string. For example, each part may be one of the input data items and the tokens of the input token string after the data item until the next data item, or until the end of the input token string in the case where the data item is a part of the last data item of the query input. For example, some of the parts may be the "task example" parts discussed previously and / or one of the parts may include the subject data item. For example, the query input may be "[image of a cat] This is a cat [Image of a dog] This". The first part of the query input is "[Image of a cat] This is a cat" (the part is the task example part; [Image of a cat] represents an image of a cat), and the second part of the query input is "[Image of a dog] This" (where [Image of a dog] is the subject data item, i.e., an image of a dog). For each part, the modality network generates at least one corresponding compressed representation of the corresponding data item, as described above. In the case of masking, one or more of the gated crisscross attention layers may be associated with each portion. For a given portion, one or more associated gated crisscross attention layers generate interaction terms based only on compressed representations of data items of the associated portion (i.e., compressed representations of other data items of the query input are not used), and data generated based on tokens of the input token string other than within the associated portion (i.e., tokens included in only the portion of the input token string within the associated portion are used).

[0051] Note that since masking can be applied only to interaction terms, it does not block information about the entire prompt input from flowing through the data item token processing model so that all token processing layers have access to this information.

[0052] Adaptive systems of gated crisscross attention layers may include crisscross transformer operations and / or processing by (single-layer or multi-layer) perceptrons. For example, the output of a crisscross transformer operation may be the input to a perceptron, the perceptron output represented above as The value of .

[0053] The crisscross transformer operation is based on a matrix for each gated crisscross attention layer (for each of the one or more heads) , and Collection. Matrix , and The components of and the variable parameters of the perceptron of the gated criss-cross attention layer (if any) are the variable parameters of the gated criss-cross attention layer trained during training (or at least among the variable parameters of the gated criss-cross attention layer trained during training).

[0054] The crisscross transformer operation can receive one or more of (i) a compressed representation of the data items in the query input, and (ii) the output of the previous processing layer, or in the case where the gated crisscross attention layer is the first processing layer of the stack, a hint input. As described above, the crisscross transformer operation can be represented using the matrix , and The key vector, value vector, and query vector are generated from different data inputs (for each of the one or more headers) in data inputs (i) and (ii). For example, the key vector, value vector, and query vector can be generated by respectively comparing data input (i) with the matrix and The key vector and value vector are generated by multiplying them together, and the data input (ii) can be combined with the matrix Then, as explained above, the output of the cross transformer operation is generated based on the key vector, the value vector, and the query vector, for example as ,in is the number of components in the key vector.

[0055] As described above, the prompt input includes an input token string. It may also include "tokens" (one or more tokens, e.g., tokens contained in the same vocabulary as the input token string) that indicate the presence of a data item within the query input and, optionally, the location of the data item within the query input. For example, if the query input is "[image of a cat]This is a cat [Image of a dog]This", where "[image of a cat]" is a data item that is an image of a cat, and "[Image of a dog]" is a data item that is an image of a dog, then the prompt input may be "[image of a cat]This is a cat [Image of a dog]This". This is a cat marker (<image>This is a cat tag <image>)", where " " (i.e., those 7 ASCII characters, where ASCII is the word-meta vocabulary of the input word-meta string) is a tag indicating a data item.

[0056] The technical effect of the present disclosure is to provide a query processing system that has been experimentally proven to provide an accurate way of processing multimodal query input (i.e., query input consisting of both word elements and data items of another different modality). The accuracy has been proven to be high even compared to known computer systems that have been designed for specific multimodal query processing tasks. It is high even for zero-trial query input, and it becomes even higher for few-trial query input.

[0057] Additionally, because embodiments of the present invention can employ existing word-gram processing layers and encoder networks, the amount of training data required to train them and the computational resources required are greatly reduced. The word-gram processing layer (which may include billions of variable parameters) can be trained based on a known text database, and the encoder network can be trained based on a known image database, and only a relatively small training database of multi-modal training examples is required to train the gated cross-attention layer and the modality network. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Examples of the present invention will now be described for illustration only with reference to the following drawings.

[0059] Figure 1 (a) shows the encoder network.

[0060] Figure 1 (b) shows the word-gram processing model.

[0061] Figure 2 A query processing system comprising a modality network is shown, the modality network comprising Figure 1 (a) shows the encoder network, and includes Figure 1 (b) Data item lemma processing model of the lemma processing model shown.

[0062] Figure 3 A first modal network is shown.

[0063] Figure 4 A second modality network is shown.

[0064] Figure 5 Shows Figure 2 A pair of layers in the data item token processing model of the query processing system.

[0065] Figure 6 A query input to a query processing system is shown, as well as an output word-gram string generated from the query input by the query processing system.

[0066] Figure 7 A training example for training a query processing system is shown.

[0067] Figure 8A training system for training a query processing system based on training examples is shown.

[0068] Fig. 9 A method for generating an output word-gram string based on a query input is shown.

[0069] Fig.10 A method of training a query processing system is shown.

[0070] Fig.11 The process of interacting with the query processing system is shown.

[0071] The same reference numerals in different drawings denote the same items. DETAILED DESCRIPTION

[0072] This specification describes a query processing system operable to receive a query input comprising at least an input word-gram string (a series of word-grams selected from a vocabulary) and at least one data item having a different modality than the input word-gram string. The data item typically does not include the word-grams selected from the vocabulary. The query processing system is an example of a neural network system that can be implemented by one or more computer systems in one or more locations.

[0073] The data items may include one or more images (e.g., still or video images captured by a camera) and / or audio data representing the value of an audio waveform at each of a plurality of times, e.g., sound captured by a microphone over a period of time. The query processing system includes a portion (a "modality network") that includes an encoder network for encoding the data items (e.g., as a set of features) and a compressed representation generation system for generating a compressed representation of the data items from the encoded data items. For example, where the data items include images, the encoder network may be configured to apply at least one two-dimensional convolutional layer to each image. Where the data items include audio data, the encoder network may be configured to apply a one-dimensional convolution to the audio data.

[0074] The encoder network can be an encoder network that has been pre-trained in a different neural network system. For example, Figure 1(a) shows a neural network system including an encoder network 11 and an optional output layer 12. The encoder network 11 receives a data item and generates an output therefrom. The optional output layer 12 (if present) receives the output of the encoder network 11 and generates the output of the neural network system. The encoder network 11 can be trained using a database of data items and corresponding expected outputs, so that when a data item is input to the encoder network 11, the encoder network 11 and (if present) the output layer 12 generate the corresponding expected output. During the training process, the output layer 12 (if present) can also be optionally trained. In the case where the output layer 12 is present, the encoder network can be trained to extract features of the data item it receives, and the output layer 12 processes those features to generate the expected output (the encoded data item). After training, the output layer 12 (if present) can be discarded. The encoded data item can have a smaller number of components than the data item from which the encoded data item is generated. The encoder network 11 can include one or more stacked convolutional layers, and the encoded data item can be a (e.g., two-dimensional) feature map.

[0075] In the experiments reported below, the encoder network 11 employed is the “f6” model taken from A. Brock et al., “High-Performance Large-Scale Image Recognition Without Normalization”, 2021. The encoder network 11 is pre-trained using a contrastive objective on a dataset of image and text pairs, using two contrastive losses from A. Radford et al., “Learning Transferable Visual Models From Natural Language Supervision”, 2021. The output of the final stage, a 2D spatial grid of features, is “flattened” to form a 1D sequence.

[0076] The query processing system further includes a data item word unit processing model, which includes a word unit processing layer taken from the word unit processing model. The word unit processing model can be a model that has been pre-trained. For example, Figure 1 (b) shows a suitable word-gram processing model, which includes an integer number n of word-gram processing layers arranged in a stack (sequence) . The first word-gram processing layer receives an input word-gram string. Except for the first word-gram processing layer, each word-gram processing layer receives the output of the previous word-gram processing layer. The output of the nth word-gram processing layer is the output word-gram string of the word-gram processing model. The word-gram processing layers can constitute a "language model" (e.g., a "large language model"), trained on a large database of data such as natural language data, so that when an input word-gram string (a sequence of words from a vocabulary) is input to the first word-gram processing layer, the output word-gram string is an appropriate response. For example, if the input word-gram string is a question, the output word-gram string can be an appropriate answer. In the experiments reported below, the word-gram processing model is the Chinchilla model from "Training compute-optimallarge language models" by J. Hoffmann et al., 2022. Alternatively, the token processing layer can be a layer generated by another publicly known language model training method (e.g., Brown et al, “Languagemodels are few-shot learners”, in Conference on Neural Information Processing Systems, 2020).

[0077] Go to Figure 2 , shows the structure of the query processing system 200. The query processing system 200 is configured to receive a query input 201 including one or more data items and an input word character string as input. In one case, Figure 2 As shown, the query input 201 may include two data items 203, 205. However, it should be understood that any number of data items may be present.

[0078] The query processing system 200 extracts two data items 203, 205 from the query input 201 and processes each data item using a modal network. Figure 2 As shown, the query processing system includes two modality networks 207, 209, but alternatively, where all data items have the same modality, there may be only a single modality network which in turn receives the data items 203, 205 and generates a corresponding output from each data item.

[0079] Note that if there are multiple data items in the query input 201, and if they have different corresponding modalities, a modality network may be provided for each corresponding modality, and each modality network is configured to receive the data items of the corresponding modality (e.g., sequentially if there are multiple data items of the corresponding modality in the query input). For example, the modality network 207 may be configured to receive a data item that is a query input of a single image (e.g., a still image), and the modality network 209 may be configured to receive a data item that is a query input of audio data. In another example, the modality network 207 may be configured to receive a data item that is a query input of a single image (e.g., a still image), and the modality network 209 may be configured to receive a data item that is a video (i.e., a sequence of images) optionally with a soundtrack of audio data. There may be any desired number of modality networks for each corresponding data item modality that the query processing system 200 is configured to process.

[0080] Each modality network 207, 209 is configured to generate one or more corresponding compressed representations of the data item from each data item it receives. Each modality network may include (e.g., Figure 1 (a) a pre-trained encoder network 11 and a compressed representation generation system in the form of corresponding resamplers 210, 211. For example, each resampler 210, 211 can be a "perceptor" as described in A. Jaegle et al., "Perceiver: General Perception with Iterative Attention", 2021. Alternatively, the resamplers 210, 211 can be as described below with reference to Figure 3 and Figure 4 described.

[0081] The resamplers 210 , 211 take as input a variable number of image, video, or audio features produced from a data item by the corresponding encoder network 11 and produce a compressed representation of the data item with a fixed number of outputs (eg, visual outputs).

[0082] Figure 3 The structure of a possible modality network 207 is shown for the case where data item 203 is an image, including a corresponding "perceiver" resampler 210. Note that if data item 205 is an image, then modality network 209 has the same structure (indeed, as mentioned above, a single network can play the role of both modality networks 207, 209 when data items 203, 205 have the same modality).

[0083] Resampler 210 receives the output of encoder 11 (an encoded version of data item 203) and generates therefrom a data item represented as The perceptron resampler also receives a predetermined number of latent vectors (i.e., sets of latent values), each of which is represented as , which is learned (trained) during the training of the modality network 207 described below.

[0084] The resampler 210 also includes one or more "resampler layers" 300. For simplicity, the resampler 210 is Figure 3 300. The compressed representation generation system may alternatively include multiple resampler layers, each having a Figure 3 The structure shown in 300 in FIG. The resampler layer 300 may be arranged as a stack of resampler layers, each layer having Figure 3 Each resampler layer 300 (except the first resampler layer) receives the output of the previous resampler layer in the stack.

[0085] The resampler layer 300 (or, if there are multiple resampler layers 300, each resampler layer of the stack) receives the output of the corresponding encoder network 11, for example after transformation to the output of the encoder network 11, such as a flattening operation (i.e., converting, for example, a 2-dimensional array into a 1-D sequence).

[0086] Each resampler layer in the resampler layer 300 may perform a transformer operation as defined above; i.e., it comprises one or more transformer blocks or self-attention layers. To this end, it may comprise a unit 301 which uses the flattened data , and use the latent vector The key vector is generated by different latent vectors in Sum value vector This process uses the key matrix and value matrix as described above. Unit 302 uses the latent vector X and the value matrix to generate the query vector The attention unit 303 combines the , and The result is added to the latent vector by adding unit 305. The result is passed to an adaptive unit, such as a multi-layer feed-forward (FFW) perceptron 307. The output of the FFW perceptron 307 is added to the output of the addition unit 305 by a further addition unit 311. The result is an output vector for each latent vector X, providing a different corresponding compressed representation of the data item. The output of each resampler layer 300 may have the same dimensionality as the dimensionality of the latent variable X. This dimensionality may be much lower than the dimensionality of the encoder network 11.

[0087] Figure 4 The example in which the input data item 203 is a plurality of video frames is shown. Figure 3 The data item 203 is split into individual frames 41, 42, 43. Each of these frames is sequentially encoded by the encoder network 11 to generate a corresponding encoded frame. Corresponding data 44, 45, 46 are added to each of the encoded frames, thereby specifying the position of the frame in the data item 203. The resulting data is input to a resampler 210, which may have the same structure as described above with reference to FIG. Figure 3 Note that although the input to the resampler 210 is different from Figure 3 , but for a given latent vector , the output is the same number of output tokens.

[0088] The operation of the perceptron resampler can be represented by the following pseudocode: Let x_f denote the output of the corresponding encoder network 11. This is an array with dimension [T, S, d], where T is an integer variable representing the degree (i.e., the number of encoder networks 11 or the number of times a single encoder network is used), S is an integer representing the number of spatial localizations, and d is another integer indicating the number of feature values ​​for each localization. Let x denote the R learned latent variables, so this is an array with dimension [R, d]. The number of layers of the resampler is denoted as num_layers. First, the time embedding is added and the result is flattened: x_f=x_f + time_embeddings x_f = flatten(x_f) This produces an array of dimension [T*S, d] Then, for each integer value in the range 1 to num_layers : x=x+attention_i(q=x, kv=concatenation([x_f, x])) x=x+ffw_i(x). Here, attention_i represents the attention unit 303 performed by the i-th resampler layer 300 based on the corresponding key, value and query matrices, and ffw_i represents the operation performed by the corresponding perceptron 307 of the i-th resampler layer 300.

[0089] Therefore, the resampler 210 converts the spatial visual features (in Figure 4The resampler 210 maps a variable size grid of visual features (spatial-temporal features in the case of ) to a fixed number of output tokens. Each resampler 210 has a set of learned latent vectors as queries, and the keys and values ​​are the concatenations of the visual features with the learned latent vectors.

[0090] Return to Figure 2 The query processing system 200 further includes a data item word processing model 220. This includes stacked layers, the stacked layers are Figure 1 (b) The word processing model of the word processing layer 131, 132, …, 13n is composed of the number The number of gated crisscross attention layers 231, 232, …, 23j is interlaced. Can be equal to the number of word processing layers , but in the variant, and can be different from each other, for example so that there can be any number of token processing layers between any given pair of gated cross-attention layers.

[0091] The outputs from the resamplers 210, 211 are used as control inputs for the gated cross-attention layers. Optionally, the same outputs from the resamplers 210, 211 can be used as control inputs for all gated layers. This was achieved in some successful experiments where 64 latent vectors were used and the 6 resampler layers and the corresponding 64 vectors output by the perceptron resamplers 210, 211 were used together as control inputs for all gated crisscross attention layers. However, this is not the only possibility. For example, there may be corresponding different (learned) latent vectors to generate corresponding control inputs for corresponding ones of the gated crisscross attention layers.

[0092] The input word-gram string is used as a prompt input to the data item word-gram processing model 220 and is fed to the first processing layer of the stack, which may be the first gated crisscross attention layer 231. Data Passage ( Figure 2 (from the middle to the top) passes through the data item word-gram processing model 220 to generate an output word-gram string, which is the output of the data item word-gram processing model 220.

[0093] Gated Cross-Attention Layer Each receives at least one of the compressed representations from the modality networks 207 , 209 and performs a gating operation based on the received compressed representation.

[0094] Figure 6 Shows Figure 2 The structure of two consecutive layers 523, 513 of the data item processing model 220. Layer 523 is a gated cross attention layer, which is a layer Layer 513 is one of the word unit processing layers 131, 132, …, one of the.

[0095] The gated crisscross attention layer 523 receives visual input 51, 52 (compressed representation) from the corresponding modality network in the modality network 207, 209. It further receives a vector, which is the language input 53 and can be represented as y. If the gated crisscross attention layer 523 is the first layer of the stack, the language input 53 is typically the input word-unit string (prompt input) or a part thereof. Otherwise, the language input 53 to the gated crisscross attention layer 523 is the output of the previous layer from the data item word-unit processing model 220.

[0096] The gated cross-attention layer 523 applies the attention function 501 as described above, where the visual inputs 51, 52 may define key and value vectors, and a query vector is defined based on the language input 53. The output of the attention function 501 is added to the language input 53 by the addition unit 502. The result is then input to the FFW network 503, such as a multi-layer perceptron. The output of the FFW network 503 is represented as This is input to the tanh gating unit 504, which applies a component-wise operation where each component is multiplied by the gating parameter , and the tanh function is applied to the result. The output of the FFW network 504 is added to its input by the addition unit 505. This produces the output of the gated crisscross attention layer 523, which can be written as .

[0097] As in some known systems, the word-gram processing layer 513 includes a self-attention layer 506. The attention layer operates on the key vector, value vector, and query vector generated from the input to the word-gram processing layer 513. The output of the self-attention layer 506 is added to its input by an addition unit 507, and the result is fed to a feed-forward (e.g., multi-layer perceptron) layer 508. The output of the FFW network 508 is added to the input of the FFW network 508 by an addition unit 509. This produces the output of the word-gram processing layer 513.

[0098] Note that if the gating parameter is close to zero, then the output of the gated cross attention layer 523 is very close to the language input 53, so the query processing system 200 performs the same Figure 1 (b) shows an almost identical function to the trained word-gram processing model shown in FIG. However, in use (i.e., after the query processing system 200 has been trained), the value of the gating parameter is significantly higher than zero, so that the compressed representations (e.g., visual input) 51 and 52 affect the output of the word-gram processing layer 513.

[0099] Figure 6 200. The query input 61 includes two data items 203, 205 in the form of still images: still images of a cat and a dog, respectively. As explained above, the data items 203, 205 are input to corresponding modality networks 207, 209, or are continuously input to the same modality network to produce corresponding compressed representations of the data items 203, 205. Since the data items 203, 205 are in the form of still images, the query processing system 200 uses Figure 3 If the data item is a video, the modality network can be Figure 4 The form shown.

[0100] The words of the query input 61 are used to generate an input word-meta string for the query input of the query processing system 200. Optionally, the query input may include one or more "tags", such as for each data item 203, 205 of the query input 61, to indicate the presence of the associated data item. For example, the input word-meta string may be the following ASCII characters: " This is a very cute dog. Optionally, additional tags may be provided. For example, tags may be added before any images and at the end of the query input (e.g., <eoc>, meaning "end of chunk"). Tokens can be formed from token strings from the vocabulary or from outside the vocabulary.

[0101] The query processing system 200 processes the input word-gram string by passing the input word-gram string through layers 231, 131, 232, 132, ..., 23j, 13n in sequence, wherein the gated layers 231, 232, ..., 23j are controlled based on the visual input generated by the modality networks 207, 209 based on the corresponding data items 203, 205, i.e., the two still images of the query input 61. In this way, the query processing system 200 generates an output word-gram string such as Figure 6 The output word-meta string 62 is an ASCII string that reads "a very serious cat". This is a logical grammatical continuation of the query input 61. The first image 203 and the associated text "This is a very cute dog" constitute the "task example". The output word-meta string is the result of performing the task illustrated by the task example on the second data item 205 ("subject data item").

[0102] We will now consider training a query processing system such as Figure 2 The method can be implemented by one or more computer systems in one or more locations. As explained above, the encoder network 11 and the word unit processing layer 131, 132, ..., 13n can be pre-trained as Figure 1 The encoder network 11 and the word-gram processing layers 131, 132, ..., 13n are not additionally trained during the training of the other layers of the query processing system 200. This means that the amount of data required in the training database is small, for example much smaller than the training database used earlier for training the word-gram processing layer and / or the encoder network.

[0103] Training is based on a training database of "multimodal" training examples. Figure 7 A training example 71 is shown. This optionally contains one or more task examples. A single task example 701 is shown, including a data item (a photo of a dog) and text associated with the data item ("This is a picture of my dog"). Different training examples in the training examples may include different numbers of task examples, or no task examples. The training example 71 further includes a portion 702 which is a data item (a photo of a cat) and text associated with the data item ("This is a picture of my cat"). If the training example contains a task example, the text of portion 702 has the same semantic relationship as the data item of portion 702, as the text of (each of) the training examples has the same semantic relationship as the data item of that training example.

[0104] Can be Figure 8 In the system shown, the modality networks 207, 209 and gated cross attention layers 231, 232, ..., 23j are jointly trained. The system uses a Figure 7 The training database 801 shown is composed of training examples, each training example including at least one data item and a word-gram string ("multi-modal" training examples).

[0105] During training, a series of training iterations are performed. In each iteration, one or more training examples in the training examples can be selected. In a typical implementation, multiple training examples are selected and regarded as a batch, but for simplicity, the case of selecting a single training example is discussed here. The splitter unit 802 receives the selected training example and divides the word-gram string of the selected training example into a first part as an earlier local part of the word-gram string and a second part as a later local part of the word-gram string. The data item of the selected training example and the first part of the word-gram string of the selected training example together constitute the query input of the query processing system 200. The data item is input to the modal network 207, 209 of the query processing system 200, and the first part of the word-gram string is used to generate a prompt input for the data item word-gram processing model 220 of the query processing system 200. Therefore, based on the selected training example, the data item word-gram processing model 220 generates an output word-gram string from the compressed representation generated by the modal network and from the prompt input.

[0106] The difference unit 803 receives the second part of the word-gram string of the selected training example and the output word-gram string from the splitter unit, and calculates a measure of difference (the difference between them).

[0107] The update unit 804 then generates an update to the query processing system 200 - namely, an update to the modal network and the gated cross-attention layer, which increases the statistical correlation between the output word-gram string and the second part of the word-gram string of the training example. The process can be repeated for different selections of locations in the word-gram string of the selected training example at which the splitter unit 802 split the word-gram string into the first part and the second part, and the update generated by the update unit 804 can be, for example, to reduce the difference of multiple selections of these selections, such as the average of the selections.

[0108] For example, if the token string of a training example is represented as And the data items of the training items are represented as , then the modification can be such as adding , in is the number of characters in the word-meta string of the training example, Represents the first characters, Indicates the beginning of a word string characters, Represents the first characters before the data item, and Representing data items word processing network based on The prompt input and the modal network from the data items The compressed representation generated to generate the output token string starts with probability.

[0109] In practice, as described above, each of the training iterations is preferably performed using multiple training examples. For example, the training examples in the training database may be split into groups ("data sets") of training examples in the training database. There may be an integer Dataset. The dataset is represented as ,in . This is assigned the corresponding scalar weight , which can be the same for all batches, but can alternatively be different for different batches (e.g., lower for batches of training examples taken from sources with lower correlations between text and images). If all datasets are of the same quality, then there may be no value in splitting the training examples (e.g., can be set to 1). In each training iteration, In each of the datasets, randomly select the corresponding batch Training examples are provided, and variable parameters of a compressed representation generation system and a gated criss-cross attention layer are updated so as to reduce a loss function that indicates a failure of a data item token processing model to predict consecutive tokens among tokens in a batch of training examples.

[0110] For example, the loss function could be the expected negative log-likelihood of the second part of a word-gram string of a training example given the first part, over all possible positions that could partition the word-gram string of the training example The average of: in represents the expected value. The gradient of the loss function can be calculated over all is computed on the dataset in order to perform an update step on the variable parameters of both the gated criss-cross attention layer and the modality network.

[0111] Initially, during the training process, the gating parameters The value of can be chosen to be low (e.g., 0.01) so that the influence of the data item on the output word-unit string is correspondingly low. Thus, at the beginning of training, the gated cross-attention layer does not interfere too much with the operation of the trained word-unit processing layers 231, 232, ... 23j of the data item word-unit processing network 220. During training, as the gated cross-attention layer gradually learns to cooperate with the word-unit processing layer, the value of the gating parameter is gradually increased, for example, to =1.

[0112] Preferably, in both training and operation, "masking" is applied to at least one (or all) of the gated cross-attention layers 231, 232, ...23j. This means that instead of the gated cross-attention layers generating interaction terms based on compressed representations of all data items of the query input and data derived from all prompt inputs, the interaction terms are generated based on compressed representations of only a (true) subset of the data items in the query input and only a (true) subset of the prompt inputs. For example, the query input may include (or consist of) multiple (non-overlapping) parts, each part including only one of the data items, and may also include multiple word-lemmas of the input word-lemma string. For example, some of the parts may be the "task example" parts discussed previously and / or one of the parts may include a subject data item. For each part, the modal network generates at least one corresponding compressed representation of the corresponding data item, as described above. In the case of masking, one or more of the gated cross-attention layers may be associated with each of the parts. For a given portion, one or more associated gated crisscross attention layers generate interaction terms based only on compressed representations of data items of the associated portion (i.e., compressed representations of other data items of the query input are not used), without employing data generated based on tokens of the input token string other than within the associated portion (i.e., only tokens included in the portion of the input token string within the associated portion are used). Note that since masking is preferably applied only to interaction terms, it does not block information about the entire prompt input from flowing through the data item token processing model so that all token processing layers have access to that information.

[0113] Go to Fig. 9 , a method 900 is shown as an example of the present disclosure. The method can be performed by one or more computers in one or more locations. The method 900 is performed by, for example, Figure 2 The method is performed by the query processing system 200 of the query processing system 200. The method is performed on a query input having one or more data items and an input word character string.

[0114] In step 901, a query input (such as Figure 6 Each data item of the query input 61) is input into a corresponding modal network to generate one or more compressed representations of each data item. If the data items have different modalities, there is typically at least one modal network for data items of each modality. For a given data item modality, there may be one or more modal networks for data items of that modality.

[0115] In step 902, an input word-gram string is used to generate a prompt input. This may include inserting at least one token of at least one data item of the query input into the prompt input.

[0116] In step 903, using Figure 2 The data item word unit processing model 220 of the data item word unit processing model processes the prompt input, and the data item word unit processing model also receives the compressed representation generated in step 901 to control the gating layer of the data item processing model. The output of the data item processing model is an output word unit string, such as Figure 6 The output word string 62 is obtained.

[0117] Go to Fig.10 , a method 1000 is shown as an example of the present disclosure. The method can be performed by one or more computers in one or more locations. The method 1000 generates a query processing system, such as Figure 2 Query processing system 200.

[0118] In step 1001, by passing the word processing layer (In such Figure 1 (b) Trained and untrained gated cross-attention layers in the system Interleave, using e.g. Figure 1 The trained language model of the model shown in (b) is used to form a data item word processing model, such as Figure 2 The data item word-unit processing model 220 is used.

[0119] In step 1002, a query processing system such as query processing system 200 is formed using the data item token processing model formed in step 1001 and a modality network such as modality networks 207, 209. The modality network employs the trained encoder network 11 and (e.g., untrained) resamplers 210, 211.

[0120] In step 1003, such as by Figure 8 The system shown jointly trains the resamplers 210, 211 and the gating layer . Train resamplers 210, 211 and gating layers The training of the corresponding key, value and query matrices includes the training of the resampler 210 further including the training of the latent vector.

[0121] use Figure 2 We experimented with the query processing system 200 and demonstrated its ability to perform different and challenging tasks. For example, Fig.11 shows a dialog between a human user and an embodiment of the system. The query processing system is based on a large language model, i.e., Figure 1 The word-gram processing system shown in (b) is the Chinchilla model of J. Hoffmann et al., implemented with 80 billion variable parameters.

[0122] Initially, a user generates a query input comprising a data item 111 (a still image) and an input word-meta string 112 (the word "Which city is this?").

[0123] The query processing system processes the query input to generate an output word-meta string 113 (the word "This is a streetsign in New York").

[0124] The user then generates a new query input comprising data item 111, input word-meta string 112, output word-meta string 113 and new input word-meta string 114. The new input word-meta string consists of the words "Which street corner it is?".

[0125] The query processing system processes the new query input to generate a new output word element string 115 (the word "It is the intersection of Madison Avenue and East 42 nd Street (which is the intersection of Madison Avenue and East 42nd Street)").

[0126] Thus, a dialog is performed between the human user and the query processing system, wherein the human user obtains successive information based on the data item of the initial query input. The dialog may continue in a series of additional steps, wherein the user concatenates the data item 111, the input word-element string and the corresponding generated output string of any previous step, and the new word-element input string at each step to generate a new query input, which is input to the query processing system.

[0127] For a system of one or more computers to be configured to perform a specific operation or action, it is meant that the system has installed thereon software, firmware, hardware, or a combination thereof, which, in operation, causes the system to perform the operation or action. For one or more computer programs to be configured to perform a specific operation or action, it is meant that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0128] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier, for execution by a data processing device or for controlling the operation of the data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device. A computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. However, a computer storage medium is not a propagation signal.

[0129] The term "stack" of layers refers to a sequence of layers, each layer receiving data input and producing data output. Each layer except the first layer receives at least a portion of the output of the previous layer in the sequence as at least a portion of its input. Thus, data flows through the stack from the first layer to the last layer in the sequence, and the output of the stack of layers includes the output of the last layer in the sequence.

[0130] The term "data processing apparatus" covers all kinds of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The apparatus may include special-purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0131] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script, or code) may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, such as files storing one or more modules, subroutines, or portions of code. A computer program may be deployed to execute on one computer located at one site or on multiple computers distributed across multiple sites and interconnected by a communications network.

[0132] As used in this specification, an "engine" or "software engine" refers to a software-implemented input / output system that provides outputs distinct from inputs. An engine may be a coded block of functionality, such as a library, platform, software development kit ("SDK"), or object. Each engine may be implemented on any suitable type of computing device, such as a server, mobile phone, tablet computer, notebook computer, music player, e-book reader, laptop or desktop computer, PDA, smart phone, or other fixed or portable device including one or more processors and a computer-readable medium. Additionally, two or more of the engines may be implemented on the same computing device, or on different computing devices.

[0133] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special-purpose logic circuit systems, and the device can also be implemented as a special-purpose logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). For example, the processes and logic flows can be performed by the device and can also be implemented as a graphics processing unit (GPU).

[0134] For example, a computer suitable for executing a computer program can be based on a general or special microprocessor or both, or any other kind of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for executing or implementing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include or be operably coupled to receive data from or transmit data to or both of one or more large-capacity storage devices for storing data, such as disks, magneto-optical disks, or optical disks. However, a computer does not need to have such a device. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0135] Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0136] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0137] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component, such as a data server, or includes a middleware component, such as an application server, or includes a front-end component, such as a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by digital data communication of any form or medium, such as a communication network. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), such as the Internet.

[0138] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0139] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or content that can be claimed, but as descriptions of features that can be specific to a particular embodiment of a particular invention. Certain features described in the context of separate embodiments in this specification may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable sub-combination. In addition, although features may be described above as working in certain combinations and even initially claimed as such, in some cases, one or more features from the claimed combination may be removed from the combination, and the claimed combination may be directed to a sub-combination or a variation of the sub-combination.

[0140] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all of the operations shown be performed, to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0141] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing may be advantageous.< / eoc>

Claims

1. A computer-implemented method of generating an output word-gram string based on a query input comprising an input word-gram string and one or more data items, the input word-gram string and the output word-gram string being strings of word-grams selected from a word-gram vocabulary, and the data items having modalities other than the word-grams selected from the word-gram vocabulary, the method comprising: Upon receiving data items of the modality, inputting each data item of the query input into a trained modality network to generate one or more compressed representations of each data item; generating a prompt input including the input word-meta string of the query input; as well as The prompt input is input into a data item word-gram processing model, the data item word-gram processing model has a plurality of processing layers arranged as a stack, the output word-gram string is the output of the data item word-gram processing model, the processing layers include a plurality of word-gram processing layers and a plurality of gated cross-attention layers, each gated cross-attention layer is arranged to receive at least one of the compressed representations, the word-gram processing layers are interleaved with the gated cross-attention layers.

2. The computer-implemented method of claim 1 , wherein: The word-gram processing layer is operable to provide a word-gram string processing model together with the gated cross-attention layer to receive an input word-gram string and generate a corresponding output word-gram string.

3. A computer-implemented method according to claim 1 or claim 2, comprising: Generate an output token string based on the query input; as well as Perform the following set of steps at least once: Based on the query input and the output word-unit string, forming a new query input; and A new output word-gram string is generated based on the new query input.

4. A computer implemented method of training a query processing system for generating an output word-gram string based on a query input comprising an input word-gram string and one or more data items, the input word-gram string and the output word-gram string being strings of word-grams selected from a word-gram vocabulary, and the data items having modalities other than the word-grams selected from the word-gram vocabulary, The method employs a word-gram processing model and a database of training examples, the word-gram processing model comprising a stack of word-gram processing layers, the stack of word-gram processing layers being configured to receive an input word-gram string and generate a corresponding output word-gram string, each training example comprising at least one data item and at least one word-gram string; The method comprises: forming a data item word-gram processing model by interleaving the word-gram processing layer with a gated cross-attention layer, the data item word-gram processing model being configured to generate an output word-gram string upon receiving a prompt input as a word-gram string; The query processing system is formed, including: (a) a modality network configured to receive the data items of the query input to generate one or more compressed representations of each data item; and (b) the data item word-gram processing model, the data item word-gram processing model being configured to receive a prompt input of the input word-gram string comprising the query input, and each gated criss-cross attention layer being arranged to receive at least one of the compressed representations; and Using the training database, train: the modal network, and The multiple gated cross-attention layers.

5. The computer-implemented method of claim 4, wherein: The training trains the query processing system when the encoder of the modal network receives the at least one data item of any training example among the training examples and the data item word-gram processing model receives a prompt input including a first part of the word-gram string of the training example to generate an output of the query processing system, which is positively statistically correlated with a subsequent part of the word-gram string of the training example.

6. A computer-implemented method according to any preceding claim, wherein: The modal network includes: an encoder configured to encode a data item received by the encoder to generate an encoded data item, and A compressed representation generating system is arranged to receive the encoded data item and generate an output, the output of the modal network being based on the output of the compressed representation generating system.

7. A computer-implemented method according to claim 6, when dependent on claim 4, wherein: The encoder has been trained to encode data items received by the encoder to generate encoded data items, and training the modal network and the plurality of gated cross-attention layers includes training the compressed representation generation system without further training the encoder.

8. A computer-implemented method according to claim 6 or claim 7, wherein: The compressed representation generation system comprises a stack of one or more resampler layers, each resampler layer is adapted to perform an attention operation taking a key vector, a value vector, and a query vector, a subset of which are based on the encoded data item and the remainder of which are based on the output of a previous resampler layer in the resampler layer or, in the case of the first resampler layer of the stack, on the set of input potential values, The output of the modality network is based on the output of the last resampler layer of the stack of resampler layers.

9. The computer-implemented method of claim 8, wherein: The key vector and value vector of each resampler layer are based on the encoded data item and a potential input, the potential input being the output of a previous resampler layer in the resampler layer, or in the case of the first resampler layer of the stack, the set of input potential values, and the query vector is based on the potential input.

10. The computer-implemented method of claim 8 or 9, wherein: Each resampler layer further comprises a perceptron arranged to receive the output of the attention operation and generate an output, the output of the modality network being based on the output of the perceptron of the last resampler layer of the stack.

11. A computer-implemented method according to any preceding claim, wherein: The prompt input further includes one or more corresponding marking items for each data item in the query input, the one or more marking items indicating the presence of the data item in the query input.

12. The computer-implemented method of claim 11, wherein: The location of each marked item in the prompt input indicates the location of the corresponding data item in the query input.

13. A computer-implemented method according to any preceding claim, wherein: Each gated criss-cross attention layer generates its output as the component-wise sum of: a first input, the first input being the output of a previous processing layer in the stack of processing layers, or the hint input in the case where the gated crisscross attention layer is the first processing layer in the stack of processing layers, and An interaction term based on the output of the compressed representation generation system received by the gated criss-cross attention layer and at least a portion of the first input to the gated criss-cross attention layer.

14. A computer-implemented method according to claim 13, when dependent on claim 4, wherein: The interaction term has a magnitude that is positively dependent on a value of a gating parameter, and the training comprises incrementally increasing the learning parameter.

15. A computer-implemented method according to claim 13 or claim 14, comprising: In the case where the query input includes multiple parts, each part includes one of the data items, and for each part: The modality network generates at least one corresponding compressed representation of the corresponding data item, and At least one of the gated criss-cross attention layers generates the interaction term based solely on the compressed representation of the corresponding data item, without employing data generated based on word-lemmas of the input word-lemma string other than within the portion.

16. A computer-implemented method according to any preceding claim, wherein: Each gated crisscross attention layer comprises a crisscross attention layer that employs a key vector, a value vector, and a query vector, a subset of which are based on the at least one compressed representation received by the gated crisscross attention layer, and the remainder of the key vector, value vector, and query vector are based on an output of a previous processing layer in the stack of processing layers, or, if the gated crisscross attention layer is the first processing layer in the stack of processing layers, based on the prompt input.

17. The computer-implemented method of claim 16, wherein: The key vector and value vector of each gated criss-cross attention layer are obtained based on the at least one compressed representation received by the gated criss-cross attention layer, and the query vector of each gated criss-cross attention layer is based on the output of the previous processing layer in the stack of processing layers, or based on the hint input when the gated criss-cross attention layer is the first processing layer in the stack of processing layers.

18. The computer-implemented method of claim 16 or 17, wherein: The gated crisscross attention layer further includes a perceptron that receives the output of the crisscross attention layer, and the output of the gated crisscross attention layer is based on the output of the perceptron.

19. A computer-implemented method for generating a compressed representation of a data item having a modality, the method comprising processing the data item using a modality network, the modality network comprising: an encoder configured to encode the data item to generate an encoded data item, and a compressed representation generating system arranged to receive said encoded data item and to generate an output, wherein the compressed representation generation system comprises a stack of one or more resampler layers, each resampler layer being configured to perform an attention operation that employs a key vector, a value vector, and a query vector, each of which is based on at least one of the encoded data item and a potential input, the potential input being an output of a previous resampler layer in the resampler layer or, in the case of the first resampler layer in the stack, a set of input potential values, the at least one of which is based on both the encoded data item and the potential input, The output of the modality network is based on the output of the last resampler layer of the stack of resampler layers.

20. The computer-implemented method of claim 19, wherein: The key vector and value vector of each resampler layer are based on the encoded data item and the potential input, and the query vector of each resampler layer is based on the potential input.

21. A computer-implemented method according to claim 19 or claim 20, wherein: Each resampler layer is configured to add the potential input to the output of the attention operation.

22. A computer-implemented method according to any one of claims 19 to 21, wherein: Each resampler layer further comprises a perceptron arranged to receive the output of the attention operation and to generate an output, the output of the modality network being based on the output of the perceptron of the last resampler layer of the stack of resampler layers.

23. A computer-implemented method of generating an output word-gram string based on a query input comprising an input word-gram string and one or more data items, the input word-gram string and the output word-gram string being strings of word-grams selected from a word-gram vocabulary, and the data items having modalities other than word-grams selected from the word-gram vocabulary, the method comprising: (a) generating one or more compressed representations of each data item by processing the data item using a modality network, the modality network comprising: an encoder configured to encode the data item to generate an encoded data item, and a compressed representation generating system arranged to receive said encoded data item and to generate an output, wherein the compressed representation generation system comprises a stack of one or more resampler layers, each resampler layer being configured to perform an attention operation, the attention operation taking a key vector, a value vector, and a query vector, the key vector, the value vector, and the query vector each being based on at least one of the encoded data item and a potential input, the potential input being the output of a previous resampler layer in the resampler layer or, in the case of the first resampler layer of the stack, a set of input potential values, the at least one of the key vector, the value vector, and the query vector being based on both the encoded data item and the potential input, the output of the modal network being based on the output of a last resampler layer of the stack of resampler layers; (b) generating a prompt input including the input word-meta character string of the query input; and (c) inputting the prompt input and the compressed representation of each data item into a data item word-gram processing model configured to generate the output word-gram string based on the prompt input and the compressed representation of each data item.

24. A computer-implemented method according to any one of claims 1 to 18 or 23, wherein: The output word-gram string is a response to a query about the content of a subject data item that is one of the data items in the query input, the query being defined based on the input word-gram string.

25. The computer-implemented method of claim 24, wherein: The query input includes one or more task example parts in addition to the subject data item, each task example part including a corresponding data item and a corresponding segment of the input word string, and for each task example part, the corresponding part of the input word string is the response to the query when the query is about the content of the corresponding data item.

26. A computer-implemented method according to claim 24 or 25, wherein: The query input is a question, and the response to the query is an answer to the question.

27. A computer-implemented method according to any one of claims 1 to 18 or 23 to 26, wherein: The word-gram vocabulary includes symbols of a natural language writing system.

28. A computer-implemented method according to any preceding claim, wherein: The modality includes at least one of: (i) a still image or a video image, or (ii) a sound signal.

29. A system comprising one or more computers and one or more storage devices storing instructions which, when executed by the one or more computers, cause the one or more computers to perform the method of any preceding claim.

30. One or more computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to implement the method of any one of claims 1 to 28.