System and method for generating multi-modal data using single tower architecture with data generation subsystem

Abstract

A computer-implemented method of generating multi-modal data. The method comprises the following steps of: generating an output sequence of multi-modal lexical elements from regression by using a lexical element generation neural network; and in response to the next multi-modal lexical element being an image start lexical element, generating an image using the image generation subsystem conditioned by features representing the current sequence of multi-modal lexical elements obtained from the lexical element generation neural network. The method further comprises: processing the image to convert pixels of the image into a sequence of image lexical elements, each image lexical element comprising a block encoding of values of pixels in different regions of the image, the block encoding mapping a set of values of the pixels to respective image lexical elements; and adding the sequence of image lexical elements behind the current output sequence of the multi-modal lexical elements as the following multi-modal lexical elements in the output sequence of the multi-modal lexical elements.

Description

Cross-reference to related applications

[0001] This application claims priority to U.S. Provisional Application No. 63 / 640,140, ​​filed April 29, 2024. The disclosure of the prior application is considered part of the disclosure of this application and is incorporated herein by reference. Background Technology

[0002] This manual relates to using machine learning models to process data.

[0003] Machine learning models receive input and generate outputs, such as predicted outputs, based on the received input.

[0004] A neural network is a machine learning model that uses one or more layers of non-linear units to predict the output from a given input. In addition to the output layer, some neural networks also include one or more hidden layers. The output of each hidden layer serves as the input to the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates its output from the received input based on the current values ​​of its corresponding set of parameters. Summary of the Invention

[0005] This specification describes a method for generating multimodal data, implemented as a computer program on one or more computers at one or more locations. A method for training a system to generate multimodal data items is also described. A corresponding system is also described. Some implementations of the described techniques address the "negative transfer" problem, where training on multiple modalities adversely affects performance on individual modalities.

[0006] In a first aspect, a computer-implemented method for generating multimodal data using a system is described. The system includes a word-generating neural network and a data (e.g., image) generation subsystem. The data (e.g., image) generation subsystem may include an image-generating neural network; it can implement a diffusion model.

[0007] This method involves: receiving a cue sequence as an input sequence defining multimodal lexical units; and using a lexical generation neural network to process the input sequence of multimodal lexical units to generate an output sequence of multimodal lexical units. A multimodal lexical unit can represent a data element of one of multiple modalities.

[0008] The output sequence of generating multimodal lexical units includes, for each successive position in the output sequence of multimodal lexical units, the following autoregressive steps: using a lexical generation neural network to process a combined sequence of the input sequence of multimodal lexical units and the current output sequence of multimodal lexical units to generate the next multimodal lexical unit in the output sequence of multimodal lexical units; and appending the next multimodal lexical unit to the current output sequence of multimodal lexical units.

[0009] This method involves, in response to the next multimodal lexical being an image-starting lexical: generating an image; processing the image to convert its pixels into a sequence of image lexical units; and appending the sequence of image lexical units to the current output sequence of the multimodal lexical units as the next multimodal lexical unit in the output sequence of the multimodal lexical units. The image is generated using an image generation subsystem. The image is generated conditioned on features representing the current output sequence of multimodal lexical units obtained from a lexical generation neural network.

[0010] In a second aspect, a method for training a system to generate multimodal data items is described. The system includes a word-generating neural network and a data (e.g., image) generation subsystem, which includes the data (e.g., image) generation neural network, such as a diffusion model neural network. The system may also include a block encoder.

[0011] The method includes: obtaining a set of multimodal training examples, each training example including text or audio data and at least one training image.

[0012] The method includes generating, for each training example, a corresponding sequence of real-valued multimodal lexical units representing text or audio data and training images.

[0013] Generating ground truth multimodal terms representing training images may include: using a block encoder to process the training image to transform the pixels of the training image into a sequence of image terms. Each image term may include a block encoding of the values ​​of pixels in different regions (e.g., patches or tiles) of the training image, which maps a set of pixel values ​​to a corresponding image term.

[0014] In some implementations of this system, during training and / or inference, the block encoder feeds the “raw” pixel values ​​from each region into the word generation neural network. That is, the block encoding process may simply involve dividing an image or other data item into regions or patches to determine the pixel values ​​within a region or patch, which are then provided to the word generation neural network as one or more terms. For example, in some implementations, fixed regions or patches of the image (i.e., fixed blocks of pixels) correspond to terms, and the representation of this term is independent of previous portions of the image representation. The block encoder can encode regions or pixel blocks independently of other regions or pixel blocks of the image. Encoding can be constant because the encoding of a particular block of pixel values ​​is always the same (during inference). In some implementations, the number of terms representing the image varies depending on the image size (number of pixels). In some implementations, the block encoder can be differentiable; this does not preclude constant encoding (during inference, but not training).

[0015] The sequence of ground truth terms includes an image start term. The image start term can be positioned before (e.g., immediately preceding) the ground truth multimodal term representing the training image. The term representing the training image can be, for example, a term representing the first patch of the training image. Generally, an image patch corresponds to a small area of ​​the image (e.g., pixels in a region of the image). An image can be represented by multiple patches, which are combined to form the entire image.

[0016] The method further includes: determining multiple training sequences. Each training sequence comprises a subsequence of a sequence of ground-value multimodal words. In some implementations, the subsequence derived from the sequence of ground-value multimodal words (i.e., the training sequence) comprises a series of subsequences of increasing sequence length, for example, from 1 to L, where L is the length of the sequence of ground-value multimodal words. Some training sequences end with an image-starting word.

[0017] The method includes: for each training sequence, in response to determining that the last ground truth multimodal term in the subsequence is an image starting term, using an image generation subsystem to generate a predicted image conditioned on features representing the subsequence.

[0018] The method includes: when the last ground truth multimodal word in a subsequence is not an image-starting word, using a word generation neural network to process the subsequence to generate a predicted next multimodal word. Alternatively, the method may include: when the last ground truth multimodal word in a subsequence is an image-starting word, using a word generation neural network to process the subsequence to generate a predicted next multimodal word. In other words, when the last ground truth multimodal word in a subsequence is an image-starting word, the method may include: using an image generation subsystem to generate a predicted image; and using a word generation neural network to process the subsequence to generate a predicted next multimodal word.

[0019] The method further includes training the system based on training examples. This training uses an image reconstruction target dependent on the predicted image and the training image. The training also uses a lexical prediction target dependent on the predicted next multimodal lexical and the next ground truth multimodal lexical following the subsequence. The training includes updating the learnable parameters of the lexical generation neural network and the image generation neural network. That is, the lexical generation neural network and the image generation neural network are jointly trained. In an implementation where the system includes a block encoder, this training may (but does not necessarily) further include updating the learnable parameters of the block encoder. That is, in such an implementation, the block encoder can be jointly trained with the lexical generation neural network and the image generation neural network.

[0020] Multimodal training examples may include training examples in which at least one training image follows the text or audio data. Additionally or alternatively, multimodal training examples may include training examples in which at least one training image precedes the text or audio data.

[0021] Processing an image to convert its pixels into a sequence of image terms can be performed by a block encoder. Each image term can include a block code of the values ​​of pixels in different regions of the image, mapping a set of pixel values ​​to the corresponding image term. In such implementations, the system includes a block encoder. Generally, a block encoder is a differentiable block encoder, i.e., it has learnable parameters that can be updated using a gradient descent optimization algorithm. It can be implemented, for example, using a neural network.

[0022] In some implementations, the block encoder encodes each region or block of the image independently of other regions (blocks). That is, in some implementations, image terms do not depend on previously generated image terms. In other implementations, after training, the block encoder implements deterministic (e.g., constant) mapping (encoding), meaning it always maps the same set of pixel values ​​for an image region to the same corresponding image terms. In some implementations, the image is mapped to a certain number of image terms; the number of image terms is variable and depends on the image size, i.e., larger images are encoded with more terms.

[0023] In some implementations of the subject matter described herein, an image generation subsystem can be used to generate audio (sound) by representing audio as an image—specifically, a spectrogram. In some implementations of the subject matter described herein, the image generation subsystem is replaced by an audio generation subsystem that does not depend on intermediate steps of image generation—specifically, an audio generation subsystem that includes an audio generation neural network. For example, the audio generation subsystem may include a speech generation subsystem. Generally, any audio generation subsystem can be used. In such implementations, image terms are replaced with audio terms; these audio terms can be generated by an audio block encoder to encode temporal or time-frequency domain audio blocks. Similarly, after training, the audio block encoder can implement a deterministic (e.g., constant) mapping (encoding) from audio blocks to audio terms. Many techniques exist that can be used for audio block encoding; in some implementations, a linear audio region encoding neural network is used in a manner corresponding to a linear image region encoding neural network, i.e., encoding audio as a linear projection of temporal or time-frequency domain audio data over a temporal or time-frequency region.

[0024] The subject matter described in this specification can be implemented in specific embodiments to achieve one or more of the following advantages.

[0025] The described system implementation addresses the "negative transfer" problem, where training on multiple modalities (e.g., images and text) in a multimodal model often negatively impacts the performance of each modality compared to training on a single modality (e.g., text only). Ideally, training should not hinder the performance of other modalities but should instead improve the performance of each modality when training on multiple modalities—a phenomenon known as "positive transfer."

[0026] One implementation of the described system addresses this problem by offloading image generation to a separate model (in this implementation, a diffusion model) and encoding the images generated by that model using a block encoding scheme. Specifically (after training), this block encoding scheme uses a deterministic (e.g., constant) mapping to map the values ​​of pixels in regions of an image to a set of pixel values, and then to the corresponding image terms. Each image region is encoded independently, and the number of terms encoded for each region is variable, even when the image size is variable. In some implementations, a linear encoding scheme (i.e., a linear mapping from pixel values ​​to image term values) is used to encode the images. Combining these two approaches facilitates the generation and preservation of image details when processing multimodal terms, thereby addressing the negative transfer problem. Furthermore, during training, the block encoder can be specifically adapted to the overall model.

[0027] Images generated using the described system are conditional on features representing the current output sequence. Therefore, image generation can be conditionalized based on previously generated images. This allows for improved consistency in the generation of multiple images. A series of images (e.g., multiple images in a set) can be generated, with consistency between each image. For example, if a series of images representing a specific location is generated, each image within that set can be consistent with other images in the set, such as representing the same time of day, the same weather, the same subject (e.g., the same person, the same object category, etc.). The subject matter described herein can be used to perform prompt-based image editing, where a series of output images can be updated based on one or more text prompts.

[0028] Generally, once trained on a large corpus, the system can perform a wide range of tasks even without fine-tuning. After training, the system can be deployed to perform tasks. For example, it can be deployed in an environment that allows users to request that the system process specified multimodal inputs to generate corresponding outputs. Users can make requests, for example, through a user interface or an application programming interface (API). Requests can be transmitted from the user device (e.g., via a data communication network, such as the Internet) to one or more computers implementing the system (e.g., in a data center). The system can process the multimodal inputs specified by the user request to generate corresponding outputs and then transmit the outputs to the user device (e.g., via a data communication network).

[0029] In some implementations, after training, the system can be prompted to describe specific tasks to be performed by it. For example, if the prompt includes an image, it could specify "Generate a caption," "Generate a description," "Answer the following question: [about the image or video]," or "Detect a person." In cases where the system is used for agent control tasks, the prompt could be defined as "Take the knife out of the drawer," or "Q: What action should the robot take to take the knife out of the drawer?" Alternatively, the prompt could provide one or more examples of the task to be performed. The system can be trained on various natural and / or computer languages, and the prompt can then specify the language to be used.

[0030] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of this subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0031] Figure 1 An example system for generating multimodal data is shown.

[0032] Figure 2 This is a flowchart of an example process for generating multimodal data.

[0033] Figure 3A and Figure 3B This illustrates the specific implementation of the image generation subsystem.

[0034] Figure 4A and Figure 4B Experimental inferences associated with the system used to generate multimodal data are shown.

[0035] Figure 5 This is a flowchart of an example process used to train a system to generate multimodal data.

[0036] The same reference numerals and names indicate the same elements in each figure. Detailed Implementation

[0037] Figure 1 An example system 100 for generating multimodal data is shown, implemented as a computer program on one or more computers at one or more locations. The system includes a word-generating neural network 102 and a data generation subsystem 104, such as... Figure 1 The example shows an image generation subsystem 104. Optionally, the system further includes a block encoder (not shown).

[0038] As used in this specification, "modal" refers to a data type. Multimodal data is data that includes two or more different data types, such as data that includes two or more of the following different data types: text data, audio data such as speech data (specifically, data that defines an audio waveform), or image data (specifically, data that defines the pixel values ​​of pixels in a still or moving image).

[0039] Text data can represent text in natural or computer languages. Text can be received, for example, as a series of encoded characters (e.g., characters encoded in UTF-8); such "characters" can include Chinese and other similar characters, as well as logograms, syllabograms, and the like. Text can be processed to represent it as a series of text tokens in a vocabulary, where each text token represents a word, word fragment, or character in a natural or computer language. Computer languages ​​can be any form of language used for communicating with computers, such as markup languages, command or configuration languages, data exchange languages ​​(such as JSON), or programming languages. There are many ways to represent text as a series of text tokens; one way is to use a text tokenizer.

[0040] As used in this specification, a "lexical" is a data element composed of numerical vectors and having a specified dimension. Generally, image lexicals and multimodal lexicals can have the same dimension. A multimodal lexical can represent a data element of one modality among multiple modalities (e.g., representing text, image, or audio). For a set of multimodal lexicals, each of the multiple multimodal lexicals can represent a data element of one modality among multiple modalities (e.g., representing text, image, or audio). As above, generally, each multimodal lexical can have the same dimension. Therefore, a multimodal lexical can refer to multiple lexicals of a single modality among multiple modalities, or multiple lexicals of two or more modalities among multiple modalities.

[0041] Images can be still or moving (e.g., video), and can be 2D or 3D. Images can be associated with light in the electromagnetic spectrum (e.g., visible, infrared, or ultraviolet light). Images can be monochrome, color, or hyperspectral images, or LiDAR images, in which case "pixels" can include points in a point cloud. It is typically represented by the pixel values ​​of an image (more specifically, voxels in a 3D image). Images can also be audio spectrograms, i.e., images representing the time-frequency of audio.

[0042] The images described herein (whether during training or inference) can be images of real-world environments, such as those captured from the real world by a camera or microphone or other image or audio sensor. Depending on the type of image captured, the objects represented in the image can be physical real-world objects or real-world sounds (e.g., spoken words).

[0043] System 100 is configured to receive a prompt sequence of input sequence 106 that defines or includes multimodal lexical terms.

[0044] The input sequence of multimodal lexical units as described herein may include multimodal lexical units representing combinations of two or more of the following: text data elements, audio data elements, or image data elements. Text data elements may be, for example, words or word slices as previously described. Audio data elements may represent time segments of audio (e.g., audio waveforms). Image data elements may include pixel values ​​of images or regions (e.g., patches) of images. Similarly, the output sequence 116 of multimodal lexical units as described herein may include multimodal lexical units representing combinations of two or more of the following: text data elements, audio data elements, or image data elements.

[0045] In some implementations, the input sequence comprises multiple or various lexical units that are interleaved with each other across different modalities. A modality can be any two or more of the following: text data elements, audio data elements, and / or image data elements. Generally, the number of lexical units in each modality can be different, and therefore the interleaving of lexical units from different modalities may not be strictly alternating. More generally, assuming the modalities of the lexical units are type "A" (e.g., representing text) and "B" (e.g., representing image or audio), the input sequence can alternate between chunks or clusters of type "A" lexical units and chunks or clusters of type "B" lexical units. A specific example of an input sequence of multimodal lexical units could be "A, A, A, B, B, B, B, A, A, B, B, B, B".

[0046] In some implementations, the input sequence includes an "image start" or "image origin" (referred to as "boi" in this document) terminology to mark the beginning of one or more image terms in the input sequence. Generally, the "boi" terminology also marks or delineates the transition between terms of a first modality and terms of a second different modality (e.g., between terms of type "A" and "B").

[0047] In some implementations, one or more image terms in the input sequence comprise “raw” (i.e., unprocessed) pixel patches of an image. These “raw” pixel patches can be aggregated to form an image. This helps improve inference performance because no information is lost from the one or more images associated with the one or more image terms.

[0048] In some implementations, the input sequence includes a “sequence start” (“bos”) term and a “sequence end” (“eos”) term to mark the beginning and end of the sequence, respectively.

[0049] In some implementations, the lexical generation neural network adds lexical positional encoding to each multimodal lexical in the input sequence (e.g., audio lexical, image lexical, and / or text lexical). Any suitable positional encoding can be used, such as relative positional encoding, rotational positional encoding (such as RoPE), or absolute positional encoding.

[0050] The lexical generation neural network 102 is configured to process the input sequence 106 of multimodal lexical units to generate an output sequence 116 of multimodal lexical units. The modality of the output sequence can be any two or more of the following: text data elements, audio data elements, and / or image data elements. Each successive lexical unit 108 generated by the lexical generation neural network is appended to the current output sequence 110 of lexical units (which may include different or the same type of multimodal lexical units).

[0051] In some implementations, the current output sequence 110 includes output lexical units at each position preceding the position of the next multimodal lexical unit that the lexical generation neural network in the output sequence will predict (e.g., representing text, image, or audio).

[0052] In some implementations, the word generation neural network 102 generates the output sequence of multimodal words by autoregressively processing a combination sequence of the input sequence of multimodal words and the current output sequence of the words for each successive position in the output sequence of multimodal words.

[0053] In some implementations, the word-generating neural network 102 autoregressively generates multiple text words (in) before autoregressively generating the word "boi". Figure 1 The lexical units are labeled t0, t1, and t2. It will be understood that in some implementations, the first lexical unit generated autoregressively by the lexical generation neural network can be the "boi" lexical unit.

[0054] A data (e.g., image) generation subsystem 104 is configured to process an input image and generate an output image in response to a word generation neural network autoregressively generating the word "boi". That is, the image generation subsystem is triggered to generate the output image conditioned on the word generation neural network generating the word "boi". The generated output image is conditioned on features representing the current output sequence of words obtained from the word generation neural network. The data (e.g., image) generation subsystem 104 may include, for example, a diffusion model or an autoregressive model conditioned on features representing the current output sequence of words.

[0055] In the specific implementation, the features used to conditionally condition the image generation subsystem are determined from the output features of the "boi" terminator, which is used to generate the summarizing multimodal terminator. The "boi" terminator is a convenient choice for generating the summarizing multimodal terminator because, as the terminator generation neural network autoregressively generates successive terms, the "boi" terminator already represents or provides a summary of all preceding terms generated by the terminator generation neural network, and because there are no target text terms to predict for the "boi" terminator. This means that the values ​​of the output features of the "boi" terminator can be assigned or otherwise processed during training.

[0056] Alternative implementations for generating summarizing multimodal lexical units are envisioned, such as using all preceding lexical units generated by a lexical unit generation neural network 102 (or all preceding lexical units before the sentence's starting lexical unit) to generate the summarizing multimodal lexical units. This could involve pooling features (e.g., mean pooling or max pooling) from the output features of preceding lexical units in the current output sequence. In a further implementation, a combination of the pooled features and the features derived from the "boi" lexical unit can be used to generate summarizing multimodal lexical units for conditionalizing image generation.

[0057] In some implementations, the system is configured to process images generated by the image generation subsystem 104 to convert the pixels of the image into a sequence 114 of image terms. Each image term may include a block code of the values ​​of pixels in different regions of the image, which maps a set of pixel values ​​to a corresponding image term. Each image term in one or more image terms represents a patch or region of the image, which, when aggregated, forms the output image 112. That is, an image term may be or represents a patch or region of the output image 112.

[0058] The system is then configured to append a sequence 114 of image terms to the current output sequence of terms (i.e., immediately following the "boi" term) as the next term in the output sequence of multimodal terms. The current output sequence then includes multimodal terms of the first modality (i.e., text) and multimodal terms of the second modality (i.e., image).

[0059] Image processing, which converts the pixels of an image into a sequence of image terms, can be accomplished by a block encoder (…). Figure 1 (Not shown in the image) is executed. Each image term may include block encoding of the values ​​of pixels in different regions of the image, which maps a set of pixel values ​​to the corresponding image term.

[0060] In some implementations, the system or block encoder encodes each region or block of the image independently of other regions (e.g., blocks). Each block or region of the image mapped to image terms or encoded as image terms can have a fixed size or dimension. Therefore, in some implementations, the number of image terms generated by the system or block encoder is variable and depends on the image size. That is, the larger the resolution of the image, the more terms are encoded.

[0061] In some implementations, any given image lexicon generated by the system or block encoder does not depend on any previously generated image lexicon (e.g., the same image or a different image generated by the image generation subsystem).

[0062] In some implementations, the encoding can be consistent or deterministic, such as constant, because the system or encoder encodes the same set of values ​​for pixels in an image region into the same corresponding image terms.

[0063] In some implementations, the block encoder is a differentiable block encoder, meaning it has learnable parameters that can be updated using a gradient descent optimization algorithm. It can be implemented, for example, using a neural network.

[0064] In some implementations, the lexical generation neural network 102 is configured to continue processing a combined sequence comprising an input sequence of multimodal lexical units and a current output sequence of multimodal lexical units to generate or cause the generation of further lexical units to be appended to the output sequence of multimodal lexical units. These further lexical units may include text lexical units, “boi” lexical units, or image lexical units generated from a further output image generated by the image generation subsystem 104 in response to the lexical generation neural network generating “boi” or “eos” lexical units. The “eos” lexical unit is the final lexical unit of the output multimodal lexical unit sequence.

[0065] In some implementations, the sequence of processing an image to convert its pixels into image terms includes: processing the image to divide it into a set of regions or tiles; and a linear projection that determines each image term as the value of a pixel in a corresponding region. That is, the implementation of block coding for processing the output image into corresponding image terms does not use a discrete lookup table. In this implementation, the linear projection proceeds from a dimension determined by multiple pixel values ​​in the regions or tiles to a dimension that matches the dimension of each multimodal term in the multimodal vocabulary.

[0066] In some implementations, the multimodal data of the input and output sequences includes audio data. The audio data (which can represent spoken words) can include values ​​of the audio waveform, such as the instantaneous amplitude values ​​of the waveform. The audio data can be represented as a spectrogram (i.e., a time-frequency representation of the instantaneous amplitude of the audio waveform). In other words, in some implementations, the image includes an audio spectrogram.

[0067] In this implementation, the system is configured to convert an audio spectrogram into time-series audio data of an audio waveform. The spectrogram can be, for example, a Mel spectrogram. The time-series audio data of the audio waveform can represent the instantaneous amplitude values ​​of the audio waveform. The audio waveform can include the waveform of speech in natural language.

[0068] Since a spectrogram is a form of image, the image generation subsystem should not be interpreted as limited to outputting images associated with a specific frequency range of the electromagnetic spectrum (e.g., visible light, infrared light, ultraviolet light, etc.). That is, the image generation subsystem can generate an image (in the form of a spectrogram) representing audio data, which is then block-encoded into image terms (e.g., one or more audio terms) as described above.

[0069] In some implementations, in addition to or instead of the image generation subsystem, the data generation system 104 may include an audio generation system, such as an audio diffusion model or an autoregressive audio generation model.

[0070] The word generation neural network can be a Transformer neural network, such as a so-called decoder-only Transformer neural network, i.e., a neural network represented by a series (e.g., one or more) self-attention neural network layers. An example Transformer model is described in Vaswani et al. (arXiv:1706.0372). More specifically, a neural network can include one or more Transformer layer blocks. As used herein, a Transformer layer block is a collection of one or more attention neural network layers.

[0071] For example, one or more neural network layers in a transformer layer block may include one or more attentional or self-attention neural network layers that each use an attention mechanism to apply attention or self-attention operations; these attentional or self-attention neural network layers may be followed by feedforward neural network layers.

[0072] In some implementations, the word generation neural network 102 may include one or more self-attention neural network layers. The self-attention neural network layer has an attention layer input for each element of the input and is configured to apply an attention mechanism to the attention layer input to generate an attention layer output for each element of the input. Many different attention mechanisms are available. For example, a self-attention operation can be a self-attention operation that applies an attention mechanism to the elements of an embedding (as a representation of entities as an ordered set of numerical values) to update each element of that embedding. For example, the input embedding may be used to determine a query vector and a set of key-value vector pairs, and the updated embedding may include a weighted sum of values ​​weighted by a similarity function between the query and each corresponding key.

[0073] Typically, to apply self-attention operations in an attention layer, each attention mechanism uses one or more attention heads. Each attention head generates a set of queries, a set of keys, and a set of values, and then applies any of the various variations of query-key-value (QKV) attention (e.g., a dot product attention function or a scaled dot product attention function) to generate output using the queries, keys, and values.

[0074] As a specific example, in the attention head of a self-attention neural network layer, the attention mechanism can be configured to apply each of the query transformation, key transformation, and value transformation to the attention layer input for each embedding of the input sequence X to derive the corresponding query vector, key vector, and value vector for determining the updated embedding. The query transformation, key transformation, and value transformation can be any corresponding linear transformation or any other suitable learned transformation. For example, the attention head can generate the updated embedding for each input position by computing a weighted sum of values, weighted by a similarity function between the query and the corresponding key of the input position. The similarity function can include, for example, a dot product, cosine similarity, or other similarity metrics.

[0075] When attention heads use positional encoding, the application of the dot product attention function, the query, the calculation of the key and value, or both, depends on the relative or absolute position of the corresponding query, key, and value embedded in the input sequence.

[0076] For example, an implementation of RoPE could involve: for a given query at a corresponding input position, determining a query rotation matrix that represents the absolute or relative position of the corresponding input position of the query, such as an index of an input position in a sequence; for a given key at a corresponding input position, determining a key rotation matrix that similarly represents the absolute or relative position of the corresponding input position of the key, such as an index of an input position in a sequence; and multiplying the query rotation matrix, the key rotation matrix, the query (vector), and the key (vector) to determine a weight value between the query and the key, which depends on the relative distance between the position corresponding to the key and the position corresponding to the query.

[0077] As another example, an ALiBi implementation could involve adding a linear bias matrix to the weights determined by the combination of the key and the query.

[0078] When the attention head does not use positional encoding, the application of the dot product attention function and the computation of queries, keys, and values ​​are both independent of the relative or absolute positions embedded in the input sequence corresponding to the queries, keys, and values.

[0079] Each query, key, and value can be a vector containing one or more vector elements. When multiple attention heads exist, the attention block then combines the outputs of the multiple attention heads, for example, by concatenating the outputs and optionally processing the concatenated outputs through a linear layer.

[0080] For the local attention mechanism, for each location, the location used to generate the query, key, and value for that location is defined by the local window size of the local attention mechanism; that is, the non-zero attention weight for a given location is computed only for the location within the local window of that location.

[0081] In some cases, because the attention applied by the attention layer is causal, the system can store embeddings or key-value pairs that have been computed for earlier input position steps when generating output for any given attention mechanism, rather than recompiling the embeddings (or key-value pairs) for earlier time steps.

[0082] Therefore, in these cases, updating the corresponding embedding by applying the attention mechanism to the corresponding embedding of each input position in the input position means updating the corresponding embedding of the last input position in the current input sequence using the key and value already retrieved from memory (e.g., from a "cache") or the embedding of other input positions. Storing the key and value in memory for later reuse can be referred to as storing the key and value in a "KV cache".

[0083] In some implementations, some or all of the layers in a neural network may include other types of layers besides attention layers, such as normalization layers, residual connection layers, feedforward layers, etc.

[0084] During inference, the transformer can operate in autoregressive mode.

[0085] In autoregressive mode, the transformer generates an output sequence of terms by processing the most recently generated terms in the output sequence at each of multiple time steps to generate new output terms to be added to the output sequence.

[0086] In some implementations, the image generation subsystem includes a diffusion model image generation subsystem. That is, the input image is generated by sampling the values ​​of the image's pixels or the latent vector representation of the image from a noise distribution. Generating the image then involves initializing the image or latent vector representation by sampling the values ​​of the image's pixels or the latent vector representation of the image from the noise distribution. Image generation may also include: at each of a series of time steps, determining an updated version of the image or its latent vector representation (e.g., by processing the time step and processing the image or its latent vector representation at that time step), and using an image generation neural network conditioned on features representing the current output sequence of multimodal terms to determine a noise-reduced version of the image or its latent vector representation. An example of implementing a diffusion model in the latent variable space is described in arXiv:2112.10752.

[0087] In the case of using a diffusion model to generate moving images, this can be accomplished in various ways. As one example, the time axis can be viewed as an additional spatial dimension. As another example, techniques such as those described in arXiv:2402.09470 can be used.

[0088] In some implementations, as follows: Figure 3A As shown in more detail, the image generation neural network has a U-net architecture. Generally, the U-Net architecture maps inputs of a given dimension to outputs of the same dimension. The U-net architecture has one or more cross-attention neural network layers. In its implementation, using the image generation subsystem to generate predicted images conditioned on features representing the current output sequence includes: using one or more cross-attention neural network layers to focus on features of the current output sequence obtained by processing the current output sequence (e.g., the "boi" word) using a word generation neural network.

[0089] Cross-attention neural network layers can be similar to the self-attention neural network layers described above, but in which the query is derived from one embedding, and the keys and values ​​are derived from different embeddings. For example, the query can be obtained from features generated by U-Net, and the keys and values ​​can be obtained from features of the current output sequence.

[0090] In some implementations, such as when the lexical generation neural network has a Transformer neural network architecture, the features of the current output sequence may include, for example, features of the final self-attention neural network layer of the lexical generation neural network, or features of subsequent linear layers, or features of subsequent softmax layers (“soft lexical”).

[0091] In some implementations, the features of each successive multimodal lexical unit—specifically, the output features—can be cached when generating each successive multimodal lexical unit for later use by the cross-attention neural network layer.

[0092] Now, an example task that can be implemented by System 100 is described:

[0093] Tasks may include generating an image specified by a prompt.

[0094] The prompt may include an image. The task may include generating a modified version of the image. The modifications to be performed may be described by the prompt.

[0095] The prompt may include an image. The task may include an optical character recognition task, which involves generating an output sequence of multimodal lemmas representing words or characters in an image.

[0096] The prompt may include an image. The task may include generating an output sequence of multimodal lexical units representing answers to questions about the image.

[0097] The prompt may include an image and identify one or more objects in the image. The task may include generating an output sequence of multimodal lemmas that define the presence, location, orientation, or count of one or more objects in the image.

[0098] Hints may include images. Tasks may include generating an output sequence of multimodal terms that describe the content of an image or classify the content of an image into one or more categories.

[0099] In some implementations, the cue sequence may include text and / or audio defining the audio generation task. The time-series audio data of the audio waveform can define the audio specified by the cue sequence. The audio may include spoken words from natural language.

[0100] In some implementations, the cue sequence may include text or audio defining an image generation task or image processing task. The image may define the result of the task. The task may include generating an image specified by the cue.

[0101] The prompt may include an image, and the task may include generating an edited version of the image, wherein the edits to be performed are described by the prompt.

[0102] The prompt may include an image, and the task may include an optical character recognition task involving generating an output sequence of multimodal lemmas representing words or characters in the image.

[0103] The prompt may include an image, and the task may include generating an output sequence of multimodal lexical units representing answers to questions about the image.

[0104] The prompt may include an image and identify one or more objects in the image, and the task may include generating an output sequence of multimodal lemmas that define the presence, location, orientation, or count of one or more objects in the image.

[0105] The prompt may include an image, and the task may include generating an output sequence of multimodal lexical units that describe the content of the image or classify the content of the image into one or more categories.

[0106] The prompt may include an image and define the goal of a mechanical agent acting in a real-world environment, and the task may include generating an output sequence of multimodal lexical units that define one or more actions to be performed by the mechanical agent to achieve the goal. In some implementations, the system is configured to cause the mechanical agent to perform one or more actions defined by the output sequence of multimodal lexical units.

[0107] Figure 2 This is a flowchart of an example process 200 for generating multimodal data. Figure 2 The process can be performed by one or more computers in one or more locations (e.g. Figure 1 The system shown in the figure is implemented.

[0108] In step 202, the method includes: receiving a prompt sequence that defines or includes an input sequence 106 of multimodal terms.

[0109] As described above, the input sequence of multimodal lexical units can include multimodal lexical units representing a combination of two or more of the following: text data elements, audio data elements, or image data elements. Text data elements can be, for example, words or word slices as previously described. Audio data elements can represent time segments of audio (e.g., audio waveforms). Image data elements can include pixel values ​​of an image or a region (e.g., a patch) of an image.

[0110] In some implementations, one or more image terms in the input sequence comprise “raw” (i.e., unprocessed) pixel patches of images. This improves inference because no information is lost from the one or more images associated with the one or more image terms.

[0111] In some implementations, the method further includes adding a lexical positional encoding to each multimodal lexical unit in the input sequence (e.g., audio lexical units, image lexical units, and / or text lexical units). Any suitable positional encoding can be used, such as relative positional encoding, rotational positional encoding (such as RoPE), or absolute positional encoding.

[0112] In step 204, the method includes: using a lexical generation neural network 102 to regressively generate the next multimodal lexical from an output sequence 116 of multimodal lexical by processing a combined sequence of an input sequence including multimodal lexical and a current output sequence of multimodal lexical. The current output sequence can initially be empty or null. The modality of the output sequence can be any two or more of the following: text data elements, audio data elements, and / or image data elements.

[0113] In step 206, the method includes appending the next multimodal lexical 108 to the current output sequence 110. This step represents an update to the current output sequence to include the last previous multimodal lexical generated by the lexical generation neural network 102.

[0114] In step 208, it is determined whether the next multimodal lexical is "image start" or "image beginning" (i.e., the "boi" lexical). If the determination is negative, the method continues to repeat step 204 (or optionally together with step 216), i.e., generating a new multimodal lexical. Therefore, steps 204 to 206 can be repeated for each successive position in the output sequence of multimodal lexical ...

[0115] In step 210, the image generation subsystem generates an image conditioned on features representing the current output sequence of multimodal words obtained from the word generation neural network (and which includes the word "boi"). That is, the image generation subsystem is triggered to generate the output image conditioned on the word generation neural network generating the word "boi". The generated output image is conditioned on features representing the current output sequence of words obtained from the word generation neural network.

[0116] In a specific implementation, the features used to conditionally condition the image generation subsystem are determined from or otherwise obtained from the output features of the "boi" terminology, which is used to generate the summarizing multimodal terms. The "boi" terminology is a convenient choice for generating the summarizing multimodal terms because, as the terminology generation neural network autoregressively generates successive terms, the "boi" terminology already represents or provides a summary of all preceding terms generated by the terminology generation neural network, and because there are no target text terms to predict for the "boi" terminology. For example, the current output sequence including the "boi" terminology is taken as input to the terminology generation neural network 102. The output features of the "boi" terminology are then used to conditionally condition the image generation subsystem.

[0117] Alternative implementations for generating summarizing multimodal lexical units are envisioned, such as using all preceding lexical units generated by a lexical generation neural network 102 (or all preceding lexical units before the sentence's starting lexical unit) to generate the summarizing multimodal lexical units. This could involve pooling features from the output features of preceding lexical units in the current output sequence. In a further implementation, a combination of the pooled features and features derived from the "boi" lexical unit can be used to generate summarizing multimodal lexical units for conditionalizing image generation.

[0118] In some implementations, the method includes using a word-generating neural network to process a combined sequence including image-starting words to generate features of a summary multimodal word. The summary multimodal word can be generated as described above, i.e., autoregressively, such that it includes features summarizing the current output sequence. Generating an image using an image generation subsystem conditioned on features representing the current output sequence of multimodal words obtained from the word-generating neural network can include generating an image conditioned on features of the summary multimodal word. The generation and use of the summary multimodal word can be performed in combination with or as an alternative to any of the other methods for conditionalizing image generation described herein.

[0119] In step 212, a sequence of image terms is generated from the output image. That is, the pixels of the image are processed into a sequence of image terms. Each image term includes a block code of the values ​​of pixels in different regions (or patches) of the image, which maps a set of pixel values ​​to the corresponding image term.

[0120] In some implementations, the sequence of processing an image to convert its pixels into image terms includes: processing the image to divide it into a set of regions or patches that tile the image; and a linear projection that determines each image term as the value of a pixel in a corresponding region. That is, the implementation of block coding for processing the output image into corresponding image terms does not use a discrete lookup table. In this implementation, the linear projection proceeds from a dimension determined by multiple pixel values ​​in the regions or patches to a dimension that matches the dimension of each multimodal term in the multimodal vocabulary.

[0121] In step 214, a sequence of image terms is appended to the current output sequence of multimodal terms as the next multimodal term in the output sequence of multimodal terms. This step represents an update to the current output sequence to include any image terms generated by the image generation subsystem in response to the terminology generation neural network generating the "boi" terminology.

[0122] In optional step 216, the method includes: determining whether the next multimodal lexical is a "sequence end" lexical. If the determination is positive, the method proceeds to step 218. If the determination is negative, the method proceeds to step 204.

[0123] In step 218, the method ends.

[0124] In this manner, performing steps 204 to 206 or steps 204 to 214 results in the input sequence 106 of multimodal lexical units being processed to generate an output sequence of multimodal lexical units.

[0125] In some implementations, the method further includes: continuing to process the combined sequence after appending the sequence of image terms to the current output sequence of multimodal terms to generate further multimodal terms of the output sequence of multimodal terms. That is, steps 204 to 206 and 204 to 214 can be repeated to continue processing the combined sequence including the input sequence of multimodal terms and the current output sequence of multimodal terms to generate or cause the generation of further terms for appending to the output sequence of multimodal terms. The further terms may include text terms, “boi” terms, or image terms generated from a further output image generated by the image generation subsystem 104 in response to the generation of “boi” or “eos” terms by the terminology generation neural network. The “eos” term is the final term of the output multimodal term sequence. As noted above, the “eos” term triggers the end of the method.

[0126] As noted above, in some implementations, the multimodal data of the input and output sequences includes audio data. Audio data (which can represent spoken words) can include values ​​of the audio waveform, such as the instantaneous amplitude values ​​of the waveform. Audio data can be represented as a spectrogram (i.e., a time-frequency representation of the instantaneous amplitude of the audio waveform). In other words, in some implementations, the image includes an audio spectrogram.

[0127] In such implementations, the method further includes converting an audio spectrogram into time-series audio data of an audio waveform. The spectrogram may be, for example, a Mel spectrogram. The time-series audio data of the audio waveform may represent the instantaneous amplitude values ​​of the audio waveform. The audio waveform may include the waveform of speech in natural language.

[0128] As described above, it will be understood that a spectrogram is a form of image, and method 200 may involve outputting an audio image that is then block-encoded into image lexics (e.g., one or more audio lexics).

[0129] The example prompt sequence is as described above.

[0130] Figure 3AThe specific implementation of the image generation subsystem 304 is shown. The image generation subsystem includes a diffusion model neural network 306 (also referred to as the "diffusion model" in this paper) with a U-ViT architecture (Ronneberger et al., "U-Net: Convolutional Networks for Biomedical Image Segmentation", arXiv:1505.04597), which maps an input image 311 of a given dimension (e.g., a 128x128 matrix) to an output image 312 of the same dimension (e.g., a 128x128 matrix).

[0131] Figure 3A The current output sequence 310 generated by the lexical generation neural network 302 is also shown, which is truncated to show only its output layer. Figure 3B A diagram illustrating the image generation subsystem 304 together with the word generation neural network 302 is shown, illustrating the input to the word generation neural network 302, where the image start word is the final word in the input sequence of words. For simplicity, only the autoregressive arrows pointing to direct neighbors are visualized. In the example diagram shown, the word generation neural network 302 is combined with a diffusion model neural network 306 for image generation. The diffusion model neural network 306 is conditioned on all output features of the word generation neural network 302, thus supporting end-to-end training on interleaved sequences. When the image start word is generated during inference, the diffusion model neural network 306 is triggered to generate the next image. In the example diagram shown, new image words are passed to the word generation neural network 302 in parallel, which allows for bidirectional attention masks between image words. In the example implementation, image words are encoded using a single linear layer, and no separate image encoder is used. In the example implementation, for the input of the language modality, the text is lexicalized, and each word is encoded using an embedding lookup table. In the example implementation, for an image, the image is patchified into a sequence of tokens and forwarded through a single linear layer to obtain the same hidden size as text tokens. The token generation neural network 302 in the example diagram has a single-tower architecture. A single-tower architecture can offer the following benefits: high training efficiency (single training cycle and task), reduced hyperparameters, and allows the model to flexibly allocate model complexity to different modalities. The latter can be significant in terms of scale. For example, with a unified architecture, only a single model is scaled, and therefore scaling strategies such as those used for language modeling can be followed.

[0132] The diffusion model is conditioned on features representing the current output sequence of lexical units (which can include any combination of text lexical units, audio lexical units, or image lexical units).

[0133] In the implementation shown, by using a cross-attention layer (in Figure 3A Conditioning (illustrated as 313) is implemented by including normalized output features into each visual transformer (ViT) block. Features for cross-attention can be generated based on the summarizing multimodal lexical units described above (e.g., using the "boi" lexical unit and / or all preceding lexical units) generated by the lexical unit generation neural networks 102, 302 for generating summarizing multimodal lexical units. Optionally, this involves pooling features from the output features of preceding lexical units in the current output sequence to generate pooled features 319. In a further implementation, the combination of pooled features 319 and features derived from the "boi" lexical unit can be used to generate summarizing multimodal lexical units for conditionalizing image generation.

[0134] Generally, U-ViT can include a set of ResNet blocks and a set of Transformer blocks in the downlink path, the uplink path that matches the downlink path, and the intermediate path, which includes another set of Transformer blocks. That is, U-ViT can include one or more ResNet blocks (He et al., “Deep residual learning for image recognition”, Proc. IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016) and one or more self-attention layers.

[0135] In the specific example shown, the diffusion model comprises four ResNet blocks 315 split across two layers and three self-attention ViT blocks 317 split across two further layers. The ResNet blocks are positioned at the higher resolution layers (e.g., 128x128 and 64x64), while the self-attention ViT blocks are split across two lower resolution layers (e.g., 32x32 and 16x16).

[0136] In some implementations, ResNet blocks in the downlink and uplink paths can be concatenated to preserve high-resolution information that might otherwise be lost at lower-resolution layers. This is in... Figure 3A The middle part is marked with an arrow labeled "Jump (connection)".

[0137] Generally, a diffusion model can be used to generate an output image 312 from a diffusion input (e.g., an initial noisy image 311). In some implementations, the diffusion neural network may include a convolutional neural network, such as U-Net, or other architectures that map an input of a given dimension to an output of the same dimension. In some implementations, the diffusion neural network may additionally or alternatively include a Transformer neural network that processes the input through a set of self-attention layers to generate the output.

[0138] At each of one or more update iterations, the diffusion model processes the diffusion input for that update iteration, including the current noisy data item for that update iteration, to generate a denoised output. At the first time step, the current noisy data item is the initial noisy data item. At each update iteration, the current data item up to that update iteration is updated using the denoised output generated by the diffusion neural network, thereby generating an updated current data item. The current noisy data item corresponds to the updated noisy data item generated in the previous iteration. In this way, the diffusion neural network is used to perform a reverse diffusion process across one or more update iterations to generate output data items.

[0139] A trained diffusion neural network can process the diffusion input for that update iteration, including the current data item (up to that update iteration), at any given update iteration to generate a denoised output for that update iteration. In some implementations, the denoised output is an estimate of the noise component of the current data item (i.e., the noise that needs to be combined with the output data item to generate the current data item). In other implementations, the denoised output is an estimate of the output data item given the current data item, i.e., an estimate of the data item that would be generated by removing the noise component of the current data item.

[0140] It will be understood that diffusion models can have any suitable architecture that allows a neural network to map a diffused input, including noisy data items, to a denoised output. For example, a diffusion model can have one or more further layers, including one or more transformer neural networks and / or convolutional neural networks.

[0141] exist Figure 3A In the implementation shown, the diffusion model is configured as a conditional model to generate a denoised output conditioned on a conditional input. As mentioned above, the conditional input includes features representing the current output sequence of multimodal lexical units obtained from the lexical generation neural network (and including the "boi" lexical unit). That is, the diffusion model is configured to generate an output image 112 with features characterized by the conditional input.

[0142] More generally, conditional input can include or represent one or more different types of input in one or more different modalities, such as any combination of text data elements, audio data elements, and image data elements. In some implementations, conditional input can include one or more images or other sensor data captured from the real-world environment.

[0143] The diffusion model neural network can be conditional on the input (e.g., features representing the current output sequence 310 of multimodal lexical units obtained from the lexical generation neural network 302) in any of a variety of ways.

[0144] For example, noisy data items can be concatenated with or otherwise combined with a conditional input and processed by the input layer of a diffusion model neural network. For instance, the diffusion input can include multiple channels, where the initial values ​​of one or more channels are taken from the conditional input, and the initial values ​​of the remaining one or more channels are noisy data items. The diffusion input can be generated by concatenating the noisy data items with the conditional input along the channel dimension. Alternatively, the diffusion input can be generated by including one or more conditional embeddings generated from the conditional input at some locations in the diffusion input and including one or more embeddings from the noisy data items at the remaining locations.

[0145] Conditioned inputs can be taken as inputs to one or more intermediate layers or the final layer of a diffusion model neural network, for example. The conditional inputs can be combined with the outputs of one or more layers of the diffusion model neural network, and the result is processed by subsequent layers of the diffusion model neural network.

[0146] For example, conditional inputs can be combined by one or more cross-attention layers of the model, such as... Figure 3A As shown. That is, a diffusion model can include one or more cross-attention layers, each of which cross-attentions one or more embeddings. Each of the one or more cross-attention neural network layers can be similar to a self-attention neural network layer, but in which the query is derived from one embedding, and the keys and values ​​are derived from different embeddings. For example, a query can be obtained from features generated by U-Net, and keys and values ​​can be obtained from features of the current output sequence. As used in this specification, an embedding is an ordered set of numerical values, such as a vector of floating-point values ​​or other types of values.

[0147] The diffusion model input at any given update iteration can also include data defining the noise level for that iteration. Typically, each update iteration has a corresponding time step t, and the noise level for that iteration depends on that time step. For example, the noise level can be a decreasing function of time step t. Examples of such functions include linear functions, cosine functions, and sigmoid functions. In these cases, the data identifying the noise level, time step, or both can be embedded using an appropriate neural network (e.g., a multilayer perceptron (MLP)) and used to condition the diffusion neural network, as described above, for example, for conditional input.

[0148] In some implementations, noisy data items can be initialized; that is, a first instance of a noisy data item can be generated by sampling the value of each element in the data item from a corresponding noise distribution (e.g., a Gaussian distribution or a different noise distribution). For example, an initial noisy data item can be generated by sampling the initial value of each of the multiple embeddings included in the initial noisy data item from a corresponding noise distribution (e.g., a Gaussian distribution or another predetermined distribution). Thus, the initial noisy data item includes multiple embeddings, where the initial value of each embedding is sampled from the corresponding noise distribution.

[0149] When configured as a conditional model, the diffusion model input for the first update iteration includes an initial noisy data item and a conditional input, and may further include data defining the noise level for that iteration. The output data item is then generated by updating the noisy data item at each of the multiple update iterations. In other words, the output data item is the data item following the last iteration of the multiple update iterations. The noisy data item at the first update iteration is the initial noisy data item.

[0150] When configured as a conditional model, the diffusion model input for each subsequent update iteration includes the current noisy data item and the conditional input, and may further include data defining the noise level for that iteration. An updated current noisy data item is generated in each update iteration. The current noisy data item corresponds to the updated noisy data item generated in the previous iteration.

[0151] At each update iteration, the denoised output generated by the diffusion neural network is used to update the current data item up to that update iteration, thereby generating the updated current data item. For example, the system can use the denoised output to determine an initial estimate of the output data item and then apply an appropriate diffusion sampler to the initial estimate to update the current data item. For example, when the denoised output is a prediction of the output data item, it can be used directly as the initial estimate. When the denoised output is a prediction of a noise component, the initial estimate can be determined from the current data item, the denoised output, and the noise level for the current update iteration. Any suitable diffusion sampler (e.g., a DDPM (Denoising Diffusion Probability Model) sampler, a DDIM (Denoising Diffusion Implicit Model) sampler, or another suitable sampler) can be used to update the current noisy data item with an estimate to generate the updated current data item. DDPM is discussed, for example, in Ho et al.'s arXiv:2006:11239.

[0152] After the last update iteration, the updated current data item can be taken as the output data item. Alternatively, after the last iteration, the initial estimate can be taken directly as the updated current data item (without using a sampler).

[0153] In some implementations, the number of update iterations is fixed. In others, the number of iterations can be adjusted based on the latency requirements for generating the output data items; that is, the number of iterations is chosen such that the output data items will be generated to meet the latency requirements. In still other cases, the number of iterations can be determined based on the computational resource consumption requirements for generating the output data items. For example, this requirement could be the maximum number of floating-point operations (FLOPS) to be performed as part of generating the final output data items.

[0154] As described above, the reverse diffusion process is performed across update iterations by updating the current data item at each iteration. Each update iteration corresponds to a different point in time within a time interval (e.g., an interval between zero and one, or another suitable time interval). This point in time is also referred to as the time step t or time index t. For example, update iterations can be evenly spaced across the time interval, i.e., evenly spaced at regular intervals within the interval, or they can be arranged within the time interval according to different schemes.

[0155] For the first update iteration, the current data item is the noisy initial data item. For each subsequent update iteration, the current data item is the data item updated in the previous update iteration, that is, the updated current data item from the previous update iteration.

[0156] As described above, the noise component of the current data item is added to the output data item to generate the current data item. For example, at an iteration with time index t (i.e., the time point (“time step”) corresponding to the update iteration is t), the current data item x t It can be represented as ,in α is the noise component, and x0 is the output data item. α can be determined based on a pre-defined schedule (e.g., linear, quadratic, cosine, etc.) across time index t. t and σ t In one example, α t = and σ t It can be a value between zero and one, where the value is taken from a pre-determined schedule table across time indices.

[0157] For example, at the update iteration corresponding to the inverse diffusion time index t, the current noisy data item x is updated based on the denoised output for that update iteration. t The current noisy data item after being updated will be referred to as the updated noisy data item x. t-1 The updated current data item for the final update iteration can be taken as the output data item x0. At any given update iteration, the current noisy data item provided as part of the diffusion input will be the updated current noisy data item already generated in the immediately preceding update iteration. For the first iteration, the current noisy data item is the initial noisy data item. At each update iteration, the current data item up to that update iteration is updated using the denoised output generated by the diffusion neural network. For example, when the denoised output is a prediction of the output data item, in order to generate the updated noisy data item x0... t-1 The denoised output can be projected onto the noise level corresponding to the time index t-1.

[0158] For example, at each iteration except the last iteration, a denoised output and a diffusion sampler (used to generate the initial estimate of the output data item) are used to generate an updated estimate of the current noisy data item. The system can use any suitable diffusion sampler (e.g., a DDPM (Denoising Diffusion Probability Model) sampler, a DDIM (Denoising Diffusion Implicit Model) sampler, or another suitable sampler) to update the current noisy data item with an estimate to generate the updated current data item. DDPM is discussed, for example, in Ho et al.'s arXiv:2006:11239. For the last iteration, the estimate can be either the initial estimate generated using the denoised output or generated using the sampler. When the denoised output is an estimate of the noise component, the initial estimate of the output data item can be determined as (x...t - σ t 𝜀̂) / α t , where ᝜�̂ is the denoised output. When the denoised output is an estimate of the output data item, it can be used as the initial estimate.

[0159] Some implementations utilize guidance when performing the backdiffusion process. That is, the backdiffusion process is sometimes a guided backdiffusion process. For example, it can be guided by a classifier or without classifier guidance. Classifier-free guidance is described, for example, in arXiv:2207.12598 by Ho and Salimans.

[0160] A diffusion neural network can be trained on a set of training data items using a denoising score matching objective (an example of an image reconstruction objective) to generate a denoised output. The denoising score matching objective can measure the error between (i) the denoised output and (ii) the target denoised output, such as mean squared error, L1 error, L2 error, or different types of error, generated by processing a diffusion input that includes noisy data items generated by adding sampled noise to the training data items, the target denoised output being generated from the training data items, from the sampled noise, or both. For example, when the denoised output is an estimate of the noise component of the current data item, the target denoised output can be the sampled noise. As another example, when the denoised output is an estimate of the target data item, the target denoised output can be the target data item.

[0161] Specifically, to train a diffusion neural network using a score-matching objective, the training method may include: (i) sampling data items from a set of training data items; (ii) sampling one or more corresponding conditional inputs for that data item; (iii) sampling at training time steps t, for example, sampling randomly and uniformly from a time interval or according to different distributions over the time interval; and (iv) sampling noise from a noise distribution (which may be, for example, a Gaussian noise distribution). Sampling is performed. Then, the system can compare the target data item x0 with the sampled noise based on the sampled time step t. Combining to generate noisy data item x t For example, by setting noisy data items Then, the system can use a diffuse neural network to process data including noisy data items x. tThe system takes data at specified time steps and conditional inputs as inputs to generate a denoised output. The system can then calculate the error between the denoised output and the target denoised output, and use this error to train a diffusion neural network, for example, by determining the gradient of the error and then using the gradient to update the parameters of the diffusion neural network by applying an optimizer to (at least) the gradient. As a specific example, a denoising score matching target can measure (i) the error between the denoised output and (ii) the target denoised output, such as mean squared error, L1 error, L2 error, or different types of error.

[0162] Figure 4A and Figure 4B Example inference results associated with system 100 are shown in the figure.

[0163] exist Figure 4A The text perplexity evaluation results for the following three models are shown: Large Language Model (LLM), Transformer model, and Transformer with diffusion model (e.g., System 100). The Transformer with diffusion model can be essentially as described above, for example in... Figure 3A Each model was trained by uniformly sampling training batches from the M3W and C4 datasets. For LLM, all image terms were removed during training, so that LLM was trained to input and output only text terms. The transformer model was trained to input sequences of images and text terms and output sequences of images and text terms. For both the transformer model and the transformer + diffusion model, no loss was applied to the image terms (i.e., they are either raw or unprocessed patches of image pixels). All models were trained using the same batch size, number of training iterations, and number of terms. Note that LLM actually sees more text terms than the multimodal terminology due to its image terminology removal, by 12% in this example.

[0164] As from Figure 4A As can be seen, although the evaluation results of the transformer and diffusion model are worse than those of the LLM-only model (which can be explained by the fact that the LLM is trained on more text lexical units), the transformer and diffusion model does show an improvement over the transformer-only model. This indicates that the transformer and diffusion model described in this paper can facilitate positive transfer of performance.

[0165] Figure 4BThe diagram shows a series of images (IMG1, IMG2, IMG3, IMG4) generated by system 100 based on the initial prompt sequence “a house next to an empty road. A lot of snow is on the roof.” Then, further prompt sequences (e.g., “the backsideview of the house,” “side perspective,” and “close-up shot of the red shed”) are input as further prompt sequences, causing the system to generate further corresponding images (i.e., IMG2, IMG3, IMG4). As shown, each subsequently generated image is substantially consistent with both the previously generated image and the previously input text prompt.

[0166] Figure 5 A method 500 for training a system 100 to generate multimodal data, implemented by a computer program on one or more computers at one or more locations, is illustrated. The system includes a word-generating neural network 102 and an image generation subsystem 104. In a specific implementation, the word-generating neural network is a transformer neural network, as described above, and / or the image generation subsystem is a diffusion model generation subsystem, as described above.

[0167] In step 502, the method includes: obtaining a set of multimodal training examples. Each training example may include two or more different data types: text data, audio data (such as speech), or image data. Generally, training examples include text or audio data and at least one training image (which may in turn include audio data). The multimodal training examples represent training input sequences for the system.

[0168] In some implementations, obtaining a set of multimodal training examples may include: obtaining training examples comprising i) text or first audio data and ii) second audio data. The text or first audio data and the second audio data may be semantically related. The method may further include: converting the second audio data of each training example into an audio spectrogram to obtain a training image for that training example.

[0169] Generally, the system can be trained based on image reconstruction objectives and lexical prediction objectives. The image reconstruction objective can depend on the image predicted by the system and the training image (which the system is intended to predict). The lexical prediction objective can depend on the predicted next multimodal lexical (e.g., the next text lexical in the output sequence) and the next ground truth multimodal lexical (which the predicted multimodal lexical is intended to predict). As described in more detail below, training can involve teacher-forcing methods.

[0170] In some implementations, the system is trained end-to-end based on a combined objective of image reconstruction and word prediction. Specifically, the combined objective involves adding the image reconstruction objective and the word prediction objective together.

[0171] In step 504, the method includes generating a sequence of ground truth multimodal lexical units corresponding to each training example in a set of multimodal training examples. Each ground truth multimodal lexical unit may represent training data (e.g., text or audio data and at least one training image) for the corresponding training example. The sequence of ground truth multimodal lexical units includes an image start (“boi”) lexical unit in the sequence, which is positioned in the sequence before the first ground truth multimodal lexical unit representing the training image in the sequence of multimodal lexical units. Generally, the training data may include examples of interleaved image-text sequences, or audio-text sequences, or combinations thereof. An example is the WebLI dataset, Web Language Image, arXiv:2305.18565v1 by Chen et al. (which has various versions with different sizes).

[0172] In some implementations, generating the corresponding sequence of ground truth multimodal terms involves inserting the image-starting "boi" terminator into the sequence before the ground truth multimodal terminator representing the training image. In other implementations, the image-starting terminator may already exist.

[0173] In some implementations, generating ground truth multimodal terms representing training images includes: processing the training image using a block encoder to convert the pixels of the training image into a sequence of image terms, each image term including a block encoding of the values ​​of pixels in different regions of the training image, which maps a set of pixel values ​​to a corresponding image term. That is, each image term in the sequence may include a block encoding of the values ​​of pixels in different regions of the training image, which maps a set of pixel values ​​to a corresponding image term.

[0174] In some implementations, one or more ground truth multimodal terms representing a training image are “raw” (i.e., unprocessed) pixel patches of the image. These “raw” pixel patches can be grouped together to form the training image.

[0175] In some implementations, the block encoder encodes each region or block of the image independently of other regions (e.g., blocks). Each block or region of the image that is mapped into image terms or encoded into image terms can have a fixed size or dimension. Therefore, in some implementations, the number of generated image terms is variable and depends on the size of the training image. That is, the larger the resolution of the image, the more terms are encoded.

[0176] In some implementations, the block encoder includes a linear image region coding neural network. The method may include using the linear image region coding neural network to linearly project a set of pixel values ​​for each region of a training image onto corresponding image terms. Training may include backpropagating gradients into the linear image region coding neural network to update its weights. The block encoder may then be associated with multiple learnable parameters.

[0177] In some implementations, any given ground truth image lexical generated does not depend on any previously generated image lexical (e.g., the same image or a different image generated by the image generation subsystem).

[0178] In some implementations, the encoding can be consistent or deterministic, such as constant, because the same set of values ​​for pixels in any given training image region is mapped to the same corresponding image lexical.

[0179] In some implementations, the block encoder is a differentiable block encoder, meaning it has learnable parameters that can be updated using a gradient descent optimization algorithm. It can be implemented, for example, using a neural network.

[0180] In step 506, for each sequence of ground truth multimodal lexical units generated in step 504, multiple training sequences are determined. That is, for any given training example, there are multiple training sequences. Each of the multiple training sequences includes a subsequence of the sequence of ground truth multimodal lexical units.

[0181] For example, if the sequence of ground truth multimodal lexical units includes the lexical unit "A, A, B, A", then multiple training sequences can include the sequences "A", "A, A", and "A, A, B". The system is then teacher-forced trained to predict the next multimodal lexical unit in each of these training sequences. That is, predict "A" for the subsequence "A", predict "B" for the subsequence "A, A", and predict "A" for the subsequence "A, A, B".

[0182] In step 508, for each training sequence determined in step 506, it is determined whether the last (or final) ground truth multimodal term in the sequence is the "boi" term. If the determination is affirmative, the method proceeds to step 510. If the determination is negative, the method proceeds to step 512.

[0183] In step 510, the method further includes generating a predicted image conditioned on features representing the subsequence. Step 510 can be performed by the image generation subsystem 104. That is, the method includes generating a predicted image using the image generation subsystem in response to the last ground truth multimodal term in the subsequence being the term "boi" (or conditioned on it).

[0184] In some implementations, using an image generation subsystem to generate a predicted image conditioned on features representing a subsequence includes: using a word generation neural network to process a training sequence including the image start word "boi" to generate features of a summary multimodal word; and using the image generation subsystem to generate a predicted image conditioned on features of the summary multimodal word.

[0185] In a specific implementation, the features used to conditionally condition the image generation subsystem are determined from the output features of the "boi" lexicon or otherwise obtained, which are used to generate the summarizing multimodal lexicon. The "boi" lexicon is a convenient choice for generating the summarizing multimodal lexicon because, as the lexicon generation neural network autoregressively generates successive lexicons, the "boi" lexicon already represents or provides a summary of all preceding lexicons generated by the lexicon generation neural network, and because the "boi" lexicon has no target text lexicons to predict.

[0186] Alternative implementations for generating summarizing multimodal lexical units are envisioned, such as using all preceding lexical units generated by the lexical unit generation neural network 102 (or all preceding lexical units before the sentence start lexical unit) to generate summarizing multimodal lexical units (instead of just the "boi" lexical unit). This could involve pooling features from the output features of preceding lexical units in the current output sequence. In a further implementation, a combination of the pooled features and the features derived from the "boi" lexical unit can be used to generate summarizing multimodal lexical units for conditionalizing image generation.

[0187] In some implementations, conditionalization performed to represent features of a subsequence can be done, for example, without classifier guidance (when the image generation neural network can be trained with or without conditionalization (e.g., with conditionalization randomly omitted)).

[0188] In step 512, the method further includes predicting the next multimodal lexical (e.g., a text lexical). Step 512 can be performed by the lexical generation neural network 102.

[0189] In step 514, the system is trained based on an image reconstruction objective and a lexical prediction objective. The image reconstruction objective may depend on the image predicted by the system and the training image (which the image is intended to predict). The lexical prediction objective may depend on the predicted next multimodal lexical (e.g., the next text lexical in the output sequence) and the next ground truth multimodal lexical (which the predicted multimodal lexical is intended to predict).

[0190] In some implementations, step 514 involves updating the learnable parameters of the word generation neural network and the image generation neural network. In some implementations, step 514 also involves updating the learnable parameters of the block encoder (if this block encoder is implemented as a neural network).

[0191] Lexical prediction objectives can include self-supervised objectives. Many different types of self-supervised objective functions exist. As an example, the system can be trained using softmax cross-entropy loss, such as teacher-coerced training with softmax cross-entropy loss. As another example, the system can be trained using autoregressive negative log-likelihood (NLL) loss, such as... For a multimodal input consisting of an input sequence encoded as L tokens, where the Lth token is... Multimodal lexical Conditioned by the preceding multimodal lexical unit As another example, the system can be trained using masking loss, for instance, when the system needs to predict the loss of masked data (such as masked words). As yet another example, the system can be trained using a self-supervised objective function that includes a contrastive loss function (a contrastive loss function that depends on positive examples and one or more negative examples).

[0192] There are many image reconstruction objectives that can be used, such as mean squared error objectives, or, in some implementations, diffusion model objectives as described later.

[0193] As described above, in some implementations, training is performed using batches of interleaved text and image sequences. Since the number of images per sequence can vary significantly across sequences, one image is randomly selected per sequence / batch element to train the image generation neural network (e.g., a diffusion model). For conditionalization, all words following the image start word in the image (including patches) are masked. Training is performed using a teacher-forced method, while maintaining a consistent batch size for the diffusion model. Training is performed end-to-end on these samples by summing the image diffusion and the average negative log-likelihood of the text words. These two losses can be weighted. The training was found to be quite robust to the weighting of these two losses, showing stable results across a weighting range of 1 to 10.

[0194] In some implementations, the image generation subsystem is trained to generate images based on learned representations output by the word generator neural network. That is, the image generation subsystem learns to generate images using conditionalizations generated by the word generator neural network. Since the image generation subsystem is trained to better predict ground truth multimodal words (which depend on any previous image words), the word generator neural network and the image generation subsystem can be jointly improved.

[0195] In some implementations, this training involves backpropagating the gradient of the image reconstruction target from the image generation subsystem to the word generation neural network. That is, the image generation subsystem and the word generation neural network are trained "jointly." If the image generation subsystem includes a diffusion model, this diffusion model can be derived from a sequence of image generation steps or from a random selection of t steps.

[0196] In some implementations, the training includes backpropagating the gradients of the image reconstruction target and the gradients of the word prediction target through a word generation neural network to the block encoder to update the learnable parameters of the block encoder (if the block encoder is implemented as a neural network).

[0197] As already described, the lexical generation neural network may include one or more self-attention neural network layers. During training, while the self-attention neural network layer is processing multimodal lexical units (e.g., when predicting the next multimodal lexical unit in step 514), a causal mask may be applied to the self-attention neural network layer.

[0198] In some implementations, while the self-attention neural network layer is processing multimodal lexical units (e.g., when predicting the next text or audio lexical unit), the lexical generation neural network can apply a causal mask to the self-attention neural network layer, that is, so that at each time step, the self-attention neural network layer only sees past inputs in the sequence of processed inputs.

[0199] In some implementations, images are generated by a separate model (e.g., image generation subsystem 104, which, as noted above, may include a diffusion model). Since the diffusion model generates images (rather than individual image terms) and all image terms associated with an image can be passed to the term generation neural network in parallel, bidirectional attention can be used during training while the self-attention neural network layer is processing image terms; that is, there is no causal attention mask.

[0200] In some implementations, the image generation subsystem can be a diffusion model image generation subsystem, such as... Figure 3A As shown. In such an implementation, using an image generation subsystem to generate a predicted image may include: sampling a time step from a distribution; generating a noisy version of the training image by adding noise to the training image at a level determined by the time step, the added noise being defined as a noisy image added to the training image; and using an image generation neural network to process the noisy version of the training image and the time step to generate the predicted image.

[0201] The predicted image can represent a noisy image. The image reconstruction objective can depend on the differences (e.g., mean squared error) between the predicted image and the noisy image (or patches thereof). The predicted image can represent a reconstructed version of the training image. The image reconstruction objective can depend on the differences between the predicted image and the training image.

[0202] Generally, any diffusion model loss can be used. The noise to be added for a given time step can be determined based on a predefined noise schedule.

[0203] In some implementations, the diffusion model can be a latent variable diffusion model. That is, the method can involve generating a noisy version of a training image or a noisy latent vector representation of a training image by adding noise to the training image or the latent vector representation of the training image at a level determined by a time step, where the added noise is defined as a noisy image added to the training image, or a noisy latent vector added to the latent vector representation of the training image. The method can also involve using an image generation neural network to process the noisy version of the training image or the noisy latent vector representation of the training image along with a time step to generate a predicted image or a latent vector representation of the predicted image. The predicted image can represent a noisy image or a reconstructed version of the training image; or a noisy latent vector or a reconstructed version of the latent vector representation of the training image. The image reconstruction objective can accordingly depend on the difference between the predicted image and the noisy image, or the difference between the latent vector representation of the predicted image and the noisy latent vector; or it can depend on the difference between the predicted image and the training image, or the difference between the latent vector representation of the predicted image and the latent vector representation of the training image.

[0204] In some implementations, image generation neural networks 104 and 304 are as described above. Figure 3A As described above. That is, the image generation neural network 304 has a U-net architecture, such as described above. The U-net architecture has one or more cross-attention neural network layers. Using the image generation subsystem to generate predicted images conditioned on features representing subsequences may include: using one or more cross-attention neural network layers to focus on features of the subsequence obtained by processing the subsequence using a word-generating neural network.

[0205] In some implementations, the method includes: replacing the image generation subsystem comprising the image generation neural network with a second image generation subsystem comprising the second larger image generation neural network; freezing the learnable parameters of the word generation neural network (and optionally a block encoder if the block encoder includes learnable parameters); and training the second larger image generation neural network (e.g., which includes one or more layers in its U-Net architecture) using further multimodal training examples to update the learnable parameters of the second larger image generation neural network, each further training example comprising text or audio data and at least one training image. This can provide a computationally cheaper implementation where training is performed end-to-end using a smaller diffusion head, and a larger diffusion head is then trained on the frozen model.

[0206] In some implementations, the methods described above can be combined with pre-trained image generation neural networks and / or pre-trained word generation neural networks. For example, existing large-scale pre-trained LLM and diffusion models can be fine-tuned in this setting.

[0207] Now, an example task that can be achieved by system 100 (e.g., after it has been trained) will be described.

[0208] In some implementations, the method includes using the system to perform an audio generation task after training. In such implementations, the method may include using the system to process a cue sequence of input sequences that define multimodal lexical units for the system. The cue sequence may include text or audio defining the audio generation task. Time-series audio data of the audio waveform may define the audio specified by the cue sequence. The audio may include spoken words of natural language.

[0209] The method may include using the system after training to perform an image generation task or an image processing task. In such an implementation, the method may include using the system to process a cue sequence of input sequences that define multimodal lexical units for the system, wherein the cue sequence includes text or audio defining the image generation task or the image processing task, and wherein the image defines the result of the task.

[0210] Tasks may include generating an image specified by a prompt.

[0211] The prompt may include an image. The task may include generating a modified version of the image. The modifications to be performed may be described by the prompt.

[0212] The prompt may include an image. The task may include an optical character recognition task, which involves generating an output sequence of multimodal lemmas representing words or characters in an image.

[0213] The prompt may include an image. The task may include generating an output sequence of multimodal lexical units representing answers to questions about the image.

[0214] The prompt may include an image and identify one or more objects in the image. The task may include generating an output sequence of multimodal lemmas that define the presence, location, orientation, or count of one or more objects in the image.

[0215] Hints may include images. Tasks may include generating an output sequence of multimodal terms that describe the content of an image or classify the content of an image into one or more categories.

[0216] The prompt may include an image and define the goal of the mechanical agent acting in a real-world environment. The task may include generating an output sequence of multimodal lexical units that define one or more actions to be performed by the mechanical agent to achieve the goal. In some implementations, the system is configured to cause the mechanical agent to perform one or more actions defined by the output sequence of multimodal lexical units.

[0217] The method can be executed to perform an audio generation task before further training. The method can also be executed to perform an image generation or image processing task before further training. That is, further training (e.g., fine-tuning) can be performed following the training methods described herein.

[0218] Below are a few further examples of machine learning tasks that can be performed by the system 100 trained as described in this article. In the examples below, the task could also be an audio processing task (where appropriate), in the case of referring to an image processing task.

[0219] Generally, cue sequences can include text and / or audio, such as speech that defines (after training) the task to be performed by the system.

[0220] As an example, a task could include generating an image specified by a prompt. The prompt could specify the content of the image; that is, the prompt could include a description of the image to be generated, or (especially where the prompt includes a still or moving image) the prompt could specify that the image should depict a prediction of what will happen next.

[0221] As another example, the prompt may include an image, and the task may involve generating an edited or modified version of the image, where the edits or modifications to be performed are described by the prompt. Some example modifications include generating another perspective or view of the subject depicted in the image, such as a view from a different angle or a close-up or zoomed-out view; or a change in the style of the image; or a change in the context of the image (e.g., daytime <> nighttime; rain <> no rain); and so on. This can be used to progressively refine the image.

[0222] As another example, the prompt can include an image, and the task can involve generating an output sequence of multimodal lexical units representing answers to questions about the image. In other words, the prompt can define any visual question-answering task; this can involve reasoning about the content of an image.

[0223] Hints can define queries. For example, a system can be used to detect objects in a video frame and provide information related to the detected objects in response to a query. As another example, specifically in the case of a moving image, such queries may require predictive reasoning (“What will happen next?”), counterfactual reasoning (“What will happen under different circumstances?”), explanatory inference (“Why will something happen?”), or, in general, causal reasoning. Queries may include, for example, requests for predictions of future events or states related to one or more objects in the object (e.g., “Will objects X and Y collide?”), or requests for conditional or counterfactual information related to one or more objects in the object (e.g., “What event would [not] happen if object X is modified, moved, or absent?”), or requests to analyze a video frame to determine the properties or characteristics of one or more objects in the object (e.g., “How many objects of type Z are moving?”). The response can be, for example, a text answer, such as a yes / no answer, or it can define the position of an object, or be in some other format. This can be used to predict whether two objects will collide, or how to avoid such a collision. For example, the system can be used to, for example, provide warnings and / or control the movement of one or more objects.

[0224] As another example, the cue may include an image, and one or more objects in the image may be identified, for example, through a text description or in some other way. The task may involve generating an output sequence of multimodal terms that define the presence, location (e.g., bounding boxes), orientation, or count of one or more objects in the image. In the implementation of the system, the way the image is encoded (and generated) facilitates tasks involving processing fine, even pixel-level details of the image. For example, the image in the cue, or the image generated by the system, may include, for example, a segmentation mask that defines a portion of the task to be performed.

[0225] As another example, the cue may include an image, and the task may include generating an output sequence of multimodal lexical units that describe the image (e.g., an image captioning task) or classifying the image content into one or more categories. In the case that the image is a moving image, this may include action (e.g., gesture) recognition.

[0226] As another example, the prompt may include an image and define the goal of a mechanical agent acting in a real-world environment, and the task may involve generating a multimodal sequence of output lexical units that define one or more actions to be performed by the mechanical agent to achieve the goal (e.g., a task or subtask of the mechanical agent (e.g., a robot)).

[0227] Other examples of tasks that can be performed include: text-to-speech tasks, where the prompts include natural language text and the system generates corresponding audio for the speech; and speech-to-text (speech recognition) tasks, where the prompts include audio and the system generates corresponding natural language text.

[0228] Here are some further examples of the tasks.

[0229] As an example, the task may include object or action detection. Training data items may include images or videos containing one or more objects or actions, and sequences of text. The sequences of text may describe or otherwise annotate the objects or actions, and may include text that gives the bounding box coordinates of the objects or actions. After training, when the system is used for inference, the system output may include or represent text describing or otherwise annotating the detected objects or actions in the image input, and may include the bounding box coordinates of the detected objects or actions, such as "10 20 90 100 cat 20 30 100 100 dog".

[0230] As another example, the task can include a classification task, such as an object or action classification task. Training data items can include image or video items containing sequences of one or more objects or actions along with text. The text sequence can describe the object or action or otherwise classify that object or action. After training, when the system is used for inference, the system output can include data, such as text, classifying the objects or actions in the image input into one of several classes.

[0231] As another example, the task can include image or video description tasks, such as captioning tasks (as used herein, which include audio description tasks to explain what is happening in a video). Training data items can include images or videos and sequences of text describing those images or videos. After training, when the system is used for inference, the system output can include data describing the image or video, such as text. For example, the system output can provide captions or descriptions, or it can count objects in the image or video, or it can provide some other form of description.

[0232] As another example, the task could include an image or video question-answering task. Training data items could include images or videos and sequences of text describing those images or videos. After training, when the system uses this data for inference, the system output could include data (e.g., text) that answers questions about the input specified in the sequence of prompts about the text, as described above. This could be used, for example, to answer questions about visual drawings and charts or about sound.

[0233] As another example, the task can include character or word recognition tasks, such as OCR (Optical Character Recognition) tasks. Training data items can include images or videos and sequences of text, which include text depicted in the image or video or represented as speech in an audio item. After training, when the system is used for inference, the system output can include, for example, text representing characters or words in natural language.

[0234] As another example, the task can include still or moving image generation; Google DeepMindGemini is an example of a system capable of generating image output. As yet another example, an image, such as a drawing or chart, can be decoded from one or more (language) lexical units generated by the system. Training data items for such a system can include images or videos and sequences of text describing those images or videos. After training, when the system is used for inference, the system output can include image or video data, such as image data defining the pixel values ​​of still or moving images, and sequences of text input to the system's multimodal input that can describe or characterize the image or video to be generated.

[0235] As another example, the task may include a computer language text generation task. Training data items may include images or videos and sequences of text in the computer language used to generate those images or videos. After training, when the system is used for inference, the system output may include text in that computer language or another computer language used to generate or render images or videos (e.g., web pages, drawings, or charts).

[0236] In another example of a computer language text generation task, training data items may include images or videos and a sequence of computer language text used to perform tasks related to those images or videos. These tasks may involve, for example, data processing tasks that analyze the content of the images or videos to provide the results of the analysis, or searches for information related to the content of the images or videos. The computer language in the system output may include computer language used to invoke functions or call one or more external APIs. As just one example, such output could be formatted as a JSON object. As previously described, the sequence of text in the multimodal input can define the task to be performed and may include, for example, images or videos related to the task to be performed, tasks that involve manipulating a specific type of data that could benefit from accessing an API, such as mathematical data, date / time related data, scientific data, recent data that may be later than the system's training (accessible through search functions or APIs), etc. After training, when the system is used for inference, the system output may include text in that computer language or another computer language used to perform tasks related to the images or videos in the input (e.g., as described above). The method may then include: using the text in the computer language to perform the task.

[0237] Generally, when the system output includes text, this system output can be provided as speech representing the text.

[0238] In some implementations, the machine learning task includes an agent control task in which an agent interacts with the environment to perform an agent control task. In these implementations, multimodal inputs include observations characterizing the environment. For example, multimodal inputs may include a sequence of text defining a task to be performed by the agent, and images may represent observations of the environment, such as those captured from the real-world environment by a camera or other imaging device. Training data items may include a sequence of text representing one or more actions of the agent, and image observations of the environment. After training, when the system is used for inference, the system output includes an action selection output, such as text, for selecting one or more actions that the agent will perform in response to the observations. As an example, system output 122 may define the actions as text, such as “A: 132 114 128 5 25 156”, which can be converted into control signals for a mechanical agent (such as a robot), such as “…”. As another example, in addition to or instead, the action selection output can define one or more low-level skills, such as a vocabulary of previously learned skills. As before, the sequence of text input to the system can describe the task to be performed, such as "What action should the robot take to [perform task]".

[0239] In some implementations of agent control, the environment is a real-world environment, and the agent is a mechanical agent that interacts with the real-world environment, such as a robot or an autonomous or semi-autonomous land, air, or sea vehicle operating or navigating within it. Actions are performed by the mechanical agent in the real-world environment to execute tasks. For example, the agent may be a robot or other mechanical agent that interacts with the environment to complete a specific task (e.g., locating or manipulating an object of interest in the environment, moving the object of interest to a designated location in the environment, or navigating to a designated destination in the environment). In these implementations, observations may include one or more of the following: images, object position data, and sensor data used to capture observations as the agent interacts with the environment. Actions may define control signals used to control the robot or other mechanical agent, such as position, torque, or other control signals for parts of the mechanical agent, or higher-level control commands.

[0240] In some implementations of intelligent agent control, the intelligent agent can be a human intelligent agent, and the environment can be a real-world environment. For example, the intelligent agent can be a human user of a digital assistant (such as a smart speaker, smart display, or some other device used to instruct the user to perform actions). The task can be any real-world task the user wishes to perform. Observations can be obtained from an observation capture subsystem (e.g., a monitoring system such as a video camera or sound capture system) used to capture visual observations of the user performing the task. Actions can include instructions in the form of, for example, text, images, video, or audio data (such as speech) guiding the user to perform the task.

[0241] Many large training datasets exist, such as the M3W (MultiModal MassiveWeb) dataset or the C4 (Colossal Clean Crawled Corpus) dataset, or WebLI (Web Language Image, arXiv:2305.18565v1 by Chen et al.); and / or training data can be obtained from web pages. For example, a more recent version of the MultiModalMassive Web dataset. Text and images can be linearized based on location to create interwoven data sequences of images and text. Several filtering steps can be applied, such as removing non-English documents. Documents lacking images may be discarded. Further text and image filtering heuristics can be applied to remove low-quality documents and reduce duplication. Images that fail to meet specific criteria can be removed, such as being too small (width or height less than 64 pixels), having a very large aspect ratio, or being of low quality (e.g., monochrome images). After these filters, documents without any images may also be discarded.

[0242] Alternatively, a smaller but more specialized training dataset can be used, for example, to train the system for one or more specific tasks. Several examples of visual tasks are: the Visual Genome dataset for visual question answering (arXiv:1602.07332 by Krishna et al.); Objects365 ("Objects365: A large-scale, high-quality dataset for object detection" by Shao et al., IEEE / CVF International Conference on Computer Vision, pp. 8430-8439); Open Images V4 (arXiv:1811.00982 by Kuznetsova et al.); the SBU dataset ("Im2Text: Describing Images Using 1 Million Captioned Photographs" by Ordonez et al., NeurIPS 2011); and the Conceptual Captions dataset, such as V1 (2 million images) or V2 (10 million images). (Sharma et al., “Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning,” ACL 2018); and for video, Kinetics (Kay et al., arXiv:1705.06950). An example task-specific training dataset for audio data is AudioSet (Gemmeke et al., “Audio set: An ontology and human-labeled dataset for audio events,” ICASSP, IEEE, 2017, pp. 776-780). An example task-specific training dataset for agent (robot) control is described in Ebert et al., arXiv:2109.13396.

[0243] This specification uses the term "configuration" in conjunction with system and computer program components. For configuring one or more computer systems to perform a specific operation or action, it means that software, firmware, hardware, or a combination thereof are installed on the system that causes the system to perform that operation or action during operation. For configuring one or more computer programs to perform a specific operation or action, it means that one or more programs include instructions that, when executed by a data processing device, cause that device to perform that operation or action.

[0244] Embodiments of the subject matter and functional operation described in this specification may be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their equivalents), or in one or more of these. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or one or more of these. Alternatively or additionally, program instructions may be encoded on artificially generated propagation signals (e.g., machine-generated electrical, optical, or electromagnetic signals) that are generated to encode information for transmission to a suitable receiver device for execution by the data processing device.

[0245] The term "data processing device" refers to data processing hardware and encompasses all kinds of devices, apparatuses, and machines used for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The device may also be or further include special-purpose logic circuit systems, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the device may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof. Therefore, systems, artificial neural networks, or trained artificial neural networks as described herein can be implemented in hardware using electronic circuit systems, such as in a physical box. Similarly, computer code as described herein can be code used to emulate such hardware or code used in a hardware description language.

[0246] Computer programs (which may also be referred to or described as programs, software, software applications, apps, modules, software modules, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and they can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. Programs may, but do not necessarily, correspond to files in a file system and can be stored in a variety of ways. This includes being embedded within files containing other programs or data (e.g., scripts within a markup language document), residing in dedicated files, or distributed across multiple coordination files (e.g., files storing modules, subroutines, or code segments). Programs may be stored as a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordination files (e.g., files storing one or more modules, subroutines, or code segments). Computer programs can be deployed to execute (or perform) on a single computer or on multiple computers, whether located at a single site or distributed across multiple sites and interconnected via data communication networks. The specific implementation of a computer program can involve a combination of traditional programming languages ​​and specialized languages ​​or libraries designed for GPU programming or TPU, depending on the hardware platform chosen and the desired performance characteristics.

[0247] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in others, multiple engines may be installed and run on the same one or more computers. Engines are generally implemented as one or more software modules or components installed on one or more computers, which may be located at a single site or distributed across multiple locations. In some cases, one or more dedicated computers may be used for a specific engine, while in others, multiple engines may operate concurrently on the same one or more computers. Examples of engine functions within the context of AI and machine learning include data preprocessing and cleaning, feature engineering and extraction, model training and optimization, inference and prediction generation, and post-processing of results. The specific design and implementation of an engine will depend on the overall architecture and the distribution of computational tasks across various hardware components, including CPUs, GPUs, TPUs, and other specialized processors.

[0248] The processes and logic flows described in this specification can be executed by one or more programmable computers, which execute one or more computer programs to perform functions by manipulating input data and generating output. Additionally, graphics processing units (GPUs) and tensor processing units (TPUs) can be used to implement aspects of these processes and logic flows concurrently, thereby significantly accelerating execution. This approach offers significant advantages for computationally intensive tasks common in AI and machine learning applications, such as matrix multiplication, convolution, and other operations exhibiting high parallelism. By leveraging the parallel processing capabilities of GPUs and TPUs, significant speedup and efficiency gains can be achieved compared to relying solely on CPUs. Alternatively, or in combination with programmable computers and specialized processors, these processes and logic flows can also be implemented using dedicated logic circuit systems (e.g., FPGAs or ASICs), or through a combination of dedicated logic circuit systems and one or more programmable computers, to achieve even higher performance or energy efficiency in specific use cases.

[0249] A computer suitable for executing computer programs can be based on a general-purpose microprocessor or a special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The components of a computer are the central processing unit for executing or carrying out instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated into a special-purpose logic circuit system. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or will be operatively coupled to receive data from or transfer data to such one or more mass storage devices, or both. However, a computer need not have such devices. Furthermore, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0250] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0251] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a web page to a web browser in response to a request received from a web browser on the user's device. Furthermore, the computer can interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and receiving responsive messages from the user in response.

[0252] Data processing devices used to implement machine learning models may also include, for example, dedicated hardware accelerator units for handling the general and computationally intensive portions of machine learning training or production (i.e., inference) workloads.

[0253] Computers capable of executing computer programs can be based on general-purpose microprocessors, special-purpose microprocessors, or a combination of both. They can also utilize any other type of central processing unit (CPU). Additionally, graphics processing units (GPUs), tensor processing units (TPUs), and other machine learning accelerators can be employed to enhance performance, particularly for tasks involving artificial intelligence and machine learning. These accelerators typically work in conjunction with the CPU to handle specialized computations, while the CPU manages overall system operation and other tasks. Generally, the CPU receives instructions and data from read-only memory (ROM), random access memory (RAM), or both. The basic components of a computer include a CPU for executing instructions and one or more memory devices for storing instructions and data. The specific configuration of the processing unit and memory will depend on factors such as the complexity of the AI ​​model, the amount of data being processed, and the desired performance and latency requirements. Implementations can be carried out on a wide range of computing platforms, from small embedded devices with limited resources to large-scale data center systems with high-performance computing capabilities. Systems may include storage devices such as hard disks, SSDs, or flash memory for persistent data storage.

[0254] Computer-readable media suitable for storing computer program instructions and data encompass all forms of non-volatile memory, media, and storage devices. Examples include semiconductor memory devices such as read-only memory (ROM), solid-state drives (SSDs), and flash memory devices; hard disk drives (HDDs); optical media; and optical discs such as CDs, DVDs, and Blu-ray discs. The specific type of computer-readable media used will depend on factors such as the size of the data, access speed requirements, cost considerations, and the desired level of portability or durability.

[0255] To facilitate user interaction, embodiments of the subject matter described herein can be implemented on computing devices equipped with display devices (such as liquid crystal displays (LCDs) or organic light-emitting diode (OLED) displays) for presenting information to users. Input can be provided by the user through various means, including keyboards, touchscreens, voice commands, gesture recognition, or other input modalities, depending on the specific device and application. Additional input methods may include sound, voice, or tactile input, while feedback to the user may take the form of visual, auditory, or tactile feedback. Furthermore, the computer can interact with the user by exchanging documents with the user's device or application. This may involve sending web content or data in response to a request, or sending and receiving text messages or other forms of messages via mobile devices or messaging platforms. The selection of input and output modalities will depend on the specific application and the desired form of user interaction.

[0256] Machine learning models can be implemented and deployed using machine learning frameworks such as TensorFlow or JAX. These frameworks provide comprehensive tools and libraries that facilitate the development, training, and deployment of machine learning models.

[0257] Embodiments of the subject matter described in this specification can be implemented within a computing system comprising one or more components, depending on the specific application and requirements. These components may include: backend components, such as backend servers or cloud-based infrastructure; optional middleware components, such as middleware servers or application programming interfaces (APIs), to facilitate communication and data exchange; and frontend components, such as client devices having a user interface through which users can interact with the implemented subject matter, a web browser, or an app. For example, the described functionality may be implemented only on the client device (e.g., for on-device machine learning) or deployed as a combination of frontend and backend components for more complex applications. Where present, these components may be interconnected using any form or medium of digital data communication, such as communication networks like local area networks (LANs) or wide area networks (WANs), including the Internet. The specific system architecture and component selection will depend on factors such as the scale of the application, the need for real-time processing, data security requirements, and the desired user experience.

[0258] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components (e.g., as a data server), or middleware components (e.g., an application server), or front-end components (e.g., a client computer having a graphical user interface, web browser, or app that a user can interact with through an implementation of the subject matter described in this specification), or any combination of one or more such back-end components, middleware components, or front-end components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0259] Computing systems can include clients and servers. Clients and servers are typically geographically separated (e.g., remotely connected) and generally interact via a communication network. The specific type of network (such as a local area network (LAN), wide area network (WAN), or the Internet) will depend on the accessibility and scale of the application. The client-server relationship is established by computer programs running on the respective computers and designed to communicate with each other using appropriate protocols. These protocols may include HTTP, TCP / IP, or other specialized protocols, depending on the nature of the exchanged data and the system's security requirements. In some embodiments, the server transmits data or instructions to a user device, such as a computer, smartphone, or tablet, acting as a client. The client device can then process the received information, display the results to the user, and potentially send the data or feedback back to the server for further processing or storage. This allows for dynamic interaction between the user and the system, enabling a wide range of applications and functionalities. The client-server relationship arises from computer programs running on the respective computers that establish a client-server relationship between them. In some embodiments, the server transmits data (e.g., HTML pages) to the user device, for purposes such as displaying data to a user interacting with the device acting as a client and receiving user input from that user. Data generated on the user's device can be received from the device at the server, such as the results of user interactions.

[0260] While this specification contains numerous details of specific implementations, these details should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features that may be characteristic of particular embodiments of a particular invention. Certain features described in the context of individual embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, one or more features from a claimed combination may, in some cases, be removed from that combination, and the claimed combination may involve sub-combinations or variations thereof.

[0261] Similarly, although operations are depicted in the accompanying drawings and described in a specific order in the claims, this should not be construed as requiring such operations to be performed in the specific order shown or in sequential order, or requiring all illustrated operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0262] Specific embodiments of this subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require a specific order or sequence to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous.

Claims

1. A computer-implemented method of generating multi-modal data using a system comprising a token generation neural network and an image generation subsystem comprising an image generation neural network, the method comprising: receiving a prompt sequence defining an input sequence of multi-modal tokens, and processing the input sequence of multi-modal tokens using the token generation neural network to generate an output sequence of multi-modal tokens, wherein a multi-modal token represents a data element of one of a plurality of modalities; wherein generating the output sequence of multi-modal tokens comprises, for each successive position in the output sequence of multi-modal tokens, autoregressively: processing a combined sequence comprising the input sequence of multi-modal tokens and a current output sequence of multi-modal tokens using the token generation neural network to generate a next multi-modal token of the output sequence of multi-modal tokens, and appending the next multi-modal token to the back of the current output sequence of multi-modal tokens; the method further comprising, in response to the next multi-modal token being an image start token: generating an image using the image generation subsystem conditioned on features representing the current output sequence of multi-modal tokens obtained from the token generation neural network; processing the image to convert pixels of the image into a sequence of image tokens, each image token comprising a block encoding of values of the pixels in a different region of the image, the block encoding mapping a group of values of the pixels to a respective image token; and appending the sequence of image tokens to the back of the current output sequence of multi-modal tokens as a next multi-modal token in the output sequence of multi-modal tokens.

2. The method of claim 1, further comprising: processing the combined sequence continues after appending the sequence of image tokens to the back of the current output sequence of multi-modal tokens to generate further multi-modal tokens of the output sequence of multi-modal tokens.

3. The method of claim 1 or 2, wherein the token generating neural network comprises one or more self-attention neural network layers, the method further comprising: bidirectional attention is used when the self-attention neural network layer is processing the image tokens.

4. The method of any one of claims 1 to 3, wherein the input sequence of multi-modal tokens comprises multi-modal tokens representing textual, audio, or image data elements, and wherein the output sequence of multi-modal tokens comprises multi-modal tokens representing textual or audio data elements.

5. The method of any of claims 1 to 4, wherein processing the image to convert pixels of the image into a sequence of image tokens comprises: processing the image to divide the image into a set of regions that tile the image, and determining each image token as a linear projection of the values of the pixels in each respective region.

6. The method of any one of claims 1 to 5, wherein the image generation subsystem is a diffusion model image generation subsystem, and wherein generating the image comprises: initializing the image or a latent vector representation of the image by sampling values of the pixels of the image or the latent vector representation from a noise distribution; and At each time step in a sequence of time steps: determine an updated version of the image or the latent vector representation of the image by processing the time step and processing the image or the latent vector representation of the image at the time step, determine a noise-reduced version of the image or the latent vector representation of the image using the image generation neural network conditioned on the features representing the current output sequence of multi-modal tokens.

7. The method of any one of claims 1 to 6, wherein the image generation neural network has a U-net architecture comprising one or more cross-attention neural network layers, and wherein generating a predicted image using the image generation subsystem conditioned on features representing the current output sequence comprises: attending to features of the current output sequence obtained by processing the current output sequence using the token generation neural network using the one or more cross-attention neural network layers.

8. The method of any one of claims 1 to 7, comprising: processing the combined sequence comprising the image starting token using the token generation neural network to generate features of a summary multi-modal token; and wherein generating an image using the image generation subsystem conditioned on features of the current output sequence of multi-modal tokens obtained from the token generation neural network comprises generating the image using the image generation subsystem conditioned on the features of the summary multi-modal token.

9. The method of any one of claims 1 to 8, wherein the image comprises an audio spectrogram; the method further comprising: convert the audio spectrogram to time-series audio data of an audio waveform.

10. The method of claim 9, wherein the prompt sequence comprises text or audio defining an audio generation task; and wherein the time-series audio data of an audio waveform defines audio specified by the prompt sequence.

11. The method of any one of claims 1 to 8, wherein the prompt sequence comprises text or audio defining an image generation task or an image processing task; and wherein the image defines a result of the task; in particular wherein: i) the task comprises generating an image specified by a prompt; ii) the prompt comprises an image, and the task comprises generating a modified version of the image, wherein the modification to be performed is described by the prompt; iii) the prompt comprises an image, and the task is an optical character recognition task involving generating an output sequence of multi-modal tokens representing words or characters in the image; iv) the prompt comprises an image, and the task comprises generating an output sequence of multi-modal tokens representing an answer to a question about the image; v) the prompt comprises an image and identifies one or more objects in the image, and the task comprises generating an output sequence of multi-modal tokens defining a presence, location, orientation, or count of one or more of the objects in the image; vi) the prompt comprises an image and the task comprises generating an output sequence of multi-modal tokens that describe content of the image or classify content of the image into one or more categories of a plurality of categories; vii) the prompt comprises an image and defines a goal of a mechanical agent acting in a real-world environment, and the task comprises generating an output sequence of multi-modal tokens that define one or more actions to be performed by the mechanical agent to achieve the goal.

12. A computer-implemented method of training a system for generating multi-modal data items, the system comprising a token generation neural network, an image generation subsystem comprising an image generation neural network, and a block encoder, the method comprising: obtaining a set of multi-modal training examples, each training example comprising text or audio data and at least one training image, and for each training example: generating a corresponding sequence of ground-truth multi-modal tokens representing the text or audio data and the training image, wherein generating the ground-truth multi-modal tokens representing the training image comprises processing the training image using the block encoder to convert pixels of the training image into a sequence of image tokens, each image token comprising a block encoding of values of the pixels in a different region of the training image, the block encoding mapping a set of values of the pixels to a respective image token; and wherein the sequence of ground-truth multi-modal tokens includes an image start token preceding the ground-truth multi-modal tokens representing the training image; and determining a plurality of training sequences, each of the plurality of training sequences comprising a sub-sequence of the sequence of ground-truth multi-modal tokens; and for each training sequence: in response to determining that a last ground-truth multi-modal token in the sub-sequence is the image start token, generating a predicted image using the image generation subsystem conditioned on a feature representing the sub-sequence, and when the last ground-truth multi-modal token in the sub-sequence is not the image start token, processing the sub-sequence using the token generation neural network to generate a predicted next multi-modal token; and training the system based on the training examples using i) an image reconstruction objective that depends on the predicted image and the training image, and ii) a token prediction objective that depends on the predicted next multi-modal token and a next ground-truth multi-modal token following the sub-sequence, the training comprising updating learnable parameters of the token generation neural network and the image generation neural network.

13. The method of claim 12, wherein the multi-modal training examples comprise training examples in which the at least one training image follows the text or audio data and training examples in which the at least one training image precedes the text or audio data.

14. The method of claim 12 or 13, wherein the training comprises updating learnable parameters of the block encoder, and wherein the training comprises backpropagating gradients of the image reconstruction objective and the token prediction objective through the token generation neural network into the block encoder to update the learnable parameters of the block encoder.

15. The method of any one of claims 12 to 14, wherein the block encoder comprises a linear image region encoding neural network, the method comprising: projecting the set of values of the pixels of each region of the training image linearly to a corresponding image token using the linear image region encoding neural network; and wherein the training comprises backpropagating the gradients into the linear image region encoding neural network to update weights of the linear image region encoding neural network.

16. The method of any one of claims 12 to 15, wherein generating a predicted image using the image generation subsystem conditioned on features representing the subsequence comprises: processing the training sequence including the image start token using the token generation neural network to generate features of an abstract multimodal token; and generating the predicted image using the image generation subsystem conditioned on the features of the abstract multimodal token.

17. The method of any one of claims 12 to 16, wherein the image generation subsystem is a diffusion model image generation subsystem, wherein generating a predicted image using the image generation subsystem comprises: sampling a time step from a distribution; generating a noisy version of the training image by adding noise to the training image at a level determined by the time step, the added noise defining a noise image added to the training image; and processing the noisy version of the training image and the time step using the image generation neural network to generate the predicted image; and wherein the predicted image represents a reconstructed version of the training image or the noise image, and wherein the image reconstruction objective depends on a difference between the predicted image and the noise image, or a difference between the predicted image and the training image, respectively.

18. The method of any one of claims 12 to 17, wherein the image generation neural network has a U-net architecture comprising one or more cross-attention neural network layers, and wherein generating the predicted image using the image generation subsystem conditioned on features representing the subsequence comprises: attending to features of the subsequence obtained by processing the subsequence using the token generation neural network using the one or more cross-attention neural network layers.

19. The method of any one of claims 12 to 18, further comprising: replacing the image generation subsystem comprising the image generation neural network with a second image generation subsystem comprising a second, larger image generation neural network; freezing the learnable parameters of the token generation neural network and the block encoder; and training the second larger image generation neural network using further multi-modal training examples to update learnable parameters of the second larger image generation neural network, each further training example comprising text or audio data and at least one training image.

20. The method of any one of claims 12 to 19, wherein generating the corresponding sequence of ground truth multi-modal tokens comprises: inserting the image start token in the sequence prior to the ground truth multi-modal tokens representing the training image.

21. The method of any one of claims 12 to 20, wherein obtaining the set of multi-modal training examples comprises obtaining training examples comprising i) text or first audio data and ii) second audio data, wherein the text or first audio data and the second audio data are semantically related; the method further comprising: converting the second audio data of each training example into an audio spectrogram to obtain the training image for the training example.

22. The method of claim 21, further comprising: using the system to perform an audio generation task after the training, the method comprising: processing, using the system, a prompt sequence defining an input sequence of multi-modal tokens for the system, wherein the prompt sequence comprises text or audio defining the audio generation task; and wherein the time series audio data of the audio waveform defines audio specified by the prompt sequence.

23. The method of any one of claims 12 to 22, further comprising: using the system to perform an image generation task or an image processing task after the training, the method comprising: processing, using the system, a prompt sequence defining an input sequence of multi-modal tokens for the system, wherein the prompt sequence comprises text or audio defining an image generation task or an image processing task, and wherein the image defines a result of the task; in particular wherein: i) the task comprises generating an image specified by the prompt; ii) the prompt comprises an image, and the task comprises generating a modified version of the image, wherein the modification to be performed is described by the prompt; iii) the prompt comprises an image, and the task is an optical character recognition task involving generating an output sequence of multi-modal tokens representing words or characters in the image; iv) the prompt comprises an image, and the task comprises generating an output sequence of multi-modal tokens representing an answer to a question about the image; v) the prompt comprises an image and identifies one or more objects in the image, and the task comprises generating an output sequence of multi-modal tokens defining a presence, location, orientation or count of one or more of the objects in the image; vi) the prompt comprises an image, and the task comprises generating an output sequence of multi-modal tokens describing content of the image or classifying content of the image into one or more of a plurality of categories; vii) the prompt comprises an image and defines a goal of a mechanical agent acting in a real-world environment, and the task comprises generating an output sequence of multi-modal tokens defining one or more actions to be performed by the mechanical agent to achieve the goal.

24. The method of claim 22 or 23, performed prior to further training to perform the audio generation task or to perform the image generation task or the image processing task.

25. A system comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of the respective method of any one of claims 1-24.

26. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations of the respective method of any one of claims 1-24.

Images