Training image representation neural network using cross-modal interface

Abstract

To provide a method of training an image representation neural network that is configured to receive an input image and to process the input image to generate a representation of the input image as a set of text tokens from a vocabulary of text tokens, a system and a storage medium.SOLUTION: A training system acquires a set of one or more training images 102, generates a text representation 112, for each of the training images 102, using an image representation neural network 120, generates an output image 132 from the text representation 112, using a text-conditional image generation neural network 130, and trains the image representation neural network 120 based on a difference between the output image 132 and a ground truth output.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This specification relates to image processing using machine learning models. [Background technology]

[0002] As an example, a neural network is a machine learning model that employs one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to another layer in the network, such as the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current value of its respective set of weights. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Kudo et al., arXiv:1808.06226 [Non-patent document 2] Salimans and Ho arXiv:2202.00512, 2022, Section 4, Appendix D Summary of the Invention [Means for solving the problem]

[0004] This specification describes a system for training an image representation neural network implemented as a computer program on one or more computers at one or more locations and configured to receive an input image and process the input image to generate a representation of the input image as a set of text tokens from a vocabulary of text tokens.

[0005] Specifically, the system trains an image representation neural network by using text as a "cross-modal interface" between the output of the image representation neural network and a text-conditioned image generation neural network.

[0006] Once an image representation neural network is trained, the representations generated by the trained image representation neural network can be used for any of a variety of downstream tasks. For example, the representation of a new image generated by the downstream neural network can be provided as input to the downstream neural network, i.e., in place of the new image, without the downstream neural network being required to be capable of processing image data.

[0007] Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages.

[0008] This specification describes techniques for effectively training an image representation neural network that maps an input image to a set of text tokens that describe the input image. Specifically, the system uses text as a "latent space" during training by employing a text-to-image generative neural network, e.g., a diffusion neural network, to process the text generated by the image representation neural network to generate output that is used to train the image representation neural network. As a result, the latent text generated by the image representation neural network is an accurate and comprehensive description of the input image, albeit with potentially mixed semantic concepts, resulting in a "scrambled caption" of the input image. This allows the latent text to be used, after training, as an effective representation of the corresponding image for any of a variety of downstream tasks. Advantageously, this training requires no additional supervision beyond the image itself.

[0009] More specifically, recent text-to-image generative models excel at converting arbitrary rich text, e.g., dozens of words, into highly detailed images that closely follow the prompt. In other words, these generative models are capable of processing complex text into visually coherent outputs. By employing one of these text-to-image generative models as a decoder in an autoencoder framework that uses text as a latent representation during training (with an image representation neural network used as the encoder), the optimized image representation neural network can explore a wide latent space of text, unlocking the vast visual-linguistic knowledge encapsulated within the generative model and producing high-quality text representations.

[0010] Once trained, the representations produced by an image representation neural network can be used for any of a variety of downstream tasks. For example, the representations can be used to "inject" image content into models not trained for image processing, such as large language models (LLMs), without requiring additional retraining of these models.

[0011] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 illustrates an exemplary training system. [Figure 2] 1 is a flow diagram of an exemplary process for training an image representation neural network. [Figure 3] 1 is a flow diagram of an exemplary process for training an image representation neural network where the image generation neural network is a diffusion neural network. [Figure 4] FIG. 1 illustrates examples of downstream tasks that can be performed after training an image representation neural network. DETAILED DESCRIPTION OF THE INVENTION

[0013] Like reference numbers and designations in the various drawings indicate like elements.

[0014] 1 is a diagram of an exemplary training system 100. Training system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, in which the systems, components, and techniques described below may be implemented.

[0015] The system 100 trains an image representation neural network 120 configured to receive an input image 102 and process the input image to generate a representation 112 of the input image as a set of text tokens from a vocabulary of text tokens.

[0016] The tokens in the vocabulary can be any suitable text tokens, such as words, parts of words, punctuation marks, characters, bytes, and optionally numbers or other text symbols contained in a corpus of text, that represent elements of text in one or more natural languages. For example, system 100 can tokenize a given sequence of words and split the sequence into tokens from the vocabulary by applying a tokenizer, such as the SentencePiece tokenizer (Kudo et al., arXiv:1808.06226) or another tokenizer.

[0017] In general, system 100 trains neural network 120 to generate text output that accurately and comprehensively describes the content of input image 102, even when the output may contain mixed semantic concepts. As explained below, such properties make the representations generated by neural network 120 effective for a variety of downstream tasks.

[0018] The image representation neural network 120 may have any suitable architecture that enables the neural network 120 to map the image 102 to a representation 112 of the input image as a set of text tokens from a vocabulary of text tokens.

[0019] As a specific example, the neural network 120 may include an image backbone neural network configured to process the input image 102 to generate a feature representation of the input image, and an encoder neural network configured to process the feature representation of the input image to generate a representation 112 of the input image 102 as a set of text tokens from a vocabulary of text tokens.

[0020] In some of these examples, the image backbone neural network may have been pre-trained on an image representation learning task, and thus training the image representation neural network 120 may include training the encoder neural network while keeping the image backbone neural network fixed.

[0021] In other of these examples, system 100 may train both the encoder neural network and the image backbone neural network during training. For example, both the image backbone neural network and the encoder neural network may be trained from randomly initialized parameter values, or the image backbone neural network may be fine-tuned from pre-trained parameter values ​​and the encoder neural network may be trained from randomly initialized parameter values.

[0022] The image backbone neural network and the encoder neural network may each have any suitable architecture.

[0023] For example, the image backbone neural network may be a vision Transformer (ViT) or a convolutional neural network that processes an input image to generate a feature representation of the input image that includes multiple feature vectors that represent the input image.

[0024] As another example, the encoder neural network can implement attention pooling to map the feature representations to text tokens. In this example, the encoder neural network can have a respective learned query corresponding to each text token in representation 112, i.e., can maintain learned query vectors as part of training neural network 120, each of which corresponds to a different one of the text tokens in representation 112.

[0025] Additionally, in this example, the encoder neural network may include a sequence of self-attention layer blocks and an output layer block.

[0026] Each self-attention layer block is configured to update the learned queries conditional on the feature representations of the input image. For example, each self-attention layer block can include a self-attention layer that applies self-attention across the learned queries to update the learned queries, and a cross-attention layer that updates each learned query by performing cross-attention on the feature representations.

[0027] In this example, to process the feature representation of an input image to generate a representation of the input image, the encoder neural network may process the trained queries through a sequence of self-attention layer blocks, and then, after processing the trained queries through the sequence of self-attention layer blocks, process each trained query using an output layer block to generate corresponding text tokens in the representation. For example, the output layer block may include a linear neural network layer that projects each trained query to one of the distinct tokens in the vocabulary. As described below, during training, the system 100 may use “approximation” to approximate a discrete sampling of vocabulary tokens when gradients need to be backpropagated to the image representation neural network 120.

[0028] Once trained, the image representation neural network 120 can be used by the inference system 150 to perform downstream tasks.

[0029] For example, after training image representation neural network 120, inference system 150 can receive query input 152 for a downstream task. Query input 152 typically includes a query image 154 and, optionally, other data, such as one or more other images, one or more inputs of different modalities, such as text or audio.

[0030] The reasoning system 150 processes the query image 154 using the image representation neural network 120 to generate a representation 156 of the query image 154 as a set of text tokens.

[0031] The inference system 150 can then provide the representation 156 of the query image as input to a downstream neural network 160 configured to perform the downstream task.

[0032] Downstream neural network 160 may generally be any neural network configured to process inputs including text tokens from a vocabulary to generate outputs for downstream tasks.

[0033] For example, downstream neural network 160 may be a language model neural network, such as a large-scale language model (LLM) neural network, or a visual language model (VLM) neural network. The LLM may be, for example, a multimodal model that processes input containing tokens representing multiple different data modalities, or a unimodal model that processes input containing text tokens.

[0034] For example, the query input 152 may include a query image 154 and text, and the downstream neural network 160 may be an LLM. Thus, providing the representation 156 of the query image 154 as input to the downstream neural network 160 may include providing the representation 156 of the query image 154 and the text from the query input 152 as input to the LLM, rather than directly providing the query image 154 as part of the input. For example, the LLM may be trained on text-only data and therefore not be able to directly process image data input.

[0035] The downstream task performed by downstream neural network 160 may be any of a variety of tasks, for example, a multimodal dialog task, whereby the images are part of the dialog input submitted to the system by the user, and the output generated by the downstream neural network is a response that is displayed to the user.

[0036] Other examples of downstream tasks include multimodal zero-shot or few-shot learning tasks.

[0037] More specifically, by utilizing the trained representation neural network 120, and because the representations generated by the downstream neural network 120 are accurate and comprehensive descriptions of the corresponding images, the downstream neural network 160 can effectively perform multimodal tasks that require operating on images even when the downstream neural network 160 is not configured to process image data. Furthermore, there is no need to retrain the downstream neural network 160 to effectively perform the downstream tasks.

[0038] Examples of downstream tasks are described in more detail below with reference to FIG.

[0039] To train the image representation neural network 120, the system 100 uses a cross-modal interface, i.e., an interface that maps the text generated by the model to the image modality. As described below, this cross-modal interface is exploited using a text-conditioned image generation neural network 130.

[0040] The text-conditioned image generation neural network 130 is a neural network that receives input including conditional text and processes the input to generate an output that defines an output image 132, e.g., an output image described by the conditional text.

[0041] For example, the text-conditioned image generation neural network 130 may be a text-conditioned spreading neural network.

[0042] In this example, the output of the text-conditional diffusion neural network that defines the output image is the denoised output.

[0043] A text-conditional diffusion neural network is a neural network that can be used to perform a de-diffusion process to generate an output image from given conditional inputs, including text.

[0044] To perform the dediffusion process, the system initializes representations of the output image. For example, the system can sample each value in each representation from a noise distribution, such as a Gaussian distribution.

[0045] The system then uses a conditional diffusion neural network to update the representation in each of multiple dediffusion steps (also called "iterations" or "update iterations"). Each dediffusion step is associated with a noise level for the iteration. Typically, each update iteration has a corresponding time step t, and the noise level for the iteration varies depending on the time step. For example, the noise level can be a decreasing function of the time step t. Examples of such functions include linear, cosine, and sigmoid functions. Thus, early iterations are associated with higher noise levels, and later iterations are associated with lower noise levels, with the diffusion neural network gradually "denoising" the representation to generate the final representation.

[0046] As part of the update at any given step, the system generates a denoised output of the despreading step.

[0047] The system then uses the denoised output of the despreading step to update the representation of the output image.

[0048] For example, the system may map the denoised output to an initial updated representation and then apply a diffusion sampler, such as a Denoising Diffusion Probabilistic Model (DDPM) sampler, a Denoising Diffusion Implicit Model (DDIM) sampler, or another suitable sampler, to the initial updated representation to generate the updated representation.

[0049] Optionally, after the final despreading iteration, the system can forgo the use of the spreading sampler and instead use the initial updated representation as the updated representation.

[0050] The system processes the diffusion inputs of the de-diffusion step, including a representation of the output image and conditional inputs, using a diffusion neural network to generate a de-noised output, which may be used as a final de-noised output or may be combined with one or more other de-noised outputs, e.g., through classifier-free guidance, to generate a final de-noised output.

[0051] More specifically, the diffusion neural network may be any suitable diffusion neural network configured to receive an input including a current (noisy) representation of the image and a conditional input including text to generate a denoised output.

[0052] For example, the diffusion neural network may include a text encoder neural network configured to process a text input to generate an encoded representation of the text input, and an image diffusion neural network configured to generate an output image over multiple sampling steps conditioned on the encoded representation of the text input.

[0053] In this example, the vocabulary of text tokens is the input vocabulary of the text encoder neural network.

[0054] In some implementations, the diffusion neural network performs the diffusion process in pixel space, and thus the image ("representation") operated on and generated by the diffusion neural network has a value for each pixel that specifies a color value, e.g., an RGB value or another color encoding scheme.

[0055] Examples of such spreading neural networks include Imagen.

[0056] In some other implementations, the diffusion neural network performs the diffusion process in a latent space, e.g., a latent space that is lower dimensional than pixel space, i.e., the image ("representation") operated on by the diffusion neural network is a latent image, and the values ​​of the image's pixels are learned as latent values ​​rather than color values.

[0057] Examples of such diffusion neural networks include mobileDiffusion.

[0058] In these implementations, during training, the diffusion neural network can be associated with an image encoder to encode training images into a latent space, and after training, to generate new target images, the diffusion neural network can be associated with a decoder neural network that receives inputs including latent representations of images and decodes the latent representations to reconstruct the images. For example, both the encoder and decoder neural networks can be convolutional neural networks, self-attention neural networks, or can include both convolutional and self-attention layers.

[0059] The diffusion neural network may have any suitable architecture that enables the neural network to map a diffusion input comprising an input representation of a data item, and to map the input representation to a denoised output having the same dimensionality as the input representation.

[0060] For example, the diffusion neural network may be a convolutional neural network, such as a U-Net, or any other architecture that maps one input of a given dimension to an output of the same dimension.

[0061] As another example, the diffusion neural network may be a Transformer neural network that processes a diffused input through a set of self-attention layers to produce a denoised output.

[0062] As yet another example, a spreading neural network can include both convolutional and self-attention layers.

[0063] The diffusion neural network can be conditioned based on the conditioning input in any of a variety of ways.

[0064] As an example, a diffusion neural network can use an encoder neural network to generate one or more embeddings that represent the conditioning inputs, and the diffusion neural network can include one or more cross-attention layers that each cross-attend to one or more embeddings.

[0065] As another example, a diffusion neural network can include one or more other types of neural network layers that are conditional on one or more embeddings. Examples of such layers include Feature-wise Linear Modulation (FiLM) layers, layers with conditional gate activation functions, etc.

[0066] As another example, as described in more detail below, the output of the encoder when encoding the conditional inputs can be combined with features of a representation of the output image, e.g., through a weighted sum, and the combined features can be processed by the rest of the diffusion neural network.

[0067] An embedding, as used herein, is an ordered collection of numbers, for example a vector of floating point values ​​or other types of values.

[0068] As mentioned above, the conditional input typically includes text. For textual conditional inputs, the diffusion neural network can use a text encoder neural network, such as a Transformer neural network or a recurrent neural network, to generate a fixed or variable number of text embeddings that represent the conditional text.

[0069] The diffusion input for any given update iteration may also include data defining the noise level of the iteration. Typically, each update iteration has a corresponding time step t, and the noise level of the iteration varies depending on the time step. For example, the noise level may be a decreasing function of the time step t. Examples of such functions include linear, cosine, and sigmoid functions. In such cases, data identifying the noise level, the time step, or both may be embedded using an appropriate neural network, such as a multi-layer perceptron (MLP), and used to condition the diffusion neural network as described above for the conditional input.

[0070] In general, the denoised output produced by the diffusion neural network defines an estimate of the final image given the current image.

[0071] In some implementations, the denoising output is an estimate of the noise component of the current image, i.e., the noise that needs to be combined with, e.g., added to or subtracted from, the final image to generate the current image.

[0072] In some other implementations, the denoising output is an estimate of the final image given the current image, i.e., an estimate of the image that would result from removing the noise component of the current image.

[0073] In yet other implementations, the system parameterizes the denoising output differently, for example using the v parameterization (Salimans and Ho arXiv:2202.00512, 2022, Section 4, Appendix D) or another suitable parameterization.

[0074] Specifically, this specification generally describes generative neural network 130 as a diffusion neural network. However, more generally, generative neural network 130 may be any suitable generative neural network capable of mapping conditional inputs, including text, to an output image, such as an autoregressive generative neural network, a non-autoregressive masked token generative neural network, a regularized flow model, a generator of a generative adversarial neural network, etc.

[0075] Generally, the text-conditioned image generation neural network is pre-trained for the text-conditioned image generation task, e.g., by system 100 or another system. For example, if the text-conditioned image generation neural network is a diffusion neural network, system 100 or another system may pre-train the diffusion neural network for the text-conditioned image generation task by optimizing any suitable diffusion model training objective, e.g., a score-matching objective.

[0076] The use of the text-conditioned image generation neural network 130 to train the image representation neural network 120 is described in more detail below with reference to FIGS.

[0077] 2 is a flow diagram of an exemplary process 200 for training an image representation neural network. For convenience, process 200 is described as being performed by one or more computer systems located at one or more locations. For example, a training system such as training system 100 shown in FIG. 1 can be suitably programmed in accordance with this specification to perform process 200.

[0078] The system can repeatedly perform iterations of process 200 on different training images to train the image representation neural network.

[0079] Specifically, the system obtains a set of one or more training images, step 202. For example, the system can sample one or more training images from a larger set of training data.

[0080] The system then performs steps 204 and 206 for each training image in the set.

[0081] Specifically, the system processes the training images using an image representation neural network to generate training representations of the training images, step 204. As previously mentioned, the training representation of a given training image is a set of text tokens from a vocabulary of text tokens.

[0082] The system then processes a text input comprising a set of text tokens in a training representation of a training image using a text-conditioned image generation neural network to generate an output that defines an output image (step 206).

[0083] The system then trains an image representation neural network based on the objective function (step 208).

[0084] The objective function includes, for each training image in the set, a first term that measures the difference between (i) a ground truth output corresponding to the training image and (ii) an output of the text-conditioned image generation neural network generated by processing a text input comprising the set of text tokens in the training representation of the training image. In some cases, the objective function also includes one or more additional terms, such as a regularization term or an auxiliary loss term, or both.

[0085] As mentioned above, in some cases, the text-conditioned image-generating neural network may be a text-conditioned diffusion neural network. In this example, the output of the text-conditional diffusion neural network that defines the output image is a denoised output, and the ground truth output is a ground truth denoised output corresponding to the training image.

[0086] Training the text-conditioned image generation neural network when it is a text-conditioned diffusion neural network is described below with reference to FIG.

[0087] In other cases, the text-conditioned image generative neural network may be a different type of generative neural network that directly generates the output image.

[0088] In some of these examples, the ground truth output is the training image, and the output image is the output of a text-conditioned image generation neural network generated by processing a text input with a set of text tokens in the training representation of the training image.

[0089] In other of these examples, the ground truth output may be a set of visual tokens, e.g., discrete tokens, representing the training image, and the output of the text-conditioned image generation neural network generated by processing a text input with a set of text tokens in the training representation of the training image is a set of visual tokens, e.g., discrete tokens, representing the output image.

[0090] Generally, the system trains an image representation neural network while holding the text-conditioned image generation neural network fixed. More specifically, the system back-propagates the gradient of the objective function through the text-conditioned image generation neural network to the image representation neural network to calculate the gradient of the objective function with respect to the parameters of the image representation neural network. The system can then apply an optimizer, such as Adam, rmsProp, or Adafactor, to the gradient to update the parameters of the image representation neural network (while holding the text-conditioned image generation neural network fixed).

[0091] To efficiently perform backpropagation during training, the system utilizes Gumbel-softmax as a continuous relaxation to backpropagate gradients from the decoder through discrete latents, i.e., Gumbel-softmax can be used to allow gradients to be backpropagated through discrete selections of text tokens in the representations generated by the image representation neural network. The relaxation becomes stronger as temperature τ → 0. The system can utilize an annealing schedule for temperature τ of Gumbel-softmax to improve training stability.

[0092] 3 is a flow diagram of an exemplary process 300 for training an image representation neural network, where the text-conditioned image generation neural network is a text-conditioned diffusion neural network. For convenience, process 300 is described as being performed by one or more computer systems located at one or more locations. For example, a training system such as training system 100 shown in FIG. 1 can be suitably programmed in accordance with this specification to perform process 300.

[0093] The system samples the noise level (step 302). For example, the system may sample a time step t from a predetermined interval, such as the interval between 0 and 1 or the interval between 0 and T (where T is a positive constant), as described above, and then associate the time step with a corresponding noise level σ. t That is, as mentioned above, each time step within the predetermined interval has a corresponding noise level.

[0094] This system uses a noise level of σ t We extract the noisy image χ from the training image χ by applying noise ε to the training image χ according to t is generated (step 304).

[0095] That is, the system samples noise ε from a noise distribution, e.g., a Gaussian or other suitable distribution, and then calculates the noise level σ t Combine the noise ε with the training image χ0 according to

[0096] For example, the noisy image χ t can satisfy the following: χ t =α t χ0+σ t ε In the above equation, α t teeth

[0097]

number

[0098] may be equal to

[0099] The system then processes the noisy image and a text input comprising a set of text tokens in the training representation of the training image using a text-conditioned diffusion neural network to generate a denoised output (step 306).

[0100] As mentioned above, the denoising output defines an estimate of the training image given the noisy image and the text input.

[0101] For example, as discussed above, the denoised output may be (i) an estimate of the training image, (ii) an estimate of the noise applied to the training image to generate the noisy image, or (iii) an estimate of the v predicted output generated from the noise and the training image.

[0102] After performing process 300 for each of the training images in the set, the system trains an image representation neural network based on an objective function that includes a first term that measures the difference between (i) the ground truth output corresponding to the training image and (ii) the denoised output, for each training image in the set.

[0103] The ground truth output generally depends on the type of denoising output produced by the diffusion neural network.

[0104] For example, if the denoised output is an estimate of the training images, the ground truth output may be the training images.

[0105] As another example, if the denoising output is an estimate of the noise applied to the training images to generate the noisy images, the ground truth output may be the sampled noise described above with reference to step 304.

[0106] As yet another example, if the denoised output is an estimate of the v-predicted output, the ground truth output may be the v-predicted output generated from noise and training images.

[0107] FIG. 4 illustrates an example 400 of downstream tasks that may be performed after training of the image representation neural network 120 (referred to in the drawings as "encoder").

[0108] As shown in example 400, image representation neural network 120 receives an image 402 of a dog and generates a textual representation 410 of image 402. As can be seen from the example, representation 410 is an informative text that blends comprehensive semantic concepts present in the image to create a "scrambled caption." That is, because representation 410 blends semantic concepts, it is not written in the style of a human user, yet representation 410 is nevertheless an accurate and comprehensive description of image 402.

[0109] As previously mentioned, during training, the representations 410 are provided as inputs to the text-conditioned image generation neural network to generate outputs that define output images 404 for use in training the neural network 120.

[0110] After training, the text representation 410 can be used for any of a variety of tasks.

[0111] That is, the text representation 410 can act as a flexible interface between different modalities. For example, the text representation 410 may be used as a description of, or in place of, the image 402, allowing various visual language applications to be performed on the image 402.

[0112] As one example 420, text representation 410 can serve as a "transferable" prompt that can be effectively provided to different text-to-image tools ("generative models"), such that each tool will generate output images that match the meaning of input image 402, despite different configurations and training paradigms. That is, providing the same traditional text prompt to different text-to-image tools would typically result in the tools generating different images with different meanings because the text-to-image tools are configured differently. However, by providing text representation 410 instead, each of the tools will instead generate output images that match the meaning of images generated by the other tools, because text representation 410 is accurate and comprehensive.

[0113] As another example 430, the text representation 410 enables a text-only chatbot to participate in a multimodal dialog, i.e., to effectively capture information representing images in sequence, despite being configured to process only text. Specifically, the text representation 410 can be used as part of an input prompt to a text-only LLM along with a query about an input image, thereby enabling the text-only LLM to accurately answer the query about the input image even though the LLM is not trained to process image data.

[0114] As yet another example 440, the text representations 410 can be used to inject image context into off-the-shelf large-scale language models to perform visual understanding tasks, e.g., free-form visual question answering, by prompting the LLM with few-shot examples, each containing a respective text representation generated by the neural network 120.

[0115] As used herein, the term "configured" is used in reference to computing systems and environments, as well as computer program components. A computing system or environment is considered "configured" to perform particular operations or actions if it includes the necessary software, firmware, hardware, or a combination thereof, enabling it to perform those operations or actions during operation. For example, configuring a system may involve installing a software library with a particular algorithm, updating firmware with new instructions for processing data, or adding hardware components to enhance processing capabilities. Similarly, one or more computer programs are "configured" to perform particular operations or actions if they contain instructions that, when executed by a computing device or hardware, cause the device to perform the intended operations or actions.

[0116] The embodiments and functional operations described herein can be implemented in various forms, including digital electronic circuitry, software, firmware, computer hardware (including the disclosed structures and their structural equivalents), or any combination thereof. The subject matter can be realized as one or more computer programs, which are essentially modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or controlling the operation of a computing device or hardware. The storage medium can be a storage device, such as a hard drive or solid-state drive (SSD), a storage medium, a random-access memory device, or a serial-access memory device, or a combination thereof. Additionally or alternatively, the program instructions can be encoded in a transmission signal, such as a machine-generated electrical, optical, or electromagnetic signal, designed to carry information for transmission to a receiving device or system for execution by the computing device or hardware. Furthermore, implementations can leverage emerging technologies, such as quantum computing or neuromorphic computing, for specific applications and can be deployed in distributed or cloud-based environments where components reside on different machines or within a cloud infrastructure.

[0117] The term "computing device or hardware" refers to physical components involved in data processing and encompasses all types of devices and machines used for this purpose. Examples include a processor or processing unit, a computer, multiple processors or computers working in conjunction, and specialized processing hardware such as a graphics processing unit (GPU), a tensor processing unit (TPU), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). In addition to hardware, a computing device or hardware may also include code that creates an execution environment for computer programs. This code can take the form of processor firmware, a protocol stack, a database management system, an operating system, or a combination of these elements. Embodiments may particularly benefit from leveraging the parallel processing capabilities of GPUs in the context of general-purpose computing on graphics processing units (GPGPUs), where code specifically designed for GPU execution, often referred to as kernels or shaders, is employed. Similarly, TPUs excel at performing optimized tensor operations, which are important for many machine learning algorithms. By leveraging these accelerators and their specialized programming models, the system can significantly speed up and streamline tasks involving artificial intelligence and machine learning, particularly in areas such as computer vision, natural language processing, and robotics.

[0118] A computer program, also referred to as software, an application, a module, a script, code, or simply a program, can be written in any programming language, including compiled or interpreted, declarative or procedural languages. It can be deployed in a variety of forms, such as a standalone program, a module, a component, a subroutine, or any other unit suitable for use in a computing environment. A program may or may not correspond to a single file in a file system and can be stored in a variety of ways, including being embedded within a file with other programs or data (e.g., a script within a markup language document), residing in a dedicated file, or being distributed across multiple coordinated files (e.g., files storing modules, subprograms, or code segments). A computer program can be executed on a single computer or across multiple computers, whether located at a single site or distributed across multiple sites and interconnected through a data communications network. A particular implementation of a computer program may include a combination of traditional programming languages ​​and specialized languages ​​or libraries designed for GPGPU programming or TPU utilization, depending on the selected hardware platform and desired performance characteristics.

[0119] As used herein, the term “engine” broadly refers to a software-based system, subsystem, or process designed to perform one or more specific functions. An engine is typically implemented as one or more software modules or components installed on one or more computers and can be located at a single site or distributed across multiple locations. In some cases, one or more dedicated computers may be used for a particular engine, while in other cases, multiple engines may run simultaneously on the same computer or computers. Examples of engine functions in the context of AI and machine learning can include data preprocessing and cleaning, feature engineering and extraction, model training and optimization, inference and prediction generation, and post-processing of results. The specific design and implementation of an engine varies depending on its overall architecture and the distribution of computational tasks across various hardware components, including CPUs, GPUs, TPUs, and other specialized processors.

[0120] The processes and logic flows described herein can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. Additionally, graphics processing units (GPUs) and tensor processing units (TPUs) can be utilized to enable simultaneous execution of aspects of these processes and logic flows, significantly improving performance. This approach offers significant advantages for computationally intensive tasks commonly found in AI and machine learning applications, such as matrix multiplication, convolution, and other operations that exhibit a high degree of parallelism. By leveraging the parallel processing capabilities of GPUs and TPUs, significant speedups and efficiency improvements can be achieved compared to relying solely on CPUs. Alternatively, or in combination with programmable computers and specialized processors, these processes and logic flows can also be implemented using specialized processing hardware, such as field programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs), further improving performance or energy efficiency in certain use cases.

[0121] A computer capable of executing a computer program can be based on a general-purpose microprocessor, a special-purpose microprocessor, or a combination of both. Any other type of central processing unit (CPU) can also be utilized. Additionally, graphics processing units (GPUs), tensor processing units (TPUs), and other machine learning accelerators can be employed to improve performance, particularly for tasks related to artificial intelligence and machine learning. These accelerators often work in conjunction with the CPU, handling specialized calculations while the CPU manages overall system operation and other tasks. Typically, a CPU receives instructions and data from read-only memory (ROM), random access memory (RAM), or both. Elements of a computer include a CPU for executing instructions and one or more memory devices for storing instructions and data. The specific configuration of the processing unit and memory varies depending on factors such as the complexity of the AI ​​model, the amount of data to be processed, and the desired performance and latency requirements. Embodiments can be implemented on a wide range of computing platforms, from small, resource-limited embedded devices to large-scale data center systems with high-performance computing capabilities. The system may include storage devices such as hard drives, SSDs, flash memory, etc. for persistent data storage.

[0122] Computer-readable media suitable for storing computer program instructions and data encompass all forms of non-volatile memory, media, and memory devices. Examples include read-only memory (ROM), semiconductor memory devices such as solid-state drives (SSDs) and flash memory devices, hard disk drives (HDDs), optical media, optical discs such as CDs, DVDs, and Blu-ray discs. The specific type of computer-readable medium used will depend on factors such as the size of the data, access speed requirements, cost considerations, and the level of portability or permanence desired.

[0123] To facilitate user interaction, embodiments of the subject matter described herein can be implemented on a computing device equipped with a display device, such as a liquid crystal display (LCD) or organic light-emitting diode (OLED) display, for presenting information to the user. Input can be provided by the user through a variety of means, including a keyboard, touchscreen, voice command, gesture recognition, or other input modalities, depending on the particular device and application. Additional input methods can include acoustic, voice, or tactile input, and feedback to the user can take the form of visual, auditory, or tactile feedback. Furthermore, the computer can interact with the user by exchanging documents with the user's device or application. This can include sending web content or data upon request, or sending and receiving text messages or other forms of messages through a mobile device or messaging platform. The choice of input and output modalities will depend on the particular application and the desired form of user interaction.

[0124] Machine learning models can be implemented and deployed using machine learning frameworks such as TensorFlow or JAX, which provide comprehensive tools and libraries that make it easy to develop, train, and deploy machine learning models.

[0125] Embodiments of the subject matter described herein can be implemented within a computing system comprising one or more components, depending on the particular application and requirements. These may include back-end components such as a back-end server or cloud-based infrastructure, optional middleware components such as a middleware server or application programming interface (API) for facilitating communication and data exchange, and front-end components such as a client device with a user interface, web browser, or app through which a user can interact with the implemented subject matter. For example, the described functionality may be implemented solely on the client device (e.g., for on-device machine learning) or may be deployed as a combination of front-end and back-end components for more complex applications. These components, if present, may be interconnected using any form or medium of digital data communication, such as a communications network, such as a local area network (LAN) or a wide area network (WAN), including the Internet. The specific system architecture and component selection will depend on factors such as the scale of the application, the need for real-time processing, data security requirements, and the desired user experience.

[0126] A computing system can include clients and servers that are geographically separated but can interact through a communications network. The specific type of network, such as a local area network (LAN), a wide area network (WAN), or the Internet, depends on the scope and scale of the application. The client-server relationship is established through computer programs running on the respective computers, which are designed to communicate with each other using appropriate protocols. These protocols may include HTTP, TCP / IP, or other specialized protocols, depending on the nature of the data being exchanged and the security requirements of the system. In certain embodiments, the server sends data or instructions to a user's device, such as a computer, smartphone, or tablet, which acts as a client. The client device can then process the received information, display the results to the user, and possibly send data or feedback back to the server for further processing or storage. This allows for dynamic interaction between users and the system, enabling a wide range of applications and functionality.

[0127] While this specification contains details of many specific implementations, these should not be construed as limiting the scope of the invention or what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features that are described herein in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable subcombination. Furthermore, even if features are described above as functioning in a particular combination and originally claimed as such, one or more features from a claimed combination may in some cases be deleted from that combination, and the claimed combination may be directed to subcombinations or variations of the subcombination.

[0128] Similarly, while acts are depicted in the figures and claimed in a particular order, it should not be understood that such acts need to be performed in the particular order or sequential order shown, or that all of the acts shown need to be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated into a single software product or packaged into multiple software products.

[0129] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. By way of example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0130] 100 Training System 102 input images 112 Expression 120 Image Representation Neural Network 130 Text-Conditioned Image Generation Neural Network 132 output images 150 Inference Systems 152 Query Input 154 query images 156 Expression 160 Downstream Neural Network 200 processes 300 processes 400 examples 402 images 404 Output Image 410 Text Expressions 420 examples 430 examples 440 examples

Claims

1. A method implemented by one or more computers, comprising: training an image representation neural network configured to receive an input image and process the input image to generate a representation of the input image as a set of text tokens from a vocabulary of text tokens, the training comprising: obtaining a set of one or more training images; For each training image in the set, processing the training images using the image representation neural network to generate training representations of the training images as a set of text tokens from the vocabulary of text tokens; processing a text input comprising the set of text tokens in the training representation of the training image using a text-conditioned image generation neural network to generate an output defining an output image; training the image representation neural network based on an objective function including, for each training image in the set, a first term that measures the difference between (i) a ground truth output corresponding to the training image and (ii) the output of the text-conditioned image generation neural network that was generated by processing the text input comprising the set of text tokens in the training representation of the training image; A method comprising:

2. 2. The method of claim 1 , wherein the text-conditioned image generation neural network has been pre-trained on a text-conditioned image generation task, and wherein training the image representation neural network based on the objective function comprises training the image representation neural network while holding the text-conditioned image generation neural network fixed.

3. 2. The method of claim 1 , wherein the text-conditioned image generation neural network is a text-conditional diffusion neural network, the output of the text-conditional diffusion neural network that defines the output image is a denoised output, and the ground truth output is a ground truth denoised output that corresponds to the training image.

4. processing a text input comprising the set of text tokens in the training representation of the training image using a text-conditioned image generation neural network to generate an output defining an output image; sampling the noise level; generating noisy images from the training images by applying noise to the training images according to the noise level; processing the noisy image and the text input using the text-conditioned diffusion neural network to generate the denoised output, the denoised output defining an estimate of the training image given the noisy image and the text input; The method of claim 3, comprising:

5. The noise-removed output is (i) the estimates of the training images; (ii) an estimate of the noise applied to the training images to generate the noisy image; or (iii) an estimate of the v predicted output generated from the noise and the training images; 5. The method of claim 4, wherein the method is one of:

6. The text conditional neural network a text encoder neural network configured to process the text input to generate an encoded representation of the text input; an image spreading neural network configured to generate the output image over multiple sampling steps conditioned on the encoded representation of the text input; The method of claim 3, comprising:

7. The method of claim 5 , wherein the vocabulary of text tokens is an input vocabulary of the text encoder neural network.

8. The image representation neural network an image backbone neural network configured to process the input image to generate a feature representation of the input image; an encoder neural network configured to process the feature representation of the input image to generate the representation of the input image; The method of claim 1 , comprising:

9. 9. The method of claim 8, wherein the image backbone neural network is pre-trained on an image representation learning task, and wherein training the image representation neural network based on the objective function comprises training the encoder neural network while keeping the image backbone neural network fixed.

10. The method of claim 8 , wherein training the image representation neural network based on the objective function comprises training the encoder neural network and the image backbone neural network.

11. the encoder neural network has a respective trained query corresponding to each text token in the representation, the encoder neural network comprises a sequence of self-attention layer blocks and an output layer block, and processing the feature representation of the input image to generate the representation of the input image comprises: processing the learned query through the sequence of self-attention layer blocks, each self-attention layer block configured to update the learned query conditional on the feature representation of the input image; after processing the trained queries through the sequence of self-attention layer blocks, processing each trained query using the output layer block to generate the corresponding text tokens in the representation; The method of claim 8, comprising:

12. The method of claim 11 , wherein the output layer block is a linear neural network layer.

13. After training the image representation neural network, receiving a query input for a downstream task comprising a query image; processing the query image using the image representation neural network to generate a representation of the query image as a set of text tokens; providing the representation of the query image as an input to a downstream neural network configured to perform the downstream task; The method of claim 1 , comprising:

14. 14. The method of claim 13, wherein the query input comprises the query image and text, and the downstream neural network is a large-scale language model, and wherein providing the representation of the query image as input to the downstream neural network configured to perform the downstream task comprises providing the representation of the query image and the text from the query input as inputs to the large-scale language model.

15. The method of claim 14 , wherein the downstream task is a multimodal dialog task.

16. The method of claim 14 , wherein the downstream task is a zero-shot task or a multimodal few-shot learning task.

17. 17. A system comprising one or more computers and one or more storage devices storing instructions operable, when executed by the one or more computers, to cause the one or more computers to perform the operations of the respective methods described in any one of claims 1 to 16.

18. One or more computer storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform the respective method operations of any one of claims 1 to 17.

Images