Multimodal neural network with only decoder language model

Abstract

Methods, systems, and devices, including computer programs encoded on computer storage media, for training a multi-modal neural network using a contrastive loss and an image description generation loss.

Description

Background Technology

[0001] This manual relates to using machine learning models to process input.

[0002] As an example, a neural network is a machine learning model that uses one or more layers of non-linear units to predict an output from a received input. In addition to the output layer, some neural networks also include one or more hidden layers. The output of each hidden layer is used as input to another layer in the network (e.g., the next hidden layer or the output layer). Each layer of the network generates an output from the received input based on the current values ​​of a corresponding set of weights. Summary of the Invention

[0003] This specification describes a system implemented as a computer program on one or more computers that uses a multimodal neural network to process multimodal inputs, including both visual input (i.e., images or multiple video frames from a video) and text.

[0004] As will be described below, the system can use both contrastive learning loss and captioning loss to jointly pre-train a multimodal neural network.

[0005] Specific embodiments of the subject matter described in this specification may be implemented in order to achieve one or more of the following advantages.

[0006] This specification describes a multimodal neural network with an architecture that allows the neural network to be jointly pre-trained using contrastive loss and descriptive generation loss.

[0007] Unlike other methods, neural networks use a shared decoder-only language model neural network to compute both text embeddings for contrastive loss and text description data for description generation loss.

[0008] In particular, previous methods used separate components to compute the text-based inputs to both losses under the assumption that the conflicting nature of the two tasks (i.e., the fact that contrastive learning uses unconditional sequence-level text representations to describe the probability of generating an optimized text sequence conditioned on previous words and their corresponding images) would make using a unified representation problematic or detrimental to overall performance.

[0009] On the other hand, the described techniques utilize a unified representation generated from a decoder-only language model and demonstrate that joint training for these diverse target tasks is possible. Therefore, these techniques maximize weight sharing in the language model's neural network, i.e., by sharing more parameters of the model's parameters between the contrastive loss and the description generation loss, while simultaneously achieving state-of-the-art performance on a variety of downstream multimodal tasks. This results in a more compact model with fewer parameters and therefore lower memory and storage requirements. The model can be deployed on devices with limited computational resources, such as mobile or edge devices.

[0010] Furthermore, using the same core architecture makes it very easy to extend to open vocabulary object detection and video text tasks, and unlike previous work, these extensions are made with minimal modifications. For example, the described techniques can be used to achieve state-of-the-art results on a variety of multimodal tasks, such as image-text and text-image retrieval, video question answering, and open vocabulary detection.

[0011] Furthermore, many existing methods use separate visual input encoders or separate passes through a single visual encoder to compute the contrast loss and image description generation loss. The described method computes both losses using only a single pass through a single visual encoder. Since the visual encoder is typically the most computationally expensive component of the system, computed contrast loss and description generation loss using only a single pass through the visual encoder significantly reduces training latency, FLOPs, and computational resource consumption.

[0012] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0013] Figure 1A An example neural network system is shown.

[0014] Figure 1B An example of training a multimodal neural network is shown.

[0015] Figure 2 This is a flowchart of an example process for training a multimodal neural network.

[0016] Figure 3 This is a flowchart of an example process for performing training steps during the training of a multimodal neural network.

[0017] Figure 4 An example of performing training steps is shown.

[0018] The same reference numerals and names in the various figures indicate the same elements. Detailed Implementation

[0019] Figure 1A An example neural network system 100 is shown. The neural network system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, in which the systems, components and techniques described below can be implemented.

[0020] System 100 is a system that uses a multimodal neural network 110 to process multimodal inputs including both visual input 102 (i.e., an image or multiple video frames from a video) and text. In other words, system 100 processes multimodal inputs including the intensity values ​​of pixels from an image or multiple video frames and the corresponding text sequence.

[0021] The multimodal neural network 110 includes: (i) a visual encoder neural network 112 configured to process a visual input 102 comprising one or more images to generate an encoded representation 114 of the visual input 102; and (ii) a decoder-only language model neural network 120.

[0022] Generally, system 100 can pre-train a multimodal neural network 110 on both contrastive loss and descriptive generation loss.

[0023] The contrast loss can depend on the encoded representation generated by the visual encoder, for example, on the embedding generated based on the encoded representation and the text embedding generated based on the representation generated by the language model neural network, while the description generation loss can depend on the encoded representation generated by the visual encoder and the score generated by the language model neural network.

[0024] After the multimodal neural network 110 has been pre-trained, the multimodal neural network 110, the visual encoder 112, the language model 120, or a combination thereof can be used for downstream tasks.

[0025] In some implementations, downstream tasks can be performed in a zero-shot manner, that is, without further training of any of the components of the multimodal neural network 110.

[0026] In some other implementations, downstream tasks can be performed after one or more components of the multimodal neural network 110 have been fine-tuned (i.e. further trained) on labeled training data used for the downstream task.

[0027] As an example, system 100 can fine-tune network 110 on labeled training data without modifying the architecture of multimodal neural network 110.

[0028] As another example, the system can keep any part of the visual encoder 112 and the language model neural network fixed for downstream tasks, while learning one or more additional layers that (i) process the encoded representations generated by the visual encoder 112 to generate outputs for downstream tasks, (ii) generate inputs to the visual encoder, or (iii) both.

[0029] In some examples, the downstream task is a visual classification task that requires classifying visual input (e.g., video or image) into one of a set of categories that correspond to different object types.

[0030] In some other examples, the downstream task is a visual action recognition task, which requires classifying video input into one action category from a set of action categories.

[0031] In some examples, the downstream task is a cross-modal retrieval task that requires (i) retrieving one or more text sequences most similar to the visual input or (ii) retrieving one or more visual input sequences most similar to the text sequence.

[0032] In some examples, the downstream task is a multimodal understanding task. For instance, this task could be a visual question answering (VQA) task that requires generating answers to questions posed about visual input.

[0033] In some examples, the downstream task is an image description generation task that requires generating textual descriptions for visual input.

[0034] In some examples, the downstream task is an open vocabulary object detection task that requires detecting objects in the input image.

[0035] Generally, System 100 can be trained for downstream tasks in any appropriate manner, i.e., using any suitable supervised learning loss function suitable for downstream tasks.

[0036] Generally, the language model neural network 120 is configured to process the current text sequence 104 to generate an output that defines new lexical units 128 to be appended to the current text sequence 104.

[0037] The output defining the new lexical unit 128 to be appended to the current text sequence 104 typically includes the corresponding score for each lexical unit in the lexical unit vocabulary. The lexical unit vocabulary can include any of the following: characters, subwords, words, punctuation marks, symbol lexical units (e.g., #, $, and other symbols), mathematical symbols, etc. The lexical unit vocabulary can also include one or more special lexical units appended to the input text sequence processed by the neural network, such as sequence start lexical units, sequence end lexical units, designated "class" lexical units, etc.

[0038] During training, the language model neural network 120 can generate the corresponding output for each of the multiple tokens in the input sequence in a single forward pass (i.e., in parallel) by processing a single “current sequence” 104 representing the entire input text sequence.

[0039] After training, the language model neural network 120 can be used to regressively generate text sequences by: processing the current text sequence 104 up to that time step at each time step; and then updating the current text sequence 104 by selecting lexical units from the vocabulary using the output for the current text sequence and then appending the selected lexical units to the end of the current text sequence 104.

[0040] The visual encoder neural network 112 is a neural network that has parameters (“visual encoder neural network parameters” or “visual encoder parameters”) and receives visual input 102 (i.e., input image or video) and processes the visual input 102 according to the parameters to generate an encoded representation 114 of the visual input 102.

[0041] Generally, the encoded representation 114 includes corresponding embeddings (also referred to as "updated lexical units") for each of the plurality of pieces in the visual input 102 (e.g., each of the plurality of spatial pieces (regions) for each image in the images of the visual input 102); or, in some cases where the visual input 102 includes multiple images, each of the plurality of spatiotemporal pieces (regions) of the visual input 102.

[0042] As used in this specification, “embedded” is a vector of numeric values ​​(e.g., floating-point values ​​or other values) with a predetermined dimension. The space of possible vectors with a predetermined dimension is called the “embedded space”.

[0043] The visual encoder neural network 112 can have any suitable architecture that allows the neural network 112 to map the input visual input 102 to the encoded representation 114. For example, the visual encoder neural network 112 can be a convolutional neural network. As another example, the visual encoder neural network 112 can be a visual Transformer neural network with one or more self-attention layers. As yet another example, the visual encoder neural network 112 can be a neural network with a hybrid of both convolutional layers and self-attention layers.

[0044] When the encoder neural network 112 is a visual Transformer or another neural network that generates the initial embedding for each piece of visual input 102 and updates the initial embedding to generate the encoded representation 114, the encoder neural network 112 can use the cropped positional embedding during training to improve the generalization of the neural network to downstream tasks after training.

[0045] The cropped location embedding causes the model to treat the input image as a "crop" from a larger image. Specifically, cropped location embeddings of patches of the given input image are generated by generating random cropped versions of the larger image, where the cropped versions have the same size as the given input image. Then, cropped location embeddings of patches of the given input image are generated by assigning the location embeddings of corresponding patches from the cropped versions of the larger image as cropped location embeddings of patches of the input image.

[0046] The cropped position embedding is described in more detail in the following literature: Kim, et al, Region-AwarePretraining for Open-Vocabulary Object Detection with Vision Transformers, available at arXiv:2305.07011.

[0047] The language model neural network 120 can have any suitable architecture that allows the language model neural network 120 to map the lexical units in the text sequence 104 to the output of the next defined lexical unit 128.

[0048] In a specific example, the language model neural network 120 can have an attention-based architecture, such as the architecture of a decoder-only Transformer neural network.

[0049] In this example, the language model neural network 120 may include a sequence of layers containing one or more self-attention layers, wherein each self-attention layer is configured to receive a corresponding current representation of each text word in the current text sequence as input, and process the corresponding current representation to generate a corresponding updated representation of each text word in the current text sequence as output by applying a self-attention mechanism to the corresponding current representation.

[0050] The self-attention mechanism on the corresponding current representation refers to the attention mechanism that computes queries, keys, and values ​​from the corresponding current representation.

[0051] After training, each self-attention layer can apply a causal masking self-attention mechanism to the corresponding current representation to generate the updated representation.

[0052] The self-attention mechanism of causal masking on the corresponding current representation refers to the following attention mechanism: no given position in the current text sequence pays attention to any position after the given position in the current text sequence, that is, no non-zero attention weight is given to any position after the given position in the current text sequence.

[0053] This contrasts with self-attention mechanisms with bidirectional masks, where any given position in the current text sequence pays attention to all positions in the current text sequence; that is, it has non-zero attention weights for all positions in the current text sequence, regardless of whether those positions are after a given position in the current text sequence.

[0054] Each self-attention layer may optionally (e.g., by leveraging a position-wise feedforward neural network, by applying layer normalization, by leveraging residual connections, etc.) apply other operations to the representation as part of updating that representation.

[0055] Optionally, during training, system 100 can switch between applying causal masking and not applying causal masking during the processing of different inputs to neural network 120.

[0056] In this example, the corresponding current representation received as input by the first self-attention layer in the sequence of layers is the corresponding embedding of each text word in the current text sequence (e.g., generated by the embedding layer of the language model neural network 120), and the corresponding current representation received as input by each subsequent layer (i.e., each layer after the first layer in the sequence) is the corresponding updated representation of the text word in the current text sequence generated by the previous layer in the sequence of layers as output.

[0057] Additionally, the neural network 120 includes one or more cross-attention layers as part of a sequence of layers. Each cross-attention layer processes the corresponding current representation by applying a cross-attention mechanism between the corresponding current representation of a text word in the current text sequence, which is received as input by the cross-attention layer, and the (generated) input derived from the encoded representation of the visual input, to generate a corresponding updated representation of each text word in the current text sequence as output.

[0058] "Cross-attention" refers to the attention mechanism that uses the query derived from the corresponding current representation of the text words in the current text sequence as input, and the keys and values ​​derived from the input generated from the encoded representation of the visual input.

[0059] Specific examples of self-attention mechanisms, including self-attention, cross-attention, and causal masking, that can be employed by the system are described in the following literature: Vaswani et al. “Attention is all you need”, 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA; Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, SharanNarang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXivpreprint arXiv:1910.10683, 2019; Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. Towards a human-like open-domain chatbot. CoRR, abs / 2001.09977, 2020; and Tom B Brown, BenjaminMann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, ArvindNeelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Languagemodels are few-shot learners. arXiv preprint arXiv:2005.14165, 2020, Hua, et al., Transformer Quality in Linear Time, arXiv preprint arXiv:2202.10447, 2022.

[0060] In some implementations, the input derived from the encoded representation (also referred to below as the description-generated representation) is an embedding in the encoded representation 114.

[0061] In some other implementations, the neural network 110 applies one or more transformations to the encoded representation 114 to generate inputs for the cross-attention layer.

[0062] For example, neural network 110 can apply linear layers or other learned transformations to project the encoded representation 114 to have the same dimensions as the representation generated by language model neural network 120.

[0063] Therefore, the updated representation generated by a given cross-attention layer is a multimodal representation that depends on the visual input and the text terms in the current text sequence. Each of these cross-attention layers may optionally (e.g., by utilizing a position-wise feedforward neural network, by applying layer normalization, by utilizing residual connections, etc.) apply other operations to the representation as part of the updated representation.

[0064] As an example, the sequence of attention layers can alternate between self-attention layers and cross-attention layers. As another example, the sequence of attention layers can include a cross-attention layer after every two, three, or four self-attention layers.

[0065] Therefore, due to the presence of the cross-attention layer, the representation generated by the last attention layer in the sequence is a multimodal representation, as described above.

[0066] To generate the score distribution, the neural network 120 may also include an output layer block.

[0067] The output layer block is a set of one or more neural network layers, such as one or more fully connected layers followed by a softmax layer. This set of one or more neural network layers is configured to receive one or more of the corresponding updated representations of text words in the current text sequence as outputs generated by the last subsequent attention layer in the sequence of subsequent attention layers, and to process the one or more corresponding updated representations to generate an output that defines new words to be appended to the current text sequence, i.e., to generate a score distribution of words in the vocabulary.

[0068] As a specific example, the output layer block may include a single fully connected layer (“vocabulary embedding layer”), optionally followed by a softmax layer, which maps the corresponding updated representation to the score distribution.

[0069] For example, during training, when the current output sequence is the entire training sequence, the output layer block can generate the corresponding score distribution for each text word in parallel by processing the updated representation of the word immediately preceding that text word in the training sequence for each text word to generate the score distribution for that text word. In this example, the system can augment the training sequence by starting with a specified sequence of words before using the language model neural network to process the training sequence.

[0070] After training, while system 100 is operating autoregressively, the output layer block can generate a single score distribution for the current output sequence by processing the updated representation of the last word in the current output sequence. System 100 can then use the score distribution generated by the output layer block to select the next word to add to the current output sequence. For example, system 100 can select the word with the highest score in the score distribution, or it can sample words from the score distribution.

[0071] As described above, system 100 or another training system can pre-train the multimodal neural network 110 on both the contrastive loss and the descriptive generation loss.

[0072] Figure 1B Example 150 of training a multimodal neural network 110 is shown.

[0073] As shown above, during training, the system receives training pairs including images and text sequences (“a white cliff with plants on it”).

[0074] The system uses a visual encoder neural network 112 to process images to generate image features.

[0075] The system also uses a decoder-only language model neural network 120 to process text sequences to generate text features for images.

[0076] The system can then use image features and text features to calculate the contrast loss, while using text features to generate scores for calculating the contrast loss, as will be described in more detail below.

[0077] Figure 2 This is a flowchart of an example process 200 for training a multimodal neural network. For convenience, process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a properly programmed neural network system (e.g., neural network system 100 of Figure 1) can perform process 200.

[0078] The system obtains a training dataset consisting of multiple image-text sequence pairs (step 202).

[0079] In other words, each pair in the training dataset includes a visual input and an input text sequence.

[0080] Specifically, the input text sequence has been determined by the system or an external source to describe the content of the visual input or otherwise relate to it. In other words, the visual input and the input text sequence have been determined to be semantically similar.

[0081] For example, within a given training pair, the text sequence could be a text annotation of the visual input from a set of manually or automatically generated image annotations, or it could be alternative text associated with the visual input from a set of alternative text data. Alternative text is text displayed on a webpage in place of an image, for example, when an image cannot be rendered correctly or otherwise fails to load. For instance, the system could obtain alternative text data from data maintained by an internet search engine or other software that automatically crawls webpages on the internet.

[0082] The system trains a neural network on the training dataset to minimize the overall loss function (step 204).

[0083] Generally, the overall loss function includes: (i) a contrastive loss, which measures the similarity between an image embedding generated from an encoded representation produced by a visual encoder neural network and a text embedding generated from the output of a language model neural network; and (ii) an image description generation loss, which measures the quality of a text description generated by a language model neural network given an encoded representation produced by a visual encoder neural network.

[0084] For example, quality can be measured based on the true value of the expected output of the language model's neural network or the target text description. More specifically, quality can be measured based on the scores assigned to the lexical units in the target text description by the language model's neural network.

[0085] Specifically, in order to train a neural network, the system can iterate the training process repeatedly on different batches of training pairs sampled from the training dataset to update the parameters of the visual encoder neural network, the language model neural network, or both.

[0086] In other words, at each iteration of the training process, the system can obtain a batch of training pairs (e.g., by sampling the batch from a larger training dataset) and use one or more training pairs from the batch to update the parameters of the visual encoder neural network and the language model neural network.

[0087] The system can continue to iterate the training process until the termination criteria for training the neural network are met, such as until the parameters have converged, until a threshold amount of clock time has elapsed, or until a threshold number of iterations of the process has been performed.

[0088] The following text is for reference only. Figure 3 An example describing the execution of the training process.

[0089] Generally, as described above, a decoder-only language model neural network includes (i) one or more self-attention layers and (ii) one or more cross-attention layers. Therefore, utilizing cross-attention layers introduces a dependency on the input image into the representation of the input text sequence generated by the language model neural network.

[0090] To take this into account, the contrastive loss measures the similarity between an image embedding generated from an encoded representation produced by a visual encoder neural network and a text embedding generated from the output of a language model neural network with one or more cross-attention layers disabled. This ensures that the text embedding used for the contrastive loss is independent of the input image.

[0091] Moreover, autoregressive causal masking can be suitable for image description generation loss, but may reduce the quality of the representations generated for use in contrastive loss.

[0092] With this in mind, contrastive loss can measure the similarity between image embeddings generated from encoded representations produced by a visual encoder neural network and text embeddings generated from the output of a language model neural network with one or more cross-attention layers disabled and a self-attention mechanism applied with bidirectional masking.

[0093] On the other hand, image description generation loss can measure the quality of text descriptions generated by a language model neural network with cross-attention layers enabled and self-attention mechanisms applied, given an encoded representation generated by a visual encoder neural network.

[0094] The following text is for reference only. Figure 3 The description modifies the operations performed during training iterations to allow for the use of appropriate quantities to evaluate both losses.

[0095] Figure 3 This is a flowchart of an example process 300 for performing training iterations during the training of a multimodal neural network. For convenience, process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a properly programmed neural network system (e.g., neural network system 100 of Figure 1) can perform process 300.

[0096] The system obtains one or more training pairs (step 302).

[0097] The system uses a visual encoder neural network to process each image in the batch to generate an encoded representation of the image, and generates an embedding of the image based on the encoded representation (step 304).

[0098] This system can generate image embeddings of images in any of a variety of ways.

[0099] For example, the system can generate image embeddings by processing encoded representations using one or more learned operations jointly learned with the training of a neural network.

[0100] For example, the system can use a linear layer to process the encoded representation and, for example, pool the output of the linear layer along the spatial dimension to generate an image embedding. As another example, the system can, for example, pool the encoded representation along the spatial dimension to generate an initial embedding and apply a linear layer to the initial embedding to generate an image embedding.

[0101] The system uses a language model neural network with the cross-attention layer disabled to process each text sequence in the batch to generate an embedding for that text sequence (step 306). That is, the system performs step 306 to generate embeddings for text sequences of any image in the batch, independent of the images, by disabling (i.e., bypassing) the cross-attention layer, such that the cross-attention layer is not used during the processing of the text sequence. Optionally, the system can also configure the masking of the self-attention layer as a bidirectional mask.

[0102] This system can generate text embeddings of text sequences in any of a variety of ways.

[0103] As an example, the system can apply global pooling on the sequence dimension of the updated representation generated by the last decoder layer before the output subnet in the sequence.

[0104] Therefore, in this example, only the output subnet is not shared between the contrastive loss and the descriptive generation loss, thereby maximizing parameter sharing and improving the quality of training.

[0105] For each pair, the system processes the text sequence in the pair using a language model neural network conditioned on the images in the batch and with causal masks on a self-attention layer to generate a corresponding score for each word in the text sequence (step 308). Specifically, the system can process the text sequence in the pair by using a language model neural network with cross-attention layers enabled and receiving input derived from encoded representations of the images in the pair, while ensuring that the language model neural network is image-conditioned.

[0106] Therefore, as part of the execution process 300, the system performs two forward propagations through the language model neural network: one forward propagation independent of the training image for generating embeddings of the text sequence, and the other forward propagation conditioned on the corresponding training image for generating the corresponding scores of the lexical units in the text sequence.

[0107] The system determines the gradient of the contrast loss, which is computed using the embeddings of images in the batch and the embeddings of text sequences in the batch (step 310).

[0108] The goal of contrastive loss is to train visual encoders and language models so that they can embed image and text inputs into a representation space (i.e., the space of image and text embeddings) so that inputs with similar semantics are mapped to nearby points regardless of their modality.

[0109] Therefore, the system can train neural networks 112 and 120 on a contrastive loss that is suitable for batches including visual input x. i and text sequence y i All training for x encourages i Text embedding and y i The image embeddings are closer together, while being further away from all other visual inputs and all other embeddings of text segments in the batch.

[0110] The following section will describe a specific example of contrastive loss.

[0111] Based on the embedding of images and text segments in mini-batches, an N×N similarity matrix A is calculated, where A i;j It represents x i Embedding and y j The value of the similarity of the embeddings. For example, A i;j It can be x i Embedding and y j The dot product between the embeddings.

[0112] The system can then use the gradient of the contrastive loss computed using matrix A to train both the language model neural network and the visual encoder neural network. For example, the contrastive loss could be a cross-entropy loss on the rows and columns of A, where diagonal terms are considered correct classes and other terms are considered incorrect classes. A specific example of this type of loss is: in is the softmax temperature of the scaling log (e.g., used to steepen or weaken the softmax distribution in the rows and columns of A), and N is the total number of training pairs in the batch. In some cases, the system normalizes the contrastive and unimodal representations of the visual input and text sequences in the batch before computing matrix A.

[0113] Since this loss is minimized, for all pairs in the batch, x i and y i The embeddings become closer together, while becoming farther away from all other visual inputs and all other embeddings of the text segments in the batch, thereby achieving the goal of contrastive learning.

[0114] As another example, the system can alternatively use focus contrast loss.

[0115] The focus contrast loss can be expressed as: in It is a normalized version of the image embedding of the image in the i-th pair of the batch, and It is the normalized version of the text embedding of the text sequence in the j-th pair of the batch.

[0116] Alternatively, the focus contrast loss can be expressed as the sum of the text-to-image focus loss and the image-to-text focus loss: The system determines the gradient of the image description generation loss, which is computed for each pair using the corresponding score for each word in the text sequence (step 312).

[0117] As mentioned above, for a given training pair, the image description generation loss is typically based on the scores assigned to lexical units in the training text sequence by the language model neural network (when conditioned on the visual input in the training pair).

[0118] As a specific example, the description generation loss for a given training pair can be given by the following: The overall description generation loss is the average of the description generation losses of the training pairs in the batch, and T is the total number of positions in the training text sequence of the training pair. The word at position t in the training text sequence is assigned from a score distribution generated conditioned on the words preceding the word at position t in the training text sequence and the visual input x in the training pair. The score.

[0119] The system determines the overall gradient of the overall loss function based on the gradients of the contrast loss and the image captioning generation loss (step 314). For example, this overall gradient could be a weighted sum of the contrast loss and the image captioning generation loss.

[0120] The system uses the global gradient to train the multimodal neural network (step 316). For example, the system can apply an optimizer to the global gradient and the current values ​​of the network parameters of the multimodal neural network to update the current values ​​of the network parameters.

[0121] Figure 4 Example 400 of performing training iterations is shown.

[0122] As shown in Example 400, the system performs three forward propagations during a given training iteration.

[0123] Specifically, the system performs a forward propagation 410 through a visual input encoder neural network to generate an encoded representation of the image and an image embedding for each image in the batch.

[0124] The system performs a first forward propagation 420 through a language model neural network without using cross attention and with bidirectional interrogation on a self-attention layer to generate a text embedding for each text sequence in the text sequence. As can be seen from Example 400, the text embedding is computed independently of any image in the batch.

[0125] The system can then use text embeddings and image embeddings to evaluate the contrast loss, as described above.

[0126] The system also performs a second forward propagation 430 of a language model neural network with cross-attention enabled and causal masking to generate the scores necessary to compute the image description generation (“generative”) loss. Due to the use of cross-attention, the generative loss features are conditioned on the corresponding input image.

[0127] In some implementations, the system trains a visual input encoder on training data including image-text pairs, while the downstream task requires video processing. In these implementations, the system can use the visual input encoder to process image patch embeddings during training, and process the embeddings of “video tubes,” each representing a spatiotemporal patch covering a corresponding spatial region within multiple video frames, during downstream fine-tuning and while performing the downstream task. The system can extract the video tubes and project them onto the embeddings using learned components acquired as part of the downstream training.

[0128] This specification uses the term "configuration" in conjunction with system and computer program components. For a system of one or more computers to be configured to perform a specific operation or action, it means that software, firmware, hardware, or a combination thereof are installed on the system to cause the system to perform that operation or action during operation. For configuring one or more computer programs to perform a specific operation or action, it means that one or more programs include instructions that, when executed by a data processing device, cause that device to perform that operation or action.

[0129] Embodiments of the subject matter and functional operation described in this specification may be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their equivalents), or in combinations thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by a data processing device or for controlling the operation of the data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination thereof. Alternatively or additionally, program instructions may be encoded on artificially generated propagation signals (e.g., machine-generated electrical, optical, or electromagnetic signals) that are generated to encode information for transmission to a suitable receiver device for execution by the data processing device.

[0130] The term "data processing device" refers to data processing hardware and encompasses all kinds of devices, apparatuses, and machines used for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The device may also be or further include a dedicated logic circuit system, such as an FPGA (Field-Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit). In addition to hardware, the device may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0131] A computer program (which may also be referred to or described as a program, software, software application, app, module, software module, script, or code) can be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not necessarily, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions). A computer program can be deployed to execute on a single computer or on multiple computers located at a site or distributed across multiple sites and interconnected via a data communication network.

[0132] In this specification, the term "database" is used broadly to refer to any collection of data: data that does not need to be structured in any particular way, or does not need to be structured at all, and can be stored on storage devices in one or more locations. Thus, for example, an indexed database may include multiple collections of data, each of which can be organized and accessed differently.

[0133] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines may be installed and run on the same one or more computers.

[0134] The processes and logic flows described in this specification can be executed by one or more programmable computers, which execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by a dedicated logic circuit system, such as an FPGA or ASIC, or by a combination of a dedicated logic circuit system and one or more programmable computers.

[0135] A computer suitable for executing computer programs may be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the central processing unit for making or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated into a special-purpose logic circuit system. Generally, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or operatively coupled to receive data from or transfer data to such one or more mass storage devices, or both. However, a computer does not necessarily need to have such devices. Furthermore, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0136] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CD-ROMs and DVD-ROMs.

[0137] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser in response to a request received from a web browser on the user's device. Additionally, the computer can interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and receiving responsive messages from the user in response.

[0138] Data processing devices used to implement machine learning models may also include, for example, dedicated hardware accelerator units for handling the general and computationally intensive portions of machine learning training or production (i.e., inference, workloads).

[0139] Machine learning models can be implemented and deployed using machine learning frameworks (such as the TensorFlow framework or the Jax framework).

[0140] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components (e.g., as a data server), or middleware components (e.g., an application server), or front-end components (e.g., a client computer having a graphical user interface, web browser, or app that a user can interact with through an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0141] A computing system may include clients and servers. Clients and servers are typically geographically separated and interact via a communication network. The client-server relationship is established by computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., HTML pages) to a user device, for example, for the purpose of displaying data to a user interacting with the device acting as a client and receiving user input from that user. Data generated at the user device, such as the result of user interaction, may be received at the server from the device.

[0142] While this specification contains numerous details of specific implementations, these details should not be construed as limiting the scope of any invention or the scope that may be claimed, but rather as descriptions of features that may be characteristic of particular embodiments of a particular invention. Certain features described in the context of individual embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as operating in certain combinations and even initially claimed in this way, in some cases one or more features from the claimed combination may be removed from the combination, and the claimed combination may involve sub-combinations or variations thereof.

[0143] Similarly, although operations are depicted in the accompanying drawings and described in a specific order in the claims, this should not be construed as requiring such operations to be performed in the specific order shown or in sequential order, or requiring all shown operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0144] This specification also provides the subject matter of the following clauses:

[0145] Clause 1. A method for training a neural network to perform one or more multimodal tasks, said neural network comprising:

[0146] A visual encoder neural network, configured to process a visual input comprising one or more images to generate an encoded representation of the visual input; and

[0147] A decoder-only language model neural network, wherein the decoder-only language model neural network is configured to process a current text sequence to generate an output defining new lexical units to be appended to the current text sequence, wherein the current text sequence includes corresponding text lexical units at each of one or more input positions, and wherein the training includes:

[0148] Obtain a training dataset, which includes multiple image-text sequence pairs; and

[0149] The neural network is trained on the training dataset to minimize an overall loss function comprising: (i) a contrastive loss that measures the similarity between an image embedding generated from an encoded representation produced by the visual encoder neural network and a text embedding generated from the output of the language model neural network; and (ii) an image description generation loss that measures the quality of a text description generated by the language model neural network given an encoded representation produced by the visual encoder neural network.

[0150] Clause 2. The method described in Clause 1 further includes:

[0151] After training the neural network, it is used to perform downstream multimodal tasks.

[0152] Clause 3. The method as described in Clause 2, wherein using the neural network to perform downstream multimodal tasks includes:

[0153] A downstream neural network, comprising the visual encoder and the language model neural network, is trained on training data used for the downstream multimodal task.

[0154] Clause 4. The method as described in Clause 3, wherein using the neural network to perform the downstream multimodal task comprises:

[0155] A trained downstream neural network is used to process the task input for the downstream multimodal task to generate the task output for the downstream multimodal task.

[0156] Clause 5. The method of Clause 2, wherein using the neural network to perform a downstream multimodal task comprises: using a trained neural network to process task inputs for the downstream multimodal task to generate task outputs for the downstream multimodal task.

[0157] Clause 6. The method as described in Clause 5, wherein using the neural network to perform a downstream multimodal task comprises: using the neural network to perform the downstream multimodal task without fine-tuning the trained neural network.

[0158] Clause 7. The method as described in any of the preceding clauses, wherein the language model neural network comprises a sequence of layers, wherein each layer in the sequence receives a corresponding current representation of each text word in the current text sequence as input, and generates a corresponding updated representation of each text word in the current text sequence as output, and wherein the sequence of layers comprises:

[0159] One or more self-attention layers, each configured to receive a corresponding current representation of each text word in the current text sequence as input, and to process the corresponding current representation by applying a self-attention mechanism to the corresponding current representation of each text word in the current text sequence to generate a corresponding updated representation of each text word in the current text sequence as output; and

[0160] One or more cross-attention layers, each configured to apply a cross-attention mechanism between the corresponding current representation of the text word in the current text sequence received as input by the cross-attention layer and the input derived from the encoded representation of the visual input.

[0161] Clause 8. The method as described in Clause 7, wherein the contrast loss measures the similarity between an image embedding generated from an encoded representation produced by the visual encoder neural network and a text embedding generated from the output of the language model neural network with the one or more cross-attention layers disabled.

[0162] Clause 9. The method as described in Clause 8, wherein the contrastive loss measures the similarity between an image embedding generated from an encoded representation produced by the visual encoder neural network and a text embedding generated from the output of the language model neural network with the one or more cross-attention layers disabled and the self-attention mechanism applied with a bidirectional mask.

[0163] Clause 10. The method as described in Clause 8 or Clause 9, wherein the image description generation loss measures the quality of the text description generated by the language model neural network with the cross-attention layer enabled and the self-attention mechanism applied, given an encoded representation generated by the visual encoder neural network.

[0164] Clause 11. The method of any one of Clauses 7 to 10, wherein each text embedding is generated based on an updated representation generated by the last layer in the sequence.

[0165] Clause 12. The method of any one of Clauses 7 to 11, wherein training the neural network comprises:

[0166] Obtain a batch of multiple image-text sequence pairs;

[0167] The visual encoder neural network is used to process each image in the batch to generate an encoded representation of the image, and an embedding of the image is generated based on the encoded representation;

[0168] The language model neural network with the cross-attention layer disabled is used to process each text sequence in the batch to generate the embedding of the image;

[0169] For each pair, the text sequence in the pair is processed using the language model neural network that has the cross-attention layer enabled and receives input derived from the encoded representation of the image in the pair and has a causal mask on the self-attention mechanism to generate a corresponding score for each word in the word sequence.

[0170] The gradient of the contrast loss is determined, and the gradient of the contrast loss is calculated using the embeddings of the images in the batch and the embeddings of the text sequence in the batch;

[0171] The gradient of the image description generation loss is determined, and the gradient of the image description generation loss is calculated for each pair using the corresponding score of each word in the word sequence;

[0172] The overall gradient of the overall loss function is determined based on the gradient of the contrast loss and the gradient of the image description generation loss; and

[0173] The neural network is trained using the global gradient.

[0174] Clause 13. The method of any one of Clauses 7 to 10, wherein processing each text sequence in the batch to generate the embedding of the image using the language model neural network with the cross-attention layer disabled comprises:

[0175] The language model neural network, with the cross-attention layer disabled and bidirectional masking on the self-attention mechanism, is used to process each text sequence in the batch to generate the embedding of the image.

[0176] Clause 14. The method as described in any one of Clauses 7 to 13, wherein,

[0177] The corresponding current representation received as input by the first layer in the sequence is the corresponding embedding of each text word in the current text sequence, and

[0178] The corresponding current representation received as input by each layer after the attention layer in the sequence is the corresponding updated representation of the text words in the current text sequence generated as output by the previous layer in the sequence.

[0179] Clause 15. A system as described in any of the preceding clauses, wherein defining the output for attaching a new lexical to the current text sequence comprises a score distribution that assigns a corresponding score to each lexical in the vocabulary of text lexicals.

[0180] Clause 16. The method as described in Clause 15, wherein, when subject to Clause 15, the language model neural network includes a lexicon embedding layer configured to map the output of the last layer in the sequence to the score distribution.

[0181] Clause 17. The method as described in any of the preceding clauses, wherein the encoded representation comprises a corresponding updated lexical for each of a plurality of pieces of the visual input.

[0182] Clause 18. The method as described in any of the preceding clauses, wherein the visual encoder neural network is a visual Transformer neural network.

[0183] Clause 19. The method as described in any of the preceding clauses, wherein each image embedding is generated by processing a corresponding encoded representation using one or more learned operations jointly learned with the training of the neural network.

[0184] Clause 20. The method as described in Clause 20, wherein generating each image embedding comprises:

[0185] A linear layer is used to process the corresponding encoded representation, and the output of the linear layer is pooled; or

[0186] Pooling the encoded representation to generate an initial embedding and applying a linear layer to the initial embedding to generate the image embedding.

[0187] Clause 21. The method described in any of the preceding clauses, wherein the contrast loss is a focal contrast loss.

[0188] Clause 22. The method described in any of the preceding clauses, when subject to Clause 18, wherein the visual Transformer neural network uses pruned position embeddings.

[0189] Clause 23. A system comprising:

[0190] One or more computers; and

[0191] A storage device for storing one or more instructions, which, when executed by the one or more computers, cause the one or more computers to perform a corresponding operation of the method as described in any one of Clauses 1 to 22.

[0192] Clause 24. One or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform a corresponding operation of the method as described in any one of Clauses 1 to 22.

[0193] Specific embodiments of this subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the figures do not necessarily require a specific order or sequence to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous.

Claims

1. A method for training a neural network to perform one or more multimodal tasks, the neural network comprising: A visual encoder neural network, configured to process a visual input comprising one or more images to generate an encoded representation of the visual input; as well as A decoder-only language model neural network, wherein the decoder-only language model neural network is configured to process a current text sequence to generate an output defining new lexical units to be appended to the current text sequence, wherein the current text sequence includes corresponding text lexical units at each of one or more input positions, and wherein the training includes: Obtain a training dataset, which includes multiple image-text sequence pairs; and The neural network is trained on the training dataset to minimize an overall loss function comprising: (i) a contrastive loss that measures the similarity between an image embedding generated from an encoded representation produced by the visual encoder neural network and a text embedding generated from the output of the language model neural network; and (ii) an image description generation loss that measures the quality of a text description generated by the language model neural network given an encoded representation produced by the visual encoder neural network.

2. The method of claim 1, further comprising: After training the neural network, it is used to perform downstream multimodal tasks.

3. The method as described in claim 2, wherein, Using the neural network to perform downstream multimodal tasks includes: A downstream neural network, comprising the visual encoder and the language model neural network, is trained on training data used for the downstream multimodal task.

4. The method of claim 3, wherein, Using the neural network to perform the downstream multimodal task includes: A trained downstream neural network is used to process the task input for the downstream multimodal task to generate the task output for the downstream multimodal task.

5. The method of claim 2, wherein, Using the neural network to perform a downstream multimodal task includes: using a trained neural network to process task inputs for the downstream multimodal task to generate task outputs for the downstream multimodal task.

6. The method of claim 5, wherein, Using the neural network to perform downstream multimodal tasks includes using the neural network to perform the downstream multimodal tasks without fine-tuning the trained neural network.

7. The method as described in any of the preceding claims, wherein, The language model neural network includes a sequence of layers, wherein each layer in the sequence receives a current representation of each text word in the current text sequence as input, and generates an updated representation of each text word in the current text sequence as output, and wherein the sequence of layers includes: One or more self-attention layers, each configured to receive a corresponding current representation of each text word in the current text sequence as input, and to process the corresponding current representation by applying a self-attention mechanism to the corresponding current representation of each text word in the current text sequence to generate a corresponding updated representation of each text word in the current text sequence as output; and One or more cross-attention layers, each configured to apply a cross-attention mechanism between the corresponding current representation of the text word in the current text sequence received as input by the cross-attention layer and the input derived from the encoded representation of the visual input.

8. The method of claim 7, wherein, The contrast loss measures the similarity between an image embedding generated from an encoded representation produced by the visual encoder neural network and a text embedding generated from the output of the language model neural network with one or more cross-attention layers in the language model neural network disabled.

9. The method of claim 8, wherein, The contrast loss measures the similarity between an image embedding generated from the encoded representation produced by the visual encoder neural network and a text embedding generated from the output of the language model neural network with one or more cross-attention layers disabled and the self-attention mechanism applied with bidirectional masking.

10. The method of claim 8 or claim 9, wherein, The image description generation loss measures the quality of the text description generated by the language model neural network with the cross-attention layer enabled and the self-attention mechanism applied, given the encoded representation generated by the visual encoder neural network.

11. The method according to any one of claims 7 to 10, wherein, Each text embedding is generated based on the updated representation generated by the last layer in the sequence.

12. The method according to any one of claims 7 to 11, wherein, Training the neural network includes: Obtain a batch of multiple image-text sequence pairs; The visual encoder neural network is used to process each image in the batch to generate an encoded representation of the image, and an embedding of the image is generated based on the encoded representation; The language model neural network with the cross-attention layer disabled is used to process each text sequence in the batch to generate an embedding of the text sequence; For each pair, the text sequence in the pair is processed using the language model neural network that has the cross-attention layer enabled and receives input derived from the encoded representation of the image in the pair and has a causal mask on the self-attention mechanism to generate a corresponding score for each word in the word sequence. The gradient of the contrast loss is determined, and the gradient of the contrast loss is calculated using the embeddings of the images in the batch and the embeddings of the text sequence in the batch; The gradient of the image description generation loss is determined, and the gradient of the image description generation loss is calculated for each pair using the corresponding score of each word in the word sequence; The overall gradient of the overall loss function is determined based on the gradient of the contrast loss and the gradient of the image description generation loss; and The neural network is trained using the global gradient.

13. The method according to any one of claims 7 to 10, wherein, Processing each text sequence in the batch to generate the image embedding using the language model neural network with the cross-attention layer disabled includes: The language model neural network, with the cross-attention layer disabled and bidirectional masking on the self-attention mechanism, is used to process each text sequence in the batch to generate the embedding of the image.

14. The method according to any one of claims 7 to 13, wherein, The corresponding current representation received as input by the first layer in the sequence is the corresponding embedding of each text word in the current text sequence, and The corresponding current representation received as input by each layer after the attention layer in the sequence is the corresponding updated representation of the text words in the current text sequence generated as output by the previous layer in the sequence.

15. The system as described in any of the preceding claims, wherein, The output, which is defined to be appended to the current text sequence, includes a score distribution that assigns a corresponding score to each text lexicon in the vocabulary of the text lexicons.

16. The method of claim 7, wherein, when subordinate to claim 15, The language model neural network includes a lexicon embedding layer configured to map the output of the last layer in the sequence to the score distribution.

17. The method as claimed in any of the preceding claims, wherein, The encoded representation includes a corresponding updated lexical for each of the multiple pieces of the visual input.

18. The method as claimed in any of the preceding claims, wherein, The visual encoder neural network is a visual Transformer neural network.

19. The method as claimed in any of the preceding claims, wherein, Each image embedding is generated by processing the corresponding encoded representation using one or more learned operations jointly learned with the training of the neural network.

20. The method of claim 20, wherein, Generating each image embedding includes: A linear layer is used to process the corresponding encoded representation, and the output of the linear layer is pooled; or Pooling the encoded representation to generate an initial embedding and applying a linear layer to the initial embedding to generate the image embedding.

21. The method as claimed in any of the preceding claims, wherein, The contrast loss is the focal contrast loss.

22. The method of any of the preceding claims, when subordinate to claim 18, wherein, The visual Transformer neural network uses cropped position embeddings.

23. A system comprising: One or more computers; as well as A storage device for storing one or more instructions, which, when executed by the one or more computers, cause the one or more computers to perform a corresponding operation of the method as described in any one of claims 1 to 22.

24. One or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform a corresponding operation of the method as claimed in any one of claims 1 to 22.

Images