Contrast commentary adding neural network
Through the design of CoCa neural network, single-stage pre-training is performed using comparative learning and commentary addition loss, which solves the problem of high computational overhead of pre-training multimodal model, and realizes high-efficiency, low-energy consumption pre-training and excellent performance of downstream tasks. .
Patent Information
- Application Number
- CN202380036307.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-05-03
- Filing Date
- 2023-04-28
- Publication Date
- 2024-12-06
AI Technical Summary
The prior art has high computational overhead in the pre-training process of multimodal models, resulting in high energy consumption and high carbon footprint, and it is difficult to effectively utilize supervision signals on multiple data sources and modalities.
A contrasting commentary addition (CoCa) neural network is proposed to pre-train by combining the contrasting learning loss and commentary addition loss, omitting the cross attention of the decoder layer, sharing the calculation graph to reduce computational overhead, and view all tags Unified training is used for text to achieve single-stage pre-training from scratch.
Significantly reduces the energy consumption and carbon footprint of the pre-training process, improves pre-training efficiency, enables performance equivalent to or better than traditional techniques with fewer FLOPs and training iterations, and supports zero-test or low-level task-specific adaptations The downstream tasks assigned.
Smart Images

Figure CN119096248A_ABST
Abstract
Description
Background Art
[0001] This specification involves processing input using a machine learning model.
[0002] As an example, a neural network is a machine learning model that uses one or more layers of nonlinear units to predict outputs for received inputs. In addition to the output layer, some neural networks also include one or more hidden layers. The output of each hidden layer is used as input to another layer in the network (e.g., the next hidden layer or the output layer). Each layer of the network generates an output from the received input according to the current value of the corresponding set of weights. Summary of the invention
[0003] This specification describes a system implemented as a computer program on one or more computers that processes multimodal inputs including both visual input (i.e., images or multiple video frames from a video) and text using a contrastive captioning neural network. As described below, the neural network is referred to as a "contrastive captioning" neural network because it can be pre-trained using both a contrastive learning loss and a captioning loss.
[0004] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.
[0005] This specification describes a contrastive caption addition (CoCa) neural network that has an architecture that allows the neural network to be pre-trained jointly using contrastive objectives and caption addition losses. Unlike a standard encoder-decoder transformer (in which all decoder layers focus on the encoder output), CoCa omits the cross-attention in the first set of decoder layers, so that the first set of decoder layers encodes a unimodal text representation. In other words, CoCa has a decoder (language model neural network) that has multiple initial self-attention layers without any cross-attention layers. CoCa then cascades cross-attention to the remaining decoder layers of the visual encoder to generate a multimodal image text representation. Therefore, CoCa effectively decouples the language model neural network into a unimodal text decoder, followed by a multimodal text decoder.
[0006] A contrastive loss is applied between unimodal visual and textual embeddings, while a caption addition loss is applied on the multimodal decoder output for predicting text tokens.
[0007] By sharing the same computational graph, i.e., the same neural network architecture, both training objectives can be computed efficiently with minimal computational overhead. That is, the number of computations required to compute both losses is obtained through a single forward pass through the CoCa network.
[0008] This allows neural networks to be pre-trained from scratch on image-text pairs in a unified form (e.g., including one or more of web-scale alternative text data or annotated images) in a single stage, seamlessly unifying natural language supervision for representation learning.
[0009] In other words, for each training pair, the system applies both a contrast objective between the outputs of the visual encoder and the unimodal text decoder, and a caption addition objective at the output of the multimodal decoder.
[0010] Furthermore, CoCa can be trained on both image annotation data and noisy image text data by simply treating all labels as text. Therefore, the generation loss on image annotation text provides a fine-grained training signal similar to the single encoder cross entropy loss method, effectively incorporating all three pre-training paradigms into a single unified method.
[0011] Furthermore, due to CoCa’s decoupled decoder (language model) design, two training losses can be considered efficiently. Since the unidirectional language model is trained on complete sentences with causal masking, the decoder can efficiently generate outputs for both the contrastive loss and the generative loss with a single forward pass (compared to two passes for bidirectional methods).
[0012] Therefore, most of the computation is shared between the two losses, and CoCa incurs only minimal overhead compared to the standard encoder-decoder model. On the other hand, while many existing methods utilize multiple stages to train model components on various data sources and / or modalities, CoCa directly utilizes various data sources (e.g., using both annotated images and noisy alternative text images) for pre-training from scratch end-to-end by treating all labels as text to achieve both contrastive and generative objectives.
[0013] Thus, the described techniques achieve improved pre-training efficiency, i.e., can achieve performance equal to or better than conventional techniques using fewer FLOPs and fewer training iterations.
[0014] Pre-training large multimodal models that can be used for real-world tasks usually results in large carbon dioxide (CO2) emissions and large electricity usage, for example because the datasets on which the pre-training is performed are extremely large and the models have a large number of parameters. By reducing the number of FLOPs that need to be performed and performing fewer training iterations for the reasons mentioned above, the described techniques significantly reduce the CO2 footprint of the pre-training process, while also significantly reducing the electricity consumed by the pre-training process.
[0015] Additionally, this pre-training scheme allows neural networks to achieve state-of-the-art performance on a wide range of downstream tasks through zero-shot transfer or minimal task-specific adaptation. Specific examples of downstream tasks are described in more detail below.
[0016] The details of one or more embodiments of the subject matter of this specification are set forth in the following drawings and the description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 An example neural network system is shown.
[0018] Figure 2 is a flow chart of an example process for training a contrasting transcript adding neural network.
[0019] Figure 3 The training of the contrast transcript addition neural network is shown.
[0020] Figure 4 The adaptation of the contrasting caption adding neural network to various downstream tasks is shown.
[0021] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION
[0022] Figure 1 An example neural network system 100 is shown. Neural network system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below may be implemented.
[0023] System 100 is a system that processes multimodal input including both visual input 102 (ie, an image or multiple video frames from a video) and text using a contrastive transcript addition neural network 110 .
[0024] As will be described below, neural network 110 is referred to as a “contrastive captioning” neural network because neural network 110 may be pre-trained using a combination of both a contrastive learning loss and a captioning loss.
[0025] The contrasting transcript adding neural network 110 comprises: (i) a visual encoder neural network 112 configured to process a visual input 102 comprising one or more images to generate an encoded representation 114 of the visual input 102; and (ii) a language model neural network 120 comprising a set of initial unimodal neural network layers 122 and a set of subsequent neural network layers 126, which include both cross-modal layers and unimodal layers.
[0026] That is, when processing a multimodal input including both a visual input 102 and a text sequence, the representation generated by the initial layer 122 within the language model neural network 120 is a unimodal representation 124 that depends only on the text sequence, while the representation generated by the subsequent layer 126 is a multimodal representation that depends on both the representation 124 of the text sequence generated by the initial layer 122 and the visual input 102.
[0027] Generally speaking, the language model neural network 120 is configured to process a current text sequence 104 to generate outputs defining new word-grams 128 to be appended to the current text sequence 104 .
[0028] The output of defining new word-grams 128 to be appended to the current text sequence 104 typically includes a respective score for each word-gram in a word-gram vocabulary. The word-gram vocabulary may include any of the following: characters, subwords, words, punctuation marks, symbol word-grams (e.g., #, $, and other symbols), mathematical symbols, etc. The word-gram vocabulary may also include one or more special word-grams appended to the input text sequence processed by the neural network, e.g., a start-of-sequence word-gram, an end-of-sequence word-gram, a designated "class" word-gram, etc.
[0029] During training, the language model neural network 120 may generate a corresponding output for each of multiple tokens in the input sequence in a single forward pass (ie, in parallel) by processing a single “current sequence” 104 that represents the entire input text sequence.
[0030] After training, the language model neural network 120 can be used to regressively generate text sequences by: processing the current text sequence 104 up to that time step at each time step; and then updating the current text sequence 104 by selecting a word-gram from the vocabulary using the output for the current text sequence and then appending the selected word-gram to the end of the current text sequence 104.
[0031] Visual encoder neural network 112 is a neural network having parameters (“visual encoder neural network parameters” or “visual encoder parameters”) and that receives visual input 102 and processes visual input 102 according to the parameters to generate an encoded representation 114 of visual input 102 .
[0032] In general, the encoded representation 114 includes a corresponding embedding (also referred to as an "updated word-gram") for each of a plurality of patches in the visual input 102 (e.g., for each of a plurality of spatial patches (regions) for each of an image of the visual input 102, or in some cases where the visual input 102 includes multiple images, for each of a plurality of spatiotemporal patches (regions) of the visual input 102).
[0033] As used in this specification, an "embedding" is a vector of numerical values (eg, floating point values or other values) having a predetermined number of dimensions. The space of possible vectors having a predetermined number of dimensions is referred to as an "embedding space."
[0034] The visual encoder neural network 112 may have any suitable architecture that allows the neural network 112 to map the input visual input 102 to the encoded representation 114. For example, the visual encoder neural network 112 may be a convolutional neural network. As another example, the visual encoder neural network 112 may be a visual Transformer neural network with one or more self-attention layers. As yet another example, the visual encoder neural network 112 may be a neural network with a mixture of both convolutional layers and self-attention layers.
[0035] The language model neural network 120 may have any suitable architecture that allows the language model neural network 120 to map tokens in a text sequence to corresponding unimodal representations 124 for each of the tokens, and then map the unimodal representations 124 to outputs that define the next token 128.
[0036] In a specific example, the language model neural network 120 can have an attention-based architecture, such as the architecture of a decoder-only Transformer neural network.
[0037] In this example, the set of initial unimodal neural network layers 122 may include a sequence of initial attention layers, wherein each initial attention layer is configured to receive a corresponding current representation of each of the text tokens in the current text sequence as input, and process the corresponding current representation to generate a corresponding updated representation of each of the text tokens in the current text sequence as output. For example, each initial attention layer may apply a causal masked self-attention mechanism with respect to the corresponding current representation to generate a corresponding updated representation.
[0038] The self-attention mechanism with respect to the corresponding current representation refers to the attention mechanism that computes the query, key, and value from the corresponding current representation.
[0039] The causal masked self-attention mechanism with respect to the corresponding current representation refers to the following attention mechanism, in which any given position in the current text sequence does not pay attention to any position after the given position in the current text sequence, that is, does not have a non-zero attention weight for any position after the given position.
[0040] Each attention layer can optionally apply other operations to the representation as part of updating the representation (e.g., by utilizing a position-wise feed-forward neural network, by applying layer normalization, by utilizing residual connections, etc.).
[0041] In this example, the corresponding current representations received as input by the first initial attention layer in the sequence of initial attention layers are the corresponding embeddings (e.g., as generated by the embedding layer of the language model neural network 120) of each of the text tokens in the current text sequence, and the corresponding current representations received as input by each subsequent initial attention layer (i.e., each initial attention layer after the first initial attention layer in the sequence of initial attention layers) are the corresponding updated representations of the text tokens in the current text sequence generated as output by the previous initial attention layer in the sequence of initial attention layers.
[0042] Therefore, the corresponding unimodal representation of the text word in the current text sequence is the corresponding updated representation of the text word in the current text sequence generated as output by the last initial attention layer in the sequence of initial attention layers.
[0043] More specifically, when the language model neural network 120 has an attention-based architecture, the initial attention layer includes multiple self-attention layers but does not include any cross-attention layers to ensure that the corresponding updated representation of the text tokens in the current text sequence is a unimodal representation that depends only on the current text sequence and not on the visual input.
[0044] In addition, the set of subsequent neural network layers 126 includes a sequence of subsequent attention layers, wherein each subsequent attention layer is configured to receive as input a corresponding current representation of each of the text tokens in the current text sequence, and to process the corresponding current representation to generate as output a corresponding updated representation of each of the text tokens in the current text sequence.
[0045] Therefore, the corresponding current representations received as input by the first subsequent attention layer in the sequence of subsequent attention layers are the corresponding unimodal representations (generated by the initial attention layer) of each of the text tokens in the current text sequence, and the corresponding current representations received as input by each subsequent attention layer after the first subsequent attention layer in the sequence of subsequent attention layers are the corresponding updated representations of the text tokens in the current text sequence generated as output by the previous subsequent attention layer in the sequence of subsequent attention layers.
[0046] In this example, as with the initial attention layer, the sequence of subsequent attention layers also includes one or more self-attention layers. That is, for one or more of the subsequent neural network layers, processing the corresponding current representation to generate a corresponding updated representation of each of the text tokens in the current text sequence as an output includes: applying a causal masked self-attention mechanism. Each of these attention layers can optionally apply other operations to the representation as part of updating the representation (e.g., by utilizing a position-by-position feedforward neural network, by applying layer normalization, by utilizing residual connections, etc.).
[0047] Unlike the initial layer, the sequence of subsequent attention layers also includes one or more cross-modal layers. Each cross-modal layer processes the corresponding current representation by applying a cross-attention mechanism between the input derived from the encoded representation of the visual input and the corresponding current representation of the text tokens in the current text sequence received as input by the cross-modal layer to generate a corresponding updated representation of each of the text tokens in the current text sequence as output.
[0048] "Cross attention" between the input derived from the encoded representation of the visual input and the corresponding current representations of text tokens in the current text sequence received as input by the cross-modal layer refers to an attention mechanism using keys and values derived from a query derived from the corresponding current representations of text tokens in the current text sequence and the input generated from the encoded representation of the visual input.
[0049] Specific examples of self-attention, cross-attention, and causal masking self-attention mechanisms that can be employed by the system are described in Vaswani et al., “Attention is all you need”, 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA; Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu, Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019; Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le’s Towards a human-like open-domain chatbot. CoRR, abs / 2001.09977, 2020; and Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, PranavShyam, Girish Sastry, Amanda Askell et al.’s Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020, Hua et al., Transformer Quality in Linear Time, arXiv preprint arXiv:2202.10447, 2022. .
[0050] In some implementations, the input derived from the encoded representation (also referred to below as the transcript representation) is an embedding in the encoded representation 114 .
[0051] In some other implementations, the neural network 110 applies one or more transformations to the encoded representation 114 to generate inputs provided to the cross-modal layer. Figure 3 An example of these transformations is described in more detail.
[0052] Thus, the updated representation generated by a given cross-modal layer is a multimodal representation that depends on the visual input and the text tokens in the current text sequence. Each of these cross-modal layers can optionally apply other operations to the representation as part of updating the representation (e.g., by utilizing a position-wise feed-forward neural network, by applying layer normalization, by utilizing residual connections, etc.).
[0053] As an example, the sequence of subsequent attention layers may alternate between self-attention layers and cross-modal layers. As another example, the sequence of subsequent attention layers may include a cross-modal layer after every two, three, or four self-attention layers.
[0054] Therefore, due to the presence of the cross-modal layer, the representation generated by the last subsequent attention layer in the sequence is a multimodal representation, as described above.
[0055] To generate a score distribution, the set of subsequent neural network layers 126 may also include an output layer block.
[0056] The output layer block is a collection of one or more neural network layers, e.g., one or more fully connected layers followed by a softmax layer, configured to receive one or more of the corresponding updated representations of text tokens in a current text sequence generated as output by a last subsequent attention layer in a sequence of subsequent attention layers, and to process the one or more corresponding updated representations to generate an output defining a new token to be appended to the current text sequence, i.e., to generate a score distribution within the tokens in the vocabulary.
[0057] For example, during training, when the current output sequence is the entire training sequence, the output layer block can generate a corresponding score distribution for each of the text tokens in parallel by processing, for each text token, an updated representation of the token immediately preceding the text token in the training sequence to generate a score distribution for the text token. In this example, the system can augment the training sequence with a specified sequence start token before processing the training sequence using a language model neural network.
[0058] After training, when the system 100 operates autoregressively, the output layer block can generate a single score distribution for the current output sequence by processing the updated representation of the last token in the current output sequence. The system 100 can then use the score distribution generated by the output layer block to select the next token to be added to the current output sequence. For example, the system 100 can select the token with the highest score in the score distribution, or can sample tokens from the score distribution.
[0059] In general, the system 100 or another training system may pre-train the contrastive captioning neural network 110 based on both the contrastive loss and the captioning loss.
[0060] The contrastive loss may depend on the encoded representation generated by the visual encoder 112 and the representation 124 generated by the initial neural network layer 122 , while the caption addition loss may depend on the encoded representation 124 generated by the visual encoder and the representation generated by the subsequent neural network layer 126 .
[0061] That is, due to the architecture of the neural network 110, the contrastive caption addition neural network 110 can be effectively pre-trained by jointly using both the contrastive loss and the caption addition loss without increasing the number of forward passes that need to be performed through the language model neural network 120 and the visual encoder neural network 112.
[0062] References below Figure 2 and Figure 3 The pre-trained neural network 110 is described in more detail.
[0063] After the comparison commentary adding neural network 110 has been pre-trained, the visual encoder 112, the initial layer 122, the subsequent layers 126, or some combination thereof can be used for downstream tasks.
[0064] In some implementations, the downstream tasks can be performed in a zero-trial fashion, ie, without further training any of the components of the contrast transcript addition neural network 110 .
[0065] In some other implementations, the downstream task can be performed after fine-tuning one or more of the components of the contrast caption adding neural network 110 on labeled training data for the downstream task.
[0066] For example, the system 100 can keep the visual encoder 112 and any portion of the language model neural network 120 used for a downstream task fixed, while learning a customized attention pooling layer and optionally one or more additional output layers that receive the output of the attention pooling layer, the output of one of the layers of the language model neural network 120, or both that are specific to the downstream task.
[0067] As another example, the system 100 may also fine-tune any portion of the visual encoder 112 and the language model neural network 120 for downstream tasks.
[0068] In some examples, the downstream task may be an image or video processing task.
[0069] In some examples, the downstream task is a visual classification task that requires classifying visual input into one of a set of categories, each corresponding to a different object type.
[0070] In some other examples, the downstream task is a visual action recognition task that requires classifying a video input into one of a set of action categories.
[0071] In some examples, the downstream task is a cross-modal retrieval task that requires (i) retrieving one or more text sequences that are most similar to a visual input or (ii) retrieving one or more visual inputs that are most similar to a text sequence.
[0072] In some examples, the downstream task is a multimodal understanding task. For example, the task may be a visual question answering task (VQA) that requires generating answers to questions posed about visual input.
[0073] In some examples, the downstream task is an image captioning task that requires generating a textual caption for a visual input.
[0074] References below Figure 4 Describe downstream tasks in more detail.
[0075] Figure 2 2 is a flow chart of an example process 200 for training a narration adding neural network. For convenience, process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a neural network system (e.g., Figure 1 The neural network system 100) can perform process 200.
[0076] The system can repeatedly perform iterations of process 200 on different batches of training examples to update parameters of the visual encoder neural network, the language model neural network, or both.
[0077] That is, at each iteration of process 200, the system obtains batches of training pairs (e.g., by sampling batches from a larger set of training data) and updates parameters of the visual encoder neural network and the language model neural network using the batches of one or more training pairs.
[0078] The system may continue to perform iterations of process 200 until a termination criterion for training of the neural network has been met, e.g., until parameters have converged, until a threshold amount of wall clock time has elapsed, or until a threshold number of iterations of process 200 have been performed.
[0079] The system obtains a batch of one or more training pairs (step 202).
[0080] Each training pair consists of a visual input and an input text sequence.
[0081] Specifically, the input text sequence has been determined by the system or an external source to describe the content of the visual input or otherwise be related to the visual input. In other words, the visual input and the input text sequence have been determined to be semantically similar.
[0082] For example, in a given training pair, the text sequence may be a text annotation of a visual input from a collection of manually or automatically generated image annotations, or may be an alternative text associated with the visual input from a collection of alternative text data. Alternative text is text that is displayed in place of an image on a web page, for example, if the image cannot be rendered correctly or otherwise fails to load. For example, the system may obtain alternative text data from data maintained by an Internet search engine or other software that automatically crawls web pages on the Internet.
[0083] For each training pair, the system processes the corresponding visual input and the corresponding text sequence in the training pair using the contrasting caption adding neural network (step 204).
[0084] Specifically, for each training pair, the system processes the visual input in the training pair using a visual encoder neural network to generate an encoded representation of the visual input.
[0085] The system processes the text sequences in the training pairs using a set of initial neural network layers to generate a corresponding unimodal representation of each of the text tokens in the text sequence. The representation is called "unimodal" because it depends only on the text tokens in the text sequence and not on the visual input.
[0086] The system processes the corresponding unimodal representations of the text tokens in the text sequence using a set of subsequent neural network layers to generate a corresponding score distribution over the vocabulary of the text tokens for each of a plurality of text tokens from the corresponding text sequence. As described above, the system can generate these score distributions for each of the plurality of text tokens in parallel.
[0087] For each training pair, processing the text sequence in the training pair using an initial set of neural network layers and processing the corresponding unimodal representation using a subsequent set of neural network layers is performed in a single forward pass through the language modeling neural network. That is, the system only needs to perform a single forward pass through the language modeling neural network to generate both the unimodal representation (to be used to calculate the contrastive loss) and the score distribution (to be used to calculate the commentary addition loss).
[0088] The system trains a neural network to minimize a loss function that includes (i) a contrastive learning loss term based on the similarity between a contrastive representation derived from an encoded representation of the visual input and one or more unimodal representations of text tokens from each of the text sequences in the training pair; and (ii) a caption addition loss term based on a corresponding score distribution of multiple text tokens in the corresponding text sequence for each training pair (step 206).
[0089] That is, the system can calculate the gradient of the loss function with respect to the parameters of the visual encoder neural network and the language model neural network, for example by back-propagation, and then apply the optimizer to the gradient to update the parameters of the visual encoder neural network and the language model neural network.
[0090] Because, as described above, the system only needs to perform a forward pass of words through the language model neural network to generate both the unimodal representations (which will be used to calculate the contrastive loss) and the score distribution (which will be used to calculate the caption addition loss), the system can calculate the quantities required for each of the two losses using different outputs of the same forward pass.
[0091] Therefore, even though the neural network is being trained on both the contrastive loss and the caption addition loss, only a single forward pass through the visual encoder neural network and the language model neural network is required to evaluate both losses.
[0092] In more detail, the contrastive loss is based on a “contrastive representation” of each of the visual inputs in the batch, and for each text sequence in the batch, based on one or more of the unimodal representations of one or more of the text sequences in the batch.
[0093] As a specific example, each text sequence in a batch can include the same designated word-gram located at the same position within each text sequence. For example, the system or another system can enhance each text sequence with a designated word-gram (e.g., a "CLS" word-gram placed at the end of each text sequence). The system can then calculate the contrastive loss using the unimodal representation of the designated word-gram.
[0094] Reference below Figure 3 Describe the computational contrast representation in more detail.
[0095] The goal of the contrastive loss is to train the visual encoder 112 and the language model 120 so that they can embed image and text inputs into a representation space (i.e., the space of contrastive representations and unimodal representations) so that inputs with similar semantics are mapped to nearby points regardless of their modality.
[0096] Thus, the system can train neural network 112 and neural network 120 for batches including visual input x i and the text sequence y i This encourages x i The embedding (i.e., contrastive representation) of i The embeddings of a given token (the unimodal representation of a given token) are closer together, while being further away from all other embeddings of all other visual inputs and text segments in the batch.
[0097] Next, a specific example of the contrast loss 130 will be described.
[0098] Based on the embeddings of the images and text segments in the mini-batch, we compute the N×N similarity matrix A, where A i;j It means x i The embedding of j The embedding similarity value of . For example, A i;j It can be x i The embedding of j The dot product between the embeddings of .
[0099] The system can then train the language model neural network and the visual encoder neural network using the gradient of the contrastive loss computed using the matrix A. For example, the contrastive loss can be a cross entropy loss over the rows and columns of A, where the diagonal entries are considered the correct class and the other entries are considered incorrect classes. A specific example of such a loss is: , in is a softmax temperature that scales logarithms (e.g., to steepen or attenuate the softmax distributions in the rows and columns of A), and N is the total number of training pairs in the batch. In some cases, before computing the matrix A, the system normalizes the contrastive and unimodal representations of the visual inputs and text sequences in the batch.
[0100] Since this loss is minimized, for all pairs in the batch, x i and i The embeddings of t become closer together, while becoming farther away from all other embeddings of all other visual inputs and text segments in the batch, thereby achieving the goal of contrastive learning.
[0101] For each training pair and for each of a plurality of tokens from a corresponding text sequence, a caption addition loss term measures the quality of the corresponding score distribution of the token relative to the corresponding token in the text sequence. As described above, since the score distribution is generated using the output of a subsequent attention layer of the language model neural network, the score distribution depends on both the visual input and the text sequence.
[0102] As a specific example, the caption addition loss for a given training pair can be given by: , where the overall caption addition loss is the average of the caption addition losses for the training pairs in a batch, T is the total number of positions in the training text sequence in the training pairs, and is the score distribution assigned to the token at position t in the training text sequence conditioned on the token preceding it and the visual input x in the training pair score.
[0103] The overall loss for pre-training can be, for example, a weighted sum of the caption addition loss and the contrastive loss.
[0104] Figure 3 is a diagram showing an example of training of the neural network 110 based on contrastive and caption-adding losses.
[0105] Specifically, Figure 3 An example of how to train the neural network 110 on a training pair comprising an image 310 and a text sequence 320 “two dogs running in a field” is shown.
[0106] like Figure 3 As shown, the system processes image 310 using visual encoder neural network 112 to generate an encoded representation of image 310.
[0107] The system then generates two representations from the encoded representation: a contrastive representation to be used for the contrastive loss as described above; and a caption-added representation to be used to condition the cross-modal layer within the language model neural network 120 (i.e., input derived from the encoded representation) as described above.
[0108] The system may generate the comparative representation in any of a variety of ways.
[0109] As an example, the system can use the embedding of a specified token within the encoded representation as a contrastive representation. That is, when dividing a given visual input into patches, the neural network 112 can add a placeholder patch that does not correspond to any of the patches in the visual input. The system can then use the embedding of this placeholder patch as a contrastive representation.
[0110] As another example, the system can generate a contrastive representation using pooling. As an example, the system can apply a pooling operation (e.g., global average pooling (GAP)) on the embeddings in the encoded representation, and use the resulting pooled embedding as the contrastive representation.
[0111] As yet another example, the system can generate contrastive representations using learned attention pooling.To perform learned attention pooling, the system can incorporate an attention pooling layer within the contrastive caption adding neural network 110.
[0112] The attention pooling layer focuses attention on the updated tokens (i.e., the embeddings in the encoded representation) and the set of learned query tokens to generate corresponding updated query tokens for each of the learned query tokens in the second set. That is, the system generates queries for the attention mechanism applied by the attention pooling layer using the learned query tokens, and generates keys and values for the attention mechanism using the embeddings in the encoded representation. Thus, the output of the attention pooling layer is an updated query token for each query token in the set of query tokens.
[0113] The set of learned query tokens and the parameters of the attention mechanism are learned jointly during pre-training using the parameters of the language model neural network and the visual encoder neural network.
[0114] To generate the contrastive representation, the system may use a set of learned query tokens having only a single query token, and may use updated query tokens of the single query token as the contrastive representation.
[0115] The system may generate the transcript addition representation in any of a number of ways.
[0116] As an example, the system may directly use the encoded representation as a caption to add representation.
[0117] As another example, the system may: incorporate another attention pooling layer whose learned set of query vectors has multiple query vectors; and then use the updated query tokens generated by the attention pooling layer as the encoded representation.
[0118] When used, attention pooling layers can act as “task adapters” that (i) ensure that the captioning task receives more fine-grained input that individually represents different regions within the visual input, while the contrasting task receives a global representation that represents the entire visual input; and (ii) allow the output of the visual encoder to adapt in a different learned way for each of the tasks, thereby improving the quality of pre-training in many cases.
[0119] The system then processes the text sequence 320 using the language model neural network 120 as described above. Specifically, the language model neural network 120 first generates unimodal representations by processing the text sequence 320 using an initial layer 122 ("unimodal text decoder"), and then processes those unimodal representations conditioned on the caption-added representations via cross-attention using a subsequent layer 126 ("multimodal text decoder") to generate outputs defining word units in the text sequence 320.
[0120] exist Figure 3 In the example of , the system then computes the contrastive loss using the unimodal representation of the "[CLS]" token and the contrastive representation generated using attention pooling, while computing the caption addition loss using the output of the tokens in the text sequence 320 as described above.
[0121] Figure 4 It is shown how the contrast caption adding neural network 110 can be used for various downstream tasks.
[0122] like Figure 4 As shown, the system first performs pre-training 402 of the "CoCa" neural network 110 as described above.
[0123] The system may then perform zero-trial, feature-freezing, or fine-tuning downstream adaptation 404 to utilize at least a portion of the CoCa neural network 110 for downstream tasks.
[0124] That is, in some implementations, the neural network 110 can adapt the downstream task in a zero-trial fashion (ie, without further training any of the components of the neural network 110 or any additional components compared to the transcript addition).
[0125] In some other implementations, the downstream task can be performed after fine-tuning one or more of the components of the contrast caption adding neural network 110 on labeled training data for the downstream task (eg, through supervised learning).
[0126] For example, for frozen feature adaptation, the system can keep the visual encoder 112 and any parts of the language model neural network 120 for the downstream task fixed, while learning a customized attention pooling layer for the task, and optionally, one or more additional output layers that receive the output of the attention pooling layer, the output of one of the layers of the language model neural network 120, or both specific to the downstream task.
[0127] As another example, for fine-tuning adaptation, the system may also fine-tune any portion of the visual encoder 112 and the language model neural network 120 for downstream tasks.
[0128] In some examples, the downstream task is a visual classification task 406 that requires classifying the visual input into one of a set of categories, each corresponding to a different object type. In this example, the system can use at least the visual encoder 112, and then process the encoded representation generated by the visual encoder 112 to generate the classification.
[0129] For example, the system can perform zero-trial visual classification by processing the text labels of a set of categories using a language model neural network to generate a unimodal representation of each category and processing the visual input using the visual encoder 112 to generate a contrastive representation of the visual input. The system can then select the category with the unimodal representation that is most similar to the contrastive representation as the classification of the visual input.
[0130] In some other examples, the downstream task is a visual action recognition task that requires classifying a video input into one of a set of action categories.
[0131] In these examples, the system can use only the visual encoder 112, and can learn a customized attention pooling layer and one or more output layers for the visual action recognition task. As a specific example, the system can take multiple frames of a video and feed each frame individually into the shared visual encoder. For frozen feature evaluation or fine-tuning, the system can learn an additional pooler on top of the spatial and temporal feature tokens using a softmax cross entropy loss. Note that the pooler has a single query token, so pooling all spatial and temporal tokens is not computationally expensive.
[0132] In some examples, the downstream task is a cross-modal alignment task 408 that requires (i) retrieving one or more text sequences that are most similar to a visual input or (ii) retrieving one or more visual inputs that are most similar to a text sequence. In these examples, the system can use the initial layers of the visual encoder neural network 112 and the language model neural network (unimodal text decoder). For example, for zero-try video text retrieval, the system can use a simple approach in which the system computes the average embedding of a collection of frames of a video (frames are uniformly sampled from the video) and uses the average embedding as a representation of the video.
[0133] In some examples, the downstream task is a multimodal understanding task 410. In these examples, the system can use the entire language model neural network 120 (unimodal decoder and multimodal decoder) and the visual encoder 112.
[0134] For example, the task may be a visual question answering task (VQA) that requires generating answers to questions posed about visual input.
[0135] As another example, a downstream task is an image captioning task that requires generating a text caption for a visual input.
[0136] Thus, after performing downstream adaptation 404, the system can receive new input for the downstream task and process the new input using the downstream task neural network to generate a task output for the downstream task. Depending on the downstream task, the downstream task neural network can include one or more of (i) a visual encoder, (ii) an initial set of neural network layers, or (iii) a subsequent set of neural network layers to generate the task output for the downstream task.
[0137] Table 1 shows an example of the performance of CoCa relative to the state of the art on two downstream tasks: image classification (left) and video action recognition (right). Table 1 shows the performance of downstream adaptation using a freezing technique (where components of CoCa are not further trained) and a fine-tuning technique (where components of CoCa are fine-tuned). Table 1
[0138] As can be seen from Table 1, the described technique is competitive with the state-of-the-art in the zero-fine-tuning case, and outperforms the state-of-the-art on both tasks in the fine-tuning case.
[0139] This specification uses the term "configuration" in connection with system and computer program components. For a system of one or more computers to be configured to perform a specific operation or action, it means that the system has installed thereon software, firmware, hardware, or a combination thereof that, in operation, causes the system to perform that operation or action. For one or more computer programs to be configured to perform a specific operation or action, it means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform that operation or action.
[0140] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, program instructions may be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical, optical, or electromagnetic signal), generating an artificially generated propagation signal to encode information for transmission to a suitable receiver device for execution by a data processing device.
[0141] The term "data processing equipment" refers to data processing hardware and encompasses all kinds of equipment, devices and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The equipment may also be or include a dedicated logic circuit system, for example, an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the equipment may optionally include code that creates an execution environment for a computer program, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
[0142] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script or code, may be written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages); and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, subroutines or code portions). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.
[0143] In this specification, the term "database" is used broadly to refer to any collection of data: the data need not be structured in any particular way, or at all, and may be stored on storage devices in one or more locations. Thus, for example, an index database may include multiple collections of data, each of which may be organized and accessed in a different way.
[0144] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally speaking, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same one or more computers.
[0145] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a dedicated logic circuit system (e.g., an FPGA or ASIC) or by a combination of a dedicated logic circuit system and one or more programmed computers.
[0146] A computer suitable for executing a computer program can be based on a general or special microprocessor or both, or any other kind of central processing unit. Generally speaking, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by a special logic circuit system or incorporated into the special logic circuit system. Generally speaking, the computer will also include one or more large-capacity storage devices for storing data, such as magnetic disks, magneto-optical disks or optical disks, or be operatively coupled to receive data from a large-capacity storage device or transfer data to a large-capacity storage device or both. However, the computer does not need to have such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.
[0147] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CD ROM and DVD-ROM disks.
[0148] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, a computer may interact with a user by transmitting documents to and receiving documents from a device used by a user; for example, by transmitting a web page to a web browser on a user's device in response to a request received from the web browser. In addition, a computer may interact with a user by transmitting a text message or other form of message to a personal device (e.g., a smart phone running a messaging application), and receiving a responsive message from the user in response.
[0149] A data processing device used to implement a machine learning model may also include, for example, dedicated hardware accelerator units for processing general-purpose and computationally intensive parts of machine learning training or production (i.e., inference, workloads).
[0150] Machine learning models can be implemented and deployed using a machine learning framework (e.g., the TensorFlow framework or the Jax framework).
[0151] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.
[0152] A computing system may include a client and a server. The client and the server are usually remote from each other and typically interact through a communication network. The relationship between the client and the server is due to computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server sends data (e.g., an HTML page) to a user device, for example, to display data to a user interacting with a device acting as a client and to receive user input therefrom. Data generated at a user device can be received from the device at the server, for example, due to user interaction.
[0153] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on the scope of any invention or the scope that may be claimed, but should be interpreted as descriptions of features that may be unique to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although features may be described above as working in certain combinations and even initially claimed as such, one or more features from the claimed combination may be deleted from the combination in some cases, and the claimed combination may involve a sub-combination or a variant of a sub-combination.
[0154] Similarly, although operations are depicted in the drawings and described in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in a continuous order, or that all of the operations shown be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
[0155] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or ordered order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.
Claims
1. A system comprising: One or more computers and one or more storage devices storing instructions, the instructions, when executed by the one or more computers, causing the one or more computers to implement a contrasting commentary adding neural network, the neural network comprising: a visual encoder neural network configured to process a visual input comprising one or more images to generate an encoded representation of the visual input; and A language model neural network, wherein the language model neural network is configured to process a current text sequence to generate an output defining a new word-gram to be appended to the current text sequence, wherein the current text sequence comprises a respective text word-gram at each of one or more input positions, and wherein the language model neural network comprises: a set of initial neural network layers configured to process an input including each text token in the current text sequence to generate a respective unimodal representation of each of the text tokens in the current text sequence, the unimodal representation being independent of the visual input; and a set of subsequent neural network layers configured to process inputs comprising the corresponding unimodal representations of the text tokens in the current text sequence to generate the output defining the new token to be appended to the current text sequence, wherein the subsequent neural network layers include one or more cross-modal layers conditioned on the encoded representation of the visual input.
2. The system of claim 1, wherein: The set of initial unimodal neural network layers comprises a sequence of initial attention layers, Each initial attention layer is configured to receive as input a respective current representation of each of the text tokens in the current text sequence, and to process the respective current representation to generate as output a respective updated representation of each of the text tokens in the current text sequence, the respective current representation received as input by a first initial attention layer in the sequence of initial attention layers is a respective embedding of each of the text tokens in the current text sequence, and The corresponding current representation received as input by each initial attention layer after the first initial attention layer in the sequence of initial attention layers is a corresponding updated representation of the text word in the current text sequence generated as output by the previous initial attention layer in the sequence of initial attention layers.
3. A system as claimed in claim 2, wherein the corresponding unimodal representation of the text word-gram in the current text sequence is the corresponding updated representation of the text word-gram in the current text sequence generated as output by the last initial attention layer in the sequence of initial attention layers.
4. The system of claim 2 or claim 3, wherein processing the respective current representations to generate as output respective updated representations of each of the text tokens in the current text sequence comprises: Applying self-attention mechanism with causal masking.
5. A system as claimed in any preceding claim, wherein defining the output of the new word-gram to be appended to the current text sequence comprises a score distribution that assigns a respective score to each text word-gram in a vocabulary of text word-grams.
6. A system as claimed in any preceding claim, wherein: The set of subsequent neural network layers comprises a sequence of subsequent attention layers, Each subsequent attention layer is configured to receive as input a respective current representation of each of the text tokens in the current text sequence, and to process the respective current representations to generate as output a respective updated representation of each of the text tokens in the current text sequence, the respective current representation received as input by a first subsequent attention layer in the sequence of subsequent attention layers is the respective unimodal representation of each of the text tokens in the current text sequence, and The corresponding current representation received as input by each subsequent attention layer after the first subsequent attention layer in the sequence of subsequent attention layers is a corresponding updated representation of the text word in the current text sequence generated as output by the previous subsequent attention layer in the sequence of subsequent attention layers.
7. The system of claim 6, wherein the set of subsequent neural network layers further comprises: An output layer block, the output layer block being configured to receive one or more of the corresponding updated representations of the text word-grams in the current text sequence generated as output by a last initial subsequent attention layer in the sequence of subsequent attention layers, and to process the one or more corresponding updated representations to generate the output defining the new word-gram to be appended to the current text sequence.
8. The system of claim 6 or claim 7, wherein for one or more of the subsequent neural network layers, processing the respective current representations to generate as output respective updated representations of each of the text tokens in the current text sequence comprises: Applying self-attention mechanism with causal masking.
9. The system of any one of claims 6 to 8, wherein the one or more cross-modal layers are each a respective one of the subsequent attention layers in the sequence of subsequent attention layers, and wherein for each cross-modal layer, processing the respective current representation to generate as output a respective updated representation of each of the text tokens in the current text sequence comprises: A cross-attention mechanism is applied between an input derived from the encoded representation of the visual input and the corresponding current representations of the text tokens in the current text sequence received as input by the cross-modal layer.
10. A system as claimed in any preceding claim, wherein the encoded representation comprises a respective updated word-gram for each of a plurality of patches of the visual input.
11. The system of claim 10, wherein the comparing commentary adding neural network further comprises: A first attention pooling layer that applies attention to the updated terms and a first set of learned query terms to generate a corresponding updated query term for each of the learned query terms, and wherein each cross-modal layer receives the corresponding updated query term as input.
12. The system of claim 11 when appended to claim 9, wherein the input derived from the encoded representation is the corresponding updated query token.
13. The system of any one of claims 10 to 12, wherein the visual encoder neural network is a visual Transformer neural network.
14. The system of any one of claims 10 to 13, wherein the comparison commentary adding neural network further comprises: A second attention pooling layer that applies attention to the updated terms and a second set of learned query terms to generate a corresponding updated query term for each of the learned query terms in the second set.
15. The system of claim 14, wherein the second set of learned query tokens includes only a single learned query token.
16. A method of training a comparative commentary adding neural network as claimed in any preceding claim, the method comprising: Obtaining a set of one or more training pairs, each of the one or more training pairs comprising a corresponding visual input and a corresponding text sequence; For each training pair, using the contrasting commentary adding neural network to process the corresponding visual input and the corresponding text sequence in the training pair, including: processing the visual input in the training pair using the visual encoder neural network to generate an encoded representation of the visual input; processing the text sequence in the training pair using the set of initial neural network layers to generate a respective unimodal representation of each of the text tokens in the text sequence; and processing the respective unimodal representations of the text tokens in the text sequence using the set of subsequent neural network layers to generate, for each text token in a plurality of text tokens from the respective text sequence, a respective score distribution over the vocabulary of the text tokens; and The neural network is trained to minimize a loss function comprising: (i) a contrastive learning loss term based on a similarity between a contrastive representation derived from the encoded representation of the visual input and a unimodal representation of one or more of the text tokens from each of the text sequences in the training pairs; and (ii) a caption-addition loss term based, for each training pair, on the respective score distributions of the plurality of text tokens in the respective text sequences.
17. The method of claim 16, wherein for each training pair, processing the text sequences in the training pair using the set of initial neural network layers and processing the corresponding unimodal representations using the set of subsequent neural network layers is performed in a single forward pass through the language model neural network.
18. A method as claimed in claim 16 or claim 17, wherein for each training pair and for each word-gram of the plurality of word-grams from the corresponding text sequence, the caption addition loss term measures the quality of the corresponding score distribution of the word-gram relative to the corresponding word-gram in the text sequence.
19. A method as described in any one of claims 16 to 18, wherein each training text sequence in each training pair includes the same designated word-gram, and wherein the contrastive learning loss term is based on the similarity between the contrastive representation derived from the encoded representation of the visual input in the training pair and the corresponding unimodal representation of the designated word-gram in the training text sequence in the training pair.
20. A method as claimed in any one of claims 16 to 19 as appended to claim 15, wherein the contrast representation of each visual input is the updated query token of a single query token in the second set.
21. A method as claimed in any one of claims 16 to 19 when dependent on claim 10, wherein the contrastive representation of each visual input is generated by pooling the corresponding updated word-grams for each of the plurality of patches of the visual input.
22. The method of any one of claims 16 to 21, further comprising: After the training, a downstream task is performed using one or more of (i) the visual encoder, (ii) the initial set of neural network layers, or (iii) the subsequent set of neural network layers.
23. The method of claim 22, further comprising: After the training and before performing a downstream task using one or more of (i) the visual encoder, (ii) the initial set of neural network layers, or (iii) the subsequent set of neural network layers, fine-tune one or more components of the contrast caption adding neural network on labeled training data for the downstream task.
24. The method of claim 22, further comprising: After the training and before performing a downstream task using one or more of (i) the visual encoder, (ii) the initial set of neural network layers, or (iii) the subsequent set of neural network layers, fine-tune a downstream neural network including the one or more components of the contrast caption adding neural network on labeled training data for the downstream task.
25. The method of claim 24, wherein fine-tuning the downstream neural network comprises training one or more additional components of the downstream neural network while keeping fixed one or more of (i) the visual encoder, (ii) the initial set of neural network layers, or (iii) the subsequent set of neural network layers.
26. A method performed by one or more computers, the method comprising: Receive new input for downstream tasks; as well as The new input is processed using a downstream task neural network, the downstream task neural network comprising one or more of (i) a visual encoder, (ii) an initial set of neural network layers, or (iii) a subsequent set of neural network layers for generating a task output for the downstream task, wherein the one or more of (i) a visual encoder, (ii) an initial set of neural network layers, or (iii) a subsequent set of neural network layers for generating a task output for the downstream task has been trained by performing the corresponding operations described in any one of claims 16 to 25.
27. One or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to implement the comparative commentary adding neural network of any one of claims 1 to 15.
28. A method comprising: The corresponding operations performed by the comparative commentary adding neural network according to any one of claims 1 to 15.
29. A system comprising: one or more computers; as well as One or more storage devices storing instructions which, when executed by the one or more computers, cause the one or more computers to perform corresponding operations of the method of any one of claims 16 to 26.
30. One or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform corresponding operations of the method of any one of claims 16 to 26.
Citation Information
Patent Citations
Training a document classification neural network
US10528866B1
