Systems and methods for visual and linguistic representation learning

Abstract

Embodiments described herein provide visual and language (V+L) systems and methods for learning visual and language representations. In particular, a method can include receiving a training dataset comprising a plurality of image samples and a plurality of text samples; encoding the plurality of image samples into a plurality of encoded image samples and the plurality of text samples into a plurality of encoded text samples; computing a first loss objective based on the plurality of encoded image samples and the plurality of encoded text samples; encoding a first subset of the plurality of encoded image samples and a second subset of the plurality of encoded text samples into a plurality of encoded image-text samples; computing a second loss objective based on the plurality of encoded image-text samples; and updating the V+L model based at least in part on the first loss objective and the second loss objective.

Description

[0001] Related applications

[0002] This application claims priority to U.S. nonprovisional application No. 17 / 370,524, filed July 8, 2021, and U.S. provisional patent application No. 63 / 193,286, filed May 26, 2021, the entire contents of which are expressly incorporated herein by reference. Technical Field

[0003] The publicly available texts involve machine learning models and neural networks as a whole, and more specifically, visual and language representation learning. Background Technology

[0004] Visual and pre-trained learning (VLP) aims to learn multimodal representations from large-scale image-text pairs, which can improve downstream vision and language (V+L) tasks such as predicting image-text retrieval, image-text relationships, visual question answering, or natural language visual reasoning. While VLP methods have made some progress in vision and language tasks, existing VLP frameworks may be subject to several limitations, including low modeling efficiency, high annotation and / or computational overhead, and / or overfitting to noise.

[0005] Therefore, there is a need for improved VLP systems and methods that avoid the drawbacks associated with existing VLP methods. Attached Figure Description

[0006] Figure 1 This is a simplified diagram of a computing device for implementing a vision and language pre-training (VLP) method according to the implementation scheme described herein.

[0007] Figure 2 This is a simplified diagram of the process flow for training a VLP system according to the implementation scheme described herein.

[0008] Figure 3 It is implemented based on some implementation schemes. Figure 1 A simplified logic flowchart of an exemplary method of a submodule in the example.

[0009] Figures 4A to 4B This is a simplified diagram of the model architecture for using a VLP system in downstream tasks, based on the implementation scheme described herein.

[0010] In the accompanying drawings, elements with the same reference numerals have the same or similar functions. Detailed Implementation

[0011] Machine learning methods have been applied to vision and language (V+L) tasks. These methods typically utilize vision and language pre-training (VLP) to learn multimodal representations from large-scale image-text pairs. However, existing VLP frameworks can be limited by several factors. First, image features and word token embeddings reside in very different spaces, which can make it difficult for multimodal encoders to learn to model the interactions between features and embeddings. Second, existing VLP frameworks require bounding box annotations for pre-training and / or high-resolution images, resulting in high annotation and / or computational overhead. Third, the image-text datasets used to train existing VLP methods can be noisy, leading to overfitting to noise and consequently, performance degradation.

[0012] Given the need for improved VLP systems and methods that avoid the drawbacks associated with existing VLP approaches, the implementation described in this paper provides VLP systems and methods, such as a method for pre-training V+L models utilizing intermediate image-text contrast (ITC) loss. For example, training inputs are fed into a unimodal image and text encoder to transform into unimodal outputs, and the ITC loss is computed by calculating the loss between the predicted similarity and ground truth similarity of the unimodal outputs from the image-text pairs. The ITC loss is computed at least in part based on the representations output by the unimodal image and text encoders, which can then be updated at least in part based on the ITC loss. Thus, image and text features can be aligned during the training process based on the ITC loss, making it easier for the multimodal encoder to perform cross-modal learning. Furthermore, training based on the ITC loss can improve the unimodal encoder's ability to understand the semantic meaning of images and text. A common embedding space for images and text can also be learned based on the ITC loss, enabling the image-text matching target to find samples containing more information.

[0013] In one implementation, the VLP system and method described herein employs momentum distillation (MoD) to generate pseudo-targets to capture visual concepts that may not be described by ground-value text. MoD leverages momentum models to generate pseudo-targets as additional supervision during training and feeds these pseudo-targets to train the image encoder, text encoder, and multimodal encoder, thereby allowing improved learning under noisy supervision and the use of larger unfixed training datasets.

[0014] As used herein, the term “network” can include any hardware- or software-based framework, including any artificial intelligence network or system, neural network or system, and / or any training or learning model implemented thereon or together with it.

[0015] As used herein, the term "module" can include a hardware- or software-based framework that performs one or more functions. In some implementations, a module may be implemented on one or more neural networks.

[0016] VLP Systems and Methods

[0017] Figure 1 This is a simplified diagram of a computing device used to implement a VLP system for training vision and learning (V+L) models, according to some implementation schemes. Figure 1 As shown, computing device 100 includes a processor 110 coupled to memory 120. The operation of computing device 100 is controlled by processor 110. Although computing device 100 is shown as having only one processor 110, it should be understood that processor 110 may represent one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), graphics processing units (GPUs), etc., within computing device 100. Computing device 100 may be implemented as a standalone subsystem, added to a board within a computing device, and / or a virtual machine.

[0018] Memory 120 may be used to store software executed by computing device 100 and / or one or more data structures used during operation of computing device 100. Memory 120 may include one or more types of machine-readable media. Some common forms of machine-readable media may include floppy disks, flexible disks, hard disks, magnetic tape, any other magnetic media, CD-ROMs, any other optical media, punched cards, paper tape, any other physical media with a perforated pattern, RAM, PROM, EPROM, flash memory EPROM, any other memory chip or cartridge, and / or any other media that a processor or computer is adapted to read from therefrom.

[0019] Processor 110 and / or memory 120 can be arranged in any suitable physical configuration. In some embodiments, processor 110 and / or memory 120 may be implemented on the same board, in the same package (e.g., system-in-package), on the same chip (e.g., system-on-a-chip), etc. In some embodiments, processor 110 and / or memory 120 may include distributed, virtualized, and / or containerized computing resources. According to this embodiment, processor 110 and / or memory 120 may reside in one or more data centers and / or cloud computing facilities.

[0020] In some examples, memory 120 may include a non-transitory tangible machine-readable medium containing executable code that, when run by one or more processors (e.g., processor 110), can cause said one or more processors to perform the methods described in more detail herein. For example, as shown, memory 120 includes instructions for VLP module 130, which can be used to implement and / or simulate systems and models, and / or implement any methods further described herein. In some examples, VLP module 130 may receive several inputs via data interface 115, such as image input 142 and text input 144 from a user. Data interface 115 may be any of a user interface that receives image and text input from a user, or a communication interface that can receive or retrieve image and text input from a database. VLP module 130 may generate output 150, such as one or more output image-text pairs.

[0021] In some embodiments, VLP module 130 includes an image encoder module 131 and a text encoder module 132. Specifically, the image encoder module is configured to form an encoding of image input 142. The text encoder module is configured to form an encoding of text input 144. In some embodiments, VLP module 130 includes a multimodal encoder 133. The multimodal encoder is configured to receive encodings of image input and text input. The multimodal encoder is configured to fuse the encodings of image input and text input. In some embodiments, VLP module 130 includes a momentum module 134. During training, the momentum module is configured to receive the output from the multimodal encoder and perform momentum distillation (MoD), which generates a pseudo-target of the output, such as an exponential moving average version of the output.

[0022] Some examples of computing devices (such as computing device 100) may include non-transitory tangible machine-readable media that include executable code. Some common forms of machine-readable media may be, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media with a perforated pattern, RAM, PROMs, EPROMs, flash memory EPROMs, any other memory chips or cartridges, and / or any other media that a processor or computer is adapted to read from therefrom.

[0023] Figure 2 This is a simplified diagram of the process flow for training a V+L model using one or more loss objectives, based on some implementation schemes. For example... Figure 2 As shown, image input 210 is passed to feedforward image encoder 212 to generate embedding 214. Input image I is encoded as a sequence of embeddings 214, such as {v cls ,v1,...vN}, where v cls This is an embedding of the [CLS] token. Text input 220 is passed to a feedforward text encoder 222 to generate embedding 224. For example, the text encoder transforms the input text T into a sequence of embeddings 224, such as {w cls ,w1,...w N}

[0024] V+L model 200 may include an image encoder 212, a text encoder 222, and a multimodal encoder 240. An image-text contrastive loss 230 may be generated to align the unimodal representations of image-text pairs before fusion at the multimodal encoder 240. An image-text matching loss 242 (using a hard negative example 250 mined through contrastive similarity) and a masked language modeling loss 244 are applied to learn the multimodal interactions between images and text. To improve learning in the presence of noisy data, a momentum model 260 (e.g., a moving average version of the base model) may be used as additional supervision to generate pseudo-targets during training of V+L model 200.

[0025] Image encoder 212 and text encoder 222 may include one or more feedforward layers and one or more self-attention layers. Multimodal encoder 240 may include one or more feedforward layers, one or more cross-attention layers, and one or more self-attention layers. For example, a 12-layer transformer may be used for image encoder 212, and a 6-layer transformer may be used for both text encoder 222 and multimodal encoder 240. Text encoder 222 is initialized using the first 6 layers of a BERT model, and multimodal encoder is initialized using the last 6 layers of a BERT model. Image features and text features can be fused through cross-attention at each layer of multimodal encoder 240.

[0026] Encoding 214 from image encoder 212 and encoding 224 from text encoder 222 are used to generate a first loss objective that includes an image-text contrastive learning (ITC) loss function 230, so that encoding 214 from image encoder 212 and encoding 224 from text encoder 222 can be aligned and compared. The purpose of image-text contrastive learning (ITC) is to learn a better unimodal representation before the fusion of encoding 214 from image encoder 212 and encoding 224 from text encoder 222.

[0027] To generate the image-text contrastive learning (ITC) loss for each image and text, the similarity between each image and text in multiple image-text pairs and the unpaired image and text can be generated. For example, this can be achieved by using a similarity function s = g v (v cls ) T gw (w cls This is used to calculate the image-to-text and text-to-image similarity for each image and each text pair, ensuring that paired images and texts have a high similarity score. v and g w It is a linear transformation that maps the [CLS] embedding of encoding 214 from image encoder 212 and the [CLS] embedding of encoding 224 from text encoder 222 to a normalized lower-dimensional (256-d) representation.

[0028] Image-Text Contrastive Learning (ITC) loss can be further used to merge the most recent M image-text representations of encoded image samples and encoded text samples from the autovariable unimodal encoder 260 into two queues. The normalized features from encodings 214 and 224 of the autovariable unimodal encoder 260 are represented as g′ v (v′ cls ) and g′ w (w′ cls ). We can use s(I,T)=g v (v cls ) T g′ w (w′ cls Image-text similarity can be calculated using s(T,I). To calculate text-image similarity. In certain aspects, image-to-text similarity and text-to-image similarity can be normalized using softmax as follows:

[0029]

[0030] Where τ is a learnable temperature parameter.

[0031] Truth-valued one-hot similarity can be expressed as and Negative pairs have a probability of 0, and positive pairs have a probability of 1.

[0032] The image-text contrastive learning (ITC) loss function is calculated as the average expected sum of the cross-entropy between the computed softmax-normalized image-to-text similarity and the labeled ground truth image-to-text similarity.

[0033] For example, the image-text contrast (ITC) learning loss can be defined as the cross-entropy H between the predicted similarity p and the ground truth one-hot similarity y between encoded image samples and encoded text samples, such as:

[0034]

[0035] In one implementation, the encodings from image encoder 212 and text encoder 222 are further passed to feedforward multimodal encoder 240 to generate encoded image-text samples. Multimodal encoder 240 is configured to generate a second loss objective, which includes an image-text matching (ITM) loss 242 and a masked language modeling (MLM) loss 244. ITM loss 242 is calculated based on the expected cross-entropy between the predicted image-text match of the encoded image-text sample and its corresponding ground truth image-text match. ITM loss 242 can be generated using a hard negative example 250 obtained through contrastive similarity mining of the encoded image-text samples.

[0036] Image-text matching (ITM) 242 predicts the two-class probabilities of an encoded image-text sample, for example, whether a pair of images and text in the encoded image-text sample is positive (matched) or negative (mismatched). The output embedding of the [CLS] token of the multimodal encoder 240 can be used as a joint representation of a pair of images and text in the encoded image-text sample, and can be appended with a fully connected (FC) layer, followed by a softmax function, to predict the two-class probabilities p of that image-text pair. itm (That is, whether the image-text pair is positive or negative). The ITM loss can be the cross-entropy H between the predicted two-class probabilities of the image-text pair and the ground truth one-hot two-class probabilities, such as:

[0037]

[0038] Where y itm It is a 2-dimensional one-hot vector representing the truth label.

[0039] The multimodal encoder 240 is also configured to generate a masked language modeling (MLM) loss 244 to learn the multimodal interaction between the image input 210 and the text input 220. The MLM loss 244 can be defined as a loss function between the predicted probabilities of one or more masking tokens in the encoded image-text sample and the ground truth identifiers of the one or more masking tokens in the encoded image-text sample.

[0040] Masked Language Modeling (MLM) uses both the image from the encoded image-text sample and the contextual text to predict masked words in the encoded image-text sample. The input token can be randomly masked with a predetermined probability (such as 15%) and replaced with a special token [mask]. For example, the replacement is 10% random token, 10% unchanged and 80% [mask].

[0041] The MLM learning loss 244 can be the cross-entropy H between the predicted probabilities of masked tokens in encoded image-text samples and the distribution of ground-valued one-hot vocabularies, such as:

[0042]

[0043] in It can be used to represent masked text. It can be used to represent the model prediction probability of a masking token, and y msk It is a unique hot word distribution, where the truth token has a probability of 1.

[0044] Before being encoded into encoded image-text samples by a multimodal encoder, a subset of encoded image and text samples can be selected, at least in part, based on negative example mining. Hard negative examples can be sampled for the ITM task with zero computational overhead. Negative image-text pairs are hard if they share similar semantics but differ in fine-grained details. The contrastive similarity from Equation (1) can be used to find hard negative examples. For each image in a mini-batch, a negative text can be sampled from the same batch following the contrastive similarity distribution, where text more similar to the image has a higher chance of being sampled. Similarly, a hard negative example image can be sampled for each text.

[0045] In some implementations, the visual and language learning (V+L) model will be updated at least in part based on a first loss objective and a second loss objective, for example, based on a combination of the first and second loss objectives, such as a combination of ITC loss, MLM loss, and ITM loss, expressed as:

[0046]

[0047] In one implementation, the final loss target can be a weighted sum of ITC loss, MLM loss, and ITM loss, where weighting coefficients are configured empirically or predefined.

[0048] In one implementation, to improve learning, momentum distillation (MoD) is used as an alternative to the original noisy input data used for training the model to generate pseudo-targets, such as when noisy input data is available for training the model. For all encoders (e.g., image encoder 212, text encoder 222, and multimodal encoder 240), the pseudo-targets are generated by momentum model 260. The momentum model is a continuously evolving teacher model that comprises exponential moving average versions of all encoders (including unimodal and multimodal encoders).

[0049] During training, basic vision and language models can be trained to match their predictions with those from the momentum model. Specifically, to modify ITC, pseudo-targets generated by the momentum model can be used to adjust image-text similarity, such as... Similarly, pseudo-targets generated by momentum models can be used to adjust text-image similarity, such as... The soft pseudo-target q can be generated by replacing s with s' in formula (1). i2t and q t2i The ITC-MoD loss can be generated by modifying the ITC using a MoD pseudo-target, such as by defining it as:

[0050]

[0051] Similarly, to modify the MLM, one can do so by, for example, through The momentum model predicts the probability of generating masking tokens. MLM-MoD loss can be generated by modifying the MLM with a MoD spurious target, such as defined as:

[0052]

[0053] In some implementations, the visual and language learning (V+L) model will be updated at least in part based on a first loss target and a second loss target, for example, based on a combination of a first loss target and a second loss target modified by a pseudo-target generated by a momentum model.

[0054] Figure 3 It is based on some implementation schemes shown for implementation Figure 1 A simplified logic flowchart of the visual and language representation learning method 300 in submodules 131 to 134 is provided. One or more of the processes 310 to 360 of method 300 may be implemented, at least in part, in the form of executable code stored on a non-transitory tangible machine-readable medium, which, when run by one or more processors, causes said one or more processors to execute one or more of the processes 310 to 360. In some embodiments, method 300 may correspond to the method used by module 130.

[0055] At process 310, it can be, for example, via Figure 1 The data interface 115 receives a training dataset comprising multiple image samples and multiple text samples. In some embodiments, at least one of the multiple image samples corresponds to at least one of the multiple text samples;

[0056] At process 320, the image encoder can encode the plurality of image samples into a plurality of encoded image samples. At process 320, the text encoder can encode the plurality of text samples into a plurality of encoded text samples. The encoding by the image encoder or the text encoder can occur simultaneously or at different times. For example, the encoding by the image encoder can occur before the encoding by the text encoder. For another example, the encoding by the image encoder can occur after the encoding by the text encoder. In some embodiments, the image encoder is a transformer. In other embodiments, the text encoder is a transformer.

[0057] At process 330, a first loss target can be calculated based on the plurality of encoded image samples and the plurality of encoded text samples. The first loss target may include an image-text contrast loss (ITC) target, which refers to a loss function between the predicted similarity and the corresponding ground truth similarity between the encoded image samples and the encoded text samples.

[0058] In additional and alternative embodiments, method 300 or process 330 may further include: forming a momentum model using momentum distillation (MoD); generating a plurality of modeled image samples and a plurality of modeled text samples using the momentum model; including the plurality of modeled image samples in the plurality of image samples; including the plurality of modeled text samples in the plurality of text samples; and using the modeled image samples and modeled text samples to generate a first target, such as an ITC loss target.

[0059] At process 340, the multimodal encoder can encode a first subset of the plurality of encoded image samples and a second subset of the plurality of encoded text samples into a plurality of encoded image-text samples. In some embodiments, the multimodal encoder is a transformer. The first and second subsets can be selected at least in part based on negative example mining or mining negative image-text pairs that share similar semantics but differ in fine-grained details. Negative image-text pairs can be selected at least based on the contrastive similarity distribution from Equation (1).

[0060] At process 350, a second loss objective can be computed based on the plurality of encoded image-text samples, and the second loss objective includes an image-text matching (ITM) loss objective and a masked language modeling (MLM) loss objective. The ITM loss can be a loss function between the predicted image-text match of the encoded image-text sample and the corresponding ground truth image-text match of the encoded image-text sample. The MLM loss can be a loss function between the prediction of the masking token in the encoded image-text sample and the ground truth vocabulary distribution of the encoded image-text sample.

[0061] In additional and alternative implementations, method 300 or process 350 may also include using modeled image samples and modeled text samples derived from an autovariable model to generate a second objective, such as an MLM loss objective.

[0062] At process 360, the V+L model can be updated at least partially based on a first loss objective and a second loss objective. For example, updating the V+L model includes updating the image encoder, text encoder, and multimodal encoder based on a combination of the first and second loss objectives. In another example, updating the V+L model includes updating the image encoder and text encoder at least partially based on the first loss objective; and updating the multimodal encoder at least partially based on the second loss objective.

[0063] In other implementations, method 300 may also include fine-tuning the V+L model for tasks selected from the group consisting of: image-text retrieval task, image-to-text retrieval (TR) task, text-to-image retrieval (IR) task, visual entailment (VE) task, visual question answering (VQA) task, and real natural language visual reasoning (NLVR) task.

[0064] In one implementation, the lower bound of mutual information (MI) between different “views” of an image-text pair can be maximized.

[0065] Formally, given two random variables a and b, mutual information (MI) measures their dependence and is defined as follows:

[0066] MI(a,b)=H(α)-H(α|b)-H(b)-H(b|a)(8)

[0067] A self-supervised learning method has been proposed to maximize the lower bound of mutual information, called InfoNCE:

[0068]

[0069] Where s(a,b) is the scoring function (e.g., the dot product between two representations), and Includes positive samples b extracted from the proposed distribution and Negative example samples. An alternative version of the ITC loss with one-hot labels (a variation of formula (2)) could be:

[0070]

[0071] make Minimization can be viewed as maximizing a symmetric version of InfoNCE. Therefore, ITC treats the two modalities (i.e., I and T) as different views of the image-text pair and trains a unimodal encoder to maximize the MI between the image and text for each positive pair.

[0072] MLM can be interpreted as maximizing the MI between the masking word token and its masking context (i.e., image + masking text). Specifically, an alternative version of the MLM loss with one-hot labels (a variation of Equation (3)) could be:

[0073]

[0074] in It is a lookup function that maps word tokens y to vectors, and It is a complete vocabulary set, and It is a function that returns the final hidden state of the multimodal encoder corresponding to the masking token. Therefore, MLM considers the two views of the image-text pair to be: (1) a randomly selected word token, and (2) the image plus the context text with the word masked.

[0075] Both ITC and MLM generate views by extracting partial information from image-text pairs. Momentum distillation improves upon both ITC and MLM by generating distinct views from the entire proposed distribution. For ITC, alternative views of image-text pairs are generated by finding semantically similar images and text in the training dataset. For MLM, alternative views of masked words are generated from the entire vocabulary set. Therefore, MoD can be considered as performing data augmentation on the original views. MoD generates a set of distinct views that do not exist in the original image-text pairs, which can improve the generalization performance of the model.

[0076] Exemplary system architecture and performance

[0077] Exemplary experiments are conducted to evaluate the performance of VLP systems (e.g., pre-trained vision and learning models or V+L models) on downstream tasks. In some implementations, the pre-trained V+L model can be fine-tuned and applied to one or more downstream tasks, including image-text retrieval, visual entailment, visual question answering, and real natural language visual reasoning.

[0078] The V+L model consists of a BERT with 123.7M parameters and a ViT-B / 16 with 85.8M parameters. The model was pre-trained for 30 epochs on eight NVIDIA A100 GPUs using a batch size of 512. The AdamW optimizer with a weight decay of 0.02 was used. Further details of the AdamW optimizer are provided in Loshchilov’s “Decoupled Weight Decay Regularization” (arXiv preprint arXiv:1711.05101, 2017), which is explicitly incorporated herein by reference in its entirety. The learning rate was preheated to 1e in the first 1000 iterations. -4 And decays to 1e according to cosine modulation. -5 .

[0079] For example, the pre-training data was generated using two web datasets (concept titles and SBU titles) and two in-domain datasets (COCO and Visual Genome). The total number of unique images was 4.0M, and the number of image-text pairs was 5.1M. To demonstrate that the V+L model can scale with larger-scale web data, a noisier 12M concept dataset was also included, increasing the total number of images to 14.1M. 2 .

[0080] During pre-training, random image clippings with a resolution of 256×256 were used as input, and RandAugment was also applied. Further details of RandAugment are provided in "RandAugment: Practical automated data augmentation with areduced search space," published by Cubuk et al. at the 2020 CVPR Workshops, pp. 702-703, which is explicitly incorporated herein by reference in its entirety. Color variations were removed from RandAugment because text typically contains color information.

[0081] During fine-tuning, the image resolution was increased to 384×384, and positional encoding of image patches was interpolated. The momentum parameter used to update the momentum model was set to 0.995, and the queue size used for image-text contrastive learning was set to 65536. The distillation weight α increased linearly during the first epoch.

[0082] Image-text retrieval comprises two subtasks: image-to-text retrieval (TR) and text-to-image retrieval (IR). The V+L model is evaluated on the Flickr30K and COCO benchmarks after fine-tuning using training samples from each of the Flickr30K and COCO datasets. For zero-shot retrieval on Flickr30K, the V+L model fine-tuned on COCO is evaluated.

[0083] During fine-tuning, the ITC loss (Equation (2)) and the ITM loss (Equation (4)) are jointly optimized. ITC learns the image-text scoring function based on the similarity of unimodal features, while ITM models the fine-grained interaction between the image and the text to predict the matching score. Since the downstream dataset contains multiple texts for each image, the ground truth label for ITC is modified to consider multiple positives in the queue, where each positive has a ground truth probability of 1 / # positive.

[0084] During inference, feature similarity scores s are first calculated for all image-text pairs. itc Then select the top k candidates and use them to calculate their ITM scores. itm The V+L model is used for sorting. Its inference speed is much faster than methods that require calculating ITM scores for all image-text pairs.

[0085] Visual entailment (SNLI-VE) is a fine-grained visual reasoning task that predicts whether the relationship between an image and text is entailed, neutral, or contradictory. Visual entailment can be considered a three-way classification problem. Multilayer perceptrons (MLPs) can be used to predict class probabilities on the multimodal encoder representation of [CLS] tokens.

[0086] Visual question answering (“VQA”) requires a model to predict the answer given an image and a question. Unlike existing writings that formulate VQA as a multi-answer classification problem, VQA can be framed as an answer generation problem. Specifically, a 6-layer transformer decoder can be used to generate the answer.

[0087] Figures 4A to 4B This is a simplified diagram of the model architecture for using a VLP system in downstream tasks, based on the implementation scheme described herein. Figure 4A As shown, will be with Figure 2A essentially identical model is used for visual question answering, the difference being the addition of an autoregressive decoder 450 to generate an answer given an image-question embedding. An image encoder 420 encodes an image input 410 into an image embedding, and a text encoder 422 encodes a question input 412 into a question embedding. The image embedding is passed to a multimodal encoder 430 via a cross-attention input 440 to generate a multimodal image-question embedding using the question embedding from the text encoder 422. The autoregressive answer decoder 450 receives the multimodal image-question embedding via the cross-attention input 440, and a sequence start token ([CLS]) 460 is used as the initial input token for the decoder. Similarly, a sequence end token ([SEP]) is appended to the end of the decoder output, indicating completion of generation. The answer decoder 450 is initialized using pre-trained weights from the multimodal encoder 430 and fine-tuned using a language modeling loss. To ensure a fair comparison with existing methods, the answer decoder 450 is restricted to generating only from 3,192 candidate answers during inference.

[0088] like Figure 4B As shown, real-world natural language visual inference uses a model to predict whether text accurately describes a pair of images. A natural extension can be made to the multimodal encoder 470, enabling inference on two images 490 and 492. These two images 490 and 492 can be fed into two image encoders 494 and 496, which share all parameters to generate embeddings fed into the multimodal encoder 470. Text input 475 can also be fed into a text encoder 485 to generate embeddings that enter the multimodal encoder 470. Each layer of the multimodal encoder 470 is replicated as having two consecutive transformer blocks 480, each block containing a self-attention layer, a cross-attention layer, and a feedforward layer (see [link to documentation]). Figure 2 The multimodal blocks 480 can also share cross-attention layers. The two multimodal blocks 480 within each layer are initialized using the same pre-trained weights, and the two cross-attention layers share the same linear projection weights for keys and values.

[0089] During training, the two multimodal blocks 480 receive two different sets of image embeddings for image pairs 490 and 492. The MLP classifier can learn the multimodal encoder representation of the [CLS] token to predict "true" or "false".

[0090] Additional pre-training steps can be performed to prepare a new multimodal encoder for image-pair inputs. The text assignment (TA) task can be designed as follows: given a pair of images and text, the model needs to assign the text to the first image, the second image, or not assign the text to either. This can be considered a three-way classification problem, and fully connected (FC) layers can be used for [CLS] representation to predict the assignment category. Using 4M images, the model is pre-trained using text alignment (TA) for only one epoch.

[0091] As shown in Table 1, the V+L model is evaluated in downstream tasks (including image-text contrastive learning, contrastive hard negative example mining, and momentum distillation). Table 1 shows the performance of different variants of the V+L model on downstream tasks. Adding ITC significantly improves the performance of the pre-trained model across all tasks compared to the baseline pre-trained task (MLM+ITM). The proposed hard negative example mining improves ITM by finding training samples with more information. Furthermore, it performs well for both ITC and MLM, as well as for all downstream tasks (image-to-text retrieval or TR, text-to-image retrieval or IR, visual entailment or VE, visual question answering or VQA, and real natural language visual reasoning or NLVR). 2 Adding momentum distillation improves learning. The V+L model can efficiently utilize noisier Web data, such as 14M pre-training images, to improve pre-training performance.

[0092] Table 1: Evaluation of the V+L model on four downstream V+L tasks.

[0093]

[0094]

[0095] Table 1 reports the average R@1, R@5, and R@10 values ​​for text retrieval (TR) and image retrieval (IR). Furthermore, in Table 1, ITC refers to Image-Text Contrast Learning, MLM refers to Masked Language Modeling, and ITM... 难 Refers to image-text matching mined using contrastive hard negative examples. MoD: Momentum distillation.

[0096] Tables 2 and 3 report the results for fine-tuned and zero-shot image-text retrieval, respectively. The V+L model achieves state-of-the-art performance, outperforming other methods trained on significantly larger datasets. Given a substantial improvement in the V+L model, it can be trained on a much larger scale of Web image-text pairs, increasing the number of training images from 4M to 14M.

[0097] Table 2: Fine-tuned image-text retrieval results on the Flickr30K and COCO datasets.

[0098]

[0099] Table 3: Results of zero-sample image-text retrieval on Flickr30K.

[0100]

[0101]

[0102] Table 4 reports a comparison with existing methods for other V+L understanding tasks. Using 4M pre-trained images, the V+L model achieves state-of-the-art performance. Using 14M pre-trained images, the V+L model significantly outperforms existing methods, including those requiring additional object labels or adversarial data augmentation. Compared to VILLA, the V+L model achieves an absolute improvement of 2.47% on the VQA test-std and on NLVR. 2 It achieves an absolute improvement of 3.84% on the Test-P and an absolute improvement of 1.88% on the SNLI-VE test. Because the V+L model is detector-free and requires lower resolution images, it also enjoys a much faster inference speed compared to existing methods (more than 10 times faster than UNITER or VILLA).

[0103] Table 4: Comparison with existing techniques for downstream visual-language tasks.

[0104]

[0105] Visual localization aims to locate regions in an image that correspond to specific text descriptions. The V+L model is shown to achieve visual localization by probing its attention without requiring training with any bounding box annotations. Experiments are conducted on the widely used RefCOCO+ dataset. The pre-trained model is fine-tuned on the RefCOCO+ training set using only image-text supervision. The same fine-tuning strategy is followed for image-text retrieval. The results are reported in Table 5.

[0106] Table 5: Weakly supervised visual localization on the RefCOCO+ dataset.

[0107]

[0108] Table 6 examines the impact of various design choices on image-text retrieval. This is because contrastive similarity scores are used during inference. itc To filter the top k candidates, k can be changed to report its impact. Generally, s itm The final sorted result is insensitive to changes in k. This is because, using s alone... itcGood recall has been achieved, therefore the top k candidates are most likely to contain the correct candidate. It has also been confirmed that the proposed hard-negation example mining can improve retrieval performance.

[0109] Table 6: Ablation study on fine-tuning image-text retrieval.

[0110]

[0111] Table 7 examines the impact of text assignment (TA) pre-training and parameter sharing on NLVR. 2 The impact of TA was tested. Three sharing strategies were tested: (1) two consecutive multimodal blocks sharing all parameters, (2) sharing only the cross-attention (CA) layer, and (3) no sharing. Sharing the entire block achieved better performance without TA. Sharing the cross-attention layer achieved the best performance when using TA to pre-train a model for image-pair inputs.

[0112] Table 7: Ablation study of the impact of text assignment (TA) pre-training and parameter sharing on NLVR2.

[0113]

[0114]

[0115] This specification and the accompanying drawings illustrating aspects of the invention, embodiments, implementations, or applications should not be considered limiting. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of this specification and the claims. In some cases, well-known circuits, structures, or techniques have not been shown or described in detail to avoid obscuring the embodiments in the disclosure. Similar reference numerals in two or more figures denote identical or similar elements.

[0116] This specification sets forth specific details of some embodiments consistent with the disclosed text. Numerous specific details are set forth to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that some embodiments can be practiced without some or all of these specific details. The specific embodiments disclosed herein are intended as illustrative rather than limiting. Those skilled in the art will recognize that other elements fall within the scope and spirit of the disclosed text, although such other elements are not specifically described herein. Furthermore, to avoid unnecessary repetition, the one or more features shown and described in association with one embodiment may be incorporated into other embodiments unless otherwise specifically described or when one or more features would render an embodiment ineffective.

[0117] Although exemplary embodiments have been shown and described, various modifications, alterations, and substitutions are contemplated within the foregoing disclosure, and in some cases, some features of the embodiments may be employed without correspondingly using other features. Many variations, substitutions, and modifications will be recognized by those skilled in the art. Therefore, the scope of the invention should be defined only by the appended claims and should be interpreted broadly in accordance with the scope of the embodiments disclosed herein.

Claims

1. A method for training a visual and language learning model, said visual and language learning model including an image encoder, a text encoder, and a multimodal encoder, said method comprising: The system receives a training dataset comprising multiple image samples and multiple text samples via a data interface, wherein at least one of the multiple image samples corresponds to at least one of the multiple text samples; The plurality of image samples are encoded into a plurality of encoded image samples by an image encoder, and the plurality of text samples are encoded into a plurality of encoded text samples by a text encoder; Based on the plurality of encoded image samples and the plurality of encoded text samples, a first loss target is calculated; A multimodal encoder encodes a first subset of the plurality of encoded image samples and a second subset of the plurality of encoded text samples into a plurality of encoded image-text samples; The first subset and the second subset are selected based at least in part on mining negative image-text pairs through the contrastive similarity of the encoded image-text samples; Based on the multiple encoded image-text samples, a second loss target is calculated; and The visual and language learning models are updated based at least in part on the first loss target and the second loss target.

2. The method according to claim 1, wherein the first loss objective includes an image-text contrast loss objective, wherein the image-text contrast loss objective is the average expected sum of the cross-entropy between the calculated softmax-normalized image-to-text similarity and the labeled ground truth image-to-text similarity and the calculated softmax-normalized text-to-image similarity and the labeled ground truth text-to-image similarity.

3. The method of claim 1, wherein the second loss objective comprises an image-text matching loss objective and an MLM loss objective, the image-text matching loss objective being calculated as the cross-entropy between the predicted two-class probabilities of the image-text pair and the ground truth one-hot two-class probabilities, and the MLM loss objective being calculated as the cross-entropy between the predicted probabilities of one or more masking tokens in the encoded image-text sample and the ground truth identifiers of the one or more masking tokens in the encoded image-text sample.

4. The method of claim 1, wherein updating the visual and language learning model comprises: The image encoder and the text encoder are updated at least in part based on the first loss target; and The multimodal encoder is updated based at least in part on the second loss objective.

5. The method according to claim 1, further comprising: Momentum models are formed using momentum distillation; The momentum model was used to generate multiple modeled image samples and multiple modeled text samples; The plurality of modeled image samples are included in the plurality of image samples; and The plurality of modeled text samples are included in the plurality of text samples.

6. The method of claim 1, wherein the image encoder, the text encoder, and the multimodal encoder each include a converter.

7. The method according to claim 1, further comprising: For tasks selected from the following groups, the visual and language learning models are fine-tuned: image-text retrieval task, image-to-text retrieval task, text-to-image retrieval task, visual entailment task, visual question answering task, and real natural language visual reasoning task.

8. A system for training visual and language learning models, comprising: Non-transitory memory; and One or more processors, coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform operations, said operations including: The system receives a training dataset comprising multiple image samples and multiple text samples via a data interface, wherein at least one of the multiple image samples corresponds to at least one of the multiple text samples; The plurality of image samples are encoded into a plurality of encoded image samples by an image encoder, and the plurality of text samples are encoded into a plurality of encoded text samples by a text encoder; Based on the plurality of encoded image samples and the plurality of encoded text samples, a first loss target is calculated; A multimodal encoder encodes a first subset of the plurality of encoded image samples and a second subset of the plurality of encoded text samples into a plurality of encoded image-text samples; The first subset and the second subset are selected based at least in part on mining negative image-text pairs through the contrastive similarity of the encoded image-text samples; Based on the multiple encoded image-text samples, calculate the second loss target; and The visual and language learning models of the image encoder, the text encoder, and the multimodal encoder are updated at least in part based on the first loss target and the second loss target.

9. The system of claim 8, wherein updating the visual and language learning model comprises: The image encoder and the text encoder are updated at least in part based on the first loss objective, and the multimodal encoder is updated at least in part based on the second loss objective.

10. The system of claim 8, wherein the operation further comprises: Momentum models are formed using momentum distillation; The momentum model was used to generate multiple modeled image samples and multiple modeled text samples; The plurality of modeled image samples are included in the plurality of image samples; and The plurality of modeled text samples are included in the plurality of text samples.

11. The system of claim 8, wherein the image encoder, the text encoder, and the multimodal encoder each include a converter.

12. The system of claim 8, wherein the operation further comprises fine-tuning the visual and language learning model for a task selected from the group consisting of: image-text retrieval task, image-to-text retrieval task, text-to-image retrieval task, visual entailment task, visual question answering task, and real natural language visual reasoning task.

13. A non-transitory machine-readable medium having machine-readable instructions stored thereon, the machine-readable instructions being executable to cause a system to perform operations, the operations including: The system receives a training dataset comprising multiple image samples and multiple text samples via a data interface, wherein at least one of the multiple image samples corresponds to at least one of the multiple text samples; The plurality of image samples are encoded into a plurality of encoded image samples by an image encoder, and the plurality of text samples are encoded into a plurality of encoded text samples by a text encoder; Based on the plurality of encoded image samples and the plurality of encoded text samples, a first loss target is calculated; A multimodal encoder encodes a first subset of the plurality of encoded image samples and a second subset of the plurality of encoded text samples into a plurality of encoded image-text samples; The first subset and the second subset are selected based at least in part on mining negative image-text pairs through the contrastive similarity of the encoded image-text samples; Based on the multiple encoded image-text samples, a second loss target is calculated; and The image encoder, the text encoder, and the multimodal encoder are updated at least in part based on the first loss target and the second loss target.

Images