Multi-language multi-mode pre-training method based on one-to-K contrast learning

Abstract

The invention provides a multi-language multi-mode pre-training method based on one-to-K contrast learning. For a pre-training task, a universal framework in multi-language and multi-mode pre-training is adopted, the universal framework is composed of a multi-language text encoder f, a visual encoder g and a fusion encoder phi with image-to-text cross attention, images and texts are pre-trained in a small-batch proportion of 1 to K, K is larger than or equal to 2, and K texts in different languages are aligned with the images at the same time; and superposing a multi-language image-text matching and cross-modal masking language modeling pre-training task on a new one-to-K comparative learning normal form, and finally realizing a pre-training process of a multi-language multi-modal model of comparative learning of which the input image and text are one-to-K on the basis of balancing positive samples and negative samples through difficult sample mining.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, and more specifically, to a multi-language multi-modal pre-training method based on 1-to-K contrastive learning. Background Art

[0002] Recently, significant progress has been made in the multimodal and multilingual fields, leading to an increased interest in more general multilingual multimodal scenarios. In the multilingual multimodal field, researchers first explored multilingual multimodal pre-training and then multilingual multimodal retrieval as the first independent downstream task. The goal of multilingual multimodal retrieval is to achieve image-text retrieval in multilingual scenarios through a single model, preventing the high latency of translating from other languages ​​to English in real-time web search. Generally speaking, modern dense retrieval matches query results through a specific distance metric (such as Euclidean distance or cosine similarity), which means that dense retrieval methods should make queries and semantically similar candidates closer than other random candidates in a high-dimensional space. Therefore, the core of the retrieval task is to align the semantic spaces of the query and candidate sets, whether they are in different languages ​​or different modalities. Recent studies have shown that contrastive learning based on paired data is effective in cross-lingual and cross-modal retrieval tasks. For example, CLIP only aligns different modalities for pre-training through contrastive learning, and performs significantly in zero-shot cross-modal retrieval.

[0003] One existing technique follows the model architecture of the multilingual domain, where a pre-trained multimodal model (e.g., CLIP) is required. The pre-trained model is then adjusted to a multilingual version by aligning the representations of English and non-English text while freezing the backbone of vision and English text. The representative of these methods is Multilingual CLIP, and their essential idea is to use English as a bridge between images and other languages.

[0004] The multimodal alignment process is so difficult that English text cannot be aligned well with images. Specifically, in the text modality (unimodal) scenario, the loss value can drop by 5 to 6 orders of magnitude, while in multimodal contrastive learning, the loss value can only drop by 2 orders of magnitude. This means that the alignment between English text and images is not ideal, and if English text is used to connect images and text in other languages, there is a risk of error propagation in the intra-modal alignment, resulting in worse alignment between non-English text and images. This problem is widespread in pre-training, so the impact of this problem is global and can be revealed by the uneven performance under different language settings.

[0005] Another existing approach follows the model architecture of the multimodal domain, where multilingual image-text pairs are required. Due to the difficulty of collecting multilingual image-text pairs in practice, a translation model is usually used to translate the English text in the existing image-text pairs into other languages. Then, at most one non-English text is added to the image to form an image-text pair, consistent with the input form of the multimodal model. The representative of these methods is UC2. The idea behind these methods is to align images with text in one language at a time to improve cross-lingual performance.

[0006] The representations obtained by popular multilingual text encoders are not semantically aligned after being compressed by t-SNE. Instead, they remain irregularly distributed. Therefore, when the model only sees text in one language at a time, the alignment of the images may not be beneficial for all languages, which can lead to inconsistent performance between semantically similar texts in different languages. Since this problem arises dynamically in different instances of different languages ​​during the pre-training phase, the impact of this problem is local. (1) When retrieving the corresponding text in different languages ​​using the same image, very different ranking results will be obtained; (2) When retrieving the same image using text in different languages, very different retrieval results will be obtained. Summary of the invention

[0007] The purpose of the embodiments of the present disclosure is to provide a multilingual multimodal pre-training method based on 1-to-K contrastive learning, which is used to solve the problem that the existing multilingual multimodal pre-training methods applied to multilingual multimodal pre-training will lead to inconsistent sorting and recall.

[0008] In a general aspect, a multilingual multimodal pre-training method based on 1-to-K contrastive learning is provided. For a pre-training task, a general framework in multilingual multimodal pre-training is adopted, which is composed of a multilingual text encoder f, a visual encoder g, and a fusion encoder φ with image-to-text cross-attention. The image and text are pre-trained in a small batch ratio of 1 to K, K ≥ 2, and K texts in different languages ​​are aligned with the image at the same time; and multilingual image-text matching and cross-modal masked language modeling pre-training tasks are superimposed on the new 1-to-K contrastive learning paradigm. On the basis of balancing positive samples and negative samples through hard sample mining, the pre-training process of a multilingual multimodal model for contrastive learning of 1-to-K input images and texts is finally realized;

[0009] The specific method of aligning K texts of different languages ​​with images at the same time is: directly forming a tuple of texts and images of all languages ​​as input, using t and i to represent the normalized text and image representations respectively, and the optimization objective of 1-to-K contrastive learning is expressed as:

[0010]

[0011] Where K is the number of languages, N is the number of negative examples, i2t represents the training objective from image to text, t2i represents the training objective from text to image, kcl represents one-pair k contrastive learning, represents image representation, represents the text representation vector, i represents the image representation vector, τ represents the temperature parameter, j represents the number of the anchor sample in the batch, n represents the number of other samples in the batch, and T represents the transpose of the representation vector.

[0012] The specific method of the difficult sample mining is: adopting the difficult sample mining strategy for both positive samples and negative samples, and in the direction of image to text, the training model selects the correct one from the difficult positive text sample and the difficult negative text sample. The loss function of multilingual image text matching can be expressed as:

[0013]

[0014] Similarly, for the text-to-image direction, the matching target can be expressed as:

[0015]

[0016] Among them, mitm represents multilingual image-text matching, i2t represents the loss from image to text, t2i represents the loss from text to image, u represents the last layer representation sequence obtained by the image-text pair input model, cls represents the representation at the cls position, p represents the positive image-text pair, nt represents the image-text pair consisting of image and negative text, ni represents the image-text pair consisting of text and negative image, ψ represents the subsequent MLP, which maps the representation to a 2-dimensional score representation, the 0th dimension represents the probability of mismatch, and the 1st dimension represents the probability of match.

[0017] The purpose of the multimodal masked language model task is to reconstruct the masked word tags using text context information and image information. The loss function is expressed as:

[0018]

[0019] A special marker MASK is used to replace 15% of the word markers in each text. cmlm represents the multimodal masked language model, mask represents masking out part of the text in the input, ρ represents the cosine similarity, and w j represents the representation of a word in the vocabulary, W represents the representation of the entire vocabulary, Represents the representation of the jth position in the representation sequence obtained by the masked text, Represents the word representation at the jth position.

[0020] The final optimization goal is expressed as:

[0021]

[0022] The technical effects to be achieved by the embodiments of the present invention are:

[0023] First, this scheme proposes a novel 1-to-K contrastive learning loss. Second, this scheme uses two pre-training tasks based on hard example mining, image-text matching and masked language model, to train the fusion encoder. Finally, better multi-language and multi-modal retrieval performance is obtained with the same training time.

[0024] Based on the previous multi-language or multi-modal pre-training paradigm, the performance of multi-language and multi-modal retrieval is further improved, making the retrieval more accurate and effective, and reducing the inference cost of practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and other objects and features of the present disclosure will become more apparent from the following description in conjunction with the accompanying drawings.

[0026] Figure 1 is a schematic diagram showing an example of a zero-sample detector based on a large model in the prior art;

[0027] Figure 2 is a schematic diagram showing the comparison of the effect of removing the fusion encoder on xFlickr&CO according to an embodiment of the present disclosure;

[0028] Figure 3 is a schematic diagram showing the result of evaluating a checkpoint every 5 rounds on Multi30K according to an embodiment of the present disclosure;

[0029] Figure 4 is a schematic diagram showing a retrieval instance analysis according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] The following specific embodiments are provided to help the reader obtain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear. For example, the order of operations described herein is only an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, for greater clarity and simplicity, the description of features known in the art may be omitted.

[0031] The features described herein can be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein have been provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein, which will be clear after understanding the disclosure of the present application.

[0032] As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more.

[0033] Although terms such as "first", "second", and "third" may be used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions should not be limited by these terms. Instead, these terms are only used to distinguish one member, component, region, layer, or portion from another member, component, region, layer, or portion. Therefore, without departing from the teachings of the examples described herein, the first member, first component, first region, first layer, or first portion referred to in the examples may also be referred to as the second member, second component, second region, second layer, or second portion.

[0034] In the specification, when an element (such as a layer, a region or a substrate) is described as being “on”, “connected to” or “coupled to” another element, the element may be directly “on”, “connected to” or “coupled to” another element, or one or more other elements may be present therebetween. Conversely, when an element is described as being “directly on”, “directly connected to” or “directly coupled to” another element, there may be no other elements present therebetween.

[0035] The terms used herein are only used to describe various examples and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. The terms "comprise", "include" and "have" indicate the presence of the described features, quantities, operations, components, elements and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof.

[0036] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by a person of ordinary skill in the art to which the present disclosure belongs after understanding the present disclosure. Unless explicitly defined as such herein, terms (such as those defined in a general dictionary) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and should not be interpreted in an idealized or overly formal manner.

[0037] Furthermore, in the description of examples, when it is considered that a detailed description of a well-known related structure or function would cause vague interpretation of the present disclosure, such a detailed description will be omitted.

[0038] Figure 1 is a schematic diagram showing a multi-language multi-modal pre-training method based on 1-to-K contrastive learning according to an embodiment of the present disclosure.

[0039] In order to achieve the above-mentioned invention object, the technical framework adopted by the present invention is as follows Figure 1 shown.

[0040] This scheme proposes a simple but effective multilingual multimodal pre-training paradigm, 1-to-K contrastive learning, such as Figure 1 As shown. Specifically, when images and texts are pre-trained in a mini-batch ratio of 1 to K (K ≥ 2) instead of 1:1, each image is aligned with K texts in different languages ​​simultaneously. Under this paradigm, all languages ​​are aligned with vision at the same time, and no language is used as a bridge between vision and other languages. This paradigm in principle eliminates intra-modal error propagation and inter-modal optimization direction bias. In addition, this scheme also adopts two commonly used pre-training tasks that capture fine-grained correlations between modalities, multi-lingual image-text matching (MITM) and cross-modal masked language modeling (CMLM), which are better superimposed on the new 1-to-K contrastive learning paradigm with the help of hard example mining.

[0041] This section will introduce the 1-to-K contrastive learning loss proposed in this paper:

[0042] To solve the inconsistency problem, the key is that texts in all languages ​​should be aligned with semantically similar images at the same time. Obviously, it is impossible to do this through paired data, that is, randomly sampling one text from all languages ​​and combining it with the corresponding image to form an image-text pair. Therefore, an effective method is to directly form a tuple of texts and images in all languages ​​as input. Each tuple contains an image and its corresponding text description, and multiple tuples are multiple images and their corresponding text descriptions. Each tuple is first encoded by the image and text encoders to obtain the corresponding image i and text representation t, and the obtained representation is used. Therefore, this scheme proposes a 1-to-k contrastive learning method to solve this problem. For simplicity, let t and i represent the normalized text and image representations, respectively. Then, the optimization objective of 1-to-k contrastive learning can be expressed as follows

[0043]

[0044] where K is the number of languages ​​and N is the number of negative examples. It is worth noting that there is literature on multi-positive contrastive learning in other fields, where all positive terms are accumulated in the numerator and the overall positive probability is approximately equal to 1. In contrast, this scheme further sets the label of each positive term to 1 / K to ensure equal contribution from each language.

[0045] Multi-language multi-modal pre-training technology based on 1-to-K contrastive learning:

[0046] Based on the proposed 1-to-k contrastive learning, this paper further proposes a multilingual multimodal pre-training model named CCRK. Specifically, this paper combines 1-to-k contrastive learning with multilingual image-text matching (MITM) and cross-modal masked language modeling (CMLM) tasks. For MITM and CMLM tasks, hard sample mining techniques are introduced to balance positive and negative samples. Specifically, the language with lower similarity between images and various languages ​​is used as the hard samples in CMLM and MITM. This paper adopts a general framework in multilingual multimodal pre-training, which consists of a multilingual text encoder f, a visual encoder g, and a fusion encoder φ with image-to-text cross-attention.

[0047] First, this scheme introduces hard example mining. In all languages, the cross-attention mechanism between image representation and text representation greatly increases the pre-training time. Therefore, this scheme adopts a hard example mining strategy for both positive and negative samples. This method allows the model to focus only on how to reconstruct the hardest positive sample in the multimodal masked language model task and distinguish the hardest negative sample in the multilingual image-text matching task. The multilingual image-text matching task is a binary classification task, whose goal is to identify whether the semantics of a given image-text pair match. This task is generally considered to be a bidirectional prediction problem of image-text. Specifically, in the image-to-text direction, the training model selects the correct one from the hard positive text sample and the hard negative text sample. The loss function of multilingual image-text matching can be expressed as:

[0048]

[0049] Similarly, for the text-to-image direction, the matching target can be expressed as:

[0050]

[0051] The goal of the multimodal masked language model task is to reconstruct masked word tags using text context information and image information. The loss function can be expressed as:

[0052]

[0053] This solution adopts the approach used in BERT and uses a special token [MASK] to replace 15% of the word tokens in each text.

[0054] Since contrast loss, image text matching, and masked language model have been verified in many previous works, and some recent methods optimize them together and achieve convergence at the same time, this scheme directly adds them together without using additional hyperparameters to weight different losses. Therefore, the final optimization objective can be expressed as:

[0055]

[0056] Exploring variants that only use contrastive learning:

[0057] In fact, the multilingual multimodal pre-training technique based on 1-to-K contrastive learning is proposed to ensure that the model's parameter amount and pre-training tasks are similar to other baseline models. However, neither the multilingual image-text matching task, the multimodal masked language model task, nor the fusion encoder are required. Therefore, this scheme further compares the effects of 1-to-K and 1-to-1 contrastive learning on Recall@K and MRV indicators (the lower the MRV indicator, the better) without the fusion encoder, while other settings are consistent with the normal method of this scheme. Figure 2 It can be seen that 1-to-K contrastive learning can still achieve better results on xFlickr&CO and Multi30k.

[0058] Exploration of losses and performance:

[0059] To better understand why our approach works, we report the 1:1 contrast loss and 1:k contrast loss of our approach and our approach without 1:K contrast learning during pre-training. In addition, we evaluate checkpoints every 5 epochs on Multi30K in the zero-shot setting and plot the results in Figure 3 It is shown in Figure 2 that 1-vs-k contrastive learning performs better on all checkpoints.

[0060] Example analysis:

[0061] Figure 4 Some examples from the xFlickr&CO dataset are shown. Each column represents the top 1 image searched in the corresponding language. In the CCLM method compared with this solution, most of these examples did not retrieve the correct image, while the method of this solution made the correct image retrieved in the top 1 of most languages, whether it was the relatively simple example 3 or the more complex example 5, which proved the superiority of this solution.

[0062] Experimental verification:

[0063] Our technique is extensively tested on multilingual multimodal retrieval tasks. We evaluate our method on four popular datasets, including xFlickr&CO, WIT, Multi30K, and COCO. Although the images in xFlickr&CO come from the original Flickr30K and COCO, the multilingual texts in xFlickr&CO are manually re-annotated. Therefore, the performance of xFlickr&CO is not strongly correlated with that of Multi30K and COCO. For xFlickr&CO and WIT, we use two protocols to evaluate our model: fine-tuning on the English training set (zero-shot) and fine-tuning on 100 other language instances based on the English model (few-shot). For Multi30K and COCO, we also use two evaluation protocols: fine-tuning on the English training set (zero-shot) and fine-tuning on each language training set (fine-tuning). Note that the results of WIT in the few-shot scenario are not reported because IGLUE does not provide a corresponding evaluation protocol.

[0064] Table 3 Zero-sample experimental results

[0065]

[0066] Table 4 Experimental results of other settings

[0067]

[0068] From the experimental results, it can be seen that the method in this paper has excellent accuracy in multimodal and multilingual retrieval tasks, surpassing previous advanced methods in all indicators and improving the retrieval performance.

[0069] In summary, the present invention can further improve the performance of multi-language and multi-modal retrieval based on the previous multi-language or multi-modal pre-training paradigm, making the retrieval more accurate and effective, and can also reduce the inference cost of practical applications.

[0070] Although some embodiments of the present disclosure have been shown and described, it will be appreciated by those skilled in the art that modifications may be made to the embodiments without departing from the principles and spirit of the present disclosure, the scope of which is defined by the claims and their equivalents.

Claims

1. A multi-language multi-modal pre-training method based on 1-to-K contrastive learning, characterized in that: For a pre-training task, a general framework in multilingual multimodal pre-training is adopted, and the texts and images of all languages ​​are directly formed into a tuple as input, which is composed of a multilingual text encoder f using a bidirectional attention Transformer architecture based on the first six layers of XBERT, a visual encoder g using a visual Transformer architecture based on the Swin-Transformer, and a fusion encoder φ with image-to-text cross-attention based on the bidirectional cross-attention Transformer architecture of the last six layers of XBERT. The image i and text t are pre-trained in a small batch ratio of 1 to K, K ≥ 2, and K texts in different languages ​​are aligned with the image at the same time; and the multilingual image-text matching and cross-modal masked language modeling pre-training tasks are superimposed on the new 1-to-K contrastive learning paradigm. On the basis of balancing positive and negative samples through difficult example mining, the pre-training process of a multilingual multimodal model for contrastive learning of 1-to-K input images and texts is finally realized; The specific method of aligning K texts of different languages ​​with images at the same time is: directly forming a tuple of texts and images of all languages ​​as input, using t and i to represent normalized text and image representations respectively, and the optimization objective of 1-to-k contrastive learning is expressed as: Where K is the number of languages, N is the number of negative examples, i2t represents the training objective from image to text, t2i represents the training objective from text to image, kcl represents one-pair k contrastive learning, represents image representation, represents the text representation vector, i represents the image representation vector, τ represents the temperature parameter, j represents the number of the anchor sample in the batch, n represents the number of other samples in the batch, and T represents the transpose of the representation vector.

2. A multi-language multi-modal pre-training method based on 1-to-K contrastive learning as claimed in claim 1, characterized in that: The specific method of the difficult sample mining is: a difficult sample mining strategy is adopted for both positive samples and negative samples. In the image-to-text direction, the training model selects the correct one from the difficult positive text sample and the difficult negative text sample. The image-to-text loss function of multilingual image-to-text matching can be expressed as: Similarly, for the text-to-image direction, the matching target can be expressed as: Among them, mitm represents multilingual image-text matching, i2t represents the loss from image to text, t2i represents the loss from text to image, u represents the last layer representation sequence obtained by the image-text pair input model, cls represents the representation at the cls position, p represents the positive image-text pair, nt represents the image-text pair consisting of image and negative text, ni represents the image-text pair consisting of text and negative image, ψ represents the subsequent MLP, which maps the representation to a 2-dimensional score representation, the 0th dimension represents the probability of mismatch, and the 1st dimension represents the probability of match; The purpose of the multimodal masked language model task is to reconstruct the masked word tags using text context information and image information. The loss function is expressed as: A special marker MASK is used to replace 15% of the word markers in each text. cmlm represents the multimodal masked language model, mask represents masking out part of the text in the input, ρ represents the cosine similarity, and w j represents the representation of a word in the vocabulary, W represents the representation of the entire vocabulary, Represents the representation of the jth position in the representation sequence obtained by the masked text, Represents the word representation of the jth position; the final optimization goal is expressed as:

Images