Systems and methods for interlanguage transfer in natural language processing

By training a supervised alignment model on rich resource language pairs using XLM-R and incorporating BERT scores with normalization and contrastive learning, the method addresses inefficiencies in interlanguage sentence alignment, achieving high accuracy and efficiency across diverse language pairs.

JP2024542440AActive Publication Date: 2024-11-15SALESFORCE INC
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2024527807
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2022-01-21
Filing Date
2022-11-14
Publication Date
2024-11-15
Estimated Expiration
2042-11-14

AI Technical Summary

Technical Problem

Existing interlanguage sentence alignment systems face challenges in achieving efficient and accurate alignment due to the scarcity of training data for low-resource languages, with unsupervised methods offering low accuracy and supervised methods requiring difficult-to-obtain large datasets, leading to inefficiencies in parallel data collection across languages.

Method used

A pre-trained multilingual model, such as XLM-R, is used to train a supervised alignment model on rich resource language pairs, incorporating a BERT score for semantic similarity and a normalization layer to counteract popular sentence effects, and employs contrastive learning with within-batch negatives to improve accuracy and efficiency.

Benefits of technology

The proposed method enhances the accuracy of interlanguage sentence alignment, achieving performance comparable to models trained on all language pairs while using fewer resources, and maintains effectiveness across various language pairs, including non-English-centric scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024542440000001_ABST
    Figure 2024542440000001_ABST
Patent Text Reader

Abstract

The embodiments described herein provide a cross-language sentence alignment framework that is trained only on rich-resource language pairs. To obtain an accurate aligner, a pre-trained multilingual model is used and a classifier is trained on parallel data from the rich-resource language pair. This trained classifier can then be used for cross-language transfer with low-resource languages.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] [Cross reference] This application claims priority under U.S. Patent Law to U.S. Provisional Patent Application No. 63 / 279,476, filed November 15, 2021, which is a continuation of U.S. Nonprovisional Patent Application No. 17 / 581,380, filed January 21, 2022, each of which is expressly incorporated by reference in its entirety.

[0002] [Technical field] The present disclosure relates generally to machine learning models and neural networks for natural language processing (NLP), and more specifically to zero-shot inter-language transfer with one rich-resource language pair for low-resource sentence retrieval in NLP. [Background technology]

[0003] Cross-lingual sentence alignment refers to aligning an original sentence in a source language to a translation sentence in a target language. For example, because the grammatical rules of the source and target languages ​​are different, the words in the original sentence and the words in the translation sentence may be in different orders, and therefore it is necessary to align words that have the same meaning in different languages. Cross-lingual sentence alignment can be used in various applications, including (1) mining parallel corpora for machine translation systems, (2) filtering existing parallel corpora to enhance their quality, and (3) estimating translation quality (Quality Estimation tasks), for example, in scenarios for evaluating already generated translations or for deciding among potential candidates. However, to train a cross-lingual sentence alignment model, the training data of different language sources can be quite different. For example, common languages ​​such as English, Spanish, and French often have richer training datasets, while less common languages ​​such as Mongolian and Malay have scarce resources. [Brief description of the drawings]

[0004] [Figure 1] FIG. 1 is a simplified block diagram illustrating a training framework for an alignment model according to one or more embodiments described herein. [Diagram 2] FIG. 2 is a simplified block diagram illustrating aspects of the BERT score calculation shown in module 106 of FIG. 1 , according to one or more embodiments described herein. [Diagram 3] FIG. 2 is a simplified logical flow diagram illustrating a method for training an aligner model for cross-language sentence alignment according to one embodiment described herein. [Figure 4] 1 is a simplified diagram of a computing device for implementing inter-language sentence alignment according to some embodiments. [Diagram 5] 14 shows various performance results of an aligner model described herein compared to a baseline model in a previous study, according to one or more embodiments described herein. [Figure 6] 14 shows various performance results of an aligner model described herein compared to a baseline model in a previous study, according to one or more embodiments described herein. [Figure 7] 14 shows various performance results of an aligner model described herein compared to a baseline model in a previous study, according to one or more embodiments described herein. [Figure 8] 14 shows various performance results of an aligner model described herein compared to a baseline model in a previous study, according to one or more embodiments described herein. [Figure 9] 14 shows various performance results of an aligner model described herein compared to a baseline model in a previous study, according to one or more embodiments described herein. [Figure 10]14 shows various performance results of an aligner model described herein compared to a baseline model in a previous study, according to one or more embodiments described herein. [Figure 11] 14 shows various performance results of an aligner model described herein compared to a baseline model in a previous study, according to one or more embodiments described herein. [Figure 12] 14 shows various performance results of an aligner model described herein compared to a baseline model in a previous study, according to one or more embodiments described herein. [Figure 13] 14 shows various performance results of an aligner model described herein compared to a baseline model in a previous study, according to one or more embodiments described herein.

[0005] In the figures and appendices, elements with the same name have the same or similar functions. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] As used herein, the term "network" may include any hardware or software-based framework, including any artificial intelligence network or system, neural network or system, and / or any training or learning model implemented thereon or in conjunction therewith.

[0007] As used herein, the term "module" may include a hardware or software-based framework that performs one or more functions. In some embodiments, a module may be implemented on one or more neural networks.

[0008] As used herein, the term "rich source" or "low resource" refers to the size of available data. For example, if the parallel data size of a language pair exceeds a certain threshold (e.g., 20,000, 30,000, etc.), such a language pair is considered a rich source. Similarly, if the parallel data size of a language pair is less than a certain threshold (e.g., 10,000 (10K), 20,000 (20K), etc.), such a language pair is considered a low source.

[0009] Cross-language sentence alignment provides a key aspect in machine translation related tasks. Existing systems include two main streams of work: one is fully unsupervised, and the other is supervised using data across many language pairs. Unsupervised methods have the advantage of not collecting any parallel data, but they can only achieve relatively low accuracy on benchmark datasets such as Tatoeba (Artetxe et al., Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond, Transactions of the Association for Computational Linguistics, 7:597-610, 2019), which is evaluated on 36 language pairs including several low-resource language pairs. On the other hand, supervised methods presuppose data access to a large number of low-resource language pairs, which is by definition difficult to obtain and whose quality is hard to guarantee. This all-or-nothing data choice between unsupervised and supervised methods leaves a large gap as to whether zero-shot cross-lingual transfer is effective for such tasks. Therefore, efficiently distributing efforts for parallel data collection across languages ​​remains a challenge.

[0010] In view of the need to improve the efficiency and accuracy of inter-language sentence alignment, the embodiments described herein provide an inter-language sentence alignment framework that is trained only on rich-resource language pairs that are extremely rich in resources (as long as one language pair works). To obtain an accurate aligner, a pre-trained multilingual model is used as a classifier to be trained on parallel data from the rich-resource language pair. This trained classifier can then be used for inter-language transfer with low-resource languages.

[0011] For example, an XLM-RoBERTa (XLM-R) network (described in Conneau et al., Unsupervised cross-lingual representation learning at scale, in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, Association for Computational Linguistics, 2020) can be employed as a pre-trained multilingual model for alignment tasks. A supervised version of the BERT score can be calculated as the semantic similarity between sentences, and a normalization layer is incorporated into the architecture of XLM-R to counter the popular sentence effect, for example, when some sentences in the source language tend to yield high scores on any sentence in the target language. In this way, when trained on any single rich-resource language pair, the model matches the performance (e.g., within 2.0 of the accuracy difference) of a model trained on all language pairs under the same data budget.

[0012] In one embodiment, to further narrow the narrow gap between using one language pair and using all pairs for training while respecting the constraint of only rich resources, the number of language pairs is scaled up with the top k rich resource language pairs. This upgrade further improves the accuracy of the model, which is only 0.4 shy compared to training on all language pairs under the same data budget.

[0013] In one embodiment, the model is trained on language pairs that are not English-centric, e.g., language pairs in which none of the sentences in the language pair are English. The training performance of the model is not significantly affected when English is not the anchor language. The model performs similarly as long as the amount of parallel data is above a certain threshold.

[0014] 1 is a simplified block diagram illustrating a training framework for an alignment model according to one or more embodiments described herein. FIG. 1 illustrates a first batch 102 of source language sentences, e.g., S={S1, S2, ..., S M} and a second batch 104 of target language sentences, e.g., T={T1, T2, ..., T N} are input to the alignment model 110 for alignment.

[0015] Specifically, the alignment model 110 determines whether each sentence in the batches 102 and 104 matches, e.g., for all i=1,...,M and j=1,...,N, S i 102a is T j 104a. The alignment model 110 may build a Transformer-based model on top of XLM-R that is pre-trained on the monolingual CC-100 dataset covering 100 languages.

[0016] The alignment model 110 may include an embedding model 105, a BERT score calculation module f() 106, and a normalization layer 118. i 102a and T j Given 104a, the context embedding model 105 encodes the tokens in the source sentence and the target sentence into two vector representations, respectively. Specifically, the embedding model 105 may employ BERT (Devlin et al., BERT: Pre-training of deep bidirectional transformers for language under-standing, in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, 201) or ELMo (Peters et al., Deep contextualized word representations, in proceedings of NAACL-HLT, 2018), which generates different vector representations for the same word in different sentences depending on the surrounding words that form the context of the target word. For example, the embedding model 105 may employ a BERT encoder that tokenizes the input text into a series of word pieces, and unknown words are split into several commonly observed strings. A representation for each word chunk is computed using a Transformer encoder by repeatedly applying alternating self-attention and nonlinear transformations.

[0017] Next, as further shown in FIG. 2, a BERT score calculation module 106 calculates a BERT score for the input sentences 102a and 104a using the cosine similarity between the context embedding representations from the embedding model 105, optionally weighted by the inverse sentence frequency score.

[0018] 2 is a simplified block diagram illustrating aspects of the BERT score calculation shown in module 106 of FIG. 1, in accordance with one or more embodiments described herein. For example, given an input sentence S i Given 102a, a sequence of vectors s =<s1,s2,…,sk> is the input sentence S i 102a. Similarly, for a sequence of vectors t= <t1,t2,…,t l > is the input text T j 102b's context embedding 105b is shown.

[0019] First, the context embedding t = <t1,t2,…,t l Context embedding to match the vector in s = <s1,s2,…,s k A recall score is calculated by matching each vector in <s1,s2,…,s k Each vector in and the context embedding <t1,t2,…,t l Calculate pairwise cosine similarities 205a between vectors in

number

[0020] Similarly, the context embedding s = <s1,s2,…,s k Context embedding to match vectors in t = <t1,t2,…,t l A precision score is calculated by matching each vector in <t1,t2,…,t l Each vector in and the context embedding <s1,s2,…,s k Calculate the pairwise cosine similarities 205b between vectors in

number

[0021] The BERT score is then calculated as follows:

number

[0022] Thus, the BERT score acts as a shallow cross-attention layer while being much faster than full cross-attention. The resulting model is also independent of the order of the two sentences.

[0023] Returning to FIG. 1, the calculated BERT scores of the input sentences 102a and 104a are fed to a normalization layer 118. In bitext alignment, it is often observed that some sentences in one language tend to have high similarity scores with any sentence in the other language. This phenomenon makes the ranking inaccurate. To counteract this bias, a scaled average of the similarity scores between each sentence in one language and all sentences in the other language is subtracted from the BERT scores. In other words, for the source sentences S i 102a and target sentence T j The normalized similarity 115 between 104a and 104b is calculated as follows:

number

[0024] In the inference stage, two sentences S i , T jThe determination of whether Similiary(i,j) is aligned may be performed, for example, by a thresholding operation. For example, if Similarity(i,j) is greater than a predefined threshold, then the source sentence S i and target sentence T j is considered to be consistent.

[0025] During training, a challenge to training the aligner model 110 with only positive parallel data is the lack of carefully designed negative examples. To address this challenge, a contrastive learning approach can be adopted, where the aligner model 110 is trained on a classification task with within-batch negatives. For example, given a batch S={S1, S2, ..., S N}, and a batch of target language sentences T={T1,T2,…,T N}, S i For each i, T i If S and T are aligned with each other, the pairwise semantic similarities between S and T are calculated to obtain N similarities for the positive alignment and N for the negative alignment. 2 -N similarities are obtained (N in total) 2 similarities are computed for each positive logit. During training, these similarity scores are used as logits, pairing each positive logit with every negative logit. These logits are then used to compute the contrastive loss 120, which is then used to update the aligner model 110 via the backpropagation path 125.

[0026] 3 is a simplified logic flow diagram illustrating a method for training an aligner model for cross-language sentence alignment according to one embodiment described herein. One or more of the processes 302-312 of the method 300 may be implemented, at least in part, in the form of executable code stored on a non-transitory tangible machine-readable medium that, when executed by one or more processors, may cause the one or more processors to perform one or more of the processes 302-312. In some embodiments, the method 300 may correspond to a method used by the module 430.

[0027] In step 302, a training dataset is received. The training dataset includes at least a first sentence in a source language (e.g., sentence 102a in FIG. 1 ) and a second sentence in a target language (e.g., sentence 104a in FIG. 1 ) that is a translated version of the first sentence. For example, the source language and the target language belong to a group of rich source languages, such as English-Spanish, English-French, etc.

[0028] In one embodiment, the training dataset can be (1) an English-centric dataset, such as OPUS-100; (2) a non-English-centric language dataset, e.g., v2021-08-07 Tatoeba Challenge. OPUS-100 is English-centric, meaning that all training pairs include English on either the source or target side. The corpus covers 100 languages ​​(including English). Languages ​​for training are selected based on the amount of parallel data available in OPUS. The OPUS collection consists of multiple corpora ranging from movie subtitles to GNOME documentation to the Bible. OPUS-100 contains approximately 55 million sentence pairs. For example, 99 language pairs are selected to train the aligner model, 44 of which are selected from one million (1M) sentence pairs in the training data, 73 are selected from at least one hundred thousand (100k), and 95 are selected from at least ten thousand (10k). Following the selection of OPUS-100, the training data for each language pair in New-Tatoeba is capped at one million (1M) to facilitate easier comparison with OPUS-trained models.

[0029] In step 304, a positive input pair including a first sentence and a second sentence and a number of negative input pairs are formed, each negative pair including the first sentence and a respective unmatched sentence in the target language. For example, given a batch S of sentences in the source language, S={S1, S2, ..., S N}, and a batch of target language sentences T={T1,T2,…,T N}, S i For each i, T i If S and T are aligned with each other, the pairwise semantic similarities between S and T are calculated to obtain N similarities for the positive alignment and N for the negative alignment. 2 -N similarities are obtained (N in total) 2 similarities are calculated).

[0030] In step 306, the pre-trained multilingual model may be used to calculate pairwise token-level similarity between the two sentences in each positive or negative input pair. For example, the pairwise token-level similarity between the two sentences may be calculated as a BERT score, as described in connection with FIG.

[0031] In step 308, a loss target is calculated based on the calculated pairwise token-level similarities associated with the positive input pair and multiple negative input pairs. For example, the similarity scores are used as logits to pair each positive logit with all negative logits. These logits are then used to calculate a contrastive loss between the positive and negative pairs.

[0032] In step 310, the pre-trained multilingual model is updated based on the loss target.

[0033] In step 312, an alignment task may be performed between the source language and another language different from the target language using the updated pre-trained multilingual model. For example, a pre-trained multilingual model trained using only rich language pairs (e.g., English-Spanish, English-French, etc.) may then be used to align rare source language pairs (e.g., English-Mongolian, English-Malay, etc.). In some implementations, during the inference stage, an aligner model (pre-trained multilingual model trained by method 300) may be used to align rare source language pairs without using English as an anchor language, e.g., where none of the languages ​​in the rare language pair is English.

[0034] In one embodiment, the training method 300 for the aligner model may be executed on a single A100 graphics processor (GPU). For example, the training batch size is 64 and the evaluation batch size is 256. These are examples of the maximum numbers that can fit into a batch on one GPU. A smaller training batch size may result in reduced performance, while a larger batch size can provide more in-batch negatives for each positive example.

[0035] 4 is a simplified diagram of a computing device for implementing inter-language sentence alignment, according to some embodiments. As shown in FIG. 4, the computing device 400 includes a processor 410 coupled to a memory 420. The operation of the computing device 400 is controlled by the processor 410. Although the computing device 400 is shown with only one processor 410, it is understood that the processor 410 may represent one or more central processing units, multi-core processors, microprocessors, microcontrollers, digital signal processors, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), graphics processing units (GPUs) and / or the like within the computing device 400. The computing device 400 may be implemented as a standalone subsystem, as a board added to a computing device, and / or as a virtual machine.

[0036] The memory 420 may be used to store software executed by the computing device 400 and / or one or more data structures used during operation of the computing device 400. The memory 420 may include one or more types of machine-readable media. Some common forms of machine-readable media may include a floppy disk, a flexible disk, a hard disk, a magnetic tape, any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with a pattern of holes, a RAM, a PROM, an EPROM, a FLASH-EPROM, any other memory chip or cartridge, and / or any other medium adapted to be read from by a processor or computer.

[0037] The processor 410 and / or memory 420 may be arranged in any suitable physical arrangement. In some embodiments, the processor 410 and / or memory 420 may be implemented on the same substrate, in the same package (e.g., a system-in-package), on the same chip (e.g., a system-on-chip), and / or the like. In some embodiments, the processor 410 and / or memory 420 may comprise distributed, virtualized, and / or containerized computing resources. Consistent with such embodiments, the processor 410 and / or memory 420 may be located in one or more data centers and / or cloud computing facilities.

[0038] In some examples, the memory 420 may include a non-transitory tangible machine-readable medium including executable code that, when executed by one or more processors (e.g., processor 410), may cause the one or more processors to perform methods described in further detail herein. For example, as shown, the memory 420 includes instructions for a paraphrase generation module 430 that may be used to implement and / or emulate the systems and models and / or to implement any of the methods described further herein. In some examples, the interlanguage transfer module 430 may receive an input 440, such as, for example, an input text in a source language and / or a target language, via a data interface 415. The data interface 415 may be either a user interface that receives articulated audio input or a communication interface that may receive or retrieve previously stored training text from a database. The interlanguage transfer module 430 may generate an output 450, such as an alignment of the input 440 with a sentence in the target language that corresponds to the input 440.

[0039] In some embodiments, the interlanguage transfer module 430 may implement an embedding module 431 (e.g., similar to 105 in FIG. 1), a BERT score module 232 (e.g., similar to 106 in FIG. 1), a normalization module 233 (e.g., similar to 118 in FIG. 1), and a contrastive loss module 434 (e.g., 120 in FIG. 1). In some examples, the interlanguage transfer module 430 and sub-modules 431-434 may be implemented using hardware, software, and / or a combination of hardware and software.

[0040] Some examples of computing devices, such as computing device 400, may include a non-transitory tangible machine-readable medium that includes executable code that, when executed by one or more processors (e.g., processor 410), may cause the one or more processors to perform the process of method 300. Some common forms of machine-readable media that may include the process of method 300 are, for example, a floppy disk, a flexible disk, a hard disk, a magnetic tape, any other magnetic medium, a CD-ROM, any other optical medium, a punch card, a paper tape, any other physical medium having a pattern of holes, a RAM, a PROM, an EPROM, a FLASH-EPROM, any other memory chip or cartridge, and / or any other medium adapted to be read from by a processor or a computer.

[0041] Exemplary Performance As explained above, the aligner model may be trained by a training dataset of language pairs selected from OPUS-100 and / or the v2021-08-07 Tatoeba Challenge. In one embodiment, for training purposes, all languages ​​from both CC-100 and OPUS-100 are selected without assuming the set of language pairs to be tested. Since OPUS-100 was sampled without considering the Tatoeba and BUCC evaluation data, any examples from OPUS where either the source or target are in the test set may be removed.

[0042] 5-13 show various performance results of the aligner models described herein compared to baseline models in previous studies, according to one or more embodiments described herein.

[0043] In some embodiments, the aligner model may be tested via a cross-language sentence retrieval task that retrieves matching sentences in the target language from a set of sentences. Three evaluation datasets are used. The first dataset is the Tatoeba dataset from the XTREME benchmark (Hu et al., Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization, in International Conference on Machine Learning, pages 4411-4421, 2020), called Tatoeba-36 because it contains 36 language pairs. The second dataset is the v2021-08-07 Tatoeba Challenge,4, called New-Tatoeba. For evaluation purposes, only language pairs with more than 1K examples combined in the dev and test sets are retained. Following Tatoeba-36, where most language pairs have 1K test examples, 1K examples are randomly sampled for each language pair in New Tatoeba. The resulting evaluation set contains 223 language pairs, including 49 pairs that are English-centric and 174 pairs that are not English-centric. The third dataset is BUCC 2018 (Zweigenbaum et al., Overview of the third BUCC shared task: Spotting parallel sentences in comparable corpora, in Proceedings of 11th Workshop on Building and Using Comparable Corpora, pages 39-42, 20188), which is a cross-lingual bitext mining task. This task is included because the two Tatoeba datasets are both ranking tasks, while BUCC requires a universal threshold because during web mining, each sentence in the source language does not necessarily have a translation in the target language.Therefore, these datasets contain many distraction sentences.,The drawback of BUCC is that it only contains four language pairs,,all of which are fairly rich resources.,Hence, BUCC serves only as a sanity check for the aligner,model.

[0044] Exemplary baseline models for comparison with the aligner model include VECO (Luo et al., VECO: Variable and flexible cross-lingual pre-training for language understanding and generation, in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3980-3994, 2021) and ERNIE-M (Ouyang et al., ERNIE-M: Enhanced multilingual representation by aligning cross-lingual semantics with monolingual corpora, in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 27-38, 2021). Like the aligner model described herein (referred to as "OneAligner"), ERNIE-M is built on top of XLM-R. It is trained on a monolingual corpus and a parallel corpus containing 96 languages. VECO shares the same model size as the aligner model and is trained with a monolingual corpus and a parallel corpus containing 50 languages ​​(presumably to avoid capacity dilution). The monolingual data is extracted from CC-100, while the bilingual data is collected from the OPUS website. There are 6.4 billion (6.4G) parallel examples covering 879 language pairs. Figure 6 shows basic statistics for each model.

[0045] The aligner model ("OneAligner") is trained on the entire OPUS-100 dataset with and without the one million (1M) budget. Figure 5 shows that both models achieve state-of-the-art results on the Tatoeba-36 dataset. Figures 5 and 6 also show that the aligner model is more data-efficient compared to the other two models.

[0046] Figure 7 shows the performance of Tatoeba-36 against models trained on the OPUS-100 dataset for each language pair at the intersection of OPUS-100 and CC-100 languages. Performance is quite consistent across language pairs, suggesting that it is possible to fine-tune OneAligner on almost any language pair at hand and arrive at a model that performs similarly.

[0047] Figure 8 shows the scatter plot of Figure 7 against data availability for each language pair. After reaching a certain data size threshold (somewhere between 10k and 20k), all language pairs perform similarly. This is partly expected because the aligner model design does not introduce new parameters to XLM-R and eliminates the need to train randomly initialized parameters.

[0048] Similar to Fig. 7, Fig. 9 shows the accuracy of aligner models trained on each of the top 16 rich-resource non-English-centric pairs from the New-Tatoeba dataset. Again, performance is consistent across language pairs, indicating that models trained on non-English language pairs can still obtain similar performance on the English-centric evaluation set.

[0049] Figure 10 addresses the reverse question: whether models trained on English-centric data perform similarly on non-English evaluation data. In Figure 10, when compared column-wise, OneAligner shows similar performance whether it is trained on English-centric language pairs or whether there is an overlap between the fine-tuning and evaluation languages. When comparing models evaluated on English-centric or non-English-centric language pairs, both models perform better on English-centric language pairs. This is because English dominates the training data during pre-training of XLM-R.

[0050] For example, rather than splitting in a bipolar fashion between "English-centric" and "non-English-centric", one can describe the setting with a spectrum and explore X-centric, where X can be any language. Accuracy for language X is defined as the average of the accuracy of all language pairs that contain X.

[0051] Figure 11 shows a scatter plot of the performance of Top-1-Eng New-Tatoeba against monolingual data size for each language in the CC-100 dataset. Similar to Figure 8, the performance of New-Tatoeba correlates positively with monolingual data size up to a certain data threshold (somewhere up to 10.0 GB).

[0052] The single-pair Tatoeba results are already satisfactory. However, what if we aimed for even better performance without violating the assumption of only rich resources? We found that adding other rich-resource pairs could help. Unfortunately, OPUS-100 does not provide a ranking of which language pairs are the most rich-resource (recall that in this dataset the size of each language pair is capped at 1 million (1M) and the original paper did not include pre-sampling data statistics), so we use the New-Tatoeba dataset and rank each English-centric pair based on its availability.

[0053] Figure 12 shows the performance of the top 1 to top 32 rich-resource language pair combinations on the Tatoeba dataset. In Figure 12, performance increases monotonically as more language pairs are added until an accuracy of 94.0 is reached, which is only 0.4 points below the best performance achieved when training with all language pairs under a fixed one million (1M) budget. Note that the least resource-rich language UK in the top 32 list is still within the "highest" resource range defined in the Tatoeba Challenge, containing about 34 million training examples, so this training never violates the rich-resource restriction on training data. Thus, at least for the sentence retrieval task, the marginal cost of improving that 0.4 point accuracy does not seem to justify the effort of extensively collecting more low-resource parallel data with more language pairs.

[0054] Figure 13 shows the BUCC F1 scores of the two Top1 models compared to previous work. Both models outperform VECO by 1.2 points. Recall that the English-centric Top1 model is trained on en-es. In other words, the model has not seen a single parallel example between en and each of the BUCC target languages ​​{de,fr,ru,zh}, whereas VECO is trained extensively on each of the language pairs. This result is consistent with the observation that the aligner model can perform cross-language transfer at a comparable performance to the intra-language model, regardless of whether the fine-tuning language pair is English-centric or not.

[0055] This description and the accompanying drawings illustrating aspects, embodiments, implementations, or applications of the present invention should not be construed as limiting. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of the present specification and claims. In some instances, well-known circuits, structures, or techniques have not been shown or described in detail so as not to obscure the embodiments of the present disclosure. Similar numbers in two or more figures represent the same or similar elements.

[0056] In this description, specific details are described that describe some embodiments consistent with the present disclosure. Numerous specific details are described to provide a thorough understanding of the embodiments. However, it will be apparent to one skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are meant to be illustrative, not limiting. One skilled in the art may recognize other elements that are not specifically described herein but are within the scope and spirit of the present disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in connection with one embodiment may be incorporated into other embodiments, unless specifically described otherwise or if one or more features render the embodiment non-functional.

[0057] Although exemplary embodiments have been shown and described, a wide range of modifications, changes, and substitutions are contemplated in the foregoing disclosure, and in some cases, some features of the embodiments may be employed without the corresponding use of other features. Those skilled in the art will recognize many variations, alternatives, and modifications. Accordingly, the scope of the present invention should be limited only by the scope of the following claims, which should be appropriately interpreted broadly and consistent with the scope of the embodiments disclosed herein.

Claims

1. 1. A method for interlingual sentence alignment, comprising: receiving, via a communications interface, a training data set including at least a first sentence in a source language and a second sentence in a target language that is a translated version of the first sentence; forming a positive input pair including the first sentence and the second sentence, and a plurality of negative input pairs each including the first sentence and a respective non-matching sentence in the target language; Computing pairwise token-level similarities between two sentences in each positive input pair or negative input pair via a pre-trained multilingual model; calculating a loss target based on the calculated pairwise token-level similarities associated with the positive input pairs and the plurality of negative input pairs; updating the pre-trained multilingual model based on the loss target; performing an alignment task between the source language and another language different from the target language using the updated pre-trained multilingual model; The method includes:

2. generating a first context embedding and a second context embedding of the two sentences in each of the positive input pairs or the negative input pairs via a context embedding encoder; The method of claim 1 further comprising:

3. The pairwise token-level similarity between the two sentences is computing a first set of pairwise cosine similarities between every vector in the first context embedding and a first vector in the second context embedding; determining a first maximum similarity measure among the first set of pairwise cosine similarities; determining a first score based on the first maximum similarity; The method of claim 2, wherein the calculated value is:

4. computing a second set of pairwise cosine similarities between every vector in the second context embedding and a second vector in the first context embedding; determining a second maximum similarity measure among the second set of pairwise cosine similarities; determining a second score based on the second maximum similarity; The method of claim 3 further comprising:

5. calculating a similarity score based on the first score and the second score; The method of claim 4 further comprising:

6. normalizing the calculated similarity scores by subtracting from the calculated similarity scores a scaled average of the similarity scores between each of the source language sentences and all of the target language sentences, and between each of the target language sentences and all of the source language sentences. The method of claim 5 further comprising:

7. 2. The method of claim 1 , wherein the loss goal is calculated as a contrastive loss using the calculated pairwise token-level similarity for each positive input pair or negative input pair as an output logit from the pre-trained multilingual model.

8. The method of claim 1 , wherein the source language and the target language belong to a group of rich source languages.

9. The method of claim 1 , wherein the other language is a rare source language, and the pre-trained multilingual model is not trained on any language samples of the rare source language.

10. The method of claim 1 , wherein neither the source language nor the target language is English.

11. 1. A system for interlingual sentence alignment, comprising: a communications interface for receiving a training data set including at least a first sentence in a source language and a second sentence in a target language that is a translated version of the first sentence; a memory storing a pre-trained multilingual model and a plurality of processor-executable instructions; a processor that executes the plurality of processor-executable instructions to perform operations; and the operation comprises: forming a positive input pair including the first sentence and the second sentence, and a plurality of negative input pairs each including the first sentence and a respective non-matching sentence in the target language; Calculating pairwise token-level similarities between two sentences in each positive input pair or negative input pair via the pre-trained multilingual model; calculating a loss target based on the calculated pairwise token-level similarities associated with the positive input pairs and the plurality of negative input pairs; updating the pre-trained multilingual model based on the loss target; and using the updated pre-trained multilingual model to perform an alignment task between the source language and another language different from the target language; Including, the system.

12. The operation includes: generating a first context embedding and a second context embedding of the two sentences in each of the positive input pairs or the negative input pairs via a context embedding encoder; The system of claim 11 further comprising:

13. The pairwise token-level similarity between the two sentences is Computing a first set of pairwise cosine similarities between every vector in the first context embedding and a first vector in the second context embedding; determining a first maximum similarity measure among the first set of pairwise cosine similarities; determining a first score based on the first maximum similarity; Computing a second set of pairwise cosine similarities between every vector in the second context embedding and a second vector in the first context embedding; determining a second maximum similarity measure among the second set of pairwise cosine similarities; and determining a second score based on the second maximum similarity; and calculating a similarity score based on the first score and the second score; normalizing the calculated similarity scores by subtracting from the calculated similarity scores a scaled average of the similarity scores between each of the source language sentences and all of the target language sentences, and between each of the target language sentences and all of the source language sentences; The system of claim 12, wherein the calculation is performed by:

14. 12. The system of claim 11, wherein the loss goal is calculated as a contrastive loss using the calculated pairwise token-level similarity for each positive input pair or negative input pair as an output logit from the pre-trained multilingual model.

15. 1. A non-transitory processor-readable storage medium storing a plurality of processor-executable instructions for inter-language sentence alignment, the instructions comprising: receiving, via a communications interface, a training data set including at least a first sentence in a source language and a second sentence in a target language that is a translated version of the first sentence; forming a positive input pair including the first sentence and the second sentence, and a plurality of negative input pairs each including the first sentence and a respective non-matching sentence in the target language; Computing pairwise token-level similarities between two sentences in each positive input pair or negative input pair via a pre-trained multilingual model; calculating a loss target based on the calculated pairwise token-level similarities associated with the positive input pairs and the plurality of negative input pairs; updating the pre-trained multilingual model based on the loss target; and using the updated pre-trained multilingual model to perform an alignment task between the source language and another language different from the target language; 16. A non-transitory processor-readable storage medium that is executed by a processor to perform operations including:

Citation Information

Patent Citations

  • Word vector alignment method and word vector alignment model training method

    CN111291563A

  • Word alignment score calculation apparatus, word alignment device, and computer program

    JP2015170168A

  • Training Deep Neural Networks Based on Pairwise Similarity Distributions

    JP2020504891A

  • Cross-Lingual Regularization for Multilingual Generalization

    US20200285706A1