Systems and methods for interlanguage transfer in natural language processing

Abstract

The embodiments described herein provide a cross-language sentence alignment framework that is trained only on rich-resource language pairs. To obtain an accurate aligner, a pre-trained multilingual model is used and a classifier is trained on parallel data from the rich-resource language pair. This trained classifier can then be used for cross-language transfer with low-resource languages.

Description

[Technical Field]

[0001] [Cross reference] This application claims priority under the U.S. Patent Act to U.S. Provisional Patent Application No. 63 / 279,476 filed on 15 November 2021, and to U.S. Non-Provisional Patent Application No. 17 / 581,380 filed on 21 January 2022, which are expressly incorporated herein by reference in their entirety.

[0002] [Technical field] This disclosure relates in general to machine learning models and neural networks for natural language processing (NLP), and more specifically to zero-shot inter-language transfers using a single rich-resource language pair for low-resource sentence extraction in NLP. [Background technology]

[0003] Cross-lingual sentence alignment refers to aligning a source text in a source language with a target language translation. For example, because the grammatical rules of the source and target languages ​​differ, the order of words in the source text and the translation may differ, and therefore, words that have the same meaning in different languages ​​need to be aligned. Cross-lingual sentence alignment can be used in a variety of applications, including (1) mining parallel corpora for machine translation systems, (2) filtering existing parallel corpora to improve their quality, and (3) estimating translation quality in scenarios such as evaluating already generated translations or deciding from potential candidates (Quality Estimation tasks). However, training data from different language sources for training cross-lingual sentence alignment models can be extremely different. For example, common languages ​​such as English, Spanish, and French often have richer training datasets, while less common languages ​​such as Mongolian and Malay often lack resources. [Brief explanation of the drawing]

[0004] [Figure 1] This is a simplified block diagram illustrating a training framework for an alignment model according to one or more embodiments described herein. [Figure 2] This is a simplified block diagram showing an embodiment of the BERT score calculation shown in module 106 of Figure 1, according to one or more embodiments described herein. [Figure 3] This is a simplified logical flowchart illustrating a method for training an aligner model for interlingual sentence alignment according to one embodiment described herein. [Figure 4] This is a simplified diagram of a computing device for implementing interlingual text alignment, according to several embodiments. [Figure 5] This document presents various performance results of the aligner models described herein, compared to baseline models in previous studies, based on one or more embodiments described herein. [Figure 6] This document presents various performance results of the aligner models described herein, compared to baseline models in previous studies, based on one or more embodiments described herein. [Figure 7] This document presents various performance results of the aligner models described herein, compared to baseline models in previous studies, based on one or more embodiments described herein. [Figure 8] This document presents various performance results of the aligner models described herein, compared to baseline models in previous studies, based on one or more embodiments described herein. [Figure 9] This document presents various performance results of the aligner models described herein, compared to baseline models in previous studies, based on one or more embodiments described herein. [Figure 10]Shows various performance results of the aligner model described herein compared to the baseline model in previous studies, according to one or more embodiments described herein. [Figure 11] Shows various performance results of the aligner model described herein compared to the baseline model in previous studies, according to one or more embodiments described herein. [Figure 12] Shows various performance results of the aligner model described herein compared to the baseline model in previous studies, according to one or more embodiments described herein. [Figure 13] Shows various performance results of the aligner model described herein compared to the baseline model in previous studies, according to one or more embodiments described herein.

[0005] In the figures and appendices, elements with the same name have the same or similar functions.

Mode for Carrying Out the Invention

[0006] As used herein, the term "network" can include any hardware or software-based framework, including any artificial intelligence network or system, neural network or system, and / or any training or learning model implemented thereon or together therewith.

[0007] As used herein, the term "module" can include a hardware or software-based framework that performs one or more functions. In some embodiments, a module can be implemented on one or more neural networks.

[0008] As used herein, the terms "rich source" or "low resource" refer to the size of the available data. For example, if the size of the parallel data for a language pair exceeds a certain threshold (e.g., 20,000, 30,000, etc.), such a language pair is considered a rich source. Similarly, if the size of the parallel data for a language pair is smaller than a certain threshold (e.g., 10,000 (10K), 20,000 (20K), etc.), such a language pair is considered a low source.

[0009] Inter-language sentence alignment provides an important aspect in machine translation related tasks. Existing systems include two main work flows. One is completely teacherless, and the other is teacher-assisted using data spanning multiple language pairs. The teacherless approach has the advantage of not collecting any parallel data, but it can only achieve relatively low accuracy in benchmark datasets such as Tatoeba (Artetxe et al., Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond, Transactions of the Association for Computational Linguistics, 7:597-610, 2019) which are evaluated on 36 language pairs including multiple low-resource language pairs. On the other hand, the teacher-assisted approach assumes access to data for a large number of low-resource language pairs, which, by definition, is difficult to obtain and guarantee its quality. This all-or-nothing data selection between the teacherless approach and the teacher-assisted approach leaves a large gap regarding whether zero-shot inter-language transfer is effective for such tasks. Therefore, it is still difficult to efficiently distribute the effort for collecting parallel data between languages.

[0010] In light of the need to improve the efficiency and accuracy of interlingual sentence alignment, the embodiments described herein provide an interlingual sentence alignment framework that is trained only on rich resource language pairs, which are extremely resource-rich (as long as one language pair is functional). To obtain an accurate aligner, a pre-trained multilingual model is used as a classifier to be trained on parallel data from rich resource language pairs. This trained classifier can then be used for interlingual transfer with low-resource languages.

[0011] For example, the XLM-RoBERTa (XLM-R) network (described in Conneau et al., Unsupervised cross-lingual representation learning at scale, in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, Association for Computational Linguistics, 2020) can be employed as a pre-trained multilingual model for alignment tasks. A supervised version of the BERT score can be computed as semantic similarity between sentences, and a normalization layer is incorporated into the XLM-R architecture to counteract the popular sentence effect, for example, when some sentences in the source language tend to yield high scores for any given sentence in the target language. In this way, when trained on any single rich resource language pair, the model will perform (e.g., with a precision difference of 2.0 or less) to match a model trained on all language pairs under the same data budget.

[0012] In one embodiment, while adhering to the constraint of using only rich resources, the number of language pairs is scaled up to the top k rich resource language pairs to further narrow the narrow gap between using one language pair for training and using all pairs. This upgrade further improves the model's accuracy, falling just 0.4 short (shy) compared to training on all language pairs under the same data budget.

[0013] In one embodiment, the model is trained on language pairs that are not English-centric, for example, language pairs where none of the sentences are in English. The model's training performance is largely unaffected even when English is not used as the anchor language. The model performs similarly as long as the amount of parallel data exceeds a certain threshold.

[0014] Figure 1 is a simplified block diagram showing a training framework for an alignment model according to one or more embodiments described herein. Figure 1 shows the first batch 102 of statements in the source language, e.g., S={S1,S2,…,S M} and the second batch of sentences in the target language, for example T={T1,T2,…,T N This indicates that the values ​​are input to the alignment model 110 for alignment.

[0015] Specifically, the alignment model 110 checks whether each sentence in batches 102 and 104 matches, for example, for all i=1,...,M and j=1,...,N, S i 102a is T j It is possible to determine whether it is aligned with 104a. The alignment model 110 can be built on top of XLM-R, and is a Transformer-based model pre-trained on the single-language CC-100 dataset covering 100 languages.

[0016] The alignment model 110 may include an embedding model 105, a BERT score calculation module f() 106, and a normalization layer 118. Two input sentences S i 102a and T j Given 104a, the context embedding model 105 encodes the tokens in the source and target sentences into two vector representations, respectively. Specifically, the embedding model 105 may employ BERT (Devlin et al., BERT: Pre-training of deep bidirectional transformers for language under-standing, in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, 201) or ELMo (Peters et al., Deep contextualized word representations, in proceedings of NAACL-HLT, 2018), which generates different vector representations for the same word in different sentences depending on the surrounding words that form the context of the target word. For example, the embedding model 105 may employ a BERT encoder that tokenizes the input text into a series of word fragments, with unknown words being split into several commonly observed strings. The representation of each word fragment is computed using a Transformer encoder by repeatedly applying self-aware transformations and nonlinear transformations alternately.

[0017] Next, as further shown in Figure 2, the BERT score calculation module 106 calculates BERT scores for input sentences 102a and 104a using the cosine similarity between context embedding representations from the embedding model 105, which are optionally weighted by the inverse sentence frequency score.

[0018] FIG. 2 is a simplified block diagram showing an aspect of BERT score calculation shown in module 106 of FIG. 1, according to one or more embodiments described herein. For example, for an input sentence S i when 102a is provided, a sequence of vectors s=<s1, s2, …, sk> represents the contextual embedding 105a of the input sentence S i 102a. Similarly, a sequence of vectors t=<t1, t2, …, t l > represents the contextual embedding 105b of the input sentence T j 102b.

[0019] First, a recall score is calculated by matching each vector in the contextual embedding s=<s1, s2, …, s l > to a vector in the contextual embedding t=<t1, t2, …, t k > that matches it. Specifically, the pairwise cosine similarity 205a between each vector in <s1, s2, …, s k > and the vectors in the contextual embedding <t1, t2, …, t l > is calculated, and the maximum similarity 208a is taken:

Equation

[0020] Similarly, a precision score is calculated by matching each vector in the contextual embedding t=<t1, t2, …, t k > to a vector in the contextual embedding s=<s1, s2, …, s l > that matches it. Specifically, the pairwise cosine similarity 205b between each vector in <t1, t2, …, t l > and the vectors in the contextual embedding <s1, s2, …, s k > is calculated, and the maximum similarity 208a is taken:

Equation

[0021] Next, calculate the BERT score as follows:

number

[0022] Therefore, the BERT score functions as a shallow cross-attention layer, while being much faster than full cross-attention. The resulting model is also independent of the order of the two sentences.

[0023] Referring back to Figure 1, the calculated BERT scores for input sentences 102a and 104a are fed into the normalization layer 118. In bitext alignment, it is often observed that some sentences in one language tend to have high similarity scores with any given sentence in the other language. This phenomenon makes the ranking inaccurate. To compensate for this bias, the scaled mean of the similarity scores between each sentence in one language and all sentences in the other language is subtracted from the BERT score. In other words, the source sentence S i 102a and target sentence T j The normalized similarity score of 115 between 104a and 104a is calculated as follows:

number

[0024] In the inference stage, there are two sentences S i , T jThe determination of whether or not they are aligned can be made, for example, by a thresholding operation. For example, if Similar(i,j) is greater than a predefined threshold, the source statement S i and target sentence T j This can be considered to be a match.

[0025] During training, a challenge to training the aligner model 110 using only positive parallel data is the lack of carefully designed negative examples. To address this challenge, a contrast learning method can be employed, and the aligner model 110 is trained on a classification task that includes negatives within a batch. For example, a batch of sentences in the source language S={S1,S2,…,S N}, and batch of sentences in the target language T={T1,T2,…,T N Regarding}, S i For each i, T i If aligned, pairwise semantic similarity between S and T is calculated, and N similarities are obtained for positive alignments and N for negative alignments. 2 -N similarity scores are obtained (total of N) 2 (The similarity scores of each individual are calculated). During training, these similarity scores are used as logits, with each positive logit paired with all negative logits. These logits are then used to calculate the contrast loss 120, which is then used to update the aligner model 110 via the backpropagation path 125.

[0026] Figure 3 is a simplified logical flowchart illustrating a method for training an aligner model for interlingual sentence alignment according to one embodiment described herein. One or more of processes 302-312 of Method 300 may be implemented in the form of executable code stored in a non-temporary tangible machine-readable medium, which, when executed by one or more processors, causes one or more processors to execute one or more of processes 302-312. In some embodiments, Method 300 may correspond to a method used by Module 430.

[0027] In step 302, the training dataset is received. The training dataset includes at least a first sentence in the source language (e.g., sentence 102a in Figure 1) and a second sentence in the target language which is the translated version of the first sentence (e.g., sentence 104a in Figure 1). For example, the source and target languages ​​belong to the group of rich source languages, such as English-Spanish or English-French.

[0028] In one embodiment, the training dataset could be (1) an English-centric dataset such as OPUS-100, or (2) a non-English-centric language dataset, e.g., v2021-08-07 Tatoeba Challenge. OPUS-100 is English-centric, meaning that all training pairs contain English on either the source or target side. The corpus covers 100 languages ​​(including English). The languages ​​for training are selected based on the amount of parallel data available in OPUS. The OPUS collection consists of multiple corpora ranging from movie subtitles to GNOME documentation and the Bible. OPUS-100 contains approximately 55 million sentence pairs. For example, 99 language pairs could be selected to train an aligner model, of which 44 are selected from 1 million (1M) sentence pairs in the training data, 73 from at least 100,000 (100k) sentence pairs, and 95 from at least 10,000 (10k) sentence pairs. Following the selection of OPUS-100, the training data for each language pair in New-Tatoeba is capped at 1 million (1M) to facilitate comparison with models trained on OPUS.

[0029] In step 304, a positive input pair containing the first and second sentences, and several negative input pairs are formed. Each negative pair contains the first sentence and the corresponding mismatched sentences in the target language. For example, a batch of sentences in the source language S = {S1, S2, ..., S N}, and batch of sentences in the target language T={T1,T2,…,T N Regarding}, S i For each i, T i If aligned, pairwise semantic similarity between S and T is calculated, and N similarities are obtained for positive alignments and N for negative alignments. 2 -N similarity scores are obtained (total of N) 2 (The similarity score is calculated for each item.)

[0030] In step 306, a pre-trained multilingual model may be used to calculate pairwise token-level similarity between two sentences within each positive or negative input pair. For example, pairwise token-level similarity between two sentences may be calculated as the BERT score, as illustrated in Figure 2.

[0031] In step 308, the loss target is calculated based on the calculated pairwise token-level similarity associated with the positive input pairs and multiple negative input pairs. For example, the similarity scores are used as logits, with each positive logit paired with all the negative logits. These logits are then used to calculate the contrast loss between the positive pairs and the negative pairs.

[0032] In step 310, the pre-trained multilingual model is updated based on the loss target.

[0033] In step 312, the updated pre-trained multilingual model may be used to perform the alignment task between a source language and another language different from the target language. For example, a pre-trained multilingual model trained only with rich language pairs (e.g., English-Spanish, English-French, etc.) may then be used to align rare source language pairs (e.g., English-Mongolian, English-Malay, etc.). In some implementations, during the inference phase, the aligner model (the pre-trained multilingual model trained by method 300) may be used to align rare source language pairs without using English as the anchor language, for example, when none of the languages ​​in the rare language pair are English.

[0034] In one embodiment, the training method 300 for the aligner model may be executed on a single A100 graphics processor (GPU). For example, the training batch size is 64 and the evaluation batch size is 256. These are the maximum number of examples that can fit into a batch on a single GPU. A smaller training batch size may result in reduced performance, while a larger batch size can provide more in-batch negatives for each positive example.

[0035] Figure 4 is a simplified diagram of a computing device for implementing interlingual sentence alignment in several embodiments. As shown in Figure 4, the computing device 400 includes a processor 410 coupled to memory 420. The operation of the computing device 400 is controlled by the processor 410. Although the computing device 400 is shown with only one processor 410, it is understood that the processor 410 may represent one or more central processing units, multicore processors, microprocessors, microcontrollers, digital signal processors, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), graphics processing units (GPUs), and / or similar within the computing device 400. The computing device 400 may be implemented as a standalone subsystem, as a board added to a computing device, and / or as a virtual machine.

[0036] Memory 420 may be used to store software executed by computing device 400 and / or one or more data structures used during the operation of computing device 400. Memory 420 may include one or more types of machine-readable media. Some common forms of machine-readable media may include floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media having a pattern of holes, RAM, PROMs, EPROMs, FLASH-EPROMs, any other memory chips or cartridges, and / or any other media adapted for a processor or computer to read from there.

[0037] The processor 410 and / or memory 420 can be located in any suitable physical arrangement. In some embodiments, the processor 410 and / or memory 420 may be mounted on the same board, in the same package (e.g., system-in-package), on the same chip (e.g., system-on-chip), and / or similar. In some embodiments, the processor 410 and / or memory 420 may include distributed, virtualized, and / or containerized computing resources. Consistent with such embodiments, the processor 410 and / or memory 420 may be located in one or more data centers and / or cloud computing facilities.

[0038] In some examples, memory 420 may include non-temporary tangible machine-readable media containing executable code that, when executed by one or more processors (e.g., processor 410), can cause one or more processors to perform methods further described herein. For example, as shown, memory 420 may include instructions for a paraphrase generation module 430 that can be used to implement and / or emulate systems and models, and / or to implement any of the methods further described herein. In some examples, the interlanguage transfer module 430 may receive input 440, such as input text in a source language and / or target language, via a data interface 415. The data interface 415 may be either a user interface that receives articulated audio input, or a communication interface that can receive or retrieve previously stored training text from a database. The interlanguage transfer module 430 may produce output 450, such as an alignment of the input 440 with a sentence in the target language.

[0039] In some embodiments, the interlingual transfer module 430 may implement the embedding module 431 (e.g., similar to 105 in Figure 1), the BERT score module 232 (e.g., similar to 106 in Figure 1), the normalization module 233 (e.g., similar to 118 in Figure 1), and the contrast loss module 434 (e.g., 120 in Figure 1). In some examples, the interlingual transfer module 430 and submodules 431-434 may be implemented using hardware, software, and / or a combination of hardware and software.

[0040] Some examples of computing devices, such as computing device 400, may include a non-temporary tangible machine-readable medium containing executable code that, when executed by one or more processors (e.g., processor 410), causes one or more processors to execute the process of method 300. Some common forms of machine-readable medium that may contain the process of method 300 are, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media having a pattern of holes, RAM, PROMs, EPROMs, FLASH-EPROMs, any other memory chips or cartridges, and / or any other media adapted for a processor or computer to read from there.

[0041] Exemplary performance As described above, the aligner model can be trained with a training dataset of language pairs selected from OPUS-100 and / or the v2021-08-07 Tatoeba Challenge. In one embodiment, all languages ​​from both CC-100 and OPUS-100 are selected for training purposes without assuming a set of language pairs to be tested. Since OPUS-100 was sampled without considering Tatoeba and BUCC evaluation data, any examples from OPUS where either the source or target is present in the test set may be excluded.

[0042] Figures 5 to 13 show various performance results of the aligner models described herein, compared to baseline models in previous studies, using one or more embodiments described herein.

[0043] In some embodiments, the aligner model can be tested via an interlingual sentence extraction task that extracts matching sentences in a target language from a set of sentences. Three evaluation datasets are used. The first dataset is the Tatoeba dataset from the XTREME benchmark (Hu et al., Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization, in International Conference on Machine Learning, pages 4411-4421, 2020), and is called Tatoeba-36 because it contains 36 language pairs. The second dataset is v2021-08-07 Tatoeba Challenge,4, and is called New-Tatoeba. For evaluation purposes, only language pairs with 1000 (1K) or more examples combined in the dev and test sets are retained. Following Tatoeba-36, where most language pairs have 1000 (1K) test examples, 1000 (1K) samples are randomly sampled for each language pair in New Tatoeba. The resulting evaluation set includes 223 language pairs, with 49 English-centric pairs and 174 non-English-centric pairs. The third dataset is BUCC 2018 (Zweigenbaum et al., Overview of the third BUCC shared task: Spotting parallel sentences in comparable corpora, in Proceedings of 11th Workshop on Building and Using Comparable Corpora, pages 39-42, 2018). This is an interlingual bitext mining task. This task is included because both Tatoeba datasets are ranking tasks, while BUCC requires a universal threshold because each sentence in the source language does not necessarily have a translation in the target language during web mining.Therefore, these datasets contain many distraction sentences. The drawback of BUCC is that it only includes four language pairs, all of which are fairly rich resources. Therefore, BUCC only serves as a health check for aligner models.

[0044] Exemplary baseline models for comparison with the aligner model include VECO (Luo et al., VECO: Variable and flexible cross-lingual pre-training for language understanding and generation, in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3980-3994, 2021) and ERNIE-M (Ouyang et al., ERNIE-M: Enhanced multilingual representation by aligning cross-lingual semantics with monolingual corpora, in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 27-38, 2021). Similar to the aligner model described herein (referred to as "OneAligner"), ERNIE-M is built upon XLM-R. It is trained on a single-language corpus and a parallel corpus containing 96 languages. VECO shares the same model size as the aligner model and is trained using a single-language corpus and a parallel corpus containing 50 languages ​​(presumably to avoid capacity dilution). Single-language data is extracted from CC-100, while bilingual data is collected from the OPUS website. There are 6.4 billion (6.4G) parallel examples covering 879 language pairs. Figure 6 shows the basic statistics for each model.

[0045] The aligner model ("OneAligner") is trained across the entire OPUS-100 dataset, with or without a 1 million (1M) budget. Figure 5 shows that both models achieve state-of-the-art results on the Tatoeba-36 dataset. Figures 5 and 6 also show that the aligner model is more data-efficient compared to the other two models.

[0046] Figure 7 shows the performance of Tatoeba-36 for each language pair in the common part of the OPUS-100 and CC-100 languages, compared to models trained on the OPUS-100 dataset. The performance is perfectly consistent across language pairs, suggesting that OneAligner can be fine-tuned for almost any language pair at hand to arrive at a similarly performing model.

[0047] Figure 8 shows a scatter plot of Figure 7 against the data availability of each language pair. After reaching a certain data size threshold (somewhere between 10,000 (10k) and 20,000 (20k)), all language pairs perform similarly. This is partly expected because the aligner model design does not introduce new parameters into XLM-R and eliminates the need to train randomly initialized parameters.

[0048] Similar to Figure 7, Figure 9 shows the accuracy of the aligner model trained on each of the top 16 rich resource non-English-centric pairs from the New-Tatoeba dataset. Again, the performance is consistent across language pairs, indicating that the model can still achieve similar performance on an English-centric evaluation set even when trained on non-English language pairs.

[0049] Figure 10 addresses the inverse question of whether a model trained on English-centric data performs similarly on non-English evaluation data. In Figure 10, when compared column by column, OneAligner performs similarly regardless of whether it is trained on English-centric language pairs or whether there is overlap between the fine-tuning language and the evaluation language. Comparing each model evaluated on English-centric or non-English-centric language pairs, both models perform better on English-centric language pairs. This is because English dominates the training data during XLM-R pre-training.

[0050] For example, instead of bipolar partitioning between "English-centered" and "non-English-centered" areas, the settings can be described using a spectrum, and an X-center can be searched, where X is any language. The precision of language X is defined as the average precision of all language pairs containing X.

[0051] Figure 11 shows a scatter plot of Top-1-Eng New-Tatoeba's performance against the single-language data size for each language in the CC-100 dataset. Similar to Figure 8, New-Tatoeba's performance is positively correlated with the single-language data size up to a certain data threshold (somewhere up to 10.0 GB).

[0052] The results for a single pair of Tatoeba are already satisfactory. However, what if we aimed for even better performance without violating the assumption of rich resources only? It turned out that adding other rich resource pairs could be helpful. Unfortunately, OPUS-100 does not provide a ranking of which language pairs are the richest resources (recall that the size of each language pair in this dataset is capped at 1 million (1M), and the original paper did not include pre-sampling data statistics), so we use the New-Tatoeba dataset and rank each English-centric pair based on its availability.

[0053] Figure 12 shows the performance of the top 1 to top 32 rich resource language pair combinations on the Tatoeba dataset. In Figure 12, performance increases monotonically as more language pairs are added until it reaches an accuracy of 94.0, which is only 0.4 points below the best performance achieved when training with all language pairs under a fixed 1 million (1M) budget. It should be noted that this training does not violate the rich resource limit on training data, as even the least resource-rich language UK in the top 32 list is still within the "highest" resource range defined in the Tatoeba Challenge, containing approximately 34 million training examples. Therefore, at least for the sentence extraction task, the marginal cost of improving accuracy by 0.4 points does not seem to justify the effort to collect more low-resource parallel data with more language pairs over a wide area.

[0054] Figure 13 shows the BUCC F1 scores of two Top1 models compared to previous studies. Both models outperform VECO by 1.2 points. Recall that the English-centric Top1 models are trained on en-es. In other words, the models do not see a single parallel example between en and each of the BUCC target languages ​​{de,fr,ru,zh}, while VECO is extensively trained on each of the language pairs. This result is consistent with the observation that aligner models can perform interlanguage transfer with comparable performance to intralanguage models, regardless of whether the fine-tuning language pair is English-centric.

[0055] This description and accompanying drawings illustrating aspects, embodiments, implementations, or applications of the present invention should not be construed as limiting. Various mechanical, compositional, structural, electrical, and operational modifications can be made without departing from the spirit and scope of this specification and the claims. In some cases, well-known circuits, structures, or techniques are not shown or described in detail so as not to obscure the embodiments of this disclosure. Similar numbers in two or more figures represent the same or similar elements.

[0056] This description provides specific details illustrating several embodiments consistent with the present disclosure. Numerous specific details are provided to provide a complete understanding of the embodiments. However, it will be apparent to those skilled in the art that some embodiments may be carried out without some or all of these specific details. The specific specific embodiments disclosed herein are meant to be illustrative, not limiting. Those skilled in the art will recognize other elements not specifically described herein but within the scope and spirit of the disclosure. In addition, to avoid unnecessary repetition, one or more features shown and described in relation to one embodiment may be incorporated into other embodiments unless otherwise specifically described, or if one or more features would render the embodiment non-functional.

[0057] While exemplary embodiments have been shown and described, a wide range of modifications, alterations, and substitutions are contemplated in the foregoing disclosure, and in some cases, some features of the embodiments may be adopted without corresponding use of other features. Those skilled in the art will recognize many variations, substitutions, and modifications. Therefore, the scope of the present invention should be limited only by the following claims, which should be interpreted to broadly coincide with the scope of the embodiments disclosed herein.

Claims

1. A method for interlingual text alignment performed by a computing device, The steps include receiving a training dataset via a communication interface, which includes at least a first sentence in the source language and a second sentence in the target language that is a translated version of the first sentence; The steps of forming a positive input pair including the first sentence and the second sentence, and a plurality of negative input pairs, each including the first sentence and the respective mismatched sentences in the target language, A step of calculating pairwise token-level similarity between two sentences in each positive input pair or negative input pair via a pre-trained multilingual model, wherein the pairwise token-level similarity between the two sentences is: Based on the cosine similarity between the vector embeddings of the two sentences, a similarity score is calculated between the two sentences. The steps are calculated by subtracting the scaled average of the similarity scores between each sentence of the source language and all sentences of the target language in the training dataset, and between each sentence of the target language and all sentences of the source language, from the calculated similarity score. The steps include: calculating a loss target based on the calculated pairwise token-level similarity associated with the positive input pair and the plurality of negative input pairs; The steps include updating the pre-trained multilingual model based on the loss target, The steps include: performing an alignment task between the source language and another language different from the target language using the updated pre-trained multilingual model; Methods that include...

2. The vector embeddings of the two sentences mentioned above are: The method according to claim 1, wherein the computation is performed by the step of generating a first context embedding and a second context embedding of the two sentences within the respective positive input pair or negative input pair via a context embedding encoder.

3. The pairwise token-level similarity between the two sentences is, The steps include: calculating a first set of pairwise cosine similarities between all vectors in the first context embedding and a first vector in the second context embedding; The steps include determining a first maximum similarity among the first set of pairwise cosine similarities, A step of determining a first score based on the first maximum similarity, The method according to claim 2, calculated by...

4. The steps include: calculating a second set of pairwise cosine similarities between all vectors in the second context embedding and the second vector in the first context embedding; The steps include determining the second maximum similarity in the second set of pairwise cosine similarities, The steps include determining a second score based on the second maximum similarity score and The method according to claim 3, further comprising:

5. Step of calculating the similarity score based on the first score and the second score. The method according to claim 4, further comprising:

6. The method according to claim 1, wherein the loss target is calculated as a contrast loss using the calculated pairwise token-level similarity for each positive or negative input pair as the output logit from the pre-trained multilingual model.

7. The method according to claim 1, wherein the source language and the target language belong to the group of rich source languages.

8. The method according to claim 1, wherein the other language is a rare-source language, and the pre-trained multilingual model is not trained on any language samples of the rare-source language.

9. The method according to claim 1, wherein neither the source language nor the target language is English.

10. It is a system of interlingual sentence alignment, A communication interface that receives a training dataset comprising at least a first sentence in the source language and a second sentence in the target language which is a translated version of the first sentence, Memory for storing pre-trained multilingual models and multiple processor-executable instructions, A processor that executes the plurality of processor-executable instructions in order to perform an operation The operation is provided, A positive input pair comprising the first sentence and the second sentence, and a plurality of negative input pairs, each comprising the first sentence and the respective mismatched sentences in the target language, Compute the pairwise token-level similarity between two sentences in each positive input pair or negative input pair via the pre-trained multilingual model, wherein the pairwise token-level similarity between the two sentences is: Based on the cosine similarity between the vector embeddings of the two sentences, a similarity score is calculated between the two sentences. The calculation is performed by subtracting the scaled average of the similarity scores between each sentence of the source language and all sentences of the target language in the training dataset, and between each sentence of the target language and all sentences of the source language, from the calculated similarity score. The loss target is calculated based on the calculated pairwise token-level similarity associated with the positive input pair and the plurality of negative input pairs, Updating the pre-trained multilingual model based on the aforementioned loss target, Using the updated pre-trained multilingual model, perform an alignment task between the source language and another language different from the target language. A system that includes this.

11. The vector embeddings of the two sentences mentioned above are: The system according to claim 10, wherein the first and second context embeddings of the two sentences are computed by generating them within the respective positive input pair or negative input pair via a context embedding encoder.

12. The pairwise token-level similarity between the two sentences is, Calculate a first set of pairwise cosine similarities between all vectors in the first context embedding and the first vector in the second context embedding, Determining the first maximum similarity among the first set of pairwise cosine similarities, Determining a first score based on the first maximum similarity, Calculate a second set of pairwise cosine similarities between all vectors in the second context embedding and the second vector in the first context embedding, Determining the second maximum similarity in the second set of pairwise cosine similarities, A second score is determined based on the second maximum similarity score, The similarity score is calculated based on the first score and the second score, The system according to claim 11, calculated by:

13. The system according to claim 10, wherein the loss target is calculated as a contrast loss using the calculated pairwise token-level similarity for each positive or negative input pair as the output logit from the pre-trained multilingual model.

14. A non-temporary processor-readable storage medium for storing multiple processor-executable instructions for interlingual sentence alignment, wherein the multiple processor-executable instructions are: The system receives a training dataset via a communication interface, which includes at least a first sentence in the source language and a second sentence in the target language that is a translated version of the first sentence. A positive input pair comprising the first sentence and the second sentence, and a plurality of negative input pairs, each comprising the first sentence and the respective mismatched sentences in the target language, Compute the pairwise token-level similarity between two sentences in each positive input pair or negative input pair via a pre-trained multilingual model, wherein the pairwise token-level similarity between the two sentences is: Based on the cosine similarity between the vector embeddings of the two sentences, a similarity score is calculated between the two sentences. The calculation is performed by subtracting the scaled average of the similarity scores between each sentence of the source language and all sentences of the target language in the training dataset, and between each sentence of the target language and all sentences of the source language, from the calculated similarity score. The loss target is calculated based on the calculated pairwise token-level similarity associated with the positive input pair and the plurality of negative input pairs, Updating the pre-trained multilingual model based on the aforementioned loss target, Using the updated pre-trained multilingual model, perform an alignment task between the source language and another language different from the target language. A non-temporary processor-readable storage medium that is executed by the processor to perform operations including [specific actions].

Images