Pre-training method and device of cross-lingual language model

Abstract

The present disclosure proposes a computer-implemented method and device for pre-training a cross-lingual language model, the method comprising: in response to a stop condition not being met, for a relevance ranking batch of cross-lingual query and document pairs, predicting relevance ranking using a cross-lingual language model with a relevance ranking model head; updating the cross-lingual language model using known relevance ranking and predicted relevance ranking; for a query language modeling batch of cross-lingual query and document pairs, inputting a query and a corresponding document into the cross-lingual language model with multiple tokenizations and some of the multiple tokenizations being masked tokenizations to predict the masked tokenizations in the query; and updating the cross-lingual language model based on a comparison of the predicted results of the cross-lingual language model and actual masked tokenizations; and in response to the stop condition being met, outputting a pre-trained cross-lingual language model.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to systems and methods for computer learning that can provide improved computer performance, features, and uses. More specifically, the present disclosure relates to search systems and methods, particularly systems and methods for cross-language search. BACKGROUND

[0002] Cross-lingual ad-hoc retrieval (CLIR) refers to the task of retrieving documents in a target language L s written in a source language L t The better the CLIR functionality of a search engine, the more extensive the impact, as the search engine can satisfy more users' information needs across the language barrier.

[0003] Recently, the use of monolingual pre-trained language models based on Transformer neural networks (e.g., BERT, see Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). Minneapolis, MN, 4171-4186 (hereinafter “Devlin et al. (2019)”) for English has shown state-of-the-art performance for ad-hoc retrieval. For example, almost all of the top competitors in the MS MARCO passage and document retrieval tasks rely on Transformer-based pre-trained language models. At the same time, multilingual language models (e.g., mBERT (Devlin et al. (2019)) and XLM (Alexis Conneau and Guillaume Lample. 2019. Cross-lingual Language Model Pretraining. In Advances in Neural Information Processing Systems (NeurIPS). Vancouver, Canada, 7057-7067 (hereinafter “Conneau et al. (2019)”) have been shown to perform well on a variety of downstream cross-lingual tasks such as cross-lingual text classification, cross-lingual named entity recognition, and supervised / unsupervised machine translation. However, the wave of multilingual language models has yet to benefit CLIR. SUMMARY

[0004] According to an aspect of the disclosure, there is provided a computer-implemented method of pre-training a cross-lingual language model, the method comprising:

[0005] In response to the stop condition not being reached, the following steps are performed: for a batch of cross-lingual query and document pairs of relevance ranking, using a relevance ranking model head, predicting relevance ranking using the cross-lingual language model, wherein the batch of cross-lingual query and document pairs of relevance ranking includes at least one positive sample cross-lingual query and document pair in which the languages of the query and the document are different and the document is relevant to the query, and at least one negative sample cross-lingual query and document pair in which the languages of the query and the document are different and the document is irrelevant to the query, the relevance ranking is a ranking for relevance between the document and the corresponding query; updating the cross-lingual language model using the known relevance ranking and the predicted relevance ranking; for a batch of cross-lingual query and document pairs of query language modeling, inputting a query and a corresponding document having a plurality of tokens and some of the plurality of tokens being masked tokens into the cross-lingual language model to predict the masked tokens in the query, wherein in the batch of cross-lingual query and document pairs of query language modeling, the languages of the query and the document are different and the document is relevant to the query; and updating the cross-lingual language model based on a comparison of the prediction results of the cross-lingual language model with the actual masked tokens; and

[0006] In response to the stop condition being reached, outputting the pre-trained cross-lingual language model.

[0007] According to another aspect of the present disclosure, there is provided a computer-implemented method of pre-training a cross-lingual language model, the method comprising:

[0008] initializing the cross-lingual language model using a pre-trained multilingual language representation model, wherein a global window for attention is used for query tokens and a sliding window for attention is used for at least some document tokens;

[0009] In response to the stop condition not being reached, the following steps are performed: for a batch of cross-lingual query and document pairs of relevance ranking, using a relevance ranking model head, predicting relevance ranking using the cross-lingual language model, wherein the batch of cross-lingual query and document pairs of relevance ranking includes at least one positive sample cross-lingual query and document pair in which the languages of the query and the document are different and the document is relevant to the query, and at least one negative sample cross-lingual query and document pair in which the languages of the query and the document are different and the document is irrelevant to the query, the relevance ranking is a ranking for relevance between the document and the corresponding query; updating the cross-lingual language model using the known relevance ranking and the predicted relevance ranking; for a batch of cross-lingual query and document pairs of query language modeling, inputting a query and a corresponding document having a plurality of tokens and some of the plurality of tokens being masked tokens into the cross-lingual language model to predict the masked tokens in the query, wherein in the batch of cross-lingual query and document pairs of query language modeling, the languages of the query and the document are different and the document is relevant to the query; and updating the cross-lingual language model based on a comparison of the prediction results of the cross-lingual language model with the actual masked tokens; and

[0010] In response to the stop condition being reached, outputting the pre-trained cross-lingual language model.

[0011] According to another aspect of the present disclosure, there is provided an electronic device comprising:

[0012] at least one processor; and

[0013] a memory communicatively connected to the at least one processor; wherein

[0014] the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above method.

[0015] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform the above method.

[0016] According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the above method.

[0017] This summary outlines some features and advantages of embodiments of the present disclosure; however, the present disclosure also proposes other features, advantages and embodiments, or other features, advantages and embodiments will be apparent to a person skilled in the art based on the drawings, specification and claims of the present disclosure. Therefore, it should be understood that the scope of the present disclosure should not be limited by the specific embodiments disclosed in the summary. BRIEF DESCRIPTION OF DRAWINGS

[0018] Reference will be made to embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. These drawings are intended to be used in illustration, not limitation. While the present disclosure is generally described in the context of these embodiments, it should be understood that the purpose of the present disclosure is not to limit the scope of the present disclosure to these specific embodiments. The items in the figures are not drawn to scale.

[0019] Figure 1 An "INSERT" according to embodiments of the present disclosure is shown.

[0020] Figure 2 A cross-lingual language model according to embodiments of the present disclosure is illustrated.

[0021] Figure 3 A method of pre-training data generation according to embodiments of the present disclosure is shown.

[0022] Figure 4 A multi-lingual data generation method according to embodiments of the present disclosure is shown.

[0023] Figure 5 A pre-training query language modeling task system for cross-lingual retrieval according to embodiments of the present disclosure is shown.

[0024] Figure 6A method of pre-training a cross-lingual language model using a query language modeling task is shown, in accordance with an embodiment of the present disclosure.

[0025] Figure 7 A pre-training system for cross-lingual retrieval utilizing a relevance ranking task is illustrated, in accordance with an embodiment of the present disclosure.

[0026] Figure 8 A method of pre-training a cross-lingual language model with global + sliding window attention using a relevance ranking task is shown, in accordance with an embodiment of the present disclosure.

[0027] Figure 9 A method of pre-training a cross-lingual LM + GSW attention model is shown, in accordance with an embodiment of the present disclosure.

[0028] Figure 10 A method for converting a pre-trained multilingual language representation model to a cross-lingual language model (LM) with global + sliding window (GSW) attention (cross-lingual LM + GSW attention) model is shown, in accordance with an embodiment of the present disclosure.

[0029] Figure 11 An alternative method of pre-training a cross-lingual LM + GSW attention model is shown, in accordance with an embodiment of the present disclosure.

[0030] Figure 12 A cross-lingual information retrieval (CLIR) system is illustrated, in accordance with an embodiment of the present disclosure.

[0031] Figure 13 A method of training a cross-lingual information retrieval (CLIR) system using a pre-trained cross-lingual language model (LM) with global + sliding window (GSW) attention (cross-lingual LM + GSW attention) model is shown, in accordance with an embodiment of the present disclosure.

[0032] Figure 14 A cross-lingual question answering (CLQA) system is shown, in accordance with an embodiment of the present disclosure.

[0033] Figure 15 A method of fine-tuning / training a cross-lingual question answering (CLQA) system using a pre-trained cross-lingual language model (LM) with global + sliding window (GSW) attention (cross-lingual LM + GSW attention) model is shown, in accordance with an embodiment of the present disclosure.

[0034] Figure 16 Results of CLIR performance on CLEF containing various models, including model embodiments in accordance with embodiments of the present disclosure.

[0035] Figure 17 Results containing the CLQA performance of various models (including model embodiments according to embodiments of the present disclosure) on multilingual question answering datasets. By convention, the numerical values are F1 / Exact Match scores (%) in percentile format. The best performance for each pair of languages is marked in bold. “*” indicates statistically significant improvement over mBERT (p<0.05).

[0036] Figure 18 CLIR and CLQA performance of models pre-trained with different objectives (QLM, RR, QLM+RR) according to embodiments of the present disclosure are shown.

[0037] Figure 19 CLIR performance of model embodiments pre-trained with QLM objective under different parameters (masking region, masking probability) according to embodiments of the present disclosure are shown. Round = 0 corresponds to base mBERT.

[0038] Figure 20 A simplified block diagram of a computing device / information processing system according to embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0039] In the following description, specific details are set forth in order to provide a thorough understanding of the present disclosure. However, persons having ordinary skill in the art will appreciate that the present disclosure can be practiced without these details. Moreover, persons having ordinary skill in the art will appreciate that embodiments of the present disclosure described hereinafter can be implemented in a variety of ways, such as a process, an apparatus, a system, a device, or a method on a tangible computer readable medium.

[0040] The components or modules shown in the figures are illustrative of exemplary embodiments of the present disclosure and are intended to avoid obscuring the present disclosure. It will also be understood that throughout the discussion, components can be described as independent functional units, which can include sub-units, but those skilled in the art will recognize that a variety of components or portions thereof can be divided into separate components, or integrated together, including, for example, integrated in a single system or component. It should be noted that the functions or operations discussed for the present disclosure can be implemented as components. The components can be implemented as software, hardware, or a combination thereof.

[0041] Furthermore, connections between components or systems within the figures are not limited to direct connections, rather data can be modified, re-formatted, or otherwise changed by intermediary components. Also, additional or fewer connections can be used. It is also noted that the terms "coupled," "connected," "communicatively coupled," "interface," "interface connection," or any variant thereof are intended to include direct and indirect connections between components, wireless connections, and any combination thereof. It is further noted that any communication, such as a signal, response, reply, acknowledgement, message, query, etc., can include one or more information exchanges.

[0042] References in the specification to "one or more embodiments," "preferred embodiments," "an embodiment,” “embodiments,” etc., mean that a particular feature, structure, characteristic, or function described in connection with the embodiment is included in at least one embodiment of the disclosure and can be included in more than one embodiment. The various appearances of the phrase in

[0043] Certain terms are used throughout the description and claims which refer to particular features or aspects of the embodiments as described. As one skilled in the art will appreciate, the function of a feature can be performed by equivalents. Use of such terms in the claims is merely for descriptive purposes and is not intended to limit the scope of the application to a particular or uniquely-structured application, as such equivalents are within the scope of the application. Also, the use of certain terminology or expressions will not exclude the use of other terminology or expressions that describe or perform a similar function. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted in an overly literal or restricted way unless expressly so defined herein.

[0044] In one or more embodiments, the stopping condition can include: (1) a set number of iterations have been performed; (2) a certain amount of processing time has been reached; (3) convergence (e.g., a difference between successive iterations is less than a first threshold); (4) divergence (e.g., a degradation in performance); (5) an acceptable result has been reached; and (6) all data has been processed.

[0045] Those skilled in the art will recognize that (1) certain steps can be optional; (2) steps can not be limited to the particular order described herein; (3) certain steps can be performed in different orders; and (4) certain steps can be performed concurrently.

[0046] Any headings used herein are for organizational purposes only and are not to be construed as limiting the scope of the application or the claims. Each reference / document mentioned in this patent document is hereby incorporated by reference in its entirety.

[0047] It should be noted that any experiments and results provided herein are provided for illustrative purposes only and are illustrative of one or more particular embodiments under certain conditions. Accordingly, these experiments and their results are not to be used to limit the scope of the disclosure of the current patent document.

[0048] Most related researches focus on the success of cross-language relevance transfer. First for a language L s , monolingual retrieval in L t , where the available training data for L t is usually less. This task, although very important, is different from cross-language ad-hoc retrieval. In the Cross-Lingual Question Answering (CLQA) literature, cross-language relevance transfer is directly referred to as Cross-Lingual Transfer (XLT for short), while the "real" cross-language task of different question and context languages is referred to as Generalized Cross-Lingual Transfer (G-XLT). In this patent document, this naming convention is inherited.

[0049] Current CLIR methods usually use learning-to-rank with a neural match model combined with pre-obtained Cross-Lingual word Embedding (CLE). Some attempts to employ multilingual language models for CLIR show much poorer performance of such models compared to learning-to-rank with CLE. The use of language models is a gap between monolingual (English) ad-hoc retrieval and cross-language ad-hoc retrieval.

[0050] Therefore, there is a need for a system and method to address the gap in language models and language model performance that can be used for various purposes such as CLIR and CLQA.

[0051] Existing research on cross-language retrieval cannot well utilize large-scale pre-trained language models, such as multilingual BERT and XLM. Lack of cross-language passage-level relevance data for fine-tuning and lack of query-document style pre-training are some key factors leading to this problem. Therefore, the present disclosure proposes two new pre-training task embodiments for retrieval to further pre-train cross-language language models for downstream retrieval tasks such as cross-language ad hoc retrieval CLIR and cross-language question answering CLQA. In one or more embodiments, remote supervision data is constructed from multilingual text using segment alignment to support retrieval-oriented language model pre-training. In one or more embodiments, fine-tuning of the language model is directly performed on a portion of the evaluation set by enabling the Transformer to accept longer sequences. Experiments show that the model embodiments significantly improve the general multilingual language model at least in the cross-language retrieval setting and the cross-language transfer setting.

[0052] In one or more embodiments, remote supervision data from multilingual resources is constructed using segment alignment to support retrieval-oriented language model pre-training. The present disclosure also proposes an embodiment of fine-tuning of the language model directly on a portion of the evaluation set by enabling the Transformer to accept longer sequences. Experiments based on multiple data sets show that the embodiments significantly improve the general multilingual language model at least in the cross-language retrieval setting and the cross-language transfer setting.

[0053] System, method and computer readable medium embodiments for pre-training a cross-language language model can include: for a batch of cross-language query and document pairs for relevance ranking, using the cross-language language model to predict relevance ranking with a relevance ranking model head, where the relevance ranking is a ranking for relevance between a document and a corresponding query; updating the cross-language language model using known relevance ranking and predicted relevance ranking; for a batch of cross-language query and document pairs for query language modeling, inputting a query and a corresponding document with multiple tokens and some of the tokens being masked tokens into the cross-language language model to predict the masked tokens in the query, where the languages of the query and the document are different and the document is relevant to the query in the batch of query language modeling; and updating the cross-language language model based on a comparison of the predicted results of the cross-language language model and the actual masked tokens. In one or more embodiments, in response to not reaching a stopping condition, repeating the above steps; in response to reaching the stopping condition, outputting the pre-trained cross-language language model. It should be noted that whether to train the relevance ranking first or the query language modeling first is not limited here.

[0054] In one or more embodiments, the system, method, and computer-readable medium embodiments include obtaining a relevance ranking batch by sampling a batch of cross-lingual query and document pairs, where the relevance ranking batch includes (1) at least one positive sample cross-lingual query and document pair in which the query and document are in different languages and the document is relevant to the query, and (2) at least one negative sample cross-lingual query and document pair in which the query and document are in different languages and the document is irrelevant to the query.

[0055] In one or more embodiments, the system, method, and computer-readable medium embodiments include obtaining a query language modeling batch by sampling a batch of cross-lingual query and document pairs in which the query and document are in different languages and the document is relevant to the query.

[0056] In one or more embodiments, the cross-lingual language model includes global plus sliding window attention, where a global window for attention is used for query tokenization and a sliding window for attention is used for at least some document tokenization.

[0057] In one or more embodiments, the cross-lingual language model is initially obtained by performing the following steps: obtaining a pre-trained multilingual language representation model that includes an attention module and that is trained with a Masked Language Modeling (MLM) task objective; replacing the attention module in the pre-trained multilingual language representation model with global plus sliding window attention to obtain a modified language model; updating the modified language model using the MLM task objective and a dataset to improve performance of the modified language model relative to the pre-trained multilingual language representation model; and outputting the updated modified language model as a pre-trained cross-lingual language model.

[0058] In one or more embodiments, at least one sampling step includes sampling from a plurality of different languages to train the pre-trained cross-lingual language model to accept as input and output as output each of the plurality of different languages.

[0059] In one or more embodiments, cross-lingual data can be obtained by performing the following steps: matching a plurality of segments of one or more multilingual resources that are related to a same topic; and extracting a sentence or a portion of a sentence from a matching segment in a first language as a query and using a matching segment or a portion of the matching segment in a language different from the first language as a cross-lingual document.

[0060] In one or more embodiments, the pre-trained cross-lingual language model can be used for one or more downstream tasks. For example, in one or more embodiments, a CLIR model comprising a pre-trained cross-lingual language model and a cross-lingual information retrieval (CLIR) model head is trained by performing the following steps: initializing parameters or weights of the CLIR model head with random values, model weights, or a combination of the random values and the model weights, where the model weights are from a relevance ranking model head used in pre-training; training using positive query-document pairs and negative query-document pairs to update parameters or weights of the pre-trained cross-lingual language model, parameters or weights of the CLIR model head, or parameters or weights of both the cross-lingual language model and the CLIR model head; and outputting the trained CLIR model. For another example, in one or more embodiments, a CLQA model comprising a pre-trained cross-lingual language model and a cross-lingual question answering (CLQA) model head is trained by performing the following steps: initializing parameters or weights of the CLQA model head; training using training samples to update parameters or weights of the pre-trained cross-lingual language model, parameters or weights of the CLQA model head, or parameters or weights of both the pre-trained cross-lingual language model and the CLQA model head; and outputting the trained CLQA model.

[0061] System, method, and computer-readable medium embodiments for pre-training a cross-lingual language model can include, in response to a stop condition not being reached, performing the following steps: (1) for a cross-lingual query and document pair, training the cross-lingual language model using a query language modeling objective, where at least some query tokens of a query input to the cross-lingual language model are masked query tokens in the cross-lingual query and document pair, the query language modeling objective involves predicting the masked query tokens for a query given a corresponding relevant cross-lingual document; and (2) for a cross-lingual query and document pair, training the cross-lingual language model with a relevance ranking model head to predict a relevance ranking, the relevance ranking being a ranking for relevance between a document and a corresponding query; and in response to the stop condition being reached, outputting the pre-trained cross-lingual language model. Note that it does not matter whether the relevance ranking is trained first or the query language modeling is trained first.

[0062] In one or more embodiments, the cross-lingual language model can include global plus sliding window attention, where a global window for attention is used for query tokens and a sliding window for attention is used for at least some document tokens.

[0063] In one or more embodiments, the pre-trained multilingual language representation model can be used to initialize the cross-lingual language model.

[0064] In one or more embodiments, the system, method, and computer-readable medium embodiments can initialize a cross-lingual language model by performing the following steps: obtaining a pre-trained multi-lingual language representation model, the pre-trained multi-lingual language representation model including an attention module, and the pre-trained multi-lingual language representation model trained with a Masked Language Modeling (MLM) task objective; replacing the attention module in the pre-trained multi-lingual language representation model with a global plus sliding window attention to obtain a modified language model; updating the modified language model using the MLM task objective and a dataset to improve performance of the modified language model relative to the pre-trained multi-lingual language representation model; and outputting the updated modified language model as a cross-lingual language model for pre-training.

[0065] In one or more embodiments, the system, method, and computer-readable medium embodiments can obtain a set of cross-lingual query and document pairs for training a cross-lingual language model with the correlation ranking model head by performing the following steps: sampling a set of cross-lingual query and document pairs, where the set of cross-lingual query and document pairs includes: at least one positive sample cross-lingual query and document pair in which the query and document are in different languages and the document is relevant to the query; and at least one negative sample cross-lingual query and document pair in which the query and document are in different languages and the document is not relevant to the query.

[0066] In one or more embodiments, the system, method, and computer-readable medium embodiments can obtain a set of cross-lingual query and document pairs for training a cross-lingual language model with the query language model objective by performing the following steps: sampling a set of cross-lingual query and document pairs in which the query and document are in different languages and the document is relevant to the query.

[0067] In one or more embodiments, at least one sampling step includes sampling from a plurality of different languages to train the pre-trained cross-lingual language model to accept as input and output as output each of the plurality of different languages.

[0068] In one or more embodiments, the pre-trained cross-lingual language model can be used for one or more downstream tasks, such as Cross-Lingual Information Retrieval (CLIR) and / or Cross-Lingual Question Answering (CLQA).

[0069] A. GENERAL INTRODUCTION

[0070] In one or more embodiments, to address the shortcomings of existing cross-lingual (CL) language models (LMs), the differences between pre-training and applying a cross-lingual LM are considered. A prerequisite for retrieval using a cross-lingual LM is a good cross-lingual alignment representation at multiple text segment (i.e., word, sentence, paragraph, and document) levels. Conneau et al. (2020) (Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Emerging Cross-lingual Structure in Pretrained Language Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL). Online, 6022-6034) point out that representations from different languages of monolingual BERTs can linearly map each other at word and sentence levels, and the success of unified cross-lingual LMs mainly lies in the parameter sharing in the upper encoder. Both mBERT and XLM focus on word-level and sentence-level tasks during pre-training: the Masked Language Modeling (MLM) task trains the model to fill in the blanks of monolingual sentences, while the Translation Language Modeling (TLM) task makes the model to challenge to fill in the blanks in pairs of parallel sentences. These two modeling tasks perform well in word-level and sentence-level tasks, but perform poorly in retrieval tasks, which indicates that the representation of longer sequences can not be well aligned in cross-lingual LMs.

[0071] To this end, at least two new pre-training objectives are presented herein for better aligning the representation of longer texts and better modeling the query-document interaction.

[0072] In one or more embodiments, the Query Language Modeling (QLM) task masks some query tokens and requires the model embodiment to predict the masked tokens based on the query context and the full relevant foreign document. In one or more embodiments, the masking probability is significantly improved compared to the autoencoder language modeling task to strengthen the reference to cross-lingual long sequences.

[0073] In one or more embodiments, the Relevance Ranking (RR) task operates on a coarser-grained representation and directly resembles the cross-lingual ad-hoc retrieval task. For a query and multiple foreign documents, the model embodiment is required to rank these documents based on the relevance level.

[0074] At least some of the cross-lingual LM embodiments herein are not fully “self-supervised” due to the use of some knowledge of query-document relevance for two pre-training objectives. Embodiments of a simple and effective method for constructing such distantly weakly supervised data from multilingual data are also described herein. Specifically, in one or more embodiments, for multiple segments of multilingual versions of data entities, matching is performed based on cross-lingual representations of segment titles. For each pair of matched document segments, a sentence is extracted from one segment as a query and the other segment is used as a relevant document. Text segments such as those from Wiki are a source of more fine-grained semantics in various retrieval datasets, and in one or more embodiments, the concept is extended to multilingual text. In one or more embodiments, millions of raw matching segments are generated for each language pair. Thus, for the pre-training task in the embodiments herein, this data extraction scheme is a good balance point between quantity and quality.

[0075] The pretrain-finetune-inference paradigm for zero-shot retrieval using language models is re-evaluated herein. Due to the quadratic time and memory complexity of the full self-attention mechanism of Transformers, there is a small upper limit on the input sequence length for Transformers and language models (e.g., 512 tokens for BERT). However, in most cases, 512 tokens is not enough to encode queries and full documents when performing fine-tuning. Referring to Figure 1 , Figure 1 The distribution of document lengths in the dataset is shown for English and French documents. In Figure 1In this context, the positions of the 512 tokenized words are marked as 115 for reference. In practice, considering the length of the query, a document with 512 tokens is not appropriate. Current monolingual retrieval research either truncates the document so that the input sequence meets the size requirement or fine-tunes a language model on a passage-level set and then post-aggregates the relevance scores of the document segments to reason over longer collections of text. However, it should be noted that truncating the document results in some degree of information loss. Furthermore, there is no multilingual passage-level relevance dataset like MS MARCO for English retrieval. The embodiments herein seek to fine-tune LMs for downstream retrieval tasks directly on the evaluation set (similar to non-retrieval cross-lingual tasks) but also seek to minimize information loss during processing. To this end, in one or more embodiments, the self-attention mechanism in the Transformer is replaced by a global + sliding window (GSW) attention to unlock the ability of the cross-lingual LM to process longer sequences in a “Transformer-internal” fashion. Note that there is also a “Transformer-external” solution where the original Transformer slides over the document and a parameterized saturation function aggregates multiple windows and outputs a score. In contrast, the embodiments herein are more computationally efficient, especially considering that large-scale pre-training is performed in addition to fine-tuning.

[0076] Some contributions of the embodiments include, but are not limited to, the following contributions.

[0077] - Embodiments are presented herein for two new retrieval-oriented tasks for pre-training cross-lingual language models. In one or more embodiments, weakly supervised data is constructed to support pre-training of cross-lingual LMs with these tasks.

[0078] - Global + sliding window attention is used in the cross-lingual language model embodiments to better align longer text representations across languages during the pre-training phase and the fine-tuning phase while minimizing information loss.

[0079] - The embodiments are extensively evaluated on downstream CLIR and CLQA tasks. Detailed experiments are also conducted to support the explanation of each component from an empirical perspective. For CLIR, the embodiments achieve 13.9% to 29.7% MAP improvement over vanilla mBERT re-ranking in all 12 language pairs on the benchmark CLEF dataset. For cross-lingual QA datasets, F1 is improved by 1.7 and 2.8 points in the XLT setting (German and Spanish) and by 3.6 to 9.8 points in the G-XLT setting (6 language pairs) over mBERT on the multilingual QA dataset.

[0080] For convenience, the following table is provided.

[0081] Table 1: Abbreviations commonly used in this disclosure.

[0082]

[0083] 1 Cross-lingual language models are presented in (1) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised Cross-lingual Representation Learning at Scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL). Online, 8440-8451 (hereinafter “Conneau et al. (2020b)”); and (2) Conneau et al. (2019).

[0084] B. Related Work

[0085] 1. Cross-lingual ad hoc retrieval

[0086] Cross-lingual ad hoc retrieval has been considered as a combination of machine translation and monolingual ad hoc retrieval. Initial translation resources came from the field of statistical machine translation (SMT). Some early works used word-by-word translation. Some developers used a conversion table from SMT to convert queries into structured queries of structured probabilities. CLIR methods gradually shifted to using cross-lingual word embeddings as translation resources. There are generally two approaches to obtaining CLE: pseudo-bilingual and post-projection. Some developers proposed heuristic methods for cross-lingual ad hoc retrieval using CLE. Recently, the combination of neural match models and CLE was proposed for document re-ranking and achieved impressive performance on standard benchmarks.

[0087] 2. Pre-trained LMs for monolingual IR

[0088] Pre-trained language models have revolutionized the field of human language technology in general. Here we take BERT as an example, but other language models can be used. There are two main approaches to apply BERT to ad-hoc retrieval: (i) single-tower: pack the query-document pair into one sequence separated by [sep] tokens and feed into one BERT encoder. During encoding, each query / document token can attend to the whole sequence (also called cross-attention). The output of the [cls] token is used to predict ranking scores; and (ii) two-tower: query and document are encoded using separate BERT encoders. The matching score is the similarity (e.g., cosine similarity) between the two sequence embeddings. Two-tower models are more efficient in indexing query and document vector representations and are typically used for the first stage retrieval, while single-tower models with full cross-attention are typically used for the last stage of document re-ranking. For document re-ranking, introducing term-level matching signals from word embeddings in addition to the cross-attention output [cls] vector can provide additional improvements. The embodiments here address the re-ranking problem and, for simplicity, the embodiments use a single-tower model without resorting to term-level matching.

[0089] 3. Cross-lingual pre-trained LMs

[0090] Cross-lingual pre-trained language models are able to encode text from multiple languages simultaneously. mBERT (Devlin et al. (2019)) adopts the same model structure and training objectives as BERT and is pre-trained on Wikipedia in 100+ languages. In addition to the Masked Language Modeling (MLM) objective, the XLM model (Conneau et al. (2019)) is pre-trained with the Translation Language Modeling (TLM) objective to leverage parallel sentence resources if available: a pair of parallel sentences is randomly masked and the language model is challenged to predict the masked tokens by attending to local and remote foreign context. XLM-RoBERTa (Conneau et al. (2020b)) improves upon XLM by introducing more training data. Two additional word-level and sentence-level tasks are proposed to pre-train Unicoder. Evaluation on a range of word-level and sentence-level cross-lingual transfer tasks shows that these cross-lingual LMs have important practicality for transferring language knowledge from high-resource languages to low-resource languages.

[0091] In the retrieval side, there are also some works on cross-lingual transfer for ad-hoc retrieval and question answering. But unlike cross-lingual transfer, using a single tower model for cross-lingual retrieval requires the language model to encode two sequences from different languages (CLIR: query / document, CLQA: question / context) in the same process. Some developers reported that their attempts to use a single tower model and CEDR-like matching model for CLIR were unsuccessful, stating that leveraging a multilingual pre-trained model does not provide high gains for CLIR and further research is needed to fine-tune or train. There is only one detailed report on CLIR using a single tower model (Zhuolin Jiang, Amro El-Jaroudi, William Hartmann, Damianos G. Karakos, and Lingjun Zhao. 2020. Cross-lingual Information Retrieval with BERT. In Proceedings of the workshop on Cross-Language Search and Summarization of Text and Speech (CLSSTS@LREC). Marseille, France, 26-31 (hereinafter “Jiang et al. (2020)”), which presents a method that decomposes the query into terms and the document into sentences. Thus, the complexity of their model is quadratic based on vanilla BERT cross-attention, and thus far from practical. In contrast, the embodiments herein are able to encode the full query and document in both languages at once.

[0092] C. Model Structure Embodiments

[0093] Figure 2 The structure of an embodiment of a cross-lingual LM 205 is shown. The sentencepiece tokenization 220 can attend to neighboring tokens in a w-token window, while the query tokenization has global attention. Although other values can be used, for illustration in Figure 2 the experiments, the w-token window was set to w = 2. In the experiments, larger window sizes were used (w = {32, 64, 128, 256}).

[0094] As shown, the input is a packed sequence containing one query 210 and one document 220, separated by a [sep] token. To encourage learning representations for unknown languages, unlike XLM, the embodiments do not supplement language-specific embeddings. Instead, in one or more embodiments, the input segment embeddings are to let the model embodiments distinguish between the two parts of the input.

[0095] As Figure 1As shown, the majority of documents in the CLIR dataset (and generally in practical applications of CLIR) exceed the input length limit of mBERT and XLM. The cross-lingual LM embodiments herein are built to encode more document content at the pre-training, fine-tuning, and inference stages. The embodiments employ an attention mechanism similar to that used in Longformer (Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 (2020) (hereinafter “Longformer (2020)”) to replace the full self-attention of mBERT, such that each Transformer block can encode longer sequences. Compared to other solutions that slide vanilla Transformers over long documents to fine-tune monolingual language models, the embodiments herein are more computationally efficient, especially at the pre-training stage.

[0096] Longformer is a long-document transformer where O(n 2 ) complexity self-attention is replaced by a series of linear attention mechanisms. Specifically, in one or more embodiments, a “global + sliding window” (GSW) attention is used. In embodiments, query tokens are allowed to have global attention, and document tokens are restricted to sliding window attention. In each Transformer, as shown in Figure 2 , all query tokens still attend to any other token in the sequence, but document tokens attend to tokens in a w-token wide window. In one or more embodiments, specific tokens like [sep] and [cls] also have global attention. An example of a Transformer is discussed in Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All You Need. In Advances in Neural Information Processing Systems (NIPS). Long Beach, CA, 5998-6008. The Transformer can compute attention scores as follows:

[0097]

[0098] where the input is of dimension dk query and keys and dimensions d v values. Compute dot products of query with all keys, divide each by Vd k and apply softmax function to obtain weights of these values. Q represents query matrix, K represents key matrix, V represents value matrix, all of which can be used to compute attention function for query set.

[0099] On the other hand, GSW embodiments can use two sets of projections: {Qs, Ks, Vs} and {Qg, Kg, Vg}, {Qs, Ks, Vs} are used to compute attention scores for sliding window attention, {Qg, Kg, Vg} are used to compute attention scores for global attention. Intuitively, one can view GSW as a reallocation of computational power: instead of letting two distant document tokens attend to each other, embodiments can let the query token attend more to document tokens. In practice, using a single tower model, a comparison between (a) full self-attention with maximum sequence length of 512 and (b) GSW attention with maximum sequence length of 1024 and window size of 64, found that (i) they consume similar GPU memory; (ii) GSW runs slightly slower; and (iii) GSW is able to encode documents that are more than twice as long. An empirical comparison of the effectiveness of the two is given below.

[0100] D. Retrieval-oriented cross-lingual language model pre-training embodiments

[0101] One goal of language model pre-training for specific tasks is to further improve the performance of the model on downstream tasks by leveraging weakly supervised data that is applied to the specific task modeling objective. This section describes weakly supervised data construction and retrieval-oriented cross-lingual modeling tasks.

[0102] 1. Data

[0103] General requirements for ideal pre-training data for the tasks in this paper include: (i) each positive example contains a pair of short text (query) and long text (document) in different languages; (ii) the query and document are semantically related; and (iii) the number of training examples is large enough.

[0104] Figure 3A pre-training data generation method according to embodiments of the disclosure is shown. There is a trade-off between the granularity of semantic relevance and the number of available training examples, with data with a heading (e.g., a Wikipedia section) having a good balance point. To this end, multilingual sections of corresponding documents in different languages are matched (305). It should be noted that in one or more embodiments, these sections can not be direct language translations, but rather represent sections discussing the same topic(s). Sentences (which can be one or more word phrases) are extracted (310) from the matched sections in the first language as queries, and the matched sections (or portions thereof) in the other language are extracted as corresponding documents. The motivation of this approach is to discuss the semantic granularity in monolingual Wikipedia. While this approach can be considered conceptually similar to an inverse cloze task (ICT), i.e., extracting a sentence from a Wikipedia paragraph as a query, with the rest of the paragraph as a document, there are some key differences. Key differences include (but are not limited to): (i) the embodiments herein extend from monolingual to multilingual; and (ii) longer texts (sections vs. paragraphs) are kept as documents, which is more similar to the downstream retrieval task.

[0105] However, even for Wikipedia content, accurate cross-lingual section alignment information is typically not available. Multilingual content for the same entity (e.g., multilingual Wikipedia pages) are typically not translations of each other, and they are typically organized with different structures. In fact, multilingual Wikipedia section alignment is itself an active research problem. In one or more embodiments, an efficient and effective section alignment method based on cross-lingual word embeddings (CLE) is employed.

[0106] Figure 4 A multilingual data generation method according to embodiments of the disclosure is shown. Assume Page s and Page t are two documents (e.g., two Wikipedia pages) in source and target languages, respectively, for the same entity. The heading of a section can be defined as the heading immediately preceding the section. For each section Sec s in Page s (i), the heading embedding of the section is obtained (405). In one or more embodiments, the embedding for the heading can be obtained by averaging the CLEs (removing stop words) of all the terms of its heading. The embedding for each section Sec t (j) in Page t can be similarly obtained (410). The heading embedding from the source language document can be compared (415) with the heading embedding from the target language document. If the heading embedding of Secs (i) is similar to Sec t(j) the title embedding similarity (e.g., cosine similarity or other similarity measure) is greater than a threshold η, then these segments can be considered (420) matching segments. The underlying assumption is that: (i) the title is an accurate summary of the segment content; (ii) matching segments are related to the same aspect of the same entity. Conceptually, the relevance of matching segments is “lower-bounded”, so in the worst case, two segments are related to different aspects of the same entity, which is still acceptable for retrieval. Note that for both languages, in one or more embodiments, one segment is allowed to match at most one foreign segment, and if there is a conflict, the highest matching pair can be chosen. The quality of the data can be reflected by the performance of the pre-trained model on downstream tasks, as described in the experimental section below.

[0107] It should be noted that the model embodiment and the data construction method embodiment support any language present in the multilingual Wikipedia. For ease of demonstration and evaluation, four languages {English (En), Spanish (Es), French (Fr), German (De)} are chosen. Using FastText CLE, η is set to η = 0.3 (note that CLE generally has lower cosine similarity than monolingual embeddings). A filtering is implemented so that a pair of matching segments has at least five sentences in both segments. The number of aligned segments created is listed in Table 2.

[0108] Table 2: Number of aligned segments in each language pair.

[0109]

[0110] 2. Pre-training task embodiments

[0111] For the large amount of cross-lingual query-document relevance data described, at least two new pre-training tasks or objectives are developed for cross-lingual retrieval: Query Language Modeling (QLM) and Relevance Ranking (RR).

[0112] a) Query Language Modeling (QLM) embodiment

[0113] Figure 5 A pre-training system for cross-lingual retrieval utilizing the query language modeling task is illustrated diagrammatically in accordance with embodiments of the present disclosure. Figure 6A method of pre-training a cross-lingual language model with global + sliding window attention using a query language modeling task is shown, in accordance with embodiments of the present disclosure. For a pair of cross-lingual query 510 and document 520, a certain percentage of query tokens 515 are masked (605), which is random. The output from the CL LM + GSW attention model 505 is input (610) to a model head (not shown) that outputs a prediction of the masked tokens (e.g., predicted words 525). For example, in one or more embodiments, the final hidden vectors corresponding to the masked query tokens are fed into an output softmax over the vocabulary to generate the final output words 525. In one or more embodiments, the cross-lingual LM + GSW attention model embodiment is updated or trained (620) based on a comparison of the predictions of the cross-lingual LM + GSW attention model embodiment to the actual masked tokens.

[0114] In one or more embodiments, if 15% of the query tokens are masked, the QLM will be easier than the MLM because there is an extra full foreign document to support the predictions. To this end, in one or more embodiments, the masking probability can be increased to at least 30% to strengthen the attention from the query to the foreign document. If the document tokens are also masked, the QLM reduces to approximately the TLM. However, masking the document tends not to promote cross-lingual attention because the information from the short foreign query is negligible to help complete the long document. Thus, in one or more embodiments, masking the query tokens only, at a higher probability, best promotes cross-lingual query-document interaction understanding. Empirical experiments provided in § F.3.b below give reasons for these choices.

[0115] b) Relevance ranking (RR) embodiments

[0116] Figure 7 A pre-training system for cross-lingual retrieval with a relevance ranking task is shown diagrammatically, in accordance with embodiments of the present disclosure. One or more embodiments of this task can be viewed as similar to the cross-lingual zero-shot retrieval task, but with significant differences. For example, the data has a coarser-grained semantics than the fine-tuning data.

[0117] For a pair of cross-lingual query 710 and document 720, a model 705 encodes the packed sequence QD + (e.g., sequence 735), a model head (e.g., a learnable weight matrix W 722) multiplies the output hidden vector of the [cls] token, and produces a ranking score S + 725. Irrelevant documents can also be sampled to form the sequence QD -740, and a ranking score S-730 is obtained in a similar manner. The model 705 can be optimized with a cross-entropy loss that supports multiple negative examples. In one or more embodiments, one negative example can be sampled for each positive example, although different ratios can be used. For a query Q and a document D, where Q e Sec s (i) e Page s and D = Sec t (j) e Page t , then {Sec t (k)} k≠j is considered a hard negative example. To avoid repeating the same negative document in multiple training rounds with almost no fragments in Page t , the sampling probability of hard negative examples can be dynamically adjusted according to the source entity from which the positive query-document pair is created. In one or more embodiments, the sampling probability of a hard negative example is set to a T = 1 - (3 / 4) T , where T is the number of fragments in Page t . With a probability of 1 - a T , a fragment of language L t can be randomly sampled as an irrelevant document.

[0118] Figure 8 An alternative method of pre-training a cross-lingual language model with global + sliding window attention using a relevance ranking task is shown, according to embodiments of the present disclosure. In one or more embodiments, a batch of cross-lingual query and document pairs are sampled (805), where at least one cross-lingual query and document pair is a positive example and at least one cross-lingual query and document pair is a negative example. Because the queries are cross-lingual, the language of the query is one language and the language of the corresponding document is another language. A positive example means that the query and the corresponding document in the query and document pair are relevant, and a negative example means that the query and the corresponding document in the query and document pair are irrelevant. The cross-lingual query and document pairs are input (810) into a cross-lingual language model (LM) with global + sliding window (GSW) attention (cross-lingual LM + GSW attention) embodiment (e.g., model 705). The cross-lingual LM + GSW attention embodiment is used (815) to generate an encoded output, which is input (820) into a model (e.g., model (722) that predicts a relevance ranking (e.g., S + or S - ) for the relevance between the query and the document. With the known relevance ranking and the predicted relevance ranking, the cross-lingual LM + GSW attention embodiment can be updated (825) with a loss function (e.g., cross-entropy loss).

[0119] 3. Pre-training embodiments

[0120] Figure 9 A method of pre-training a cross-lingual LM+GSW attention model is shown, according to embodiments of the disclosure. Pre-training a language model from scratch can incur high computational cost. In one or more embodiments, pre-training a retrieval-oriented language model embodiment can start from a pre-trained language model (905). For example, a public mBERT checkpoint (available at huggingface.co / bert-base-multilingual-uncased) can be used as a starting point, which has been pre-trained on over 100 languages using the MLM objective, but other language models can also be used. Thus, the cross-lingual LM embodiment is pre-trained with three objectives (MLM, QLM, and RR) implicitly. The following discussed Figure 10 A method embodiment with additional details for initializing a cross-lingual language model (cross-lingual LM+GSW) embodiment with global+sliding window attention is shown.

[0121] The following naming convention is used to identify different variants of model embodiments: “(model, objective, number of rounds)”. A model with full self-attention is referred to as mBERT directly, while a model with GSW attention is referred to as mBERT-GSW. For example, (mBERT-GSW, QLM-RR, 10 rounds) is a cross-lingual LM+GSW attention model pre-trained with QLM and RR for 10 rounds.

[0122] When pre-training with two objectives, in one or more embodiments, in each iteration, the relevance ranking (RR) is trained (910) with language pairs in random order, and then the query language modeling (QLM) is trained (915) with language pairs in random order. In one or more embodiments, each round contains 32K positive query-document pairs for each language pair for each objective. The training embodiments use 16 Nvidia VOLTA (16 GB) GPUs. Adam is used for model optimization. The learning rate is set to 1e-5, and the batch size is set to 32. In one or more embodiments, a maximum of 20 rounds is set for training. Training mBERT takes approximately 24 hours, while training mBERT-GSW (window size = 64, maximum sequence length = 1024) takes approximately 40 hours.

[0123] Figure 10A method of converting a pre-trained multilingual language representation model to a cross-lingual language model (LM) with global + sliding window (GSW) attention (cross-lingual LM + GSW attention) model is shown in accordance with embodiments of the present disclosure. Given (1005) an initial pre-trained multilingual language representation model (e.g., mBERT), in one or more embodiments, an mBERT-GSW with 1024 tokenization is obtained as initialization by replacing (1010) the attention module in mBERT to GSW attention and copying (1015) the position embeddings from the first 512 tokenization positions to the next 512 tokenization positions. Due to the non-optimized position embeddings from copying, the inexperienced mBERT-GSW can have worse language modeling capability (reflected in high per character bits per character (BPC) metric). Thus, in one or more embodiments, an MLM task can be used on a dataset (e.g., a WikiText-103 dataset) and 2K gradient updates are performed (1020) so that the mBERT-GSW has a similar BPC as the base mBERT. Upon completion of training, the pre-trained modified model (i.e., pre-trained cross-lingual LM + GSW attention embodiment) is output. In one or more embodiments, the output pre-trained cross-lingual LM + GSW attention embodiment can then be used in training with RR and QLM tasks as described above.

[0124] Figure 11 An alternative method of pre-training a cross-lingual LM + GSW attention model is shown in accordance with embodiments of the present disclosure. In one or more embodiments, a pre-trained multilingual language representation model is converted (1110) to a cross-lingual language model using global + sliding window attention (i.e., a cross-lingual LM + GSW attention embodiment). For example, Figure 10 An example method for performing this conversion is shown.

[0125] In one or more embodiments, a batch of cross-lingual query and document pairs is sampled (1115) in which at least one cross-lingual query and document pair is a positive sample (i.e., relevant to a corresponding document (in another language)) and at least one cross-lingual query and document pair is a negative sample (i.e., not relevant to a corresponding document) in the batch of cross-lingual query and document pairs. This batch is used to train (1120) the cross-lingual LM + GSW attention embodiment with a relevance ranking (RR) objective / task.

[0126] In one or more embodiments, a batch of cross-language query and document pairs is also sampled (1125), wherein a certain percentage of query tokens are masked. It should be noted that positive samples from the RR training batch can be used for this batch. Other data from previous samples (e.g., using queries but identifying the correct corresponding documents) can also be used. This batch is used to train (1130) a cross-language LM+GSW attention embodiment using a Query Language Modeling (QLM) objective / task. In one or more embodiments, a loss function, such as cross-entropy loss, can be used during training.

[0127] In one or more embodiments, another batch may be obtained (1135), and the process returns to step 1115. Alternatively, if no additional batch is available for that round, a query (1140) is performed to determine whether to start another round. If another round begins, the process returns to step 1115; otherwise, a pre-trained cross-lingual LM+GSW attention embodiment is output (1145). The output pre-trained cross-lingual LM+GSW attention embodiment can then be fine-tuned for a specific task such as CLIR or CLQA.

[0128] E. Fine-tuning the CL application model using a pre-trained cross-lingual LM+GSW attention model example.

[0129] As described above, pre-trained cross-lingual language models with global sliding window attention (cross-lingual LM+GSW attention) can be used for various cross-lingual applications. As examples, this section presents embodiments of cross-lingual information retrieval and cross-lingual question answering models.

[0130] 1. CLIR Model Implementation Examples

[0131] Figure 12 A cross-language information retrieval (CLIR) system according to an embodiment of this disclosure is shown. For example... Figure 12 As shown, model embodiment 1202 includes a CLIR model head 1210 that receives input from the CLLM+GSW attention model embodiment 1205. The CLLM+GSW attention model embodiment 1205 receives query input 1215 and document 1220. The purpose of model 1202 is to identify relevant documents (if any) for the input query 1215. The CLIR task can be viewed as similar to a cross-language relevance ranking task. Therefore, those skilled in the art will recognize the above references... Figure 7 and Figure 8 The training aspects discussed are applicable here.

[0132] Figure 13A method for fine-tuning / training a cross-lingual information retrieval (CLIR) system according to an embodiment of this disclosure is shown, the system using a pre-trained cross-lingual language model (LM) with global + sliding window (GSW) attention (cross-lingual LM+GSW attention). For a CLIR model 1202 including a pre-trained cross-lingual LM+GSW attention model embodiment 1205 and a CLIR model head 1210, the parameters / weights of the CLIR model head 1210 can be initialized (1305) using random values, model weights from the RR model head used in pre-training (as a warm start), or a combination of both.

[0133] After initializing the model, the entire CLIR model can be trained (1310) using query-document pairs (e.g., positive and negative samples) to fine-tune the CL LM+GSW model 1205 and train the CLIR model head 1210. For example, for a cross-language query and document pair, the query-document sequence can be encoded using model 1205, and the model head 1210 multiplied by the output hidden vector of [cls] segmentation to produce a relevance score. Updates can be performed using cross-entropy loss, and training can continue until a stopping condition is met. Once the stopping condition is met, the trained CLIR model 1202 is output and can be used for the CLIR task. In one or more embodiments, the CLIR model can set a threshold for relevance and output top-ranked relevant documents, which can be listed in order of relevance score.

[0134] 2. CLQA Model Implementation Examples

[0135] Figure 14 A cross-language question-answering (CLQA) system according to an embodiment of this disclosure is illustrated. For example... Figure 14 As shown, model embodiment 1402 includes a CLQA model head 1410 that receives input from CL LM+GSW attention model embodiment 1405. CL LM+GSW attention model embodiment 1405 receives question input 1415 and document segmentation 1425 for each document from a document set 1420. The purpose of model 1402 is to identify relevant parts (start segmentation and end segmentation) of relevant documents (if any). In one or more embodiments, the CLQA model head may include two feedforward neural networks (each feedforward neural network may include one or more layers), one feedforward neural network taking the output of the cross-lingual language model and predicting the start segmentation in the document, and the other feedforward neural network taking the output of the cross-lingual language model and predicting the end segmentation in the document.

[0136] Figure 15A method of fine-tuning / training a Cross-Language Question Answering (CLQA) system using a pre-trained Cross-Language Language Model (LM) with Global + Sliding Window (GSW) attention (Cross-Language LM + GSW attention) model is shown in accordance with embodiments of the present disclosure. For a CLQA model 1402 including a pre-trained Cross-Language LM + GSW attention model embodiment 1405 and a CLQA model head 1410, the parameters / weights of the CLQA model head 1410 can be initialized (1505) with random values. After the model is initialized, the CLQA model can be trained (1510) using training data including input questions and respective start token-end token of documents that answer the input questions. In one or more embodiments, an error can be determined, where the error represents a sum of an error in identifying the start token plus an error in identifying the end token in the document. The update can be performed using cross-entropy loss, and the training can continue until a stopping condition is reached. Once the stopping condition is reached, the trained CLQA model 1402 will be output and available for QA tasks.

[0137] F. Experiments

[0138] It should be noted that these experiments and results are provided by way of illustration and are illustrative of one or more particular embodiments under certain conditions. Thus, neither these experiments nor their results should be taken as limiting the disclosed scope of the present patent application.

[0139] 1. Cross-Language Zero-Shot Retrieval

[0140] a) Evaluation Data and Metrics

[0141] The gold standard CLIR dataset CLEF was chosen to evaluate the effectiveness of cross-language zero-shot retrieval. The collection for zero-shot retrieval was used. In this dataset, 200 topics in different languages were treated as queries for retrieving news articles in different languages. Four languages were chosen from our pre-training data, resulting in 12 cross-lingual query-document evaluation pairs. Following standard practice, queries were created by concatenating the title and description of each topic. Queries that did not have relevant documents were removed. A first-stage retrieval model like BM25 was not used to obtain top candidates for re-ranking, as such an approach would require a query translation module and could introduce bias. Since the human evaluators see the top-ranked documents when creating relevance labels, we directly used all the labeled documents relevant to a query as candidate documents for re-ranking. Mean Average Precision (MAP) at query level was reported. Statistical significance differences in MAP were determined using a two-tailed paired t-test with p < 0.05. Table 3 shows the statistics of the CLEF dataset.

[0142] Table 3: Statistics of CLEF: number of queries (#queries), average number of documents per query related to it (#pos), average number of documents per query not related to it (#neg), and average number of document tokens after mBERT tokenization (doc.length) for each language.

[0143]

[0144] b) Competitive methods

[0145] The model embodiments were compared to a number of recent competitive CLIR methods.

[0146] (i) BWE-AGG: This is an unsupervised method that first constructs query and document embeddings by summing the CLEs of their constituent terms (see Robert Litschko, Goran Glavas, Simone Paolo Ponzetto, and Ivan Vulic. 2018. Unsupervised Cross-Lingual Information Retrieval Using Monolingual Data Only. In Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval (SIGIR). Ann Arbor, MI, 1253-1256) (hereinafter “Litschko et al. (2018)”). Candidate documents are ranked according to the cosine similarity of their embeddings to the query embedding. There are two variants for constructing document embeddings based on different summing weights: BWE-AGG-ADD uses uniform weights for all terms, and BWE-ADD-IDF weights document terms with IDF in the target language set. FastText embeddings were used.

[0147] (ii) TbT-QT-QL: This is an unsupervised query translation method based on CLEs (see Litschko et al. (2018)). Each source language query term is translated to the target language term that is closest to it in the CLE space. Thus, the CLIR task is reduced to a monolingual retrieval task, and the translated query is used with a query likelihood model. Galago (lemurproject.org / galago.php) was used to construct the inverted index and retrieve documents. FastText embeddings were used for query translation.

[0148] (iii) DRMM and K-NRM: These select two matching models from early neural CLIR research (see Puxuan Yu and James Allan. 2020. A Study of Neural Matching Models for Cross-lingual IR. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval (SIGIR). Virtual Event, China, 1637-1640 (hereinafter “Yu et al. (2020)”). They construct query-document interactions at the term level based on CLE, but output matching scores using different pooling methods. These two models are implemented on top of Matchzoo (see Jiafeng Guo, Yixing Fan, Xiang Ji, and Xueqi Cheng. 2019. MatchZoo: A Learning, Practicing, and Developing System for Neural Text Matching. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). Paris, France, 1297-1300 (hereinafter “Guo et al. (2019)”). For CLE, both fastText embeddings and smart-shuffing bilingual word embeddings are tested. The former is an example of post-projecting CLEs and aligns fastText embeddings trained on monolingual Wikipedia in 44 languages into a space using the relaxed CSLS method. Smart-shuffing is a pseudo-bilingual approach, but instead of randomly mixing words in parallel sentences, it leverages word-level parallel data (i.e., translation lexicons) to guide the process to bridge the “translation gap”. Smart-shuffing embeddings are used with a window size set to 10. Note that smart-shuffing embeddings are bilingual and only overlap with the evaluation languages on {En & Fr, En & De}. Therefore, we report its performance on four query-document language pairs.

[0149] (iv) mBERT: uses a public checkpoint of multilingual BERT (huggingface.co / bert-base-multilingual-uncased). It was initially pre-trained on the top 102 languages of Wikipedia dumps with a Masked Language Modeling (MLM) objective and a Next Sentence Prediction (NSP) objective.

[0150] (v) XLM-R: uses a public checkpoint (huggingface.co / xlm-roberta-base). It was initially pre-trained on a CommonCrawl corpus of 100 languages with a MLM objective.

[0151] c) Evaluation details

[0152] As mentioned before, all labeled query-document pairs were used as hard negatives on CLEF and the MAP of the reranking was reported. For unsupervised methods, all queries were tested. For methods that require training, five-fold cross-validation was employed to overcome the small number of queries per language pair. Evaluation was done separately per language pair. Specifically, each training (fine-tuning) round contained all positive query-document pairs. Each positive document was paired with a randomly sampled hard negative document and optimized using a pairwise cross-entropy loss. The maximum number of training (fine-tuning) rounds was set to 20 rounds. When the model produced the best MAP on the validation set, the MAP on the test set was recorded. For DRMM, the bin size was set to 30 and the histogram mode to “log count”. For KNRM, the number of Gaussian kernels was set to 20 (plus one for exact match) and sigma was set to 0.1. For fine-tuning of the Transformer-based models (mBERT, XLM, and the current patent document’s embodiments), only the last three encoder layers were fine-tuned to avoid overfitting. Furthermore, the parameters of the last three encoder layers were reinitialized (“reset”) before fine-tuning for better stability.

[0153] d) Results

[0154] Table 4 summarizes the overall results of all competing CLIR models on all evaluated language pairs, as shown in Figure 16 Detailed analysis is provided below.

[0155] Table 4 Figure 16 describes the performance of CLIR on CLEF. The numbers are MAP. The best performance per language pair is marked in bold. “*” indicates a statistically significant improvement compared to mBERT (paired t-test, p < 0.05).

[0156] Unsupervised methods: BWE-AGG and TbT-QT-QL are unsupervised CLIR methods based on fast-text CLE. In most cases, TbT-QT-QL outperforms BWE-AGG to a large extent, which is consistent with the results reported in previous studies. However, both studies only experiment with English as the query language, while the experiments in this paper are more comprehensive. It was found that in some cases ({De-Es, De-Fr, Es-De, Fr-De}), the performance of BWE-AGG was close to or slightly better than that of TbT-QT-QL. TbT-QT-QL relies heavily on the quality of top-term translation. It is suspected that the German embedding is inconsistent with the Spanish / French embedding and cannot provide top-quality nearest neighbor term translation.

[0157] Neural matching: DRMM and KNRM represent the category of neural matching. The number reported by Yu et al. (2020) dropped significantly. One major difference between their evaluation and the evaluation in this paper is that they truncated the documents to the first 500 terms, while the tests in this paper retained all content. If the same truncation strategy is adopted, similar performance is also observed in their report: the performance of neural matching is significantly better than that of unsupervised methods, but still lower than the mBERT baseline in this patent document. This indicates that KNRM and DRMM cannot handle long documents well. This is the first time that an empirical comparison between smart-mixed embeddings and fast-text embeddings has been made. It was observed that the lexical coverage of smart-mixed in the CLEF collection was small, which may be an important factor in its poor performance.

[0158] General-purpose language models: This category includes the XLM-R, mBERT, and mBERT-GSW embodiments. The poor performance of XLM-R is consistent with other people’s findings. The results here report for the first time the good performance of mBERT on CLIR. The performance of XLM-R is much worse than mBERT, which seems counterintuitive: their model structures and pre-training objectives (MLM) are similar, but XLM-R has much more training data, and is reported to outperform mBERT on a variety of cross-lingual tasks. Control experiments were performed to rule out the part-of-speech tagger and text case as factors. It is suspected that the failure of XLM-R for CLIR is due to the way the pre-training data is input to the model: unlike BERT, XLM-R receives a tokenized stream, so sequences in a mini-batch can contain more than two consecutive sentences. This can be fine for word-level tasks like extractive QA (as shown in §F.2), but can cause confusion for tasks like CLIR that require alignment of representations of long texts. Comparing mBERT and the mBERT-GSW embodiment, in most cases, it is observed that mBERT-GSW is statistically significantly better than mBERT. This suggests that the benefits of accepting longer input sequences are not limited to the pre-training phase, but also include the fine-tuning and inference phases. The less information is lost during the fine-tuning process, the greater the difference at inference time.

[0159] Retrieval-oriented language models: This category describes cross-lingual LMs that were pre-trained with QLM objectives and RR objectives. By comparing (mBERT, QLM-RR) and vanilla mBERT, it can be seen that there is a significant improvement in re-ranking effectiveness for all language pairs. This demonstrates (i) that the pre-training objective embodiments are effective for downstream CLIR tasks; (ii) that the quality of the pre-training weakly-supervised data constructed using fragment alignment from the multilingual Wikipedia is high, and the learned knowledge generalizes to non-Wikipedia sets. By comparing retrieval-oriented LMs that utilize GSW attention (mBERT-GSW embodiment, QLM-RR) and those that utilize self-attention (mBERT, QLM-RR), it is observed that there is an additional statistically significant improvement. This suggests that locally expanding the input length of the Transformer to encode more context can provide additional benefits for ad-hoc retrieval, which is an additional benefit of retrieval-oriented pre-training. The entire model embodiments (mBERT-GSW, QLM-RR, 20) provide up to 29.7% MAP improvement over vanilla mBERT re-ranking.

[0160] 2. Cross-lingual question answering

[0161] Cross-lingual extractive question answering is a word-level retrieval task that does not have an explicit connection to any of our pre-training objectives. Therefore, it can better demonstrate the generalization ability of the language model pre-training strategies we propose.

[0162] a) Evaluation datasets

[0163] The Multilingual Question Answering (MLQA) dataset was chosen for testing. There is no specialized training data in the dataset, and following standard practice, a second MLQA training dataset was leveraged for fine-tuning, and the development and test sets from the first MLQA dataset were used for evaluation in two settings. (i) zero-shot XLT: the development / test set is monolingual QA in a language different from the fine-tuning language; (ii) G-XLT: the language of the questions and context / answers in the development / test set differ. Note that G-XLT can not be zero-shot learning in terms of language, as the language of the query or context can be English. Since there is no French data in the MLQA dataset, the test results are for G-XLT with six language pairs, and for XLT with two languages (Spanish and German). The F1 score and exact match score are the metrics, and are standard metrics for extractive QA. The MLQA dataset is much larger than CLEF, so all encoder layers of the language model can be fine-tuned. Results on the test set are reported when the model obtains the best F1 score on the development set. Statistical significance testing was done using a two-tailed paired t-test with p < 0.05.

[0164] b) Results

[0165] Table 5( Figure 17 ) summarizes the overall results for cross-lingual question answering using the MLQA dataset from the previous section. XLM-R performs slightly better than mBERT in the XLT setting, but much worse in the G-XLT setting, compared to the general-purpose language model. The performance of the tested mBERT-GSW implementation is slightly better than mBERT.

[0166] The tested language model implementations that were pre-trained with QLM and RR obtain statistically significant improvements over mBERT when fully trained, with the improvements being more significant in G-XLT than in XLT. This can be because these models are strictly pre-trained with bilingual query-document pairs. In other words, XLT is a zero-shot learning task, and more difficult to improve upon, in terms of our pre-training data.

[0167] A cross-task comparison was also done (Table 4( Figure 16 ) and Table 5( Figure 17), and it was observed that the mBERT-GSW embodiment consistently outperformed mBERT on CLIR when pre-training the same number of rounds with the same targets, but the two models performed similarly on QA. The documents (“context”) in the MLQA dataset are passages, which are generally shorter than the news articles in CLEF. Therefore, pre-training on longer text does not seem to provide an additional gain for QA, as the sequence length of the fine-tuning and test data is small.

[0168] 3. Ablation studies and parameter analysis

[0169] a) Utility of pre-training tasks

[0170] Experiments were conducted to study the impact of each pre-training target on downstream tasks. To improve efficiency, language models that limit the input sequence to the first 512 tokens (with self-attention) were used instead of language models that limit the input to 1024 tokens (with GSW attention). More than two model embodiments were pre-trained, one with only the QLM target and another with only the RR target. In the CLIR and CLQA tasks, the performance of all compared models was recorded in each available language pair. There were no significant differences between language pairs, so two language pairs were chosen for each task to demonstrate. The results are shown in Figure 18

[0171] Figure 18 CLIR and CLQA performance of models pre-trained with different targets (QLM, RR, QLM+RR) according to embodiments of the present disclosure are shown. The same evaluation strategy was adopted and the performance under different pre-training rounds is reported. Note that the starting point (pre-training round = 0) in all subplots refers to the base mBERT model that was not pre-trained with RR or QLM.

[0172] In the case of CLIR, both RR and QLM provide positive gains for retrieval, with the former being more effective than the latter. This is not surprising given that RR has the same form as on-the-fly retrieval using a language model. It was observed that there is complementarity between the two targets, as QLM+RR outperforms both targets individually. For CLQA, RR and QLM also provide benefits to the downstream task. However, in this case, QLM is more effective than RR. The two pre-training targets are also mutually beneficial, as the LM pre-trained with both targets performs best in terms of F1.

[0173] Therefore, it can be seen that QLM and RR each have a positive impact on the downstream cross-lingual retrieval task, and that these positive impacts are additive, so the performance of language models pre-trained with both tasks is best.

[0174] ​b) QLM mask

[0175] There are two options to distinguish QLM from other pre-training tasks: (i) mask only query tokenization; (ii) increase the probability of masking from 15% to 30%. Control experiments are conducted here to justify these options. Five model instances are pre-trained with only the QLM objective under different settings. Masking queries with a probability of 30% as the baseline. Two model instances are trained with different query masking probabilities (15% and 45%). The other two model instances are trained with a masking probability of 30%, but different regions (document vs. query plus document) are masked. Their performances on CLEF are shown as Figure 19

[0176] Masking probability: Comparing the circular plot 1905, the linear plot 1910, and the diamond plot 1915, it can be observed that p = 0.15 is always outperformed by higher query masking probabilities. This indicates that p = 0.15 makes the QLM task too easy for the language model to learn cross-lingual query-document interaction. When the model is well trained, query masking probability p = 0.45 performs slightly better than p = 0.30, which indicates that further increasing the query masking probability can provide additional gains.

[0177] Masking region: Comparing the linear plot 1910, the square plot 1920, and the star plot 1925, it seems that masking only document tokenization greatly harms the performance of CLIR when starting pre-training, and cannot recover to the level of vanilla mBERT reranking after 20 pre-training iterations. Masking only queries and masking both queries and documents can bring improvements, but the former significantly outperforms if the model is well trained.

[0178] Therefore, the results show that masking only query tokenization with a higher probability (> 0.3) is a better setting for the QLM task in one or more embodiments.

[0179] c) Sentence and document-level semantic alignment

[0180] To investigate whether the proposed retrieval-oriented cross-lingual modeling task embodiments can facilitate cross-lingual alignment at a coarser-grained semantic level, the pre-trained LMs are evaluated on two cross-lingual alignment tasks, namely cross-lingual sentence alignment (XSA) and cross-lingual document alignment (XDA). XSA / XDA require that parallel sentences / documents should have embeddings that are close in the representation space. The LM is utilized to encode sentences (documents) as “[cls] content [sep]”, and the last hidden state of the [cls] tokenization is taken as the sentence (document) embedding.

[0181] Metrics: For sentence sets ​and the corresponding set of parallel (translated) sentences The top-level precision (P@1) is measured, which is defined as:

[0182]

[0183] wherein, is an indicator function, and Sim is a function that measures the similarity of two cross-lingual sentences. The CSLS metric with a neighborhood size of 10 is used as the Sim function, which is an improved version of the cosine similarity metric. Compared to cosine similarity, CSLS has two advantages for evaluating cross-lingual word alignment: (i) CSLS is a symmetric measure, which means that switching and does not affect the degree of alignment; and (ii) CSLS can alleviate the hubness problem.

[0184] Data: The XSA and XDA evaluations use parallel sentences and documents. For XSA, the cross-lingual natural language inference (XNLI) dataset is used. It contains 10K sentences that are artificially translated into 15 languages. Four languages that overlap with the pretraining languages in this paper are selected and form six evaluation pairs. For XDA, another dataset is used, which contains approximately 86,000 documents in six languages. Among the four languages in the pretraining languages used in this paper, German is not in the corpus. Therefore, English, French, and Spanish data are used and form three language pairs. As in Sections F.3.a and F.3.b, to improve efficiency, the LM with an input limit of 512 tokens is used.

[0185] Results: The XSA and XDA performance of the base mBERT and the XSA and XDA performance of the cross-lingual LM embodiments, in which the cross-lingual LM embodiments are additionally pre-trained for 20 rounds of our retrieval-oriented modeling tasks, are reported in Tables 6 and 7. It can be observed that the cross-lingual sentence and document alignments generated by the base mBERT are poor except between the Spanish and French documents. For XSA, QLM and RR improve over the base mBERT. The improvement is more significant for QLM, and this effect is additive for XSA. For XDA, QLM and RR work similarly, and combining the two modeling tasks together can further improve the cross-lingual document alignment. The difference between XSA and XDA is understandable considering that QLM focuses more on sentence-level semantics (sentence completion for foreign documents), while RR focuses more on document-level semantics (ranking of documents according to foreign sentences). In all cases, pre-training the model with RR or QLM on the weakly supervised data from Wikipedia can significantly improve cross-lingual coarse-grained semantic alignment. In addition to cross-lingual retrieval, this can also be used to improve more applications.

[0186] Table 6: Cross-lingual sentence alignment (XSA) results on the datasets. The numbers are P@1 in percentage (%).

[0187]

[0188] Table 7: Cross-lingual document alignment (XDA) results on the datasets. The numbers are P@1 in percentage (%).

[0189]

[0190] d) Impact of window size in GSW attention

[0191] One of the assumed key parameters in global + sliding window (GSW) attention is the window size w. In the context of cross-lingual retrieval, it represents the number of adjacent tokenizations that the document tokenization can "attend" to in a single Transformer layer. While theoretically, GSW is more efficient (linear vs. quadratic) than full self-attention, in practice, a too large window size can render this effort ineffective. Therefore, it is important to evaluate how the setting of the window size impacts the performance of the desired task. To this end, four different window sizes w = {32, 64, 128, 256} are chosen for experimentation. For all four model instantiations, the input sequence limit is set to 1024 tokenizations, which are pre-trained on the English-French portion of the weakly supervised data (described in Section D.1) using QLM plus RR pre-training modeling task and evaluated on the CLEF dataset (described in Section F.1) for CLIR in both directions (En→Fr and Fr→En). The results are seen in Table 8. It is observed that there is no statistically significant difference in the downstream CLIR performance caused by different window sizes w in GSW attention. Given the randomness in model training, it can be concluded that the window size w in GSW attention has no apparent impact on the CLIR task. In practice, when faced with trade-offs, a longer sequence length can be prioritized over a larger window size.

[0192] Table 8: Impact of window size w in GSW attention on CLIR performance. The numbers are MAP.

[0193]

[0194] G. Some observations

[0195] This paper shows that the lack of cross-lingual passage-level correlation data and the lack of proper query-document style pretraining are some key reasons for poor performance when employing multilingual language models for CLIR. To overcome these difficulties, two new pretraining objective instantiations are developed to improve Transformer-based cross-lingual language models for the retrieval task. In addition, this paper presents instantiations of a method for creating fine-grained cross-lingual query-document style weakly supervised data and uses this method to newly create a dataset from multilingual text to support large-scale pretraining. The instantiations employ global + sliding window attention to allow the language model to effectively encode longer documents in all three stages (pretraining, fine-tuning, and inference). Extensive experiments demonstrate the effectiveness of the instantiations' contributions to cross-lingual ad hoc retrieval and cross-lingual extractive question answering. Detailed ablation studies prove the reasonableness of model and parameter choices. It is also found that the model instantiations can significantly improve cross-lingual coarse-grained semantic alignment, which can lead to wider applications beyond retrieval.

[0196] H. Computing system embodiments

[0197] In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solutions comply with relevant laws and regulations and do not violate public order and good customs.

[0198] According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0199] In one or more embodiments, various aspects of this patent document may be addressed to, may include, or may be implemented on one or more information processing systems (or computing systems). Information processing systems / computing systems may include any tool or set of tools that can be used to calculate, account for, determine, classify, process, send, receive, retrieve, initiate, route, switch, store, display, communicate, present, detect, record, reproduce, process, or utilize information, intelligence, or data of any form. For example, a computing system may be or may include a personal computer (e.g., a laptop computer), a tablet computer, a mobile device (e.g., a personal digital assistant (PDA), a smartphone, a phablet, a tablet computer, etc.), a smartwatch, a server (e.g., a blade server or a rack server), a network storage device, a camera, or any other suitable device, and may vary in size, shape, performance, functionality, and price. A server may be a cloud server, a server for a distributed system, or a server incorporating blockchain technology. A computing system may include random access memory (RAM), one or more processing resources (such as a central processing unit (CPU) or hardware or software control logic unit), read-only memory (ROM), and / or other types of memory. Additional components of a computing system may include one or more drives (e.g., hard disk drives, solid-state drives, or both), one or more network ports for communicating with external devices, and various input and output (I / O) devices (such as keyboards, mice, styluses, touchscreens, and / or video displays). The computing system may also include one or more buses for transmitting communication between the various hardware components.

[0200] Figure 20 A simplified block diagram of an information processing system (or computing system) according to an embodiment of this disclosure is shown. It should be understood that the functions shown for system 2000 can be used to support various embodiments of the computing system; however, it should be understood that the computing system can be configured differently and include different components, including, for example... Figure 20 The diagram shows a number of fewer or more components.

[0201] like Figure 20 As shown, the computing system 2000 includes one or more central processing units (CPUs) 2001 that provide computing resources and control the computer. The CPU 2001 may be implemented using a microprocessor or the like, and may also include one or more graphics processing units (GPUs) 2002 and / or floating-point coprocessors for mathematical calculations. In one or more embodiments, one or more GPUs 2002 may be incorporated into a display controller 2009, such as part of one or more graphics cards. The system 2000 may also include system memory 2019, which may include RAM, ROM, or both.

[0202] likeFigure 20 As shown, many controllers and peripherals can also be provided. Input controller 2003 represents an interface to various input devices 2004, such as a keyboard, mouse, touchscreen, and / or stylus. The computing system 2000 can also include a storage controller 2007 for interfacing with one or more storage devices 2008 that include storage media such as magnetic tapes or disks, or optical media that can be used to record the instruction programs of operating systems, utilities, and applications, which can include embodiments of programs implementing aspects of the present disclosure. According to the present disclosure, the storage devices 2008 can also be used to store processed data or data to be processed. The system 2000 can also include a display controller 2009 for providing an interface to a display device 2011, which can be a cathode ray tube (CRT), a thin film transistor (TFT), an organic light emitting diode, an electroluminescent panel, a plasma panel, or any other type of display. The computing system 2000 can also include one or more peripheral controllers or interfaces 2005 for one or more peripheral devices 2006. Examples of peripheral devices can include one or more printers, scanners, input devices, output devices, sensors, etc. A communications controller 2014 can interface with one or more communication devices 2015, which enables the system 2000 to connect to remote devices through any of a variety of networks, including the Internet, cloud resources (e.g., Ethernet Cloud, Fibre Channel over Ethernet (FCoE) / Data Center Bridging (DCB) Cloud, etc.), local area networks (LANs), wide area networks (WANs), storage area networks (SANs), or through any suitable electromagnetic carrier signals, including infrared signals. As described in embodiments, the computing system 2000 includes one or more fans or fan trays 2018 and one or more cooling subsystem controllers 2017 that monitor the heat tolerance temperature of the system 2000 (or components thereof) and operate the fans / fan trays 2018 to help regulate the temperature.

[0203] In the illustrated system, all major system components can be connected to bus 2016, which can represent more than one physical bus. However, various system components can or can not be in physical proximity to one another. For example, input data and / or output data can be remotely transmitted from one physical location to another. Additionally, programs implementing various aspects of the present disclosure can be accessed from a remote location (e.g., a server) over a network. Such data and / or programs can be transmitted by any of a variety of machine-readable media, including, for example: magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as Compact Discs (CDs) and holographic devices; magneto-optical media; and hardware devices specially configured to store or store and execute program code, such as Application Specific Integrated Circuits (ASICs), Programmable Logic Devices (PLDs), flash devices, other non-volatile memory (NVM) devices (such as devices based upon 3D magnetic memory), and ROM and RAM devices.

[0204] Various aspects of the present disclosure can be encoded upon one or more non-transitory computer-readable media for one or more processors or processing units to execute so that the steps are performed. It should be noted that the one or more non-transitory computer-readable media should include volatile and / or non-volatile memory storage. It should be noted that alternative implementations are possible, including hardware implementations or software / hardware implementations. Hardware implemented functions can be implemented using ASICs, programmable arrays, digital signal processing circuits, etc. Thus, the "means" term in any claim is intended to encompass software and hardware implementations. Similarly, the term "computer-readable medium" as used herein includes software and / or hardware having instructions programmed thereon or a combination thereof. In view of these alternatives for implementing the functionality, it should be understood that the drawings and appended description provide functional information for programmers skilled in the art to write program code (i.e., software) and / or to fabricate circuits (i.e., hardware) to perform the processing required.

[0205] It should be noted that embodiments of the present disclosure can also relate to a computer program product, which has a non-transitory tangible computer-readable medium having instructions for performing various computer-implemented operations. The medium and the computer code can be specially designed and constructed for the purposes of the present disclosure, or they can be of the kind known or used by those having ordinary skill in the relevant arts, given the task. Examples of tangible computer-readable media include, but are not limited to: magnetic media, such as hard disks, floppy disks, and magnetic tape; optical media such as Compact Disc (CD) and holographic devices; magneto-optical media; and hardware devices that are specially configured to store and perform program code, such as Application Specific Integrated Circuits (ASICs), Programmable Logic Devices (PLDs), flash memory devices, other nonvolatile memory (NVM) devices, such as 3D magnetic memory-based devices, and ROM and RAM devices. Examples of computer code include machine code, such as produced by a compiler, and files containing higher-level code, such as an interpreted or interpreted programming language. Embodiments of the present disclosure can be implemented in whole or in part within a program module that is executed on a processing device. Examples of program modules include libraries, programs, routines, objects, components, and data structures. In a distributed computing environment, program modules can be physically located in local or remote settings or both.

[0206] Those skilled in the art will recognize that the practice of the present disclosure is not limited to a particular computing system or programming language. Those skilled in the art will also recognize that the above-described elements can be physically and / or functionally separated into modules and / or sub-modules, or combined together.

[0207] Those skilled in the art will appreciate that the above examples and embodiments are exemplary, and not limiting to the scope of the present disclosure. All alternatives, enhancements, equivalents, combinations, and improvements thereto, which are apparent to those skilled in the art upon reading the description and studying the drawings, are intended to be encompassed within the true spirit and scope of the present disclosure. It should also be noted that the elements of any claims can be arranged in different permutations and combinations, including having multiple dependencies, configurations, and combinations thereof.

Claims

1. A computer-implemented method of pre-training a cross-lingual language model, the method comprising: in response to a stopping condition not being met, performing the following steps: for a relevance ranking batch of cross-lingual query and document pairs, using a relevance ranking model head, predicting relevance rankings using the cross-lingual language model, wherein the relevance ranking batch includes at least one positive sample cross-lingual query and document pair in which the query and document are in different languages and the document is relevant to the query and at least one negative sample cross-lingual query and document pair in which the query and document are in different languages and the document is not relevant to the query, the relevance rankings are rankings of relevance between documents and respective queries, the cross-lingual language model includes global plus sliding window attention, a global window for attention is used for query tokenization, and a sliding window for attention is used for at least some document tokenization; updating the cross-lingual language model using known relevance rankings and predicted relevance rankings; for a query language modeling batch of cross-lingual query and document pairs, inputting a query and a respective document into the cross-lingual language model to predict a masked token in the query, the query and document are in different languages and the document is relevant to the query in the query language modeling batch, the query has multiple tokens and some of the multiple tokens are masked tokens; and updating the cross-lingual language model based on a comparison of the predicted result of the cross-lingual language model and the actual masked token; and in response to the stopping condition being met, outputting the pre-trained cross-lingual language model. initially obtaining the cross-lingual language model by performing the following steps:

2. The computer-implemented method of claim 1, wherein, obtaining a pre-trained multilingual language representation model, the pre-trained multilingual language representation model includes an attention module, and the pre-trained multilingual language representation model is trained using a masked language modeling (MLM) task objective; replacing the attention module in the pre-trained multilingual language representation model with a global plus sliding window attention to obtain a modified language model; updating the modified language model using the MLM task objective and a dataset to improve performance of the modified language model relative to the pre-trained multilingual language representation model; and outputting the updated modified language model as the cross-lingual language model for pre-training. obtaining at least one of the relevance ranking batch and the query language modeling batch by performing the following steps:

3. The computer-implemented method of claim 1, wherein, sampling from a plurality of different languages to train the pre-trained cross-lingual language model to accept each of the plurality of different languages as input and output each of the plurality of different languages as output. at least one sampling step includes:

4. The computer-implemented method of claim 1, wherein, matching a plurality of snippets of one or more multilingual resources that are related to a same topic; and extracting a sentence or a portion of a sentence from a matching snippet in a first language as a query and using a matching snippet or a portion of the matching snippet in a language different from the first language as a cross-lingual document. ​ 5. The computer-implemented method of claim 1, further comprising: for a cross-language information retrieval (CLIR) model, the CLIR model comprising a pre-trained cross-lingual language model and a CLIR model head: initializing parameters or weights of the CLIR model head using a random value, a model weight, or a combination of the random value and the model weight, wherein the model weight is from a relevance ranking model head used in pre-training; training using positive query-document pairs and negative query-document pairs to update parameters or weights of the pre-trained cross-lingual language model, parameters or weights of the CLIR model head, or parameters or weights of both the cross-lingual language model and the CLIR model head; and outputting the trained CLIR model.

6. The computer-implemented method of claim 1, further comprising: for a cross-lingual question answering (CLQA) model, the CLQA model comprising a pre-trained cross-lingual language model and a CLQA model head: initializing parameters or weights of the CLQA model head; training using training samples to update parameters or weights of the pre-trained cross-lingual language model, parameters or weights of the CLQA model head, or parameters or weights of both the pre-trained cross-lingual language model and the CLQA model head; and outputting the trained CLQA model.

7. A computer-implemented method of pre-training a cross-lingual language model, the method comprising: initializing a cross-lingual language model using global plus sliding window attention using a pre-trained multilingual language representation model, wherein a global window for attention is used for query tokenization and a sliding window for attention is used for at least some document tokenization; in response to a stop condition not being reached, performing the following steps: for a cross-lingual query and document pair, training the cross-lingual language model using a query language model objective, wherein at least some query tokenization of a query input into the cross-lingual language model in the cross-lingual query and document pair is masked query tokenization, the query language model objective involving predicting the masked query tokenization for a query given a respective relevant cross-lingual document, and for a cross-lingual query and document pair, training the cross-lingual language model using a relevance ranking model head to predict a relevance ranking, the relevance ranking being a ranking for relevance between a document and a respective query; and in response to a stop condition being reached, outputting the pre-trained cross-lingual language model.

8. The computer-implemented method of claim 7, wherein, initializing a cross-lingual language model using global plus sliding window attention using a pre-trained multilingual language representation model comprises: obtaining a pre-trained multilingual language representation model, the pre-trained multilingual language representation model comprising an attention module, and the pre-trained multilingual language representation model being trained using a masked language modeling (MLM) task objective; replacing the attention module in the pre-trained multilingual language representation model with global plus sliding window attention to obtain a modified language model; updating the modified language model using the MLM task objective and the dataset to improve performance of the modified language model relative to the pre-trained multilingual language representation model; and outputting the updated modified language model as a pre-trained cross-lingual language model.

9. The computer-implemented method of claim 7, further comprising: obtaining cross-lingual query and document pairs for training a cross-lingual language model with the relevance ranking model head by performing the following steps: sampling a set of cross-lingual query and document pairs, wherein the set of cross-lingual query and document pairs includes: at least one positive sample cross-lingual query and document pair in which the query and document are in different languages and the document is relevant to the query, and at least one negative sample cross-lingual query and document pair in which the query and document are in different languages and the document is not relevant to the query; and obtaining cross-lingual query and document pairs for training a cross-lingual language model with the query language model objective by performing the following steps: sampling a set of cross-lingual query and document pairs in which the query and document are in different languages and the document is relevant to the query.

10. The computer-implemented method of claim 7, wherein, At least one sampling step includes: sampling from a plurality of different languages to train the pre-trained cross-lingual language model to accept as input and output as output each of the plurality of different languages.

11. The computer-implemented method of claim 7, further comprising: for a cross-lingual information retrieval (CLIR) model, the CLIR model including a pre-trained cross-lingual language model and a CLIR model head: initializing parameters or weights of the CLIR model head using a random value, a model weight, or a combination of the random value and the model weight, wherein the model weight is from the relevance ranking model head used in pre-training; training using positive query-document pairs and negative query-document pairs to update parameters or weights of the pre-trained cross-lingual language model, parameters or weights of the CLIR model head, or parameters or weights of both the cross-lingual language model and the CLIR model head; and outputting the trained CLIR model.

12. The computer-implemented method of claim 7, further comprising: for a cross-lingual question answering (CLQA) model, the CLQA model including a pre-trained cross-lingual language model and a CLQA model head: initializing parameters or weights of the CLQA model head; training using training samples to update parameters or weights of the pre-trained cross-lingual language model, parameters or weights of the CLQA model head, or parameters or weights of both the pre-trained cross-lingual language model and the CLQA model head; and outputting the trained CLQA model.

13. An electronic device, comprising: at least one processor; and memory that is communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-12.

14. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are for causing the computer to perform the method of any one of claims 1-12.

15. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-12.

Images