Information processing device, pre-training data creation method, and program

Abstract

This information processing device comprises a pre-training data creation unit that creates pre-training data for a word alignment model by collecting, from a set of multilingual text, text pairs that refer to the same object or text pairs that are semantically similar, and assigning a word alignment to the text pairs.

Description

Information processing device, pre-training data creation method, and program

[0001] The present invention relates to the technical field of word correspondence.

[0002] Identifying words or sets of words that are translations of each other in two sentences that are translations of each other is called word alignment.

[0003] For example, Non-Patent Document 1 discloses a technique for fine-tuning a trained multilingual model with bilingual data using an objective function that improves the accuracy of word alignment. Also, Non-Patent Document 2 discloses a technique for determining word alignment using cross-language span prediction based on a trained multilingual model.

[0004] Zi-Yi Dou and Graham Neubig. Word alignment by fine-tuning embeddings on parallel corpora. In Proceedings of EACL-2021, pp. 2112-2128, 2021. Masaaki Nagata, Katsuki Chousa, and Masaaki Nishino. A supervised word alignment method based on cross-language span prediction using multilingual bert. In Proceedings of EMNLP-2020, pp. 555-565, 2020. Masoud Jalili Sabet, Philipp Dufter, Franc, ois Yvon, and Hinrich Schutze. Simalign: High quality word alignments without parallel training data using static and contextualized embeddings. In Findings of EMNLP2020, pp. 1627-1642, 2020.

[0005] Conventional word alignment methods require bilingual data with manually assigned word alignments or large amounts of bilingual data as training data for a word alignment model. In particular, the supervised word alignment method based on cross-lingual span prediction disclosed in Non-Patent Document 2 is highly accurate, but requires manually created correct answer data for word alignments to train the word alignment model. Generally, correct answer data for word alignments is a scarce language resource, and creating it requires a certain amount of time, cost, and technical skill.

[0006] The present invention has been made in view of the above points, and has an object to provide a technique for automatically creating pre-training data for a word alignment model.

[0007] According to the disclosed technology, there is provided an information processing device including a pre-training data creation unit that creates pre-training data for a word alignment model by collecting text pairs that refer to the same subject or semantically similar text pairs from a collection of multilingual texts and assigning word alignments to the text pairs.

[0008] According to the disclosed technology, it is possible to automatically create pre-training data for a word alignment model.

[0009] FIG. 1 is a diagram illustrating an example of the configuration of an information processing device 100. FIG. 2 is a diagram illustrating a processing flow during training. FIG. 3 is a diagram illustrating a processing flow during inference. FIG. 4 is a diagram illustrating an example of the configuration of an information processing device 200. FIG. 5 is a diagram illustrating an example of the configuration of an information processing device 300. FIG. 6 is a diagram illustrating an overall flow of pre-training of a word alignment model. FIG. 7 is a diagram illustrating an overall flow of pre-training of a word alignment model. FIG. 8 is a diagram illustrating an example of the configuration of an information processing device 400 that performs supervised word alignment using language span prediction. FIG. 9 is a diagram illustrating an example of the hardware configuration of an information processing device. FIG. 10 is a diagram illustrating experimental results. FIG. 11 is a diagram illustrating experimental results.

[0010] Hereinafter, an embodiment of the present invention (the present embodiment) will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment. In the following description, references are indicated by numbers such as [1], and the names of the documents corresponding to the numbers are listed at the end of the specification.

[0011] In the description of the specific technology according to the present embodiment, text pairs that mention the same entity are collected to learn the correspondence between Wikipedia entities, but the word correspondence data obtained from unsupervised word correspondence does not necessarily have to be text pairs that mention the same entity. Note that the term "entity" in the specification may also be referred to as "target."

[0012] (Outline of the embodiment) As described above, conventional word alignment methods require bilingual data to which word alignments have been manually assigned, or a large amount of bilingual data, as training data.

[0013] In contrast to this, in the technology according to the present embodiment, the information processing device 100 described below relaxes the conditions required for training data of a word alignment model, namely, "word alignments must be correct, all words in a bilingual text pair must be fully aligned, and the training data must be parallel in the first place," thereby creating a large amount of training data that is "noisy, has partial word alignments, and is not necessarily bilingual (non-parallel)," and pre-trains the word alignment model.

[0014] Specifically, the information processing device 100 uses interlingual links in Wikidata and Wikipedia to collect pairs of texts that mention the same entity (such as a person, place name, or event) for pre-training word correspondences between different words (or word sets) that mention the same subject. The collected text pairs are not limited to text pairs that mention the same entity, but may also be semantically similar text pairs.

[0015] Furthermore, the information processing device 100 automatically assigns word correspondence to text pairs that refer to the same entity or that are semantically similar using an unsupervised word correspondence technique, and uses the word correspondence obtained by the automatic word correspondence as pre-training data for the word correspondence model only for word pairs that are expected to have relatively high accuracy based on frequency of occurrence or parts of speech.

[0016] The pre-trained word alignment model created in this way can achieve word alignment with higher accuracy than conventional unsupervised word alignment even in its original state (zero-shot).Furthermore, by fine-tuning this pre-trained word alignment model using manually generated word alignment data, it can achieve the world's highest accuracy in word alignment.

[0017] Generally, a machine learning method that learns a model from a large amount of automatically generated data that is not necessarily correct (including errors) is called weakly supervised learning. In this embodiment, the information processing device 100 executes the weakly supervised learning method for the word correspondence model.

[0018] In order to facilitate understanding of the technology according to the present embodiment, the conventional technology and its problems will first be described in detail below, and then the technology according to the present embodiment will be described in detail.

[0019] (Regarding the prior art) <Unsupervised word alignment, semi-supervised word alignment> Around 2020, word alignment methods using pre-trained multilingual models such as multilingual BERT[2] and XML-R[1] began to be proposed. Pre-trained multilingual models are trained using monolingual data in each language through masked language modeling tasks, and do not use bilingual data for training. Multilingual BERT and XML-R support around 100 languages.

[0020] SimAlign [8] uses a pre-trained multilingual model to find contextualized word embeddings for each word in an input sentence pair, and then finds word alignments based on the similarity of the word pairs. Since it does not use manually created ground truth data or bilingual data, it is an unsupervised word alignment method and can find word alignments for any language pair supported by the pre-trained multilingual model.

[0021] The accuracy of word alignment by SimAlign is higher than that of GIZA++[6] and FastAlign[4], which use a large number of parallel sentences based on statistical translation models to find word alignments.

[0022] AWESOME [3] is a method for fine-tuning multilingual models trained on bilingual data using objective functions that improve word alignment accuracy. Objective functions include a masked language model, a translation language model, and a self-training objective function. The self-training objective function encourages changes in the embeddings of word pairs that are considered to correspond in this way so that they are more similar to each other.

[0023] Generally, machine learning methods that use unlabeled data to improve labeling accuracy are called semi-supervised learning. AWESOME uses bilingual data, but does not use manually created correct word alignment data, making it a semi-supervised word alignment method. In general, AWESOME has higher word alignment accuracy for language pairs of fine-tuned bilingual data than SimAlign.

[0024] <Supervised word alignment based on cross-lingual span prediction> SpanAlign[5] is a method for finding word alignments using cross-lingual span prediction based on a trained multilingual model. Similar to SQuAD (The Stanford Question Answering Dataset) style question answering, an output layer that predicts the start and end points of spans is added on top of the trained multilingual model, and the system predicts the word sequence (span) in a sentence in one language that corresponds to the words in the sentence in the other language enclosed by delimiters (e.g., '¶').

[0025] The model for cross-language span prediction is created by fine-tuning a word alignment model that adds an output layer for span prediction, which predicts the start and end points of a span, on top of a trained multilingual model, using span prediction ground truth data created from word alignment ground truth data.Since span prediction changes depending on the direction of the source and destination languages, the two-way span prediction is symmetrical to reduce word alignment errors.

[0026] SpanAlign is a supervised word alignment method that uses manually created correct answer data for word alignments and does not use bilingual data. With correct answer data for word alignments of approximately 300 sentence pairs, it significantly surpasses conventional accuracy and achieves the world's highest accuracy.

[0027] Supervised word alignment methods based on cross-lingual span prediction are highly accurate, but require manually generated ground truth data to train the word alignment model. Given ground truth data, supervised word alignment (SpanAlign) can achieve higher word alignment accuracy than semi-supervised word alignment (AWESOME). However, ground truth data is generally a scarce linguistic resource, and creating it requires a certain amount of time, money, and technical expertise.

[0028] Therefore, compared to unsupervised word alignment (SimAlign), which does not require correct answer data or bilingual data for word alignment, and semi-supervised word alignment (AWESOME), which only requires bilingual data, supervised word alignment (SpanAlign) has the problem that it can be applied to fewer language pairs.

[0029] (Summary of the Problem and Solution) The above-mentioned conventional "supervised word alignment method based on cross-language span prediction" can achieve the world's highest accuracy for language pairs for which manually created correct answer data for word alignment exists, but there is a problem in that this method cannot be applied to language pairs for which manually created correct answer data for word alignment does not exist. The problem to be solved in this embodiment is to solve this problem.

[0030] In this embodiment, to solve the above problems, the information processing device 100 automatically generates a large amount of word alignment data containing errors, and pre-trains a word alignment model using the word alignment data to improve the robustness and generality of span prediction-based word alignment with respect to language pairs and target domains. Furthermore, by fine-tuning the pre-trained word alignment model using manually generated word alignment data, higher accuracy than conventional supervised word alignment is achieved. The technology related to this embodiment will be described in detail below.

[0031] (Device Configuration Example) Fig. 1 shows a configuration example of an information processing device 100 according to this embodiment. As shown in Fig. 1, the information processing device 100 according to this embodiment includes a pre-training data creation unit 2, a pre-training data DB (database) 3, a pre-training unit 4, a multilingual model DB 5, a word correspondence model DB 6, a word correspondence correct answer data DB 7, a training unit 8, a trained word correspondence model DB 9, and a word correspondence inference unit 10. Fig. 1 also shows a multilingual text DB 1 that is accessed by the information processing device 100.

[0032] Each DB in the information processing device 100 has a function of storing (memorizing) data, and the DB may be replaced with a "storage unit" or a "storage unit," etc. An overview of each unit in FIG.

[0033] The multilingual text DB1 is a DB that stores multilingual text with hyperlinks. Specifically, for example, the multilingual text with hyperlinks is Wikipedia, and the multilingual text DB1 is a system on the Internet that realizes Wikipedia.

[0034] The pre-training data creating unit 2 creates pre-training data to be used for pre-training the word alignment model.

[0035] The pre-training data DB 3 stores pre-training data created by the pre-training data creation unit 2. The multilingual model DB 5 stores pre-trained multilingual models (specifically, model parameters, etc.). Examples of pre-trained multilingual models include multilingual BERT and XLM-R.

[0036] The pre-training unit 4 pre-trains the pre-trained multilingual model read from the multilingual model DB 5 as a word correspondence model by pre-training the pre-trained multilingual model using pre-training data read from the pre-training data DB 3. The pre-trained word correspondence model is stored in the word correspondence model DB 6.

[0037] The word correspondence correct data DB7 stores correct data used to train the word correspondence model. This correct data is, for example, manually created data. The training unit 8 trains the pre-trained word correspondence model read from the word correspondence model DB6 using the correct data read from the word correspondence correct data DB7. The trained word correspondence model is stored in the trained word correspondence model DB9.

[0038] The word correspondence inference unit 10 uses a word correspondence model read from the trained word correspondence model DB 9 to determine word correspondences for the input sentence pair and outputs information indicating the word correspondences. During inference, the information indicating word correspondences output by the word correspondence model is not limited to specific information. For example, it may output each word pair constituting the word correspondence in the input sentence pair, or it may output a word in one input sentence of the input sentence pair that corresponds to a specific word (a word specified by the user) in the other input sentence, or it may output information indicating word correspondences other than these. Note that in the configuration shown in FIG. 1 , functional units other than the word correspondence inference unit 10 may be provided outside the information processing device 100. In this case, the word correspondence inference unit 10 in the information processing device 100 uses a word correspondence model read from the trained word correspondence model DB 9 external to the information processing device 100 to determine word correspondences for the input sentence pair and output information indicating the word correspondences.

[0039] <Regarding Hyperlinked Multilingual Text> As described above, in this embodiment, pre-training data for a word alignment model is created from hyperlinked multilingual text.

[0040] A hyperlinked multilingual text is typically a collection of texts, such as Wikipedia, that have links from words in the text to other texts that use those words as headings, and that provide a means of mutually referencing headings (words) that have the same meaning in different languages, such as Wikipedia's interlanguage links.

[0041] <Regarding Pre-Trained Multilingual Model> In this embodiment, a word correspondence model is created by pre-training a pre-trained multilingual model as a word correspondence model.

[0042] The pre-trained multilingual model in this embodiment is a language model that is pre-trained using monolingual text in at least two target languages ​​for which word correspondence is sought. In this embodiment, multilingual BERT is used, but this is not limited to this. Any pre-trained multilingual model that can output word embedding vectors that take context into account for multilingual text, such as XLM-RoBERTa (XLM-R), may be used.

[0043] (Processing flow during training) The processing flow of "pre-training of word correspondence model and training of word correspondence model" executed by the information processing device 100 shown in Fig. 1 will be described with reference to the flowchart in Fig. 2. Note that a detailed processing example relating to the creation of pre-training data will be described later.

[0044] In S101 (step 101), the pre-training data creation unit 2 creates pre-training data for a word alignment model from a multilingual text with hyperlinks. The created pre-training data is stored in the pre-training data DB 3.

[0045] In S102, the pre-training unit 4 creates a pre-trained word correspondence model by training the pre-trained multilingual model read from the multilingual model DB 5 using pre-training data for the word correspondence model read from the pre-training data DB 3. The created pre-trained word correspondence model is stored in the word correspondence model DB 6.

[0046] In S103, the training unit 8 creates a trained word correspondence model by training (which may be called fine tuning) the pre-trained word correspondence model read out from the word correspondence model DB 6 using the word correspondence correct answer data read out from the word correspondence correct answer data DB 7. The created trained word correspondence model is stored in the trained word correspondence model DB 9.

[0047] (Flow of Inference) Next, the processing flow of "word correspondence inference" executed by the information processing device 100 shown in FIG. 1 will be described with reference to the flowchart of FIG.

[0048] In S201, a pair of a source language sentence and a target language sentence (input sentence pair) is input to the word correspondence inference unit 10.

[0049] In S202, the word correspondence inference unit 10 infers word correspondence from the input sentence pair using a trained word correspondence model read out from the trained word correspondence model DB 9. In S203, the word correspondence inference unit 10 outputs the inferred word correspondence.

[0050] (Other Device Configuration Examples) The configuration of the information processing device in this embodiment is not limited to the configuration of the information processing device 100 shown in FIG.

[0051] Another configuration example is shown in Fig. 4. The configuration of an information processing device 200 shown in Fig. 4 is the same as the configuration of the information processing device 100 shown in Fig. 1 except that the "word correspondence correct answer data DB 7, training unit 8, and trained word correspondence model DB 9" are removed.

[0052] The operation of the information processing device 200 shown in FIG. 4 is the same as the operation of the information processing device 100 shown in FIG. 1 except that the step of fine-tuning a pre-trained word correspondence model using correct answer data for word correspondence to create a trained word correspondence model is omitted.

[0053] 4, the pre-trained word correspondence model can be used as is for inferring word correspondence in the word correspondence inference unit 10. In this case, since no correct answer data for word correspondence is used, this is called zero-shot word correspondence.

[0054] 1 and the information processing device 200 in Fig. 4, the configuration for performing pre-training / training and the configuration for performing inference may be separate devices. A device having a configuration for performing pre-training / training may be called a training device, and a device having a configuration for performing inference may be called an inference device.

[0055] 5, an information processing device 300 having only a pre-training data creation unit 2 may be used. In this case, pre-training of a word alignment model is performed in a separate device using pre-training data created using the information processing device 300. The information processing device 300 may also be called a pre-training data creation device.

[0056] (Regarding "word correspondence") As will be described later, this embodiment also uses pre-training data that predicts pairs of words with the same meaning in two semantically non-equivalent sentences in the same language. Therefore, the input for inference may be a pair of sentences in the same language.

[0057] Generally, finding pairs of words with the same meaning in a pair of sentences in the same language is called monolingual word alignment. In contrast to this, finding pairs of words with the same meaning in a pair of sentences in different languages ​​is called bilingual word alignment. In the field of natural language processing, for historical reasons, "word alignment" refers to bilingual word alignment unless otherwise specified.

[0058] As described above, this embodiment can infer not only bilingual word correspondences but also monolingual word correspondences. Therefore, in the specification and claims of this application, unless otherwise specified, or unless the context clearly indicates whether monolingual word correspondences or bilingual word correspondences are intended, the term "word correspondence" shall be taken to encompass both monolingual word correspondences and bilingual word correspondences.

[0059] 6 and 7, which show specific examples, an overall flow of pre-training of a word association model executed by the information processing device 100 (or the information processing device 200) will be described. Detailed processing for creating pre-training data will be described later.

[0060] First, the pre-training data generating unit 2 collects text pairs that refer to the same entity from hyperlinked multilingual texts (specifically, Wikipedia, etc.), as shown in FIG. 6(a).

[0061] Next, the pre-training data creation unit 2 assigns word alignments to the text pairs (annotates word alignments) as shown in Figures 6(b) and 6(c). A text pair to which word alignments have been assigned is a text pair that includes a pair of phrases with the same meaning. In this embodiment, as will be described later, there are two methods for identifying word alignments: (b) Wiki word alignments, which identify word alignments based on hyperlinks; and (c) Common word alignments, which identify word alignments based on word embedding similarities and parts of speech (POS tags).

[0062] The set of text pairs annotated with word correspondences obtained as described above is stored in the pre-training data DB 3. Fig. 7(d) shows an image of the set of text pairs annotated with word correspondences.

[0063] The pre-training unit 4 performs pre-training for creating a word alignment model on a pre-trained multilingual model using text pairs (pre-training data) read from the pre-training data DB 3. In (e) of Figure 7 (upper part of Figure 7), "Transformer Encoder" is shown as an example of a pre-trained multilingual model.

[0064] The pre-training unit 4 inputs text pairs into a pre-trained multilingual model and trains the multilingual model so that it can identify correct words (words in the target language sentence that form word correspondences) that correspond to words sandwiched between specific symbols in the source language sentence. Training involves optimizing parameters so that the multilingual model (word correspondence model) can output correct answers. Note that the operation in FIG. 7( e) is basically the same as the operation during inference, except for the parameter optimization.

[0065] (Supervised word alignment using linguistic span prediction) Here, in order to facilitate understanding of the difference between the technology according to this embodiment and the conventional technology, supervised word alignment using linguistic span prediction based on the conventional technology will be described. Supervised word alignment using linguistic span prediction is the technology on which the technology according to this embodiment is based. Note that the pre-training data created by the technology according to this embodiment can be used not only for word alignment using span prediction (SpanAlign), but also for general supervised word alignment.

[0066] An example of the device configuration of an information processing device 400 that performs supervised word correspondence using language span prediction is shown in Fig. 8. As shown in Fig. 8, a training unit 8 trains a pre-trained multilingual model (such as multilingual BERT or XLM-R) read from a multilingual model DB 5 using correct word correspondence data to create a word correspondence model. A word correspondence inference unit 10 uses this trained word correspondence model to infer word correspondence.

[0067] Here, we assume the use of SpanAlign [5] as a specific technique for supervised word alignment using linguistic span prediction. In SpanAlign, word alignment is formulated as a cross-lingual span prediction problem similar to the SQuAD-style question-answering task [7]. In SQuAD, a "context," such as a paragraph selected from Wikipedia, and a "question" are given, and the question-answering system predicts a "span" (substring) within the context as the "answer." Similarly, given a target sentence as the context and a word in a source sentence as the question, a word alignment system predicts a word or word sequence that is the translation of a word in the source sentence as the span of the target sentence.

[0068] 7(e), which has been described as a technique according to the present embodiment, is also an example of determining word alignment using cross-language span prediction, a conventional technique, if only this portion is viewed. In this example, the target language is Japanese and the source language is English. With SpanAlign, a phrase in a source language sentence for which word alignment is desired is surrounded by special symbols, and the start and end points of the corresponding target language phrase are predicted. In this example, the phrase for which word alignment is desired is "Ashikaga Yoshimitu," the special symbol is "¶," and the start point and end point of the corresponding phrase are "Ashikaga" and "Yoshimitsu."

[0069] To find word alignments for all words in a pair of sentences in different languages, SpanAlign finds the corresponding word sequence in the target sentence for every word in the source sentence, then finds the corresponding word sequence in the source sentence for every word in the target sentence, and then symmetrizes the two-way span prediction to infer accurate word alignments.

[0070] In supervised word alignment using cross-lingual span prediction (SpanAlign), pairs of source and target language sentences are input from manually created ground truth data of word alignments, and information on the words in the source language sentences for which word alignment is desired and their corresponding words in the target language is obtained, and these are used as training data for the word alignment model.

[0071] In the conventional technique, the correct answer data for word alignment is a translation (parallel) of the input sentence pair, and all word alignments are fully aligned, and the word alignments are correct.

[0072] (Regarding training data for word correspondence in this embodiment) Unlike the above-mentioned conventional technology, in this embodiment, by relaxing the above-mentioned three requirements of correct answer data for word correspondence as follows (1) to (3), it is possible to automatically generate a large amount of training data for a word correspondence model using span prediction.

[0073] (1) Word correspondence may be incorrect (correct → noisy) (2) Correspondence may not be assigned to all words in the input sentence pair (fully-aligned → partially-aligned) (3) The input sentence pair may not be bilingual (parallel → non-parallel) Below, two examples of pre-training data creation method 1 and pre-training data creation method 2 will be described as examples of the operation of the pre-training data creation unit 2. In the description, reference will be made as appropriate to FIG. 6 which shows specific examples. In this embodiment, both pre-training data creation method 1 and pre-training data creation method 2 are used, but only one of them may be used.

[0074] Pre-training data creation method 1 is an example that uses Wikipedia's interlanguage links. Pre-training data creation method 2 is an example that uses unsupervised word correspondence. Note that the pre-training data creation method is not limited to these, and any method may be used as long as it can create span prediction data that satisfies the above requirements (1) to (3).

[0075] For example, from a collection of multilingual texts that do not have hyperlinks and in which it is possible to distinguish text pairs that refer to the same entity or that are semantically similar, it is possible to collect text pairs that refer to the same entity or that are semantically similar, and create pre-training data for a word alignment model by assigning word alignments to the text pairs.

[0076] (Pre-training Data Creation Method 1) First, we will explain pre-training data creation method 1. In pre-training data creation method 1, the pre-training data creation unit 2 creates pre-training data using text pairs that refer to the same entity.

[0077] 6(a) and (b) show a specific example of creating pre-training data using Wikipedia. First, as shown in Fig. 6(a), the pre-training data creation unit 2 collects pairs of paragraphs (sentences) that co-mention the same entity from Wikipedia.

[0078] The pre-training data creation unit 2 in this embodiment uses the mechanism of Wikidata (https: / / www.wikidata.org / wiki / Wikidata:Main_Page) to determine whether different expressions refer to the same entity.

[0079] In Wikidata, items that represent topics, concepts, objects, etc. are uniquely identified by a number (identifier) ​​called a "QID" that begins with the letter Q. This identifier makes it possible to determine whether multiple terms (e.g., "Ashikaga Yoshimitsu" and "Ashikaga Yoshimitsu") refer to the same entity.

[0080] For example, for each of a plurality of identifiers, the pre-training data creation unit 2 collects from Wikipedia a pair of two texts that represent the same entity corresponding to the identifier as a pair of texts corresponding to the entity, thereby collecting a set of "pairs of two texts that represent the same entity."

[0081] In addition to using the identifiers described above, Wikipedia interlanguage links may be used to determine whether the same entity is referenced. For example, the pre-training data creation unit 2 collects the text of the Wikipedia page for "Ashikaga Yoshimitsu" and the text of the "Ashikaga Yoshimitsu" page linked to by the interlanguage link to the English page on that page as a "pair of two texts representing the same entity."

[0082] For example, in FIG. 6(a), "Ashikaga Yoshimitsu was the third shogen..." and "Ashikaga Yoshimitsu was the third shogen..." are examples of a text pair that refer to the same entity in different languages.

[0083] The pre-training data creation unit 2 can also collect text pairs that refer to the same entity in the same language from Wikipedia. The examples of text pairs in Fig. 6(a) are "Ashikaga Yoshimitsu was the third shogen ..." and "in 1368 Yoshimitsu was appointed shogun ..."

[0084] The pre-training data creation unit 2 uses a text pair that refers to the same entity to create pre-training data for bilingual word correspondence between English and Japanese, as shown in FIG. 6( b), in which "Ashikaga Yoshimitsu" in "Ashikaga Yoshimitsu was the third shogen ..." is the source of word correspondence, and "Ashikaga Yoshimitsu" in "Ashikaga Yoshimitsu was a shogun in the early Muromachi period ..." is the target of word correspondence.

[0085] Similarly, the pre-training data creation unit 2 creates monolingual word-compatible pre-training data from "Ashikaga Yoshimitsu" in "Ashikaga Yoshimitsu was the third shogen ..." and "Yoshimitsu" in "in 1368 Yoshimitsu was appointed shogun ...".

[0086] In addition, in the method of determining the above-mentioned word correspondence (e.g., "Ashikaga Yoshimitsu" and "Ashikaga Yoshimitsu") from a pair of texts that refer to the same entity, the word correspondence may be a pair of words that correspond to the identifiers used to collect the text pairs, or a pair of words that are the targets of the hyperlinks used to collect the text pairs (the co-mentioned entities shown in FIG. 6(a)). Furthermore, word correspondence may be determined by other methods.

[0087] (Pre-training Data Creation Method 2) Next, creation of pre-training data using unsupervised word correspondence will be described as pre-training data creation method 2. Pre-training data creation method 2 can be applied to any text pair (text pair that mentions the same entity or semantically similar text pair) not limited to being obtained from Wikipedia, but here, as a specific example, a case where it is applied to a text pair that mentions the same entity in Wikipedia, as shown in Figures 6(a) and 6(c), will be described.

[0088] In the pre-training data creation method 1, the word correspondences obtained from text pairs that refer to the same entity in Wikipedia are mainly for proper nouns, and no word correspondences for common words are obtained.

[0089] Therefore, in the pre-training data creation method 2, the pre-training data creation unit 2 uses a trained multilingual model to create pre-training data having word correspondences for common words using unsupervised word correspondences.

[0090] For example, the pre-training data creation unit 2 infers word correspondence between a pair of texts based on the similarity of word embeddings obtained from a trained multilingual model, similar to the unsupervised word alignment SImAlign. Specifically, for example, if the similarity between the word embeddings in one text and the word embeddings in the other text in the pair is high, it can be determined that these words correspond to each other.

[0091] "war" and "wer" in Fig. 6(c) are an example of word correspondences obtained by pre-training data creation method 2. Alternatively, word correspondences may be obtained by presenting a text pair to a large-scale language model such as ChatGpt and querying it for word correspondences (corresponding word pairs).

[0092] Furthermore, the pre-training data creation unit 2 automatically assigns parts of speech to pairs of texts that refer to the same entity in Wikipedia using a part-of-speech (POS) tagger.

[0093] For example, if one of the languages ​​in the target text pair is English, an English part-of-speech tagger is used to tag each word in the English text with a part of speech related to a common word, such as a noun, verb, adjective, adverb, article, preposition, auxiliary verb, or conjunction. In this embodiment, a word tagged with a part of speech by the POS tagger is called a "common word."

[0094] The pre-training data creation unit 2 maintains a word correspondence (word pair) obtained by a trained multilingual model if at least one of the words is a "common word," and discards the word correspondence if neither word in the word correspondence (word pair) is a "common word."

[0095] Furthermore, the pre-training data creation unit 2 may use common word lists (e.g., https: / / www.wordfrequency.info / ), which are lists of frequently occurring words, to maintain a word correspondence (word pair) obtained by the trained multilingual model if at least one of the words is present in the list, and may discard the word correspondence (word pair) if neither word is present in the list.

[0096] (Hardware Configuration Example) Any of the devices (information processing devices 100, 200, 300, etc.) described in this embodiment can be realized, for example, by causing a computer to execute a program. This computer may be a physical computer or a virtual machine on the cloud.

[0097] That is, the device can be realized by executing a program corresponding to the processing performed by the device using hardware resources such as a CPU and memory built into a computer. The program can be recorded on a computer-readable recording medium (such as a portable memory) and stored or distributed. The program can also be provided via a network such as the Internet or email.

[0098] Fig. 9 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 9 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected via a bus B. The computer may further include a GPU.

[0099] The program that realizes the processing on the computer is provided by a recording medium 1001, such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0100] The memory device 1003 reads and stores a program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes functions related to the device in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) or the like according to the program. The input device 1007 is composed of a keyboard, mouse, buttons, a touch panel, etc., and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0101] (About the Experiment) An experiment was conducted to confirm the effect of the technology according to the present embodiment, and the experiment will be described below. In the following description of the experiment, the technology according to the present invention will be referred to as the "proposed method."

[0102] <Experimental Data> First, we collected paragraphs from Wikipedia dumps in English, German, French, Romanian, Chinese, and Japanese. Interlanguage link information was extracted from Wikidata. Wikipedia2Vec (https: / / wikipedia2vec.github.io / wikipedia2vec / ) was used to extract paragraphs and entity reference relationships.

[0103] Although the technology according to this embodiment can be applied to any language pair included in Wikipedia, since most of the available word alignment benchmarks are focused on English, we focused on English-based paragraph pairs, namely, De-En (German-English), Fr-En (English-French), Ro-En (Romanian-English), Zh-En (Chinese-English), and Ja-En (Japanese-English). In addition, we also created monolingual data for English.

[0104] To remove overly long paragraphs, we used the multilingual SentencePiece model flores101_mm100_615M (https: / / github.com / flairNLP / flair) to split the text into subwords, and processed paragraphs with more than 30 subwords and less than 158 subwords.

[0105] Additionally, since semantically dissimilar paragraphs have few corresponding words, text pairs in different languages ​​were filtered using the multilingual sentence embedding model LaBSE (https: / / huggingface.co / sentence-transformers / LaBSE) to select text pairs with a similarity score of 0.75 or higher. Similarly, monolingual English text pairs were filtered using the sentence embedding model pcl-bert-base-uncased (https: / / github.com / qiyuw / PeerCL) to calculate similarity.

[0106] Finally, we randomly sampled 2 million pairs of texts. We then applied unsupervised word alignment to 200,000 pairs of texts and generated general word alignment data using an English part-of-speech tagger (flair / upos-multi, https: / / huggingface.co / flair / upos-multi).

[0107] <Benchmark Data> The technology according to this embodiment was evaluated on word alignment datasets for five language pairs: Chinese-English (Zh-En), Japanese-English (Ja-En), German-English (De-En), Romanian-English (Ro-En), and English-French (En-Fr), which are the same benchmarks used in [5].

[0108] <Evaluation Measure for Word Alignment Accuracy> In this embodiment, the F1 score expressed by equation (1) is used as the evaluation measure for word alignment, which has equal weighting on precision and recall.

[0109] Since some prior studies only report the AER (Alignment Error Rate, word error rate) [6], we also use the AER to compare the prior studies with the technology according to this embodiment.

[0110] In AER, the manually created correct word data consists of sure correspondences (sure, S) and possible correspondences (possible, P), where S ⊆ P. The precision, recall, and AER of word correspondence A are defined as follows:

[0111]

[0112]

[0113] <Pre-training and Fine-tuning of Word Alignment Model> In this embodiment, the pre-training of the word alignment model starts with a trained multilingual model and uses word alignment pre-training data to perform 100,000 steps of continual pre-training, including a 2,000-step warm-up. bert-base-multilingualcased (https: / / huggingface.co / bert-base-multilingual-cased) is used for Zh-En and Ja-En, and xlm-roberta-base (https: / / huggingface.co / xlm-roberta-base) is used for De-En, En-En, and Ro-En. The learning rate is 1e-6, the maximum sequence length is 384, and the batch size is 96. In either case, the pre-trained model is a 12-layer transformer encoder with a hidden layer size of 768 and 12 attention heads.

[0114] When supervised word alignment is performed using the trained word alignment model of this embodiment, fine tuning is performed using ground truth data for word alignment, and the number of epochs is 5.

[0115] In addition, we also conducted experiments on zero-shot word alignment, which does not fine-tune using ground truth data for word alignment, and few-shot word alignment, which fine-tunes using a small amount of ground truth data. In the few-shot experiments, we used ground truth data of 32 sentence pairs and performed fine-tuning for 250 epochs.

[0116] <Comparison of word alignment accuracy> Figure 10 shows a comparison of the proposed method (WSPAlign) and conventional methods for supervised word alignment (fine-tuned using ground truth data for word alignment) for five language pairs. The conventional methods include Giza++ and FastAlign, which are word alignment methods based on statistical translation models; SimAlign, which is unsupervised word alignment based on a trained multilingual model; AWESOME, which is semi-supervised word alignment using a trained multilingual model and bilingual data; and SpanAlign, which is supervised word alignment using manually created ground truth data for word alignment and a trained multilingual model. The values ​​in Figure 10 are those reported in the paper for each benchmark data set.

[0117] For all benchmark datasets across all language pairs, our weakly supervised word alignment method (WSPAlign) outperforms all previous methods. For example, compared to the previous highest accuracy, supervised word alignment (SpanAlign), our proposed method (WSPAlign) achieves an F1 improvement of 3.3-6.1 points and an AER improvement of 1.5-6.1 points. Note that a larger F1 is better, and a smaller AER is better. <Zero-shot and Few-shot Word Alignment> To examine the effectiveness of manually generated ground truth data for word alignment in our proposed method, we used a pre-trained word alignment model to calculate the F1 accuracy for three cases: zero-shot (no ground truth data used), few-shot (fine-tuned with 32 sentence pairs of ground truth data), and fine-tuned with all ground truth data. Figures 11 and 12 show the F1 results.

[0118] In Figures 11 and 12, WSPAlign-M6 is the result when pre-trained using pre-training datasets for six languages ​​(five language pairs), WSPAlign-E is the result when pre-trained using only the English pre-training dataset, SpanAlign is the highest accuracy of supervised word alignment SpanAlign achieved using all ground truth data, SimAlign is the accuracy of unsupervised word alignment, and FastAlign and Giza++ are the accuracy of word alignment using a statistical translation model.

[0119] As shown in Figure 12 (c) and (d), for English-German (De-En) and Romanian-English, the zero-shot word alignment accuracy of the proposed method WSPAlign exceeds that of the unsupervised word alignment method SimAlign, and with a small amount of correct answer data (32 sentence pairs), it surpasses the accuracy of SpanAlign, the highest accuracy achieved so far.

[0120] <Effect of Monolingual Pre-Training Data> In Figures 11 and 12, when comparing pre-training on five language pairs (WSPAlign-M6) with pre-training on a monolingual language (WSPAlign-E), it can be seen that there is not much difference in the maximum achievable accuracy when using a small number of ground truth data (32 sentence pairs) or when using all ground truth data. Of course, because it uses cross-lingual pre-training data, WSPAlign-M6 has higher word alignment accuracy.

[0121] This shows that the pre-training of word correspondence in the proposed method has the aspects of gaining knowledge of cross-linguistic correspondence relationships and improving span prediction ability, and that span prediction ability can be improved even with monolingual pre-training data alone.

[0122] Furthermore, since the accuracy of span prediction for same-language text pairs is improved by using monolingual pre-training data, the proposed method can also be applied to monolingual word matching.

[0123] (Summary of the Technology According to the Present Embodiment) Conventional word alignment methods require bilingual data with manually assigned word alignments or a large amount of bilingual data as training data. In contrast, the technology according to the present embodiment relaxes the requirements for training data for a word alignment model, such as "the word alignments must be correct, all words in the bilingual text pair must be fully aligned, and the training data must be parallel," thereby creating a large amount of training data that is "noisy, partially aligned, and not necessarily bilingual," and pre-training a word alignment model.

[0124] Specifically, in the technology according to this embodiment, although word correspondence using cross-language span prediction is highly accurate, it can only be applied to language pairs for which correct answer data for word correspondence exists. In order to solve this problem, a large amount of pre-training data for word correspondence is automatically created from multilingual text with hyperlinks (specifically, Wikipedia).

[0125] Any method can be used to create pre-training data for word alignments, as long as it relaxes the properties of manually created correct answer data for word alignments, such as "the alignments are correct, all words are fully aligned, and word alignments are assigned to parallel text pairs in the first place." As specific examples, we have shown the method of using text pairs that mention the same entity on Wikipedia, as well as the method of using unsupervised word alignments to assign word alignments to text pairs, as mentioned above.

[0126] A trained multilingual model pre-trained with pre-trained data for word correspondence can infer word correspondence as is (zero shot), but when further fine-tuned with manually created ground truth data for word correspondence, it can achieve the world's highest accuracy.

[0127] Cross-language span prediction requires knowledge for crossing languages ​​and knowledge for span prediction, and the knowledge for span prediction can be learned even from monolingual pre-training data. Therefore, the technology according to this embodiment can also be used as a method for realizing monolingual word correspondence.

[0128] (Effects of the Technology According to the Present Embodiment) The technology according to the present embodiment makes it possible to automatically generate a large amount of pre-training data for a word alignment model, thereby achieving word alignment that is more robust, highly accurate, and applicable in a wider range than conventional techniques.

[0129] The following additional notes are provided regarding the above-described embodiments.

[0130] <Additional Notes> (Additional Item 1) An information processing device comprising: a pre-training data creation unit that creates pre-training data for a word alignment model by collecting text pairs that refer to the same subject or semantically similar text pairs from a collection of multilingual texts and assigning word alignments to the text pairs. (Additional Item 2) The information processing device according to Additional Item 1, wherein the pre-training data creation unit collects the text pairs using an identifier that identifies the subject. (Additional Item 3) The information processing device according to Additional Item 1, wherein the pre-training data creation unit assigns word alignments to the text pairs using a multilingual model. (Additional Item 4) The information processing device according to Additional Item 3, wherein the pre-training data creation unit selects word alignments to be used as the pre-training data from a plurality of word alignments assigned using the multilingual model based on word appearance frequencies or parts of speech. (Additional Item 5) The information processing device according to Additional Item 1, wherein word alignments for input text pairs are inferred using the word alignment model pre-trained using the pre-training data. (Supplementary Item 6) The information processing device according to Supplementary Item 1, further comprising: a pre-training unit that pre-trains the word alignment model using the pre-training data created by the pre-training data creation unit. (Supplementary Item 7) The information processing device according to Supplementary Item 6, further comprising: a training unit that further trains the word alignment model pre-trained by the pre-training unit using word alignment supervised answer data. (Supplementary Item 8) The information processing device according to Supplementary Item 1, wherein the text pairs to which the word alignments have been assigned are text pairs including a pair of words having the same meaning. (Supplementary Item 9) An information processing device comprising: a word alignment inference unit that infers word alignments for an input text pair using the word alignment model that has been pre-trained using pre-training data for a word alignment model created by collecting text pairs that refer to the same subject or semantically similar text pairs from a collection of multilingual texts and assigning word alignments to the text pairs, and further trained using word alignment supervised answer data.(Supplementary Item 10) A pre-training data creation method executed by an information processing device, comprising: collecting text pairs that refer to the same subject or semantically similar text pairs from a collection of multilingual texts, and assigning word correspondences to the text pairs to create pre-training data for a word correspondence model. (Supplementary Item 11) A non-transitory storage medium storing a program for causing a computer to function as the information processing device according to any one of Supplementary Items 1 to 9.

[0131] (References) [1] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised Cross-lingual Representation Learning at Scale. arXiv:1911.02116, 2019. [2] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the NAACL-2019, pp. 4171-4186, 2019. [3] Zi-Yi Dou and Graham Neubig. Word alignment by fine-tuning embeddings on parallel corpora. In Proceedings of EACL-2021, pp. 2112-2128, 2021. [4] Chris Dyer, Victor Chahuneau, and Noah A. Smith. A Simple, Fast, and Effective Reparameterization of IBM Model 2. In Proceedings of the NAACL-HLT-2013, pp. 644-648, 2013. [5] Masaaki Nagata, Katsuki Chousa, and Masaaki Nishino. A supervised word alignment method based on cross-language span prediction using multilingual bert. In Proceedings of EMNLP-2020, pp. 555-565, 2020.[6] Franz Josef Och and Hermann Ney. A Systematic Comparison of Various Statistical Alignment Models. Computational Linguistics, Vol. 29, No. 1, pp. 19-51, 2003. [7] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Proceedings of EMNLP-2016, pp. 2383-2392, 2016. [8] Masoud Jalili Sabet, Philipp Dufter, Franc, ois Yvon, and Hinrich Schutze. Simalign: High quality word alignments without parallel training data using static and contextualized embeddings. In Findings of EMNLP2020, pp. 1627-1642, 2020. Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and variations are possible within the scope of the gist of the present invention as set forth in the claims.

[0132] This patent application claims priority based on Japanese Patent Application No. 2024-091646, filed on June 5, 2024, the entire contents of which are incorporated herein by reference.

[0133] 100 to 400 Information processing device 1 Multilingual text DB 2 Pre-training data creation unit 3 Pre-training data DB 4 Pre-training unit 5 Multilingual model DB 6 Word correspondence model DB 7 Word correspondence correct answer data DB 8 Training unit 9 Trained word correspondence model DB 10 Word correspondence inference unit 1000 Drive device 1001 Recording medium 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input device 1008 Output device

Claims

1. An information processing device having a pre-training data creation unit that collects text pairs that refer to the same subject or that are semantically similar from a collection of multilingual texts, and creates pre-training data for a word alignment model by assigning word alignments to the text pairs.

2. The information processing device according to claim 1, wherein the pre-training data creation unit collects the text pairs using an identifier that identifies the target.

3. The information processing device according to claim 1, wherein the pre-training data creation unit assigns word correspondences to the text pairs using a multilingual model.

4. The information processing device according to claim 3, wherein the pre-training data creation unit selects word correspondences to be used as the pre-training data from a plurality of word correspondences assigned using the multilingual model based on the frequency of appearance of the words or the parts of speech.

5. The information processing device according to claim 1, wherein the word alignment model pre-trained with the pre-training data is used to infer word alignments for pairs of input texts.

6. The information processing device according to claim 1, further comprising a pre-training unit that performs pre-training of the word alignment model using the pre-training data created by the pre-training data creation unit.

7. The information processing device according to claim 6, further comprising a training unit that further trains the word correspondence model that has been pre-trained by the pre-training unit using correct word correspondence data.

8. The information processing device according to claim 1, wherein the text pair to which the word correspondence is assigned is a text pair including a pair of words having the same meaning.

9. An information processing device comprising a word correspondence inference unit that infers word correspondence for an input text pair by collecting text pairs that refer to the same subject or that are semantically similar from a collection of multilingual texts, assigning word correspondences to the text pairs, and using a word correspondence model that has been pre-trained with pre-training data for the word correspondence model, and further trained using correct word correspondence data.

10. A pre-training data creation method executed by an information processing device, which creates pre-training data for a word alignment model by collecting text pairs that refer to the same subject or semantically similar text pairs from a collection of multilingual texts and assigning word alignments to the text pairs.

11. A program for causing a computer to function as an information processing device according to any one of claims 1 to 9.

Images