Corresponding device, learning device, corresponding method, learning method, and program

Abstract

To accurately perform corresponding processing for identifying a pair of mutually corresponding information in two pieces of sequence information.SOLUTION: A corresponding device includes: a question generation unit configured to receive a first language text and a second language text as input and generate a span prediction problem between the first language text and the second language text; and a span prediction unit configured to predict a span to be an answer to the span prediction problem generated by the question generation unit, using a span prediction model created using data consisting of span prediction problems between first and second languages and their answers.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for identifying pairs of corresponding sentence sets (one or more sentences) in two corresponding documents. [Background technology]

[0002] Identifying pairs of corresponding sentences in two documents that correspond to each other is called sentence alignment. A sentence alignment system generally consists of a mechanism for calculating the similarity score between sentences in two documents, and a mechanism for identifying sentence alignments for the entire document from the candidates for sentence alignment obtained by that mechanism and their scores. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Brian Thompson and Philipp Koehn. Vecalign: Improved sentence alignment in linear time and space. In Proceedings of EMNLP-2019, pp. 1342-1348, 2019. Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional technologies for matching sentences do not use context information when calculating the similarity between sentences. Furthermore, in recent years, a method for calculating similarity using vector representations of sentences using neural networks has achieved high accuracy, but this method cannot effectively utilize information on a word-by-word basis because the sentences are converted into a single vector representation at once. This results in poor accuracy.

[0005] In other words, the conventional techniques have not been able to accurately identify pairs of corresponding sentence sets in two documents that are in a corresponding relationship with each other. Note that this problem is not limited to documents and can also occur in sequence information.

[0006] The present invention has been made in consideration of the above-mentioned points, and has an object to provide a technique that enables accurate correspondence processing to identify pairs of information that correspond to each other in two pieces of sequence information. [Means for solving the problem]

[0007] According to the disclosed technology, a problem generator that receives a first language text and a second language text as input and generates a span prediction problem between the first language text and the second language text, a span prediction unit that predicts a span that is an answer to the span prediction problem generated by the problem generation unit, using a span prediction model created using data consisting of a span prediction problem between a first language and a second language and its answer; A corresponding device is provided, comprising: Effect of the Invention

[0008] The disclosed technology provides a technology that enables accurate matching processing to identify pairs of information that correspond to each other in two series information. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing the configuration of an apparatus according to a first embodiment. [Diagram 2] 1 is a flowchart showing an overall flow of processing. [Diagram 3] 13 is a flowchart illustrating a process for training a cross-language span prediction model. [Figure 4] 13 is a flowchart showing a process of generating sentence alignment. [Diagram 5] FIG. 2 is a diagram illustrating a hardware configuration of the device. [Figure 6]FIG. 11 is a diagram showing an example of sentence corresponding data. [Figure 7] FIG. 13 shows the average number of sentences and tokens in each dataset. [Figure 8] FIG. 13 is a diagram showing the F1 score for the entire correspondence relationship. [Figure 9] FIG. 13 is a diagram showing sentence alignment accuracy evaluated for each number of source and target language sentences in the alignment relationship. [Figure 10] FIG. 13 is a diagram showing a comparison result of translation accuracy when the amount of bilingual text pairs used for learning is changed. [Figure 11] FIG. 11 is a diagram showing the configuration of an apparatus according to a second embodiment. [Figure 12] 1 is a flowchart showing an overall flow of processing. [Figure 13] 13 is a flowchart illustrating a process for training a cross-language span prediction model. [Figure 14] 13 is a flowchart showing a process for generating word correspondences. [Figure 15] FIG. 11 is a diagram showing an example of word correspondence data. [Figure 16] FIG. 13 is a diagram showing examples of questions from English to Japanese. [Figure 17] FIG. 13 is a diagram illustrating an example of span prediction. [Figure 18] FIG. 13 is a diagram showing an example of symmetrization of word correspondence. [Figure 19] FIG. 13 is a diagram showing the number of data items used in an experiment. [Figure 20] FIG. 1 is a diagram showing a comparison between a conventional technique and a technique according to an embodiment. [Figure 21] FIG. 13 illustrates the effect of symmetrization. [Figure 22] FIG. 1 illustrates the importance of context for source language words. [Figure 23] FIG. 13 shows word matching accuracy when trained using a subset of Chinese and English training data. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention (the present embodiment) will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0011] In the following, examples 1 and 2 will be described as the present embodiment. In examples 1 and 2, the alignment is mainly described using a text pair between different languages ​​as an example, but this is just an example, and the present invention is not limited to the alignment of a text pair between different languages, but can also be applied to the alignment of a text pair in the same language between different domains. An example of the alignment of a text pair in the same language is the alignment of sentences / words in a colloquial style and sentences / words in a business style.

[0012] Since language is also a type of "domain," alignment of text pairs between different languages ​​is an example of alignment of text pairs between different domains.

[0013] Furthermore, a sentence, a document, or a paragraph is a sequence of tokens, and may be called sequence information. In this specification, the number of sentences that are elements of a "sentence set" may be multiple or may be one.

[0014] Example 1 First, a first embodiment will be described. In the first embodiment, the problem of identifying sentence alignment is regarded as a set of problems (cross-language span prediction) of independently predicting a set of consecutive sentences (span) in a document in one language that corresponds to a set of consecutive sentences in a document in another language, and a cross-language span prediction model is learned using a neural network from pseudo-correct answer data created by an existing method, and the prediction result is subjected to mathematical optimization within the framework of a linear programming problem, thereby realizing highly accurate sentence alignment. Specifically, a sentence alignment device 100 described later executes the process related to this sentence alignment. More specifically, the linear programming used in the first embodiment is integer linear programming. Unless otherwise specified, the "linear programming" used in the first embodiment means "integer linear programming".

[0015] In the following, a reference technique related to sentence matching will be described first in order to facilitate understanding of the technique according to the first embodiment, followed by a description of the configuration and operation of the sentence matching device 100 according to the first embodiment.

[0016] The reference numbers and names of reference documents related to the reference techniques of Example 1 are listed at the end of Example 1. In the following description, the numbers of related reference documents are indicated as "[1]" etc.

[0017] (Example 1: Description of the Reference Technology)

[0018] As mentioned above, a sentence alignment system generally consists of a mechanism for calculating the similarity scores between sentences in two documents and a mechanism for identifying sentence alignments for the entire document from the candidates for sentence alignment obtained by this mechanism and their scores.

[0019] Regarding the former mechanism, conventional methods use similarity that does not take into account context, based on sentence length [1], bilingual dictionaries [2, 3, 4], machine translation systems [5], multilingual sentence vectors [6] (Non-Patent Document 1 mentioned above), etc. For example, Thompson et al. [6] propose a method to obtain language-independent multilingual sentence vectors using a method called LASER, and calculate the similarity score of sentences from the cosine similarity between those vectors.

[0020] Regarding the latter mechanism for identifying sentence correspondences across an entire document, many conventional techniques, such as those proposed by Thompson et al. [6] and Uchiyama et al. [3], use dynamic programming (DP) techniques that assume monotonicity of sentence correspondences.

[0021] Uchiyama et al. [3] proposed a sentence alignment method that takes into account the score of document alignment. In this method, a document in one language is translated into the other language using a bilingual dictionary, and the documents are aligned based on BM25 [7]. Next, sentence alignment is performed from the obtained document pairs using a sentence similarity called SIM and alignment using DP. SIM is defined based on the relative frequency of words that correspond one-to-one between two documents using a bilingual dictionary. In addition, the average of the SIMs of the sentence alignments in the corresponding documents is used as a score AVSIM that represents the reliability of the document alignment, and the product of SIM and AVSIM is used as the final sentence alignment score. This makes it possible to perform robust sentence alignment even when the document alignment is not very accurate. This method is commonly used as a sentence alignment method between English and Japanese.

[0022] (Example 1: Issues) In the conventional techniques described above, no context information is used when calculating the similarity between sentences. Furthermore, in recent years, methods that calculate similarity using vector representations of sentences using neural networks have achieved high accuracy, but these methods cannot effectively utilize word-by-word information because sentences are converted into a single vector representation at once. This can result in a loss of accuracy in matching sentences.

[0023] In addition, most of the conventional techniques perform global optimization using dynamic programming that assumes the correspondence is monotonic. However, not all of the correspondences in actual bilingual documents are monotonic. In particular, it is known that legal documents contain non-monotonic sentence correspondences, and the conventional techniques lose accuracy when used on such documents.

[0024] A technique for solving the above problems and enabling highly accurate sentence matching will be described below as a first embodiment.

[0025] (Overview of the technology according to the first embodiment) In the first embodiment, sentence alignment is first converted into a cross-language span prediction problem. Cross-language span prediction is realized by fine-tuning a multilingual language model that has been pre-trained using monolingual data related to at least the pair of languages ​​being handled, using pseudo sentence alignment correct answer data created by an existing method. In this case, a sentence from one document and another document are input to the model, so that the context before and after the span can be taken into account when making predictions. In addition, by using a multilingual language model that uses a structure called self-attention, word-by-word information can be utilized.

[0026] Next, in order to identify correspondences that are consistent across the entire document, the scores of candidates for sentence alignment based on span prediction are symmetrized and then global optimization is performed using linear programming. This improves the reliability of the results of asymmetric cross-language span prediction, and makes it possible to identify non-monotonic sentence alignments. With this method, highly accurate sentence alignment is achieved in the first embodiment.

[0027] (Device configuration example) FIG. 1 shows a sentence matching device 100 and a pre-learning device 200 in the first embodiment. The sentence matching device 100 is a device that executes a sentence matching process using the technology according to the first embodiment. The pre-learning device 200 is a device that learns a multilingual model from multilingual data. Note that both the sentence matching device 100 and a word matching device 300 described later may be called "matching devices."

[0028] As shown in FIG. 1, the sentence matching device 100 includes a cross-language span prediction model training unit 110 and a sentence matching execution unit 120.

[0029] The cross-language span prediction model training unit 110 has a document correspondence data storage unit 111, a sentence correspondence generation unit 112, a sentence correspondence pseudo-answer data storage unit 113, a cross-language span prediction question answer generation unit 114, a cross-language span prediction pseudo-answer data storage unit 115, a span prediction model training unit 116, and a cross-language span prediction model storage unit 117. Note that the cross-language span prediction question answer generation unit 114 may be called a question answer generation unit.

[0030] The sentence correspondence execution unit 120 has a cross-language span prediction question generation unit 121, a span prediction unit 122, and a sentence correspondence generation unit 123. Note that the cross-language span prediction question generation unit 121 may be called a question generation unit.

[0031] The pre-training device 200 is a device related to existing technology. The pre-training device 200 has a multilingual data storage unit 210, a multilingual model learning unit 220, and a pre-trained multilingual model storage unit 230. The multilingual model learning unit 220 reads monolingual texts in at least two target languages ​​or domains for which sentence correspondence is required from the multilingual data storage unit 210, thereby learning a language model, and stores the language model in the pre-trained multilingual model storage unit 230 as a pre-trained multilingual model.

[0032] In the first embodiment, since it is sufficient that a pre-trained multilingual model trained by some means is input to the cross-language span prediction model training unit 110, it is also possible to use, for example, a general-purpose pre-trained multilingual model that is publicly available, without providing the pre-training device 200.

[0033] The pre-trained multilingual model in the first embodiment is a language model that is pre-trained using at least monolingual text in each language for which sentence correspondence is required. In the present embodiment, XLM-RoBERTa is used as the language model, but is not limited to this. Any pre-trained multilingual model that can make predictions for multilingual texts taking into account word-level information and context information, such as multilingual BERT, may be used. In addition, since the model is compatible with multiple languages, it is called a "multilingual model," but it is not essential to train in multiple languages. For example, pre-training may be performed using texts from multiple domains in the same language.

[0034] The sentence matching device 100 may be called a learning device. The sentence matching device 100 may not include the cross-language span prediction model training unit 110, but may include the sentence matching execution unit 120. An apparatus that is solely equipped with the cross-language span prediction model training unit 110 may be called a learning device.

[0035] (Overview of the operation of the sentence correspondence device 100) 2 is a flowchart showing the overall operation of the sentence matching device 100. In S100, a pre-trained multilingual model is input to the cross-language span prediction model training unit 110, and the cross-language span prediction model training unit 110 trains a cross-language span prediction model based on the pre-trained multilingual model.

[0036] In S200, the cross-language span prediction model trained in S100 is input to the sentence alignment execution unit 120, which uses the cross-language span prediction model to generate and output sentence alignments for the input document pair.

[0037] <s100> The process of training the cross-language span prediction model in S100 will be described with reference to the flowchart in Fig. 3. As a premise of the flowchart in Fig. 3, it is assumed that a pre-trained multilingual model has already been input and that the pre-trained multilingual model is stored in the storage device of the cross-language span prediction model training unit 110. It is also assumed that the sentence-corresponding pseudo-superficial-answer data storage unit 111 stores sentence-corresponding pseudo-superficial-answer data.

[0038] In S101, the cross-language span prediction question answer generation unit 114 reads sentence-corresponding pseudo-correct answer data from the sentence-corresponding pseudo-correct answer data storage unit 113, generates cross-language span prediction pseudo-correct answer data from the read sentence-corresponding pseudo-correct answer data, i.e., a pair of a cross-language span prediction question and its pseudo answer, and stores it in the cross-language span prediction pseudo-correct answer data storage unit 113.

[0039] Here, the pseudo-correct answer data for sentence correspondence includes, for example, a document in the first language, a document in the second language corresponding thereto, and data showing correspondence between a set of sentences in the first language and a set of sentences in the second language, when a sentence correspondence is sought between a first language and a second language. The data showing correspondence between a set of sentences in the first language and a set of sentences in the second language is, for example, data showing correspondence such that, when a document in the first language=(sentence 1, sentence 2, sentence 3, sentence 4) and a document in the second language=(sentence 5, sentence 6, sentence 7, sentence 8), (sentence 1, sentence 2) corresponds to (sentence 6, sentence 7), and (sentence 1, sentence 2) corresponds to (sentence 5, sentence 6).

[0040] As described above, pseudo-superficial answer data for sentence correspondence is used in Example 1. The pseudo-superficial answer data for sentence correspondence is obtained by performing sentence correspondence using an existing method from data of document pairs that have been manually or automatically aligned.

[0041] In the configuration example shown in FIG. 1, data on document pairs that have been manually or automatically aligned is stored in a document alignment data storage unit 111. The data is document alignment data written in the same language (or domain) as the document pair for which the sentence alignment is sought. From this document alignment data, a sentence alignment generation unit 112 generates pseudo-correct answer data for sentence alignment using an existing method. More specifically, sentence alignment is sought using the technology of Uchiyama et al. [3] described in the reference technology. In other words, sentence alignment is sought from the document pair by matching using the inter-sentence similarity called SIM and DP.

[0042] In addition, instead of the pseudo-correct data corresponding to the sentence, manually created correct data corresponding to the sentence may be used. Moreover, the "pseudo-correct data" and the "correct data" may be collectively referred to as "correct data."

[0043] In S102, the span prediction model training unit 116 trains a cross-language span prediction model from the cross-language span prediction pseudo-ground-truth data and the pre-trained multilingual model, and stores the trained cross-language span prediction model in the cross-language span prediction model storage unit 117.

[0044] <s200> Next, the process of generating sentence alignment in S200 will be described with reference to the flowchart in Fig. 4. Here, it is assumed that the cross-language span prediction model has already been input to the span prediction unit 122 and stored in the storage device of the span prediction unit 122.

[0045] In S201, a document pair is input to the cross-language span prediction problem generation unit 121. In S202, the cross-language span prediction problem generation unit 121 generates a cross-language span prediction problem from the input document pair.

[0046] Next, in S203, the span prediction unit 122 uses the cross-language span prediction model to perform span prediction on the cross-language span prediction question generated in S202 to obtain an answer.

[0047] In S204, the sentence alignment generation unit 123 performs global optimization from the answers to the cross-language span prediction questions obtained in S203 to generate sentence alignments. In S205, the sentence alignment generation unit 123 outputs the sentence alignments generated in S204.

[0048] It should be noted that the "model" in this embodiment is a neural network model, and specifically, is made up of weight parameters, functions, and the like.

[0049] (Hardware configuration example) The sentence matching device and learning device in Example 1, and the word matching device and learning device in Example 2 (collectively referred to as "devices") can be realized by, for example, having a computer execute a program describing the processing contents described in this embodiment (Example 1, Example 2). Note that this "computer" may be a physical machine or a virtual machine on the cloud. When a virtual machine is used, the "hardware" described here is virtual hardware.

[0050] The above program can be recorded on a computer-readable recording medium (such as a portable memory) and stored or distributed. The above program can also be provided via a network such as the Internet or email.

[0051] Fig. 5 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 5 has a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, etc., which are all connected to each other via a bus B.

[0052] The program for realizing the processing on the computer is provided by a recording medium 1001 such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0053] The memory device 1003 reads out and stores the program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes the functions related to the device in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network. The display device 1006 displays a GUI (Graphical User Interface) or the like according to a program. The input device 1007 is composed of a keyboard and mouse, buttons, a touch panel, or the like, and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0054] (Example 1: Explanation of specific processing contents) The process contents of the sentence matching device 100 in the first embodiment will be described in more detail below.

[0055] <Formulation from sentence alignment to span prediction> In the first embodiment, sentence alignment is formulated as a cross-language span prediction problem similar to the SQuAD-format question answering task [8]. First, the formulation from sentence alignment to span prediction is explained using an example. In relation to the sentence alignment device 100, the cross-language span prediction model and its learning in the cross-language span prediction model training unit 110 are mainly explained here.

[0056] A question-answering system performing an SQuAD-style question-answering task is given a "context," such as a paragraph selected from Wikipedia, and a "question," and the system predicts the "span" in the context as the "answer."

[0057] Similar to the above span prediction, the sentence matching execution unit 120 in the sentence matching device 100 of the first embodiment regards the target language document as a context and a set of sentences in the source language document as a question, and predicts the set of sentences in the target language document, which is the translation of the set of sentences in the source language document, as the span of the target language document. For this prediction, the cross-language span prediction model in the first embodiment is used.

[0058] --About the cross-language span prediction question answer generator 114-- In the first embodiment, the cross-language span prediction model learning unit 110 of the sentence matching device 100 performs supervised learning of the cross-language span prediction model, but correct answer data is required for learning. In the first embodiment, the cross-language span prediction question answer generation unit 114 generates this correct answer data as pseudo correct answer data from the sentence-matched pseudo correct answer data.

[0059] Fig. 6 shows examples of cross-language span prediction questions and answers in Example 1. Fig. 6(a) shows a monolingual question answering task in the SQuAD format, and Fig. 6(b) shows a sentence alignment task from bilingual documents.

[0060] The cross-language span prediction problem and answer shown in Fig. 6(a) consists of a document, a question (Q), and an answer (A) to that. The cross-language span prediction problem and answer shown in Fig. 6(b) consists of an English document, a Japanese question (Q), and an answer (A) to that.

[0061] As an example, if the target document pair is an English document and a Japanese document, the cross-language span prediction question answer generation unit 114 shown in Figure 1 generates multiple documents (contexts) and question-answer pairs as shown in Figure 6(b) from the sentence-corresponding pseudo-answer data.

[0062] As described later, in the first embodiment, the span prediction unit 122 of the sentence correspondence execution unit 120 uses the cross-language span prediction model to make predictions in each direction, from the first language document (question) to the second language document (answer) and from the second language document (question) to the first language document (answer). Therefore, when training the cross-language span prediction model, bidirectional pseudo-answer data may be generated to perform bidirectional training so that predictions can be made in this way.

[0063] Note that performing prediction in both directions as described above is just an example. It is also possible to perform prediction in only one direction, such as prediction from the first language document (question) to the second language document (answer) or prediction from the second language document (question) to the first language document (answer).

[0064] --Definition of the cross-linguistic span prediction problem-- The definition of the cross-language span prediction problem in the first embodiment will be described in more detail. Let F be a source language document consisting of tokens of length N, and let F={f 1 ,f 2 ,...,f N }, and let E = {e 1 ,e 2 ,...,e M }.

[0065] In the first embodiment, the cross-language span prediction problem is to find a source language sentence Q={f i ,f i+1 ,...,f j }, the target text R of span (k, l) in the target document E is k ,e k+1 ,...,e l }. The "source language sentence Q" may be one sentence or multiple sentences.

[0066] In the sentence alignment in the first embodiment, not only one sentence can be aligned, but also multiple sentences can be aligned. In the first embodiment, any consecutive sentences in a source language document are input as a source language sentence Q, so that one-to-one and many-to-many alignments can be handled in the same framework.

[0067] --About the span prediction model learning unit 116-- The span prediction model training unit 116 trains the cross-language span prediction model using the pseudo-correct answer data read from the cross-language span prediction pseudo-correct answer data storage unit 115. That is, the span prediction model training unit 116 inputs a cross-language span prediction question (question and context) to the cross-language span prediction model, and adjusts the parameters of the cross-language span prediction model so that the output of the cross-language span prediction model becomes a correct answer (pseudo-correct answer). This parameter adjustment can be performed using existing technology.

[0068] The trained cross-language span prediction model is stored in the cross-language span prediction model storage unit 117. In addition, the sentence matching execution unit 120 reads out the cross-language span prediction model from the cross-language span prediction model storage unit 117 and inputs it to the span prediction unit 122.

[0069] --About the pre-trained model BERT-- Here, a pre-trained model BERT, which is expected to be used as the pre-trained multilingual model in Example 1, will be described. BERT [9] is a language representation model that uses an encoder based on Transformer to output a word embedding vector for each word in an input sequence, taking into account the surrounding context. Typically, the input sequence is one sentence or two sentences concatenated with a special symbol between them.

[0070] BERT pre-trains a language representation model from large-scale language data using a task of learning a masked language model that predicts masked words in the input sequence both forward and backward, and a next sentence prediction task that determines whether two given sentences are adjacent or not. By using such pre-training tasks, BERT can output word embedding vectors that capture features related to linguistic phenomena not only within a single sentence but also across two sentences. Note that language representation models such as BERT are sometimes simply called language models.

[0071] It has been reported that adding an appropriate output layer to pre-trained BERT and fine-tuning with the training data of the target task can achieve the highest accuracy in various tasks such as text semantic similarity, natural language inference (textual entailment recognition), question answering, named entity extraction, etc. Note that the above-mentioned fine-tuning means, for example, using the parameters of pre-trained BERT as the initial values ​​of the target model (a model with an appropriate output layer added to BERT) to train the target model.

[0072] In tasks that use pairs of sentences as input, such as semantic text similarity, natural language inference, and question answering, BERT is given input by concatenating two sentences using special symbols, such as '[CLS] first sentence [SEP] second sentence [SEP]'. Here, [CLS] is a special token used to create a vector that summarizes the information of the two input sentences, and is called the classification token, and [SEP] is a token that represents the division of sentences, and is called the separator token.

[0073] In a task where one input sentence is used to predict the span of the other sentence based on the other sentence, such as in question answering (QA), the vector output by BERT for [CLS] is used to predict whether or not there is a span to be extracted in the other sentence, and the vector output by BERT for each word in the other sentence is used to predict the probability that that word is the start point of the span to be extracted and the probability that that word is the end point of the span to be extracted.

[0074] BERT was originally created for English, but now BERTs for various languages, including Japanese, have been created and made publicly available. In addition, multilingual BERT, a general-purpose multilingual model created using monolingual data for 104 languages ​​extracted from Wikipedia, has been made publicly available.

[0075] Furthermore, a cross-language language model called XLM has been proposed, which is pre-trained using a fill-in-the-blank language model with bilingual texts. It has been reported to be more accurate than multilingual BERT in applications such as cross-language text classification, and pre-trained models are publicly available.

[0076] --About the cross-linguistic span prediction model-- The cross-language span prediction model in the first embodiment selects a span (k, l) of the target language text R corresponding to the source language sentence Q from the target language document E during both learning and sentence matching execution.

[0077] In the sentence alignment generation unit 123 (or the span prediction unit 122) of the sentence alignment execution unit 120, a correspondence score ω ijkl Let p be the probability of the starting position. 1 and the probability of the end position p 2 Using the product of these, the calculation is as follows:

[0078]

number

[0079] In the cross-language span prediction model of the first embodiment, a source language sentence Q and a target language document E are combined and one piece of sequence data as shown below is input.

[0080] [CLS] Source text Q [SEP] Target document E [SEP] The cross-language span prediction model in Example 1 is a model in which two independent output layers are added to a pre-trained multilingual model, and then fine-tuned using training data for the task of predicting spans between a target language document and a source language document. These output layers have a probability p 1 Or the probability of being in the final position p 2 Predict.

[0081] <About span prediction> Next, the operation of the sentence corresponding execution unit 120 will be described in detail.

[0082] --Cross-language span prediction question generation unit 121 and span prediction unit 122-- The cross-language span prediction problem generation unit 121 creates a span prediction problem of the form "[CLS] source language sentence Q[SEP] target language document E[SEP]" for each input document pair (source language document and target language document) and outputs it to the span prediction unit 122.

[0083] As described later, in Example 1, bidirectional prediction is performed, so if a document pair is a first language document and a second language document, the cross-language span prediction problem generation unit 121 may generate a span prediction problem from the first language document (question) to the second language document (answer) and a span prediction problem from the second language document (question) to the first language document (answer).

[0084] The span prediction unit 122 receives each question (question and context) generated by the cross-language span prediction question generation unit 121 and generates an answer (predicted span) and a probability p 1 , p 2 Calculate the answer (predicted span) and probability p for each question. 1 , p 2 to the sentence alignment generation unit 123.

[0085] --About the sentence matching unit 123-- For example, the sentence alignment generation unit 123 calculates the best answer span (^k, ^l) for the source language sentence as an alignment score ω ijkl The sentence alignment generating unit 123 may output the selection result and the source language sentence as a sentence alignment.

[0086]

number

[0087] More specifically, in the first embodiment, the sentence alignment unit 123 calculates the no-alignment score φ using the predicted value at the position of "[CLS]". ij Calculate this score and the corresponding span score ω ijkl Whether or not a corresponding target language text exists can be determined based on the magnitude of . For example, the sentence alignment execution unit 120 may not use a source language sentence for which no corresponding target language text exists as a source language sentence for sentence alignment generation.

[0088] Here, the unpaired score φ is calculated using the predicted value at the position of "[CLS]". ij "Calculating ω" is essentially calculating the corresponding score ω when the (start position, end position) of "[CLS]" in the sequence data input to the cross-language span prediction model is regarded as the answer span. ijkl Score φ ij This is equivalent to:

[0089] Although the answer span predicted by the cross-language span prediction model does not necessarily match the sentence boundaries in the document, the prediction result needs to be converted into a sentence sequence in order to optimize and evaluate sentence alignment. Therefore, in the first embodiment, the sentence alignment generation unit 123 obtains the longest sentence sequence that is completely contained in the predicted answer span, and the sequence is used as the prediction result at the sentence level.

[0090] --Optimization of predicted span by linear programming using the sentence matching unit 123-- Next, a description will be given of an example of a method for identifying many-to-many correspondences with high accuracy from the correspondence scores described above, which is executed by the sentence correspondence generation unit 123. Below, the problems with this method and detailed processing of this method will be described.

[0091] <Challenges> When sentence alignments obtained by cross-language span prediction using a cross-language span prediction model (e.g., sentence alignments obtained by equation (2)) are directly used, the following problems arise.

[0092] Because cross-language span prediction models independently predict the span of the target language text, there will be span overlaps in many of the predicted correspondences.

[0093] In identifying many-to-many correspondences, it is very important to determine the span of the source language sentence to be input, but it is not obvious how to select an appropriate span.

[0094] <Details of the correspondence identification method> To solve these problems, linear programming is introduced in Example 1. Global optimization using linear programming can ensure span consistency and maximize the correspondence score across the entire document. Preliminary experiments showed that converting the score into a cost and minimizing that cost achieved higher accuracy than maximizing the score, so in Example 1, the problem is formulated as a minimization problem.

[0095] In addition, since the cross-language span prediction problem is asymmetric as it is, in the first embodiment, the source language document and the target language document are swapped to solve the same span prediction problem, and the same correspondence score ω′ ijkl and the no-match score φ´ kl The method calculates the score for each direction and obtains a maximum of two prediction results for the same correspondence. By symmetrizing the scores for both directions, the reliability of the prediction results is increased, which is expected to lead to improved accuracy in sentence alignment.

[0096] If the first language document is the source document and the second language document is the target document, the correspondence score from the span (i,j) of the source sentence in the first language document to the span (k,l) of the target text in the second language document is ω ijkl where the second language document is the source document and the first language document is the target document, and the correspondence score from the span (k,l) of the source sentence in the second language document to the span (i,j) of the target text in the first language document is ω´ ijkl Also, φ ij is the score indicating that there is no span in the second language document that corresponds to span (i,j) in the first language document, and φ kl is a score indicating that there is no span in the first language document that corresponds to span (k, l) in the second language document.

[0097] In this embodiment, ω ijkl And ω´ ijkl The symmetrical score in the form of a weighted average of the above is defined as follows:

[0098]

number

[0099] In the first embodiment, a sentence alignment is defined as a set of span pairs with no overlapping spans in each document, and the sentence alignment generation unit 123 identifies sentence alignment by solving the problem of finding a set that minimizes the sum of the costs of the alignment relationships using linear programming. The linear programming in the first embodiment is formulated as follows.

[0100]

number

[0101]

number

[0102]

number

[0103]

number

[0104] y ijkl is a binary variable that indicates whether spans (i,j) and (k,l) correspond to each other, and the value is 1 when they correspond. ij ,b′ kl is a binary variable that indicates whether the spans (i,j) and (k,l) are unpaired, and a value of 1 indicates that there is no pairing. ij b ij , Σφ´ kl b´ kl In both cases, the cost increases as the number of non-matching cases increases.

[0105] Equation (6) is a constraint that ensures that for each sentence in the source document, that sentence appears in only one span pair in the correspondence. Equation (7) is a similar constraint for the destination document. These two constraints ensure that there is no overlap of spans in each document, and that each sentence is associated with some correspondence, including no correspondence.

[0106] In formula (6), any x corresponds to any source sentence. Formula (6) implies that the constraint is imposed on all source sentences that for all spans containing any source sentence x, the sum of the correspondence between any target span for that span and the pattern where x has no correspondence is 1. The same is true for formula (7).

[0107] The cost of correspondence, c ijkl is calculated from the score Ω as follows:

[0108]

number

[0109] Candidate spans of target text obtained when one source sentence is input, and their scores ω ijkl There are a number of candidates proportional to the square of the number of tokens in the target language document. Since attempting to calculate all of them as candidates would result in extremely high calculation costs, in the first embodiment, only a small number of candidates with high scores for each source language sentence are used in optimization calculations using linear programming. For example, N (N≧1) may be determined in advance, and the N candidates with the highest scores may be used for each source language sentence.

[0110] In preliminary experiments, increasing the number of candidates used for each input from one did not result in an improvement in sentence alignment accuracy, so in the experiments described below, only the highest-scoring candidate was used as the span candidate for each source sentence.

[0111] --- Filtering low-quality data taking document correspondence information into account --- When bilingual text data extracted by sentence alignment is actually used in downstream tasks, low-quality bilingual texts are often removed depending on the sentence alignment score and cost. One of the reasons for this low-quality alignment is that the alignment of automatically extracted bilingual documents is sometimes incorrect and unreliable. However, the sentence alignment scores and costs explained so far do not take into account the accuracy of document alignment.

[0112] Therefore, in the first embodiment, a document matching cost d is introduced, and the sentence matching generation unit 123 calculates the document matching cost d and the sentence matching cost c ijkl The document matching cost d is calculated by dividing formula (4) by the number of extracted sentence matchings as follows:

[0113]

number

[0114] Regarding removing low-quality bilingual sentences, for example, a document 1 in a first language and a document 2 in a second language are input to the sentence alignment execution unit 120, and the sentence alignment generation unit 123 obtains one or more bilingual sentence data that are aligned with each other. The sentence alignment generation unit 123, for example, selects d×c ijkl If d×c is larger than the threshold, it is judged to be of low quality and is not used (removed). ijkl Alternatively, only a certain number of pieces of bilingual text data may be used in ascending order of the value of .

[0115] (Effects of Example 1) The sentence alignment device 100 described in the first embodiment can achieve sentence alignment with higher accuracy than conventional methods. In addition, the extracted bilingual sentences contribute to improving the translation accuracy of the machine translation model. Below, experiments on sentence alignment accuracy and machine translation accuracy that demonstrate these effects will be described. Below, the experiment on sentence alignment accuracy will be described as Experiment 1, and the experiment on machine translation accuracy will be described as Experiment 2.

[0116] <Experiment 1: Comparison of sentence matching accuracy> We used actual Japanese and English newspaper articles in automatic bilingual documents to evaluate the accuracy of sentence alignment in Example 1. To confirm the difference in accuracy due to different optimization methods, we optimized the results of cross-language span prediction using two methods, dynamic programming (DP) [1] and linear programming (ILP, the method in Example 1), and compared them. In addition, we used the method by Thompson et al. [6], which has achieved the highest accuracy in various languages, and the method by Uchiyama et al. [3], which is the de facto standard method between Japanese and English, as the baseline.

[0117] The evaluation scale is F, which is a common scale for sentence alignment. 1 We used the score. Specifically, we used the strict value in the script "https: / / github.com / thompsonb / vecalign / blob / master / score.py". This metric is calculated according to the number of exact matches between the correct and predicted correspondences. However, even though automatically extracted bilingual documents contain unmatched sentences as noise, this metric does not directly evaluate the extraction accuracy of unmatched sentences. Therefore, to perform a more detailed analysis, we calculated the Precision / Recall / Frequency for each number of matching source and target language sentences. 1 Evaluation by score was also performed.

[0118] <Experiment 1: Experimental Data> For Experiment 1, we purchased and used newspaper articles from the Yomiuri Shimbun and its English edition, The Japan News (formerly the Daily Yomiuri). From these data, we created sentence alignment datasets both automatically and manually.

[0119] First, we automatically created 2,989 document alignment data using the method of Uchiyama et al. [3] from 317,491 Japanese articles and 3,878 English articles published in 2012. We then performed sentence alignment on the document alignment data using the method of Uchiyama et al. [3], and used the resulting pseudo-ground-truth sentence alignment data as training data for the cross-language span prediction model.

[0120] For the development and evaluation data, 157 bilingual documents consisting of 131 articles and 26 editorials were created by manually searching for Japanese articles corresponding to 182 English articles published between 2013 / 02 / 01-2013 / 02 / 07 and 2013 / 08 / 01-2013 / 08 / 07. Next, sentence alignment was performed manually from each bilingual document, resulting in 2,243 many-to-many sentence alignments. In this experiment, 15 articles from the data were used for development, another 15 articles were used for evaluation, and the remaining data was reserved. Figure 7 shows the average number of sentences and tokens in each dataset.

[0121] <Experiment 1: Experimental results> Figure 8 shows the F 1 The scores are shown below. Regardless of the optimization method, the results of cross-language span prediction show higher accuracy than the baseline. This shows that the extraction of sentence alignment candidates and score calculation using cross-language span prediction works more effectively than the baseline. In addition, since the results using two-way scores are better than the results using only one-way scores, it can be confirmed that symmetrizing the scores is very effective for sentence alignment. Next, comparing the scores of DP and ILP, ILP achieves much higher accuracy. This shows that optimization using ILP can identify sentence alignments better than optimization using DP, which assumes monotonicity.

[0122] Figure 9 shows the accuracy of sentence alignment evaluated for each number of source and target language sentences in the alignment. In Figure 9, the values ​​in N rows and M columns are the Precision / Recall / F 1 The hyphen indicates that the corresponding score does not exist in the test set.

[0123] Here too, the results of sentence alignment using cross-lingual span prediction outperform the baseline results for all pairs. Furthermore, except for 1-to-2 alignments, the accuracy of the ILP optimization is higher than that of the DP optimization. In particular, the F 1 The scores are very high, at 80.0 and 95.1, which shows a significant improvement over the baseline. This result shows that the technology of Example 1 can identify unmatched sentences with very high accuracy and is very effective for bilingual documents that contain such sentences.

[0124] In this experiment, an NVIDIA Tesla K80 (12GB) was used. In the test set, it took about 1.9 seconds to predict the span for each input, and the average time required for optimization using linear programming for a document was 0.39 seconds. Traditionally, dynamic programming has been used, which requires less computational effort than linear programming in terms of time complexity, but these results show that optimization can also be performed in a practical amount of time using linear programming.

[0125] <Experiment 2: Comparison of machine translation accuracy> Next, we will explain Experiment 2. Bilingual data extracted by sentence alignment is essential for learning cross-language models that are primarily based on machine translation systems. Therefore, in order to evaluate the effectiveness of downstream tasks in Example 1, we conducted an experiment to compare the accuracy of Japanese-English machine translation models using bilingual sentences automatically extracted from actual newspaper article data. In this experiment, we compared the following five methods. The words in parentheses indicate the notations used in the legend in Figure 10.

[0126] Cross-Lingual Span Prediction + ILP (ILP w / o doc) Cross-language span prediction + ILP + document correspondence cost (ILP) · Cross-linguistic span prediction + DP (monotonic DP) · Thompson et al.'s method [6] (vecalign) ·Uchiyama et al.'s method [3] (utiyama) In Experiment 2, we evaluated a pre-trained machine translation model using the JParaCrawl corpus

[10] and fine-tuned it using extracted bilingual data. We used BLEU

[11] , a commonly used metric in machine translation, as the evaluation metric.

[0127] <Experiment 2: Experimental Data> As in Experiment 1, data was created from the Yomiuri Shimbun and The Japan News. For the training dataset, articles published between 1989 and 2015 were used, excluding those used in development and evaluation. For automatic document alignment, the method of Uchiyama et al. [3] was used, and 110,821 bilingual document pairs were created. Bilingual sentences were extracted from bilingual documents using each method, and used in order of quality based on cost and score. For the development and evaluation dataset, the same data as in Experiment 1 was used, with 168 bilingual sentences from 15 articles as development data and 238 bilingual sentences from 15 articles as evaluation data.

[0128] <Experiment 2: Experimental results> Figure 10 shows the results of a comparison of translation accuracy when the amount of bilingual sentence pairs used for training is changed. It can be seen that the results of the sentence alignment method using cross-language span prediction achieve higher accuracy than the baseline. In particular, the method using ILP and document alignment cost achieves a maximum BLEU score of 19.0 points, which is 2.6 points higher than the best result of the baseline. These results show that the technology of Example 1 works effectively for automatically extracted bilingual documents and is useful in downstream tasks.

[0129] When we focus on the part with a small amount of data, we can see that the method using document correspondence cost achieves the same or higher translation accuracy than other methods using only ILP or DP. This shows that using document correspondence cost is useful for improving the reliability of sentence correspondence cost and removing low-quality correspondences.

[0130] (Summary of Example 1) As described above, in the first embodiment, the problem of identifying a pair of corresponding sets of sentences (or sentences) in two corresponding documents is regarded as a set of problems of independently predicting, as a span, a set of consecutive sentences in a document in one language that corresponds to a set of consecutive sentences in a document in another language (cross-language span prediction problems), and global optimization of the prediction results is performed by integer linear programming, thereby achieving highly accurate sentence alignment.

[0131] The cross-language span prediction model of the first embodiment is created by fine-tuning a pre-trained multilingual model created using only monolingual texts for each of a plurality of languages, for example, using pseudo-correct answer data created by an existing method. By using a model that uses a structure called self-attention for the multilingual model and inputting a combined source language sentence and target language document into the model, it is possible to take into account the context before and after the span and information on a token-by-token basis during prediction. Compared to conventional methods that use bilingual dictionaries and vector representations of sentences, which do not use such information, it is possible to predict candidates for sentence correspondence with high accuracy.

[0132] It should be noted that the cost of creating correct answer data is very high. On the other hand, the sentence matching task requires more correct answer data than the word matching task described in the second embodiment. Therefore, in the first embodiment, good results are obtained by using pseudo correct answer data as correct answer data. The use of pseudo correct answer data allows for supervised learning, which makes it possible to learn a high-performance model compared to an unsupervised model.

[0133] In addition, the integer linear programming used in the first embodiment does not assume monotonicity of the correspondence. Therefore, it is possible to obtain sentence correspondence with a very high accuracy compared to the conventional method that assumes monotonicity. In this case, by using a score that symmetrizes the two-way scores obtained from the asymmetric cross-language span prediction, the reliability of the prediction candidates is improved, which contributes to further improvement of the accuracy.

[0134] The technology of automatically identifying sentence correspondences using two documents that correspond to each other as input has various effects related to natural language processing technology. For example, as in Experiment 2, by mapping sentences in a document in one language (e.g., Japanese) to sentences in a document translated into another language that have a parallel translation relationship based on the sentence correspondences, training data for a machine translator between the two languages ​​can be generated. Alternatively, by extracting pairs of sentences that have the same meaning based on sentence correspondences from a document and a document rewritten in simple language in the same language, the data can be used as training data for a paraphrase generator or a vocabulary simplifier.

[0135] [References for Example 1] [1] William A. Gale and Kenneth W. Church. A program for aligning sentences in bilingual corpora. Computational Linguistics, Vol. 19, No. 1, pp. 75-102, 1993. [2] Takehito Utsuro, Hiroshi Ikeda, Masaya Yamane, Yuji Matsumoto, and Makoto Nagao. Bilingual text, matching using bilingual dictionary and statistics. In Proceedings of the COLING-1994, 1994. [3] Masao Utiyama and Hitoshi Isahara. Reliable measures for aligning japanese-english news articles and sentences. In Proceedings of the ACL-2003, pp. 72-79, 2003. [4] D. Varga, L. Nemeth, P. Halacsy, A. Kornai, V. Tron, and V. Nagy. Parallel corpora for medium density languages. In Proceedings of the RANLP-2005, pp. 590-596, 2005. [5] Rico Sennrich and Martin Volk. Iterative, MT-based sentence alignment of parallel texts. In Proceedings of the 18th Nordic Conference of Computational Linguistics (NODALIDA 2011), pp. 175-182, Riga, Latvia, May 2011. Northern European Association for Language Technology (NEALT). [6] Brian Thompson and Philipp Koehn. Vecalign: Improved sentence alignment in linear time and space. In Proceedings of EMNLP-2019, pp. 1342-1348, 2019. [7] S. E. Robertson and S. Walker. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In Proceedings of the SIGIR-1994, pp. 232-241, 1994. [8] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of EMNLP-2016, pp. 2383-2392, 2016. [9] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the NAACL-2019, pp. 4171-4186, 2019.

[10] Makoto Morishita, Jun Suzuki, and Masaaki Nagata. JParaCrawl: A large scale web-based English- Japanese parallel corpus. In Proceedings of The 12th Language Resources and Evaluation Conference, pp. 3603-3609, Marseille, France, May 2020. European Language Resources Association.

[11] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. Example 2 Next, a second embodiment will be described. In the second embodiment, a technique for identifying word alignment between two sentences that are translations of each other will be described. Identifying words or word sets that are translations of each other in two sentences that are translations of each other is called word alignment.

[0136] The technology of automatically identifying word correspondences between two mutually translated sentences has various applications in multilingual processing and machine translation. For example, by mapping annotations of named entities such as people, places, and organizations in a sentence in one language (e.g., English) to a sentence translated into another language (e.g., Japanese) based on word correspondences, training data for a named entity extractor in that language can be generated.

[0137] In the second embodiment, the problem of finding word correspondence between two sentences that are translations of each other is regarded as a set of problems (cross-language span prediction) of predicting words or consecutive word strings (spans) in a sentence of one language that correspond to each word in a sentence of another language, and a highly accurate word correspondence is realized by learning a cross-language span prediction model using a neural network from a small amount of correct answer data created manually. Specifically, a word correspondence device 300 described later executes the processing related to this word correspondence.

[0138] In addition to generating training data for the named entity extractor described above, applications of word matching include, for example, the following:

[0139] When translating a web page in one language (e.g. Japanese) to another language (e.g. English), the original text may contain HTML tags (e.g. anchor tags) ... ), the HTML tags can be correctly mapped by identifying the range of strings in a sentence in another language that are semantically equivalent to the range of strings enclosed in the

[0140] In machine translation, when it is desired to specify a specific translation for a particular word in an input sentence using a bilingual dictionary or the like, the translation can be controlled by determining the word in the output sentence that corresponds to the word in the input sentence based on word correspondence, and if that word is not the specified word, replacing it with the specified word.

[0141] In the following, first, various reference techniques related to word matching will be described in order to facilitate understanding of the technique according to the embodiment 2. After that, the configuration and operation of the word matching device 300 according to the embodiment 2 will be described.

[0142] The reference numbers and names of reference documents related to the reference techniques of Example 2 are listed at the end of Example 2. In the following description, the numbers of related reference documents are indicated as "[1]" etc.

[0143] (Example 2: Description of the Reference Technology) <Unsupervised word matching based on statistical machine translation models> As a reference technique, first, unsupervised word matching based on a statistical machine translation model will be described.

[0144] In statistical machine translation[1], a translation model P(E|F), which converts a sentence F in a source language to a sentence E in a target language, is decomposed using Bayes' theorem into the product of a reverse translation model P(F|E) and a language model P(E), which generates a sequence of words in the target language.

[0145]

number

[0146]

number

[0147] Let X be a sequence of words of length |X|. 1:|X| =x 1 ,x 2 ,...,x |X| Let Y be a word sequence y of length |Y|. 1:|Y| =y 1 ,y 2, ...,y |Y| Then, let A be the word correspondence from the target language to the source language. 1:|Y| =a 1 ,a 2 ,...,a |Y| Here, a j is the word y in the target sentence. j is word x in the target sentence aj This indicates that it corresponds to

[0148] In generative word alignment, the translation probability based on a word alignment A is expressed as the lexical translation probability P t (y j |...) and word correspondence probability P a (a j Decompose into a product of |...).

[0149]

number

[0150]

number

[0151] In addition, HMM-based word alignment

[25] assumes that the word alignment probability depends on the word alignment of the immediately preceding word in the target sentence.

[0152]

number

[0153] Unsupervised word alignment tools based on the model described in [1] include GIZA++

[16] , MGIZA[8], and FastAlign[6]. GIZA++ and MGIZA are based on Model 4 described in [1], and FastAlign is based on Model 2 described in [1].

[0154] <Word matching based on recurrent neural networks> Next, we explain word alignment based on recurrent neural networks. There are two types of unsupervised word alignment methods based on neural networks: applying neural networks to HMM-based word alignment [26, 21] and attention-based methods in neural machine translation [27, 9].

[0155] Regarding the method of applying neural networks to HMM-based word matching, for example, Tamura et al.

[21] used a recurrent neural network (RNN) to analyze not only the previous word matching but also the word matching history from the beginning of the sentence a< j =a 1:j-1 We propose a method to determine the target of the current word by taking into account the above, and to obtain word correspondence as a single model rather than modeling lexical translation probability and word correspondence probability separately.

[0156]

number

[0157] <Unsupervised word matching based on neural machine translation model> Next, we will explain unsupervised word matching based on a neural machine translation model. Neural machine translation realizes conversion from a source language sentence to a target language sentence based on an encoder-decoder model.

[0158] The encoder converts a source language sentence X=x of length |X| into a nonlinear transformation function enc using a neural network. 1:|X| =x 1 ,...,x |X| Let us consider a sequence of internal states s of length |X|. 1:|X| =s 1 ,...,s |X| If the number of dimensions of the internal state corresponding to each word is d, then s 1:|X| is a |X|×d matrix.

[0159]

number

[0160]

number

[0161] In neural machine translation, the introduction of an attention mechanism has greatly improved translation accuracy. The attention mechanism is a mechanism that determines which word information in the source language sentence should be used when generating each word in the target language sentence in the decoder by changing the weighting of the internal state of the encoder. The basic idea of ​​unsupervised word matching based on attention in neural machine translation is to consider this attention value as the probability that two words are translations of each other.

[0162] As an example, we will explain the source-target attention between source and target sentences in Transformer

[23] , a representative neural machine translation model. Transformer is an encoder-decoder model that combines self-attention and a feed-forward neural network to parallelize the encoder and decoder. The attention between source and target sentences in Transformer is called cross attention to distinguish it from self-attention.

[0163] Transformer uses scaled dot-product attention as attention. For a query Q∈R, lq×dk , key K∈R lk×dk , a value V∈R lk×dv is defined as follows:

[0164]

number

[0165] In cross-attention, Q, K, and V are Q ∈R d×dk ,W K ∈R d×dk ,W V ∈R d×dv is defined as the weight as follows:

[0166]

number

[0167]

number

[0168]

number

[0169] In this case, Q=[t 1:|Y| ] T W Q The weight matrix A of the cross attention between the source and target sentences is |Y|×|X| Define.

[0170]

number

[0171]

number

[0172] Generally, a Transformer uses multiple layers and multiple heads (attention mechanisms trained from different initial values), but for simplicity, the number of layers and heads is set to one here.

[0173] Garg et al. reported that the average cross-attention of all heads in the second layer from the top was closest to the correct answer for word alignment, and the word alignment distribution G p We define the cross-entropy loss as follows for the word correspondence obtained from a specific head among multiple heads using

[0174]

number

[0175] In the method of Garg et al., when calculating the loss of word correspondence, in Eq. (10), t 1:i-1 Instead, the whole target sentence t 1:|Y| We also use the word-corresponding training data G p Instead of using self-training based on Transformer, they use word matching obtained from GIZA++. They report that this approach achieves word matching accuracy that exceeds that of GIZA++ [9].

[0176] <Supervised word matching based on neural machine translation model> Next, we explain supervised word matching based on a neural machine translation model. 1:|X| and target sentence Y=y 1:|Y| For, we define a word correspondence A as a subset of the Cartesian product of word positions.

[0177]

number

[0178] Discriminative word alignment models word alignments directly from the source and target sentences.

[0179]

number

[20] . In their method, first, the sequence of internal states of the encoder in the neural machine translation model is denoted by s 1 ,...,s |X| , the sequence of the decoder's internal states is t 1 ,...,t |Y| Then we project these into a common vector space using a parameter-sharing three-layer feedforward neural network.

[0180]

number

[0181]

number

[0182]

number

[0183]

number

[0184]

number

[0185]

number

[0186] <BERT pre-trained model> For word alignment, as with sentence alignment in Example 1, the pre-trained model BERT is used, as described in Example 1.

[0187] (Example 2: Issues) The conventional word matching based on a recurrent neural network and unsupervised word matching based on a neural machine translation model described as reference technologies have only been able to achieve accuracy equivalent to or slightly higher than that of unsupervised word matching based on a statistical machine translation model.

[0188] Conventional supervised word matching based on neural machine translation models has higher accuracy than unsupervised word matching based on statistical machine translation models. However, both methods based on statistical machine translation models and methods based on neural machine translation models have the problem that they require a large amount of bilingual data (on the order of millions of sentences) to train the translation model.

[0189] A technique according to a second embodiment that solves the above problems will be described below.

[0190] (Overview of the technology according to the second embodiment) In the second embodiment, word alignment is realized as a process of calculating answers from cross-language span prediction questions. First, a pre-trained multilingual model trained from at least each monolingual data related to a language pair to which word alignment is assigned is fine-tuned using correct answer data of cross-language span prediction created from correct answers of word alignments manually, thereby training a cross-language span prediction model. Next, word alignment processing is performed using the trained cross-language span prediction model.

[0191] By the above-mentioned method, in the second embodiment, it is possible to realize highly accurate word matching from a small amount of correct answer data of word matching created manually without requiring bilingual data for pre-training a model for executing word matching. The technology according to the second embodiment will be described in more detail below.

[0192] (Device configuration example) 11 shows a word matching device 300 and a pre-training device 400 in the embodiment 2. The word matching device 300 is a device that executes a word matching process using the technology according to the embodiment 2. The pre-training device 400 is a device that learns a multilingual model from multilingual data.

[0193] As shown in FIG. 11, the word matching device 300 includes a cross-language span prediction model training unit 310 and a word matching execution unit 320.

[0194] The cross-language span prediction model training unit 310 has a word corresponding correct answer data storage unit 311, a cross-language span predicted question answer generation unit 312, a cross-language span predicted correct answer data storage unit 313, a span prediction model training unit 314, and a cross-language span prediction model storage unit 315. The cross-language span predicted question answer generation unit 312 may be called a question answer generation unit.

[0195] The word correspondence execution unit 320 has a cross-language span prediction question generation unit 321, a span prediction unit 322, and a word correspondence generation unit 323. The cross-language span prediction question generation unit 321 may be called a question generation unit.

[0196] The pre-training device 400 is a device related to existing technology. The pre-training device 400 has a multilingual data storage unit 410, a multilingual model training unit 420, and a pre-trained multilingual model storage unit 430. The multilingual model training unit 420 reads monolingual texts in at least two target languages ​​for which word correspondence is to be obtained from the multilingual data storage unit 410, thereby training a language model, and stores the language model in the pre-trained multilingual model storage unit 230 as a pre-trained multilingual model.

[0197] In addition, in the second embodiment, since it is sufficient that a pre-trained multilingual model trained by some means is input to the cross-language span prediction model training unit 310, it is also possible to use, for example, a general-purpose pre-trained multilingual model that is publicly available, without providing the pre-training device 400.

[0198] The pre-trained multilingual model in the second embodiment is a language model that is pre-trained using monolingual texts of at least two target languages ​​for which word correspondence is required. In the second embodiment, the multilingual BERT is used as the language model, but is not limited thereto. Any pre-trained multilingual model that can output a word embedding vector that takes context into account for multilingual text, such as XLM-RoBERTa, may be used.

[0199] The word matching device 300 may be called a learning device. The word matching device 300 may not include the cross-language span prediction model training unit 310, but may include the word matching execution unit 320. A device that is solely equipped with the cross-language span prediction model training unit 310 may be called a learning device.

[0200] (Overview of operation of word matching device 300) 12 is a flowchart showing the overall operation of the word correspondence device 300. In S300, a pre-trained multilingual model is input to the cross-language span prediction model training unit 310, and the cross-language span prediction model training unit 310 trains a cross-language span prediction model based on the pre-trained multilingual model.

[0201] In S400, the cross-language span prediction model trained in S300 is input to the word correspondence execution unit 320, and the word correspondence execution unit 320 uses the cross-language span prediction model to generate and output word correspondences for the input sentence pair (two sentences that are translations of each other).

[0202] <s300> The process of training the cross-language span prediction model in S300 will be described with reference to the flowchart in Fig. 13. Here, it is assumed that a pre-trained multilingual model has already been input and stored in the storage device of the span prediction model training unit 324. Also, the word-associated correct answer data storage unit 311 stores word-associated correct answer data.

[0203] In S301, the cross-language span prediction question answer generation unit 312 reads word corresponding correct answer data from the word corresponding correct answer data storage unit 311, generates cross-language span prediction correct answer data from the read word corresponding correct answer data, and stores it in the cross-language span prediction correct answer data storage unit 313. The cross-language span prediction correct answer data is data consisting of a set of pairs of cross-language span prediction questions (question and context) and their answers.

[0204] In S302, the span prediction model training unit 314 trains a cross-language span prediction model from the cross-language span prediction correct answer data and the pre-trained multilingual model, and stores the trained cross-language span prediction model in the cross-language span prediction model storage unit 315.

[0205] <s400> Next, the content of the process of generating word correspondences in S400 above will be described with reference to the flowchart in Fig. 14. Here, it is assumed that the cross-language span prediction model has already been input to the span prediction unit 322 and stored in the storage device of the span prediction unit 322.

[0206] In S401, a pair of a first language sentence and a second language sentence is input to the cross-language span prediction question generation unit 321. In S402, the cross-language span prediction question generation unit 321 generates a cross-language span prediction question (question and context) from the input sentence pair.

[0207] Next, in S403, the span prediction unit 322 uses the cross-language span prediction model to perform span prediction on the cross-language span prediction question generated in S402, and obtains an answer.

[0208] In S404, the word alignment generation unit 323 generates word alignments from the answers to the cross-language span prediction questions obtained in S403. In S405, the word alignment generation unit 323 outputs the word alignments generated in S404.

[0209] (Example 2: Explanation of specific processing contents) The process contents of the word matching device 300 in the second embodiment will be described in more detail below.

[0210] <Formulation from word correspondence to span prediction> As described above, in the second embodiment, the word matching process is executed as a cross-language span prediction problem process. First, the formulation from word matching to span prediction will be described using an example. In relation to the word matching device 300, the cross-language span prediction model training unit 310 will be mainly described here.

[0211] --About word correspondence data-- An example of Japanese and English word correspondence data is shown in Figure 15. This is an example of one word correspondence data. As shown in Figure 15, one word correspondence data is composed of five pieces of data: a token (word) string in the first language (Japanese), a token string in the second language (English), a string of corresponding token pairs, an original text in the first language, and an original text in the second language.

[0212] Both the first language (Japanese) token sequence and the second language (English) token sequence are indexed, starting with 0, which is the index of the first element (the leftmost token) in the token sequence, and then indexed with 1, 2, 3, ...

[0213] For example, the first element of the third data, "0-1", indicates that the first element of the first language, "Ashikaga", corresponds to the second element of the second language, "ashikaga". Also, "24-2 25-2 26-2" indicates that "de", "a", and "ru" all correspond to "was".

[0214] In Example 2, we formulate word alignment as a cross-lingual span prediction problem similar to the SQuAD-style question answering task

[18] .

[0215] A question-answering system performing an SQuAD-style question-answering task is given a "context," such as a paragraph selected from Wikipedia, and a "question," and the system predicts a "span" (a substring) within the context as the "answer."

[0216] In the same manner as the above span prediction, the word correspondence execution unit 320 in the word response device 300 of the second embodiment regards the target language sentence as a context and the words of the source language sentence as a question, and predicts the words or word strings in the target language sentence that are translations of the words of the source language sentence as the span of the target language sentence. For this prediction, the cross-language span prediction model in the second embodiment is used.

[0217] --About the cross-language span prediction question answer generator 312-- In the second embodiment, the cross-language span prediction model learning unit 310 of the word matching device 300 performs supervised learning of the cross-language span prediction model, but correct answer data is required for the learning.

[0218] In the second embodiment, a plurality of word correspondence data as shown in FIG. 15 are stored as correct answer data in the word correspondence correct answer data storage unit 311 of the cross-language span prediction model training unit 310 and are used for training the cross-language span prediction model.

[0219] However, since the cross-language span prediction model is a model for predicting answers (spans) from questions across languages, data is generated for learning to predict answers (spans) from questions across languages. Specifically, by inputting word correspondence data to the cross-language span prediction question answer generation unit 312, the cross-language span prediction question answer generation unit 312 generates pairs of cross-language span prediction questions (questions) and answers (spans, substrings) in SQuAD format from the word correspondence data. An example of the processing of the cross-language span prediction question answer generation unit 312 will be described below.

[0220] FIG. 16 shows an example of converting the word correspondence data shown in FIG. 15 into a span prediction problem in the SQuAD format.

[0221] First, the upper half of Fig. 16 (a) will be described. In the upper half of Fig. 16 (context, question 1, answer portion), a sentence in the first language (Japanese) of the word correspondence data is given as the context, a token "was" in the second language (English) is given as question 1, and the answer is the span "desu" of the sentence in the first language. The correspondence between "desu" and "was" corresponds to the corresponding token pair "24-2 25-2 26-2" of the third data in Fig. 15. In other words, the cross-language span prediction question answer generator 312 generates a pair of span prediction question (question and context) and answer in the SQuAD format based on the corresponding token pair of the correct answer.

[0222] As described later, in the second embodiment, the span prediction unit 322 of the word correspondence execution unit 320 uses the cross-language span prediction model to make predictions in each direction, from a first language sentence (question) to a second language sentence (answer), and from a second language sentence (question) to a first language sentence (answer). Therefore, the cross-language span prediction model is also trained to make predictions in both directions.

[0223] Note that performing prediction in both directions as described above is just one example. One-way prediction, such as prediction from a first language sentence (question) to a second language sentence (answer) or prediction from a second language sentence (question) to a first language sentence (answer), may also be performed. For example, in English education and the like, when English sentences and Japanese sentences are displayed simultaneously and an arbitrary character string (word string) in the English sentence is selected with a mouse or the like, a character string (word string) in the Japanese sentence that is the translation of the selected character string (word string) is calculated and displayed on the spot, one-way prediction may be performed.

[0224] Therefore, the cross-language span prediction question answer generation unit 312 of the second embodiment converts one word correspondence data into a set of questions predicting spans in a sentence in the second language from each token in the first language, and a set of questions predicting spans in a sentence in the first language from each token in the second language. In other words, the cross-language span prediction question answer generation unit 312 converts one word correspondence data into a set of questions consisting of each token in the first language and their respective answers (spans in sentences in the second language), and a set of questions consisting of each token in the second language and their respective answers (spans in sentences in the first language).

[0225] If one token (question) corresponds to multiple spans (answers), the question is defined as having multiple answers. In other words, the cross-language span prediction question answer generation unit 112 generates multiple answers for the question. Also, if there is no span corresponding to a token, the question is defined as having no answer. In other words, the cross-language span prediction question answer generation unit 312 determines that there is no answer for the question.

[0226] In the second embodiment, the language of the question is called the source language, and the language of the context and answer (span) is called the target language. In the example shown in Figure 16, the source language is English and the target language is Japanese, and the question is called an "English-to-Japanese" question.

[0227] If the question is a high-frequency word such as "of", it may appear multiple times in the source language sentence, so it is difficult to find the corresponding span in the target language sentence unless the context of the word in the source language sentence is taken into consideration. Therefore, the cross-language span prediction question answer generation unit 312 in the second embodiment generates a question with a context.

[0228] The lower half of Figure 16(b) shows examples of questions with the context of the source language sentence. In question 2, the two tokens "Yoshimitsu ASHIKAGA" immediately before the token "was" in the source language sentence, and the two tokens "the 3rd" immediately after it in the context have '¶' added as a boundary marker.

[0229] In question 3, the entire source language sentence is used as the context, and the token that is the question is sandwiched between two boundary symbols. As will be described later in the experiment, the longer the context added to the question, the better. Therefore, in Example 2, as in question 3, the entire source language sentence is used as the context of the question.

[0230] As described above, in Example 2, a paragraph mark '¶' is used as a boundary symbol. This symbol is called a pilcrow in English. The pilcrow belongs to the punctuation mark of the Unicode character category, is included in the vocabulary of multilingual BERT, and rarely appears in normal text. Therefore, in Example 2, the pilcrow is used as a boundary symbol that separates the question from the context. Any character or character string that satisfies similar properties may be used as the boundary symbol.

[0231] In addition, the word alignment data contains many null alignments (no alignments). Therefore, in Example 2, we use the formulation of SQuADv2.0

[17] . The difference between SQuADv1.1 and SQuADv2.0 is that we explicitly handle the possibility that the answer to the question does not exist in the context.

[0232] In other words, the SQuADV2.0 format explicitly indicates that a question that cannot be answered cannot be answered, so it can generate appropriate questions and answers (unable to answer) for null alignments (no alignment) in the word alignment data.

[0233] Since the handling of tokenization including word splitting and casing differs depending on the word correspondence data, in the second embodiment, the token sequence of the source language sentence is used only for the purpose of creating questions.

[0234] When the cross-language span prediction question answer generator 312 converts the word correspondence data into the SQuAD format, it uses the original text, not the token string, for the question and context. That is, the cross-language span prediction question answer generator 312 generates, as an answer, the start and end positions of the span, together with the word or word string of the span from the target language sentence (context), and the start and end positions serve as indexes to the character positions of the original text of the target language sentence.

[0235] In the conventional word matching methods, a token string is often used as an input. In other words, in the example of the word matching data in Fig. 15, the first two pieces of data are often the input. In contrast, in the second embodiment, both the original text and the token string are input to the cross-language span prediction question answer generation unit 312, making the system flexible in dealing with any tokenization.

[0236] The data of pairs of cross-language span prediction questions (question and context) and answers generated by the cross-language span prediction question answer generation unit 312 is stored in the cross-language span prediction correct answer data storage unit 313.

[0237] --About the span prediction model learning unit 314-- The span prediction model training unit 314 trains the cross-language span prediction model using the correct answer data read from the cross-language span prediction correct answer data storage unit 313. That is, the span prediction model training unit 314 inputs a cross-language span prediction problem (question and context) to the cross-language span prediction model, and adjusts the parameters of the cross-language span prediction model so that the output of the cross-language span prediction model becomes the correct answer. This training is performed separately for cross-language span prediction from a first language sentence to a second language sentence, and cross-language span prediction from a second language sentence to a first language sentence.

[0238] The trained cross-language span prediction model is stored in the cross-language span prediction model storage unit 315. In addition, the word correspondence execution unit 320 reads out the cross-language span prediction model from the cross-language span prediction model storage unit 315 and inputs it to the span prediction unit 322.

[0239] The cross-lingual span prediction model is described in detail below, as well as the processing of the word matching implementer 320.

[0240] <Cross-language span prediction using multilingual BERT> As already described, the span prediction unit 322 of the word alignment execution unit 320 in the second embodiment generates word alignments from input sentence pairs using the cross-language span prediction model trained by the cross-language span prediction model training unit 310. That is, word alignments are generated by performing cross-language span prediction on the input sentence pairs.

[0241] --About the cross-linguistic span prediction model-- In Example 2, the task of cross-language span prediction is defined as follows.

[0242] Source text X=x of length |X| characters 1 x 2 ...x |X| and a target language sentence Y=y of length |Y| characters 1 y 2 ...y |Y| Let x be the source token from character position i to character position j in the source sentence. i:j =x i ...x j , the target span y from character position k to character position l in the target sentence k:l =y k ...y l The task of cross-linguistic span prediction is to extract

[0243] The span prediction unit 322 of the word correspondence execution unit 320 executes the above tasks using the cross-language span prediction model trained by the cross-language span prediction model training unit 310. In the second embodiment, the multilingual BERT [5] is also used as the cross-language span prediction model.

[0244] BERT also works very well for the cross-language task in Example 2. Note that the language model used in Example 2 is not limited to BERT.

[0245] More specifically, in Example 2, as an example, a model similar to the model for the SQuADv2.0 task disclosed in [5] is used as the cross-language span prediction model. These models (the model for the SQuADv2.0 task, the cross-language span prediction model) are models that add two independent output layers to predict the start and end positions in the context to the pre-trained BERT.

[0246] In the cross-language span prediction model, the probability that each position in the target sentence will be the start and end position of the answer span is p start and p end Let x be the source span. i:j Given the target language span y k:l Score ω X→Y ijkl is defined as the product of the probability of the starting position and the probability of the ending position, and (^k,^l) that maximizes this product is defined as the best answer span.

[0247]

number

[0248]

number

[0249] The cross-language span prediction model in Example 2 and the model for the SQuADv2.0 task disclosed in Reference [5] have basically the same neural network structure. However, the model for the SQuADv2.0 task uses a monolingual pre-trained language model and fine-tunes (additional learning / transfer learning / fine-tuning) it with training data for a task of predicting spans between the same language, whereas the cross-language span prediction model in Example 2 uses a pre-trained multilingual model including two languages ​​related to cross-language span prediction and fine-tunes it with training data for a task of predicting spans between two languages.

[0250] In addition, while the existing implementation of the SQuAD model of BERT only outputs the answer string, the cross-language span prediction model of the second embodiment is configured to be able to output the start position and the end position.

[0251] Within BERT, i.e., within the cross-lingual span prediction model of Example 2, the input sequence is first tokenized by a tokenizer (e.g., WordPiece), and then CJK characters (Chinese characters) are split into units of one character.

[0252] In the existing implementation of the SQuAD model of BERT, the start position and end position are indices to tokens inside BERT, but in the cross-lingual span prediction model of Example 2, they are indices to character positions. This makes it possible to handle tokens (words) of the input text for which word correspondence is sought and tokens inside BERT independently.

[0253] FIG. 17 shows a process of predicting a target language (Japanese) span as an answer from the context of a target language sentence (Japanese) for a token "Yoshimitsu" in a source language sentence (English) as a question, using the cross-language span prediction model of the second embodiment. As shown in FIG. 17, "Yoshimitsu" is composed of four BERT tokens. Note that a "##" (prefix) indicating a connection with the previous vocabulary is added to a BERT token, which is a token inside BERT. Also, the boundary of an input token is indicated by a dotted line. Note that in this embodiment, an "input token" and a "BERT token" are distinguished. The former is a unit of word segmentation in the training data, and is indicated by a dashed line in FIG. 17. The latter is a unit of segmentation used inside BERT, and is indicated by a space in FIG. 17.

[0254] In the example shown in FIG. 17, five possible answers are displayed: "Yoshimitsu", "Yoshimitsu (Ashikaga Yoshimitsu", "Ashikaga Yoshimitsu", "Yoshimitsu (", and "Yoshimitsu (Ashikaga Yoshi"); "Yoshimitsu" is the correct answer.

[0255] In BERT, a span is predicted for each token within BERT, so the predicted span does not necessarily match the boundaries of the input tokens (words). Therefore, in the second embodiment, for a target language span that does not match the token boundaries of the target language, such as "Yoshimitsu (Ashikagayoshi", a process is performed to match a target language word that is completely included in the predicted target language span, i.e., "Yoshimitsu", "(", and "Ashikaga" in this example, with a source language token (question). This process is performed only at the time of prediction, and is performed by the word correspondence generation unit 323. During learning, learning is performed based on a loss function that compares the first candidate for span prediction with the correct answer regarding the start and end positions.

[0256] --Cross-language span prediction question generation unit 321 and span prediction unit 322-- The cross-language span prediction question generator 321 creates a span prediction question in the form of "[CLS]question[SEP]context[SEP]" in which the question and context are linked for each of the input first and second language sentences, for each question (input token (word)), and outputs it to the span prediction unit 122. However, as described above, the question is a contextual question that uses ¶ as a boundary symbol, such as "Yoshimitsu ASHIKAGA ¶ was ¶ the 3rd Seii Taishogun of the Muromachi Shogunate and reigned from 1368 to1394."

[0257] The cross-language span prediction problem generation unit 321 generates a span prediction problem from a first language sentence (question) to a second language sentence (answer) and a span prediction problem from a second language sentence (question) to a first language sentence (answer).

[0258] The span prediction unit 322 inputs each question (question and context) generated by the cross-language span prediction question generation unit 121, calculates the answer (predicted span) and probability for each question, and outputs the answer (predicted span) and probability for each question to the word correspondence generation unit 323.

[0259] The above probability is the product of the probability of the start position and the probability of the end position in the best answer span. The process of the word correspondence generating unit 323 will be described below.

[0260] <Symmetrization of word correspondence> In span prediction using the cross-language span prediction model of Example 2, target language spans are predicted for source language tokens, so the source and target languages ​​are asymmetric, similar to the model described in Reference [1]. In Example 2, a method of symmetricalizing predictions in both directions is introduced to improve the reliability of word correspondence based on span prediction.

[0261] First, for reference, a conventional example of symmetrizing word correspondence will be described. A method of symmetrizing word correspondence based on the model described in reference [1] was first proposed in reference

[16] . A representative statistical translation toolkit, Moses

[11] , implements heuristics such as intersection, union, and grow-diag-final, with grow-diag-final being the default. The intersection of two word correspondences (intersection) has high precision and low recall. The union of two word correspondences (union) has low precision and high recall. Grow-diag-final is a method of finding word correspondence that is intermediate between the intersection and union.

[0262] --About the word correspondence generation unit 323-- In the second embodiment, the word alignment generation unit 323 averages the probability of the best span for each token in two directions, and if this averages a predetermined threshold or more, it is deemed to correspond. This process is performed by the word alignment generation unit 323 using the output from the span prediction unit 322 (cross-language span prediction model). As described with reference to FIG. 17, the predicted span output as an answer does not necessarily match the word boundary, so the word alignment generation unit 323 also performs a process of adjusting the predicted span so that it corresponds to a one-way word unit. The specific process of symmetrizing word alignment is as follows.

[0263] In sentence X, the span from start position i to end position j is x i:j In sentence Y, the span from start position k to end position l is called y k:l Let token x i:j is the span y k:l The probability of predicting X→Y ijkl Let token y k:l is span x i:j The probability of predicting Y→X ijkl Let token x i:j and token y k:l Corresponding a ijkl The probability of ijkl In this embodiment, ω ijkl , x i:j The best span y predicted from ^k:^l The probability of ω X→Y ij^k^l and,y k:l The best span predicted from x ^i:^j The probability of ω Y→X ^i^jkl Calculate as the average of.

[0264]

number

[0265] The symmetrization method used in the second embodiment is called bidirectional average (bidi-avg). Bidirectional average is easy to implement, and has the same effect as grow-diag-final in that it finds a word correspondence that is intermediate between a set sum and a set intersection. Note that using the average is just one example. For example, the probability ω X→Y ij^k^l and probability ω Y→X ^i^jkl Alternatively, a weighted average of these may be used.

[0266] FIG. 18 shows (a) Japanese to English span prediction and (b) English to Japanese span prediction symmetricalized by two-way averaging (c).

[0267] In the example of FIG. 18, for example, the probability ω of the best span “language” predicted from “language” X→Y ij^k^l is 0.8, and the probability of the best span "language" predicted from "language" is ω Y→X ^i^jkl is 0.6, and the average is 0.7. Since 0.7 is above the threshold, it can be determined that "lang" and "language" correspond to each other. Therefore, the word alignment generation unit 123 generates and outputs the word pair "lang" and "language" as one of the word alignment results.

[0268] In the example of FIG. 18, the word pair "is" and "de" are predicted only in one direction (from English to Japanese), but are considered to correspond because the two-way average probability is equal to or greater than the threshold.

[0269] The threshold of 0.4 was determined through a preliminary experiment in which the learning data for Japanese and English word correspondences, described below, was divided in half, with one half used as training data and the other as test data. This value was used in all experiments described below. Since span predictions for each direction are performed independently, it may be necessary to normalize the scores for symmetry, but in the experiment, both directions were trained with a single model, so normalization was not necessary.

[0270] (Example 2: Effects of the embodiment) The word matching device 300 described in the second embodiment does not require a large amount of bilingual data for the language pair to which word matching is to be assigned, and can achieve more accurate supervised word matching than in the past using a smaller amount of teacher data (manually created correct answer data) than in the past.

[0271] (Example 2: Experimental) In order to evaluate the technology according to the second embodiment, a word matching experiment was carried out, and the experimental method and results are described below.

[0272] <Example 2: Experimental Data> Figure 19 shows the number of sentences in the training data and test data for the gold word alignments created manually for five language pairs: Chinese-English (Zh-En), Japanese-English (Ja-En), German-English (De-En), Romanian-English (Ro-En), and English-French (En-Fr). The table in Figure 19 also shows the amount of reserved data.

[0273] In the experiment using the conventional technique

[20] , Zh-En data was used, and in the experiment using the conventional technique [9], De-En, Ro-En, and En-Fr data were used. In the experiment using the technique of this embodiment, Ja-En data, which is one of the most distant language pairs in the world, was added.

[0274] The Zh-En data was obtained from the GALE Chinese-English Parallel Aligned Treebank

[12] , and includes broadcasting news, news wires, and web data. In order to approximate the experimental conditions described in

[20] as closely as possible, we used bilingual texts in which Chinese characters were tokenized, cleaned to remove mismatches and timestamps, and randomly divided the data into 80% training data, 10% test data, and 10% reserve data.

[0275] The KFTT word correspondence data

[14] was used as the Japanese-English data. Kyoto Free Translation Task (KFTT) (http: / / www.phontron.com / kftt / index.html) is a manual translation of Japanese Wikipedia articles about Kyoto, and is composed of 440,000 sentences of training data, 1166 sentences of development data, and 1160 sentences of test data. The KFTT word correspondence data is a part of the KFTT development data and test data that has been manually assigned word correspondences, and consists of 8 development data files and 7 test data files. In the experiment of the technology according to this embodiment, 8 development data files were used for training, 4 of the test data files were used for testing, and the rest were reserved.

[0276] The De-En, Ro-En, and En-Fr data are described in

[27] . The authors have published scripts for preprocessing and evaluation (https: / / github.com / lilt / alignment-scripts). In the prior art [9], these data are used for experiments. The De-En data is described in

[24] (https: / / www-i6.informatik.rwth-aachen.de / goldAlignment / ). The Ro-En and En-Fr data were provided as common tasks in the HLT-NAACL-2003 workshop on Building and Using Parallel Texts

[13] (https: / / eecs.engin.umich.edu / ). The En-Fr data was originally described in

[15] . The number of sentences in the De-En, Ro-En, and En-Fr data is 508,248,447. In this embodiment, 300 sentences were used for training for De-En and En-Fr, and 150 sentences were used for training for Ro-En. The remaining sentences were used for testing.

[0277] <Evaluation scale for word matching accuracy> In the second embodiment, the F1 score, which has equal weighting on precision and recall, is used as an evaluation measure for word correspondence.

[0278]

number

[16] , we also use the AER to compare the prior art with the technique of this embodiment.

[0279] Let a manually created gold word alignment consist of sure,S,and possible,P,, where S ⊆ P. We define the precision, recall, and AER of a word alignment,A,as follows:

[0280]

number

[0281]

number

[0282]

number

[0283] <Comparison of word matching accuracy> 20 shows a comparison between the technology according to Example 2 and the conventional technology. For all five data, the technology according to Example 2 is superior to all the conventional technologies.

[0284] For example, on the Zh-En data, the technique according to Example 2 achieves an F1 score of 86.7, which is 13.3 points higher than the F1 score of 73.4 of DiscAlign reported in

[20] , which is the current state-of-the-art accuracy of word alignment by supervised learning. The method in

[20] uses 4 million sentence-pairs of bilingual data to pre-train the translation model, whereas the technique according to Example 2 does not require bilingual data for pre-training. On the Ja-En data, Example 2 achieves an F1 score of 77.6, which is 20 points higher than the F1 score of 57.8 of GIZA++.

[0285] For the De-EN, Ro-EN, and En-Fr data, the method that currently achieves the highest accuracy in unsupervised word matching reports only AER [9], so this embodiment also evaluates AER. For comparison, we also report the AER of MGIZA and other conventional methods for the same data [22, 10].

[0286] In the experiment, both sure and possible word correspondences for the De-En data were used for training the present embodiment, but only sure was used for the En-Fr data because it was very noisy. The AER of the present embodiment for the De-En, Ro-En, and En-Fr data was 11.4, 12.2, and 4.0, which is clearly lower than the method in reference [9].

[0287] Comparing the accuracy of supervised learning with that of unsupervised learning is clearly an unfair way of evaluating machine learning. The purpose of this experiment is to show that supervised word alignment is a practical method for achieving high accuracy, since it can achieve an accuracy higher than the best accuracy reported so far using a smaller amount of correct answer data (about 150 to 300 sentences) than the correct answer data originally created by hand for evaluation.

[0288] Example 2: Effect of symmetrization In order to show the effectiveness of the bidirectional average (bidi-avg), which is the symmetrization method in the second embodiment, FIG. 21 shows the word correspondence accuracy of two-way prediction, set intersection, set union, grow-diag-final, and bidi-avg. Alignment Word correspondence accuracy is greatly affected by the orthography of the target language. In languages ​​that do not have spaces between words, such as Japanese and Chinese, the accuracy of span prediction to English is significantly higher than the accuracy of span prediction from English. In such cases, grow-diag-final is better than bidi-avg. On the other hand, in languages ​​that have spaces between words, such as German, Romanian, and French, there is no significant difference between span prediction to English and span prediction from English, and grow-diag-final is better than bidi-avg. In En-Fr data, set intersection has the highest accuracy, but this is thought to be because the data is originally noisy.

[0289] <The Importance of Source Language Context> Figure 22 shows the change in word matching accuracy when the size of the source word context is changed. We used the Ja-En data. We can see that the source word context is very important in predicting the target span.

[0290] Without context, the F1 score of Example 2 is 59.3, which is slightly higher than the F1 score of 57.6 of GIZA++. However, if only two words of context are provided, the score becomes 72.0, and if the entire sentence is provided as context, the score becomes 77.6.

[0291] Learning curve Figure 23 shows the learning curve of the word matching method of Example 2 when using Zh-En data. Naturally, the more training data there is, the higher the accuracy, but even with a small amount of training data, the accuracy is higher than that of conventional supervised learning methods. The F1 score of 79.6 for the technology according to this embodiment when training data is 300 sentences is 6.2 points higher than the F1 score of 73.4 for the method in literature

[20] , which currently has the highest accuracy, when training using 4,800 sentences.

[0292] (Summary of Example 2) As described above, in the second embodiment, the problem of determining word correspondence between two sentences which are translations of each other is regarded as a set of problems (cross-language span prediction) of independently predicting words or consecutive word strings (spans) in a sentence in one language which correspond to each word in a sentence in another language, and highly accurate word correspondence is achieved by learning (supervised learning) a cross-language span predictor using a neural network from a small amount of manually created correct answer data.

[0293] The cross-language span prediction model is created by fine-tuning a pre-trained multilingual model created using only monolingual texts for each of multiple languages, using a small amount of manually created correct answer data. Compared to conventional methods based on machine translation models such as Transformer, which require millions of bilingual sentence pairs for pre-training a translation model, the technology according to this embodiment can be applied to language pairs or areas with a small amount of available bilingual sentences.

[0294] In the second embodiment, if the manually created correct answer data is about 300 sentences, it is possible to achieve a word matching accuracy that exceeds that of conventional supervised learning and unsupervised learning. According to literature

[20] , correct answer data of about 300 sentences can be created in a few hours, so this embodiment can obtain highly accurate word matching at a reasonable cost.

[0295] In addition, in Example 2, by converting the word correspondence into a general-purpose problem, the cross-lingual span prediction task in the SQuADv2.0 format, it is possible to easily incorporate multilingual pre-trained models and cutting-edge technologies in question answering to improve performance. For example, it is possible to use XLM-RoBERTa[2] to create a model with higher accuracy, or distilmBERT

[19] to create a compact model that operates with fewer computer resources.

[0296] [References for Example 2] [1] Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. The Mathematics of Statistical Machine Translation: Parameter Estimation. Computational Linguistics,Vol. 19, No. 2, pp. 263-311, 1993. [2] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm´an, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised Cross-lingual Representation Learning at Scale. arXiv:1911.02116, 2019. [3] Alexis Conneau and Guillaume Lample. Cross-lingual Language Model Pretraining. In Proceedings of NeurIPS-2019, pp. 7059-7069, 2019. [4] John DeNero and Dan Klein. The Complexity of Phrase Alignment Problems. In Proceedings of the ACL-2008, pp. 25-28, 2008. [5] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the NAACL-2019, pp. 4171-4186, 2019. [6] Chris Dyer, Victor Chahuneau, and Noah A. Smith. A Simple, Fast, and Effective Reparameterization of IBM Model 2. In Proceedings of the NAACL-HLT-2013, pp. 644-648, 2013. [7] Alexander Fraser and Daniel Marcu. MeasuringWord Alignment Quality for Statistical Machine Translation. Computational Linguistics, Vol. 33, No. 3, pp. 293-303, 2007. [8] Qin Gao and Stephan Vogel. Parallel Implementations of Word Alignment Tool. In Proceedings of ACL 2008 workshop on Software Engineering, Testing, and Quality Assurance for Natural Language Processing, pp. 49-57, 2008. [9] Sarthak Garg, Stephan Peitz, Udhyakumar Nallasamy, and Matthias Paulik. Jointly Learning to Align and Translate with Transformer Models. In Proceedings of the EMNLP-IJCNLP-2019, pp.4452-4461, 2019.

[10] Aria Haghighi, John Blitzer, John DeNero, and Dan Klein. Better Word Alignments with Supervised ITG Models. In Proceedings of the ACL-2009, pp. 923-931, 2009.

[11] Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. Moses: Open Source Toolkit for Statistical Machine Translation. In Proceedings of the ACL-2007, pp. 177-180, 2007.

[12] Xuansong Li, Stephen Grimes, Stephanie Strassel, Xiaoyi Ma, Nianwen Xue, Mitch Marcus, and Ann Taylor. GALE Chinese-English Parallel Aligned Treebank - Training. Web Download, 2015. LDC2015T06.

[13] Rada Mihalcea and Ted Pedersen. An Evaluation Exercise for Word Alignment. In Proceedings of the HLT-NAACL 2003 Workshop on Building and Using Parallel Texts: Data Driven Machine Translation and Beyond, pp. 1-10, 2003.

[14] Graham Neubig. Kyoto Free Translation Task alignment data package. http: / / www.phontron.com / kftt / , 2011.

[15] Franz Josef Och and Hermann Ney. Improved Statistical Alignment Models. In Proceedings of ACL-2000, pp. 440-447, 2000.

[16] Franz Josef Och and Hermann Ney. A Systematic Comparison of Various Statistical Alignment Models. Computational Linguistics, Vol. 29, No. 1, pp. 19-51, 2003.

[17] Pranav Rajpurkar, Robin Jia, and Percy Liang. Know What You Don't Know: Unanswerable Questions for SQuAD. In Proceedings of the ACL-2018, pp. 784-789, 2018.

[18] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Proceedings of EMNLP-2016, pp. 2383-2392, 2016.

[19] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv:1910.01108, 2019.

[20] Elias Stengel-Eskin, Tzu ray Su, Matt Post, and Benjamin Van Durme. A Discriminative Neural Model for Cross-Lingual Word Alignment. In Proceedings of the EMNLP-IJCNLP-2019, pp. 910-920, 2019.

[21] Akihiro Tamura, Taro Watanabe, and Eiichiro Sumita. Recurrent Neural Networks for Word Alignment Model. In Proceedings of the ACL-2014, pp. 1470-1480, 2014.

[22] Ben Taskar, Simon Lacoste-Julien, and Dan Klein. A Discriminative Matching Approach to Word Alignment. In Proceedings of the HLT-EMNLP-2005, pp. 73-80, 2005.

[23] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In Proceedings of the NIPS 2017, pp. 5998-6008, 2017.

[24] David Vilar, Maja Popovi´c, and Hermann Ney. AER: Do we need to "improve" our alignments? In Proceedings of IWSLT-2006, pp. 2005-212, 2006.

[25] Stephan Vogel, Hermann Ney, and Christoph Tillmann. HMM-Based Word Alignment in Statistical Translation. In Proceedings of COLING-1996, 1996.

[26] Nan Yang, Shujie Liu, Mu Li, Ming Zhou, and Nenghai Yu. Word Alignment Modeling with Context Dependent Deep Neural Network. In Proceedings of the ACL-2013, pp. 166-175, 2013.

[27] Thomas Zenkel, Joern Wuebker, and John DeNero. Adding Interpretable Attention to Neural Translation Models Improves Word Alignment. arXiv:1901.11359, 2019. (Appendix) This specification discloses at least the corresponding device, learning device, corresponding method, program, and storage medium of the following appended items. Note that in appended items 1, 6, and 10 below, "using a span prediction model created using data consisting of a cross-domain span prediction problem and its answer, predicting a span that is an answer to the span prediction problem," "consisting of a cross-domain span prediction problem and its answer" relates to "data," and "created using ... data" relates to "span prediction model." (Additional note 1) Memory, at least one processor coupled to the memory; Including, The processor, A span prediction problem between the first domain sequence information and the second domain sequence information is generated by inputting the first domain sequence information and the second domain sequence information; A span prediction model is created using data consisting of a cross-domain span prediction problem and its answer, and a span that is an answer to the span prediction problem is predicted using the span prediction model. Compatible devices. (Additional note 2) The span prediction model is a model obtained by additionally training a pre-trained model using the data. Item 1. (Additional note 3) the series information in the first domain series information and the second domain series information is a document; The processor determines whether a sentence set of the first span corresponds to a sentence set of the second span based on a probability of predicting a second span by a question of a first span in span prediction from the first domain series information to the second domain series information, and a probability of predicting the first span by a question of the second span in span prediction from the second domain series information to the first domain series information. 3. A device according to claim 1 or 2. (Additional note 4) The processor generates correspondence between sentence sets between the first domain series information and the second domain series information by solving an integer linear programming problem such that a sum of costs of correspondence between sentence sets between the first domain series information and the second domain series information is minimized. Item 3. (Additional note 5) Memory, at least one processor coupled to the memory; Including, The processor, generating data having a span prediction problem and an answer thereto from the correspondence data having the first domain sequence information and the second domain sequence information; Using the data, generate a span prediction model Learning device. (Additional note 6) The computer a problem generation step of receiving first domain series information and second domain series information and generating a span prediction problem between the first domain series information and the second domain series information; a span prediction step of predicting a span that is an answer to the span prediction problem using a span prediction model created using data consisting of a cross-domain span prediction problem and its answer; How to deal with this issue. (Additional note 7) The computer a question answer generating step of generating data including a span prediction question and its answer from the correspondence data including the first domain sequence information and the second domain sequence information; A learning step of generating a span prediction model using the data. How to learn to do this. (Additional note 8) A program for causing a computer to function as a corresponding device according to any one of claims 1 to 4. (Additional note 9) A program for causing a computer to function as the learning device described in appended claim 5. (Additional note 10) A non-transitory storage medium storing a program executable by a computer to execute a corresponding process, The corresponding process includes: A span prediction problem between the first domain sequence information and the second domain sequence information is generated by inputting the first domain sequence information and the second domain sequence information; A span prediction model is created using data consisting of a cross-domain span prediction problem and its answer, and a span that is an answer to the span prediction problem is predicted using the span prediction model. Non-transitory storage medium. (Additional note 11) A non-transitory storage medium storing a program executable by a computer to execute a learning process, The learning process includes: generating data having a span prediction problem and an answer thereto from the correspondence data having the first domain sequence information and the second domain sequence information; Using the data, generate a span prediction model Non-transitory storage medium.

[0297] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and variations are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]

[0298] 100 sentence compatible device 110 Cross-Lingual Span Prediction Model Training Unit 111 Sentence-related data storage unit 112 Sentence matching generation unit 113 Sentence-based pseudo-correct data storage 114 Cross-Language Span Prediction Question Answer Generation Unit 115 Cross-Lingual Span Prediction Pseudo-Correct Answer Data Storage 116 Span Prediction Model Learning Unit 117 Cross-Lingual Span Prediction Model Storage 120 Statement-based execution unit 121 Monolingual Span Prediction Problem Generation Unit 122 Span Prediction Unit 123 Sentence matching generation unit 200 Pre-training device 210 Multilingual Data Storage Unit 220 Multilingual Model Learning Unit 230 Pre-trained multilingual model storage 300 Word Compatible Device 310 Cross-Lingual Span Prediction Model Training Unit 311 Word Correspondence Correct Answer Data Storage Unit 312 Cross-Language Span Prediction Question Answer Generation Unit 313 Cross-Language Span Prediction Correct Answer Data Storage 314 Span Prediction Model Learning Unit 315 Cross-Lingual Span Prediction Model Storage 320 Word Correspondence Execution Unit 321 Monolingual Span Prediction Problem Generation Unit 322 Span Prediction Unit 323 Word Correspondence Generation Unit 400 Pre-training device 410 Multilingual Data Storage Unit 420 Multilingual Model Learning Unit 430 Pre-trained multilingual model storage 1000 Drive Device 1001 Recording media 1002 Auxiliary storage 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input Device

Claims

1. a question generator that receives a first language text and a second language text as input and generates a span prediction question between the first language text and the second language text; a span prediction unit that predicts a span that is an answer to the span prediction problem generated by the problem generation unit, using a span prediction model created using data consisting of a span prediction problem between a first language and a second language and its answer; A corresponding device comprising:

2. The span prediction model is a model obtained by additionally training a pre-trained model using the data. The device of claim 1 .

3. a correspondence generation unit that determines whether the text of the first span corresponds to the text of the second span based on a probability of predicting a second span from a question of a first span in span prediction from the first language text to the second language text and a probability of predicting the first span from a question of the second span in span prediction from the second language text to the first language text.

3. A device according to claim 1 or 2, comprising:

4. The correspondence generation unit generates text correspondence between the first language text and the second language text by solving an integer linear programming problem such that a sum of costs of text correspondence between the first language text and the second language text is minimized. A corresponding device according to claim 3.

5. a question and answer generating unit that generates data having span prediction questions and their answers from correspondence data indicating correspondence between spans included in the first language text and spans included in the second language text; A learning unit that generates a span prediction model using the data; A learning device comprising:

6. A response method executed by a response device, comprising: a problem generation step of generating a span prediction problem between a first language text and a second language text by using the first language text and the second language text as input; a span prediction step of predicting a span that is an answer to the span prediction problem generated in the problem generation step, using a span prediction model created using data consisting of a span prediction problem between a first language and a second language and its answer; A method of response that includes the following.

7. A learning method executed by a learning device, comprising: a question answer generating step of generating data having span prediction questions and their answers from correspondence data indicating correspondence between spans included in the first language text and spans included in the second language text; A learning step of generating a span prediction model using the data. A learning method that provides:

8. A program for causing a computer to function as each unit in the corresponding device according to any one of claims 1 to 4, or a program for causing a computer to function as each unit in the learning device according to claim 5.

Images