Response device, learning device, response method, learning method, and program

Abstract

To accurately perform corresponding processing for identifying a pair of mutually corresponding information in two pieces of sequence information.SOLUTION: A corresponding device includes: a question generation unit configured to receive a first language text and a second language text as input and generate a span prediction problem between the first language text and the second language text; and a span prediction unit configured to predict a span to be an answer to the span prediction problem generated by the question generation unit, using a span prediction model created using data consisting of span prediction problems between first and second languages and their answers.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a technique for identifying pairs of sentence sets (one or more sentences) that correspond to each other in two documents that are in a corresponding relationship to each other. [Background technology]

[0002] Identifying pairs of corresponding sentences in two documents is called sentence alignment. A sentence alignment system generally consists of a mechanism for calculating the similarity score between sentences in two documents, and a mechanism for identifying sentence alignments for the entire document from the candidates and their scores obtained by that mechanism. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Brian Thompson and Philipp Koehn. Vecalign: Improved sentence alignment in linear time and space. In Proceedings of EMNLP-2019, pp. 1342-1348, 2019. Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional technologies for matching sentences do not use contextual information when calculating the similarity between sentences. Furthermore, in recent years, methods for calculating similarity using vector representations of sentences using neural networks have achieved high accuracy, but this method cannot effectively utilize word-level information because it converts each sentence into a single vector representation. This results in poor accuracy.

[0005] In other words, conventional techniques have not been able to accurately identify pairs of corresponding sentence sets in two documents that are in a corresponding relationship. Note that this issue can also arise in sequential information, not just documents.

[0006] The present invention has been made in consideration of the above points, and aims to provide a technique that enables accurate correspondence processing to identify pairs of information that correspond to each other in two pieces of sequence information. [Means for solving the problem]

[0007] According to the disclosed technology, there is provided a problem generator that receives a first language text and a second language text as input and generates a span prediction problem between the first language text and the second language text; a span prediction unit that predicts a span that will be an answer to the span prediction question generated by the question generation unit using a span prediction model created using data consisting of span prediction questions between a first language and a second language and their answers; A corresponding device is provided, comprising: [Effects of the Invention]

[0008] The disclosed technology provides a technology that enables accurate correspondence processing to identify pairs of information that correspond to each other in two pieces of sequence information. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating the configuration of an apparatus according to a first embodiment. [Figure 2] 10 is a flowchart showing the overall flow of processing. [Figure 3] 10 is a flowchart illustrating a process for training a cross-language span prediction model. [Figure 4] 10 is a flowchart showing a process of generating sentence alignment. [Figure 5] FIG. 2 is a hardware configuration diagram of the device. [Figure 6]FIG. 10 is a diagram illustrating an example of sentence correspondence data. [Figure 7] FIG. 10 is a diagram showing the average number of sentences and tokens in each dataset. [Figure 8] FIG. 10 is a diagram showing the F1 score for the entire correspondence. [Figure 9] FIG. 10 is a diagram showing the accuracy of sentence alignment evaluated for each number of source and target language sentences in the alignment relationship. [Figure 10] FIG. 10 is a diagram showing the results of a comparison of translation accuracy when the number of bilingual text pairs used for training is changed. [Figure 11] FIG. 10 is a diagram illustrating the configuration of an apparatus according to a second embodiment. [Figure 12] 10 is a flowchart showing the overall flow of processing. [Figure 13] 10 is a flowchart illustrating a process for training a cross-language span prediction model. [Figure 14] 10 is a flowchart showing a process for generating word correspondences. [Figure 15] FIG. 10 is a diagram illustrating an example of word correspondence data. [Figure 16] FIG. 10 is a diagram showing an example of a question from English to Japanese. [Figure 17] FIG. 10 is a diagram illustrating an example of span prediction. [Figure 18] FIG. 10 is a diagram illustrating an example of symmetrization of word correspondence. [Figure 19] FIG. 10 is a diagram showing the number of data points used in an experiment. [Figure 20] FIG. 1 is a diagram illustrating a comparison between a conventional technique and a technique according to an embodiment. [Figure 21] FIG. 10 illustrates the effect of symmetrization. [Figure 22] FIG. 1 illustrates the importance of context for source language words. [Figure 23] FIG. 10 shows word alignment accuracy when trained using a subset of Chinese and English training data. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention (the present embodiment) will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0011] In the following, examples 1 and 2 will be described as the present embodiment. In examples 1 and 2, alignment is mainly described using text pairs between different languages ​​as an example, but this is just an example, and the present invention is not limited to alignment of text pairs between different languages, but can also be applied to alignment of text pairs in the same language between different domains. An example of alignment of text pairs in the same language is alignment of colloquial sentences / words with business sentences / words.

[0012] Since language is also a type of "domain," alignment of text pairs between different languages ​​is an example of alignment of text pairs between different domains.

[0013] Furthermore, a sentence, a document, or a paragraph is a sequence of tokens, and may be called sequence information. In this specification, the number of sentences that are elements of a "sentence set" may be multiple or may be one.

[0014] Example 1 First, a first embodiment will be described. In the first embodiment, the problem of identifying sentence alignment is regarded as a set of problems (cross-language span prediction) of independently predicting a set of consecutive sentences (span) in a document in one language that corresponds to a set of consecutive sentences in a document in another language. A cross-language span prediction model is learned using a neural network from pseudo-correct answer data created by an existing method, and the prediction results are subjected to mathematical optimization within the framework of a linear programming problem, thereby realizing highly accurate sentence alignment. Specifically, a sentence alignment device 100, which will be described later, executes the processing related to this sentence alignment. More specifically, the linear programming used in the first embodiment is integer linear programming. Unless otherwise specified, the term "linear programming" used in the first embodiment means "integer linear programming."

[0015] In the following, first, a reference technique related to sentence matching will be described to facilitate understanding of the technique according to the first embodiment, followed by a description of the configuration and operation of the sentence matching device 100 according to the first embodiment.

[0016] The numbers and names of reference documents related to the reference techniques of Example 1 are listed at the end of Example 1. In the following description, the numbers of related reference documents are indicated as "[1]" etc.

[0017] (Example 1: Description of the Reference Technology)

[0018] As mentioned above, a sentence alignment system generally consists of a mechanism for calculating the similarity score between sentences in two documents, and a mechanism for identifying sentence alignments for the entire document from the candidates for sentence alignment obtained by this mechanism and their scores.

[0019] Regarding the former mechanism, conventional methods use similarity measures that do not take context into account, such as those based on sentence length [1], bilingual dictionaries [2, 3, 4], machine translation systems [5], and multilingual sentence vectors [6] (see Non-Patent Document 1 mentioned above). For example, Thompson et al. [6] proposed a method to obtain language-independent multilingual sentence vectors using a method called LASER, and calculate the similarity score of sentences from the cosine similarity between those vectors.

[0020] Regarding the latter mechanism for identifying sentence correspondences across an entire document, a dynamic programming (DP) method that assumes monotonicity of sentence correspondences is used in many conventional techniques, such as those by Thompson et al. [6] and Uchiyama et al. [3].

[0021] Uchiyama et al. [3] proposed a sentence alignment method that takes into account document alignment scores. In this method, a document in one language is translated into another using a bilingual dictionary, and the documents are aligned based on BM25 [7]. Next, sentence alignment is performed using the resulting document pairs based on the sentence similarity (SIM) and DP alignment. SIM is defined based on the relative frequency of words that correspond one-to-one between the two documents in the bilingual dictionary. Furthermore, the average SIM of the sentence alignments in the corresponding documents is used as a score representing the reliability of the document alignment, AVSIM, and the product of SIM and AVSIM is used as the final sentence alignment score. This allows for robust sentence alignment even when document alignment is not very accurate. This method is commonly used for sentence alignment between English and Japanese.

[0022] (Example 1: Issues) The conventional techniques described above do not use contextual information when calculating the similarity between sentences. Furthermore, in recent years, methods that calculate similarity using vector representations of sentences using neural networks have achieved high accuracy. However, these methods cannot effectively utilize word-level information because they convert sentences into a single vector representation. This can result in a loss of accuracy in matching sentences.

[0023] Furthermore, most conventional techniques perform global optimization using dynamic programming, which assumes monotonicity of correspondences. However, not all sentence correspondences in actual bilingual documents are monotonic. In particular, legal documents are known to contain non-monotonic sentence correspondences, and conventional techniques suffer from a loss of accuracy when used on such documents.

[0024] A technique that solves the above problems and enables highly accurate sentence matching will be described below as a first embodiment.

[0025] (Overview of the technology according to the first embodiment) In Example 1, sentence alignment is first converted into a cross-language span prediction problem. Cross-language span prediction is achieved by fine-tuning a multilingual language model pre-trained using monolingual data for at least the pair of languages ​​being addressed, using pseudo-correct sentence alignment data created using existing methods. In this case, a sentence from one document and another document are input to the model, allowing the context before and after the span to be taken into account during prediction. In addition, by using a multilingual language model that employs a structure known as self-attention, word-level information can be utilized.

[0026] Next, to identify correspondences that are consistent across the entire document, the scores of candidates for sentence alignment obtained by span prediction are symmetrized, and then global optimization is performed using linear programming. This improves the reliability of the results of asymmetric cross-language span prediction, making it possible to identify non-monotonic sentence alignments. Using this method, Example 1 achieves highly accurate sentence alignment.

[0027] (Device configuration example) FIG. 1 shows a sentence matching device 100 and a pre-training device 200 according to the first embodiment. The sentence matching device 100 is a device that executes sentence matching processing using the technology according to the first embodiment. The pre-training device 200 is a device that learns a multilingual model from multilingual data. Note that the sentence matching device 100 and a word matching device 300, which will be described later, may both be called "matching devices."

[0028] As shown in FIG. 1, the sentence matching device 100 includes a cross-language span prediction model training unit 110 and a sentence matching execution unit 120.

[0029] The cross-language span prediction model training unit 110 includes a document correspondence data storage unit 111, a sentence correspondence generation unit 112, a sentence correspondence pseudo-correct answer data storage unit 113, a cross-language span prediction question answer generation unit 114, a cross-language span prediction pseudo-correct answer data storage unit 115, a span prediction model training unit 116, and a cross-language span prediction model storage unit 117. The cross-language span prediction question answer generation unit 114 may also be called a question answer generation unit.

[0030] The sentence correspondence execution unit 120 has a cross-language span prediction question generation unit 121, a span prediction unit 122, and a sentence correspondence generation unit 123. The cross-language span prediction question generation unit 121 may also be called a question generation unit.

[0031] The pre-training device 200 is a device based on existing technology. The pre-training device 200 includes a multilingual data storage unit 210, a multilingual model training unit 220, and a pre-trained multilingual model storage unit 230. The multilingual model training unit 220 reads monolingual text in at least two target languages ​​or domains for which sentence correspondence is to be obtained from the multilingual data storage unit 210, thereby training a language model, and stores the language model in the pre-trained multilingual model storage unit 230 as a pre-trained multilingual model.

[0032] In the first embodiment, a pre-trained multilingual model trained by some means is input to the cross-language span prediction model training unit 110. Therefore, the pre-training device 200 may not be provided, and a general-purpose pre-trained multilingual model that is publicly available may be used, for example.

[0033] The pre-trained multilingual model in Example 1 is a language model that has been trained in advance using at least monolingual text in each language for which sentence correspondence is sought. In this embodiment, XLM-RoBERTa is used as the language model, but this is not limited to this. Any pre-trained multilingual model that can make predictions for multilingual text taking into account word-level information and contextual information, such as multilingual BERT, may be used. Furthermore, although this model is called a "multilingual model" because it can support multiple languages, training in multiple languages ​​is not essential; for example, pre-training may be performed using text from multiple domains in the same language.

[0034] The sentence matching device 100 may also be called a learning device. The sentence matching device 100 may also be equipped with the sentence matching execution unit 120 without the cross-language span prediction model training unit 110. An apparatus equipped solely with the cross-language span prediction model training unit 110 may also be called a learning device.

[0035] (Outline of operation of sentence correspondence device 100) 2 is a flowchart showing the overall operation of the sentence matching device 100. In S100, a pre-trained multilingual model is input to the cross-language span prediction model training unit 110, and the cross-language span prediction model training unit 110 trains a cross-language span prediction model based on the pre-trained multilingual model.

[0036] In S200, the cross-language span prediction model trained in S100 is input to the sentence alignment execution unit 120, which uses the cross-language span prediction model to generate and output sentence alignments for the input document pair.

[0037] <s100> The process of training the cross-language span prediction model in S100 will be described with reference to the flowchart in Fig. 3. The flowchart in Fig. 3 is premised on the assumption that a pre-trained multilingual model has already been input and stored in the storage device of the cross-language span prediction model training unit 110. It is also assumed that sentence-associated pseudo-superficial data is stored in the sentence-associated pseudo-superficial data storage unit 111.

[0038] In S101, the cross-language span prediction question answer generation unit 114 reads sentence-corresponding pseudo-correct answer data from the sentence-corresponding pseudo-correct answer data storage unit 113, and generates cross-language span prediction pseudo-correct answer data from the read sentence-corresponding pseudo-correct answer data, i.e., a pair of a cross-language span prediction question and its pseudo answer, and stores it in the cross-language span prediction pseudo-correct answer data storage unit 113.

[0039] Here, when finding sentence correspondence between a first language and a second language, the pseudo-correct answer data for sentence correspondence includes a document in the first language, a corresponding document in the second language, and data indicating correspondence between a set of sentences in the first language and a set of sentences in the second language. For example, when a document in the first language is (sentence 1, sentence 2, sentence 3, sentence 4) and a document in the second language is (sentence 5, sentence 6, sentence 7, sentence 8), the data indicating correspondence between a set of sentences in the first language and a set of sentences in the second language is data indicating correspondence such as (sentence 1, sentence 2) and (sentence 6, sentence 7) and (sentence 1, sentence 2) and (sentence 5, sentence 6).

[0040] As described above, pseudo-supervisory data for sentence alignment is used in Example 1. The pseudo-supervisory data for sentence alignment is obtained by using an existing method to align document pairs that have been aligned manually or automatically.

[0041] In the configuration example shown in FIG. 1, data on document pairs that have been manually or automatically aligned is stored in the document alignment data storage unit 111. This data is document alignment data written in the same language (or domain) as the document pair for which sentence alignment is sought. From this document alignment data, the sentence alignment generation unit 112 generates pseudo-correct sentence alignment data using an existing method. More specifically, sentence alignment is sought using the technology of Uchiyama et al. [3] described in the reference technology. In other words, sentence alignment is sought from the document pair using inter-sentence similarity called SIM and alignment using DP.

[0042] Note that manually created sentence-associated correct answer data may be used instead of the sentence-associated pseudo-correct answer data. Furthermore, the "pseudo-correct answer data" and the "correct answer data" may be collectively referred to as "correct answer data."

[0043] In S102, the span prediction model training unit 116 trains a cross-language span prediction model from the cross-language span prediction pseudo-ground-truth data and the pre-trained multilingual model, and stores the trained cross-language span prediction model in the cross-language span prediction model storage unit 117.

[0044] <s200> Next, the process of generating sentence alignment in S200 will be described with reference to the flowchart in Fig. 4. Here, it is assumed that the cross-language span prediction model has already been input to the span prediction unit 122 and stored in the storage device of the span prediction unit 122.

[0045] In S201, a document pair is input to the cross-language span prediction question generation unit 121. In S202, the cross-language span prediction question generation unit 121 generates a cross-language span prediction question from the input document pair.

[0046] Next, in S203, the span prediction unit 122 uses the cross-language span prediction model to perform span prediction on the cross-language span prediction question generated in S202, and obtains an answer.

[0047] In S204, the sentence alignment generation unit 123 performs global optimization based on the answers to the cross-language span prediction questions obtained in S203, and generates sentence alignments. In S205, the sentence alignment generation unit 123 outputs the sentence alignments generated in S204.

[0048] It should be noted that the "model" in this embodiment is a neural network model, and specifically, is made up of weight parameters, functions, and the like.

[0049] (Example of hardware configuration) The sentence matching device and learning device in Example 1, and the word matching device and learning device in Example 2 (collectively referred to as "devices") can be realized by, for example, making a computer execute a program describing the processing content described in the present embodiment (Example 1, Example 2). Note that this "computer" may be a physical machine or a virtual machine on the cloud. When a virtual machine is used, the "hardware" described here is virtual hardware.

[0050] The above program can be recorded on a computer-readable recording medium (such as a portable memory) and stored or distributed. The above program can also be provided via a network such as the Internet or email.

[0051] Fig. 5 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 5 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected via a bus B.

[0052] A program for realizing processing on the computer is provided by a recording medium 1001 such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0053] The memory device 1003 reads and stores a program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes the functions related to the device in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network. The display device 1006 displays a GUI (Graphical User Interface) or the like according to the program. The input device 1007 is composed of a keyboard, mouse, buttons, a touch panel, or the like, and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0054] (Example 1: Explanation of specific processing contents) The processing contents of the sentence matching device 100 in the first embodiment will be described in more detail below.

[0055] <Formulation of span prediction from sentence alignment> In Example 1, sentence alignment is formulated as a cross-language span prediction problem similar to the SQuAD-style question answering task [8]. Therefore, we first explain the formulation from sentence alignment to span prediction using an example. In relation to the sentence alignment device 100, we mainly explain the cross-language span prediction model and its training in the cross-language span prediction model training unit 110.

[0056] A question-answering system performing a SQuAD-style question-answering task is given a "context," such as a paragraph selected from Wikipedia, and a "question," and the system predicts the "span" in the context as the "answer."

[0057] Similar to the above span prediction, the sentence matching execution unit 120 in the sentence matching device 100 of the first embodiment regards the target language document as a context and a set of sentences in the source language document as a question, and predicts the set of sentences in the target language document that are translations of the set of sentences in the source language document as the span of the target language document. For this prediction, the cross-language span prediction model of the first embodiment is used.

[0058] --About the cross-language span prediction question answer generation unit 114-- In the first embodiment, supervised learning of the cross-language span prediction model is performed in the cross-language span prediction model learning unit 110 of the sentence matching device 100, and correct answer data is required for the learning. In the first embodiment, the cross-language span prediction question answer generation unit 114 generates this correct answer data as pseudo correct answer data from the sentence-matched pseudo correct answer data.

[0059] Fig. 6 shows examples of cross-language span prediction questions and answers in Example 1. Fig. 6(a) shows a monolingual question-answering task in the SQuAD format, and Fig. 6(b) shows a sentence alignment task from bilingual documents.

[0060] The cross-language span prediction problem and answer shown in Figure 6(a) consists of a document, a question (Q), and its answer (A). The cross-language span prediction problem and answer shown in Figure 6(b) consists of an English document, a Japanese question (Q), and its answer (A).

[0061] As an example, if the target document pair is an English document and a Japanese document, the cross-language span prediction question answer generation unit 114 shown in Figure 1 generates multiple pairs of documents (contexts) and questions and answers as shown in Figure 6(b) from the sentence-corresponding pseudo-answer data.

[0062] As will be described later, in the first embodiment, the span prediction unit 122 of the sentence correspondence execution unit 120 uses a cross-language span prediction model to make predictions in each direction, from a first language document (question) to a second language document (answer), and from a second language document (question) to a first language document (answer). Therefore, when training the cross-language span prediction model, bidirectional pseudo-correct answer data may be generated to perform bidirectional training so that predictions can be made in this way.

[0063] Note that the above-described bidirectional prediction is just one example. It is also possible to perform one-way prediction, such as prediction from a first language document (question) to a second language document (answer), or prediction from a second language document (question) to a first language document (answer).

[0064] --Definition of the cross-lingual span prediction problem-- The definition of the cross-language span prediction problem in Example 1 will be explained in more detail. Let F be a source language document consisting of tokens of length N, and let F={f1, f2, ..., f N }, and let E be a target language document consisting of tokens of length M: E={e1, e2, ..., e M }

[0065] The cross-language span prediction problem in Example 1 is to solve a source language sentence Q={f i ,f i+1 ,...,f j }, the target text R={e k ,e k+1 ,...,e l }. The "source language sentence Q" may be one sentence or multiple sentences.

[0066] The sentence alignment in Example 1 allows for alignment not only between one sentence and another, but also between multiple sentences. In Example 1, by inputting any consecutive sentences in a source language document as source language sentence Q, it is possible to handle one-to-one and many-to-many alignments in the same framework.

[0067] --About the span prediction model learning unit 116-- The span prediction model training unit 116 trains the cross-language span prediction model using the pseudo-correct answer data read from the cross-language span prediction pseudo-correct answer data storage unit 115. That is, the span prediction model training unit 116 inputs a cross-language span prediction problem (question and context) into the cross-language span prediction model, and adjusts the parameters of the cross-language span prediction model so that the output of the cross-language span prediction model becomes a correct answer (pseudo-correct answer). This parameter adjustment can be performed using existing technology.

[0068] The trained cross-language span prediction model is stored in the cross-language span prediction model storage unit 117. Furthermore, the sentence correspondence execution unit 120 reads the cross-language span prediction model from the cross-language span prediction model storage unit 117 and inputs it to the span prediction unit 122.

[0069] --About the pre-trained model BERT-- Here, we will explain the pre-trained model BERT, which is expected to be used as the pre-trained multilingual model in Example 1. BERT [9] is a language representation model that uses a Transformer-based encoder to output a word embedding vector for each word in an input sequence, taking into account the surrounding context. Typically, the input sequence is one sentence or two sentences concatenated with a special symbol in between.

[0070] BERT pre-trains a language representation model from large-scale language data using two tasks: a masked language model that predicts masked words in an input sequence both forward and backward, and a next-sentence prediction task that determines whether two given sentences are adjacent. By using these pre-training tasks, BERT can output word embedding vectors that capture features related to linguistic phenomena not only within a single sentence but also across two sentences. Note that language representation models such as BERT are sometimes simply called language models.

[0071] It has been reported that adding an appropriate output layer to a pre-trained BERT and fine-tuning it with the training data for the target task can achieve the highest accuracy in various tasks, such as text semantic similarity, natural language inference (textual entailment recognition), question answering, and named entity extraction. Note that fine-tuning as mentioned above means, for example, using the parameters of a pre-trained BERT as the initial values ​​for the target model (a model in which an appropriate output layer is added to BERT) to train the target model.

[0072] In tasks that use pairs of sentences as input, such as semantic text similarity, natural language inference, and question answering, BERT is given an input that combines two sentences using a special symbol, such as '[CLS] first sentence [SEP] second sentence [SEP]'. Here, [CLS] is a special token used to create a vector that aggregates information from the two input sentences, and is called a classification token, while [SEP] is a token that represents the boundary between sentences, and is called a separator token.

[0073] In a task such as question answering (QA), where two input sentences are given and one sentence is used to predict the span of the other sentence, BERT predicts from the vector output by [CLS] whether there is a span to be extracted in the other sentence, and from the vector output by BERT for each word in the other sentence, it predicts the probability that that word will be the starting point of the span to be extracted and the probability that that word will be the end point of the span to be extracted.

[0074] BERT was originally created for English, but now BERTs for various languages, including Japanese, have been created and made publicly available. In addition, a general-purpose multilingual model, multilingual BERT, was created using monolingual data for 104 languages ​​extracted from Wikipedia and is now publicly available.

[0075] Furthermore, a cross-language language model called XLM has been proposed, which is pre-trained using a gap-filling language model using bilingual texts. It has been reported to be more accurate than multilingual BERT in applications such as cross-language text classification, and pre-trained models are publicly available.

[0076] --About the cross-linguistic span prediction model-- The cross-language span prediction model in the first embodiment selects span (k, l) of target language text R corresponding to source language sentence Q from target language document E during both training and sentence matching execution.

[0077] The sentence alignment generation unit 123 (or the span prediction unit 122) of the sentence alignment execution unit 120 calculates an alignment score ω from span (i, j) of the source language sentence Q to span (k, l) of the target language text R. ijkl is calculated as follows using the product of the probability p1 of the start position and the probability p2 of the end position:

[0078]

number

[0079] In the cross-language span prediction model of the first embodiment, a source language sentence Q and a target language document E are combined and input as one piece of sequence data as follows:

[0080] [CLS] Source text Q [SEP] Target document E [SEP] The cross-language span prediction model in Example 1 is a model that adds two independent output layers to a pre-trained multilingual model and fine-tunes them using training data for the task of predicting spans between target and source documents. These output layers predict the probability p1 or p2 that each token position in the target document will be the start or end position of an answer span, respectively.

[0081] <About span prediction> Next, the operation of the sentence-corresponding execution unit 120 will be described in detail.

[0082] --Cross-language span prediction question generation unit 121 and span prediction unit 122-- The cross-language span prediction problem generation unit 121 creates a span prediction problem of the form "[CLS] source language sentence Q [SEP] target language document E [SEP]" for each input document pair (source language document and target language document) and outputs it to the span prediction unit 122.

[0083] As will be described later, in Example 1, bidirectional prediction is performed, so if a document pair is a first language document and a second language document, the cross-language span prediction problem generation unit 121 may generate a span prediction problem from the first language document (question) to the second language document (answer) and a span prediction problem from the second language document (question) to the first language document (answer).

[0084] The span prediction unit 122 inputs each question (question and context) generated by the cross-language span prediction question generation unit 121, calculates the answer (predicted span) and probabilities p1 and p2 for each question, and outputs the answer (predicted span) and probabilities p1 and p2 for each question to the sentence correspondence generation unit 123.

[0085] --About the sentence matching generation unit 123-- For example, the sentence alignment generation unit 123 calculates the best answer span (^k, ^l) for the source language sentence as an alignment score ω as follows: ijkl The sentence alignment generation unit 123 may output the result of this selection and the source language sentence as a sentence alignment.

[0086]

number

[0087] More specifically, in the first embodiment, the sentence alignment unit 123 calculates the no-alignment score φ using the value predicted at the position of "[CLS]". ij , and calculate the corresponding score of this score and the span ω ijkl Whether a corresponding target language text exists can be determined based on the magnitude of the difference. For example, the sentence alignment execution unit 120 may not use a source language sentence for which no corresponding target language text exists as a source language sentence for sentence alignment generation.

[0088] Here, the predicted value at the position of "[CLS]" is used to calculate the unpaired score φ ij "Calculating ω" essentially means calculating the corresponding score ω when the (start position, end position) of "[CLS]" in the sequence data input to the cross-linguistic span prediction model is considered as the answer span. ijkl Score φ ij This is equivalent to:

[0089] Although the answer span predicted by the cross-language span prediction model does not necessarily coincide with the sentence boundaries in the document, the prediction result needs to be converted into a sentence sequence for optimization and evaluation of sentence alignment. Therefore, in the first embodiment, the sentence alignment generation unit 123 finds the longest sentence sequence that is completely contained in the predicted answer span, and sets this sequence as the sentence-level prediction result.

[0090] --Optimization of prediction span by linear programming using the sentence correspondence generation unit 123-- Next, we will explain an example of a method for identifying many-to-many correspondences with high accuracy from the correspondence scores described above, which is executed by the sentence correspondence generation unit 123. Below, we will explain the issues with this method and the detailed processing of this method.

[0091] <Issues> When sentence alignments obtained by cross-language span prediction using a cross-language span prediction model (e.g., sentence alignments obtained by equation (2)) are directly used, the following problems arise.

[0092] · Because the cross-language span prediction model independently predicts the span of the target language text, there will be overlap in the spans of many predicted correspondences.

[0093] - Determining the span of the input source sentence is very important when identifying many-to-many correspondences, but it is not obvious how to select an appropriate span.

[0094] <Details of the correspondence identification method> To solve these problems, Example 1 introduces linear programming. Global optimization using linear programming ensures span consistency and maximizes the correspondence score across the entire document. Preliminary experiments showed that converting the score into a cost and minimizing that cost achieved higher accuracy than maximizing the score, so Example 1 formulates the problem as a minimization problem.

[0095] Furthermore, since the cross-language span prediction problem is asymmetric as it is, in Example 1, by swapping the source language document and the target language document and solving the same span prediction problem, a similar correspondence score ω' ijkl and the unmatched score φ´ kl The goal is to calculate the score for each word and obtain up to two predictions for the same correspondence. Symmetrical matching using both scores increases the reliability of the prediction results, which is expected to lead to improved accuracy in sentence alignment.

[0096] If the first language document is the source language document and the second language document is the target language document, the correspondence score from the span (i,j) of the source language sentence in the first language document to the span (k,l) of the target language text in the second language document is ω ijkl where the second language document is the source document and the first language document is the target document, and the correspondence score from the span (k,l) of the source sentence in the second language document to the span (i,j) of the target text in the first language document is ω´ ijkl Also, φ ij is the score indicating that there is no span in the second language document that corresponds to span (i,j) in the first language document, and φ kl is a score indicating that there is no span in the first language document that corresponds to span (k, l) in the second language document.

[0097] In this embodiment, ω ijkl and ω´ ijkl The symmetrical score in the form of a weighted average of the above is defined as follows:

[0098]

number

[0099] In the first embodiment, a sentence alignment is defined as a set of span pairs with no overlapping spans in each document, and the sentence alignment generation unit 123 identifies sentence alignment by solving the problem of finding a set that minimizes the sum of the costs of the alignment relationships using linear programming. The linear programming in the first embodiment is formulated as follows:

[0100]

number

[0101]

number

[0102]

number

[0103]

number

[0104] y ijkl is a binary variable that indicates whether spans (i,j) and (k,l) correspond to each other, and a value of 1 indicates that they correspond. ij ,b′ kl is a binary variable that indicates whether the spans (i,j) and (k,l) are unpaired, and when the value is 1, there is no pairing. ij b ij ,Σφ´ kl b´ kl Both of these are costs that increase as the number of unmatched cases increases.

[0105] Equation (6) is a constraint that ensures that each sentence in the source document appears in only one span pair in the correspondence. Equation (7) is a similar constraint for the target document. These two constraints ensure that there is no overlap of spans in each document, and that each sentence is associated with some correspondence, even if there is no correspondence.

[0106] In formula (6), any x corresponds to any source sentence. Formula (6) implies a constraint that for all spans containing any source sentence x, the sum of the correspondence between any target span and any of the spans and the pattern where x does not correspond is 1. The same is true for formula (7).

[0107] The cost of the correspondence, c ijkl is calculated from the score Ω as follows:

[0108]

number

[0109] Candidate spans of target text obtained when one source sentence is input, and their scores ω ijkl There are a number of candidates proportional to the square of the number of tokens in the target language document. Since attempting to calculate all of them as candidates would result in extremely high computational costs, in the first embodiment, only a small number of candidates with high scores for each source language sentence are used in the optimization calculation using linear programming. For example, N (N≧1) may be determined in advance, and the N candidates with the highest scores may be used for each source language sentence.

[0110] In preliminary experiments, increasing the number of candidates used for each input from one to several did not improve sentence alignment accuracy, so in the experiments described below, only the candidate with the highest score was used as the span candidate for each source sentence.

[0111] --- Filtering low-quality data taking document correspondence information into account --- When bilingual data extracted by sentence alignment is actually used in downstream tasks, low-quality bilingual sentences are often removed based on the sentence alignment score and cost. One of the reasons for these low-quality alignments is that the automatically extracted bilingual document alignments are sometimes incorrect and unreliable. However, the sentence alignment scores and costs described above do not take into account the accuracy of the document alignment.

[0112] Therefore, in the first embodiment, a document correspondence cost d is introduced, and the sentence correspondence generation unit 123 calculates the document correspondence cost d and the sentence correspondence cost c ijkl The document alignment cost d can be calculated as follows by dividing Equation (4) by the number of extracted sentence alignments:

[0113]

number

[0114] To remove low-quality bilingual sentences, for example, a document 1 in the first language and a document 2 in the second language are input to the sentence alignment execution unit 120, and the sentence alignment generation unit 123 obtains one or more bilingual sentence data that have been aligned. The sentence alignment generation unit 123, for example, selects d×c ijkl If d×c is larger than the threshold, it is judged to be of low quality and is not used (removed). ijkl Alternatively, only a certain number of bilingual sentence data items may be used in ascending order of the value of .

[0115] (Effects of Example 1) The sentence alignment device 100 described in the first embodiment can achieve sentence alignment with higher accuracy than conventional methods. Furthermore, the extracted bilingual sentences contribute to improving the translation accuracy of the machine translation model. Below, experiments on sentence alignment accuracy and machine translation accuracy that demonstrate these effects will be described. Below, the experiment on sentence alignment accuracy will be described as Experiment 1, and the experiment on machine translation accuracy will be described as Experiment 2.

[0116] <Experiment 1: Comparison of sentence alignment accuracy> We evaluated the accuracy of sentence alignment in Example 1 using actual bilingual Japanese and English newspaper articles. To confirm the difference in accuracy due to different optimization methods, we optimized the cross-language span prediction results using two methods: dynamic programming (DP) [1] and linear programming (ILP, the method in Example 1), and compared them. As baselines, we used the method by Thompson et al. [6], which has achieved the highest accuracy in various languages, and the method by Uchiyama et al. [3], which is the de facto standard method between Japanese and English.

[0117] The evaluation metric used was the F1 score, a common metric for sentence alignment. Specifically, we used the strict value in the script "https: / / github.com / thompsonb / vecalign / blob / master / score.py." This metric is calculated based on the number of exact matches between the correct and predicted alignments. However, even though automatically extracted bilingual documents contain unaligned sentences as noise, this metric does not directly evaluate the extraction accuracy of unaligned sentences. Therefore, for a more detailed analysis, we also evaluated the Precision / Recall / F1 score for each number of aligned source and target sentences.

[0118] <Experiment 1: Experimental Data> For Experiment 1, we purchased newspaper articles from the Yomiuri Shimbun and its English edition, The Japan News (formerly the Daily Yomiuri), and used them. From these data, we created sentence alignment datasets both automatically and manually.

[0119] First, we automatically created 2,989 document alignment data using the method of Uchiyama et al. [3] from 317,491 Japanese articles and 3,878 English articles published in 2012. Sentence alignment was then performed on the document alignment data using the method of Uchiyama et al. [3], and the resulting pseudo-correct sentence alignment data was used as training data for the cross-language span prediction model.

[0120] For the development and evaluation data, 157 bilingual documents consisting of 131 articles and 26 editorials were created by manually searching for Japanese equivalents from 182 English articles published between February 1, 2013 and February 7, 2013, and August 1, 2013 and August 7, 2013. Next, sentence alignment was performed manually from each bilingual document, resulting in 2,243 many-to-many sentence alignments. In this experiment, 15 articles were used for development, another 15 articles for evaluation, and the remaining data was reserved. Figure 7 shows the average number of sentences and tokens in each dataset.

[0121] <Experiment 1: Experimental Results> Figure 8 shows the F1 score for the entire correspondence. Regardless of the optimization method, the results of cross-language span prediction show higher accuracy than the baseline. This shows that extracting sentence alignment candidates and calculating scores using cross-language span prediction works more effectively than the baseline. Furthermore, the results using bidirectional scores are better than those using only unidirectional scores, confirming that score symmetry is very effective for sentence alignment. Next, comparing the scores of DP and ILP, ILP achieves significantly higher accuracy. This shows that optimization using ILP can identify sentence alignments better than optimization using DP, which assumes monotonicity.

[0122] Figure 9 shows the accuracy of sentence alignment evaluated for each number of source and target sentences in the alignment. In Figure 9, the values ​​in rows N and columns M represent the Precision / Recall / F1 score of the N-to-M alignment. A hyphen indicates that the alignment does not exist in the test set.

[0123] Here too, the results of sentence alignment using cross-lingual span prediction outperform the baseline results for all pairs. Furthermore, except for 1-to-2 correspondences, the accuracy of optimization using ILP is higher than that using DP. In particular, the F1 scores for unmatched sentences (1-to-0 and 0-to-1) are very high at 80.0 and 95.1, respectively, demonstrating a significant improvement over the baseline. These results demonstrate that the technology in Example 1 can identify unmatched sentences with very high accuracy and is extremely effective for bilingual documents containing such sentences.

[0124] This experiment used an NVIDIA Tesla K80 (12GB) processor. In the test set, it took approximately 1.9 seconds to predict the span for each input, and the average time required for optimization using linear programming for a document was 0.39 seconds. Traditionally, dynamic programming has been used, which requires less computational time than linear programming, but these results show that optimization can also be performed using linear programming in a practical amount of time.

[0125] <Experiment 2: Comparison of machine translation accuracy> Next, we will explain Experiment 2. Bilingual data extracted by sentence alignment is essential for training cross-language models, primarily machine translation systems. Therefore, to evaluate the effectiveness of the downstream tasks in Example 1, we conducted an accuracy comparison experiment using bilingual sentences automatically extracted from actual newspaper article data in a Japanese-English machine translation model. In this experiment, we compared the following five methods. The numbers in parentheses represent the notations used in the legend in Figure 10.

[0126] Cross-linguistic span prediction + ILP (ILP w / o doc) Cross-lingual span prediction + ILP + document correspondence cost (ILP) Cross-linguistic span prediction + DP (monotonic DP) Thompson et al.'s method [6] (vecalign) ·Uchiyama et al.'s method [3] (utiyama) In Experiment 2, we evaluated a pre-trained machine translation model using the JParaCrawl corpus

[10] and fine-tuned it using bilingual data extracted from the model. We used BLEU

[11] , a commonly used metric in machine translation, as the evaluation metric.

[0127] <Experiment 2: Experimental Data> As in Experiment 1, data was created from the Yomiuri Shimbun and The Japan News. The training dataset used articles published between 1989 and 2015, excluding those used in development and evaluation. For automatic document alignment, the method proposed by Uchiyama et al. [3] was used to create 110,821 bilingual document pairs. Bilingual sentences were extracted from the bilingual documents using each method, and were used in order of quality based on cost and score. The same data as in Experiment 1 was used for the development and evaluation datasets, with 15 articles (168 bilingual sentences) used for development data and 15 articles (238 bilingual sentences) used for evaluation data.

[0128] <Experiment 2: Experimental Results> Figure 10 shows the results of a comparison of translation accuracy when the number of bilingual sentence pairs used for training is varied. It can be seen that the results of the sentence alignment method using cross-language span prediction achieve higher accuracy than the baseline. In particular, the method using ILP and document alignment cost achieved a maximum BLEU score of 19.0 points, which is 2.6 points higher than the best result of the baseline. These results show that the technology in Example 1 works effectively on automatically extracted bilingual documents and is useful in downstream tasks.

[0129] Focusing on the small data volume, we can see that the method using document correspondence cost achieves translation accuracy at the same level or higher than other methods using only ILP or DP. This shows that using document correspondence cost is useful for improving the reliability of sentence correspondence cost and eliminating low-quality correspondences.

[0130] (Summary of Example 1) As explained above, in the first embodiment, the problem of identifying a pair of corresponding sets of sentences (or sentences) in two documents that correspond to each other is regarded as a set of problems of independently predicting, as a span, a set of consecutive sentences in a document in one language that corresponds to a set of consecutive sentences in a document in another language (cross-language span prediction problems), and highly accurate sentence alignment is achieved by performing global optimization on the prediction results using integer linear programming.

[0131] The cross-language span prediction model of Example 1 is created by fine-tuning a pre-trained multilingual model created using only monolingual text for each of multiple languages, for example, using pseudo-ground-truth data created by an existing method. By using a model that uses a structure called self-attention in the multilingual model and inputting a source language sentence and a target language document into the model in combination, it is possible to take into account the context before and after the span and information on a token-by-token basis during prediction. Compared to conventional methods that use bilingual dictionaries and vector representations of sentences, which do not utilize such information, it is possible to predict candidates for sentence correspondence with high accuracy.

[0132] It should be noted that the cost of creating correct answer data is very high. On the other hand, the sentence matching task requires more correct answer data than the word matching task described in Example 2. Therefore, in Example 1, good results are obtained by using pseudo-correct answer data as correct answer data. The use of pseudo-correct answer data enables supervised learning, which makes it possible to learn a high-performance model compared to an unsupervised model.

[0133] Furthermore, the integer linear programming used in Example 1 does not assume monotonicity of correspondence relationships. Therefore, it is possible to obtain sentence correspondences with extremely high accuracy compared to conventional methods that assume monotonicity. In this case, by using scores that symmetrize the two-way scores obtained from asymmetric cross-language span prediction, the reliability of prediction candidates is improved, contributing to further improvement of accuracy.

[0134] The technology for automatically identifying sentence alignments using two corresponding documents as input has various implications for natural language processing technology. For example, as in Experiment 2, by mapping sentences in a document in one language (e.g., Japanese) to their corresponding sentences in a document translated into another language based on the sentence alignments, training data for a machine translator between the two languages ​​can be generated. Alternatively, by extracting pairs of sentences with the same meaning based on sentence alignments from a document and its rewritten version in simpler language, this can be used as training data for a paraphrase generator or lexical simplifier.

[0135] [References for Example 1] [1] William A. Gale and Kenneth W. Church. A program for aligning sentences in bilingual corpora. Computational Linguistics, Vol. 19, No. 1, pp. 75-102, 1993. [2] Takehito Utsuro, Hiroshi Ikeda, Masaya Yamane, Yuji Matsumoto, and Makoto Nagao. Bilingual text, matching using bilingual dictionary and statistics. In Proceedings of the COLING-1994. [3] Masao Utiyama and Hitoshi Isahara. Reliable measures for matching Japanese-English news articles and sentences. In Proceedings of the ACL-2003, pp. 107-111. 72-79, [4] D. Varga, L. Nemeth, P. Halacsy, A. Kornai, V. Tron, and V. Nagy. Parallel corpora for medium density languages. In Proceedings of the RANLP-2005, pp. 107-111. 590–596, [5] Rico Sennrich and Martin Volk. Iterative, MT-based sentence alignment of parallel texts. In Proceedings of the 18th Nordic Conference on Computational Linguistics (NODALIDA 2011), pp. 175-182, Riga, Latvia, May 2011. Northern European Association for Language Technology (NEALT). [6] Brian Thompson and Philipp Koehn. Vecalign: Improved sentence alignment in linear time and space. In Proceedings of EMNLP-2019, pp. 1342-1348, 2019. [7] S. E. Robertson and S. Walker. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In Proceedings of the SIGIR-1994, pp. 232-241, 1994. [8] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of EMNLP-2016, pp. 2383-2392, 2016. [9] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the NAACL-2019, pp. 4171-4186, 2019.

[10] Makoto Morishita, Jun Suzuki, and Masaaki Nagata. JParaCrawl: A large scale web-based English- Japanese parallel corpus. In Proceedings of The 12th Language Resources and Evaluation Conference, pp. 3603-3609, Marseille, France, May 2020. European Language Resources Association.

[11] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. Example 2 Next, a second embodiment will be described. In the second embodiment, a technique for identifying word alignment between two sentences that are translations of each other will be described. Identifying words or word sets that are translations of each other in two sentences that are translations of each other is called word alignment.

[0136] The technology of automatically identifying word correspondences between two mutually translated sentences has various applications in multilingual processing and machine translation. For example, by mapping annotations of named entities (such as people's names, places' names, and organization names) in a sentence in one language (e.g., English) to a sentence translated into another language (e.g., Japanese) based on word correspondences, training data for a named entity extractor in that language can be generated.

[0137] In the second embodiment, the problem of finding word correspondence between two mutually translated sentences is treated as a set of problems (cross-language span prediction) that predict the words or consecutive word strings (spans) in a sentence in one language that correspond to each word in the sentence in another language, and highly accurate word correspondence is achieved by learning a cross-language span prediction model using a neural network from a small amount of manually created correct answer data. Specifically, a word correspondence device 300 (described later) executes the processing related to this word correspondence.

[0138] In addition to generating training data for the named entity extractor described above, applications of word correspondence include the following:

[0139] When translating a web page in one language (e.g., Japanese) to another language (e.g., English), the original text may contain HTML tags (e.g., anchor tags) ... ) and identify the range of characters in a sentence in another language that is semantically equivalent to the range of characters enclosed in the `.txt` tag, based on word correspondence, so that HTML tags can be mapped correctly.

[0140] In machine translation, if you want to specify a specific translation for a specific word in an input sentence using a bilingual dictionary or the like, you can control the translation by finding the word in the output sentence that corresponds to the word in the input sentence based on word correspondence, and if that word is not the specified word, replacing it with the specified word.

[0141] In the following, various reference techniques related to word matching will be described first to facilitate understanding of the technique according to the second embodiment, followed by a description of the configuration and operation of the word matching device 300 according to the second embodiment.

[0142] The numbers and names of reference documents related to the reference techniques of Example 2 are listed at the end of Example 2. In the following description, the numbers of related reference documents are indicated as "[1]" etc.

[0143] Example 2: Description of the Reference Technology <Unsupervised word matching based on statistical machine translation models> As a reference technique, we first explain unsupervised word matching based on a statistical machine translation model.

[0144] In statistical machine translation [1], a translation model P(E|F) that converts a sentence F in a source language (source language) to a sentence E in a target language (target language) is decomposed using Bayes' theorem into the product of a reverse translation model P(F|E) and a language model P(E) that generates a word sequence in the target language.

[0145]

number

[0146]

number

[0147] Let X be a source language sentence, x, of length |X|. 1:|X| =x1,x2,...,x |X| Let the target sentence Y be a word sequence y of length |Y|. 1:|Y| =y1,y 2, ...,y |Y| Then, let A be the word correspondence from the target language to the source language. 1:|Y| =a1,a2,...,a |Y| where a j is the word y in the target sentence. j is word x in the target sentence aj This indicates that it corresponds to

[0148] In generative word alignment, the translation probability based on a word alignment A is expressed as the lexical translation probability P t (y j |...) and word correspondence probability P a (a j Decompose into a product of |...).

[0149]

number

[0150]

number

[0151] Furthermore, HMM-based word alignment

[25] assumes that the word alignment probability depends on the word alignment of the previous word in the target sentence.

[0152]

number

[0153] Unsupervised word alignment tools based on the model described in Reference [1] include GIZA++

[16] , MGIZA [8], and FastAlign [6]. GIZA++ and MGIZA are based on Model 4 described in Reference [1], while FastAlign is based on Model 2 described in Reference [1].

[0154] <Word matching based on recurrent neural networks> Next, we will explain word alignment based on recurrent neural networks. There are two unsupervised word alignment methods based on neural networks: applying neural networks to HMM-based word alignment [26, 21] and attention-based methods in neural machine translation [27, 9].

[0155] Regarding the application of neural networks to HMM-based word correspondence, for example, Tamura et al.

[21] used a recurrent neural network (RNN) to analyze not only the previous word correspondence but also the history of word correspondence from the beginning of the sentence. j =a 1:j-1 We propose a method to determine the correspondence of the current word by taking into account the above, and to obtain word correspondence as a single model rather than modeling lexical translation probability and word correspondence probability separately.

[0156]

number

[0157] <Unsupervised word matching based on neural machine translation models> Next, we will explain unsupervised word matching based on a neural machine translation model. Neural machine translation realizes the conversion of a source language sentence into a target language sentence based on an encoder-decoder model.

[0158] The encoder generates a source language sentence X=x of length |X| by a function enc, which represents a nonlinear transformation using a neural network. 1:|X| =x1,...,x |X| Let us consider a sequence of internal states s of length |X|. 1:|X| =s1,...,s |X| If the number of dimensions of the internal state corresponding to each word is d, then s 1:|X| is a matrix of |X|×d.

[0159]

number

[0160]

number

[0161] In neural machine translation, the introduction of an attention mechanism has significantly improved translation accuracy. The attention mechanism determines which word information in the source language sentence to use when generating each word in the target language sentence in the decoder by changing the weighting of the encoder's internal state. The basic idea behind attention-based unsupervised word alignment in neural machine translation is to consider this attention value as the probability that two words are translations of each other.

[0162] As an example, we will explain the attention between source and target sentences (source-target attention) in Transformer

[23] , a representative neural machine translation model. Transformer is an encoder-decoder model that combines self-attention and a feed-forward neural network to parallelize the encoder and decoder. The attention between source and target sentences in Transformer is called cross-attention to distinguish it from self-attention.

[0163] Transformer uses scaled dot-product attention as attention. The scaled dot-product attention is lq×dk , key K∈R lk×dk , value V∈R lk×dv is defined as follows:

[0164]

number

[0165] In cross-attention, Q, K, and V are W Q ∈R d×dk ,W K ∈R d×dk ,W V ∈R d×dv is defined as the weight as follows:

[0166]

number

[0167]

number

[0168]

number

[0169] In this case, Q=[t 1:|Y| ] T W Q The weight matrix A of the cross attention between the source and target sentences is |Y|×|X| Define

[0170]

number

[0171]

number

[0172] Generally, a Transformer uses multiple layers and multiple heads (attention mechanisms trained from different initial values), but for simplicity, we will use only one layer and one head here.

[0173] Garg et al. reported that the average cross-attention of all heads in the second layer from the top was closest to the correct answer for word correspondence, and the word correspondence distribution G p We define the following cross-entropy loss for the word correspondence obtained from a specific head among multiple heads using

[0174]

number

[0175] In the method of Garg et al., when calculating the loss of word alignment, in Equation (10), the loss is calculated by using t from the beginning of the sentence to just before the jth word. 1:i-1 Instead, the whole target sentence t 1:|Y| Also, we use the training data G p Instead of using self-training based on Transformer, they use word correspondences obtained from GIZA++. They report that this method achieves word correspondence accuracy that exceeds that of GIZA++ [9].

[0176] <Supervised word alignment based on neural machine translation models> Next, we will explain supervised word alignment based on a neural machine translation model. 1:|X| and target sentence Y=y 1:|Y| For, we define a word correspondence A as a subset of the Cartesian product of word positions.

[0177]

number

[0178] Discriminative word alignment models word alignments directly from the source and target sentences.

[0179]

number

[20] . In their method, first, a sequence of internal states of the encoder in the neural machine translation model is denoted as s1,...,s |X| , the sequence of internal states of the decoder is t1,...,t |Y| Then, we project these onto a common vector space using a three-layer feedforward neural network with shared parameters.

[0180]

number

[0181]

number

[0182]

number

[0183]

number

[0184]

number

[0185]

number

[0186] <BERT pre-trained model> For word alignment, the pre-trained model BERT is used, as in the case of sentence alignment in Example 1, as described in Example 1.

[0187] (Example 2: Issues) The conventional word matching based on recurrent neural networks and unsupervised word matching based on neural machine translation models, which were explained as reference technologies, have only been able to achieve accuracy equivalent to or slightly better than that of unsupervised word matching based on statistical machine translation models.

[0188] Supervised word alignment based on conventional neural machine translation models is more accurate than unsupervised word alignment based on statistical machine translation models. However, both statistical and neural machine translation models have the problem of requiring a large amount of bilingual data (on the order of millions of sentences) to train the translation model.

[0189] A technique according to a second embodiment that solves the above problems will be described below.

[0190] (Overview of the technology according to the second embodiment) In Example 2, word alignment is realized as a process of calculating answers from cross-language span prediction questions. First, a pre-trained multilingual model trained from at least monolingual data for each language pair to which word alignments are assigned is fine-tuned using correct answer data for cross-language span prediction created from manually generated correct answers for word alignments, thereby training a cross-language span prediction model. Next, word alignment processing is performed using the trained cross-language span prediction model.

[0191] By the method described above, in the second embodiment, it is possible to realize highly accurate word correspondence from a small amount of manually created correct answer data of word correspondence without requiring bilingual data for pre-training a model for executing word correspondence. The technology according to the second embodiment will be described in more detail below.

[0192] (Device configuration example) 11 shows a word correspondence device 300 and a pre-training device 400 according to the second embodiment. The word correspondence device 300 is a device that executes word correspondence processing using the technology according to the second embodiment. The pre-training device 400 is a device that learns a multilingual model from multilingual data.

[0193] As shown in FIG. 11, the word matching device 300 includes a cross-language span prediction model training unit 310 and a word matching execution unit 320.

[0194] The cross-language span prediction model training unit 310 has a word correspondence correct answer data storage unit 311, a cross-language span prediction question answer generation unit 312, a cross-language span prediction correct answer data storage unit 313, a span prediction model training unit 314, and a cross-language span prediction model storage unit 315. The cross-language span prediction question answer generation unit 312 may also be called a question answer generation unit.

[0195] The word correspondence execution unit 320 has a cross-language span prediction question generation unit 321, a span prediction unit 322, and a word correspondence generation unit 323. The cross-language span prediction question generation unit 321 may also be called a question generation unit.

[0196] The pre-training device 400 is a device based on existing technology. The pre-training device 400 includes a multilingual data storage unit 410, a multilingual model training unit 420, and a pre-trained multilingual model storage unit 430. The multilingual model training unit 420 reads monolingual text in at least two target languages ​​for which word correspondence is to be obtained from the multilingual data storage unit 410, thereby training a language model, and stores the language model in the pre-trained multilingual model storage unit 230 as a pre-trained multilingual model.

[0197] In the second embodiment, a pre-trained multilingual model trained by some means is simply input to the cross-language span prediction model training unit 310. Therefore, the pre-training device 400 may not be provided, and a general-purpose pre-trained multilingual model that is publicly available may be used.

[0198] The pre-trained multilingual model in Example 2 is a language model that is pre-trained using monolingual text in at least two target languages ​​for which word correspondence is sought. In Example 2, multilingual BERT is used as the language model, but is not limited to this. Any pre-trained multilingual model that can output word embedding vectors that take context into account for multilingual text, such as XLM-RoBERTa, may be used.

[0199] The word correspondence device 300 may be called a learning device. The word correspondence device 300 may be provided with the word correspondence execution unit 320 without the cross-language span prediction model training unit 310. A device provided with the cross-language span prediction model training unit 310 alone may be called a learning device.

[0200] (Outline of operation of word correspondence device 300) 12 is a flowchart showing the overall operation of the word correspondence device 300. In S300, a pre-trained multilingual model is input to the cross-language span prediction model training unit 310, and the cross-language span prediction model training unit 310 trains a cross-language span prediction model based on the pre-trained multilingual model.

[0201] In S400, the cross-language span prediction model trained in S300 is input to the word correspondence execution unit 320, and the word correspondence execution unit 320 uses the cross-language span prediction model to generate and output word correspondences for the input sentence pair (two sentences that are translations of each other).

[0202] <s300> The process of training the cross-language span prediction model in S300 will be described with reference to the flowchart in Fig. 13. Here, it is assumed that a pre-trained multilingual model has already been input and stored in the storage device of the span prediction model training unit 324. Furthermore, word-associated correct answer data storage unit 311 stores word-associated correct answer data.

[0203] In S301, the cross-language span prediction question answer generation unit 312 reads word-corresponding correct answer data from the word-corresponding correct answer data storage unit 311, generates cross-language span prediction correct answer data from the read word-corresponding correct answer data, and stores it in the cross-language span prediction correct answer data storage unit 313. The cross-language span prediction correct answer data is data consisting of a set of pairs of cross-language span prediction questions (question and context) and their answers.

[0204] In S302, the span prediction model training unit 314 trains a cross-language span prediction model from the cross-language span prediction correct answer data and the pre-trained multilingual model, and stores the trained cross-language span prediction model in the cross-language span prediction model storage unit 315.

[0205] <s400> Next, the content of the process of generating word correspondences in S400 above will be described with reference to the flowchart in Fig. 14. Here, it is assumed that the cross-language span prediction model has already been input to the span prediction unit 322 and stored in the storage device of the span prediction unit 322.

[0206] In S401, a pair of a first language sentence and a second language sentence is input to the cross-language span prediction question generation unit 321. In S402, the cross-language span prediction question generation unit 321 generates a cross-language span prediction question (question and context) from the input sentence pair.

[0207] Next, in S403, the span prediction unit 322 uses the cross-language span prediction model to perform span prediction on the cross-language span prediction question generated in S402, and obtains an answer.

[0208] In S404, the word alignment generation unit 323 generates word alignments from the answers to the cross-language span prediction questions obtained in S403. In S405, the word alignment generation unit 323 outputs the word alignments generated in S404.

[0209] (Example 2: Explanation of specific processing contents) The processing contents of the word correspondence device 300 in the second embodiment will be described in more detail below.

[0210] <Formulation from word correspondence to span prediction> As described above, in the second embodiment, the word matching process is executed as a cross-language span prediction problem process. Therefore, the formulation from word matching to span prediction will be explained using an example. In relation to the word matching device 300, the cross-language span prediction model training unit 310 will be mainly explained here.

[0211] --About word correspondence data-- An example of Japanese and English word alignment data is shown in Figure 15. This is an example of one word alignment data. As shown in Figure 15, one word alignment data is composed of five pieces of data: a token (word) string in the first language (Japanese), a token string in the second language (English), a string of corresponding token pairs, the original text in the first language, and the original text in the second language.

[0212] The token sequence of the first language (Japanese) and the token sequence of the second language (English) are both indexed, starting with 0, which is the index of the first element (leftmost token) in the token sequence, and then indexed with 1, 2, 3, ...

[0213] For example, the first element of the third data, "0-1," indicates that the first element of the first language, "Ashikaga," corresponds to the second element of the second language, "ashikaga." Also, "24-2 25-2 26-2" indicates that "de," "a," and "ru" all correspond to "was."

[0214] In Example 2, we formulate word alignment as a cross-lingual span prediction problem similar to the SQuAD-style question answering task

[18] .

[0215] A question-answering system performing a SQuAD-style question-answering task is given a "context," such as a paragraph selected from Wikipedia, and a "question," and the system predicts a "span" (substring) in the context as the "answer."

[0216] In the same manner as the span prediction described above, the word correspondence execution unit 320 in the word response device 300 of the second embodiment regards the target language sentence as a context and the words of the source language sentence as a question, and predicts the word or word string in the target language sentence that is the translation of the word in the source language sentence as the span of the target language sentence. For this prediction, the cross-language span prediction model of the second embodiment is used.

[0217] --About the cross-language span prediction question answer generation unit 312-- In the second embodiment, the cross-language span prediction model learning unit 310 of the word matching device 300 performs supervised learning of the cross-language span prediction model, and correct answer data is required for the learning.

[0218] In the second embodiment, a plurality of pieces of word correspondence data such as those illustrated in FIG. 15 are stored as correct answer data in the word correspondence correct answer data storage unit 311 of the cross-language span prediction model training unit 310, and are used to train the cross-language span prediction model.

[0219] However, since the cross-language span prediction model is a model for predicting answers (spans) from questions across languages, data is generated for learning to predict answers (spans) from questions across languages. Specifically, by inputting word correspondence data to the cross-language span prediction question answer generation unit 312, the cross-language span prediction question answer generation unit 312 generates pairs of cross-language span prediction questions (questions) and answers (spans, substrings) in SQuAD format from the word correspondence data. An example of the processing of the cross-language span prediction question answer generation unit 312 will be described below.

[0220] FIG. 16 shows an example of converting the word correspondence data shown in FIG. 15 into a span prediction problem in the SQuAD format.

[0221] First, the upper half shown in (a) of Figure 16 will be described. The upper half of Figure 16 (the part with context, question 1, and answer) shows that a sentence in the first language (Japanese) of the word correspondence data is given as context, the token "was" in the second language (English) is given as question 1, and the answer is the span "de aru" of the sentence in the first language. The correspondence between this "de aru" and "was" corresponds to the corresponding token pair "24-2 25-2 26-2" of the third data in Figure 15. In other words, the cross-language span prediction question answer generation unit 312 generates pairs of span prediction questions (question and context) and answers in SQuAD format based on the corresponding token pairs of the correct answers.

[0222] As will be described later, in the second embodiment, the span prediction unit 322 of the word correspondence execution unit 320 uses a cross-language span prediction model to make predictions in each direction, from a first language sentence (question) to a second language sentence (answer), and from a second language sentence (question) to a first language sentence (answer). Therefore, the cross-language span prediction model is also trained to make predictions in both directions.

[0223] Note that performing bidirectional prediction as described above is just one example. It is also possible to perform one-way prediction, such as prediction from a first language sentence (question) to a second language sentence (answer), or prediction from a second language sentence (question) to a first language sentence (answer). For example, in English education, etc., when English sentences and Japanese sentences are displayed simultaneously and an arbitrary character string (word string) in the English sentence is selected with a mouse or the like, a Japanese character string (word string) that is its translation is calculated and displayed on the spot, a one-way prediction is sufficient.

[0224] Therefore, the cross-language span prediction question answer generation unit 312 in Example 2 converts one piece of word correspondence data into a set of questions that predict spans in sentences in the second language from each token in the first language, and a set of questions that predict spans in sentences in the first language from each token in the second language. In other words, the cross-language span prediction question answer generation unit 312 converts one piece of word correspondence data into a set of questions consisting of each token in the first language and their respective answers (spans in sentences in the second language), and a set of questions consisting of each token in the second language and their respective answers (spans in sentences in the first language).

[0225] If one token (question) corresponds to multiple spans (answers), the question is defined as having multiple answers. In other words, the cross-language span prediction question answer generation unit 112 generates multiple answers for the question. Also, if there is no span corresponding to a certain token, the question is defined as having no answer. In other words, the cross-language span prediction question answer generation unit 312 determines that there is no answer for the question.

[0226] In Example 2, the language of the question is called the source language, and the language of the context and answer (span) is called the target language. In the example shown in Figure 16, the source language is English and the target language is Japanese, and this question is called an "English-to-Japanese" question.

[0227] If the question contains a high-frequency word such as "of," it may appear multiple times in the source sentence, making it difficult to find the corresponding span in the target sentence unless the context of the word in the source sentence is taken into consideration. Therefore, the cross-language span prediction question answer generation unit 312 in the second embodiment generates questions with context.

[0228] The lower half of Figure 16(b) shows an example of a question with the context of the source sentence. In Question 2, the two tokens immediately before "Yoshimitsu ASHIKAGA" and the two tokens immediately after "the 3rd" in the context of the source sentence, which is the question, have "¶" added as a boundary marker.

[0229] In addition, in Question 3, the entire source sentence is used as the context, and the token that is the question is sandwiched between two boundary symbols. As will be explained later in the experiment, the longer the context attached to the question, the better. Therefore, in Example 2, the entire source sentence is used as the context of the question, as in Question 3.

[0230] As described above, in Example 2, the paragraph mark '¶' is used as the boundary symbol. This symbol is called a pilcrow in English. The pilcrow belongs to the punctuation category of the Unicode character category, is included in the vocabulary of multilingual BERT, and rarely appears in normal text. Therefore, in Example 2, it is used as the boundary symbol that separates the question from the context. Any character or string that satisfies similar properties may be used as the boundary symbol.

[0231] Furthermore, the word alignment data contains many null alignments (i.e., there is no corresponding target). Therefore, in Example 2, we use the formulation of SQuADv2.0

[17] . The difference between SQuADv1.1 and SQuADv2.0 is that they explicitly handle the possibility that the answer to a question does not exist in the context.

[0232] In other words, the SQuADV2.0 format explicitly indicates that questions that cannot be answered cannot be answered, so it can generate appropriate questions and answers (that cannot be answered) for null alignments (no alignment, no correspondence) in the word alignment data.

[0233] Since tokenization including word splitting and casing are handled differently depending on the word correspondence data, in Example 2, the token sequence of the source language sentence is used only for the purpose of creating questions.

[0234] When the cross-language span prediction question answer generator 312 converts the word correspondence data into the SQuAD format, it uses the original sentence, not the token sequence, for the question and context. That is, the cross-language span prediction question answer generator 312 generates the start and end positions of the span as an answer from the target language sentence (context), along with the word or word sequence of the span, and these start and end positions serve as indexes to the character positions of the original sentence of the target language sentence.

[0235] In addition, conventional word correspondence methods often use a token string as input. In other words, in the example of word correspondence data in Figure 15, the first two pieces of data are often the input. In contrast, in Example 2, both the original text and the token string are input to the cross-language span prediction question answer generation unit 312, thereby providing a system that can flexibly handle any tokenization.

[0236] The data of pairs of cross-language span prediction questions (question and context) and answers generated by the cross-language span prediction question answer generation unit 312 is stored in the cross-language span prediction correct answer data storage unit 313.

[0237] --About the span prediction model learning unit 314-- The span prediction model training unit 314 trains the cross-language span prediction model using the correct answer data read from the cross-language span prediction correct answer data storage unit 313. That is, the span prediction model training unit 314 inputs a cross-language span prediction problem (question and context) into the cross-language span prediction model, and adjusts the parameters of the cross-language span prediction model so that the output of the cross-language span prediction model becomes the correct answer. This training is performed separately for cross-language span prediction from a first language sentence to a second language sentence, and for cross-language span prediction from a second language sentence to a first language sentence.

[0238] The trained cross-language span prediction model is stored in the cross-language span prediction model storage unit 315. The word correspondence execution unit 320 also reads the cross-language span prediction model from the cross-language span prediction model storage unit 315 and inputs it to the span prediction unit 322.

[0239] The cross-lingual span prediction model is described in detail below, as well as the processing of the word matching implementer 320.

[0240] <Cross-language span prediction using multilingual BERT> As already explained, the span prediction unit 322 of the word alignment execution unit 320 in the second embodiment generates word alignments from input sentence pairs using the cross-language span prediction model trained by the cross-language span prediction model training unit 310. In other words, word alignments are generated by performing cross-language span prediction on the input sentence pairs.

[0241] --About the cross-linguistic span prediction model-- In Example 2, the task of cross-language span prediction is defined as follows:

[0242] Source text X=x1x2...x of length |X| characters |X| , and a target language sentence Y=y1y2...y of length |Y| characters |Y| Suppose there is a source token x from character position i to character position j in a source sentence. i:j =x i ...x j , the target span y from character position k to character position l in the target sentence k:l =y k ...y l The task of cross-lingual span prediction is to extract

[0243] The span prediction unit 322 of the word correspondence execution unit 320 executes the above tasks using the cross-language span prediction model trained by the cross-language span prediction model training unit 310. In the second embodiment, the multilingual BERT [5] is also used as the cross-language span prediction model.

[0244] BERT also performs extremely well for the cross-language task in Example 2. Note that the language model used in Example 2 is not limited to BERT.

[0245] More specifically, in Example 2, a model similar to the model for the SQuADv2.0 task disclosed in [5] is used as a cross-language span prediction model. These models (the model for the SQuADv2.0 task and the cross-language span prediction model) are pre-trained BERT models with two independent output layers that predict the start and end positions in the context.

[0246] In the cross-language span prediction model, the probability that each position in the target sentence will be the start and end position of the answer span is p start and p end Let the source span x i:j Given the target language span y k:l Score ω X→Y ijkl is defined as the product of the probability of the starting position and the probability of the ending position, and (^k,^l) that maximizes this product is defined as the best answer span.

[0247]

number

[0248]

number

[0249] The cross-language span prediction model in Example 2 and the model for the SQuADv2.0 task disclosed in Reference [5] have basically the same neural network structure, but differ in that the model for the SQuADv2.0 task uses a monolingual pre-trained language model and is fine-tuned (additional learning / transfer learning / fine-tuning) with training data for a task such as predicting spans between the same languages, whereas the cross-language span prediction model in Example 2 uses a pre-trained multilingual model including two languages ​​related to cross-language span prediction and is fine-tuned with training data for a task such as predicting spans between two languages.

[0250] In addition, while the existing implementation of the SQuAD model of BERT only outputs the answer string, the cross-language span prediction model of Example 2 is configured to be able to output the start position and end position.

[0251] Within BERT, i.e., within the cross-lingual span prediction model of Example 2, the input sequence is first tokenized by a tokenizer (e.g., WordPiece), and then CJK characters (Chinese characters) are split into units of one character.

[0252] In the existing implementation of the SQuAD model in BERT, the start and end positions are indices to tokens within BERT, but in the cross-lingual span prediction model in Example 2, they are indices to character positions. This makes it possible to handle tokens (words) in the input text for which word correspondence is sought and tokens within BERT independently.

[0253] FIG. 17 shows a process of predicting a target language (Japanese) span that serves as an answer for the token "Yoshimitsu" in a source language sentence (English) that serves as a question, from the context of the target language sentence (Japanese), using the cross-language span prediction model of Example 2. As shown in FIG. 17, "Yoshimitsu" is composed of four BERT tokens. Note that a "##" (prefix) is added to a BERT token, which is a token within BERT, to indicate a connection with the previous vocabulary. Also, the boundary of an input token is indicated by a dotted line. Note that in this embodiment, an "input token" and a "BERT token" are distinguished. The former is a unit of word segmentation in the training data, and is indicated by a dashed line in FIG. 17. The latter is a unit of segmentation used within BERT, and is indicated by a space in FIG. 17.

[0254] In the example shown in FIG. 17, five possible answers are displayed: "Yoshimitsu", "Yoshimitsu (Ashikaga Yoshimitsu", "Ashikaga Yoshimitsu", "Yoshimitsu (", and "Yoshimitsu (Ashikaga Yoshi"); "Yoshimitsu" is the correct answer.

[0255] In BERT, spans are predicted for each token within BERT, so the predicted spans do not necessarily match the boundaries of the input tokens (words). Therefore, in the second embodiment, for a target language span that does not match the boundaries of the target language tokens, such as "Yoshimitsu (Ashikagayoshi", the target language words that are completely included in the predicted target language span, i.e., "Yoshimitsu", "(", and "Ashikaga" in this example), are matched with the source language token (question). This process is performed only at the time of prediction, and is performed by the word alignment generation unit 323. During learning, learning is performed based on a loss function that compares the first candidate for span prediction with the correct answer regarding the start and end positions.

[0256] --Cross-language span prediction question generation unit 321 and span prediction unit 322-- The cross-lingual span prediction question generation unit 321 creates a span prediction question for each question (input token (word)) in the format "[CLS]question[SEP]context[SEP]" that links the question and context for each input first language sentence and second language sentence, and outputs it to the span prediction unit 122. However, as mentioned above, the question is a contextual question that uses ¶ as a boundary symbol, such as "Yoshimitsu ASHIKAGA ¶ was ¶ the 3rd Seii Taishogun of the Muromachi Shogunate and reigned from 1368 to 1394."

[0257] The cross-language span prediction problem generation unit 321 generates a problem of span prediction from a first language sentence (question) to a second language sentence (answer) and a problem of span prediction from a second language sentence (question) to a first language sentence (answer).

[0258] The span prediction unit 322 inputs each question (question and context) generated by the cross-language span prediction question generation unit 121, calculates the answer (predicted span) and probability for each question, and outputs the answer (predicted span) and probability for each question to the word correspondence generation unit 323.

[0259] The above probability is the product of the probability of the start position and the probability of the end position in the best answer span. The processing of the word correspondence generating unit 323 will be explained below.

[0260] <Symmetrization of word correspondence> In span prediction using the cross-language span prediction model of Example 2, target language spans are predicted for source language tokens, so the source and target languages ​​are asymmetric, similar to the model described in Reference [1]. In Example 2, a method for symmetrical bidirectional prediction is introduced to improve the reliability of word correspondence based on span prediction.

[0261] First, for reference, we will explain a conventional method for symmetrizing word alignments. A method for symmetrizing word alignments based on the model described in Reference [1] was first proposed in Reference

[16] . The representative statistical translation toolkit Moses

[11] implements heuristics such as intersection, union, and grow-diag-final, with grow-diag-final being the default. The intersection of two word alignments has high precision but low recall. The union of two word alignments has low precision but high recall. Grow-diag-final is a method for finding word alignments that is intermediate between intersection and union.

[0262] --About the word correspondence generation unit 323-- In the second embodiment, the word alignment generation unit 323 averages the probability of the best span for each token in two directions, and if this average is equal to or greater than a predetermined threshold, it is deemed to correspond. This process is performed by the word alignment generation unit 323 using the output from the span prediction unit 322 (cross-lingual span prediction model). As described with reference to FIG. 17, the predicted span output as the answer does not necessarily match the word boundary, so the word alignment generation unit 323 also performs a process of adjusting the predicted span so that it corresponds to a unidirectional word unit. The specific process of symmetrizing word alignment is as follows.

[0263] In sentence X, the span from start position i to end position j is x i:j In sentence Y, the span from start position k to end position l is defined as y k:l Let token x i:j is the span y k:l The probability of predicting X→Y ijkl Let token y k:l is span x i:j The probability of predicting Y→X ijkl Token x i:j and token y k:l Corresponding to a ijkl The probability of ω ijkl In this embodiment, ω ijkl x i:j The best span y predicted from ^k:^l The probability of ω X→Y ij^k^l and, y k:l The best span predicted from x ^i:^j The probability of ω Y→X ^i^jkl Calculated as the average of

[0264]

number

[0265] The symmetrization method used in the second embodiment is called bidirectional average (bidi-avg). Bidirectional average is easy to implement, and has the same effect as grow-diag-final in that it finds word correspondences that are intermediate between set union and set intersection. Note that using the average is just one example. For example, if the probability ω X→Y ij^k^l and probability ω Y→X ^i^jkl Alternatively, a weighted average of these may be used, or the maximum value of these may be used.

[0266] FIG. 18 shows (a) Japanese to English span prediction and (b) English to Japanese span prediction symmetrically obtained by bidirectional averaging (c).

[0267] In the example of FIG. 18, for example, the probability ω of the best span “language” predicted from “language” X→Y ij^k^l is 0.8, and the probability of the best span "language" predicted from "language" is ω Y→X ^i^jkl is 0.6, and their average is 0.7. Since 0.7 is above the threshold, it can be determined that "langaku" and "language" correspond to each other. Therefore, the word alignment generation unit 123 generates and outputs the word pair "langaku" and "language" as one of the word alignment results.

[0268] In the example of FIG. 18, the word pair "is" and "de" is predicted only in one direction (from English to Japanese), but is considered to correspond because the two-way average probability is above the threshold.

[0269] The threshold of 0.4 was determined through a preliminary experiment in which the learning data for Japanese and English word correspondences, described below, was divided in half, with one half used as training data and the other as test data. This value was used in all experiments described below. Since span predictions for each direction are performed independently, it may be necessary to normalize the scores for symmetry, but in the experiments, normalization was not necessary because both directions were trained using a single model.

[0270] (Example 2: Effects of the embodiment) The word matching device 300 described in the second embodiment does not require a large amount of bilingual data for the language pair to which word matching is assigned, and can achieve more accurate supervised word matching than before using a smaller amount of training data (correct answer data created manually) than before.

[0271] (Example 2: Experimental) In order to evaluate the technology according to the second embodiment, an experiment on word correspondence was carried out, and the experimental method and results will be described below.

[0272] <Example 2: Experimental Data> Figure 19 shows the number of sentences in the training data and test data for manually generated gold word alignments for five language pairs: Chinese-English (Zh-En), Japanese-English (Ja-En), German-English (De-En), Romanian-English (Ro-En), and English-French (En-Fr). The table in Figure 19 also shows the number of reserved data.

[0273] In the experiment using the conventional technique

[20] , Zh-En data was used, and in the experiment using the conventional technique [9], De-En, Ro-En, and En-Fr data were used. In the experiment using the technique of this embodiment, Ja-En data, which is one of the most distant language pairs in the world, was added.

[0274] The Zh-En data was obtained from the GALE Chinese-English Parallel Aligned Treebank

[12] , which includes broadcast news, news wires, and web data. To approximate the experimental conditions described in

[20] as closely as possible, we used character-tokenized bilingual texts, cleaned them by removing mismatches and timestamps, and randomly divided them into 80% training data, 10% test data, and 10% reserve data.

[0275] The KFTT word alignment data

[14] was used as Japanese-English data. The Kyoto Free Translation Task (KFTT) (http: / / www.phontron.com / kftt / index.html) is a manual translation of Japanese Wikipedia articles about Kyoto, and consists of 440,000 sentences of training data, 1,166 sentences of development data, and 1,160 sentences of test data. The KFTT word alignment data is a manual translation of a portion of the KFTT development data and test data, with word alignments assigned to the data manually. It consists of eight development data files and seven test data files. In experiments using the technology according to this embodiment, eight development data files were used for training, four test data files were used for testing, and the remaining files were reserved.

[0276] The De-En, Ro-En, and En-Fr data are described in

[27] . The authors have published preprocessing and evaluation scripts (https: / / github.com / lilt / alignment-scripts). Prior art [9] uses these data for experiments. The De-En data is described in

[24] (https: / / www-i6.informatik.rwth-aachen.de / goldAlignment / ). The Ro-En and En-Fr data were provided as common tasks for the HLT-NAACL-2003 workshop on Building and Using Parallel Texts

[13] (https: / / eecs.engin.umich.edu / ). The En-Fr data was originally described in

[15] . The De-En, Ro-En, and En-Fr data contain 508,248,447 sentences. In this embodiment, 300 sentences were used for training for De-En and En-Fr, and 150 sentences for training for Ro-En. The remaining sentences were used for testing.

[0277] <Evaluation scale for word matching accuracy> In the second embodiment, the F1 score, which has equal weighting on precision and recall, is used as the evaluation measure for word correspondence.

[0278]

number

[16] , we also use the AER to compare the prior art with the technology of this embodiment.

[0279] Let us assume that a manually created gold word alignment consists of a sure alignment (S) and a possible alignment (P), where S ⊆ P. We define the precision, recall, and AER of a word alignment A as follows:

[0280]

number

[0281]

number

[0282]

number

[0283] <Comparison of word matching accuracy> 20 shows a comparison between the technology according to Example 2 and the conventional technology. For all five data points, the technology according to Example 2 is superior to all the conventional technologies.

[0284] For example, on the Zh-En data, the technology in Example 2 achieved an F1 score of 86.7, which is 13.3 points higher than the F1 score of 73.4 of DiscAlign reported in

[20] , the current state-of-the-art accuracy for supervised word alignment. While the method in

[20] uses 4 million sentence-pair bilingual data to pre-train the translation model, the technology in Example 2 does not require bilingual data for pre-training. On the Ja-En data, Example 2 achieved an F1 score of 77.6, which is 20 points higher than the F1 score of 57.8 of GIZA++.

[0285] For the De-EN, Ro-EN, and En-Fr data, the method that currently achieves the highest accuracy in unsupervised word alignment [9] reports only AER, so we evaluate it in this embodiment. For comparison, we also report the AER of MGIZA and other conventional methods for the same data [22, 10].

[0286] In the experiment, both sure and possible word correspondences were used for the De-En data in the training of this embodiment, but only sure was used for the En-Fr data because it was very noisy. The AERs of this embodiment for the De-En, Ro-En, and En-Fr data were 11.4, 12.2, and 4.0, respectively, which are significantly lower than the method in [9].

[0287] Comparing the accuracy of supervised learning with that of unsupervised learning is clearly unfair when evaluating machine learning. The purpose of this experiment is to demonstrate that supervised word alignment is a practical method for achieving high accuracy, since it can achieve accuracy that exceeds the best previously reported accuracy using a smaller amount of ground truth data (approximately 150 to 300 sentences) than the ground truth data originally created manually for evaluation.

[0288] Example 2: Effect of symmetrization To demonstrate the effectiveness of the bidirectional average (bidi-avg) symmetry method in Example 2, Figure 21 shows the word alignment accuracy for bidirectional prediction, set intersection, set union, grow-diag-final, and bidi-avg. Alignment: Word alignment accuracy is significantly affected by the orthography of the target language. In languages ​​without spaces between words, such as Japanese and Chinese, the accuracy of to-English span prediction is significantly higher than that of from-English span prediction. In such cases, grow-diag-final is better than bidi-avg. On the other hand, in languages ​​with spaces between words, such as German, Romanian, and French, there is little difference between to-English span prediction and from-English span prediction, and grow-diag-final is better than bidi-avg. For En-Fr data, set intersection yields the highest accuracy, likely due to the inherent noise in the data.

[0289] <The Importance of Source Language Context> Figure 22 shows the change in word alignment accuracy when the size of the source word context is changed. The Ja-En data was used. It shows that the source word context is very important for predicting target span.

[0290] Without context, the F1 score of Example 2 is 59.3, which is slightly higher than the F1 score of 57.6 of GIZA++. However, by providing just two words of context, the score rises to 72.0, and by providing the entire sentence as context, the score rises to 77.6.

[0291] Learning Curve Figure 23 shows the learning curve of the word matching method of Example 2 when using Zh-En data. While it goes without saying that the more training data there is, the higher the accuracy, even with a small amount of training data, the accuracy is higher than that of conventional supervised learning methods. The F1 score of 79.6 for the technology according to this embodiment when training data was 300 sentences is 6.2 points higher than the F1 score of 73.4 for the currently highest accuracy method in Reference

[20] , which was trained using 4,800 sentences.

[0292] (Summary of Example 2) As explained above, in the second embodiment, the problem of finding word correspondences between two sentences that are translations of each other is regarded as a set of problems (cross-language span prediction) of independently predicting words or consecutive word strings (spans) in a sentence in one language that correspond to each word in the sentence in another language, and highly accurate word correspondences are achieved by training a cross-language span predictor using a neural network from a small amount of manually created correct answer data (supervised learning).

[0293] The cross-language span prediction model is created by fine-tuning a pre-trained multilingual model created using only monolingual text for each of multiple languages, using a small amount of manually created correct answer data. Compared to conventional methods based on machine translation models such as Transformer, which require millions of bilingual sentence pairs for pre-training a translation model, the technology according to this embodiment can be applied to language pairs or domains with a small amount of available bilingual sentences.

[0294] In Example 2, manually created correct answer data of about 300 sentences can achieve word correspondence accuracy that exceeds that of conventional supervised learning and unsupervised learning. According to literature

[20] , correct answer data of about 300 sentences can be created in a few hours, so this embodiment can obtain highly accurate word correspondence at a realistic cost.

[0295] In Example 2, we converted the word correspondences into a general-purpose problem, the cross-lingual span prediction task in the SQuADv2.0 format, which allows us to easily incorporate multilingual pre-trained models and cutting-edge technologies in question answering to improve performance. For example, we can use XLM-RoBERTa [2] to create a more accurate model, or distilmBERT

[19] to create a more compact model that requires fewer computer resources.

[0296] [References for Example 2] [1] Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. The Mathematics of Statistical Machine Translation: Parameter Estimation. Computational Linguistics,Vol. 19, No. 2, pp. 263-311, 1993. [2] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm´an, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised Cross-lingual Representation Learning at Scale. arXiv:1911.02116, 2019. [3] Alexis Conneau and Guillaume Lample. Cross-lingual Language Model Pretraining. In Proceedings of NeurIPS-2019, pp. 7059-7069, 2019. [4] John DeNero and Dan Klein. The Complexity of Phrase Alignment Problems. In Proceedings of the ACL-2008, pp. 25-28, 2008. [5] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the NAACL-2019, pp. 4171-4186, 2019. [6] Chris Dyer, Victor Chahuneau, and Noah A. Smith. A Simple, Fast, and Effective Reparameterization of IBM Model 2. In Proceedings of the NAACL-HLT-2013, pp. 644-648, 2013. [7] Alexander Fraser and Daniel Marcu. MeasuringWord Alignment Quality for Statistical Machine Translation. Computational Linguistics, Vol. 33, No. 3, pp. 293-303, 2007. [8] Qin Gao and Stephan Vogel. Parallel Implementations of Word Alignment Tool. In Proceedings of ACL 2008 workshop on Software Engineering, Testing, and Quality Assurance for Natural Language Processing, pp. 49-57, 2008. [9] Sarthak Garg, Stephan Peitz, Udhyakumar Nallasamy, and Matthias Paulik. Jointly Learning to Align and Translate with Transformer Models. In Proceedings of the EMNLP-IJCNLP-2019, pp.4452-4461, 2019.

[10] Aria Haghighi, John Blitzer, John DeNero, and Dan Klein. Better Word Alignments with Supervised ITG Models. In Proceedings of the ACL-2009, pp. 923-931, 2009.

[11] Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. Moses: Open Source Toolkit for Statistical Machine Translation. In Proceedings of the ACL-2007, pp. 177-180, 2007.

[12] Xuansong Li, Stephen Grimes, Stephanie Strassel, Xiaoyi Ma, Nianwen Xue, Mitch Marcus, and Ann Taylor. GALE Chinese-English Parallel Aligned Treebank - Training. Web Download, 2015. LDC2015T06.

[13] Rada Mihalcea and Ted Pedersen. An Evaluation Exercise for Word Alignment. In Proceedings of the HLT-NAACL 2003 Workshop on Building and Using Parallel Texts: Data Driven Machine Translation and Beyond, pp. 1-10, 2003.

[14] Graham Neubig. Kyoto Free Translation Task alignment data package. http: / / www.phontron.com / kftt / , 2011.

[15] Franz Josef Och and Hermann Ney. Improved Statistical Alignment Models. In Proceedings of ACL-2000, pp. 440-447, 2000.

[16] Franz Josef Och and Hermann Ney. A Systematic Comparison of Various Statistical Alignment Models. Computational Linguistics, Vol. 29, No. 1, pp. 19-51, 2003.

[17] Pranav Rajpurkar, Robin Jia, and Percy Liang. Know What You Don't Know: Unanswerable Questions for SQuAD. In Proceedings of the ACL-2018, pp. 784-789, 2018.

[18] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Proceedings of EMNLP-2016, pp. 2383-2392, 2016.

[19] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv:1910.01108, 2019.

[20] Elias Stengel-Eskin, Tzu ray Su, Matt Post, and Benjamin Van Durme. A Discriminative Neural Model for Cross-Lingual Word Alignment. In Proceedings of the EMNLP-IJCNLP-2019, pp. 910-920, 2019.

[21] Akihiro Tamura, Taro Watanabe, and Eiichiro Sumita. Recurrent Neural Networks for Word Alignment Model. In Proceedings of the ACL-2014, pp. 1470-1480, 2014.

[22] Ben Taskar, Simon Lacoste-Julien, and Dan Klein. A Discriminative Matching Approach to Word Alignment. In Proceedings of the HLT-EMNLP-2005, pp. 73-80, 2005.

[23] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In Proceedings of the NIPS 2017, pp. 5998 - 6008, 2017.

[24] David Vilar, Maja Popovi´c, and Hermann Ney. AER: Do we need to "improve" our alignments? In Proceedings of IWSLT - 2006, pp. 2005 - 212, 2006.

[25] Stephan Vogel, Hermann Ney, and Christoph Tillmann. HMM - Based Word Alignment in Statistical Translation. In Proceedings of COLING - 1996, 1996.

[26] Nan Yang, Shujie Liu, Mu Li, Ming Zhou, and Nenghai Yu. Word Alignment Modeling with Context Dependent Deep Neural Network. In Proceedings of the ACL - 2013, pp. 166 - 175, 2013.

[27] Thomas Zenkel, Joern Wuebker, and John DeNero. Adding Interpretable Attention to Neural Translation Models Improves Word Alignment. arXiv:1901.11359, 2019. (Appendix) This specification discloses at least the corresponding device, learning device, corresponding method, program, and storage medium of the following appended items. Note that in appended items 1, 6, and 10 below, "using a span prediction model created using data consisting of a cross-domain span prediction problem and its answer, predicting a span that is an answer to the span prediction problem," "consisting of a cross-domain span prediction problem and its answer" relates to "data," and "created using the data" relates to "span prediction model." (Additional note 1) Memory and at least one processor coupled to said memory; Including, The processor: Taking first domain series information and second domain series information as input, generating a span prediction problem between the first domain series information and the second domain series information; A span prediction model is created using data consisting of a cross-domain span prediction problem and its answer, and a span that is the answer to the span prediction problem is predicted. Compatible devices. (Additional note 2) The span prediction model is a model obtained by additionally training a pre-trained model using the data. Item 1. (Additional note 3) the series information in the first domain series information and the second domain series information is a document; The processor determines whether a set of sentences of the first span corresponds to a set of sentences of the second span based on a probability of predicting a second span based on a question of a first span in span prediction from the first domain series information to the second domain series information, and a probability of predicting the first span based on a question of the second span in span prediction from the second domain series information to the first domain series information. 3. A device according to claim 1 or 2. (Additional note 4) The processor generates correspondences between sentence sets between the first domain series information and the second domain series information by solving an integer linear programming problem so as to minimize the sum of costs of correspondences between sentence sets between the first domain series information and the second domain series information. Item 3. (Additional note 5) Memory and at least one processor coupled to said memory; Including, The processor: generating data having a span prediction question and an answer from the correspondence data having the first domain sequence information and the second domain sequence information; Using the data, generate a span prediction model Learning device. (Additional note 6) The computer a problem generation step of receiving first domain series information and second domain series information and generating a span prediction problem between the first domain series information and the second domain series information; a span prediction step of predicting a span that is an answer to a cross-domain span prediction problem using a span prediction model created using data consisting of the cross-domain span prediction problem and its answer; How to respond. (Additional note 7) The computer a question and answer generation step of generating data including a span prediction question and its answer from correspondence data including first domain series information and second domain series information; a learning step of generating a span prediction model using the data; A learning method to do this. (Additional note 8) A program for causing a computer to function as a corresponding device according to any one of appendixes 1 to 4. (Additional note 9) A program for causing a computer to function as the learning device described in appendix 5. (Additional note 10) A non-transitory storage medium storing a program executable by a computer to execute a corresponding process, The response process includes: Taking first domain series information and second domain series information as input, generating a span prediction problem between the first domain series information and the second domain series information; A span prediction model is created using data consisting of a cross-domain span prediction problem and its answer, and a span that is the answer to the span prediction problem is predicted. Non-transitory storage medium. (Additional note 11) A non-transitory storage medium storing a program executable by a computer to perform a learning process, The learning process includes: generating data having a span prediction question and an answer from the correspondence data having the first domain sequence information and the second domain sequence information; Using the data, generate a span prediction model Non-transitory storage medium.

[0297] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]

[0298] 100 sentence compatible device 110 Cross-Lingual Span Prediction Model Training Unit 111 Sentence-related data storage unit 112 Sentence Correspondence Generation Unit 113 Sentence-corresponding pseudo-correct data storage section 114 Cross-Language Span Prediction Question Answer Generation Unit 115 Cross-Language Span Prediction Pseudo-Correct Data Storage 116 Span Prediction Model Learning Unit 117 Cross-Lingual Span Prediction Model Storage 120 Statement-based execution unit 121 Cross-lingual span prediction problem generation unit 122 Span Prediction Unit 123 Sentence Correspondence Generation Unit 200 Pre-training device 210 Multilingual Data Storage Unit 220 Multilingual Model Learning Unit 230 Pre-trained multilingual model storage 300 Word Handling Device 310 Cross-Lingual Span Prediction Model Training Unit 311 Word Correspondence Correct Answer Data Storage 312 Cross-Language Span Prediction Question Answer Generation Unit 313 Cross-Language Span Prediction Correct Answer Data Storage 314 Span Prediction Model Learning Unit 315 Cross-Lingual Span Prediction Model Storage 320 Word Correspondence Execution Unit 321 Cross-lingual span prediction problem generation unit 322 Span Prediction Unit 323 Word Correspondence Generation Unit 400 Pre-training device 410 Multilingual Data Storage 420 Multilingual Model Learning Unit 430 Pre-trained multilingual model storage 1000 Drive Device 1001 Recording media 1002 Auxiliary storage 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input Device

Claims

1. a question generator that receives a first language text and a second language text as input and generates a span prediction question between the first language text and the second language text; a span prediction unit that predicts a span that will be an answer to the span prediction question generated by the question generation unit using a span prediction model created using data consisting of span prediction questions between a first language and a second language and their answers; A corresponding device comprising:

2. The span prediction model is a model obtained by additionally training a pre-trained model using the data. The device of claim 1 .

3. a correspondence generation unit that determines whether the text of the first span and the text of the second span correspond to each other based on a probability of predicting a second span from a question of a first span in span prediction from the first language text to the second language text and a probability of predicting the first span from a question of the second span in span prediction from the second language text to the first language text.

3. The device according to claim 1, further comprising:

4. The correspondence generation unit generates the text correspondence between the first language text and the second language text by solving an integer linear programming problem so as to minimize a sum of costs of the text correspondence between the first language text and the second language text. The device of claim 3 .

5. a question and answer generation unit that generates data having span prediction questions and their answers from correspondence data indicating correspondence between spans included in the first language text and spans included in the second language text; a learning unit that generates a span prediction model using the data; A learning device comprising:

6. A response method executed by a response device, comprising: a question generation step of receiving a first language text and a second language text as input and generating a span prediction question between the first language text and the second language text; a span prediction step of predicting a span that will be an answer to the span prediction question generated in the question generation step using a span prediction model created using data consisting of span prediction questions between a first language and a second language and their answers; A response method that includes the following.

7. A learning method executed by a learning device, a question and answer generation step of generating data having span prediction questions and their answers from correspondence data indicating correspondence between spans contained in the first language text and spans contained in the second language text; a learning step of generating a span prediction model using the data; A learning method that includes:

8. A program for causing a computer to function as each unit in the corresponding device according to any one of claims 1 to 4, or a program for causing a computer to function as each unit in the learning device according to claim 5.

Images