Word alignment apparatus, learning apparatus, word alignment method, learning method and program

Abstract

To realize supervised word alignment with higher accuracy than in the conventional art from a smaller amount of supervised data than in the conventional art.SOLUTION: A word alignment device includes a problem generation unit that receives a first language sentence and a second language sentence as pieces of input and generates a cross-language span prediction problem between the first language sentence and the second language sentence, and a span prediction unit that predicts a span that is an answer to the span prediction problem by using a cross-language span prediction model created using correct answer data including the cross-language span prediction problem and the answer thereof.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to techniques for identifying word correspondences between two sentences that are translations of each other. [Background technology]

[0002] Identifying words or sets of words in two sentences that are translations of each other is called word alignment.

[0003] The technology of automatically identifying word correspondences between two mutually translated sentences has various applications in multilingual processing and machine translation. For example, by mapping annotations of named entities such as people, places, and organizations in a sentence in one language (e.g., English) to a sentence translated into another language (e.g., Japanese) based on word correspondences, training data for a named entity extractor in that language can be generated.

[0004] Conventional word alignment has relied on a method for identifying pairs of words that are translations of each other based on statistical information about bilingual data, based on the model described in reference [1], which is used in statistical machine translation. References are listed at the end of this specification. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Elias Stengel-Eskin, Tzu ray Su, Matt Post, and Benjamin Van Durme. A Discriminative Neural Model for Cross-Lingual Word Alignment. In Proceedings of the EMNLP-IJCNLP-2019, pp.910-920, 2019. Summary of the Invention [Problem to be solved by the invention]

[0006] In machine translation, neural network methods have achieved significant improvements in accuracy compared to statistical methods, but in word matching, the accuracy of neural network methods was only equal to or slightly better than that of statistical methods.

[0007] The supervised word matching based on a conventional neural machine translation model disclosed in Non-Patent Document 1 has higher accuracy than unsupervised word matching based on a statistical machine translation model. However, both the method based on the statistical machine translation model and the method based on the neural machine translation model have a problem in that they require a large amount of bilingual data (approximately several million sentences) to train the translation model.

[0008] The present invention has been made in consideration of the above-mentioned points, and has an object to realize supervised word matching with higher accuracy than the conventional techniques using a smaller amount of training data than the conventional techniques. [Means for solving the problem]

[0009] According to the disclosed technology, a span prediction unit is provided that receives a first language sentence and a second language sentence as input, and predicts a first span that is included in the second language sentence and corresponds to a first word included in the first language sentence, using a span prediction model; The span prediction model is a model obtained by performing learning using at least a first word included in a first language sentence and a second language sentence as input, and a span included in the input second language sentence and corresponding to the first word as correct answer data. A word matching device is provided. Effect of the Invention

[0010] According to the disclosed technology, it is possible to achieve more accurate supervised word matching than conventional techniques using a smaller amount of training data than conventional techniques. [Brief description of the drawings]

[0011] [Figure 1] 1 is a diagram showing a configuration of an apparatus according to an embodiment of the present invention; [Diagram 2] 1 is a flowchart showing an overall flow of processing. [Diagram 3] 13 is a flowchart illustrating a process for training a cross-language span prediction model. [Figure 4] 13 is a flowchart showing a process for generating word correspondences. [Diagram 5] FIG. 2 is a diagram illustrating a hardware configuration of the device. [Figure 6] FIG. 11 is a diagram showing an example of word correspondence data. [Figure 7] FIG. 13 is a diagram showing examples of questions from English to Japanese. [Figure 8] FIG. 13 is a diagram illustrating an example of span prediction. [Figure 9] FIG. 13 is a diagram showing an example of symmetrization of word correspondence. [Figure 10] FIG. 13 is a diagram showing the number of data items used in an experiment. [Figure 11] FIG. 1 is a diagram showing a comparison between a conventional technique and a technique according to an embodiment. [Figure 12] FIG. 13 illustrates the effect of symmetrization. [Figure 13] FIG. 1 illustrates the importance of context for source language words. [Figure 14] FIG. 13 shows word matching accuracy when trained using a subset of Chinese and English training data. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention (the present embodiment) will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0013] In this embodiment, the problem of finding word correspondence between two sentences that are translations of each other is regarded as a set of problems (cross-language span prediction) of predicting words or consecutive word strings (spans) in a sentence of one language that correspond to each word in a sentence of another language, and a highly accurate word correspondence is realized by learning a cross-language span prediction model using a neural network from a small amount of correct answer data created manually. Specifically, a word correspondence device 100, which will be described later, executes the processing related to this word correspondence.

[0014] In addition to generating training data for the named entity extractor described above, applications of word matching include, for example, the following:

[0015] When translating a web page in one language (e.g. Japanese) to another language (e.g. English), the original text may contain HTML tags (e.g. anchor tags) ... ), the HTML tags can be correctly mapped by identifying the range of strings in a sentence in another language that are semantically equivalent to the range of strings enclosed in the

[0016] In machine translation, when it is desired to specify a specific translation for a particular word in an input sentence using a bilingual dictionary or the like, the translation can be controlled by determining the word in the output sentence that corresponds to the word in the input sentence based on word correspondence, and if that word is not the specified word, replacing it with the specified word.

[0017] In the following, first, various reference techniques related to word matching will be described in order to facilitate understanding of the technology according to the present embodiment, followed by a description of the configuration and operation of the word matching device 100 according to the present embodiment.

[0018] The numbers and titles of reference documents related to the reference technology are listed at the end of the specification. In the following description, the numbers of related reference documents are indicated as "[1]" etc.

[0019] (Explanation of reference technology) <Unsupervised word matching based on statistical machine translation models> As a reference technique, first, unsupervised word matching based on a statistical machine translation model will be described.

[0020] In statistical machine translation[1], a translation model P(E|F), which converts a sentence F in a source language to a sentence E in a target language, is decomposed using Bayes' theorem into the product of a reverse translation model P(F|E) and a language model P(E), which generates a sequence of words in the target language.

[0021]

number

[0022]

number

[0023] Let X be a sequence of words of length |X|. 1:|X| =x1,x2,...,x |X| Let Y be a word sequence y of length |Y|. 1:|Y| =y1,y 2, ...,y |Y| Then, let A be the word correspondence from the target language to the source language. 1:|Y| =a1,a2,...,a |Y| Here, a j is the word y in the target sentence. j is word x in the target sentence aj This indicates that it corresponds to

[0024] In generative word alignment, the translation probability based on a word alignment A is expressed as the lexical translation probability P t (y j |...) and word correspondence probability P a (a j Decompose into a product of |...).

[0025]

number

[0026]

number

[0027] In addition, HMM-based word alignment

[25] assumes that the word alignment probability depends on the word alignment of the immediately preceding word in the target sentence.

[0028]

number

[0029] Unsupervised word alignment tools based on the model described in [1] include GIZA++

[16] , MGIZA[8], and FastAlign[6]. GIZA++ and MGIZA are based on Model 4 described in [1], and FastAlign is based on Model 2 described in [1].

[0030] <Word matching based on recurrent neural networks> Next, we explain word alignment based on recurrent neural networks. There are two types of unsupervised word alignment methods based on neural networks: applying neural networks to HMM-based word alignment [26, 21] and attention-based methods in neural machine translation [27, 9].

[0031] Regarding the method of applying neural networks to HMM-based word matching, for example, Tamura et al.

[21] used a recurrent neural network (RNN) to analyze not only the previous word matching but also the word matching history from the beginning of the sentence a< j =a 1:j-1 We propose a method to determine the target of the current word by taking into account the above, and to obtain word correspondence as a single model rather than modeling lexical translation probability and word correspondence probability separately.

[0032]

number

[0033] <Unsupervised word matching based on neural machine translation model> Next, we will explain unsupervised word matching based on a neural machine translation model. Neural machine translation realizes conversion from a source language sentence to a target language sentence based on an encoder-decoder model.

[0034] The encoder converts a source language sentence X=x of length |X| into a nonlinear transformation function enc using a neural network. 1:|X| =x1,...,x |X| Let us consider a sequence of internal states s of length |X|. 1:|X| =s1,...,s |X| If the number of dimensions of the internal state corresponding to each word is d, then s 1:|X| is a |X|×d matrix.

[0035]

number

[0036]

number

[0037] In neural machine translation, the introduction of an attention mechanism has greatly improved translation accuracy. The attention mechanism is a mechanism that determines which word information in the source language sentence should be used when generating each word in the target language sentence in the decoder by changing the weighting of the internal state of the encoder. The basic idea of ​​unsupervised word matching based on attention in neural machine translation is to consider this attention value as the probability that two words are translations of each other.

[0038] As an example, we will explain the source-target attention between source and target sentences in Transformer

[23] , a representative neural machine translation model. Transformer is an encoder-decoder model that combines self-attention and a feed-forward neural network to parallelize the encoder and decoder. The attention between source and target sentences in Transformer is called cross attention to distinguish it from self-attention.

[0039] Transformer uses scaled dot-product attention as attention. For a query Q∈R, lq×dk , key K∈R lk×dk , a value V∈R lk×dv is defined as follows:

[0040]

number

[0041] In cross-attention, Q, K, and V are Q ∈R d×dk ,W K∈R d×dk ,W V ∈R d×dv is defined as the weight as follows:

[0042]

number

[0043]

number

[0044]

number

[0045] In this case, Q=[t 1:|Y| ] T W Q The weight matrix A of the cross attention between the source and target sentences is |Y|×|X| Define.

[0046]

number

[0047]

number

[0048] Generally, a Transformer uses multiple layers and multiple heads (attention mechanisms trained from different initial values), but for simplicity, the number of layers and heads is set to one here.

[0049] Garg et al. reported that the average cross-attention of all heads in the second layer from the top was closest to the correct answer for word alignment, and the word alignment distribution G p We define the cross-entropy loss as follows for the word correspondence obtained from a specific head among multiple heads using

[0050]

number

[0051] In the method of Garg et al., when calculating the loss of word correspondence, in Eq. (10), t 1:i-1 Instead, the whole target sentence t 1:|Y| We also use the word-corresponding training data G p Instead of using self-training based on Transformer, they use word matching obtained from GIZA++. They report that this approach achieves word matching accuracy that exceeds that of GIZA++ [9].

[0052] <Supervised word matching based on neural machine translation model> Next, we explain supervised word matching based on a neural machine translation model. 1:|X| and target sentence Y=y 1:|Y| For, we define a word correspondence A as a subset of the Cartesian product of word positions.

[0053]

number

[0054] Discriminative word alignment models word alignments directly from the source and target sentences.

[0055]

number

[20] . In their method, they first define a sequence of internal states of the encoder in the neural machine translation model as s1,...,s |X| , the sequence of the decoder's internal states is denoted by t1,...,t |Y| Then we project these into a common vector space using a parameter-sharing three-layer feedforward neural network.

[0056]

number

[0057]

number

[0058]

number

[0059]

number

[0060]

number

[0061]

number

[0062] <BERT pre-trained model> Next, we will explain the pre-trained model BERT. BERT [5] is a language representation model that uses a Transformer-based encoder to output a word embedding vector for each word in the input sequence, taking into account the surrounding context. Typically, the input sequence is a sentence or two sentences concatenated with a special symbol between them.

[0063] BERT pre-trains a language representation model from large-scale language data using a task of learning a masked language model that predicts masked words in the input sequence both forward and backward, and a next sentence prediction task that determines whether two given sentences are adjacent or not. By using such pre-training tasks, BERT can output word embedding vectors that capture features related to linguistic phenomena not only within a single sentence but also across two sentences. Note that language representation models such as BERT are sometimes simply called language models.

[0064] It has been reported that by adding an appropriate output layer to the pre-trained BERT and performing transfer learning (fine tuning) with the training data of the target task, it is possible to achieve the highest accuracy in various tasks such as semantic text similarity, natural language inference (textual entailment recognition), question answering, named entity extraction, etc. Note that the above-mentioned fine tuning means using the parameters of the pre-trained BERT as the initial values ​​of the target model (a model with an appropriate output layer added to BERT) to train the target model.

[0065] In tasks that take pairs of sentences as input, such as semantic text similarity, natural language inference, and question answering, BERT is given input that combines two sentences using special symbols, such as '[CLS] first sentence [SEP] second sentence [SEP]', where [CLS] is a special token used to create a vector that aggregates information from the two input sentences, and [SEP] is a token that represents the boundary between sentences.

[0066] In tasks such as semantic text similarity (STS), which outputs a numerical value (ranging from 0 to 5 for STS) for two input sentences, we use a neural network to predict the value from the vector output by BERT for [CLS].

[0067] In tasks such as natural language inference (NLI), which involves selecting one class from multiple classes such as "entrailment," "contradiction," or "neutral" for two input sentences, we use a neural network to predict the class from the vector output by BERT for [CLS].

[0068] In a task where one input sentence is used to predict the span of the other sentence based on the other sentence, such as in question answering (QA), the vector output by BERT for [CLS] is used to predict whether or not there is a span to be extracted in the other sentence, and the vector output by BERT for each word in the other sentence is used to predict the probability that that word is the start point of the span to be extracted and the probability that that word is the end point of the span to be extracted.

[0069] BERT was originally created for English, but now BERTs for various languages, including Japanese, have been created and made publicly available. In addition, multilingual BERT, a general-purpose multilingual model created using monolingual data for 104 languages ​​extracted from Wikipedia, has been made publicly available.

[0070] Furthermore, a cross-language language model called XLM, which is pre-trained using a fill-in-the-blank language model with bilingual texts, has been proposed. It has been reported to be more accurate than multilingual BERT in applications such as cross-language text classification, and pre-trained models are publicly available [3].

[0071] (About the assignment) The conventional word matching based on a recurrent neural network and unsupervised word matching based on a neural machine translation model described as reference technologies have only been able to achieve accuracy equivalent to or slightly higher than that of unsupervised word matching based on a statistical machine translation model.

[0072] Conventional supervised word matching based on neural machine translation models has higher accuracy than unsupervised word matching based on statistical machine translation models. However, both methods based on statistical machine translation models and methods based on neural machine translation models have the problem that they require a large amount of bilingual data (on the order of millions of sentences) to train the translation model.

[0073] The technology according to the present embodiment that solves the above problems will be described below.

[0074] (Overview of the Technology Relating to the Embodiments) In this embodiment, word alignment is realized as a process of calculating answers from cross-language span prediction questions. First, a pre-trained multilingual model trained from at least each monolingual data related to a language pair to which word alignment is assigned is fine-tuned using correct answer data of cross-language span prediction created from correct answers of word alignments manually, thereby training a cross-language span prediction model. Next, word alignment processing is performed using the trained cross-language span prediction model.

[0075] By using the above-mentioned method, in this embodiment, it is possible to realize highly accurate word matching from a small amount of correct answer data of word matching created manually without requiring bilingual data for pre-training a model for executing word matching. The technology according to this embodiment will be described in more detail below.

[0076] (Example of device configuration) 1 shows a word matching device 100 and a pre-training device 200 according to the present embodiment. The word matching device 100 is a device that executes word matching processing using the technology according to the present invention. The pre-training device 200 is a device that learns a multilingual model from multilingual data.

[0077] As shown in FIG. 1, the word matching device 100 includes a cross-language span prediction model training unit 110 and a word matching execution unit 120.

[0078] The cross-language span prediction model training unit 110 has a word corresponding correct answer data storage unit 111, a cross-language span predicted question answer generation unit 112, a cross-language span predicted correct answer data storage unit 113, a span prediction model training unit 114, and a cross-language span prediction model storage unit 115. The cross-language span predicted question answer generation unit 112 may be called a question answer generation unit.

[0079] The word correspondence execution unit 120 has a cross-language span prediction question generation unit 121, a span prediction unit 122, and a word correspondence generation unit 123. Note that the cross-language span prediction question generation unit 121 may be called a question generation unit.

[0080] The pre-training device 200 is a device related to existing technology. The pre-training device 200 has a multilingual data storage unit 210, a multilingual model learning unit 220, and a pre-trained multilingual model storage unit 230. The multilingual model learning unit 220 reads monolingual texts in at least two target languages ​​for which word correspondence is to be obtained from the multilingual data storage unit 210, thereby learning a language model, and stores the language model in the pre-trained multilingual model storage unit 230 as a pre-trained multilingual model.

[0081] In this embodiment, since it is sufficient that a pre-trained multilingual model trained by some means is input to the cross-language span prediction model training unit 110, it is also possible to use, for example, a general-purpose pre-trained multilingual model that is publicly available, without providing the pre-training device 200.

[0082] The pre-trained multilingual model in this embodiment is a language model that is trained in advance using monolingual texts of at least two target languages ​​for which word correspondence is required. In this embodiment, the multilingual BERT is used as the language model, but is not limited to this. Any pre-trained multilingual model that can output a word embedding vector that takes context into account for multilingual text, such as XLM-RoBERTa, may be used.

[0083] The word matching device 100 may be called a learning device. The word matching device 100 may not include the cross-language span prediction model training unit 110, but may include the word matching execution unit 120. A device that is solely equipped with the cross-language span prediction model training unit 110 may be called a learning device.

[0084] (Overview of operation of word matching device 100) 2 is a flowchart showing the overall operation of the word matching device 100. In S100, a pre-trained multilingual model is input to the cross-language span prediction model training unit 110, which trains a cross-language span prediction model based on the pre-trained multilingual model.

[0085] In S200, the cross-language span prediction model trained in S100 is input to the word correspondence execution unit 120, and the word correspondence execution unit 120 uses the cross-language span prediction model to generate and output word correspondences for the input sentence pair (two sentences that are translations of each other).

[0086] <s100> The process of learning the cross-language span prediction model in S100 will be described with reference to the flowchart in Fig. 3. Here, it is assumed that a pre-trained multilingual model has already been input and stored in the storage device of the span prediction model learning unit 124. Also, the word-associated correct answer data storage unit 111 stores word-associated correct answer data.

[0087] In S101, the cross-language span prediction question answer generation unit 112 reads word corresponding correct answer data from the word corresponding correct answer data storage unit 111, generates cross-language span prediction correct answer data from the read word corresponding correct answer data, and stores it in the cross-language span prediction correct answer data storage unit 113. The cross-language span prediction correct answer data is data consisting of a set of pairs of cross-language span prediction questions (question and context) and their answers.

[0088] In S102, the span prediction model training unit 114 trains a cross-language span prediction model from the cross-language span prediction correct answer data and the pre-trained multilingual model, and stores the trained cross-language span prediction model in the cross-language span prediction model storage unit 115.

[0089] <s200> Next, the content of the process of generating word correspondences in S200 above will be described with reference to the flowchart in Fig. 4. Here, it is assumed that the cross-language span prediction model has already been input to the span prediction unit 122 and stored in the storage device of the span prediction unit 122.

[0090] In S201, a pair of a first language sentence and a second language sentence is input to the cross-language span prediction problem generation unit 121. In S202, the cross-language span prediction problem generation unit 121 generates a cross-language span prediction problem (question and context) from the input sentence pair.

[0091] Next, in S203, the span prediction unit 122 uses the cross-language span prediction model to perform span prediction on the cross-language span prediction question generated in S202 to obtain an answer.

[0092] In S204, the word alignment generation unit 123 generates word alignments from the answers to the cross-language span prediction questions obtained in S203. In S205, the word alignment generation unit 123 outputs the word alignments generated in S204.

[0093] It should be noted that the "model" in this embodiment is a neural network model, and specifically, is made up of weight parameters, functions, and the like.

[0094] (Hardware configuration example) Both the word matching device and the learning device (collectively referred to as "device") in this embodiment can be realized, for example, by having a computer execute a program describing the processing contents described in this embodiment. Note that this "computer" may be a physical machine or a virtual machine on the cloud. When a virtual machine is used, the "hardware" described here is virtual hardware.

[0095] The above program can be recorded on a computer-readable recording medium (such as a portable memory) and can be stored or distributed. The above program can also be provided via a network such as the Internet or e-mail.

[0096] Fig. 5 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 5 has a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, etc., which are all connected to each other via a bus B.

[0097] The program for realizing the processing on the computer is provided by a recording medium 1001 such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0098] The memory device 1003 reads out and stores the program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes the functions related to the device in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network. The display device 1006 displays a GUI (Graphical User Interface) or the like according to a program. The input device 1007 is composed of a keyboard and mouse, buttons, a touch panel, or the like, and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0099] (Explanation of specific processing contents) The process contents of the word matching device 100 in this embodiment will be described in more detail below.

[0100] <Formulation from word correspondence to span prediction> As described above, in this embodiment, the word matching process is executed as a cross-language span prediction problem process. First, the formulation from word matching to span prediction will be explained using an example. In relation to the word matching device 100, the cross-language span prediction model training unit 110 will be mainly explained here.

[0101] --About word correspondence data-- An example of Japanese and English word alignment data is shown in Figure 6. This is an example of one word alignment data. As shown in Figure 6, one word alignment data is composed of five pieces of data: a token (word) string in the first language (Japanese), a token string in the second language (English), a string of corresponding token pairs, an original text in the first language, and an original text in the second language.

[0102] Both the first language (Japanese) token sequence and the second language (English) token sequence are indexed, starting with 0, which is the index of the first element (the leftmost token) in the token sequence, and then indexed with 1, 2, 3, ...

[0103] For example, the first element of the third data, "0-1", indicates that the first element of the first language, "Ashikaga", corresponds to the second element of the second language, "ashikaga". Also, "24-2 25-2 26-2" indicates that "de", "a", and "ru" all correspond to "was".

[0104] In this embodiment, we formulate word alignment as a cross-lingual span prediction problem similar to the SQuAD-style question answering task

[18] .

[0105] A question-answering system performing an SQuAD-style question-answering task is given a "context," such as a paragraph selected from Wikipedia, and a "question," and the system predicts a "span" (a substring) within the context as the "answer."

[0106] In the same manner as the above span prediction, the word correspondence execution unit 120 in the word response device 100 of the present embodiment regards the target language sentence as a context and the words of the source language sentence as a question, and predicts the words or word strings in the target language sentence that are translations of the words of the source language sentence as the span of the target language sentence. For this prediction, the cross-language span prediction model of the present embodiment is used.

[0107] --About the cross-language span prediction question answer generation unit 112-- In this embodiment, supervised learning of the cross-language span prediction model is performed in the cross-language span prediction model learning unit 110 of the word matching device 100, and correct answer data is required for the learning.

[0108] In this embodiment, multiple word correspondence data such as that shown in FIG. 5 are stored as correct answer data in word correspondence correct answer data storage unit 111 of cross-language span prediction model training unit 110, and are used to train the cross-language span prediction model.

[0109] However, since the cross-language span prediction model is a model for predicting answers (spans) from questions across languages, data is generated for learning to predict answers (spans) from questions across languages. Specifically, by inputting word correspondence data to the cross-language span prediction question answer generation unit 112, the cross-language span prediction question answer generation unit 112 generates pairs of cross-language span prediction questions (questions) and answers (spans, substrings) in SQuAD format from the word correspondence data. An example of the processing of the cross-language span prediction question answer generation unit 112 will be described below.

[0110] FIG. 7 shows an example of converting the word correspondence data shown in FIG. 6 into a span prediction problem in the SQuAD format.

[0111] First, the upper half of Fig. 7 (a) will be described. In the upper half of Fig. 7 (context, question 1, answer portion), a sentence in the first language (Japanese) of the word correspondence data is given as the context, a token "was" in the second language (English) is given as question 1, and the answer is the span "desu" of the sentence in the first language. The correspondence between "desu" and "was" corresponds to the corresponding token pair "24-2 25-2 26-2" of the third data in Fig. 6. In other words, the cross-language span prediction question answer generator 112 generates a pair of span prediction question (question and context) and answer in the SQuAD format based on the corresponding token pair of the correct answer.

[0112] As described later, in this embodiment, the span prediction unit 122 of the word correspondence execution unit 120 uses a cross-language span prediction model to make predictions in each direction, from a first language sentence (question) to a second language sentence (answer), and from a second language sentence (question) to a first language sentence (answer). Therefore, when training the cross-language span prediction model, training is performed so that predictions can be made in both directions.

[0113] Note that performing prediction in both directions as described above is just one example. One-way prediction, such as prediction from a first language sentence (question) to a second language sentence (answer) or prediction from a second language sentence (question) to a first language sentence (answer), may also be performed. For example, in English education and the like, when English sentences and Japanese sentences are displayed simultaneously and an arbitrary character string (word string) in the English sentence is selected with a mouse or the like, a character string (word string) in the Japanese sentence that is the translation of the selected character string (word string) is calculated and displayed on the spot, one-way prediction may be performed.

[0114] Therefore, the cross-language span prediction question answer generation unit 112 of this embodiment converts one word correspondence data into a set of questions predicting spans in a sentence in a second language from each token in a first language, and a set of questions predicting spans in a sentence in a first language from each token in a second language. In other words, the cross-language span prediction question answer generation unit 112 converts one word correspondence data into a set of questions consisting of each token in a first language and their respective answers (spans in a sentence in a second language), and a set of questions consisting of each token in a second language and their respective answers (spans in a sentence in a first language).

[0115] If one token (question) corresponds to multiple spans (answers), the question is defined as having multiple answers. In other words, the cross-language span prediction question answer generation unit 112 generates multiple answers for the question. Also, if there is no span corresponding to a token, the question is defined as having no answer. In other words, the cross-language span prediction question answer generation unit 112 determines that there is no answer for the question.

[0116] In this embodiment, the language of the question is called the source language, and the language of the context and answer (span) is called the target language. In the example shown in Figure 7, the source language is English and the target language is Japanese, and the question is called an "English-to-Japanese" question.

[0117] If the question is a high-frequency word such as "of," it may appear multiple times in the source language sentence, making it difficult to find the corresponding span in the target language sentence unless the context of the word in the source language sentence is taken into consideration. Therefore, the cross-language span prediction question answer generation unit 112 of this embodiment generates questions with context.

[0118] The lower half of Figure 7(b) shows examples of questions with the context of the source language sentence. In question 2, the two tokens "Yoshimitsu ASHIKAGA" and the two tokens "the 3rd" immediately before and after the token "was" in the context of the source language sentence, respectively, are marked with '¶' as a boundary marker.

[0119] In question 3, the entire source language sentence is used as the context, and the token that is the question is sandwiched between two boundary symbols. As will be described later in the experiment, the longer the context added to the question, the better. Therefore, in this embodiment, the entire source language sentence is used as the context of the question, as in question 3.

[0120] As described above, in this embodiment, a paragraph mark '¶' is used as a boundary symbol. This symbol is called a pilcrow in English. The pilcrow belongs to the punctuation mark of the Unicode character category, is included in the vocabulary of multilingual BERT, and rarely appears in normal text. Therefore, in this embodiment, the pilcrow is used as a boundary symbol that separates the question from the context. Any character or character string that satisfies similar properties may be used as the boundary symbol.

[0121] In addition, the word alignment data contains many null alignments (no alignments). Therefore, in this embodiment, we use the formulation of SQuADv2.0

[17] . The difference between SQuADv1.1 and SQuADv2.0 is that we explicitly handle the possibility that the answer to the question does not exist in the context.

[0122] In other words, the SQuADV2.0 format explicitly indicates that a question that cannot be answered cannot be answered, so it can generate appropriate questions and answers (unable to answer) for null alignments (no alignment) in the word alignment data.

[0123] Since tokenization, including word splitting, and casing are handled differently depending on the word correspondence data, in this embodiment, the token string of the source language sentence is used only for the purpose of creating questions.

[0124] When the cross-language span prediction question answer generator 112 converts the word correspondence data into the SQuAD format, it uses the original text, not the token string, for the question and the context. That is, the cross-language span prediction question answer generator 112 generates, as an answer, the start and end positions of the span, together with the word or word string of the span from the target language sentence (context), and the start and end positions serve as indexes to the character positions of the original text of the target language sentence.

[0125] In the conventional word matching methods, a token string is often used as an input. In other words, in the example of the word matching data in Fig. 6, the first two pieces of data are often the input. In contrast, in the present embodiment, both the original text and the token string are input to the cross-language span prediction question answer generation unit 112, making the system flexible in dealing with any tokenization.

[0126] The data of pairs of cross-language span prediction questions (question and context) and answers generated by the cross-language span prediction question answer generation unit 112 is stored in the cross-language span prediction correct answer data storage unit 113.

[0127] --About the span prediction model learning unit 114-- The span prediction model training unit 114 trains the cross-language span prediction model using the correct answer data read from the cross-language span prediction correct answer data storage unit 113. That is, the span prediction model training unit 114 inputs a cross-language span prediction problem (question and context) to the cross-language span prediction model, and adjusts parameters of the cross-language span prediction model so that the output of the cross-language span prediction model becomes a correct answer. This training is performed separately for cross-language span prediction from a first language sentence to a second language sentence, and cross-language span prediction from a second language sentence to a first language sentence.

[0128] The trained cross-language span prediction model is stored in the cross-language span prediction model storage unit 115. In addition, the word correspondence execution unit 120 reads out the cross-language span prediction model from the cross-language span prediction model storage unit 115 and inputs it to the span prediction unit 122.

[0129] The cross-language span prediction model is described in detail below, as well as the processing of the word matching implementer 120.

[0130] <Cross-language span prediction using multilingual BERT> As already described, the span prediction unit 122 of the word alignment execution unit 120 in this embodiment generates word alignments from input sentence pairs using the cross-language span prediction model trained by the cross-language span prediction model training unit 110. In other words, word alignments are generated by performing cross-language span prediction on the input sentence pairs.

[0131] --About the cross-linguistic span prediction model-- In the present embodiment, the task of cross-language span prediction is defined as follows.

[0132] Source text X=x1x2...x of length |X| characters |X| and a target language sentence Y=y1y2...y of length |Y| characters |Y| Let x be the source token from character position i to character position j in the source sentence. i:j =x i ...x j , the target span y from character position k to character position l in the target sentence k:l =y k ...y l The task of cross-linguistic span prediction is to extract

[0133] The span prediction unit 122 of the word correspondence execution unit 120 executes the above tasks using the cross-language span prediction model trained by the cross-language span prediction model training unit 110. In this embodiment, multilingual BERT [5] is used as the cross-language span prediction model.

[0134] Although BERT is originally a language model created for monolingual tasks such as question answering and natural language inference, it also works very well for the cross-language tasks in this embodiment. Note that the language model used in this embodiment is not limited to BERT.

[0135] More specifically, in this embodiment, as an example, a model similar to the model for the SQuADv2.0 task disclosed in [5] is used as the cross-language span prediction model. These models (the model for the SQuADv2.0 task, the cross-language span prediction model) are models that add two independent output layers that predict the start and end positions in the context to the pre-trained BERT.

[0136] In the cross-language span prediction model, the probability that each position in the target sentence will be the start and end position of the answer span is p start and p end Let x be the source span. i:j Given the target language span y k:l Score ω X→Y ijkl is defined as the product of the probability of the starting position and the probability of the ending position, and (^k,^l) that maximizes this product is defined as the best answer span.

[0137]

number

[0138]

number

[0139] The cross-language span prediction model in this embodiment and the model for the SQuADv2.0 task disclosed in Reference [5] have basically the same neural network structure. However, the model for the SQuADv2.0 task uses a monolingual pre-trained language model and fine-tunes it with training data for a task of predicting spans between the same language, whereas the cross-language span prediction model in this embodiment uses a pre-trained multilingual model including two languages ​​related to cross-language span prediction and fine-tunes it with training data for a task of predicting spans between two languages.

[0140] In addition, while the existing implementation of the SQuAD model of BERT only outputs the answer string, the cross-language span prediction model of this embodiment is configured to be able to output the start position and the end position.

[0141] Within BERT, i.e., within the cross-lingual span prediction model of this embodiment, the input sequence is first tokenized by a tokenizer (e.g., WordPiece), and then CJK characters (Chinese characters) are split into units of one character.

[0142] In the existing implementation of the SQuAD model of BERT, the start and end positions are indices to tokens inside BERT, but in the cross-lingual span prediction model of this embodiment, they are indices to character positions. This makes it possible to handle tokens (words) of the input text for which word correspondence is required and tokens inside BERT independently.

[0143] FIG. 8 shows a process of predicting a target language (Japanese) span as an answer from the context of a target language sentence (Japanese) for a token "Yoshimitsu" in a source language sentence (English) as a question, using the cross-language span prediction model of this embodiment. As shown in FIG. 8, "Yoshimitsu" is composed of four BERT tokens. Note that a "##" (prefix) indicating a connection with the previous vocabulary is added to a BERT token, which is a token inside BERT. Also, the boundary of an input token is indicated by a dotted line. Note that in this embodiment, an "input token" and a "BERT token" are distinguished. The former is a unit of word segmentation in the training data, and is indicated by a dashed line in FIG. 8. The latter is a unit of segmentation used inside BERT, and is indicated by a space in FIG. 8.

[0144] In the example shown in FIG. 8, five possible answers are displayed: "Yoshimitsu", "Yoshimitsu (Ashikaga Yoshimitsu", "Ashikaga Yoshimitsu", "Yoshimitsu (", and "Yoshimitsu (Ashikaga Yoshi"); "Yoshimitsu" is the correct answer.

[0145] In BERT, a span is predicted for each token within BERT, so the predicted span does not necessarily match the boundaries of the input tokens (words). Therefore, in this embodiment, for a target language span that does not match the token boundaries of the target language, such as "Yoshimitsu (Ashikagayoshi", a process is performed to match a target language word that is completely included in the predicted target language span, i.e., "Yoshimitsu", "(", and "Ashikaga" in this example, with a source language token (question). This process is performed only at the time of prediction, and is performed by the word correspondence generation unit 123. During learning, learning is performed based on a loss function that compares the first candidate for span prediction with the correct answer regarding the start and end positions.

[0146] --Cross-language span prediction question generation unit 121 and span prediction unit 122-- The cross-language span prediction question generation unit 121 creates a span prediction question in the form of "[CLS]question[SEP]context[SEP]" in which the question and context are linked for each of the input first language sentence and second language sentence, for each question (input token (word)), and outputs it to the span prediction unit 122. However, as described above, the question is a contextual question that uses ¶ as a boundary symbol, such as "Yoshimitsu ASHIKAGA ¶ was ¶ the 3rd Seii Taishogun of the Muromachi Shogunate and reigned from 1368 to1394."

[0147] The cross-language span prediction problem generation unit 121 generates a span prediction problem from a first language sentence (question) to a second language sentence (answer) and a span prediction problem from a second language sentence (question) to a first language sentence (answer).

[0148] The span prediction unit 122 inputs each question (question and context) generated by the cross-language span prediction question generation unit 121, calculates the answer (predicted span) and probability for each question, and outputs the answer (predicted span) and probability for each question to the word correspondence generation unit 123.

[0149] The above probability is the product of the probability of the start position and the probability of the end position in the best answer span. The process of the word correspondence generating unit 123 will be described below.

[0150] <Symmetrization of word correspondence> In span prediction using the cross-language span prediction model of this embodiment, target language spans are predicted for source language tokens, so the source and target languages ​​are asymmetric, as in the model described in reference [1]. In this embodiment, a method for symmetrical prediction in both directions is introduced to improve the reliability of word correspondence based on span prediction.

[0151] First, for reference, a conventional example of symmetrizing word correspondence will be described. A method of symmetrizing word correspondence based on the model described in reference [1] was first proposed in reference

[16] . A representative statistical translation toolkit, Moses

[11] , implements heuristics such as intersection, union, and grow-diag-final, with grow-diag-final being the default. The intersection of two word correspondences (intersection) has high precision and low recall. The union of two word correspondences (union) has low precision and high recall. Grow-diag-final is a method of finding word correspondence that is intermediate between the intersection and union.

[0152] --About the word correspondence generation unit 123-- In this embodiment, the word alignment generation unit 123 averages the probability of the best span for each token in two directions, and if this averages a predetermined threshold or more, it is deemed to correspond. This process is performed by the word alignment generation unit 123 using the output from the span prediction unit 122 (cross-language span prediction model). As described with reference to FIG. 8, the predicted span output as an answer does not necessarily match the word boundary, so the word alignment generation unit 123 also performs a process of adjusting the predicted span so that it corresponds on a word-by-word basis in one direction. The specific process of symmetrizing word alignment is as follows.

[0153] In sentence X, the span from start position i to end position j is x i:j In sentence Y, the span from start position k to end position l is called y k:l Let token x i:j is the span y k:l The probability of predicting X→Y ijkl Let token y k:l is span x i:j The probability of predicting Y→X ijkl Let token x i:j and token y k:l Corresponding a ijkl The probability of ijkl In this embodiment, ω ijkl , x i:j The best span y predicted from ^k:^l The probability of ω X→Y ij^k^l and,y k:l The best span predicted from x ^i:^j The probability of ω Y→X ^i^jkl Calculate as the average of.

[0154]

number

[0155] The symmetrization method used in this embodiment is called bidirectional average (bidi-avg). Bidirectional average is easy to implement, and has the same effect as grow-diag-final in that it finds a word correspondence that is intermediate between a set sum and a set intersection. Note that using the average is just one example. For example, if the probability ω X→Y ij^k^l and probability ω Y→X ^i^jkl Alternatively, a weighted average of these may be used.

[0156] Figure 9 shows (a) Japanese to English span prediction and (b) English to Japanese span prediction symmetricalized by two-way averaging (c).

[0157] In the example of FIG. 9, for example, the probability ω of the best span “language” predicted from “language” X→Y ij^k^l is 0.8, and the probability of the best span "language" predicted from "language" is ω Y→X ^i^jkl is 0.6, and the average is 0.7. Since 0.7 is above the threshold, it can be determined that "lang" and "language" correspond to each other. Therefore, the word alignment generation unit 123 generates and outputs the word pair "lang" and "language" as one of the word alignment results.

[0158] In the example of FIG. 9, the word pair "is" and "de" are predicted from only one direction (from English to Japanese), but are considered to correspond because the two-way average probability is equal to or exceeds a threshold.

[0159] The threshold of 0.4 was determined through a preliminary experiment in which the learning data for Japanese and English word correspondences, described below, was divided in half, with one half used as training data and the other as test data. This value was used in all experiments described below. Since span predictions for each direction are performed independently, it may be necessary to normalize the scores for symmetry, but in the experiment, both directions were trained with a single model, so normalization was not necessary.

[0160] (Effects of the embodiment) The word matching device 100 described in this embodiment does not require a large amount of bilingual data for the language pair to which word matching is to be assigned, and can achieve more accurate supervised word matching than before using a smaller amount of supervised data (manually created correct answer data) than before.

[0161] (About the experiment) In order to evaluate the technology according to the present embodiment, a word matching experiment was carried out, and the experimental method and results will be described below.

[0162] <About the experimental data> Figure 10 shows the number of sentences in the training data and test data for the gold word alignments created manually for five language pairs: Chinese-English (Zh-En), Japanese-English (Ja-En), German-English (De-En), Romanian-English (Ro-En), and English-French (En-Fr). The table in Figure 10 also shows the amount of reserved data.

[0163] In the experiment using the conventional technique

[20] , Zh-En data was used, and in the experiment using the conventional technique [9], De-En, Ro-En, and En-Fr data were used. In the experiment using the technique of this embodiment, Ja-En data, which is one of the most distant language pairs in the world, was added.

[0164] The Zh-En data was obtained from the GALE Chinese-English Parallel Aligned Treebank

[12] , and includes broadcasting news, news wires, and web data. In order to approximate the experimental conditions described in

[20] as closely as possible, we used bilingual texts in which Chinese characters were tokenized, cleaned to remove mismatches and timestamps, and randomly divided the data into 80% training data, 10% test data, and 10% reserve data.

[0165] The KFTT word correspondence data

[14] was used as the Japanese-English data. Kyoto Free Translation Task (KFTT) (http: / / www.phontron.com / kftt / index.html) is a manual translation of Japanese Wikipedia articles about Kyoto, and is composed of 440,000 sentences of training data, 1166 sentences of development data, and 1160 sentences of test data. The KFTT word correspondence data is a part of the KFTT development data and test data that has been manually assigned word correspondences, and consists of 8 development data files and 7 test data files. In the experiment of the technology according to this embodiment, 8 development data files were used for training, 4 of the test data files were used for testing, and the rest were reserved.

[0166] The De-En, Ro-En, and En-Fr data are described in

[27] . The authors have published scripts for preprocessing and evaluation (https: / / github.com / lilt / alignment-scripts). In the prior art [9], these data are used for experiments. The De-En data is described in

[24] (https: / / www-i6.informatik.rwth-aachen.de / goldAlignment / ). The Ro-En and En-Fr data were provided as common tasks in the HLT-NAACL-2003 workshop on Building and Using Parallel Texts

[13] (https: / / eecs.engin.umich.edu / ). The En-Fr data was originally described in

[15] . The number of sentences in the De-En, Ro-En, and En-Fr data is 508,248,447. In this embodiment, 300 sentences were used for training for De-En and En-Fr, and 150 sentences were used for training for Ro-En. The remaining sentences were used for testing.

[0167] <Evaluation scale for word matching accuracy> In this embodiment, the F1 score, which has equal weighting on precision and recall, is used as an evaluation measure for word correspondence.

[0168]

number

[16] , we also use the AER to compare the prior art with the technique of this embodiment.

[0169] Let a manually created gold word alignment consist of sure,S,and possible,P,, where S ⊆ P. We define the precision, recall, and AER of a word alignment,A,as follows:

[0170]

number

[0171]

number

[0172]

number

[0173] <Comparison of word matching accuracy> A comparison between the technology according to this embodiment and the conventional technology is shown in Figure 11. For all five data points, the technology according to this embodiment is superior to all the conventional technologies.

[0174] For example, on the Zh-En data, the technique according to the present embodiment achieves an F1 score of 86.7, which is 13.3 points higher than the F1 score of 73.4 of DiscAlign reported in

[20] , the current state-of-the-art accuracy for supervised word alignment. The method in

[20] uses 4 million bilingual sentence pairs to pre-train the translation model, whereas the technique according to the present embodiment does not require bilingual data for pre-training. On the Ja-En data, the technique according to the present embodiment achieves an F1 score of 77.6, which is 20 points higher than the F1 score of 57.8 of GIZA++.

[0175] For the De-EN, Ro-EN, and En-Fr data, the method that currently achieves the highest accuracy in unsupervised word matching reports only AER [9], so this embodiment also evaluates AER. For comparison, we also report the AER of MGIZA and other conventional methods for the same data [22, 10].

[0176] In the experiment, both sure and possible word correspondences for the De-En data were used for training the present embodiment, but only sure was used for the En-Fr data because it was very noisy. The AER of the present embodiment for the De-En, Ro-En, and En-Fr data was 11.4, 12.2, and 4.0, which is clearly lower than the method in reference [9].

[0177] Comparing the accuracy of supervised learning with that of unsupervised learning is clearly an unfair way of evaluating machine learning. The purpose of this experiment is to show that supervised word alignment is a practical method for achieving high accuracy, since it can achieve an accuracy higher than the best accuracy reported so far using a smaller amount of correct answer data (about 150 to 300 sentences) than the correct answer data originally created by hand for evaluation.

[0178] <Effect of symmetrization> In order to show the effectiveness of the bidirectional average (bidi-avg) which is the symmetrization method in this embodiment, FIG. 12 shows the word correspondence accuracy of two-way prediction, set intersection, set union, grow-diag-final, and bidi-avg. Alignment Word correspondence accuracy is greatly affected by the orthography of the target language. In languages ​​such as Japanese and Chinese that do not have spaces between words, the span prediction accuracy to English is significantly higher than the span prediction accuracy from English. In such cases, grow-diag-final is better than bidi-avg. On the other hand, in languages ​​that have spaces between words such as German, Romanian, and French, there is no significant difference between the span prediction to English and the span prediction from English, and grow-diag-final is better than bidi-avg. In En-Fr data, the set intersection has the highest accuracy, but this is thought to be because the data is originally noisy.

[0179] <The Importance of Source Language Context> Figure 13 shows the change in word matching accuracy when the size of the source word context is changed. We used the Ja-En data. We can see that the source word context is very important in predicting the target span.

[0180] When there is no context, the F1 score of this embodiment is 59.3, which is slightly higher than the F1 score of 57.6 of GIZA++. However, if only two words of context are provided, the score becomes 72.0, and if the entire sentence is provided as context, the score becomes 77.6.

[0181] Learning curve Figure 14 shows the learning curve of the word matching method of this embodiment when using Zh-En data. Naturally, the more training data there is, the higher the accuracy, but even with a small amount of training data, the accuracy is higher than that of conventional supervised learning methods. The F1 score of 79.6 for the training data of 300 sentences is 6.2 points higher than the F1 score of 73.4 for the currently highest accuracy method in literature

[20] , which was trained using 4,800 sentences.

[0182] (Summary of the embodiment) As described above, in this embodiment, the problem of determining word correspondence between two sentences that are translations of each other is treated as a set of problems (cross-language span prediction) of independently predicting words or consecutive word strings (spans) in a sentence in one language that correspond to each word in a sentence in another language, and highly accurate word correspondence is achieved by training a cross-language span predictor using a neural network from a small amount of manually created correct answer data (supervised learning).

[0183] The cross-language span prediction model is created by fine-tuning a pre-trained multilingual model created using only monolingual texts for each of multiple languages, using a small amount of manually created correct answer data. Compared to conventional methods based on machine translation models such as Transformer, which require millions of bilingual sentence pairs for pre-training a translation model, the technology according to this embodiment can be applied to language pairs or areas with a small amount of available bilingual sentences.

[0184] In this embodiment, if manually created correct answer data of about 300 sentences is enough, it is possible to achieve a word matching accuracy that exceeds that of conventional supervised learning and unsupervised learning. According to literature

[20] , correct answer data of about 300 sentences can be created in a few hours, so this embodiment can obtain highly accurate word matching at a realistic cost.

[0185] In addition, in this embodiment, by converting word correspondence into a general-purpose problem, a cross-lingual span prediction task in the SQuADv2.0 format, it is possible to easily incorporate multilingual pre-trained models and cutting-edge technologies related to question answering to improve performance. For example, it is possible to use XLM-RoBERTa[2] to create a model with higher accuracy, or distilmBERT

[19] to create a compact model that operates with fewer computer resources.

[0186] (Additional Note) This specification discloses at least the word matching device, learning device, word matching method, program, and storage medium in the following appended claims. Note that in appended claims 1, 7, and 11 below, "using a cross-language span prediction model created using correct answer data consisting of a cross-language span prediction question and its answer, predicting a span that is an answer to the span prediction question," "consisting of a cross-language span prediction question and its answer" relates to "correct answer data," and "created using the correct answer data..." relates to the "cross-language span prediction model." (Additional note 1) Memory, at least one processor coupled to the memory; Including, The processor, A first language sentence and a second language sentence are input, and a cross-language span prediction problem between the first language sentence and the second language sentence is generated; A cross-language span prediction model is created using correct answer data consisting of cross-language span prediction questions and their answers, and a span that is an answer to the span prediction question is predicted using the cross-language span prediction model. Word correspondence device. (Additional note 2) The cross-language span prediction model is a model obtained by additionally training a pre-trained multilingual model using the correct answer data consisting of the cross-language span prediction questions and their answers. 2. A word matching device according to claim 1. (Additional note 3) The processor, when predicting a span that is an answer to the span prediction problem, performing a bidirectional span prediction from the first language sentence to the second language sentence and from the second language sentence to the first language sentence; or Performing one-way prediction consisting of only span prediction from the first language sentence to the second language sentence or only span prediction from the second language sentence to the first language sentence. 3. The word matching device according to claim 1 or 2. (Additional note 4) The processor determines whether or not a word of the first span corresponds to a word of the second span based on a probability of predicting a second span by a question of a first span in span prediction from the first language sentence to the second language sentence, and a probability of predicting the first span by a question of the second span in span prediction from the second language sentence to the first language sentence. 4. A word matching device according to claim 3. (Additional note 5) Memory, at least one processor coupled to the memory; Including, The processor, Generate cross-language span prediction questions and their answers as correct answer data from word correspondence data having a first language sentence, a second language sentence, and word correspondence information; Using the correct answer data, a cross-language span prediction model is generated. Learning device. (Additional note 6) The span prediction problem has a question and a context, and the question is a context-based question to which the context of the language of the question is added via a boundary symbol. 6. The learning device according to claim 5. (Additional note 7) The computer a problem generation step of receiving a first language sentence and a second language sentence as input and generating a cross-language span prediction problem between the first language sentence and the second language sentence; a span prediction step of predicting a span that is an answer to the span prediction question using a cross-language span prediction model created using correct answer data consisting of a cross-language span prediction question and its answer; A word correspondence method. (Additional note 8) A learning method executed by a learning device, comprising: a question answer generating step of generating cross-language span prediction questions and their answers as correct answer data from word correspondence data having a first language sentence, a second language sentence, and word correspondence information; A learning step of generating a cross-language span prediction model using the correct answer data; A learning method that provides: (Additional note 9) A program for causing a computer to function as each unit in the word matching device according to any one of claims 1 to 4. (Additional note 10) A program for causing a computer to function as each part of the learning device described in appendix 5 or 6. (Additional note 11) A non-transitory storage medium storing a program executable by a computer to execute a word matching process, The word matching process includes: A first language sentence and a second language sentence are input, and a cross-language span prediction problem between the first language sentence and the second language sentence is generated; A cross-language span prediction model is created using correct answer data consisting of cross-language span prediction questions and their answers, and a span that is an answer to the span prediction question is predicted using the cross-language span prediction model. Non-transitory storage medium. (Additional note 12) A non-transitory storage medium storing a program executable by a computer to execute a learning process, The learning process includes: Generate cross-language span prediction questions and their answers as correct answer data from word correspondence data having a first language sentence, a second language sentence, and word correspondence information; Using the correct answer data, a cross-language span prediction model is generated. Non-transitory storage medium.

[0187] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and variations are possible within the scope of the gist of the present invention described in the claims.

[0188] (References) [1] Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. The Mathematics of Statistical Machine Translation: Parameter Estimation. Computational Linguistics,Vol. 19, No. 2, pp. 263-311, 1993. [2] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm´an, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised Cross-lingual Representation Learning at Scale. arXiv:1911.02116, 2019. [3] Alexis Conneau and Guillaume Lample. Cross-lingual Language Model Pretraining. In Proceedings of NeurIPS-2019, pp. 7059-7069, 2019. [4] John DeNero and Dan Klein. The Complexity of Phrase Alignment Problems. In Proceedings of the ACL-2008, pp. 25-28, 2008. [5] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the NAACL-2019, pp. 4171-4186, 2019. [6] Chris Dyer, Victor Chahuneau, and Noah A. Smith. A Simple, Fast, and Effective Reparameterization of IBM Model 2. In Proceedings of the NAACL-HLT-2013, pp. 644-648, 2013. [7] Alexander Fraser and Daniel Marcu. MeasuringWord Alignment Quality for Statistical Machine Translation. Computational Linguistics, Vol. 33, No. 3, pp. 293-303, 2007. [8] Qin Gao and Stephan Vogel. Parallel Implementations of Word Alignment Tool. In Proceedings of ACL 2008 workshop on Software Engineering, Testing, and Quality Assurance for Natural Language Processing, pp. 49-57, 2008. [9] Sarthak Garg, Stephan Peitz, Udhyakumar Nallasamy, and Matthias Paulik. Jointly Learning to Align and Translate with Transformer Models. In Proceedings of the EMNLP-IJCNLP-2019, pp.4452-4461, 2019.

[10] Aria Haghighi, John Blitzer, John DeNero, and Dan Klein. Better Word Alignments with Supervised ITG Models. In Proceedings of the ACL-2009, pp. 923-931, 2009.

[11] Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. Moses: Open Source Toolkit for Statistical Machine Translation. In Proceedings of the ACL-2007, pp. 177-180, 2007.

[12] Xuansong Li, Stephen Grimes, Stephanie Strassel, Xiaoyi Ma, Nianwen Xue, Mitch Marcus, and Ann Taylor. GALE Chinese-English Parallel Aligned Treebank - Training. Web Download, 2015. LDC2015T06.

[13] Rada Mihalcea and Ted Pedersen. An Evaluation Exercise for Word Alignment. In Proceedings of the HLT-NAACL 2003 Workshop on Building and Using Parallel Texts: Data Driven Machine Translation and Beyond, pp. 1-10, 2003.

[14] Graham Neubig. Kyoto Free Translation Task alignment data package. http: / / www.phontron.com / kftt / , 2011.

[15] Franz Josef Och and Hermann Ney. Improved Statistical Alignment Models. In Proceedings of ACL-2000, pp. 440-447, 2000.

[16] Franz Josef Och and Hermann Ney. A Systematic Comparison of Various Statistical Alignment Models. Computational Linguistics, Vol. 29, No. 1, pp. 19-51, 2003.

[17] Pranav Rajpurkar, Robin Jia, and Percy Liang. Know What You Don't Know: Unanswerable Questions for SQuAD. In Proceedings of the ACL-2018, pp. 784-789, 2018.

[18] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Proceedings of EMNLP-2016, pp. 2383-2392, 2016.

[19] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv:1910.01108, 2019.

[20] Elias Stengel-Eskin, Tzu ray Su, Matt Post, and Benjamin Van Durme. A Discriminative Neural Model for Cross-Lingual Word Alignment. In Proceedings of the EMNLP-IJCNLP-2019, pp. 910-920, 2019.

[21] Akihiro Tamura, Taro Watanabe, and Eiichiro Sumita. Recurrent Neural Networks for Word Alignment Model. In Proceedings of the ACL-2014, pp. 1470-1480, 2014.

[22] Ben Taskar, Simon Lacoste-Julien, and Dan Klein. A Discriminative Matching Approach to Word Alignment. In Proceedings of the HLT-EMNLP-2005, pp. 73-80, 2005.

[23] Ashish Vaswani, Noam Shazeer, Nick Parmar, Jacob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In Proceedings of the NIPS 2017, pp. 101-1 5998-6008,

[24] David Vilar, Maja Popovic, and Hermann Ney. AER: Do we need to "improve" our alignments? In Proceedings of IWSLT-2006, pp. 101-1 2005–212,

[25] Stephen Vogel, Hermann Ney, and Christoph Tillmann. HMM-Based Word Alignment in Statistical Translation. In Proceedings of COLING-1996, 1996.

[26] Nan Yang, Shujie Liu, Mu Li, Ming Zhou, and Nenghai Yu. Word Alignment Modeling with Context Dependent Deep Neural Network. In Proceedings of the ACL-2013, pp. 107-111. 166–175,

[27] Thomas Zenkel, Joern Wuebker, and John DeNero. Adding Interpretive Attention to Neural Translation Models Improves Word Alignment. arXiv:1901.11359, 2019.

Internet Application

[0189] 100 Environmental Protection 110 Cross-Lingual Span Prediction Model Training Unit 111 Word Correspondence Correct Answer Data Storage Unit 112 Cross-Language Span Prediction Question Answer Generation Unit 113 Cross-Language Span Prediction Correct Answer Data Storage 114 Span Prediction Model Learning Unit 115 Cross-Lingual Span Prediction Model Storage 120 Word Response Execution Unit 121 Monolingual Span Prediction Problem Generation Unit 122 Span Prediction Unit 123 Word Correspondence Generation Unit 200 Pre-learning device 210 Multilingual Data Storage Unit 220 Multilingual Model Learning Unit 230 Pre-trained multilingual model storage 1000 Drive Device 1001 Recording media 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input Device

Claims

1. a span prediction unit that receives a first language sentence and a second language sentence as input, and predicts a first span that is included in the second language sentence and corresponds to a first word included in the first language sentence, using a span prediction model; The span prediction model is a model obtained by performing learning using at least a first word included in a first language sentence and a second language sentence as input, and a span included in the input second language sentence and corresponding to the first word as correct answer data. Word correspondence device.

2. a span prediction unit that receives a first language sentence and a second language sentence as input, and predicts a first span that is included in the second language sentence and corresponds to a first word included in the first language sentence, using a span prediction model; The span prediction model is a model obtained by performing supervised learning. Word correspondence device.

3. The span prediction unit further predicts a second span that is included in the first language sentence and corresponds to a second word that is included in the second language sentence, using the span prediction model.

2. The word matching device of claim 1.

4. a word correspondence generating unit that associates words included in a first language sentence with words included in a second language sentence based on the prediction result of the first span and the prediction result of the second span; The word matching device of claim 3 further comprising:

5. The first word and the context information are distinguished by a predetermined symbol.

2. The word matching device of claim 1.

6. a question answer generation unit that receives word correspondence information, a first language sentence, and a second language sentence as input and generates a correct answer that is a span included in the second language sentence that corresponds to a word included in the first language sentence; a learning unit that uses learning data having the correct answer to learn a span prediction model; A learning device comprising:

7. Words and context information in the first language sentence are distinguished by predetermined symbols. The learning device according to claim 6.

8. A word matching method executed by a word matching device, comprising: a span prediction step of receiving a first language sentence and a second language sentence as input, and predicting a first span that is included in the second language sentence and corresponds to a first word included in the first language sentence, using a span prediction model; The span prediction model is a model obtained by performing learning using at least a first word included in a first language sentence and a second language sentence as input, and a span included in the input second language sentence and corresponding to the first word as correct answer data. Word correspondence method.

9. A word matching method executed by a word matching device, comprising: a span prediction step of receiving a first language sentence and a second language sentence as input, and predicting a first span that is included in the second language sentence and corresponds to a first word included in the first language sentence, using a span prediction model; The span prediction model is a model obtained by performing supervised learning. Word correspondence method.

10. A learning method executed by a learning device, comprising: a question answer generation step of generating a correct answer which is a span included in the second language sentence corresponding to a word included in the first language sentence, using word correspondence information, a first language sentence, and a second language sentence as input; a learning step of learning a span prediction model using the learning data having the correct answer; A learning method that provides:

11. a program for causing a computer to function as a span prediction unit that receives a first language sentence and a second language sentence as input, and predicts a first span that is included in the second language sentence and corresponds to a first word included in the first language sentence, using a span prediction model; The span prediction model is a model obtained by performing learning using at least a first word included in a first language sentence and a second language sentence as input, and a span included in the input second language sentence and corresponding to the first word as correct answer data. program.

12. a program for causing a computer to function as a span prediction unit that receives a first language sentence and a second language sentence as input, and predicts a first span that is included in the second language sentence and corresponds to a first word included in the first language sentence, using a span prediction model; The span prediction model is a model obtained by performing supervised learning. program.

13. Computer, a question answer generation unit that receives word correspondence information, a first language sentence, and a second language sentence as input and generates a correct answer that is a span contained in the second language sentence that corresponds to a word contained in the first language sentence; a learning unit that learns a span prediction model using learning data having the correct answer; A program to function as a

Images