Alignment device and program

Abstract

To provide an alignment device and a program capable of realizing a high accuracy alignment.SOLUTION: An alignment device comprises a generating part of a key distributed representation and an alignment score calculation part. The generating part of the key distributed representation generates the key distributed representation of input side words included in an input side sentence of given parallel translation sentence pair. The alignment score calculation part calculates the alignment score of the input side word and the output side word based on the degree of similarity between aligned distributed representation corresponding to the distributed representation for the alignment of the output side word included in the output side sentence different from the input side sentence in the parallel translation sentence pair and the key distributed representation of the input side word.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an alignment device and a program. [Background technology]

[0002] There is a need to automatically find the correspondence between words in each sentence included in a bilingual sentence pair.

[0003] A word alignment estimation method is described in Non-Patent Document 1. The method described in Non-Patent Document 1 does not use distributed representations corresponding to words.

[0004] Non-Patent Documents 2, 3, and 4 describe neural machine translation techniques. Non-Patent Documents 2 and 3 describe RNN-based neural machine translation techniques. RNN stands for recurrent neural network. Non-Patent Document 4 describes a Transformer-based neural machine translation technique.

[0005] Machine translation processing based on neural networks (Neural Machine Translation, NMT) is widely used. In neural machine translation, a word sequence corresponding to a target sentence is output in order, starting with the first word. Neural machine translation techniques use a probability value called attention probability for translation processing. The attention probability is a probability value for each word in a word sequence in an input sentence, i.e., a source sentence, and serves as a prediction of the next source word to be translated. In other words, when a source sentence is translated using neural machine translation to generate a target sentence (target word sequence), the correspondence between the words in the source sentence and those in the target sentence can be determined probabilistically based on the attention probability of each word in the output target word sequence. In other words, each word in the target sentence is assigned an attention probability for each word in the source sentence. Therefore, for example, it can be estimated that a certain word in the target sentence corresponds to the word in the source sentence with the highest attention probability.

[0006] One possible method is to use the attention probabilities in neural machine translation described above to determine the correspondence between words in an existing bilingual text pair. In this method, the source language sentences included in the bilingual text pair are input to the neural machine translation process. In the neural machine translation process, the target language sentences included in the bilingual text pair are forcibly output, and the attention probabilities corresponding to each word are calculated. These attention probabilities include information about the word correspondence between the source language and the target language.

[0007] However, when attention probability is calculated as described above, there is a problem that the quality of alignment estimation is low because information about the output words in the target language is not used to estimate the alignment.To solve this problem, the method described in Non-Patent Document 5 uses information about the output target language words to calculate attention probability. [Prior art documents] [Non-patent literature]

[0008] [Non-Patent Document 1] Franz Josef Och, Hermann Ney, A systematic comparison of various statistical alignment models, Computational Linguistics, Vol. 29, No. 1, pp.19-51, 2003. [Non-Patent Document 2] Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio, Neural machine translation by jointly learning to align and translate, Proceedings of ICLR, 2015. [Non-Patent Document 3] Thang Luong, Hieu Pham, Christopher D. Manning, Effective approaches to attention-based neural machine translation, Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp.1412-1421, Lisbon, Portugal, September 2015, Association for Computational Linguistics. [Non-Patent Document 4] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin, Attention is all you need, Advances in Neural Information Processing Systems 30, pp.5998-6008, Curran Associates, Inc., 2017. [Non-Patent Document 5] Jan-Thorsten Peter, Arne Nix, Hermann Ney, Generating alignments using target foresight in attention-based neural machine translation, PBML, Vol. 108, No. 1, pp. 27-36, 2017. Summary of the Invention [Problem to be solved by the invention]

[0009] The above-described prior art has problems to be solved.

[0010] The method described in Non-Patent Document 1 does not use embedded representations corresponding to words, so there is a problem that, for example, when a word has two or more translations, the accuracy of estimating correspondences to minor translations is low. Without using embedded representations, if multiple words have different surface representations, statistics cannot be shared between those multiple words, making it difficult to estimate correspondences to minor translations.

[0011] In the method described in Non-Patent Document 5, if the distributed representations of target language words are added to the calculation of attention probability and all parameters in the neural machine translation process are optimized, information about the target language output words is directly embedded in the parameters, resulting in ineffective training of attention probability. Therefore, the method described in Non-Patent Document 5 fixes most of the parameter values ​​in the neural machine translation process and retrains only some of the parameters involved in the calculation of attention probability. The target language distributed representations used in this method embed information about the next target language word to be output using backpropagation. In other words, these distributed representations embed information about the next word in the target language, not information about the source language word in the bilingual translation. Therefore, there is a problem in that the information is not very useful for calculating word correspondence between the source language and the target language.

[0012] The present invention was developed based on the recognition of the above-mentioned problems, and aims to provide an alignment device and program that can handle minor translations and achieve highly accurate alignment by constructing distributed representations of words in the target language that contain information about words in the source language. [Means for solving the problem]

[0013] [1] In order to solve the above problem, an alignment device according to one aspect of the present invention comprises a key distributed representation generation unit that generates a key distributed representation of an input word included in an input sentence of a given bilingual sentence pair, and an alignment score calculation unit that calculates an alignment score between the input word and the output word based on the similarity between an aligned distributed representation, which is a distributed representation for aligning an output word included in an output sentence that is different from the input sentence of the bilingual sentence pair, and the key distributed representation of the input word.

[0014] [2] Furthermore, one aspect of the present invention is that the above-mentioned alignment device further includes an attention probability calculation unit that calculates an attention probability representing the probability that the input word corresponds given the output word, and the alignment score calculation unit calculates the alignment score between the input word and the output word based also on the attention probability calculated by the attention probability calculation unit.

[0015] [3] Furthermore, one aspect of the present invention is that the above-mentioned alignment device further comprises an attention probability calculation unit that calculates an attention probability representing the probability that the input word corresponds given the output word, a weighted key distributed representation calculation unit that calculates, for each output word, a weighted key distributed representation, which is the key distributed representation of the input word weighted by the attention probability, and an aligned distributed representation calculation unit that calculates the aligned distributed representation for each word type based on the weighted key distributed representation for each output word, and the alignment score calculation unit calculates the alignment score based on the aligned distributed representation calculated by the aligned distributed representation calculation unit.

[0016] [4] Also, in one aspect of the present invention, in the above-mentioned alignment device, the aligned distributed representation calculation unit calculates the aligned distributed representation by calculating the average value of the weighted key distributed representation for each word type.

[0017] [5] Furthermore, in one aspect of the present invention, in the above-mentioned alignment device, the aligned distributed representation calculation unit estimates and obtains the aligned distributed representation for the output word so that an evaluation value based on the similarity between the weighted key distributed representation corresponding to the output word and the aligned distributed representation for the output word is optimized.

[0018] [6] Another aspect of the present invention is a program for causing a computer to function as an alignment device, comprising: a key distributed representation generation unit that generates key distributed representations of input words included in input sentences of a given bilingual sentence pair; and an alignment score calculation unit that calculates an alignment score between the input word and the output word based on the similarity between aligned distributed representations, which are distributed representations for aligning output words included in output sentences that are different from the input sentences of the bilingual sentence pair, and the key distributed representations of the input words. [Effects of the Invention]

[0019] According to the present invention, the aligned distributed representations are distributed representations corresponding to words in the target language and contain information about words in the source language. The alignment device can achieve highly accurate alignment based on such aligned distributed representations. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a first block diagram showing a schematic functional configuration of an alignment apparatus according to an embodiment of the present invention. [Figure 2] FIG. 2 is a second block diagram showing a schematic functional configuration of an alignment device according to an embodiment of the present invention. [Figure 3] 2 is a schematic diagram showing an example of bilingual text pair data to be processed by the alignment device according to the embodiment. FIG. [Figure 4] 4 is a schematic diagram showing an example of the results of the alignment device according to the embodiment determining the correspondence between words in the bilingual sentences shown in FIG. 3. FIG. [Figure 5] FIG. 2 is a block diagram showing an example of the internal configuration of the alignment device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0021] Next, an embodiment of the present invention will be described with reference to the drawings.

[0022] 1 and 2 are block diagrams showing a schematic functional configuration of an alignment device according to this embodiment. The alignment device 1 includes a bilingual sentence pair supply unit 11, a key distributed representation generation unit 13, a query distributed representation generation unit 14, a value distributed representation generation unit 15, an attention probability calculation unit 18, a weighted key distributed representation calculation unit 21, a weighted value distributed representation calculation unit 22, an aligned distributed representation calculation unit 25, an alignment score calculation unit 26, an output unit 29, a source language sentence input unit 31, a distributed representation conversion unit 32, a target language sentence input unit 33, a distributed representation conversion unit 34, an aligned distributed representation conversion unit 35, and a target language word generation probability calculation unit 37 (decoder). Note that the bilingual sentence pair supply unit 11 and the output unit 29 are omitted from FIGS. 1 and 2.

[0023] Fig. 1 shows the functional configuration used by the alignment apparatus 1 in the processing phase of constructing aligned distributed representations, while Fig. 2 shows the functional configuration used by the alignment apparatus 1 in the processing phase of calculating word correspondences.

[0024] As shown in Figure 1, functional units used in the process phase of constructing aligned shared representations include a key shared representation generation unit 13, a query shared representation generation unit 14, a value shared representation generation unit 15, an attention probability calculation unit 18, a weighted key shared representation calculation unit 21, a weighted value shared representation calculation unit 22, an aligned shared representation calculation unit 25, a source language sentence input unit 31, a shared representation conversion unit 32, a target language sentence input unit 33, a shared representation conversion unit 34, and a target language word generation probability calculation unit 37. In the aligned shared representation construction phase, the alignment device 1 constructs aligned shared representations based on a large number of bilingual sentence pairs. The aligned shared representation calculation unit 25 outputs aligned shared representations. These aligned shared representations are referenced in the word correspondence calculation phase shown in Figure 2.

[0025] As shown in Figure 2, the functional units used in the processing phase for calculating word alignment include a key shared representation generation unit 13, an aligned shared representation calculation unit 25, a source language sentence input unit 31, a shared representation conversion unit 32, a target language sentence input unit 33, and an aligned shared representation conversion unit 35. In the word alignment calculation phase, the alignment apparatus 1 calculates alignment scores for the input bilingual text pairs. In other words, the alignment apparatus 1 estimates the alignment of the bilingual text pairs.

[0026] Each of these functional units can be realized, for example, by a computer and a program. Each functional unit also has a storage means, as needed. The storage means is, for example, a memory for storing program variables or allocated by the execution of the program. Non-volatile storage means, such as a magnetic hard disk drive or a solid-state drive (SSD), may also be used, as needed. At least some of the functions of each functional unit may also be realized as a dedicated electronic circuit, rather than as a program.

[0027] The alignment device 1 of this embodiment calculates alignment between bilingual sentences. The alignment device 1 includes an encoder (not shown) and a decoder (not shown) for translation processing. The encoder and decoder are implemented using neural networks. The alignment device 1 calculates attention probability based on the similarity between a distributed representation (key distributed representation) obtained by linearly transforming the distributed representation of the input word (word in the source language sentence) and a distributed representation (query distributed representation) obtained by linearly transforming the distributed representation of the decoder.

[0028] The alignment device 1 operates based on the supplied bilingual sentences and estimates the correspondence (alignment) between words in the bilingual sentence pair. The alignment device 1 forces the decoder to output a word sequence corresponding to the target language sentence. In other words, the alignment device 1 determines the target language word as output according to the given bilingual sentence pair, regardless of the probability of the target language word calculated by the decoder. The attention probability is a value that predicts the source language word to be translated next, but when a bilingual sentence is given, the target language word to be output next is already determined, so the alignment device 1 utilizes this information.

[0029] In neural machine translation (NMT), a translation model is trained so that information about the next word in the target language is embedded in the embedded representation of a word in the target language. However, the information that the alignment device 1 needs for alignment processing is information about the bilingual (source language) word that corresponds to the word in the target language, and information about the next word in the target language is not very useful. Therefore, the alignment device 1 uses a method of calculating embedded representations that embed information about the bilingual words required to calculate the alignment.

[0030] In this embodiment, the distributed representations corresponding to each vocabulary (word type) in the target language for calculating alignment are called aligned distributed representations. As described above, the attention probability is a value for predicting the next word to be translated. Therefore, for each bilingual sentence pair, if a source language sentence is input and its corresponding target language sentence is forcibly output, the prediction may be incorrect. However, since the training data (a set of bilingual sentence pairs for training a neural machine translation model) is closed data, the above prediction is expected to be accurate on average with good accuracy in many cases for the training data.

[0031] In other words, when the alignment device 1 inputs a source language sentence of a bilingual sentence included in the set of bilingual sentence pairs and forcibly outputs the target language sentence of that bilingual sentence, it calculates a vector of the sum of a sequence of key distributed expressions weighted by attention probability (the "attention probability weighted key distributed expressions" described below).

[0032] Then, the alignment device 1 calculates the average value of the above vectors (attention probability weighted key distributed representations) for each vocabulary word in the target language. Alternatively, the alignment device 1 calculates a vector that optimizes a predetermined objective function using the above vectors. The vectors calculated here (aligned distributed representations) are distributed representations corresponding to each vocabulary word in the target language.

[0033] The alignment device 1 estimates the source language words corresponding to each word in the target language sentence by calculating the similarity score between the distributed representations (aligned distributed representations) corresponding to each vocabulary word in the target language obtained using the above method and the key distributed representations obtained from the encoder's distributed representation sequence.

[0034] The functions of each part of the alignment device 1 (see FIGS. 1 and 2) are as follows.

[0035] The bilingual sentence pair supply unit 11 supplies one or more bilingual sentence pairs. The bilingual sentence pair supply unit 11 may supply a large number of bilingual sentence pairs. A bilingual sentence pair is a pair of sentences written in two different natural languages ​​(for example, Japanese and English, but not limited to this combination). One of these sentences can be used as a source language sentence, and the other as a target language sentence. Each sentence can be represented as a sequence of words in each language. The bilingual sentence pair supply unit 11 may represent each word as a one-hot vector or as a vector of distributed representations. The two can be converted using a table or the like.

[0036] The key shared representation generation unit 13 generates key shared representations for specific words in source language sentences included in a bilingual sentence pair. That is, the key shared representation generation unit 13 generates key shared representations for input side words included in the input side (source language side) sentence of the bilingual sentence pair provided by the bilingual sentence pair supply unit 11. When the bilingual sentence pair supply unit 11 provides a source language sentence as a sequence of shared representations, the key shared representation generation unit 13 generates key shared representations (vectors) by applying a predetermined parameter matrix to shared representations (vectors) corresponding to words in the source language sentence. In this case, the conversion from the shared representations of the source language words to the key shared representations is a linear conversion.

[0037] The query distributed representation generation unit 14 generates a query distributed representation for a specific word in a target language sentence. That is, the query distributed representation generation unit 14 generates a query distributed representation for an output word included in the output (target language) sentence of the bilingual sentence pair provided by the bilingual sentence pair supply unit 11. Specifically, the query distributed representation generation unit 14 applies a predetermined parameter matrix to each distributed representation (vector) in the decoder's distributed representation sequence to generate a query distributed representation (vector). In this case, the conversion from the distributed representation of the target language word to the query distributed representation is a linear conversion.

[0038] The value distributed representation generation unit 15 generates value distributed representations for specific words in source language sentences included in a bilingual sentence pair. That is, the value distributed representation generation unit 15 generates value distributed representations for input side words included in the input side (source language side) sentence of the bilingual sentence pair provided by the bilingual sentence pair supply unit 11. When the bilingual sentence pair supply unit 11 provides a source language sentence as a distributed representation sequence, the key distributed representation generation unit 13 generates value distributed representations (vectors) by applying a predetermined parameter matrix to distributed representations (vectors) corresponding to words in the source language sentence. In this case, the conversion from the distributed representations of the source language words to value distributed representations is a linear conversion.

[0039] The attention probability calculation unit 18 calculates an attention probability that indicates the probability that an input word corresponds to an output word. The attention probability is a value that is also used in existing machine translation processes.

[0040] The weighted key distributed representation calculation unit 21 (weighted key distributed representation calculation unit) calculates, for each output word, a weighted key distributed representation, which is a key distributed representation of an input word weighted by attention probability.

[0041] The weighted value distributed representation calculation unit 22 (weighted value distributed representation calculation unit) calculates, for each output word, a weighted value distributed representation, which is a value distributed representation of the input word weighted by the attention probability.

[0042] The aligned distributed representation calculation unit 25 calculates aligned distributed representations, which are distributed representations related to words in the target language. The aligned distributed representations are distributed representations that contain information about words in the source language. The aligned distributed representation calculation unit calculates the aligned distributed representations for each word type based on the weighted key distributed representations for each output word.

[0043] The aligned shared representation calculation unit 25 may calculate the aligned shared representation by, for example, calculating an average value of weighted key shared representations for each word type. Alternatively, the aligned shared representation calculation unit 25 may estimate and obtain the aligned shared representation for the output word so that a predetermined evaluation value based on the similarity between the weighted key shared representation corresponding to the output word and the aligned shared representation for the output word is optimized.

[0044] The alignment score calculation unit 26 calculates an alignment score between an input word and an output word based on the similarity between an aligned distributed representation, which is a distributed representation for aligning an output word included in an output sentence different from the input sentence of the bilingual sentence pair, and the key distributed representation of the input word. The similarity here can be expressed as an inner product of vectors, for example. The aligned distributed representation is a distributed representation related to a word in the target language. The aligned distributed representation is a distributed representation containing information about a word in the source language. The alignment score calculation unit 26 may calculate the alignment score based on the aligned distributed representation calculated by the aligned distributed representation calculation unit 25, for example. The alignment score calculation unit 26 may also calculate the alignment score based on an aligned distributed representation, which will be described later as a modified example.

[0045] The alignment score calculation unit 26 may use the attention probability when calculating the alignment score. A specific method for doing so will be described later along with mathematical formulas. That is, the alignment score calculation unit 26 calculates the alignment score between the input word and the output word based on the attention probability calculated by the attention probability calculation unit 18.

[0046] The output unit 29 outputs information related to the results calculated by the alignment score calculation unit 26. Specifically, for example, the output unit 29 outputs the correspondence between words in a sentence on the target language side (output side) and words in a sentence on the source language side (input side). In one form, the output unit 29 outputs data of a set of pairs of word positions (numerical values, etc.) on the target language side and word positions (numerical values, etc.) on the source language side. The output unit 29 may also output data of a set of pairs of words on the target language side and words on the source language side. The output unit 29 may also output the correspondence between the two in a graphical form, for example by connecting the words in the sentence on the target language side and the words in the sentence on the source language side with lines.

[0047] The source language sentence input unit 31 acquires a source language sentence and passes it to the distributed representation conversion unit 32. The source language sentence is a sentence corresponding to a target language sentence. The source language sentence input unit 31 may acquire a source language sentence included in a bilingual sentence pair provided by the bilingual sentence pair providing unit 11.

[0048] The distributed representation conversion unit 32 converts the source language sentence acquired by the source language sentence input unit 31 into a distributed representation.

[0049] The target language sentence input unit 33 acquires a target language sentence. In the phase of constructing aligned distributed representations, the target language sentence input unit 33 passes the acquired target language sentence to the distributed representation conversion unit 34. In the phase of calculating word correspondences, the target language sentence input unit 33 passes the acquired target language sentence to the aligned distributed representation conversion unit 35. The target language sentence input unit 33 may acquire a target language sentence included in a bilingual sentence pair provided by the bilingual sentence pair providing unit 11.

[0050] The distributed representation conversion unit 34 converts the target language sentence acquired by the target language sentence input unit 33 into a distributed representation.

[0051] The aligned distributed representation conversion unit 35 converts the target language sentence acquired by the target language sentence input unit 33 into aligned distributed representations. At this time, the aligned distributed representation conversion unit 35 refers to the aligned distributed representations constructed in the aligned distributed representation construction phase.

[0052] The target language word generation probability calculation unit 37 calculates the generation probability of words belonging to the target language.

[0053] [Processing details] Next, the details of the processing performed by the alignment device 1 will be explained. First, the notation of letters used in the following processing explanation will be explained here. x1, x2, . . ., x m In this notation, 1, 2, , and m are subscripts. Also, the distributed representation h1, h2, , h m In this notation, 1, 2, ..., and m are subscripts. y1, y2, ..., y nIn this notation, 1, 2, , and n are subscripts. Also, s1, s2, , and s n In this notation, 1, 2, ..., and n are subscripts. j and y i In each of these notations, j and i are subscripts. Also, the attention probability α ij and alignment score f ij And other beta i , a i , b ij In the notation, i and j are subscripts. Also, the distributed representation k j , v j , q i In each of these notations, j and i are subscripts. Also, the matrix W k , W v and W q In each of these notations, k, v, and q are indices. Also, the aligned distributed representation d t In this notation, t (vocabulary) is a subscript.

[0054] For the word at the i-th position (position in the target sentence), the value distributed representation and key distributed representation weighted by the attention probability are denoted by c v i and c k i It may be written as:

[0055]

number

[0056] The alignment device 1 can process a bilingual sentence pair by regarding it as a pair of an input sentence and an output sentence. The input sentence is a sentence written in a source language (source sentence) in the translation process. The output sentence is a sentence written in a target language (target sentence). Each of the source sentence and the target sentence can be considered as a word string. The input sentence (source sentence) is expressed as x=x1, x2, . . . , x mThe output sentence (target language sentence) is expressed as y=y1,y2,...,y n where m is the length of the word sequence in the input sentence, and n is the length of the word sequence in the output sentence. j (1≦j≦m) and y i Each of (1≦i≦n) can be represented as a one-hot vector. Given the sentence pair x and y above, the decoder calculates the probability P(y|x) using equation (1) below.

[0057]

number

[0058] Here, we will explain the process assuming the case of neural machine translation using an RNN (recurrent neural network).

[0059] However, even in the case of neural machine translation using a Transformer, the average of the multi-head attention probabilities of a specific layer in the decoder (for example, the layer just before the final layer) can be considered the attention probability when using an RNN. In this case, the distributed representation sequence output from that specific layer is treated as the distributed representation sequence of the decoder. This allows the same processing to be performed when using a Transformer as when using an RNN.

[0060] A sequence of words in the source language sentence: x1, x2, , x m The resulting distributed representations are denoted as h1, h2, , h m The distributed representation h1,h2,···,h m Each of y is a vector, where j is the word position in the source sentence and i is the word position in the target sentence. i The attention probability used to predict ij Attention probability α ij is y iTo predict the word x in the source sentence, j is the probability of using

[0061] Key distributed representation k at position j in the source sentence j and value distributed representation v j are the weight parameter matrix W k and W v It is calculated as shown in the following equations (2) and (3).

[0062]

number

[0063]

number

[0064] The key distributed representation generation unit 13 generates a distributed representation h j Based on the key distributed representation k j In addition, the value distributed representation generation unit 15 generates the distributed representation h j Based on this, the value distributed representation v j Generate a key distributed representation k j and value distributed representation v j Each of these is a vector. Note that the weight parameter matrix W k and W v The specific value of W will be given as appropriate. k and W v The specific value of is determined, for example, by learning using a corpus.

[0065] The distributed representation sequence of the decoder is denoted as s1, s2, , s n The query distributed representation q at position i in the target sentence is i is the weight parameter matrix W q It is calculated as shown in the following equation (4).

[0066]

number

[0067] The query distributed representation generation unit 14 generates a distributed representation s i Based on the distributed representation of the query q i Generate a distributed query representation q i is a vector. Note that the weight parameter matrix W q The specific value of is the above matrix W k k and W v Similarly, it will be given accordingly.

[0068] When using the inner product to calculate similarity, the attention probability α ij is calculated as shown in the following equations (5) and (6). The operator "·" on the right side of equation (6) represents an inner product operation.

[0069]

number

[0070]

number

[0071] The attention probability calculation unit 18 calculates the above-mentioned attention probability using the key distributed representation and the query distributed representation of the given bilingual text pair.

[0072] Attention probability α ij The distributed representation obtained by weighting the value distributed representation using is calculated as shown in the following equation (7). v i The value of may be called the "attention probability weighted value distributed representation" or "weighted value distributed representation."

[0073]

number

[0074] The weighted value distributed representation calculation unit 22 calculates the weighted value distributed representation c using the attention probability aij in the given bilingual text pair and the value distributed representation. v i The distributed representation calculated by equation (7) is input to the decoder and used to calculate the probability of word yi at position i in the target sentence.

[0075] In this embodiment, the distributed representation obtained by weighting the key distributed representation using the attention probability αij is also calculated as shown in the following equation (8). k i The value of may be called the "attention probability weighted key distributed representation" or "weighted key distributed representation."

[0076]

number

[0077] The weighted key distributed representation calculation unit 21 calculates the weighted key distributed representation c using the attention probability aij in the given bilingual text pair and the key distributed representation. k i Calculate.

[0078] Next, the aligned distributed representation calculation unit 25 calculates the aligned distributed representation d corresponding to each vocabulary t in the target language. t This calculation is performed after the completion of training all parameters of NMT. ​​The aligned distributed representation calculation unit 25 calculates the aligned distributed representation d based on a large number of bilingual sentence pairs (for example, the entire training data for learning a neural machine translation model). t The aligned distributed representation calculation unit 25 calculates the aligned distributed representation d t Two typical methods for calculating are described below.

[0079] [First method for computing aligned distributed representations] In the first method, the aligned distributed representation calculation unit 25 calculates the y s of the target language sentence for all the bilingual sentences after the training of the model for neural machine translation is completed. i (position i in the sentence) k i Then, the aligned distributed representation calculation unit 25 calculates the c (attention probability weighted key distributed representation) in the vocabulary t (word type) on the target language side. k i Calculate the mean value of and convert the vector into d t Let's say.

[0080] That is, in the first method, the aligned distributed representation calculation unit 25 calculates the average value of the attention probability weighted key distributed representation for each word type (each vocabulary) as the aligned distributed representation d t It is calculated as follows.

[0081] [Second method for computing aligned distributed representations] In the second method, the aligned distributed representation calculation unit 25 calculates the optimal d t Calculate the parameters of d t As the initial value of , a random value or a value calculated by the first method is used. Specifically, the aligned distributed representation calculation unit 25 calculates d so as to maximize the value of the following equation (9). t Calculate the parameters of β in equation (9). i is as shown in formula (10). When calculating for a set of sentences, the value of formula (9) is obtained as many times as the number of sentences, and the value is obtained by multiplying all of them. In formula (10), a i and b ij The operator "·" that appears on the right-hand side of the equations that define each represents the dot product of vectors.

[0082]

number

[0083]

number

[0084] In the above equations (9) and (10), i represents the word position in the target language sentence in the bilingual sentence pair. The parameter d is used to maximize (optimize) the value of equation (9). t The estimation of can be performed using, for example, the backpropagation method.

[0085] That is, in the second method, the aligned distributed representation calculation unit 25 calculates β i The value of the aligned distributed representation for t (word type) is estimated so that the product of the values ​​of (Equation (10)) is maximized (optimized). i The value of b is as shown in equation (10). ij The sum of the values ​​of j obtained by applying an exponential function to the value of is used as the denominator, and a i It is a value whose numerator is the value obtained by applying an exponential function to the value of a. Needless to say, the exponential function exp() is a monotonically increasing function for the input value. i is c k i (Attention probability weighted key distributed representation) and word y i and the aligned distributed representation of b ij is the key distributed representation k at position j in the source sentence. j and the word y i is the dot product with the aligned distributed representation of

[0086] The d obtained above t Using the distributed representation of ij is calculated as shown in the following equation (11). ij It may be calculated as shown in the following equation (12) by taking into account the above factors.

[0087]

number

[0088]

number

[0089] The aligned distributed representation calculation unit 25 calculates the aligned distributed representation d t The alignment score calculation unit 26 calculates the alignment score f ij The alignment score f ij represents the degree of similarity between the i-th word in the target language and the j-th word in the source language.

[0090] In addition, γ in Equation (12) is the attention probability α ij The value of the hyperparameter γ is determined appropriately. For example, the alignment score f obtained using test data etc. ij The value of the parameter γ may be determined by evaluating the correctness of the value of

[0091] Using the alignment score obtained by equation (11) or equation (12), the word position in the source language sentence corresponding to the word at word position i in the target language sentence can be calculated by equation (13) below.

[0092]

number

[0093] The output unit 29 uses equation (13) to find the word position j in the source language sentence that best corresponds to the word position i in the target language sentence.

[0094] FIG. 3 is a schematic diagram showing an example of bilingual sentence pair data to be processed by the alignment device 1. A bilingual sentence pair is a pair of a sentence in a predetermined source language that can be considered as a source sentence and a sentence in another language that can be considered as a target sentence. In the example shown, the source sentence is an English sentence: "I bought croissants at a bakery with a good reputation near Ebisu station yesterday." The target sentence is a Japanese sentence: "Yesterday, I bought croissants at a bakery with a good reputation near Ebisu station." These two sentences are bilingual. Note that the source language or the target language may be any language.

[0095] FIG. 4 is a schematic diagram showing an example of the results of the alignment device 1 determining the correspondence between words in the bilingual sentences shown in FIG. 3. As shown in the figure, each of the source language sentence and the target language sentence is represented as a string of words, and the position of each word is indicated. For example, the word in the sixth position in the source language sentence is "bakery," and the word in the thirteenth position in the target language sentence is "croissant." For each word in the target language sentence, the alignment device 1 determines the position of the corresponding word in the source language sentence. For example, the first target language word, "yesterday," corresponds to the fourteenth source language word, "yesterday." The same applies to other words in the target language sentence. Note that if there is no word in the source language sentence that corresponds to a word in the target language sentence, the alignment device 1 does not output correspondence information for that word (for example, the word in the second position in the target language sentence).

[0096] In the example shown in Figure 4, the word correspondence between the target sentence and the source sentence is as follows. Note that there are cases where multiple words in one sentence correspond to a single word in the other sentence. The first word "yesterday" in the target sentence corresponds to "yesterday" in the source sentence. The third word "Ebisu" in the target sentence corresponds to "Ebisu" in the source sentence. The fourth word "eki" in the target sentence corresponds to "station" in the source sentence. The fifth and sixth words "chikai / no" in the target sentence correspond to "near" in the source sentence. The seventh word "hanyoku" in the target sentence corresponds to "reputation" in the source sentence. The eighth and ninth words "noi" in the target sentence correspond to "good" in the source sentence. The tenth and eleventh words "bakeryan-san" in the target sentence correspond to "a / bakery" in the source sentence. The twelfth word "de" in the target sentence corresponds to "at" in the source sentence. The 13th and 14th words in the target sentence, "croissants," correspond to "croissants" in the source sentence. The 15th to 18th words in the target sentence, "I bought them," correspond to "bought" in the source sentence.

[0097] The output unit 29 outputs, for example, information on the correspondence between the source language sentence and the target language sentence shown in Fig. 4. The output unit 29 may output information in another form equivalent to the information on the correspondence shown in Fig. 4.

[0098] FIG. 5 is a block diagram showing an example of the internal configuration of the alignment apparatus 1. The alignment apparatus 1 can be realized using a computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, and a bus 906. The computer itself can be realized using existing technology. The central processing unit 901 executes instructions contained in a program read from the RAM 902 or the like. In accordance with each instruction, the central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with external input / output devices. The input / output devices 904 and 905 are input / output devices. Input / output devices 904 and 905 exchange data with the central processing unit 901 via an input / output port 903. A bus 906 is a common communication path used within the computer. For example, the central processing unit 901 reads and writes data from and to RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port via the bus 906.

[0099] At least some of the functions of the alignment device 1 can be implemented by a computer. In this case, a program for implementing the functions may be recorded on a computer-readable recording medium and then loaded and executed by a computer system. The term "computer system" as used herein includes hardware such as an operating system and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, CD-ROMs, DVD-ROMs, and USB memory, as well as storage devices such as hard disks built into computer systems. In other words, a "computer-readable recording medium" may be a non-transitory computer-readable recording medium. Furthermore, the term "computer-readable recording medium" may also include media that temporarily and dynamically store programs, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, or media that store programs for a certain period of time, such as volatile memory within a computer system that serves as a server or client. The program may be designed to implement some of the functions described above, or may be capable of implementing the functions described above in combination with a program already stored in the computer system.

[0100] Although multiple embodiments have been described above, the present invention can also be implemented in the following modified examples. Multiple embodiments and modified examples may be implemented in combination to the extent that they can be combined.

[0101] [First Modification] In equation (6), the dot product of vectors is used to calculate the similarity. i ya b ijThe calculation of the dot product of vectors was also used in the calculation of finding the alignment score in equation (11) and equation (12). The calculation of the dot product of vectors was also used in the calculation of the alignment score in equation (11). In the first modified example, some or all of these may be calculated by other methods instead of the dot product. For example, "dot product of vectors / SQRT (vector size)" (SQRT() is the square root function) may be used instead of the dot product. For example, "(-1) × (squared error)" may be used instead of the dot product.

[0102] [Second Modification] In the above embodiment, the aligned shared representation calculation unit 25 calculates the aligned shared representations (d t In the second modification, the pre-calculated aligned distributed representation (d t ) is used to calculate the alignment score. In other words, the aligned distributed representations (d t ) is stored in advance in a storage device or the like. In other words, the values ​​of each aligned shared representation corresponding to the vocabulary are written in the storage device. t ) data may be calculated by the alignment apparatus 1 itself, or may be calculated by another device and passed to the alignment apparatus 1.

[0103] [Third Modification] In the above embodiment, when a bilingual sentence pair is given, the alignment device 1 processes a sentence in one language as a source language sentence (input sentence) and a sentence in the other language as a target language sentence (output sentence). In a third modified example, the alignment device 1 may reverse this input / output relationship. Furthermore, the alignment device 1 may perform both a process in which language A is the source language sentence (input sentence) and language B (A≠B) is the target language sentence (output sentence), and a process in which language B is the source language sentence (input sentence) and language A is the target language sentence (output sentence).

[0104] Then, the alignment score calculation unit 26 calculates the alignment score f of the new bilingual text pair using equations (11) and (12), etc. ij In this case, the alignment score calculation unit 26 calculates the score for the i-th word y i Depending on the value of the appropriate aligned distributed representation (word y i The distributed representation value corresponding to the

[0105] According to this embodiment (including modified examples), it is possible to automatically determine the correspondence between expressions between sentences in a bilingual text pair. Automatically determining the correspondence between sentences in a bilingual text pair enables various applications based on the bilingual text data (or at least improves the processing efficiency). As an example, it becomes possible to generate an expression dictionary (data pairing an expression in one language with an expression in another language) from the bilingual text pair data. Furthermore, as an example, if the bilingual text pair is the result of machine translation processing, it becomes possible to automatically determine whether or not a missing translation has occurred in the bilingual text pair (or at least automatically obtain information regarding the missing translation). Detecting missing translations can serve as basic information for improving the quality of machine translation technology. Other applications are also conceivable.

[0106] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Industrial Applicability]

[0107] The present invention can be used, for example, to analyze and evaluate bilingual text pair data. Such analysis and evaluation includes, for example, detecting missing translations in bilingual text pairs. However, the scope of use of the present invention is not limited to the examples given here. [Explanation of symbols]

[0108] 1 Alignment device 11. Parallel Text Supply Department 13 Key distributed expression generation unit (key distributed expression generation unit) 14 Query distributed expression generation unit (query distributed expression generation unit) 15 value distributed expression generation unit (value distributed expression generation unit) 18 Attention probability calculation unit 21 Weighted key distributed representation calculation unit (weighted key distributed representation calculation unit) 22 Weighted value distributed representation calculation unit (weighted value distributed representation calculation unit) 25 Aligned distributed representation calculation unit 26 Alignment score calculation section 29 Output section 31 Source language sentence input section 32 Distributed Representation Conversion Unit 33 Target language sentence input section 34 Distributed Representation Conversion Unit 35 Aligned distributed representation conversion unit 37 Target language word generation probability calculation unit 37 (decoder) 901 Central Processing Unit 902 RAM 903 Input / Output Ports 904,905 Input / Output Devices 906 Bus

Claims

1. a key shared representation generation unit that generates key shared representations by multiplying a vector that is a shared representation of an input word included in an input sentence of a given bilingual sentence pair by a predetermined parameter matrix determined based on a corpus; an attention probability calculation unit that calculates an attention probability representing a probability that an output word included in an output sentence different from the input sentence of the bilingual sentence pair corresponds to the output word given the output word; a weighted key distributed representation calculation unit that calculates, for each output word, a weighted key distributed representation that is the key distributed representation of the input word weighted by the attention probability; an aligned distributed representation calculation unit that calculates aligned distributed representations, which are distributed representations for each of the output words and have information about the input words, based on the weighted key distributed representations for each of the output words; an alignment score calculation unit that calculates an alignment score between the input word and the output word based on the similarity between the aligned distributed representation calculated by the aligned distributed representation calculation unit and the key distributed representation of the input word; Equipped with determining, based on the calculated alignment score, a word included in the input sentence that corresponds to a word included in the output sentence; An alignment device comprising: the aligned shared representation calculation unit calculates an average value of the weighted key shared representations calculated by the weighted key shared representation calculation unit for all given bilingual sentences for each word type, and sets the average value as the aligned shared representation. Alignment device.

2. a key shared representation generation unit that generates key shared representations by multiplying a vector that is a shared representation of an input word included in an input sentence of a given bilingual sentence pair by a predetermined parameter matrix determined based on a corpus; an attention probability calculation unit that calculates an attention probability representing a probability that an output word included in an output sentence different from the input sentence of the bilingual sentence pair corresponds to the output word given the output word; a weighted key distributed representation calculation unit that calculates, for each output word, a weighted key distributed representation that is the key distributed representation of the input word weighted by the attention probability; an aligned distributed representation calculation unit that calculates aligned distributed representations, which are distributed representations for each of the output words and have information about the input words, based on the weighted key distributed representations for each of the output words; an alignment score calculation unit that calculates an alignment score between the input word and the output word based on the similarity between the aligned distributed representation calculated by the aligned distributed representation calculation unit and the key distributed representation of the input word; Equipped with determining, based on the calculated alignment score, a word included in the input sentence that corresponds to a word included in the output sentence; 1. An alignment device comprising: The aligned distributed representation calculation unit estimates and calculates the aligned distributed representations for the output words so as to maximize a value expressed by the following formula (9) for a given bilingual sentence, [Equation 1] β in Equation (9) ｉ is expressed by the following equation (10): [Equation 2] In equations (9) and (10), i represents the position of the output word in the output sentence, In equation (10), j represents the position of the input word in the input sentence, a in formula (10) ｉ is the weighted key distributed representation (c ｋ ｉ ) and the word y ｉ Aligned distributed representation of (d ｙｉ ) and b in formula (10) ｉｊ is the key distributed representation (k ｊ ) and the word y ｉ Aligned distributed representation of (d ｙｉ ) is the dot product Alignment device.

3. A program for causing a computer to function as the alignment device according to claim 1 or 2.

Images