Machine learning program, machine learning method, and translation device

Abstract

To prevent mistranslation of polysemous words including named entities when performing machine translation.SOLUTION: A translation device 100 acquires training data including a first sentence expressed in a first language and a second sentence expressed in a second language. The translation device 100 identifies named entities and parts of speech from the first sentence. The translation device 100 generates a translation model including an attention mechanism for the named entities and the parts of speech according to the training data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a machine learning program and a machine learning method for performing machine learning on a translation model, and a translation device using the translation model. [Background technology]

[0002] Globalization has made it possible to refer to texts written in various languages. When users refer to texts written in languages ​​other than their own, there is a need to be able to easily understand such texts. To meet this need, development of machine translation, which translates texts written in a source language into texts written in a target language, is underway.

[0003] If a word or phrase containing important information in a sentence is translated incorrectly, it can affect the translation of the entire sentence. For example, a named entity may be incorrectly translated as a common noun or verb. Figure 14 is a diagram illustrating an example of an incorrect translation.

[0004] FIG. 14 shows a case where a sentence 10 in a source language (Japanese) is translated into a sentence 11 in a target language (English). "Genki" 10a included in the sentence 10 is an adjective, while "Genki" 10b included in the sentence 10 is a named entity. "Genki" 10a is correctly translated as "fine" 11a in the sentence 11, but "Genki" 10b is incorrectly translated as "fine" 11b in the sentence 11.

[0005] Prior arts 1 and 2 exist to solve the problem of named entities explained in Fig. 14. Fig. 15 is a diagram for explaining prior art 1. In Fig. 15, a sentence written in a source language (English) is designated as sentence 12, and a sentence written in a target language (German) is designated as sentence 13.

[0006] Prior art 1 uses a term dictionary for named entities to generate Append sentence 12a by assigning "0" to words contained in sentence 12 that are not included in the term dictionary, "1" to named entities in the source language that are included in the term dictionary, and "2" to named entities in the target language that are included in the term dictionary. Prior art 1 generates Replace sentence 12b by deleting named entities assigned "1" from Append sentence 12a.

[0007] In prior art 1, a vector combining the vector of sentence 12 and the vector of replace sentence 12b is input into an encoder-decoder model, and machine learning is performed so that the value output from the decoder approaches the vector of sentence 13.

[0008] In prior art 2, tags are assigned to named entities contained in a source language sentence, and a vector combining the vector of the source language word and the vector of the named entity tag is input into an encoder-decoder model, and machine learning is performed so that the value output from the decoder approaches the vector of the target language sentence. Furthermore, prior art 2 uses a chunk-level LSTM (Long Short-Term Memory) layer on the encoder side as a hidden layer. [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] Training NMT to Apply Terminology Constraints, Dinu et al.,2019 Summary of the Invention [Problem to be solved by the invention]

[0010] For example, in prior art 1, if a named entity registered in a term dictionary is a polysemous word, the meaning with the highest proportion is given priority, which may result in an incorrect translation.

[0011] FIG. 16 is a diagram for explaining the problems of prior art 1. FIG. 16 shows a case where a sentence 14 in a source language (English) is translated into a sentence 15 in a target language (Japanese). The named entity "VHL" 14a included in sentence 14 can be used as a gene name or a disease name. In the case shown in FIG. 16, the translation of VHL included in sentence 14 as the gene name is correct, but in the translation of sentence 15, VHL is erroneously translated as VHL disease (disease name).

[0012] In prior art 2, when translating words in the target language, useful named entity information may not be taken into account, or unuseful named entity information may be memorized, resulting in mistranslations of words.

[0013] In one aspect, the present invention aims to provide a machine learning program, a machine learning method, and a translation device that can prevent polysemous words, including named entities, from being mistranslated when performing machine translation. [Means for solving the problem]

[0014] In one embodiment, a machine learning program causes a computer to perform a process of acquiring training data including a first sentence expressed in a first language and a second sentence expressed in a second language, identifying named entities and parts of speech from the first sentence, and generating a translation model including an attention mechanism for the named entities and the parts of speech based on the training data. [Effects of the Invention]

[0015] When machine translation is performed, it is possible to prevent polysemous words, including named entities, from being mistranslated. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a functional block diagram illustrating a configuration of a translation device according to the first embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of the data structure of the training data. [Figure 3] FIG. 3 is a diagram illustrating an example of the data structure of the named entity dictionary. [Figure 4] FIG. 4 is a diagram illustrating an example of a data structure of the translation model according to the first embodiment. [Figure 5] FIG. 5 is a diagram for explaining the processing of the identification unit. [Figure 6] FIG. 6 is a flowchart of the process during training according to the first embodiment. [Figure 7] FIG. 7 is a flowchart of the process at the time of translation according to the first embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of translation by the translation device according to the first embodiment. [Figure 9] FIG. 9 is a functional block diagram illustrating a configuration of a translation device according to the second embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of a data structure of a translation model according to the second embodiment. [Figure 11] FIG. 11 is a flowchart of the process during training according to the second embodiment. [Figure 12] FIG. 12 is a flowchart of a process at the time of translation according to the second embodiment. [Figure 13] FIG. 13 is a diagram showing an example of the hardware configuration of a computer that realizes the same functions as the translation device according to this embodiment. [Figure 14] FIG. 14 is a diagram for explaining an example of an incorrect translation. [Figure 15] FIG. 15 is a diagram for explaining the prior art 1. In FIG. [Figure 16] FIG. 16 is a diagram for explaining the problems of the prior art 1. In FIG. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, embodiments of the machine learning program, machine learning method, and translation device disclosed in the present application will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to these embodiments. [Example]

[0018] 1 is a functional block diagram showing the configuration of a translation device according to Example 1. As shown in FIG. 1, the translation device 100 includes a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0019] The communication unit 110 performs data communication with an external device (not shown) via a network. For example, the communication unit 110 receives training data 141 (described later) from the external device.

[0020] The input unit 120 is a device or interface for inputting data. For example, a user operates the input unit 120 to input information about a source language sentence to be translated into the translation device 100.

[0021] The display unit 130 displays information output from the control unit 150. For example, the display unit 130 displays the translation result output from the control unit 250.

[0022] The storage unit 140 stores data, programs executed by the control unit 150, and the like. The storage unit 140 has training data 141, a named entity dictionary 142, and a translation model 143. The storage unit 140 is an example of a storage device, such as a hard disk or memory.

[0023] The training data 141 stores sentences in a source language and sentences in a target language in association with each other. A sentence in a source language is referred to as a "source sentence." A sentence in a target language is referred to as a "target sentence." A translation of a source sentence becomes a target sentence. A source sentence corresponds to a first sentence expressed in a first language. A target sentence corresponds to a second sentence expressed in a second language.

[0024] Fig. 2 is a diagram showing an example of the data structure of training data. As shown in Fig. 2, training data 141 associates item numbers with source language sentences and target language sentences. The item numbers are numbers that indicate records (pairs of source language sentences and target language sentences) of training data 141. The source language sentences are text data written in the source language. The target language sentences are text data written in the target language.

[0025] For example, if the source language is "English" and the target language is "Japanese," the target sentence corresponding to the source sentence "President lives in White House" is "The president lives in White House."

[0026] The named entity dictionary 142 stores named entities in the source language in association with the type of the named entity. FIG. 3 is a diagram showing an example of the data structure of the named entity dictionary. As shown in FIG. 3, the named entity dictionary 142 associates an entry number with a named entity and a type. The entry number is a number indicating a record (a pair of a named entity and a type) in the named entity dictionary. A named entity is a named entity used in the source language. A type is a type of named entity. For example, types of named entities include organizations, people's names, places, events, drug names, etc.

[0027] The translation model 143 is a machine learning model that translates a source language sentence into a target language sentence. Fig. 4 is a diagram showing an example of a data structure of the translation model according to the first embodiment. As shown in Fig. 4, the translation model 143 includes an encoder 50 and a decoder 60.

[0028] The encoder 50 includes an embedding layer 51 and a long short-term memory (LSTM) 52.

[0029] The embedding layer 51 provides a vector for the input word. For example, each word of a source language sentence of the training data 141 is input to the embedding layer 51 in order from the beginning, and each vector is output to the LSTM 52 in order.

[0030] The LSTM 52 introduces memory cells into a normal RNN (Recurrent Neural Network) and controls input, forgetting, and output with multiple gates. Every time the LSTM 52 receives a vector input from the Embedding layer 51, it outputs a hidden vector to the decoder 60.

[0031] The decoder 60 has an embedding layer 61, an LSTM 62, an attention layer 63, an affine layer 64, and a softmax layer 65.

[0032] The Embedding layer 61 provides a vector for the input word. For example, when executing machine learning, first, <bos>is input, each word of the target language sentence of the training data 141 is input to the embedding layer 61 in order from the beginning, and each vector is output to the LSTM 62 in order.

[0033] LSTM62 introduces memory cells into a conventional RNN and controls input, forgetting, and output with multiple gates. Every time LSTM62 receives a vector input from the Embedding layer 61, it outputs a hidden vector to the Attention layer 63 and the Affine layer 64.

[0034] The Attention layer 63 ignores irrelevant information and selectively focuses on parts that are highly relevant to the input information. The Attention layer 63 calculates a context vector based on the Hidden vector input from the encoder and the Hidden vector input from the LSTM 62. The Attention layer 63 outputs the calculated context vector to the Affine layer 64. The Attention layer 63 corresponds to the "attention mechanism."

[0035] The Affine layer 64 is a fully connected layer that performs calculations based on parameters. The Affine layer 64 receives as input a vector that combines the context vector output from the Attention layer 63 and the Hidden vector output from the LSTM 62, and outputs the calculation result to the Softmax layer 65.

[0036] The Softmax layer 65 normalizes the input values ​​and performs calculations so that the sum of the outputs becomes 1. The Softmax layer 65 uses the values ​​input from the Affine layer 64 to output the probability of each word in the object sentence.

[0037] The translation model 143 described in FIG. 4 is explained using the LSTMs 52 and 62, but an RNN may be used instead of the LSTM.

[0038] Returning to the explanation of Fig. 1, the control unit 150 controls the entire translation device 100. The control unit 150 includes an identification unit 151, a training unit 152, and a translation unit 153.

[0039] The identification unit 151 acquires a source language sentence and a target language sentence from the training data 141. The identification unit 151 performs morphological analysis on the source language sentence and divides it into a plurality of words. The identification unit 151 identifies named entities from among the words of the source language sentence based on each word of the source language sentence and the named entity dictionary 142.

[0040] The identification unit 151 generates named entity identification information based on the named entities identified from the source language sentence. The identification unit 151 assigns the code "O" to words included in the source language sentence that do not correspond to named entities, and assigns the code of the named entity to words that correspond to named entities.

[0041] 5 is a diagram for explaining the processing of the identification unit. In FIG. 5, the source language sentence 20 is "President lives in White House." Of these, "White House" is the identified named entity. The identification unit 151 generates named entity identification information 25 by replacing words that are not named entities with the code "O" and replacing the named entity "White House" with the code "B(Begin)-LOC, E(end)-LOC."

[0042] The identification unit 151 outputs information on the source language sentence, the named entity identification information, and the target language sentence to the training unit 152. The identification unit 151 repeatedly executes the above process for each source language sentence included in the training data 141.

[0043] During translation, when the identification unit 151 acquires a source language sentence to be translated from the input unit 120, the identification unit 151 generates named entity identification information based on the source language sentence to be translated. The identification unit 151 outputs the source language sentence and the named entity identification information to the translation unit 153. The identification unit 151 may acquire the source language sentence to be translated from an external device via the communication unit 110.

[0044] The training unit 152 trains the translation model 143, which includes an attention mechanism for named entities, based on the source language sentence, the named entity identification information, and the target language sentence.

[0045] An example of processing executed by the training unit 152 will be described with reference to FIG. 4. The training unit 152 inputs words of a source language sentence in time series to the embedding layer 51 of the encoder 50, thereby causing the LSTM 52 to output hidden vectors. By inputting words of a source language sentence in time series to the embedding layer 51, the training unit 152 outputs hidden vectors as H={h1, ,h n When the j-th word of the source language sentence is input to the embedding layer 51, the hidden vector is j Let's say.

[0046] Furthermore, the training unit 152 inputs the codes of the named entity identification information in a time series to the embedding layer 51 of the encoder 50, thereby causing the LSTM 52 to output a hidden vector. By inputting the codes of the named entity identification information in a time series to the embedding layer 51, the hidden vector output from the LSTM 52 is expressed as D={d1, ,d n When the j-th code of the named entity identification information is input to the embedding layer 51, the hidden vector is input from the LSTM 52 as d j Let's say.

[0047] The hidden vectors H and D calculated by the encoder 50 are input to the LSTM 62 and attention layer 63 of the decoder 60.

[0048] The training unit 152 first inputs the following to the embedding layer 61 of the decoder 60: <bos>After inputting the above, the words of the target language sentence of the training data 141 are input in time series, causing the embedding layer 61 to output a vector to the LSTM 62. This causes the LSTM 62 to output a hidden vector. <bos>By inputting the words of the target language sentence in time series, the hidden vector output from LSTM62 is defined as S={s1, ,s n }. The hidden vector s output from LSTM62 i-1 is the i-th word in the target sentence.

[0049] Here, the attention layer 63 calculates the attention of the source language based on the hidden vector H calculated by the encoder 50 and the hidden vector S output from the embedding layer 61. For example, the attention layer 63 calculates the attention of the source language based on equation (1). ij is calculated by equation (2). ij is, s i-1 and h j Indicates the similarity to.

[0050]

number

number

[0051] Furthermore, the attention layer 63 calculates attention for a named entity based on the hidden vector D calculated by the encoder 50 and the hidden vector S output from the embedding layer 61. For example, the attention layer 63 calculates attention for a named entity based on equation (3). sim(s i-1 ,d j ) is s i-1 and, d j The hidden vector D is defined as in equation (4).

[0052]

number

number

[0053] The attention layer 63 calculates the joint attention between the source language sentence and the named entity based on formula (5). λ included in formula (5) is set in advance to a value between 0 and 1.

[0054]

number

[0055] The Attention layer 63 calculates the context vector c based on the results of the joint attention. i Calculate the context vector c i is calculated by Equation (6). The attention layer 63 calculates the context vector c i is output to Affine layer 64.

[0056]

number

[0057] The Affine layer 64 is the context vector c output from the Attention layer 63. i and the hidden vector s output from LSTM62 i It accepts the combined vector as input and outputs the calculation result.

[0058] The training unit 152 calculates the vector v output from the Affine layer 64. i and the vector of the ith word in the target language sentence, the parameters of the encoder 50 and the decoder 60 are trained. For example, the training unit 152 trains the parameters of the encoder 50 and the decoder 60 using a backpropagation method or the like.

[0059] The training unit 152 trains the translation model 143 by repeatedly executing the above process every time it acquires information on the source language sentence, named entity identification information, and target language sentence from the identification unit 151.

[0060] Returning to the explanation of Fig. 1, the translation unit 153 translates the source language sentence to be translated into a target language sentence using the source language sentence to be translated, the named entity information of the source language sentence, and the trained translation model 143 received from the identification unit 151. In the following explanation of the translation unit 153, the source language sentence to be translated will be simply referred to as the source language sentence.

[0061] An example of processing executed by the translation unit 153 will be described with reference to FIG. 4. The translation unit 153 inputs the words of the source language sentence in time series to the embedding layer 51 of the encoder 50, thereby causing the LSTM 52 to output hidden vectors. By inputting the words of the source language sentence in time series to the embedding layer 51, the translation unit 153 outputs hidden vectors from the LSTM 52 as H={h1, ,h n When the j-th word of the source language sentence is input to the embedding layer 51, the hidden vector is j Let's say.

[0062] Furthermore, the translation unit 153 inputs the codes of the named entity identification information in a time series to the embedding layer 51 of the encoder 50, thereby causing the LSTM 52 to output a hidden vector. By inputting the codes of the named entity identification information in a time series to the embedding layer 51, the hidden vector output from the LSTM 52 is converted into a hidden vector D={d1, ,d n When the j-th code of the named entity identification information is input to the embedding layer 51, the hidden vector is input from the LSTM 52 as d j Let's say.

[0063] The hidden vectors H and D calculated by the encoder 50 are input to the LSTM 62 and attention layer 63 of the decoder 60.

[0064] Unlike the training unit 152, the translation unit 153 generates each word for each step. The translation unit 153 first adds the following to the embedding layer 61 of the decoder 60: <bos>Then, the vectors (words in the target language corresponding to the vectors) output from the Affine layer 64 are input in time series, causing the Embedding layer 61 to output the vectors to the LSTM 62. This causes the LSTM 62 to output the Hidden vectors. <bos>,We input each word at each step and the hidden vector output from LSTM62 is,S={s1,...,s n }. The hidden vector s output from LSTM62 i-1 is the i-th word in the target sentence.

[0065] The Attention layer 63 calculates the attention of the source language based on the Hidden vector H calculated by the encoder 50 and the Hidden vector S output from the Embedding layer 61. The attention of the source language is calculated using Equations (1) and (2).

[0066] The attention layer 63 calculates attention for a named entity based on the hidden vector D calculated by the encoder 50 and the hidden vector S output from the embedding layer 61. The attention for a named entity is calculated based on equation (5).

[0067] The Attention layer 63 calculates the context vector c based on the results of the joint attention. i Calculate the context vector c i is calculated by Equation (6). The attention layer 63 calculates the context vector c i is output to Affine layer 64.

[0068] The Affine layer 64 is the context vector c output from the Attention layer 63. i and the hidden vector s output from LSTM62 i The combined vector is accepted as input, and the calculation result is output to the Softmax layer 65.

[0069] The Softmax layer 65 normalizes the input values ​​and performs calculations so that the sum of the outputs becomes 1. The Softmax layer 65 outputs the probability of each word in the object sentence using the values ​​input from the Affine layer 64. The translation unit 153 generates the object sentence by repeatedly executing a process of converting each word into the object word with the highest probability based on the probability of each word output from the Softmax layer 65. The translation unit 153 may output the generated object sentence to the display unit 130 as the translation result and display it, or may transmit the translation result to an external device.

[0070] Next, a description will be given of an example of a processing procedure of the translation device 100 according to the first embodiment. The processing during training of the translation device 100 and the processing during translation will be described in this order.

[0071] Fig. 6 is a flowchart of the training process according to the present embodiment 1. As shown in Fig. 6, the identification unit 151 of the translation device 100 acquires a source language sentence and a target language sentence from the training data 141 (step S101). The identification unit 151 identifies a named entity from the source language sentence based on the named entity dictionary 142 (step S102).

[0072] The identification unit 151 generates named entity identification information based on the result of identifying the named entity (step S103). The training unit 152 of the translation device 100 inputs the source language sentence to the encoder 50 to calculate a hidden vector H (step S104). The training unit 152 inputs the named entity identification information to the encoder to calculate a hidden vector D (step S105).

[0073] The training unit 152 calculates joint attention using the Attention layer 63 (step S106). The training unit 152 calculates the Hidden State on the decoder 60 side based on the Attention information (step S107). The training unit 152 calculates the output word score of the target language (step S108). The training unit 152 calculates the error between the value output from the Affine layer 64 and the vector of the word in the target language sentence (step S109). The training unit 152 performs machine learning to reduce the error (step S110).

[0074] If the translation device 100 continues the process (step S111, Yes), the translation device 100 proceeds to step S101. On the other hand, if the translation device 100 does not continue the process (step S111, No), the translation device 100 ends the training of the translation model 143.

[0075] Fig. 7 is a flowchart of a process during translation according to the present embodiment 1. As shown in Fig. 7, the identification unit 151 of the translation device 100 acquires a source language sentence to be translated from the input unit 120 (step S201). The identification unit 151 identifies a named entity from the source language sentence based on the named entity dictionary 142 (step S202).

[0076] The identification unit 151 generates named entity identification information based on the result of identifying the named entity (step S203). The translation unit 153 of the translation device 100 inputs the source language sentence to the encoder 50 to calculate a hidden vector H (step S204). The translation unit 153 inputs the named entity identification information to the encoder to calculate a hidden vector D (step S205).

[0077] The translation unit 153 calculates joint attention using the attention layer 63 (step S206). The translation unit 153 calculates the hidden state on the decoder 60 side based on the attention information (step S207). The translation unit 153 calculates the probability of the output word in the target language (step S208). The processing in steps S207 and S208 is performed using the special character that indicates the end of a sentence. <eo>The translation unit 153 outputs the identified target language sentence to the display unit 130 (step S209).

[0078] Next, the effects of the translation device 100 according to the first embodiment will be described. When performing machine learning using the training data 141, the translation device 100 identifies named entities in the source language and generates a translation model 143 that includes an attention mechanism for the named entities. By using this translation model 143, translation can be performed while taking into consideration the balance between the meanings of words in the source language sentence and the named entities, thereby preventing erroneous translation of slang words including named entities.

[0079] Fig. 8 is a diagram showing a translation example of the translation device according to the first embodiment. For example, when a source language sentence 30 is input to the translation device 100 and translated, it is translated into a target language sentence 31. As shown in Fig. 8, "genki (adjective)" 30a in the source language sentence 30 is translated into "fine" 31a in the target language sentence 31, and "genki (named entity)" 30b in the source language sentence 30 is translated into "Genki" 31b in the target language sentence 31. In other words, the translation result of the translation device 100 according to the first embodiment is more appropriate than the translation result according to the conventional technology described with reference to Fig. 14.

[0080] Furthermore, translation device 100 performs translation using joint attention, which combines attention of the source language sentence and attention of the named entity, resulting in a smoother probability distribution and improved translation accuracy for unknown words.

[0081] In the above-mentioned first embodiment, the translation device 100 identifies named entities from the source language sentence and calculates joint attention, but this is not limited to this, and the translation device 100 may further identify parts of speech, etc., and calculate joint attention.

[0082] For example, the specifying unit 151 compares each word in the source language sentence with a part-of-speech dictionary to specify the part-of-speech of each word, and converts each word into a code (a code that uniquely indicates the part-of-speech), thereby generating part-of-speech specifying information.

[0083] The training unit 152 inputs the part-of-speech specification information in addition to the source language sentence and named entity specification information to the encoder 50, and calculates the hidden vector H of the source language sentence, the hidden vector D of the named entity specification information, and the hidden vector D' of the part-of-speech specification information.

[0084] The training unit 152 calculates the joint attention based on formula (7). In formula (7), att is attention calculated based on formula (1). att_m is attention of a named entity calculated based on formula (3). att_m' is attention of a part of speech. The attention of a part of speech is calculated by replacing the hidden vector D of the named entity shown in formula (3) with the hidden vector D' of the part of speech. The relationship between λ1, λ2, and λ3 in formula (7) is adjusted so that the sum is 1, as shown in formula (8).

[0085]

number

number

[0086] Similarly to the training unit 152, the translation unit 153 can also calculate joint attention using equation (7) and translate a source language sentence into a target language sentence. [Example]

[0087] Next, a translation device according to Example 2 will be described. The translation device 100 described in Example 1 trains an RNN-based (seq2seq) translation model, but the translation device according to Example 2 trains a Transformer-based translation model.

[0088] Fig. 9 is a functional block diagram showing the configuration of a translation device according to Example 2. As shown in Fig. 9, this translation device 200 includes a communication unit 210, an input unit 220, a display unit 230, a storage unit 240, and a control unit 250.

[0089] The explanations regarding the communication unit 210, the input unit 220, and the display unit 230 are the same as the explanations regarding the communication unit 110, the input unit 120, and the display unit 130 in the first embodiment.

[0090] The storage unit 240 stores data, programs executed by the control unit 150, and the like. The storage unit 240 has training data 241, a named entity dictionary 242, and a translation model 243. The storage unit 240 is an example of a storage device, such as a hard disk or memory.

[0091] The training data 241 stores sentences in a source language and sentences in a target language in association with each other. A sentence in a source language is referred to as a "source language sentence." A sentence in a target language is referred to as a "target language sentence." The data structure of the training data 241 is the same as the data structure of the training data 141 described in FIG. 2.

[0092] The named entity dictionary 242 associates and stores named entities in the source language with the types of the named entities. The data structure of the named entity dictionary 242 is the same as the data structure of the named entity dictionary 142 described with reference to FIG.

[0093] The translation model 243 is a machine learning model that translates a source language sentence into a target language sentence. Fig. 10 is a diagram showing an example of a data structure of the translation model according to the second embodiment. As shown in Fig. 10, the translation model 243 includes an encoder 70 and a decoder 80.

[0094] The encoder 70 includes an embedding layer 71, a self-attention layer 72, and an affine layer 73.

[0095] The Embedding layer 71 assigns a vector to an input word. For example, each word in a source language sentence of the training data 241 is input to the Embedding layer 71, starting from the beginning. The Embedding layer 71 performs Positional Embedding to assign word position information to each word vector, convert it into parallel information, and output it to the Self-Attention layer 72.

[0096] The self-attention layer 72 is a self-attention mechanism. The self-attention layer 72 calculates attention based on equation (9). In equation (9), Q E is the output result X of the embedding layer 71, and the weight W Q It is a product of K E is the output result X of the embedding layer 71, and the weight W K The d in equation (9) is multiplied by k The square root of is Q E , K. E The Self-Attention layer 72 outputs the calculation result to the Affine layer 73.

[0097]

number

[0098] The Affine layer 73 is a fully connected layer that performs calculations based on parameters. The Affine layer 73 outputs the calculation results to the decoder 80.

[0099] The decoder 80 has an embedding layer 81, a self-attention layer 82, an encoder-decoder attention layer 83, an affine layer 84, and a softmax layer 85.

[0100] The Embedding layer 81 assigns a vector to an input word. For example, during training, each word in a target language sentence of the training data 241 is input to the Embedding layer 81. The Embedding layer 81 performs Positional Embedding to assign word position information to each word vector, convert it into parallel information, and output it to the Self-Attention layer 82.

[0101] The Self-Attention layer 82 is a self-attention mechanism. The Self-Attention layer 82 calculates attention based on Equation (10). In Equation (10), Q D is the output result Y of the embedding layer 81, and the weight W Q It is a product of K D is the output result X of the embedding layer 71, and the weight W K The d in equation (10) is multiplied by k The square root of is Q D , K. D The Self-Attention layer 72 outputs the calculation result to the Affine layer 73.

[0102]

number

[0103] The Encoder-Decoder attention layer 83 is a self-attention mechanism that takes self-representation into account.

[0104] The Affine layer 84 is a fully connected layer that performs calculations based on parameters. The Affine layer 84 outputs the calculation results to the Softmax layer 85.

[0105] The Softmax layer 85 normalizes the input values ​​and performs calculations so that the sum of the outputs becomes 1. The Softmax layer 85 uses the values ​​input from the Affine layer 84 to output the probability of each word in the object sentence.

[0106] Returning to the explanation of Fig. 9, the control unit 250 controls the entire translation device 200. The control unit 250 includes a specification unit 251, a training unit 252, and a translation unit 253.

[0107] The identification unit 251 acquires a source language sentence and a target language sentence from the training data 241. The identification unit 251 performs morphological analysis on the source language sentence and divides it into a plurality of words. The identification unit 251 identifies named entities from among the words of the source language sentence based on each word of the source language sentence and the named entity dictionary 242.

[0108] The identification unit 251 generates named entity identification information based on the named entities identified from the source language sentence. The identification unit 251 assigns the code "O" to words included in the source language sentence that do not correspond to named entities, and assigns the code of the named entity to words that correspond to named entities.

[0109] The identification unit 251 outputs information on the source language sentence, the named entity identification information, and the target language sentence to the training unit 252. The identification unit 251 repeatedly executes the above process for each source language sentence included in the training data 241.

[0110] During translation, when the identification unit 251 acquires a source language sentence to be translated from the input unit 220, the identification unit 251 generates named entity identification information based on the source language sentence to be translated. The identification unit 251 outputs the source language sentence and the named entity identification information to the translation unit 253. The identification unit 251 may acquire the source language sentence to be translated from an external device via the communication unit 210.

[0111] The training unit 252 trains the translation model 243, which includes an attention mechanism for named entities, based on the source language sentence, the named entity identification information, and the target language sentence.

[0112] An example of processing executed by the training unit 252 will be described with reference to Fig. 10. The training unit 252 inputs words of a source language sentence to the embedding layer 71 of the encoder 70. As a result, the calculation result of the embedding layer 71 is input to the self-attention layer 72, the calculation result of the self-attention layer 72 is input to the affine layer 73, and an output result X1 is output from the encoder 70.

[0113] Furthermore, the training unit 252 inputs the code of the named entity identification information to the Embedding layer 71 of the encoder 70. As a result, the calculation result of the Embedding layer 71 is input to the Self-Attention layer 72, and the calculation result of the Self-Attention layer 72 is input to the Affine layer 73, and the output result X2 is output from the encoder 70.

[0114] The output results X1 and X2 calculated by the encoder 70 are input to the encoder-decoder attention layer 83 of the decoder 80. The output result X1 corresponds to the vector of a source language word. The output result X2 corresponds to the vector of a named entity in the source language.

[0115] The training unit 252 inputs the words of the target language sentence into the embedding layer 81 of the decoder 80. As a result, the calculation results of the embedding layer 81 are input into the self-attention layer 82, and the output result X3 of the self-attention layer 82 is input into the encoder-decoder attention layer 83. The output result X3 corresponds to the vector of the target language word.

[0116] The Encoder-Decoder attention layer 83 calculates joint attention based on the output results X1, X2, and X3. For example, the Encoder-Decoder attention layer 83 uses Equation (11). λ included in Equation (11) is set in advance to a value greater than or equal to 0 and less than or equal to 1.

[0117]

number

[0118] att included in Eq. (11) T is an attention calculated based on the output result X1 and the output result X3. For example, T is calculated by equation (12). In equation (12), Q E1 is the output result X3 of the Self-Attention layer 82, and the weight W Q It is a product of K E1 is the output result X1 of the encoder 70, and the weight W K The d in equation (12) is multiplied by k The square root of is Q E1 , K. E1 This indicates the dimension of

[0119]

number

[0120] att included in Eq. (11) T_ m is an attention calculated based on the output result X2 and the output result X3. For example, T_ m is calculated by equation (13). In equation (13), Q E1 is the output result X3 of the Self-Attention layer 82, and the weight W Q This is the result of multiplying K E2 is the output result X2 of the encoder 70, and the weight W K The d in equation (13) is multiplied by k The square root of is Q E1 , K. E2 This indicates the dimension of

[0121]

number

[0122] The Encoder-Decoder attention layer 83 calculates the context vector c based on the joint attention calculation result. i Calculate the context vector c i is calculated by equation (14). In equation (14), V is the output result X1 and the weight W V The Encoder-Decoder attention layer 83 multiplies the calculated context vector c i is output to the Affine layer 84.

[0123]

number

[0124] The training unit 252 trains the parameters of the encoder 70 and the decoder 80 so as to reduce the error between the vector output from the Affine layer 84 and the vector of each word in the target language sentence. For example, the training unit 252 trains the parameters of the encoder 70 and the decoder 80 using backpropagation or the like.

[0125] The training unit 252 trains the translation model 243 by repeatedly executing the above process every time it acquires information on the source language sentence, named entity identification information, and target language sentence from the identification unit 251.

[0126] Returning to the explanation of Fig. 9, the translation unit 253 translates the source language sentence to be translated into a target language sentence using the source language sentence to be translated, the named entity information of this source language sentence, and the trained translation model 243 received from the identification unit 251. In the following explanation of the translation unit 253, the source language sentence to be translated will be simply referred to as the source language sentence.

[0127] An example of processing executed by the translation unit 253 will be described with reference to Fig. 10. The translation unit 253 inputs words of a source language sentence into the embedding layer 71 of the encoder 70. As a result, the calculation result of the embedding layer 71 is input into the self-attention layer 72, the calculation result of the self-attention layer 72 is input into the affine layer 73, and an output result X1 is output from the encoder 70.

[0128] Furthermore, the translation unit 253 inputs the code of the named entity identification information to the embedding layer 71 of the encoder 70. As a result, the calculation result of the embedding layer 71 is input to the self-attention layer 72, the calculation result of the self-attention layer 72 is input to the affine layer 73, and the output result X2 is output from the encoder 70.

[0129] The output results X1 and X2 calculated by the encoder 70 are input to the encoder-decoder attention layer 83 of the decoder 80. The output result X1 corresponds to the vector of a source language word. The output result X2 corresponds to the vector of a named entity in the source language.

[0130] The translation unit 253 inputs the words of the target language sentence for each step to the embedding layer 81 of the decoder 80. During translation, the decoder 80 outputs the words of the target language sentence for each step. For example, during translation, the words of the target language sentence generated in the previous step and the context vector of the source language sentence from the encoder 70 are input to the hidden layer of the decoder 80, and the decoder 80 generates the next word. Special characters indicating the end of a sentence<End of Sentence> The process is repeated until the output of the embedding layer 81 is reached, and then the translation is completed. In other words, the information input to the embedding layer 81 is the word of the target sentence generated immediately before. As a result, the calculation result of the embedding layer 81 is input to the self-attention layer 82, and the output result X3 of the self-attention layer 82 is input to the encoder-decoder attention layer 83. The output result X3 corresponds to the vector of the target language word.

[0131] The Encoder-Decoder attention layer 83 calculates the joint attention based on the output results X1, X2, and X3 based on Equation (11). The Encoder-Decoder attention layer 83 calculates the context vector c i The Encoder-Decoder attention layer 83 calculates the calculated context vector c i is output to the Affine layer 84.

[0132] Affine layer 84 uses the context vector c i When input, it performs a calculation using the parameters and outputs the calculation result to the Softmax layer 85.

[0133] The Softmax layer 65 normalizes the input values ​​and performs calculations so that the sum of the outputs becomes 1. The Softmax layer 65 outputs the probability of each word in the object sentence using the values ​​input from the Affine layer 84. The translation unit 253 generates the object sentence by repeatedly executing a process of converting each word into the object word with the highest probability based on the probability of each word output from the Softmax layer 65. The translation unit 253 may output the generated object sentence to the display unit 230 as the translation result and display it, or may transmit the translation result to an external device.

[0134] Next, an example of a processing procedure of the translation device 200 according to the second embodiment will be described. The processing of the translation device 200 during training and the processing during translation will be described.

[0135] Fig. 11 is a flowchart of the training process according to Example 2. As shown in Fig. 11, the identification unit 251 of the translation device 100 acquires source language sentences and target language sentences from the training data 241 (step S301). The identification unit 251 identifies named entities from the source language sentences based on the named entity dictionary 242 (step S302).

[0136] The identification unit 251 generates named entity identification information based on the result of identifying the named entity (step S303). The training unit 252 of the translation device 200 inputs the source language sentence to the encoder 70 to calculate the output result X1 (step S304). The training unit 252 inputs the named entity identification information to the encoder 70 to calculate the output result X2 (step S305).

[0137] The training unit 252 calculates joint attention using the Encoder-Decoder attention layer 83 (step S306). The training unit 252 calculates the error between the value output from the Affine layer 84 and the vector of the word in the target language sentence (step S307). The training unit 252 performs machine learning to reduce the error (step S308).

[0138] If the translation device 200 continues the process (step S309, Yes), the translation device 200 proceeds to step S301. On the other hand, if the translation device 200 does not continue the process (step S309, No), the translation device 200 ends the training of the translation model 243.

[0139] Fig. 12 is a flowchart of a process during translation according to the present embodiment 2. As shown in Fig. 12, the identification unit 251 of the translation device 200 acquires a source language sentence to be translated from the input unit 220 (step S401). The identification unit 251 identifies a named entity from the source language sentence based on the named entity dictionary 242 (step S402).

[0140] The identification unit 251 generates named entity identification information based on the result of identifying the named entity (step S403). The translation unit 253 of the translation device 200 inputs the source language sentence to the encoder 70 to calculate the output result X1 (step S404). The translation unit 253 inputs the named entity identification information to the encoder 70 to calculate the output result X2 (step S405).

[0141] The translation unit 253 calculates joint attention using the Encoder-Decoder attention layer 83 (step S406). The translation unit 253 identifies a target language sentence from the output result of the Softmax layer 85 (step S407). The translation unit 253 outputs the identified target language sentence to the display unit 230 (step S408).

[0142] Next, the effects of the translation device 200 according to the second embodiment will be described. When performing machine learning using training data 241, the translation device 200 identifies named entities in the source language and generates a translation model 243 including an attention mechanism for named entities (encoder-decoder attention layer 83). By using this translation model 243, translation can be performed while taking into consideration the balance between the meanings of words in the source language sentence and the named entities, thereby preventing erroneous translation of slang words including named entities.

[0143] Next, an example of the hardware configuration of a computer that realizes the same functions as the translation device 100 (200) shown in this embodiment will be described. Fig. 13 is a diagram showing an example of the hardware configuration of a computer that realizes the same functions as the translation device according to this embodiment.

[0144] 13, computer 300 has CPU 301 that executes various types of arithmetic processing, input device 302 that accepts data input from a user, and display 303. Computer 300 also has reading device 304 that reads programs and the like from a storage medium, and interface device 305 that exchanges data with external devices and the like via a wired or wireless network. Computer 300 also has RAM 306 that temporarily stores various types of information, and hard disk drive 307. Each of devices 301 to 307 is connected to bus 308.

[0145] The hard disk drive 307 stores a specific program 307 a, a training program 307 b, and a translation program 307 c. The CPU 301 reads out the specific program 307 a, the training program 307 b, and the translation program 307 c and loads them into the RAM 306.

[0146] The specific program 307a functions as the specific process 306a, the training program 307b functions as the training process 306b, and the translation program 307c functions as the translation process 306c.

[0147] The processing of the identification process 306a corresponds to the processing of the identification units 151 and 251. The processing of the training process 306b corresponds to the processing of the training units 152 and 252. The processing of the translation process 306c corresponds to the processing of the translation units 153 and 253.

[0148] It should be noted that each of the programs 307a to 307c does not necessarily have to be stored in the hard disk drive 507 from the beginning. For example, each of the programs may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD disk, magneto-optical disk, or IC card that is inserted into the computer 300. Then, the computer 300 may read and execute each of the programs 307a to 307c.

[0149] The following supplementary notes are further disclosed regarding the embodiments including the above examples.

[0150] (Supplementary Note 1) Acquire training data including a first sentence expressed in a first language and a second sentence expressed in a second language; Identifying named entities and parts of speech from the first sentence; generating a translation model based on the training data, the translation model including an attention mechanism for the named entities and the parts of speech; A machine learning program that makes a computer perform processing.

[0151] (Supplementary Note 2) The translation model has an encoder and a decoder, the calculation result of the encoder is input to the decoder, and the decoder has the attention mechanism; based on the result of identifying the named entity, further causing the computer to execute a process of converting words included in the first sentence into a code indicating the named entity and a code not indicating the named entity, into named entity identification information; The process of generating the translation model includes a process of training the translation model using values ​​obtained by inputting a first calculation result calculated as a result of inputting the first sentence into the encoder and a second calculation result calculated as a result of inputting the named entity identification information into the encoder into the attention mechanism. 2. The machine learning program according to claim 1,

[0152] (Appendix 3) The method further causes the computer to execute a process of converting words included in the first sentence into part-of-speech identification information, based on the part-of-speech identification result, into codes corresponding to the parts of speech of the words; The process of generating the translation model includes a process of further training the translation model using a value obtained by inputting a third calculation result, which is calculated as a result of inputting the part-of-speech identification information to the encoder, into the attention mechanism. 3. The machine learning program according to claim 2,

[0153] (Appendix 4) Obtaining training data including a first sentence expressed in a first language and a second sentence expressed in a second language; Identifying named entities and parts of speech from the first sentence; generating a translation model based on the training data, the translation model including an attention mechanism for the named entities and the parts of speech; A machine learning method characterized in that the processing is executed by a computer.

[0154] (Supplementary Note 5) The translation model has an encoder and a decoder, the calculation result of the encoder is input to the decoder, and the decoder has the attention mechanism; Based on the result of identifying the named entity, further execute a process of converting words included in the first sentence into a code indicating the named entity and a code not indicating the named entity into named entity identification information; The process of generating the translation model includes a process of training the translation model using values ​​obtained by inputting a first calculation result calculated as a result of inputting the first sentence into the encoder and a second calculation result calculated as a result of inputting the named entity identification information into the encoder into the attention mechanism. 5. The machine learning method according to claim 4,

[0155] (Appendix 6) The method further causes the computer to execute a process of converting words included in the first sentence into part-of-speech identification information, which is a code corresponding to the part-of-speech of the word, based on the result of the part-of-speech identification; The process of generating the translation model includes a process of further training the translation model using a value obtained by inputting a third calculation result, which is calculated as a result of inputting the part-of-speech identification information to the encoder, into the attention mechanism. 6. The machine learning method according to claim 5,

[0156] (Appendix 7) Obtaining a first sentence expressed in a first language; Identifying named entities and parts of speech from the first sentence; generating a second sentence expressed in a second language based on the first sentence and a translation model including an attention mechanism for the named entities and the parts of speech; A translation device having a control unit including processing.

[0157] (Supplementary Note 8) The translation model has an encoder and a decoder, the calculation result of the encoder is input to the decoder, and the decoder has the attention mechanism; The translation device described in Appendix 7 is characterized in that the control unit converts words contained in the first sentence into codes indicating named entities and named entity identification information that converts words contained in the first sentence into codes not indicating named entities based on the result of identifying the named entities, and generates the second sentence using values ​​obtained by inputting into the attention mechanism a first calculation result calculated as a result of inputting the first sentence into the encoder and a second calculation result calculated as a result of inputting the named entity identification information into the encoder.

[0158] (Supplementary Note 9) The control unit further executes a process of converting words included in the first sentence into part-of-speech identification information, which is a code corresponding to the part-of-speech of the word, based on the part-of-speech identification result; The translation device described in Appendix 8 is characterized in that the second sentence is generated by further using a value obtained by inputting the third calculation result calculated as a result of inputting the part-of-speech identification information into the encoder into the attention mechanism. [Explanation of symbols]

[0159] 100,200 translation devices 110,210 Communications Department 120,220 Input section 130,230 Display section 140,240 storage section 141,241 training data 142,242 Named Entity Dictionary 143,243 translation models 150,250 Control unit 151,251 Specific part 152,252 Training Department 153,253 Translation Department< / eo> < / bos> < / bos> < / bos> < / bos> < / bos>

Claims

1. A translation model having an encoder and a decoder, wherein a calculation result of the encoder is input to the decoder, and the decoder acquires training data including a first sentence expressed in a first language and a second sentence expressed in a second language when training the translation model having an attention mechanism; Identifying named entities and parts of speech from the first sentence based on a named entity dictionary that stores named entities in the first language in association with types of named entities; converting words included in the first sentence into a code indicating a named entity and a code not indicating a named entity based on the result of identifying the named entity, into named entity identification information; Based on the result of identifying the part of speech, the words included in the first sentence are identified according to the part of speech of the words. The part of speech is converted into a code to identify the part of speech. The translation model is trained so that a value obtained by inputting a first calculation result calculated as a result of inputting the first sentence to the encoder, a second calculation result calculated as a result of inputting the named entity identification information to the encoder, and a third calculation result calculated as a result of inputting the part of speech identification information to the encoder into the attention mechanism approaches a vector of a word in the second sentence. A machine learning program that causes a computer to perform processing.

2. A translation model having an encoder and a decoder, wherein a calculation result of the encoder is input to the decoder, and the decoder acquires training data including a first sentence expressed in a first language and a second sentence expressed in a second language when training the translation model having an attention mechanism; Identifying named entities and parts of speech from the first sentence based on a named entity dictionary that associates named entities in the first language with types of named entities; converting words included in the first sentence into a code indicating a named entity and a code not indicating a named entity based on the result of identifying the named entity, into named entity identification information; Based on the result of identifying the part of speech, the words included in the first sentence are identified according to the part of speech of the words. The part of speech is converted into a code to identify the part of speech. training the translation model so that a value obtained by inputting into the attention mechanism a first calculation result calculated as a result of inputting the first sentence into the encoder, a second calculation result calculated as a result of inputting the named entity identification information into the encoder, and a third calculation result calculated as a result of inputting the part of speech identification information into the encoder approaches a vector of a word in the second sentence; A machine learning method characterized in that the processing is executed by a computer.

3. A trained translation model having an encoder and a decoder, wherein a calculation result of the encoder is input to the decoder, and the decoder obtains a first sentence expressed in a first language when translating using the translation model having an attention mechanism; Identifying named entities and parts of speech from the first sentence based on a named entity dictionary that associates named entities in the first language with types of named entities; converting words included in the first sentence into a code indicating a named entity and a code not indicating a named entity based on the result of identifying the named entity, into named entity identification information; Based on the result of identifying the part of speech, the words included in the first sentence are identified according to the part of speech of the words. The part of speech is converted into a code to identify the part of speech. calculates probabilities of words expressed in a second language based on values ​​obtained by inputting into the attention mechanism a first calculation result calculated as a result of inputting the first sentence into the encoder, a second calculation result calculated as a result of inputting the named entity identification information into the encoder, and a third calculation result calculated as a result of inputting the part of speech identification information into the encoder, and generates a second sentence based on the probabilities; A translation device having a control unit that executes processing.

Images