Natural language processing device, learning device, translation device, and program

Abstract

To provide a natural language processing device that can improve efficiency of learning of a translation model for limited vocabulary machine translation while automatically selecting a vocabulary to be limited.SOLUTION: A natural language processing device comprises: a limited vocabulary selection unit; and a learning data generation unit. The limited vocabulary selection unit reads a parallel translated sentence pair having an original language sentence and a target language sentence, and selects words, of words included in the target language sentence as a limited vocabulary, on the basis of at least either measure of importance in the target language sentence or fallibility in machine translation processing. The learning data generation unit generates a pair of input side data that is constituted to include the original language sentence and information of the selected limited vocabulary, and output side data that is constituted by the target language sentence as learning data of a vocabulary limited translation model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a natural language processing device, a learning device, a translation device, and a program. [Background technology]

[0002] Natural language machine translation using neural networks and other technologies has been put to practical use, but there is a demand for further improvements in translation accuracy. Neural machine translation is achieved by training a neural network using a large amount of bilingual data between source and target language sentences.

[0003] Machine translation using machine learning has the problem of being unable to standardize translations because translation results are output for each sentence. As an example, when machine translating an English sentence containing the expression "Tokyo Disneyland" into Japanese, the translation result may be "Tokyo Disneyland" or "TDL" depending on the sentence, resulting in a lack of consistency. One method for standardizing such translations is limited vocabulary machine translation. In limited vocabulary machine translation, some words in the target language (words that should be standardized, in other words, information about vocabulary that should be restricted) are added to the data in the source language sentence that is input into the translation model. In other words, the method attempts to control the expression of the target language sentence that is output by adding information about vocabulary constraints to the input data.

[0004] Previous research on restricted vocabulary machine translation has been divided into hard lexical constraints and soft lexical constraints. In hard vocabulary restricted machine translation, a translation model is constructed that always outputs the restricted vocabulary. In soft vocabulary restricted machine translation, a translation model is constructed that does not necessarily output the restricted vocabulary.

[0005] Non-Patent Documents 1 and 2 describe hard-type limited vocabulary machine translation techniques. Non-Patent Documents 3 and 4 describe soft-type limited vocabulary machine translation techniques.

[0006] The methods described in Non-Patent Documents 1, 2, and 3 use a model that requires a word in the target language and a corresponding word in the source language as vocabulary restriction information. That is, in the above example, data pairs of "Tokyo Disneyland" and "Tokyo Disneyland" are required. On the other hand, the method described in Non-Patent Document 4 uses a model that requires only a word in the target language, but not a word in the source language, as vocabulary restriction information. That is, in the above example, only data "Tokyo Disneyland" is required as vocabulary restriction information, and data on the source language word "Tokyo Disneyland," which is the corresponding translation of "Tokyo Disneyland," is not required. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] Chris Hokamp, ​​Qun Liu, Lexically constrained decoding for sequence generation using grid beam search, In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1535-1546, Vancouver, Canada, Association for Computational Linguistics, 2017. [Non-patent document 2] Matt Post, David Vilar, Fast lexically constrained decoding with dynamic beam allocation for neural machine translation, In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1314-1324, New Orleans, Louisiana, Association for Computational Linguistics, 2018.

Non-Patent Document 3

Non-Patent Document 4

[0008] However, the conventional limited vocabulary machine translation techniques have the following problems. For example, in soft-type restricted vocabulary machine translation techniques (Non-Patent Document 4), the vocabulary to be restricted during training of a translation model is a set of words randomly selected from the target language data. If the restricted vocabulary during training is selected in this manner, vocabulary that does not actually need to be restricted may be selected, resulting in a problem of inefficient training of the translation model. In other words, the randomly selected restricted vocabulary includes vocabulary that does not need to be restricted, preventing proper training of the translation model. This problem can occur in both hard-type and soft-type restricted vocabulary machine translation.

[0009] In order to proceed with training more appropriately, it is necessary to select vocabulary that is likely to be selected as restricted vocabulary during translation as training data. In other words, it is desirable to select only vocabulary that is likely to be restricted and use it as training data for the translation model. However, manually selecting vocabulary that is likely to be restricted is undesirable because it requires too much effort and cost.

[0010] The present invention was made based on this recognition of the problem, and aims to provide a natural language processing device, a learning device, a translation device, and a program that can increase the efficiency of learning a translation model for limited vocabulary machine translation while automatically selecting vocabulary to be restricted. [Means for solving the problem]

[0011] [1] In order to solve the above problem, a natural language processing device according to one aspect of the present invention includes a restricted vocabulary selection unit that reads a bilingual sentence pair having a source language sentence and a target language sentence, and selects, from among the words included in the target language sentence, words as restricted vocabulary based on at least one measure of importance in the target language sentence or likelihood of error in machine translation processing, and a training data generation unit that generates, as training data for a vocabulary-limited translation model, pairs of input data that includes the source language sentence and information on the selected restricted vocabulary, and output data that is composed of the target language sentence.

[0012] [2] Furthermore, in one aspect of the present invention, in the natural language processing device of [1] above, the restricted vocabulary selection unit inputs the source language sentence included in the bilingual sentence pair into a predetermined translation model to obtain a translation result sentence output by the translation model, and selects words that are included in the target language sentence included in the bilingual sentence pair but are not included in the translation result sentence as restricted vocabulary based on the fallibility measure.

[0013] [3] Furthermore, in one aspect of the present invention, in the natural language processing device of [1] or [2] above, the restricted vocabulary selection unit calculates a TF-IDF value for a word contained in the target language sentence in a set of bilingual sentence pairs, which is a set of bilingual sentence pairs, and selects restricted vocabulary using the calculated TF-IDF as the importance.

[0014] [4] Furthermore, a learning device according to one aspect of the present invention includes a learning processing unit that performs machine learning of a limited-vocabulary translation model using the learning data generated by any of the natural language processing devices described above in [1] to [3].

[0015] [5] Furthermore, a translation device according to one aspect of the present invention includes a translation processing unit having a vocabulary-limited translation model machine-learned by the learning device described in [4] above, a source language sentence input unit that passes a source language sentence to be translated to the translation processing unit, and a target language sentence output unit that outputs a target language sentence that is the translation result output by the translation processing unit based on the source language sentence.

[0016] [6] Furthermore, a translation device (302) according to one aspect of the present invention comprises: a translation processing unit (42) having a trained translation model, which performs translation processing of a source language sentence and outputs a target language sentence as a translation result; a source language sentence input unit (321) which passes the source language sentence to be translated to the translation processing unit; an external information input unit (322) which passes external information that may be a constraint when translating the source language sentence to the translation processing unit; and a target language sentence output unit (43) which outputs a target language sentence as a translation result output by the translation processing unit based on the source language sentence and the external information, wherein the translation model provided in the translation processing unit (42) has been trained in advance using training data so as to perform translation processing under the constraints represented by the external information, and outputs the target language sentence corresponding to the source language sentence under the constraints.

[0017] [7] Furthermore, in a translation device according to one aspect of the present invention, in the translation device of [6] above, the external information includes at least one of vocabulary constraint information, which is information about vocabulary constraints in the target language sentence output by the translation processing unit (42) as a translation result; dictionary information, which is information representing the correspondence between expressions in the source language sentence and expressions in the target language sentence; and grammar information, which is information about the grammatical structure of at least one of the source language sentence and the target language sentence.

[0018] [8] Furthermore, a learning device according to one aspect of the present invention includes a learning processing unit (331) that performs machine learning of a translation model using a pair of a source language sentence to be translated by the translation device (302) described above in [6] and a target language sentence output by the translation device (302) corresponding to the source language sentence as a bilingual sentence pair.

[0019] [9] Furthermore, a learning device according to one aspect of the present invention comprises a student translation processing unit (342) equipped with a student translation model machine-learned by the learning device described above in [8], a source language sentence input unit (341) that passes a source language sentence to be translated to the student translation processing unit (342), and a target language sentence output unit (343) that outputs a target language sentence that is the translation result output by the student translation processing unit (342) based on the source language sentence.

[0020]

[10] Another aspect of the present invention is a program for causing a computer to function as any one of the natural language processing devices described above in [1] to [3].

[0021]

[11] Another aspect of the present invention is a program for causing a computer to function as the learning device described in [4] or [8] above.

[0022]

[12] Another aspect of the present invention is a program for causing a computer to function as any one of the translation devices described above in [5] to [7] and [9]. [Effects of the Invention]

[0023] According to the present invention, a natural language processing device selects a more appropriate restricted vocabulary, thereby improving translation accuracy when using a limited vocabulary machine translation technique. [Brief explanation of the drawings]

[0024] [Figure 1] 1 is a block diagram showing a schematic functional configuration of a natural language processing apparatus according to a first embodiment of the present invention. [Figure 2] FIG. 2 is a schematic diagram showing an example of a translation model (Transformer) assumed by the natural language processing apparatus according to the first embodiment. [Figure 3] 1 is a schematic diagram showing a translation model assumed by a natural language processing apparatus according to a first embodiment and input / output data of the translation model. [Figure 4]FIG. 3 is a schematic diagram showing a process by a first technique for selecting restricted vocabulary by a restricted vocabulary selection unit according to the first embodiment. [Figure 5] FIG. 10 is a block diagram showing a schematic functional configuration of a natural language processing apparatus according to a second embodiment. [Figure 6] FIG. 1 is a block diagram showing an example of the internal configuration of a natural language processing apparatus according to a first embodiment or a second embodiment. [Figure 7] FIG. 10 is a schematic diagram showing an evaluation result (BLEU value, in the case of Japanese-English news translation) of the translation accuracy by the natural language processing apparatus according to the second embodiment. [Figure 8] FIG. 10 is a schematic diagram showing an evaluation result (BLEU value, in the case of English-Japanese news translation) of the translation accuracy by the natural language processing apparatus according to the second embodiment. [Figure 9] FIG. 10 is a block diagram showing a schematic functional configuration of a natural language processing apparatus according to a third embodiment. [Figure 10] 10 is a table showing a comparison of the translation accuracy when Japanese-English translation is performed using the third embodiment with that of the prior art. The evaluation value is based on BLEU. [Figure 11] 10 is a table showing a comparison of the translation accuracy when English-Japanese translation is performed using the third embodiment with that of the prior art. The evaluation value is based on BLEU. [Figure 12] This is a table showing the results of a comparison of translation speed (number of translated sentences per second) when the number of layers (number of internal parameters) of the translation model is different. DETAILED DESCRIPTION OF THE INVENTION

[0025] Next, several embodiments of the present invention will be described with reference to the drawings.

[0026] [First embodiment] In this embodiment, a technique of limited vocabulary machine translation is used as a means for solving the problem of consistent translation. That is, in this embodiment, information on the limited vocabulary in the target language sentence is added to the source language sentence on the input data side of the translation model. The natural language processing apparatus 1 of this embodiment generates training data required for training the limited vocabulary translation model. The natural language processing apparatus 1 extracts (selects) limited vocabulary from the target language sentence in the bilingual sentence pair based on appropriate criteria.

[0027] In this embodiment, the natural language processing apparatus 1 selects words to be restricted during training of a translation model based on the criteria of words that are likely to be mistranslated by existing machine translators or words that have high importance in a sentence. Furthermore, the natural language processing apparatus 1 selects these vocabulary words automatically, without relying on human intervention or judgment. Specifically, with regard to the criteria of words that are likely to be mistranslated by existing machine translators, an existing machine translator (model) is used to actually machine translate a source language sentence, and the machine translation result is compared with a correct translation (a target language sentence in a bilingual sentence pair). Words that are included in the correct translation but not in the machine translation result are then extracted as restricted vocabulary. With regard to the criteria of words that have high importance in a sentence, the importance is determined using, for example, the TF-IDF scale, and the words with high importance are extracted as restricted vocabulary.

[0028] Of these measures, the reason for using the criterion of words that are prone to errors by existing machine translators is as follows: for words that existing machine translators do not make errors on, it is reasonable to assume that the translation results will not change whether or not they are added to the restricted vocabulary. Conversely, by restricting words that are prone to mistranslation in existing machine translators, it is expected that translation accuracy will improve.

[0029] The evaluation results when restricted vocabulary is selected based on the above criteria will be explained later.

[0030] FIG. 1 is a block diagram showing a schematic functional configuration of a natural language processing apparatus according to a first embodiment. As shown in the figure, the natural language processing apparatus 1 includes a bilingual sentence pair storage unit 21, a restricted vocabulary selection unit 22, a training data generation unit 23, and a training data storage unit 24. With this configuration, the natural language processing apparatus 1 generates training data for a translation model. The translation model receives a translation target sentence (source language sentence) as input and outputs a translation result sentence (target language sentence) corresponding to the translation target sentence. Note that the source language and the target language may each be any natural language. The source language and the target language may be, for example, any of Japanese, English, German, French, Russian, Spanish, Italian, Chinese, Korean, etc., but are not limited to these.

[0031] Each of the functional units (listed above) constituting the natural language processing device 1 can be realized, for example, by a computer and a program. Furthermore, each functional unit has a storage means as needed. The storage means is, for example, a variable in the program or a memory allocated by the execution of the program. Furthermore, non-volatile storage means such as a magnetic hard disk drive or a solid state drive (SSD) may be used as needed. Furthermore, at least a part of the functions of each functional unit may be realized as a dedicated electronic circuit rather than a program. The functions of each unit are as follows:

[0032] The bilingual text pair storage unit 21 stores bilingual text pairs each consisting of a source language sentence and a target language sentence. It is desirable that the bilingual text pair storage unit 21 stores a large number of bilingual text pairs. The bilingual text pair storage unit 21 is also called a bilingual text pair corpus.

[0033] The restricted vocabulary selection unit 22 reads a bilingual sentence pair having a source language sentence and a target language sentence from the bilingual sentence pair storage unit 21. The restricted vocabulary selection unit 22 selects words from the target language sentence based on at least one of the measures of importance in the target language sentence and likelihood of error in machine translation processing. That is, the restricted vocabulary selection unit 22 selects these words as restricted vocabulary. Note that the restricted vocabulary selection unit 22 selects zero or more words from one target language sentence as restricted vocabulary.

[0034] The restricted vocabulary selection unit 22 inputs the source language sentences included in the bilingual sentence pair into a predetermined translation model to obtain translation result sentences output by the translation model. The restricted vocabulary selection unit 22 may then select, as restricted vocabulary based on the error-proneness measure, words that are included in the target language sentences included in the bilingual sentence pair but are not included in the translation result sentences.

[0035] Furthermore, the restricted vocabulary selection unit 22 may calculate the TF-IDF value for the words contained in the target language sentence in a set of parallel translation sentence pairs (a corpus stored in the parallel translation sentence pair storage unit 21), and select the restricted vocabulary using the calculated TF-IDF as the importance.

[0036] The training data generation unit 23 generates training data for training a vocabulary-restricted translation model using information on the restricted vocabulary selected by the restricted vocabulary selection unit 22. This training data is configured as pairs of input data and output data. The input data includes source language sentences included in the bilingual sentence pair and information on the restricted vocabulary selected by the restricted vocabulary selection unit 22. The output data is configured as data including target language sentences included in the bilingual sentence pair. The training data generated by the training data generation unit 23 will be described later with reference to FIG. 3.

[0037] The training data storage unit 24 stores the training data generated by the training data generation unit 23. In other words, the training data storage unit 24 stores a large amount of pairs of input data and output data generated by the training data generation unit 23. When machine learning of a translation model is performed, the training data is read from this training data storage unit 24.

[0038] FIG. 2 is a schematic diagram showing an example of a translation model assumed by the natural language processing system 1. The translation model shown here uses a Transformer. Transformers are also described in the reference [Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin: Attention Is All You Need, 2017, arXiv:1706.03762]. The Transformer is implemented using a neural network. The Transformer is configured to include an encoder and a decoder. The encoder accepts a translation target sentence as input data and passes the result of the encoding process to the decoder. The decoder generates and outputs a translation result sentence based on the information passed from the encoder. In the example shown here, the encoder accepts the Japanese sentence "I'm going to Kyoto" as input. In response, the decoder outputs the English sentence "I go to Kyoto." as the translation result. The translation model is trained using training data. The training data can be a set of pairs of source and target language sentences (parallel sentence pairs).

[0039] 3 is a schematic diagram showing a translation model assumed by the natural language processing apparatus 1 and input / output data of the translation model. As shown in the figure, the translation model is made up of an encoder 101 and a decoder 102, for example.

[0040] As shown in the figure, the data input to the encoder 101 has the structure "x1 x2...x K |c1|c2|…|c N ". This input data "x1x2...x K " is the sentence to be translated (source language sentence). x1, x2, ... x K are words that make up the source sentence. K is the length of the source sentence. Also, in this input data, N " is the data that represents the vocabulary restriction. N Each of these is a word contained in the target sentence and is a word that is subject to vocabulary restriction. N is the number of words to be restricted. The data representing the vocabulary restriction follows the source sentence above. The source sentence and the vocabulary restriction data are separated by a delimiter "|". Also, c1, c2, ...c N are separated by the delimiter "|". The delimiter "|" is a special token to represent these delimiters. N " represents a soft vocabulary restriction on the output (target language sentence) from the translation model.

[0041] The data output from the decoder 102 has the structure "y1 y2...y J " This "y1y2...y J " is the translation result sentence (target language sentence). y1, y2, ... y J are words that make up the target sentence. J is the length of the target sentence.

[0042] When training the translation model, the above input data is input to the translation model. Then, the internal parameters of the translation model are adjusted based on the difference (loss) between the output data (candidate translation result sentence) output by the translation model based on the input data and the correct answer data (target language side of the bilingual sentence pair) corresponding to the input data. The backpropagation method is used to adjust the parameters of the translation model. As shown in the figure, the input data includes vocabulary restriction information in addition to information on the source language sentence. As a result, the translation model is trained so that words included in the vocabulary restriction are included in the translation result. The natural language processing apparatus 1 of this embodiment generates the above training data (a set of pairs of input data and output data).

[0043] Note that some of the training data may not include vocabulary restriction information. For training data that does not include vocabulary restriction information, the translation model learns the relationship between source language sentences and target language sentences, but does not learn vocabulary restrictions.

[0044] When translation is executed, the above input data is input to the trained translation model. By having the model trained in advance, the vocabulary restriction information contained in the input data acts so that the restricted vocabulary is included in the target language sentence, which is the translation result. Note that the input data does not need to have vocabulary restriction information when translation is executed. In other words, when translation is executed, the input data is only the source language sentence ("x1x2...x" shown in Figure 3). K Even if the input data does not have vocabulary restriction information, the translation model will tend to output translated sentences (target language sentences) according to the restrictions expressed in the training data at the time of training.

[0045] Next, a method for selecting restricted vocabulary by the restricted vocabulary selection unit 22 in this embodiment will be described. In this embodiment, the restricted vocabulary selection unit 22 selects restricted vocabulary using two types of methods in combination. As a modified example, the restricted vocabulary selection unit 22 may use only one of these two types of methods. The first method and the second method will be described respectively.

[0046] [First method (selection of restricted vocabulary)] FIG. 4 is a schematic diagram illustrating a process performed by the restricted vocabulary selection unit 22 using a first technique for selecting a restricted vocabulary. As illustrated, the restricted vocabulary selection unit 22 selects a specific word (one or more) from among the words included in the target language sentence based on a pair of bilingual sentences retrieved from the bilingual sentence pair storage unit 21. Specifically, the restricted vocabulary selection unit 22 has an internal mechanism for translating a sentence using a translation model. Here, the translation model 221 may be any type of translation model, but is preferably the same type or similar to the translation model to be trained. For example, if the translation model to be trained using training data generated by the natural language processing apparatus 1 is a Transformer, the translation model 221 used by the restricted vocabulary selection unit 22 to use this technique is preferably also a Transformer. The restricted vocabulary selection unit 22 inputs a source language sentence included in a bilingual sentence retrieved from the bilingual sentence pair storage unit 21 to the translation model 221. The translation model 221 outputs a target language sentence (translation result) corresponding to the input source language sentence. The restricted vocabulary selection unit 22 compares the target language sentence output from the translation model 221 as the translation result with the target language sentence (correct answer) included in the bilingual sentence pair, and finds the vocabulary difference between them. Specifically, the restricted vocabulary selection unit 22 identifies a set of words that are included in the correct answer target language sentence but are not included in the target language sentence output as the translation result. The restricted vocabulary selection unit 22 then selects the identified difference as the restricted vocabulary.

[0047] For example, if the source language sentence included in the translation pair is "I went to Tokyo Disneyland yesterday.", the target language sentence included in the translation pair is "I went to Tokyo Disneyland yesterday.", and the target language sentence (translation result) output by the translation model 221 based on the above source language sentence is "I went to TDL yesterday," the restricted vocabulary selection unit 22 selects "Tokyo Disneyland," a word that is included in the target language sentence of the translation pair but not in the target language sentence of the translation result, as the restricted vocabulary.

[0048] [Second method (selection of restricted vocabulary)] In addition to the first method, the restricted vocabulary selection unit 22 uses a second method to select words contained in the target language sentence in the bilingual sentence pair as restricted vocabulary. In the second method, the restricted vocabulary selection unit 22 selects words that are considered to be important using a statistical method. For example, the restricted vocabulary selection unit 22 selects words that are considered to be important by taking statistics of words contained in target language sentences belonging to all elements of the set of bilingual sentence pairs stored in the bilingual sentence pair storage unit 21. Specifically, the restricted vocabulary selection unit 22 evaluates the importance of each word contained in the target language sentence using tf-idf. tf-idf is a statistical method that belongs to existing technology. tf-idf is an abbreviation for "Term Frequency - Inverse Document Frequency." In other words, tf-idf is calculated as the product of Term Frequency (word frequency) and Inverse Document Frequency (inverse document frequency). Specifically, tfidf, which is the tf-idf value of the ith word in the jth document, is used. i,j is calculated by the following formulas (1), (2), and (3). Note that the "document" referred to here is a target language sentence belonging to each of the bilingual text pairs stored in the bilingual text pair storage unit 21.

[0049]

number

[0050]

number

[0051]

number

[0052] In addition, n in formula (2) i,j is documented j The word t in iThe number of occurrences of

number

[0053] In equation (3), |D| is the total number of documents.

number

[0054] Using the above formula, the restricted vocabulary selection unit 22 calculates all tf-idf values ​​for each word in each document (target language sentence in a bilingual sentence pair).The restricted vocabulary selection unit 22 then determines words (target language words) with relatively high tf-idf values ​​as important words in each target language sentence.The restricted vocabulary selection unit 22 selects the set of words determined as important words for each target language sentence as the restricted vocabulary for that target language sentence.

[0055] Specifically, the restricted vocabulary selection unit 22 selects words (words in the target language) to be included in the restricted vocabulary for each bilingual sentence pair based on one of the following criteria related to the tf-idf value. For example, the restricted vocabulary selection unit 22 selects, as the restricted vocabulary for the bilingual sentence pair, words (0 or more) having a tf-idf value equal to or greater than a predetermined threshold. Alternatively, the restricted vocabulary selection unit 22 selects, as the restricted vocabulary for the bilingual sentence pair, a predetermined number (a predetermined number) of words with the highest tf-idf values ​​among the words included in the target language sentence of a bilingual sentence pair. Alternatively, the restricted vocabulary selection unit 22 selects, as the restricted vocabulary for the bilingual sentence pair, the (W·R) words with the highest tf-idf values ​​(wherein fractions after the decimal point are rounded down, rounded up, or rounded to the nearest whole number) among the words (number of words W) included in the target language sentence of a bilingual sentence pair based on a predetermined ratio R. Alternatively, the restricted vocabulary selection unit 22 uses other criteria to select, for each bilingual text pair, words (0 or more) with relatively high tf-idf values ​​as restricted vocabulary for that bilingual text pair.

[0056] When using both the first and second methods, the restricted vocabulary selection unit 22 selects, for each bilingual text pair, for example, words belonging to the union of the set of target language words selected by the first method and the set of target language words selected by the second method as the restricted vocabulary for that bilingual text pair. Alternatively, the restricted vocabulary selection unit 22 may use other methods to appropriately combine the set of target language words selected by the first method and the set of target language words selected by the second method, and set the selected words as the restricted vocabulary for that bilingual text pair.

[0057] As described above, the restricted vocabulary selection unit 22 selects restricted vocabulary for each of a large number of bilingual sentence pairs using the first method or the second method. The restricted vocabulary selection unit 22 passes information on the selected restricted vocabulary to the training data generation unit 23. This enables the training data generation unit 23 to generate training data that includes source language sentences and restricted vocabulary information (information on vocabulary restrictions related to target language sentences) on the input data side, and has target language sentences on the output data side.

[0058] As described above, the natural language processing apparatus 1 of this embodiment can generate training data that can be used to train a translation model.

[0059] [Second embodiment] Next, a second embodiment of the present invention will be described. Note that the matters already described in the previous embodiment may not be described below. Here, the description will focus on matters unique to this embodiment. The natural language processing apparatus 2 of the second embodiment has a configuration similar to that of the first embodiment, and generates training data based on a set of bilingual sentence pairs. The natural language processing apparatus 2 performs machine learning on a translation model using the generated training data. Furthermore, the natural language processing apparatus 2 uses a translation model trained using the training data to generate and output a target language sentence as a translation result based on an input source language sentence. Because the translation model is trained using information on restricted vocabulary included in the training data, the vocabulary of the translation result output by such a translation model is a restricted vocabulary.

[0060] 5 is a block diagram showing a schematic functional configuration of a natural language processing apparatus according to this embodiment. As shown in the figure, the natural language processing apparatus 2 includes a bilingual sentence pair storage unit 21, a restricted vocabulary selection unit 22, a training data generation unit 23, a training data storage unit 24, a training processing unit 31, a source language sentence input unit 41, a translation processing unit 42, and a target language sentence output unit 43. As in the first embodiment, each unit constituting the natural language processing apparatus 2 can be realized using a computer and a program. As in the first embodiment, at least some of the functions constituting the natural language processing apparatus 2 may be realized using a dedicated electronic circuit.

[0061] Among the functions of the natural language processing apparatus 2, a device configured with the learning processing unit 31 and the translation processing unit 42 may be called a learning device 7. The learning device 7 performs machine learning of the translation model provided in the translation processing unit 42, using learning data read out from the learning data storage unit 24. Also, among the functions of the natural language processing apparatus 2, a device configured with the source language sentence input unit 41, the translation processing unit 42, and the target language sentence output unit 43 may be called a translation device 8. The translation device 8 translates a source language sentence into a target language sentence, using a translation model that has been trained with the learning data read out from the learning data storage unit 24.

[0062] The bilingual sentence pair storage unit 21, the restricted vocabulary selection unit 22, the training data generation unit 23, and the training data storage unit 24 each have the same functions as the functional units of the same names that configure the natural language processing apparatus 1 of the first embodiment. That is, the restricted vocabulary selection unit 22 and the training data generation unit 23 generate training data based on the bilingual sentence pair data stored in the bilingual sentence pair storage unit 21. The generated training data is written to the training data storage unit 24. That is, the training data storage unit 24 stores the generated training data.

[0063] The learning processing unit 31 performs machine learning of the translation model held by the translation processing unit 42 described below. The learning processing unit 31 performs machine learning of the translation model using the learning data stored in the learning data storage unit 24 (learning data generated by the learning data generation unit 23). As described in the first embodiment, the learning data stored in the learning data storage unit 24 is learning data for limited vocabulary machine translation.

[0064] The source language sentence input unit 41 acquires a source language sentence to be translated, for example, from an external source, and passes the source language sentence to the translation processing unit 42 for translation. The source language sentence input unit 41 may add restricted vocabulary information when passing the source language sentence to the translation processing unit 42. The format of the data to which the restricted vocabulary information has been added is as explained with reference to FIG. 3 (first embodiment). In this case, the restricted vocabulary information may include zero or more words in the target language. In other words, c1, c2, ...c explained with reference to FIG. 3 (first embodiment) N is the information of the restricted vocabulary, and N may be 0 or greater.

[0065] The translation processing unit 42 is equipped with a translation model (limited vocabulary translation model), receives an input sentence (sentence to be translated, source language sentence) as input, and outputs an output sentence (translation result sentence, target language sentence). The translation processing unit 42 can operate in both a learning mode and a translation execution mode. In the learning mode, the translation processing unit 42 performs machine learning of the internal translation model. When the translation model is configured using a neural network, the backpropagation method, for example, can be used for the machine learning. In the translation execution mode, the translation processing unit 42 uses the trained translation model to perform translation processing of the input sentence passed from the source language sentence input unit 41, and passes the translation result to the target language sentence output unit 43.

[0066] The target language sentence output unit 43 outputs the target language sentence, which is the translation result output by the translation processing unit 42 based on the input source language sentence, to, for example, the outside.

[0067] FIG. 6 is a block diagram showing an example of the internal configuration of a natural language processing device according to the first embodiment, the second embodiment, etc. The natural language processing devices 1 and 2 can be implemented using a computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, etc., and a bus 906. The computer itself can be implemented using existing technology. The central processing unit 901 executes instructions contained in a program read from the RAM 902, etc. In accordance with each instruction, the central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with external input / output devices, etc. The input / output devices 904 and 905 are input / output devices. Input / output devices 904 and 905 exchange data with the central processing unit 901 via an input / output port 903. A bus 906 is a common communication path used within the computer. For example, the central processing unit 901 reads and writes data from and to RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port via the bus 906.

[0068] At least some of the functions of the natural language processing devices 1 and 2 in the above-described embodiments can be realized by a computer and a program. In this case, the functions can be realized by recording a program for realizing the functions on a computer-readable recording medium and loading and executing the program recorded on the recording medium into a computer system. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, CD-ROMs, DVD-ROMs, and USB memory, as well as storage devices such as hard disks built into computer systems. In other words, a "computer-readable recording medium" may be a non-transitory computer-readable recording medium. Furthermore, the term "computer-readable recording medium" may also include media that temporarily and dynamically store programs, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, or media that store programs for a certain period of time, such as volatile memory within a computer system that serves as a server or client in such cases. The program may be designed to realize some of the functions described above, or may be capable of realizing the functions described above in combination with a program already stored in the computer system.

[0069] Although multiple embodiments have been described above, the present invention can also be implemented in the following modified examples. Of the functions of the natural language processing apparatus 2 (FIG. 5) of the second embodiment, only the function of the learning device 7 may be implemented as an independent device. Similarly, of the functions of the natural language processing apparatus 2 of the second embodiment, only the function of the translation device 8 may be implemented as an independent device. Furthermore, of the functions of the natural language processing apparatus 2 of the second embodiment, the functions of the learning device 7 and the translation device 8 may both be implemented as an independent device.

[0070] [Evaluation Results (First and Second Embodiments)] The natural language processing device described above was actually constructed, and an evaluation experiment was conducted to measure translation accuracy. The results are described below.

[0071] FIG. 7 is a table of evaluation values ​​for comparing the translation accuracy of a news sentence (Japanese-English news translation) using conventional technology with that using the natural language processing device 2 of the second embodiment of the present application, when the source language is Japanese and the target language is English. The score used for evaluation is the BLEU score. BLEU stands for "Bilingual Evaluation Understudy." The methods compared are all conventional technology, and include two types: one using a translation model with no vocabulary restriction, and the other using a translation model with a vocabulary restriction and a randomly selected vocabulary restriction. On the other hand, the method described in the second embodiment achieves vocabulary restriction by using a translation model trained using training data restricted by vocabulary selected by both the first and second methods described above. The scores obtained by evaluating the translation results of the translation models using these methods are as follows: The BLEU score when a translation model with no vocabulary restriction (comparison) was used was 29.3. The BLEU score for the comparison example, where a translation model with limited vocabulary and limited by randomly selected vocabulary, was 29.5, while the BLEU score for the comparison example using the method of the second embodiment was 43.9.

[0072] FIG. 8 is a table of evaluation values ​​for comparing the translation accuracy of a news sentence (English-Japanese news translation) when the source language is English and the target language is Japanese using the conventional technology and the natural language processing device 2 of the second embodiment of the present application. The comparison targets are as described with reference to FIG. 7. The scores obtained by evaluating the translation results of the translation model using each method are as follows: When a translation model with no vocabulary restriction was used (comparison target), the BLEU score was 45.9. When a translation model with a vocabulary restriction was used and the vocabulary was restricted by randomly selected vocabulary (comparison target), the BLEU score was 37.4. On the other hand, when the method of the second embodiment was used, the BLEU score was 53.2.

[0073] As described above, in both the Japanese-English and English-Japanese news translation cases, the scores of the method according to the above embodiment are significantly higher than the scores of the conventional technology used for comparison. This demonstrates the high effectiveness of the above embodiment.

[0074] [Third embodiment] Next, a third embodiment will be described. The third embodiment is a natural language processing apparatus and the like for performing knowledge distillation using external information such as vocabulary constraint information.

[0075] Knowledge distillation is a technique used in machine translation processing. It is used to solve the following two problems. The first problem solved by knowledge distillation is training using noisy training data. Training a translation model using noisy training data results in a decrease in translation accuracy. The second problem solved by knowledge distillation is the large number of parameters a translation model has. As the number of parameters in a translation model increases, the amount of calculation required during training and translation processing increases. In other words, the calculation time during training and translation processing increases. The number of parameters in a translation model depends, for example, on the number of layers in the encoder and decoder within the translation model. To solve these two problems, conventional knowledge distillation techniques retrain a student model using the output from a teacher model.

[0076] In the knowledge distillation technique, a source language sentence is translated using a machine-learned first translation model (teacher model) to obtain a target language sentence. This process is called distillation, and the collection of bilingual sentence pairs (pseudo-bilingual data) of source and target language sentences obtained in this process is called distilled knowledge. These bilingual sentence pairs are used to perform machine learning on a second translation model (student model). As a result of training this second translation model (student model), translation processing can be performed using the second translation model (student model). This is knowledge distillation using conventional technology. Here, the number of internal parameters of the first translation model (teacher model) is relatively large, while the number of internal parameters of the second translation model (student model) is relatively small. In other words, knowledge distillation can reduce the number of parameters of a translation model.

[0077] However, the above-described conventional knowledge distillation technique still has a problem to be solved. That problem is that if the pseudo-bilingual data (data for training the student model) obtained using the teacher model contains mistranslations, the quality of the machine learning of the student model will deteriorate. When the source language sentence is Japanese and the target language sentence is English, the following situation may occur. For example, the source language sentence (Japanese sentence) to be translated is "Changes in flow rate and volume can be detected." The correct target language sentence corresponding to this source language sentence is, for example, "The change of flow rate and volume could be detected." However, the result of translation using the teacher model (target language sentence) may be, for example, "The change of flow rate could be detected." In other words, in this target language sentence, the expression corresponding to volume is missing from translation. If such a translation omission (mistranslation) exists, training the student model using the pseudo-bilingual data will have a negative impact on the student model. When the same source language sentence (Japanese sentence) is translated using the student model, an example of the translation result (target language sentence) is "The change of flow velocity and flow rate could be detected." This translation result contains a mistranslation.

[0078] In conventional knowledge distillation methods, techniques used during training of the teacher model include increasing the number of parameters the teacher model has, or using techniques during decoding (beam search or ensemble decoding), but these techniques alone were not sufficient to achieve sufficient translation accuracy.

[0079] In order to solve the problems with conventional knowledge distillation as described above, in this embodiment, external information is input to the teacher model in addition to the normal source language sentence when performing translation using the teacher model, so that the translation result using the teacher model is appropriate. An example of external information is the vocabulary constraint information described in the first and second embodiments. Note that when training the teacher model, machine learning is performed using external information as well, so that the teacher model can perform appropriate translation using external information. In other words, in this embodiment, external information (which may be called privileged information) that cannot be used by the student model is used when training the teacher model. This external information is not used in training the student model. Because the teacher model performs translation using external information, it is expected that the translation accuracy of the translated sentence (target language sentence) output by the teacher model will be greatly improved.

[0080] FIG. 9 is a block diagram showing a schematic functional configuration of a natural language processing apparatus according to this embodiment. As shown in the figure, the natural language processing apparatus 301 of this embodiment utilizes the configuration of the natural language processing apparatus 2 shown in FIG. 5 and is configured by adding additional functions. That is, as shown in the figure, the natural language processing apparatus 301 includes a bilingual sentence pair storage unit 21, a restricted vocabulary selection unit 22, a training data generation unit 23, a training data storage unit 24, a training processing unit 31, a source language sentence input unit 321, an external information input unit 322, a translation processing unit 42, a target language sentence output unit 43, a training data storage unit 324, a training processing unit 331, a source language sentence input unit 341, a translation processing unit 342, and a target language sentence output unit 343. As with the first and second embodiments, each unit constituting the natural language processing apparatus 301 can be realized using a computer and a program. Furthermore, as with the first and second embodiments, at least some of the functions constituting the natural language processing apparatus 301 may be realized using dedicated electronic circuits.

[0081] Among the functions of the natural language processing apparatus 301, a device configured with the learning processing unit 31 and the translation processing unit 42 may be referred to as a learning apparatus 7. The learning apparatus 7 performs machine learning of the translation model provided in the translation processing unit 42, using learning data read from the learning data storage unit 24. Furthermore, among the functions of the natural language processing apparatus 301, a device configured with the source language sentence input unit 321, the external information input unit 322, the translation processing unit 42, and the target language sentence output unit 43 may be referred to as a translation apparatus 302. The translation apparatus 302 translates a source language sentence into a target language sentence, using a translation model trained with learning data read from the learning data storage unit 24. This translation apparatus 302 has a configuration equivalent to the translation apparatus 8 of the configuration described in the second embodiment ( FIG. 5 ). The functions of the source language sentence input unit 41 in FIG. 5 are equivalent to the functions of the combined configuration of the source language sentence input unit 321 and the external information input unit 322 in FIG. 9. That is, the source language sentence input unit 41 in the second embodiment (FIG. 5) may add information on the restricted vocabulary when passing the source language sentence to the translation processing unit 42. The information on the restricted vocabulary may include zero or more words in the target language. In other words, c1, c2, ...c described in FIG. N was the information on the restricted vocabulary (N≧0). In this embodiment, the information on the restricted vocabulary is a typical example of "external information" that the external information input unit 322 passes to the translation processing unit 42. This "external information" will be described in more detail below. The translation device 302 writes the sentence pair between the source language sentence held by the source language sentence input unit 321 and the target language sentence obtained by the target language sentence output unit 43 into the training data storage unit 324 as training data for training the translation model held by the translation processing unit 342.

[0082] Furthermore, among the functions of the natural language processing device 301, a device configured with the learning processing unit 331 and translation processing unit 342 may be called a learning device 303. The learning device 303 performs machine learning of the translation model provided in the translation processing unit 342 using learning data read out from the learning data storage unit 324. Furthermore, among the functions of the natural language processing device 301, a device configured with the source language sentence input unit 341, translation processing unit 342, and target language sentence output unit 343 may be called a translation device 304. The translation device 304 outputs a target language sentence as a result of translating a source language sentence.

[0083] In this embodiment, the translation model held by the translation processing unit 42 is a teacher model. The translation model held by the translation processing unit 342 is a student model. The number of parameters held by the student model is set to be equal to or smaller than the number of parameters held by the teacher model.

[0084] The functions of each unit shown in Fig. 9 will now be described. Note that the functions of each unit, such as the bilingual text pair storage unit 21, restricted vocabulary selection unit 22, learning data generation unit 23, learning data storage unit 24, learning processing unit 31, translation processing unit 42, and target language sentence output unit 43, have already been described in the explanation of other embodiments.

[0085] The source language sentence input unit 321 passes the source language sentence to be translated to the translation processing unit 42. The source language sentence input unit 321 also writes the source language sentence into the training data storage unit 324 as one piece of data (source language side) that constitutes the bilingual sentence pair.

[0086] The external information input unit 322 passes external information, which is information that can serve as constraints when translating a source language sentence, to the translation processing unit 42. The external information that the external information input unit 322 passes to the translation processing unit 42 may be vocabulary constraint information, which is information about vocabulary constraints in the target language sentence that the translation processing unit 42 outputs as a translation result. Other forms of external information will be described later as modified examples. The process of translating based on vocabulary constraint information is as described in the previous embodiments. When the external information input unit 322 restricts vocabulary using vocabulary constraint information, vocabulary may be selected, for example, using the method described as a function of the restricted vocabulary selection unit 22.

[0087] The translation processing unit 42 is provided with a trained translation model (teacher model), and performs translation processing on the source language sentence passed from the source language sentence input unit 321, and outputs a target language sentence as the translation result.

[0088] The translation model (teacher model) included in the translation processing unit 42 has been trained in advance using training data so as to perform translation processing under the constraints represented by the external information passed from the external information input unit 322. This translation model (teacher model) accepts input of a source language sentence and outputs a target language sentence corresponding to the source language sentence under the constraints. As already explained, the translation model included in the translation processing unit 42 is trained by the training processing unit 31. By using appropriate training data, the training processing unit 31 causes the translation model (teacher model) to train (machine learn) so as to perform translation processing under the constraints represented by the external information passed from the external information input unit 322. The translation processing unit 42 can use a Transformer or other model as the translation model.

[0089] The target language sentence output unit 43 outputs a target language sentence that is the translation result output by the translation processing unit 42, based on the source language sentence passed from the source language sentence input unit 321 and the external information passed from the external information input unit 322. The target language sentence output unit 43 also writes the target language sentence that corresponds to the source language sentence that was the translation target into the training data storage unit 324 as one piece of data (target language side) that constitutes the bilingual sentence pair.

[0090] That is, a pair of a source language sentence passed from the source language sentence input unit 321 and a target language sentence passed from the target language sentence output unit 43 is written as a bilingual sentence pair into the training data storage unit 324. Furthermore, at least a part of the bilingual sentence pairs stored in the bilingual sentence pair storage unit 21 may also be stored in the training data storage unit 324.

[0091] The training data storage unit 324 stores training data used by the training processing unit 331 for training. The training data here is a collection of bilingual sentence pairs, each consisting of a source language sentence to be translated by the translation device 302 and a target language sentence output by the translation device 302 corresponding to the source language sentence. This bilingual sentence pair data is generated using machine translation, and is also called pseudo bilingual sentence pair data. Furthermore, as described above, the training data storage unit 324 may hold at least a portion of the originally given bilingual sentence pairs (bilingual sentence pairs stored in the bilingual sentence pair storage unit 21) as training data to be used below.

[0092] The learning processing unit 331 performs machine learning of a translation model (student model, student translation model) held by the translation processing unit 342, using learning data read from the learning data storage unit 324. In other words, the learning processing unit 331 performs machine learning of a translation model (student model, student translation model) using, as a bilingual sentence pair, a source language sentence to be translated by the translation device 302 and a target language sentence output by the translation device in response to the source language sentence. Furthermore, when the learning processing unit 331 performs machine learning of a translation model (student model) of the translation processing unit 342, it may also use bilingual sentence pairs copied from the bilingual sentence pair storage unit 21.

[0093] The source language sentence input unit 341 passes the source language sentence to be translated to the translation processing unit 342 below.

[0094] The translation processing unit 342 is provided with a translation model (student model, student translation model) that has been machine-learned by the learning device 303, and uses this translation model to translate a source language sentence into a target language sentence. Note that the translation processing unit 342 that performs translation processing using the student model may also be called a "student translation processing unit." The translation processing unit 342 may use a Transformer or other models as the translation model.

[0095] The target language sentence output unit 343 outputs the target language sentence, which is the translation result output by the translation processing unit 342 based on the source language sentence.

[0096] Instead of implementing the entire natural language processing apparatus 301 of the third embodiment as a single device, only some of its functions may be implemented as a single device. For example, only the functions of the translation apparatus 302 may be implemented. The standalone translation apparatus 302 can generate training data for training a student model. Alternatively, for example, only the training apparatus 303 may be implemented. When the standalone learning apparatus 303 is implemented, the training data is appropriately provided from an external source. Furthermore, the training results (parameter values ​​of the translation model (student model)) by the standalone learning apparatus 303 can be used in another translation apparatus. Alternatively, for example, only the translation apparatus 304 may be implemented. In this case, parameter values ​​of the translation model (student model) used by the translation apparatus 304 for translation processing may be provided from an external source. Furthermore, the functions of the translation apparatus 302, the training data storage unit 324, and the training apparatus 303 may be implemented as a single device. Such an apparatus can generate training data using a teacher model and train a student model using the training data. Furthermore, the union of the functions of the learning device 303 and the translation device 304 may be implemented as a single device. Such a device can learn a student model and perform translation processing using a trained student model. Furthermore, the union of the functions of the translation device 302, the training data storage unit 324, the learning device 303, and the translation device 304 may be implemented as a single device. Such a device can generate training data using a teacher model, train a student model using the training data, and perform translation processing using a trained student model. Furthermore, only some of the functions of the natural language processing device 301 may be implemented as a device using other combinations, etc.

[0097] [Evaluation Results (Third Embodiment)] The natural language processing apparatus 301 according to the third embodiment described above was actually constructed, and an evaluation experiment was conducted to measure translation accuracy. The results are described below. The evaluation experiment involved Japanese-to-English translation (source language: Japanese, target language: English) and English-to-Japanese translation (source language: English, target language: Japanese). The data used in the experiment was a scientific and technical paper corpus, with 2 million sentence pairs used as training data and 1,997 sentence pairs used as a test set. A Transformer model was used as the translation model. The number of encoder and decoder layers in the teacher model was 12 encoder layers and 12 decoder layers. The experiment and evaluation were conducted for the following student model encoder and decoder layers: 12 encoder layers and 12 decoder layers, 3 encoder layers and 3 decoder layers, and 1 encoder layer and 1 decoder layer. The fewer the student model layers, the smaller the number of internal parameters. When training the teacher model, beam search (width 10) and ensemble decoding (5 models) were used. The comparison targets (baseline methods) for evaluating the translation accuracy of the third embodiment are neural machine translation without knowledge distillation and neural machine translation with knowledge distillation (knowledge distillation according to conventional technology) without using external information (vocabulary constraint information).

[0098] 10 is a table of evaluation values ​​for comparing the translation accuracy of a sentence translation (Japanese-English translation) when the source language is Japanese and the target language is English in the following cases: (A) without knowledge distillation; (B) when knowledge distillation according to the conventional technology is performed (i.e., when vocabulary constraint information is not provided when translating with a teacher model); and (C) when the natural language processing apparatus 301 of the third embodiment is used (i.e., when vocabulary constraint information is provided as external information when translating with a teacher model). In this embodiment, the score used for evaluation is also the BLEU score.

[0099] As shown in Figure 10, in Japanese-English translation, regardless of whether the number of layers in the student model was 12-12, 3-3, or 1-1, the BLEU score for the method of the third embodiment (C) (knowledge distillation with vocabulary constraint function) was higher than the BLEU scores for the conventional techniques (A) and (B). When using the method of the third embodiment (C), the BLEU score is 27.0 when the number of layers in the student model is 1-1. In other words, even though the number of layers in the student model is 1-1 in this case, a translation accuracy almost equivalent to the BLEU score of 27.2 obtained in the method of (A) when the number of layers in the student model is 12-12 is obtained.

[0100] 11 is a table of evaluation values ​​for comparing the translation accuracy of a sentence translation (English-Japanese translation) when the source language is English and the target language is Japanese, in the following cases: (A) without knowledge distillation; (B) with knowledge distillation according to the conventional technology (i.e., when vocabulary constraint information is not provided when translating with a teacher model); and (C) with the natural language processing apparatus 301 of the third embodiment (i.e., when vocabulary constraint information is provided as external information when translating with a teacher model). The score used is the BLEU score.

[0101] As shown in FIG. 11, even in this English-Japanese translation, the BLEU score for the method of the third embodiment (C) (knowledge distillation with vocabulary constraint function) was higher than the BLEU scores for the conventional techniques of (A) and (B) regardless of whether the number of layers in the student model was 12-12, 3-3, or 1-1. When the method of the third embodiment (C) was used, the BLEU score for the student model with 3-3 layers was 45.3. In other words, even though the number of layers in the student model was 3-3 in this case, this was higher than the BLEU score of 44.7 for the student model with 12-12 layers in the method of (A). Furthermore, when the method of the third embodiment (C) was used, the BLEU score for the student model with 1-1 layers was 40.0. In other words, when the number of layers in the student model is 1-1, the accuracy of method (A) is not as high as the BLEU score of 44.7 when the number of layers in the student model is 12-12, but it is significantly higher than the accuracy of the conventional technology method when the number of layers in the student model is 1-1.

[0102] As described above, in both the Japanese-English and English-Japanese translation cases, the BLEU scores obtained when using the method of the third embodiment are significantly higher than the scores obtained when using the conventional technology used for comparison. This indicates that the third embodiment is highly effective.

[0103] Figure 12 is a table comparing translation speeds when the number of layers (number of internal parameters) of the translation model is different. The values ​​shown for each case represent the number of sentences that could be translated per second using a given set of computational resources. When the translation model had 12 layers (encoder) x 12 layers (decoder), the number of sentences per second was 43.8, whereas when the number of layers was 3 x 3, the number of sentences per second was nearly three times as high at 126.8. Furthermore, when the number of layers was 1 x 1, the number of sentences per second was 200.1, more than four times as high. As such, translation speed can be increased by reducing the number of layers in the student model.

[0104] In other words, by using the method of the third embodiment (knowledge distillation with vocabulary constraint function), it is possible to reduce the amount of calculation required for translation processing while ensuring translation accuracy. In other words, it becomes possible to perform accurate translation using a smaller computer.

[0105] [Modification of the third embodiment] As a modification of the third embodiment, information other than vocabulary constraint information may be used as external information. Examples of external information (privileged information) that can be used in addition to vocabulary constraint information include dictionary information and grammar information. That is, the external information input unit 322 (see FIG. 9) can pass this external information to the translation processing unit 42 instead of vocabulary constraint information. The translation processing unit 42 uses the external information passed from the external information input unit 322 to perform translation processing constrained by the external information. This improves the translation accuracy of the translation processing unit 42. That is, the quality of the pseudo-bilingual sentence pairs used by the learning processing unit 331 when training the student model (translation processing unit 342) is improved. This improves the translation accuracy of the translation processing unit 342, which performs translation using the student model.

[0106] Dictionary information, which is an example of external information, is information that represents the correspondence between expressions (words, phrases, etc.) that appear in a source language sentence and their corresponding expressions in the target language. The external information input unit 322 does not necessarily need to provide dictionary information for all expressions that appear in a source language sentence. For example, the quality of translation by the translation processing unit 42 can be improved even if the external information input unit 322 simply provides dictionary information for expressions that appear infrequently or proper nouns. As an example, the external information input unit 322 passes information on the English expression "the Dead Sea" to the translation processing unit 42 as dictionary information corresponding to the Japanese expression "Dead Sea." In other words, dictionary information is information that represents the correspondence between expressions in a source language sentence and expressions in a target language sentence.

[0107] Grammatical information, another example of external information, is information that represents the dependency structure of a sentence (structures such as grammatical structure and dependency relationships). By providing the dependency structure of a sentence in a source language sentence as external information, it is possible to prevent the dependency structure from being erroneously translated, for example, when the dependency structure cannot be uniquely identified from the surface of the source language sentence alone. Furthermore, by providing information that specifies the dependency structure of a sentence in a target language sentence as external information, it is possible to impose constraints on the word order in the target language sentence. In other words, grammatical information is information about the grammatical structure of at least one of the source language sentence and the target language sentence (information about constraints on the grammatical structure). By the external information input unit 322 providing such grammatical information, the quality of the translation by the translation processing unit 42 is improved.

[0108] Vocabulary constraint information, dictionary information, and grammar information have been described above as examples of external information that the external information input unit 322 passes to the translation processing unit 42. The external information input unit 322 may pass other external information to the translation processing unit 42. In either case, the external information that the external information input unit 322 passes to the translation processing unit 42 is information that can serve as a constraint on the result of the translation processing by the translation processing unit 42. The external information input unit 322 may also pass multiple types of external information to the translation processing unit 42. As an example, the external information input unit 322 may pass both vocabulary constraint information and grammar information to the translation processing unit 42 as external information. In such a case, the translation processing unit 42 outputs a translation result that is constrained by both of these multiple types of external information.

[0109] In other words, the external information that the external information input unit 322 passes to the translation processing unit 42 may include at least one of the following: vocabulary constraint information, which is information regarding vocabulary constraints in the target language sentence that the translation processing unit 42 outputs as a translation result; dictionary information, which is information indicating the correspondence between expressions in the source language sentence and expressions in the target language sentence; and grammatical information, which is information regarding the grammatical structure of at least one of the source language sentence and the target language sentence (information that can constrain the grammatical structure).

[0110] As another modification of the third embodiment, a model other than the Transformer (neural network) may be used as the translation model.

[0111] The natural language processing device of the third embodiment (including the above-mentioned modified examples) can also be realized using a computer and a program. An example of the internal configuration when using a computer has already been explained with reference to FIG. 6. Furthermore, the method of realizing it using a program is also as explained in the first and second embodiments.

[0112] Although several embodiments of the present invention have been described in detail above with reference to the drawings, the specific configurations are not limited to these embodiments, and designs within the scope of the present invention are also included. [Industrial Applicability]

[0113] The present invention can be used for generating training data and translating natural languages. The present invention can be used for translation processing in any industry. However, the scope of use of the present invention is not limited to the examples given here. [Explanation of symbols]

[0114] 1,2 Natural language processing system 7 Learning Device 8 Translation Device 21 Parallel Text Pair Storage Unit 22 Restricted Vocabulary Selection Section 23 Learning data generation unit 24 Learning data storage unit 31 Learning processing unit 41 Source language sentence input section 42 Translation processing section (teacher model) 43 Target language sentence output section 101 Encoder 102 decoder 221 Translation Model 301 Natural Language Processing 302 Translation Device 303 Learning Device 304 Translation Device 321 Source Language Sentence Input Unit 322 External information input section 324 Learning data storage unit 331 Learning processing unit 341 Source Language Sentence Input Unit 342 Translation Processing Unit (Student Model, Student Translation Processing Unit) 343 Target Language Sentence Output Unit 901 Central Processing Unit 902 RAM 903 Input / Output Ports 904,905 Input / Output Devices 906 Bus

Claims

1. a restricted vocabulary selection unit that reads a bilingual sentence pair having a source language sentence and a target language sentence, and selects words included in the target language sentence as restricted vocabulary based on at least one of a measure of importance in the target language sentence and a measure of likelihood of error in machine translation processing; a training data generation unit that generates pairs of input data including the source language sentence and information on the selected restricted vocabulary, and output data including the target language sentence, as training data for a vocabulary-limited translation model; A natural language processing device comprising:

2. the restricted vocabulary selection unit inputs the source language sentence included in the bilingual sentence pair into a predetermined translation model to obtain a translation result sentence output by the translation model, and selects words that are included in the target language sentence included in the bilingual sentence pair but are not included in the translation result sentence as restricted vocabulary based on the measure of error-proneness; The natural language processing device according to claim 1 .

3. the restricted vocabulary selection unit calculates a TF-IDF value in a set of bilingual sentence pairs, which is a set of bilingual sentence pairs, for words included in the target language sentence, and selects restricted vocabulary using the calculated TF-IDF as the importance. The natural language processing device according to claim 1 .

4. a learning processing unit that performs machine learning of a limited vocabulary translation model using the learning data generated by the natural language processing device according to claim 1; A learning device comprising:

5. a translation processing unit including a limited-vocabulary translation model machine-learned by the learning device according to claim 4; a source language sentence input unit that passes a source language sentence to be translated to the translation processing unit; a target language sentence output unit that outputs a target language sentence that is a translation result output by the translation processing unit based on the source language sentence; A translation device comprising:

6. a translation processing unit that is equipped with a trained translation model, performs translation processing on a source language sentence, and outputs a target language sentence as a translation result; a source language sentence input unit that passes a source language sentence to be translated to the translation processing unit; an external information input unit that passes external information that may be a constraint when translating the source language sentence to the translation processing unit; a target language sentence output unit that outputs a target language sentence that is a translation result output by the translation processing unit based on the source language sentence and the external information; Equipped with the translation model provided in the translation processing unit has been trained in advance using training data so as to perform translation processing under the constraints expressed by the external information, and outputs the target language sentence corresponding to the source language sentence under the constraints. Translation device.

7. the external information includes at least one of vocabulary constraint information, which is information regarding vocabulary constraints in the target language sentence output by the translation processing unit as a translation result; dictionary information, which is information indicating correspondences between expressions in the source language sentence and expressions in the target language sentence; and grammatical information, which is information regarding the grammatical structure of at least one of the source language sentence and the target language sentence. The translation device according to claim 6.

8. a learning processing unit that performs machine learning of a translation model using a pair of a source language sentence to be translated by the translation device according to claim 6 and a target language sentence output by the translation device corresponding to the source language sentence as a bilingual sentence pair; A learning device comprising:

9. a student translation processing unit including a student translation model machine-learned by the learning device according to claim 8; a source language sentence input unit that passes a source language sentence to be translated to the student translation processing unit; a target language sentence output unit that outputs a target language sentence that is a translation result output by the student translation processing unit based on the source language sentence; A translation device comprising:

10. Computer, The natural language processing apparatus according to any one of claims 1 to 3, A program to function as a

11. Computer, The learning device according to claim 4 or 8, A program to function as a

12. Computer, A translation device according to any one of claims 5 to 7 and claim 9. A program to function as a

Images