Sentence generation model generation device, sentence generation model, and sentence generation device

The encoder-decoder model with parallel processing capability in the sentence generation model generation device addresses the processing time issue in bilingual translation by inputting constraint data to the encoder, facilitating efficient generation of sentences with specific expressions in the second language.

JP2024168531AInactive Publication Date: 2024-12-05NTT DOCOMO INC
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023085285
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-24
Publication Date
2024-12-05
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing machine learning models for bilingual translation increase processing time due to sequential input of constraint data in the decoder, which affects the efficiency of generating sentences in a second language based on an input in a first language.

Method used

A sentence generation model generation device that uses an encoder-decoder model with parallel processing capability, where constraint data is input to the encoder along with input data, allowing the model to learn the relationship between constraint and output sentences while maintaining the arrangement order, thereby reducing processing time and enabling the generation of sentences with specific expressions.

Benefits of technology

The solution enables the generation of sentences in a second language using specific expressions while suppressing the increase in processing time, allowing for efficient and accurate bilingual translation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024168531000001_ABST
    Figure 2024168531000001_ABST
Patent Text Reader

Abstract

To obtain an output sentence in a second language using specific expressions according to an input sentence in a first language while suppressing an increase in processing time.SOLUTION: A sentence generation model generation device 10 being a device that generates a sentence generation model for generating an output sentence in a second language according to the input of an input sentence in a first language through machine learning comprises: an encoder input unit 12 which inputs vectors obtained by inputting words included in the first data and constraint data to an embedded dictionary that includes association between words in the first language and the second language and the vectors to an encoder; a decoder input unit 13 which inputs the second data which constitutes the output sentence to a decoder; a learning unit 14 which executes learning of the sentence generation model on the basis of the word-by-word errors between the sequence of words output by the decoder and the sequence of words included in the second data; and a model output unit 15 which outputs the sentence generation model learned by the learning unit.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a sentence generation model generating device, a sentence generation model, and a sentence generating device. [Background technology]

[0002] There is known a technology that generates a model by machine learning that generates an output sentence consisting of, for example, a translation in a second language according to an input sentence in a first language, and configures a translation engine, a scoring engine, etc. by using the generated model. For example, Patent Document 1 discloses a technology that generates a document corresponding to an input document by using a machine learning model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-135457 A Summary of the Invention [Problem to be solved by the invention]

[0004] In a machine learning model that outputs a translation according to an input sentence, a technique is known in which information representing a constraint is input to obtain a translation using a specific expression. The machine learning model that outputs the translation is, for example, configured with an encoder-decoder model. In this case, the information representing the constraint is input to the decoder. However, since the decoder sequentially performs processing according to the input of the constraint and words, etc. at each processing stage, the input of the information representing the constraint to the decoder extends the processing time.

[0005] Therefore, the present invention has been made in consideration of the above problems, and aims to obtain an output sentence in a second language using a specific expression in response to an input sentence in a first language, while suppressing an increase in processing time. [Means for solving the problem]

[0006] In order to solve the above problem, a sentence generation model generation device according to one aspect of the present disclosure is a sentence generation model generation device that generates a sentence generation model by machine learning, which generates an output sentence in a second language different from the first language, in response to an input of an input sentence in a first language, wherein the sentence generation model is an encoder-decoder model including a neural network and configured with an encoder and a decoder, learning data used for the machine learning of the sentence generation model includes first data, constraint data, and second data, the first data includes an arrangement of a plurality of words that constitute the input sentence, the second data includes an arrangement of a plurality of words that constitute an output sentence corresponding to the input sentence, the constraint data includes one or more constraint words that are words identified from the arrangement of words that constitute the output sentence, and the encoder generates a sentence generation model by using machine learning to generate a sentence generation model that generates an output sentence in a second language different from the first language, The sentence generation model generation device has a configuration capable of parallel processing of word sequences obtained by inputting words included in each of the first data and the constraint data into an embedded dictionary and inputting vectors obtained by the input into the encoder, the embedded dictionary including associations between words in the first language and vectors and associations between words in the second language and vectors, and converting the input words into corresponding vectors; a decoder input unit inputs at least the words constituting the second data into the decoder in accordance with their arrangement order; a learning unit that performs learning of a sentence generation model based on a word-by-word error between the word sequence output from the decoder and the word sequence included in the second data; and a model output unit that outputs the sentence generation model learned by the learning unit.

[0007] According to the above aspect, the sentence generation model is composed of an encoder-decoder model including an encoder and a decoder. In learning of the sentence generation model in which first data corresponding to an input sentence is input to the encoder and second data corresponding to an output sentence is input to the decoder, constraint data including one or more constraint words specified from the sequence of words that constitute the output sentence is input to the encoder together with the first data. Since the encoder has a configuration capable of parallel processing of the input word sequence, the processing time is prevented from being extended by inputting the constraint data. In addition, the constraint data is composed of a second language different from the first language that constitutes the first data, and vectors corresponding to each of the words included in the first data and the constraint data can be input to the encoder by referring to an embedded dictionary corresponding to both the first language and the second language. Therefore, by inputting the constraint data in which the words that constitute a desired specific expression are specified as constraint words to the encoder, the sentence generation model learns the association between the constraint data and the second data, and therefore it is possible to obtain a sentence generation model that outputs an output sentence using a specific expression consisting of the constraint words included in the constraint data. Effect of the Invention

[0008] It is possible to obtain an output sentence in a second language using a specific expression in response to an input sentence in a first language while suppressing an increase in processing time. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a functional configuration of a sentence generation model generating device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram illustrating a configuration of a sentence generation model. [Diagram 3] Fig. 3(a) is a diagram showing an example of a conventional embedded dictionary in a second language, and Fig. 3(b) is a diagram showing an example of an embedded dictionary in a second language according to the present embodiment. [Figure 4] FIG. 2 is a diagram showing an example of an embedded dictionary in a first language, which is an example of a Japanese embedded dictionary. [Diagram 5]FIG. 2 is a diagram showing an example of generation of first data, second data, and constraint data based on a corpus. [Figure 6] FIG. 1 is a diagram for explaining a schematic configuration of a Transformer, which is an example of an encoder-decoder model. [Figure 7] 1 is a block diagram showing a functional configuration of a sentence generation device according to an embodiment of the present invention; [Figure 8] FIG. 2 is a diagram illustrating a sentence generation process using a sentence generation model. [Figure 9] 11A and 11B are diagrams illustrating examples of input constraint data and output sentences that can be output based on the constraint data. [Figure 10] FIG. 13 is a diagram showing a process of evaluating a created sentence in an evaluation system constituted by the sentence generation device. [Figure 11] 4 is a flowchart showing the processing steps of a sentence generation model generating method in the sentence generation model generating device. [Figure 12] 4 is a flowchart showing the processing content of a sentence generation method in the sentence generation device. [Figure 13] FIG. 2 is a diagram showing a configuration of a sentence generation model generation program. [Figure 14] FIG. 2 is a diagram showing the structure of a sentence generation program. [Figure 15] FIG. 2 is a hardware block diagram of the sentence generation model generating device and the sentence generating device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The following describes an embodiment of a sentence generation model generating device, a sentence generation device, and a sentence generation model according to the present invention with reference to the drawings. When possible, the same parts are given the same reference numerals, and duplicated explanations will be omitted.

[0011] The sentence generation model of the present embodiment is a model constructed by machine learning for causing a computer to function and generating an output sentence in a second language different from the first language in response to an input sentence in a first language. The sentence generation model includes a neural network and is configured by an encoder-decoder model including an encoder and a decoder.

[0012] The sentence generation model generating device of this embodiment is a device that generates a sentence generation model by machine learning. The sentence generation device is a device that generates an output sentence in a second language in response to an input sentence in a first language, using the sentence generation model constructed by machine learning.

[0013] An example of a problem that can be solved by the sentence generation model generating device, sentence generation model, and sentence generation device of the present embodiment will be described.

[0014] In a system that uses AI (artificial intelligence) to correct English compositions based on given Japanese text, for example, words with low scores among the words included in the input English text are extracted, and the words following the extracted low-scoring words are corrected based on the rewritten correction text. For example, the Japanese text, "He studies English to become able to speak with foreigners," is translated into the English text, "He studies English to become able to speak with foreigners," by a normal translation engine.

[0015] If the English sentence "He study English so that can talk with American," written by a user, is input into a correction system consisting of such a translation engine, the second word entered, "study," is clearly an error and will receive a low score. Therefore, the correction system will replace the second word "study" with the higher-scoring word "studies," and further rewrite the third and subsequent words into words with higher scores, thereby outputting the corrected sentence, "He studies English to become able to speakwith foreigners."

[0016] However, since the word strings "so that" and "talk with" in the English sentence input by the user are correct expressions as English translations, it is preferable to correct the English sentence "He studies English so that he can talkwith foreigners," which is an example of an English translation using these expressions, as the corrected sentence. In this embodiment, it is possible to generate output sentences using desired expressions, and also to appropriately evaluate created sentences based on output sentences using desired expressions.

[0017] Fig. 1 is a diagram showing a functional configuration of a sentence generation model generating device according to this embodiment. The sentence generation model generating device 10 is a device that generates a sentence generation model that generates an output sentence in a second language different from the first language in response to an input sentence in a first language, by machine learning. As shown in Fig. 1, the sentence generation model generating device 10 functionally includes a constraint data generating unit 11, an encoder input unit 12, a decoder input unit 13, a learning unit 14, and a model output unit 15. Each of these functional units 11 to 15 may be configured in one device, or may be configured distributed across multiple devices.

[0018] Furthermore, the sentence generation model generation device 10 is configured to be able to access storage means such as a model storage unit 30 and a corpus storage unit 40. The model storage unit 30 and the corpus storage unit 40 may be configured within the sentence generation model generation device 10, or may be configured as separate devices outside the sentence generation model generation device 10 and accessible from the sentence generation model generation device, as shown in Fig. 1.

[0019] The model storage unit 30 is a storage means for storing a sentence generation model that has already been trained or is in the process of being trained, and can be configured with storage, memory, and the like.

[0020] The corpus storage unit 40 is a storage means that stores learning data used in machine learning of the sentence generation model and a corpus for generating learning data, and can be composed of storage, memory, etc.

[0021] Fig. 2 is a diagram showing the configuration of the sentence generation model of this embodiment. As shown in Fig. 2, the sentence generation model MD is an encoder-decoder model including a neural network and configured by an encoder en and a decoder de. The neural network configuring the encoder-decoder model is not limited, but may be, for example, a neural network called a Transformer.

[0022] The learning data used for the machine learning of the sentence generation model MD of this embodiment includes first data a, second data b, and constraint data c. The first data a includes an arrangement of a plurality of words constituting an input sentence in a first language. The second data b includes an arrangement of a plurality of words constituting an output sentence in a second language corresponding to the input sentence. The output sentence is, for example, a translation of the input sentence. The constraint data c is data including one or more constraint terms that are words in the second language identified from the arrangement of words constituting the second data b. The order of the constraint terms in the constraint data c may maintain the order of the words in the arrangement of the words in the second data b.

[0023] The sentence generation model MD further includes an embedded dictionary ED. The embedded dictionary ED includes at least associations between words and vectors, and converts (embedding) an input word into a vector of a predetermined length corresponding to the word. The embedded dictionary ED of this embodiment includes a first language embedded dictionary ed1 and a second language embedded dictionary ed2. The first language embedded dictionary ed1 includes associations between at least words and vectors in a first language. The first language may be, for example, Japanese. The second language embedded dictionary ed2 includes associations between at least words and vectors in a second language different from the first language. The second language may be, for example, English.

[0024] The encoder en receives first data a constituting an input sentence in a first language. Specifically, the first data a is divided into words, etc., by, for example, morphological analysis, etc. Each divided word, etc., is converted (embedded) into a corresponding vector by referring to the first language embedded dictionary ed1 of the embedded dictionary ED, and input to the encoder en according to the arrangement order in the first data a (input sentence).

[0025] Furthermore, the encoder en receives input of constraint data c. The constraint data c is data including constraint words, which are words identified from the sequence of words constituting the second data b, while maintaining the sequence order of the words in the second data c. Although the generation of the constraint data c will be described in detail later, the constraint data c may be data consisting of a sequence of constraint words and replacement symbols in which words or word strings other than the constraint words in the sequence of words constituting the output sentence are replaced with predetermined replacement symbols.

[0026] The constraint data c is divided into words, etc., for example, by morphological analysis, etc. Each divided word, etc. is converted (embedded) into a corresponding vector by referring to the second language embedded dictionary ed2 of the embedded dictionary ED, and input to the encoder en according to the arrangement order in the constraint data c.

[0027] The encoder en of this embodiment has a configuration capable of parallel processing of vectors of input words, etc. For example, the sentence generation model MD may be configured by a transformer to realize the encoder en capable of parallel processing of vectors of input words, etc. The encoder en processes the array of vectors of each word, etc. of the input first data a and constraint data c in parallel.

[0028] The encoder en outputs a vector indicating a calculation result based on the input of the first data a and the constraint data c (eg, the output of an intermediate layer, source-target attention, etc.) to the decoder de.

[0029] In a general encoder-decoder model, the decoder sequentially outputs a sequence of words based on the input of a vector from the encoder and a predetermined start symbol (vector) indicating the start of the output. The decoder de outputs a sequence of words (vectors) of the output sentence t based on the input of the output from the encoder en and the start symbol ss.

[0030] When the decoder de outputs a terminal symbol es, which indicates the end of the output sentence, the output sentence t is composed of the sequence of words output up to that point. In the learning phase, the second data b, which corresponds to the output sentence (the translation of the input sentence in the second language) corresponding to the first data a (input sentence), is input to the decoder de in the order of the words after the input of the start symbol ss.

[0031] FIG. 3 is a diagram for explaining an embedded dictionary of a second language, in which FIG. 3(a) is a diagram showing an example of a conventional embedded dictionary, and FIG. 3(b) is a diagram showing an example of the second language embedded dictionary ed2 of this embodiment.

[0032] As shown in Fig. 3(a), a conventional embedded dictionary has associations between words (words) and vectors (IDs) corresponding to the number of vocabulary words. In contrast, the second language embedded dictionary ed2 shown in Fig. 3(b) includes associations between subwords and vectors in addition to associations between words and vectors.

[0033] Sub-words are obtained by dividing a word into smaller units. The second language embedded dictionary ed2 may include sub-words that indicate conjugations of verbs, as shown in ID1(ing) and ID2(ed). Note that the conjugations of verbs are an example of sub-words, and sub-words may be composed of other character strings.

[0034] In the conventional embedded dictionary, for example, in the case of a verb vocabulary, all conjugated forms of each verb must be included. However, in the embedded dictionary of the present embodiment, the number of vocabulary words can be reduced by including the conjugated forms of the verb, such as "ing" and "ed", and the original form of the verb as subwords.

[0035] 4 is a diagram showing an example of the first language embedded dictionary ed1. As shown in FIG. 4, the first language embedded dictionary ed1 has associations between words or subwords in Japanese, which is an example of the first language, and vectors (IDs).

[0036] In this way, since the embedded dictionary ED for converting sentences input to the encoder en into vectors includes associations between words and subwords in a first language and vectors, and associations between words and subwords in a second language and vectors, it becomes possible to input, in addition to the first data a, constraint data c composed of a second language different from the first language constituting the first data a into the encoder en.

[0037] Furthermore, when an embedded dictionary for two languages ​​is constructed using a conventional embedded dictionary consisting of associations between words and vectors, the size of the embedded dictionary becomes large (for example, twice as large). However, in this embodiment, the embedded dictionary ED is constructed by associations between subwords and vectors, which reduces the number of vocabulary words, thereby suppressing the increase in size of the embedded dictionary due to the inclusion of embedded dictionaries for two languages.

[0038] In the following description, the term "word" means a word or a subword.

[0039] The functional units of the sentence generation model generating device 10 will be described with reference to Fig. 1 again. The constraint data generating unit 11 generates constraint data based on a corpus. The generation of constraint data and learning data including constraint data will be described with reference to Fig. 5.

[0040] FIG. 5 is a diagram showing an example of generation of first data, second data, and constraint data based on a corpus. The constraint data generation unit 11 acquires a corpus cp0 from, for example, the corpus storage unit 40. The corpus cp0 is composed of a first sentence cp01 written in a first language and a second sentence cp02 written in a second language. In the example of FIG. 5, the first sentence cp01 is a Japanese sentence, "He studies English so that he can talk with foreigners," and the second sentence cp02 is an English translation of the first sentence cp01, "He studies English so that he can talk with foreigners."

[0041] The constraint data generating unit 11 identifies the constraint words cx from the sequence of words constituting the second sentence cp02. The identification of the constraint words may be based on, for example, a designated input by a user or the like. For example, words constituting an expression that should be used as a mandatory part in the second sentence cp02, which is a translation of the first sentence cp01, may be designated as the constraint words by the designated input. Also, the identification of the constraint words may be performed randomly. In the example shown in FIG. 5, three words or word strings, "He", "so that", and "talk with", are identified as the constraint words cx.

[0042] Based on the identified constraint term cx, the constraint data generating unit 11 generates constraint data c01 including the constraint term cx while maintaining the order of the words in the second sentence cp02. As shown in Fig. 5, the constraint data c01 is data including "He", "sothat", and "talk with" while maintaining the order of the words in the second sentence cp02.

[0043] The constraint data may be data consisting of an array of constraint words and replacement symbols, in which words or word strings other than the constraint words in the array of words constituting the output sentence are replaced with a predetermined replacement symbol. The constraint data generating unit 11 replaces words or word strings other than the words specified as the constraint words cx in the word array constituting the second sentence cp02 with replacement symbols rs to generate constraint data c01 consisting of the array of constraint words cx and replacement symbols rs. In the example shown in FIG. 5, the replacement symbol rs is indicated by "* (asterisk)", and the constraint data generating unit 11 generates constraint data c01 "He * so that * talk with *" in which the three constraint words cx "He", "so that", and "talk with" and the replacement symbols rs are arranged in the same order as in the second sentence cp02.

[0044] In addition, the constraint data generation unit 11 generates first data a01 and second data b01 in the learning data based on the first sentence cp01 and the second sentence cp02, respectively, and generates learning data consisting of the first data a01, the constraint data c01, the start symbol ss, and the second data b01.

[0045] The constraint data generating unit 11 may include information indicating a relationship with the second data b01 in the constraint data. In the example shown in Fig. 5, the constraint data generating unit 11 includes a symbol cl01 indicating that the constraint data c01 is data including a constraint word cx to be used in the second data b01 (output sentence), in the constraint data c01.

[0046] As described with reference to FIG. 5, since the constraint data can be easily generated based on a corpus, an increase in the cost of obtaining learning data including the constraint data is prevented.

[0047] 1 again, the encoder input unit 12 inputs the first data a and the constraint data c to the encoder en. Specifically, the encoder input unit 12 inputs vectors obtained by inputting words included in the first data a and the constraint data c to the embedding dictionary ED, and inputs the vectors to the encoder en according to the arrangement order of the words in the first data a and the constraint data c.

[0048] 2, the encoder input unit 12 obtains vectors that can be input to the encoder en by inputting words included in the first data a to the first language embedded dictionary ed1. The encoder input unit 12 also obtains vectors that can be input to the encoder en by inputting words included in the constraint data c to the second language embedded dictionary ed2.

[0049] As described with reference to Figures 3 and 4, the embedded dictionary ED of this embodiment is configured to include an association between sub-words and vectors, so the encoder input unit 12 inputs the words and sub-words contained in the first data a and the constraint data c, respectively, into the embedded dictionary ED, and inputs the vectors obtained by inputting them into the encoder en.

[0050] The decoder input unit 13 inputs at least the words constituting the second data b to the decoder de in accordance with the arrangement order. Furthermore, the decoder input unit 13 may input a start symbol ss, which is a predetermined symbol indicating the start of output of an output sentence, to the decoder de prior to inputting the second data b.

[0051] The learning unit 14 learns the sentence generation model MD based on the error for each word between the sequence of words output from the decoder de and the sequence of words included in the second data b. Specifically, the learning unit 14 updates the weighting coefficients constituting the encoder en and the decoder de based on the error for each word between the sequence of words output from the decoder de at a stage subsequent to the input of the start symbol ss and the sequence of words included in the second data b. When the start symbol ss and the second data b are input to the decoder de in sequence according to the word order, the decoder de sequentially outputs a sequence of word vectors of the output sentence t together with the likelihood (for example, by a softmax function).

[0052] The learning unit 14 calculates the error between the sequence of words output from the decoder de and the sequence of words in the second data b for each word, and updates the weighting coefficients that make up the neural network of the encoder en and the decoder de, for example, by the error backpropagation method.

[0053] Fig. 6 is a diagram for explaining a schematic configuration of a transformer, which is an example of an encoder-decoder model. As shown in Fig. 6, when the sentence generation model MD1 (MD) is configured by a transformer, the encoder input unit 12 inputs vectors aw11, aw12,...aw1l (l is an integer of 2 or more) of words constituting the first data a1 and vectors cw11,...,cw1m (m is an integer of 2 or more) of words constituting the constraint data c1 to the input layer ila of the encoder en1 according to the arrangement order of the words. In the transformer, words are input in parallel, not sequentially as in an RNN (Recurrent Neural Network), so that input data can be processed in parallel.

[0054] In the encoder en1, self-attention sa1 for the hidden layer mla is calculated from the input layer ila, and the word vectors are converted to vectors according to the self-attention sa1. Similarly, self-attention sa2 for the output layer ola is calculated from the hidden layer mla, and the word vectors are further converted. Furthermore, source-target attention ta for the input layer ilb of the decoder de1 is calculated from the output layer ola of the encoder en1.

[0055] The decoder input unit 13 inputs the start symbol ss and vectors bw11, bw12, ..., bw1n (n is an integer equal to or greater than 2) of words constituting the second data b1 to the input layer ilb of the decoder de1 in parallel during the learning phase according to the order of the words.

[0056] In the decoder de1, similar to the encoder en1, self-attention sa3 for the hidden layer mlb from the input layer ilb is calculated, and vectors are transformed according to the self-attention sa3. Similarly, self-attention sa4 for the output layer olb is calculated from the hidden layer mlb, and vector transformation according to the self-attention sa4 is performed.

[0057] The learning unit 14 calculates the error for each word between a sequence of words t11,...,t1n (n is an integer greater than or equal to 2) based on the vector wv output at a later stage after the input of the start symbol ss and a sequence of words bw11,...,bw1n constituting the second data b1, and updates the weighting coefficients for calculating self-attention and source-target attention using the error backpropagation method.

[0058] As mentioned above, the word vectors aw11,aw12,...aw1l, the word vectors cw11,...,cw1m, and the word vectors bw11,bw12,...,bw1n are vectors of words or subwords. Also, the word sequence t11,...,t1n is a sequence of words or subwords.

[0059] 1, the model output unit 15 outputs the sentence generation model MD obtained after machine learning based on a required amount of learning data. The model output unit 15 may store the sentence generation model MD in the model storage unit 30.

[0060] Next, a process of sentence generation using the sentence generation device and the trained sentence generation model will be described with reference to FIGS.

[0061] FIG. 7 is a diagram showing a functional configuration of a sentence generation device according to this embodiment. The sentence generation device 20 is a device that generates an output sentence in a second language different from a first language in response to an input of an input sentence in a first language, using a sentence generation model constructed by machine learning. As shown in FIG. 7, the sentence generation device 20 functionally includes an input unit 21, a constraint data input unit 22, a word input unit 23, and an output unit 24. The sentence generation device 20 may further include a created sentence acquisition unit 25, a created sentence input unit 26, and a created sentence evaluation unit 27. Each of these functional units 21 to 27 may be configured in one device, or may be configured distributed across multiple devices.

[0062] The sentence generation device 20 is also configured to be able to access a model storage unit 30 that stores a trained sentence generation model. The model storage unit 30 may be configured within the sentence generation device 20, or may be configured in a separate external device.

[0063] In addition, in this embodiment, an example is shown in which the sentence generation model generating device 10 and the sentence generating device 20 are configured as separate devices (computers), but they may also be configured as an integrated device.

[0064] Fig. 8 is a diagram showing a schematic diagram of a sentence generation process by a sentence generation model. As shown in Fig. 8, the sentence generation model MD2 (MD) is a model learned and constructed by the sentence generation model generating device 10. The sentence generation model MD2 includes an encoder en2 and a decoder de2.

[0065] The sentence generation model MD (MD1, MD2), which is a model including a trained neural network, can be considered as a program that is read or referenced by a computer and causes the computer to execute specified processes and realize specified functions.

[0066] That is, the trained sentence generation model MD (MD1, MD2) of this embodiment is used in a computer having a processor and a memory. Specifically, the processor of the computer operates in accordance with instructions from the trained sentence generation model MD (MD1, MD2) stored in the memory to perform calculations on input data input to the input layer of the neural network based on trained weight coefficients (parameters) and functions corresponding to each layer, and to output results (likelihoods) from the output layer.

[0067] The input unit 21 inputs input data constituting an input sentence to an encoder in accordance with the arrangement order of the words. Specifically, the input unit 21 inputs words aw21, aw22, . . . , aw2h (h is an integer equal to or greater than 2) constituting the input data a2 to the encoder en2 in accordance with the arrangement order.

[0068] The constraint data input unit 22 inputs the input constraint data c2 to the encoder en2. Specifically, the words cw21 to cw24, ... cw2j (j is an integer equal to or greater than 2) constituting the input constraint data c2 are input to the encoder en2 according to their arrangement order. The constraint data input unit 22 may further input a symbol ct2 indicating a relationship with the output sentence. The input constraint data c2 is data including an input constraint word arbitrarily specified as a word to be used in the output sentence. The input constraint word is included in the input constraint data c2 while maintaining the arrangement order of the words in the output sentence. The input constraint word may be specified based on a designated input by a user or the like, for example.

[0069] The input constraint data c2 may be data consisting of an array of input constraint words and replacement symbols, in which words or word strings other than the input constraint words in the array of words constituting the output sentence are replaced with a predetermined replacement symbol rs. In the example shown in Fig. 8, the input constraint data c2 consists of data in which input constraint words, which are words or word strings such as "He" and "sothat", and the replacement symbol rs "* (asterisk)", are arranged in the same order as in the output sentence.

[0070] The input constraint data c2 may also include a symbol ct2, which indicates, for example, that the input constraint data c2 is data including an input constraint word to be used in the output sentence t2.

[0071] The input unit 21 and the constraint data input unit 22 input the word vectors constituting the input data a2 and the input constraint data c2 in parallel to the encoder en2. The encoder en2 outputs the calculation results of the parallel processing based on the word input to the decoder de2.

[0072] At each stage after the output from the encoder en2 is input to the decoder de2, the word input unit 23 sequentially inputs the words output from the decoder de2 in the previous stage to the decoder de2. For example, the word input unit 23 inputs a start symbol ss to the decoder de2 prior to inputting a word. The decoder de2 outputs a word tw21 at the beginning of the output sentence t2 in response to the start symbol ss. At each stage after the input of the start symbol ss, the word input unit 23 sequentially inputs the words output from the decoder de2 in the previous stage to the decoder de2. The decoder de2 sequentially outputs a sequence of words tw21, tw22, ... tw2k (k is an integer equal to or greater than 2) constituting the output sentence t2 in response to the words input sequentially.

[0073] When a terminal symbol es indicating the end of output of an output sentence is output, the output unit 24 arranges the words tw21, tw22, ... tw2k sequentially output at each stage of the decoder de2 to generate an output sentence t2. Then, the output unit 24 outputs the generated output sentence t2. The manner in which the output sentence t2 is output is not limited, and may be, for example, storage in a predetermined storage means, display on a display means, output by voice, or the like.

[0074] As described above, the words aw21, aw22, . . . , aw2h, the words cw21 to cw24, . . . cw2j, and the words tw21, tw22, . . . tw2k are words or subwords.

[0075] FIG. 9 is a diagram showing an example of input constraint data and an output sentence that can be output based on the input constraint data. In the example shown in FIG. 9, it is assumed that an input sentence "He studies English to become able to speak with foreigners (kare ha gaikokujin tohanaseruyoninarutameni eigo wo benkyo suru)" is input to the encoder en2 as input data a2. When this input sentence is translated by a normal translation engine without any constraint, for example, an output sentence "He studies English to become able to speakwith foreigners." is output. In the sentence generation device 20 of this embodiment, different output sentences can be output depending on the input constraint data input to the encoder en2.

[0076] As shown in FIG. 9, when the input constraint data "He * so that *talk with *" is input, for example, an output sentence such as "He studies English so that he can talk with foreigners" is output. When the input constraint data "He * so that *chat *" is input, for example, an output sentence such as "He studies English so that he can chat foreigners" is output. When the input constraint data "He * in order to *speak *" is input, for example, an output sentence such as "He studies English in order to become able to speak with foreigners" is output.

[0077] Moreover, when the input constraint data "He * to be able to *" is input, for example, the output sentence "He studies English to be able to speak with foreigners." is output. When the input constraint data "He * for being *" is input, for example, the output sentence "He studies English for being able to speak with foreigners." is output. When the input constraint data "His study of English * talk with *" is input, for example, the output sentence "His study of English is to be able to talk with foreigners." is output.

[0078] An evaluation system for evaluating a created sentence, which is configured by the sentence generation device 20, will be described with reference to Fig. 10 in addition to Fig. 7. Fig. 10 is a diagram showing a process of evaluating a created sentence in the evaluation system configured by the sentence generation device 20.

[0079] The decoder de3 shown in FIG. 10 outputs a likelihood indicating the likelihood of each of the words tw31, tw32, . . . , tw3i (i is an integer equal to or greater than 2) output at each stage after the input of the start symbol ss as a word constituting the output sentence t3 for each word. The evaluation system constituted by the sentence generation device 20 evaluates the created sentence created and input by the user, regarding the output sentence t3 as the correct answer. In response to the presentation of the input sentence corresponding to the output sentence t3, the user inputs, for example, a translation of the input sentence in a second language as the created sentence. Note that, although this embodiment assumes that the created sentence input by the user is evaluated, it is also possible to evaluate the created sentence created and input by a person, device, etc. other than the user.

[0080] The created sentence acquisition unit 25 acquires a created sentence r3 that is created in a second language by a user and input to the evaluation system. The created sentence r3 is made up of an array of words rw31, rw32, . . . , rw3p (p is an integer of 2 or more).

[0081] In each stage after the input of the start symbol ss, the created sentence input unit 26 sequentially inputs to the decoder de3 the word vectors of the words rw31, rw32, ..., rw3p that constitute the created sentence r3 created in the second language, instead of the words tw31, tw32, ..., tw3i output from the decoder de3 in the previous stage.

[0082] The created sentence evaluation unit 27 evaluates the created sentence r3 based on the input of the start symbol ss and the sequential input of each word rw31, rw32, ..., rw3p that constitutes the created sentence r3, and based on a comparison between the likelihood of each word rw31, rw32, ..., rw3p output from the decoder de3 at each stage after the input of the start symbol ss and the likelihood of each word tw31, tw32, ..., tw3i that constitutes the output sentence t3.

[0083] Specifically, at each output stage, the decoder de3 outputs the likelihood of each word in the entire vocabulary handled by the sentence generation model generation device 10 and the sentence generation device 20. In the phase of the sentence generation process, an output sentence t3 is constructed by arranging the words having the highest likelihood at each output stage.

[0084] At each stage of the decoder de3, the created sentence evaluation unit 27 obtains the likelihood associated with each word rw31, rw32, ..., rw3p and terminal symbol es of the created sentence r3 from the likelihood of each vocabulary output in response to the input of the start symbol ss and the words (rw31, rw32, ..., rw3p) output in the previous stage.

[0085] The created sentence evaluation unit 27 calculates and outputs an evaluation value of the created sentence r3 by comparing the likelihood of each word tw31, tw32,..., tw3i constituting the output sentence t3 with the likelihood of each word rw31, rw32,..., rw3p of the created sentence r3. The method of calculating the evaluation value is not limited, and may be based on, for example, the ratio of the likelihood of each word in each sentence t3, r3, or the sum or average of the likelihood of each sentence t3, r3.

[0086] As described above, the words tw31, tw32, . . . , tw3i and the words rw31, rw32, . . . , rw3p are words or subwords.

[0087] 10, the created sentence is evaluated based on a comparison between the likelihood of each word constituting the output sentence and the likelihood of each word obtained by sequentially inputting the words constituting the created and inputted created sentence into a decoder. This makes it possible to configure an evaluation system that evaluates the likelihood of the created sentence as a translation corresponding to the input sentence.

[0088] FIG. 11 is a flowchart showing the processing steps of the sentence generation model generating method in the sentence generation model generating device 10.

[0089] In step S1, the sentence generation model generating device 10 acquires learning data including first data a, second data b, and constraint data c. The constraint data in the learning data may be data generated in advance based on a corpus and stored in the corpus storage unit 40, or may be data generated by the constraint data generating unit 11 based on a corpus.

[0090] In step S2, the encoder input unit 12 inputs the first data a and the constraint data c to the encoder en.

[0091] In step S3, the decoder input unit 13 inputs the start symbol ss to the decoder de. Furthermore, in step S4, the decoder input unit 13 inputs the second data b to the decoder de in accordance with the arrangement order.

[0092] In step S5, the learning unit 14 calculates the error for each word between the sequence of words output from the decoder de at the stage after the input of the start symbol ss and the sequence of words included in the second data b, and updates the sentence generation model MD.

[0093] In step S6, the learning unit 14 determines whether or not machine learning based on a required amount of learning data has been completed. If it is determined that the learning has been completed, the process proceeds to step S7. On the other hand, if it is not determined that the learning has been completed, the processes of steps S1 to S6 are repeated.

[0094] In step S7, the model output unit 15 outputs the trained sentence generation model MD.

[0095] FIG. 12 is a flowchart showing the processing steps of a sentence generation method using a trained sentence generation model MD in the sentence generation device 20.

[0096] In step S11, the input unit 21 inputs the words of the input data that constitute the input sentence to the encoder of the sentence generation model MD. At the same time, the constraint data input unit 22 inputs the input constraint data to the encoder.

[0097] In step S12, the word input unit 23 inputs the start symbol ss to the decoder. In step S13, the output unit 24 obtains a word (or symbol) output from the output layer of the decoder. In step S14, the output unit 24 determines whether the output from the decoder is a terminal symbol indicating the end of the output of the output sentence. If it is determined that the output from the decoder is a terminal symbol, the process proceeds to step S16. On the other hand, if it is not determined that the output from the decoder is a terminal symbol, the process proceeds to step S15.

[0098] In step S15, the word input unit 23 inputs the word output from the previous output layer of the decoder to the current input layer of the decoder, and then the process returns to step S13.

[0099] In step S16, the output unit 24 arranges the words sequentially output from the output layer in each stage of the decoder to generate an output sentence, and in step S17, the output unit 24 outputs the output sentence.

[0100] Next, with reference to FIG. 13, a sentence generation model generating program for causing a computer to function as the sentence generation model generating device 10 of this embodiment will be described.

[0101] 13 is a diagram showing the configuration of a sentence generation model generation program. The sentence generation model generation program P1 includes a main module m10 that controls the sentence generation model generation process in the sentence generation model generation device 10, a constraint data generation module m11, an encoder input module m12, a decoder input module m13, a learning module m14, and a model output module m15. The modules m11 to m15 realize the functions of the functional units 11 to 15.

[0102] The sentence generation model generating program P1 may be transmitted via a transmission medium such as a communication line, or may be stored in a recording medium M1 as shown in FIG.

[0103] Next, with reference to FIG. 14, a sentence generation program for causing a computer to function as the sentence generation device 20 of this embodiment will be described.

[0104] 14 is a diagram showing the configuration of a sentence generation program. The sentence generation program P2 is configured to include a main module m20 that controls the sentence generation process in the sentence generation device 20, an input module m21, a constraint data input module m22, a word input module m23, and an output module m24. The sentence generation program P2 may further include a created sentence acquisition module m25, a created sentence input module m26, and a created sentence evaluation module m27. The modules m21 to m27 realize the functions of the functional units 21 to 27.

[0105] The sentence generation program P2 may be transmitted via a transmission medium such as a communication line, or may be stored in a recording medium M2 as shown in FIG.

[0106] According to the sentence generation model generating device 10, the sentence generation model generating method, and the sentence generation model generating program P1 of the present embodiment described above, the sentence generation model is configured by an encoder-decoder model including an encoder and a decoder. In the learning of the sentence generation model in which the first data corresponding to the input sentence is input to the encoder and the second data corresponding to the output sentence is input to the decoder, the constraint data including one or more constraint words specified from the sequence of words that constitute the output sentence is input to the encoder together with the first data. Since the encoder has a configuration that can process the sequence of input words in parallel, the processing time is suppressed from being extended by inputting the constraint data. In addition, the constraint data is configured in a second language different from the first language that constitutes the first data, and by referring to an embedded dictionary corresponding to both the first language and the second language, vectors corresponding to each of the words included in the first data and the constraint data can be input to the encoder. Therefore, by inputting the constraint data in which the words that constitute a desired specific expression are specified as constraint words to the encoder, the sentence generation model learns the association between the constraint data and the second data, and therefore it is possible to obtain a sentence generation model that outputs an output sentence using a specific expression consisting of the constraint words included in the constraint data.

[0107] The sentence generation model generating device, the sentence generation model, and the sentence generation device according to the present disclosure may have the following configurations. Furthermore, the actions and effects of each configuration will be described as follows.

[0108] A sentence generation model generation device according to one aspect of the present disclosure is a sentence generation model generation device that generates, by machine learning, a sentence generation model that generates an output sentence in a second language different from the first language in response to an input of an input sentence in a first language, the sentence generation model is an encoder-decoder model including a neural network and configured with an encoder and a decoder, learning data used for the machine learning of the sentence generation model includes first data, constraint data, and second data, the first data includes an arrangement of a plurality of words that constitute the input sentence, the second data includes an arrangement of a plurality of words that constitute an output sentence corresponding to the input sentence, the constraint data includes one or more constraint terms that are words identified from the arrangement of words that constitute the output sentence, and the encoder generates a sentence generation model based on the arrangement of the input words. the sentence generation model generation device has a configuration capable of parallel processing of the above, and includes an encoder input unit that inputs vectors obtained by inputting words included in each of the first data and the constraint data into an embedded dictionary to an encoder, the embedded dictionary including associations between words in a first language and vectors and associations between words in a second language and vectors, and converts the input words into corresponding vectors; a decoder input unit that inputs at least the words constituting the second data into a decoder according to their arrangement order; a learning unit that performs learning of a sentence generation model based on a word-by-word error between the word arrangement output from the decoder and the word arrangement included in the second data; and a model output unit that outputs the sentence generation model learned by the learning unit.

[0109] According to the above aspect, the sentence generation model is composed of an encoder-decoder model including an encoder and a decoder. In learning of the sentence generation model in which first data corresponding to an input sentence is input to the encoder and second data corresponding to an output sentence is input to the decoder, constraint data including one or more constraint words specified from the sequence of words that constitute the output sentence is input to the encoder together with the first data. Since the encoder has a configuration capable of parallel processing of the input word sequence, the processing time is prevented from being extended by inputting the constraint data. In addition, the constraint data is composed of a second language different from the first language that constitutes the first data, and vectors corresponding to each of the words included in the first data and the constraint data can be input to the encoder by referring to an embedded dictionary corresponding to both the first language and the second language. Therefore, by inputting the constraint data in which the words that constitute a desired specific expression are specified as constraint words to the encoder, the sentence generation model learns the association between the constraint data and the second data, and therefore it is possible to obtain a sentence generation model that outputs an output sentence using a specific expression consisting of the constraint words included in the constraint data.

[0110] In addition, in a sentence generation model generating device relating to another aspect, the embedded dictionary may include an association between subwords, which are formed by dividing words into smaller units, and vectors, and the encoder input unit may input the vectors obtained by inputting each word and subword contained in the first data and the constraint data, respectively, into the embedded dictionary, and input them to the encoder.

[0111] According to the above aspect, the embedded dictionary is configured to include associations between subwords and vectors, so that the number of words in the embedded dictionary can be reduced.

[0112] In addition, in the sentence generation model generating device according to another aspect, the sentence generation model may be configured by a Transformer.

[0113] According to the above aspect, the encoder can be configured to be capable of processing an input word sequence in parallel.

[0114] In addition, in the sentence generation model generating device according to another aspect, the order of the constraint words in the constraint data may be set to maintain the order of the words in the output sentence.

[0115] According to the above aspect, since the constraint data is composed of constraint words that maintain the order of words in the output sentence, the words and their order corresponding to the constraint words in the second data are learned as the words and their order to be used in the output sentence. Therefore, it is possible to generate a sentence generation model that can output an output sentence using a specific expression composed of constraint words.

[0116] In addition, in a sentence generation model generating device relating to another aspect, the constraint data may consist of an array of constraint words and replacement symbols in which words or word strings other than the constraint words in the array of words constituting the output sentence are replaced with predetermined replacement symbols.

[0117] According to the above aspect, since the constraint data is composed of an arrangement of the constraint words and the replacement symbols substituted for the words or word strings other than the constraint words, the words corresponding to the constraint words in the second data are learned as words to be used in the output sentence, and the words corresponding to the replacement symbols in the second data are learned as arbitrary expressions in the output sentence. Therefore, it is possible to generate a sentence generation model capable of outputting an output sentence using a specific expression composed of the constraint words.

[0118] In addition, a sentence generation model generating device according to another aspect may further include a constraint data generating unit that generates constraint data based on a corpus consisting of a first sentence written in a first language and a second sentence that is a translation of the first sentence written in a second language, the constraint data including constraint words identified from the arrangement of words constituting the second sentence, while maintaining the order of the words in the second sentence.

[0119] According to the above aspect, it is possible to obtain, as learning data, constraint data for specifying words corresponding to desired specific expressions to be used in output sentences, based on a corpus.

[0120] In addition, in a sentence generation model generating device according to another aspect, the constraint data generating unit may replace words or word strings other than the words identified as the constraint words in the word sequence constituting the second sentence with predetermined replacement symbols, thereby generating constraint data consisting of an array of the constraint words and replacement symbols.

[0121] According to the above aspect, it is possible to specify words corresponding to desired specific expressions to be used in an output sentence based on a corpus, and obtain constraint data for specifying any expression in the output sentence as learning data.

[0122] A sentence generation model according to one aspect of the present disclosure is a trained sentence generation model using machine learning to cause a computer to function and generate an output sentence in a second language different from the first language in response to an input of an input sentence in a first language, wherein the learning data used for the machine learning of the sentence generation model includes first data including an arrangement of a plurality of words constituting the input sentence, second data including an arrangement of a plurality of words constituting an output sentence corresponding to the input sentence, and constraint data, and the constraint data includes one or more constraint words which are words identified from the arrangement of words constituting the output sentence, the sentence generation model is an encoder-decoder model including a neural network and including an encoder and a decoder having a configuration capable of processing the input word arrangement in parallel, wherein vectors obtained by inputting words included in each of the first data and the constraint data into an embedded dictionary are input to the encoder, at least words constituting the second data are input to the decoder in accordance with the order of the arrangement, and the configuration of the sentence generation model is updated based on the error for each word between the arrangement of words output from the decoder and the arrangement of words included in the second data, and the embedded dictionary includes an association between words and vectors in the first language and an association between words and vectors in the second language, and converts the input word into a corresponding vector.

[0123] According to the above aspect, the sentence generation model is composed of an encoder-decoder model including an encoder and a decoder. In learning of this sentence generation model, first data corresponding to an input sentence is input to the encoder, second data corresponding to an output sentence is input to the decoder, and constraint data including one or more constraint words specified from the sequence of words that constitute the output sentence is input to the encoder together with the first data. Since the encoder has a configuration capable of processing the sequence of input words in parallel, the extension of processing time due to the constraint data being input to the encoder is suppressed. In addition, the constraint data is composed of a second language different from the first language that constitutes the first data, and vectors corresponding to each of the words included in the first data and the constraint data are input to the encoder by referring to an embedded dictionary corresponding to both the first language and the second language. Therefore, by inputting constraint data that specifies words that constitute a desired specific expression as constraint words to the encoder, the sentence generation model learns the association between the constraint data and the second data, and the sentence generation model can output an output sentence using a specific expression consisting of the constraint words included in the constraint data.

[0124] A sentence generation device according to one aspect of the present disclosure is a sentence generation device that uses a sentence generation model constructed by machine learning to generate an output sentence in a second language different from the first language in response to an input of an input sentence in a first language, and the learning data used for the machine learning of the sentence generation model includes first data including an arrangement of a plurality of words constituting the input sentence, second data including an arrangement of a plurality of words constituting an output sentence corresponding to the input sentence, and constraint data, and the constraint data includes one or more constraint words that are words identified from the arrangement of words constituting the output sentence, and the sentence generation model is an encoder-decoder model including a neural network composed of an encoder and a decoder having a configuration capable of processing the input word arrangement in parallel, and a vector obtained by inputting words included in each of the first data and the constraint data into an embedding dictionary is input to the encoder, and at least words constituting the second data are input to the decoder in accordance with the arrangement order, and the configuration of the sentence generation model is determined based on a word-by-word error between the arrangement of words output from the decoder and the arrangement of words included in the second data. the embedded dictionary includes an association between words in a first language and vectors and an association between words in a second language and vectors, and converts an input word into a corresponding vector; and the sentence generation device includes: an input unit that inputs input data constituting an input sentence to an encoder in accordance with the order of the words, the input unit inputting the vector obtained by inputting each word included in the input data to the embedded dictionary to the encoder; a constraint data input unit that inputs input constraint data to the encoder, the input constraint data including input constraint words arbitrarily specified as words to be used in an output sentence while maintaining the order of the output sentence, the constraint data input unit inputting the vector obtained by inputting each word included in the input constraint data to the embedded dictionary to the encoder; a word input unit that, at each stage after inputting the output from the encoder to the decoder, sequentially inputs the words output from the decoder in the previous stage to the decoder; and an output unit that generates an output sentence by arranging the words sequentially output at each stage of the decoder, and outputs the generated output sentence.

[0125] According to the above aspect, the sentence generation model is composed of an encoder-decoder model including an encoder and a decoder. In learning the sentence generation model, first data corresponding to an input sentence is input to the encoder, second data corresponding to an output sentence is input to the decoder, and constraint data including one or more constraint words identified from the sequence of words constituting the output sentence is input to the encoder together with the first data. This allows the trained sentence generation model to learn the association between the constraint data and the second data. In addition, since the encoder has a configuration capable of processing the sequence of input words in parallel, the input constraint data being input to the encoder is prevented from increasing the processing time. In addition, the input constraint data is composed of a second language different from the first language constituting the input data, and vectors corresponding to each of the words included in the input data and the input constraint data are input to the encoder by referring to an embedded dictionary corresponding to both the first language and the second language. Therefore, an output sentence using a desired specific expression is output by inputting the input data constituting the input sentence to the encoder and input constraint data for specifying a constraint condition in the output sentence to the encoder.

[0126] In one aspect of the sentence generation device of the present disclosure, the embedded dictionary includes an association between subwords, which are formed by dividing words into smaller units, and vectors, and the input unit inputs the vectors obtained by inputting each word and subword included in the input data into the embedded dictionary to an encoder, and the constraint data input unit inputs the vectors obtained by inputting each word and subword included in the input constraint data into the embedded dictionary to the encoder.

[0127] According to the above aspect, the embedded dictionary is configured to include associations between subwords and vectors, so that the number of words in the embedded dictionary can be reduced.

[0128] Although the present embodiment has been described in detail above, it is clear to those skilled in the art that the present embodiment is not limited to the embodiment described in this specification. The present embodiment can be implemented as a modified or altered form without departing from the spirit and scope of the present invention defined by the description of the claims. Therefore, the description in this specification is intended to be illustrative and does not have any limiting meaning on the present embodiment.

[0129] The block diagrams shown in Figs. 1 and 7 show functional blocks. These functional blocks (components) are realized by any combination of at least one of hardware and software. The method of realizing each functional block is not particularly limited. That is, each functional block may be realized by using one device that is physically or logically combined, or may be realized by using two or more devices that are physically or logically separated and directly or indirectly connected (for example, by wire, wirelessly, etc.). The functional block may be realized by combining the one device or the multiple devices with software.

[0130] Functions include, but are not limited to, judgment, determination, judgment, calculation, computation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, consideration, broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocation, mapping, assignment, etc. For example, a functional block (component) that performs the function of transmission is called a transmitting unit or a transmitter. In either case, as described above, there is no particular limitation on the method of realization.

[0131] For example, the sentence generation model generating device 10 and the sentence generation device 20 in one embodiment of the present invention may function as a computer. Fig. 15 is a diagram showing an example of a hardware configuration of the sentence generation model generating device 10 and the sentence generation device 20 according to this embodiment. Each of the sentence generation model generating device 10 and the sentence generation device 20 may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, etc.

[0132] In the following description, the term "apparatus" may be replaced with a circuit, a device, a unit, etc. The hardware configuration of the sentence generation model generation device 10 and the sentence generation device 20 may be configured to include one or more of the devices shown in the figures, or may be configured to exclude some of the devices.

[0133] Each function of the sentence generation model generating device 10 and the sentence generation device 20 is realized by loading specific software (programs) onto hardware such as a processor 1001 and a memory 1002, causing the processor 1001 to perform calculations and control communications via a communication device 1004 and the reading and / or writing of data in the memory 1002 and storage 1003.

[0134] The processor 1001 controls the entire computer by running, for example, an operating system. The processor 1001 may be configured as a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic unit, a register, etc. For example, the functional units 11 to 15, 21 to 27, etc. shown in Figs. 1 and 7 may be realized by the processor 1001.

[0135] Moreover, the processor 1001 reads out a program (program code), software modules, and data from the storage 1003 and / or the communication device 1004 to the memory 1002, and executes various processes according to the programs. As the programs, programs that cause a computer to execute at least a part of the operations described in the above-mentioned embodiments are used. For example, the functional units 11-15 and 21-27 of the sentence generation model generation device 10 and the sentence generation device 20 may be stored in the memory 1002 and realized by a control program that runs on the processor 1001. Although the above-mentioned various processes have been described as being executed by one processor 1001, they may be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented in one or more chips. The programs may be transmitted from a network via an electric communication line.

[0136] The memory 1002 is a computer-readable recording medium, and may be composed of at least one of, for example, a Read Only Memory (ROM), an Erasable Programmable ROM (EPROM), an Electrically Erasable Programmable ROM (EEPROM), a Random Access Memory (RAM), etc. The memory 1002 may also be called a register, a cache, a main memory (primary storage device), etc. The memory 1002 can store executable programs (program codes), software modules, etc. for implementing the sentence generation model generation method and the sentence generation method according to one embodiment of the present invention.

[0137] Storage 1003 is a computer-readable recording medium, and may be composed of at least one of, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray® disk), a smart card, a flash memory (e.g., a card, a stick, a key drive), a floppy disk, a magnetic strip, and the like. Storage 1003 may be referred to as an auxiliary storage device. The above-mentioned storage medium may be, for example, a database, a server, or other suitable medium including memory 1002 and / or storage 1003.

[0138] The communication device 1004 is hardware (transmission / reception device) for communicating between computers via a wired and / or wireless network, and is also called, for example, a network device, a network controller, a network card, or a communication module.

[0139] The input device 1005 is an input device (e.g., a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that accepts input from the outside. The output device 1006 is an output device (e.g., a display, a speaker, an LED lamp, etc.) that performs output to the outside. The input device 1005 and the output device 1006 may be integrated into one device (e.g., a touch panel).

[0140] Furthermore, each device such as the processor 1001 and the memory 1002 is connected by a bus 1007 for communicating information. The bus 1007 may be configured as a single bus, or may be configured as different buses between the devices.

[0141] Furthermore, the sentence generation model generating device 10 and the sentence generating device 20 may be configured to include hardware such as a microprocessor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA), and some or all of the functional blocks may be realized by the hardware. For example, the processor 1001 may be implemented by at least one of these pieces of hardware.

[0142] The notification of information is not limited to the aspects / embodiments described in the present disclosure, and may be performed using other methods. For example, the notification of information may be performed by physical layer signaling (e.g., Downlink Control Information (DCI), Uplink Control Information (UCI)), higher layer signaling (e.g., Radio Resource Control (RRC) signaling, Medium Access Control (MAC) signaling, broadcast information (Master Information Block (MIB), System Information Block (SIB))), other signals, or a combination thereof. In addition, the RRC signaling may be called an RRC message, and may be, for example, an RRC Connection Setup message, an RRC Connection Reconfiguration message, or the like.

[0143] Each aspect / embodiment described in the present disclosure may be applied to at least one of systems using LTE (Long Term Evolution), LTE-A (LTE-Advanced), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), FRA (Future Radio Access), NR (new Radio), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark)), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20, UWB (Ultra-WideBand), Bluetooth (registered trademark), and other suitable systems, and next-generation systems extended based on these. In addition, a combination of multiple systems (e.g., a combination of at least one of LTE and LTE-A and 5G, etc.) may be applied.

[0144] The order of the steps, sequences, flow charts, etc. of each aspect / embodiment described in this disclosure may be changed unless inconsistent. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0145] In the present disclosure, a specific operation performed by a base station may be performed by its upper node in some cases. In a network consisting of one or more network nodes having a base station, it is clear that various operations performed for communication with a terminal may be performed by at least one of the base station and other network nodes other than the base station (e.g., MME or S-GW, etc., but are not limited to these). Although the above example illustrates a case in which there is one other network node other than the base station, it may also be a combination of multiple other network nodes (e.g., MME and S-GW).

[0146] Information, etc. may be output from a higher layer (or a lower layer) to a lower layer (or a higher layer). Information may be input / output via multiple network nodes.

[0147] The input and output information may be stored in a specific location (e.g., memory) or may be managed in a management table. The input and output information may be overwritten, updated, or added to. The output information may be deleted. The input information may be transmitted to another device.

[0148] The determination may be based on a value represented by a single bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., comparison with a predetermined value).

[0149] Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched according to execution. In addition, notification of specific information (e.g., notification that "X is the case") is not limited to being done explicitly, but may be done implicitly (e.g., not notifying the specific information).

[0150] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described in the present disclosure. The present disclosure can be implemented in modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is intended to be illustrative and does not have any limiting meaning on the present disclosure.

[0151] Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0152] Additionally, software, instructions, etc. may be transmitted or received over a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using wired technologies, such as coaxial cable, fiber optic cable, twisted pair, and Digital Subscriber Line (DSL), and / or wireless technologies, such as infrared, radio, and microwave, these wired and / or wireless technologies are included within the definition of transmission media.

[0153] The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, the data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0154] In addition, terms explained in this disclosure and / or terms necessary for understanding this specification may be replaced with terms having the same or similar meanings.

[0155] As used in this disclosure, the terms "system" and "network" are used interchangeably.

[0156] In addition, the information, parameters, etc. described in the present disclosure may be represented by absolute values, may be represented by relative values ​​from a predetermined value, or may be represented by other corresponding information. For example, a radio resource may be indicated by an index.

[0157] The names used for the above-mentioned parameters are not limiting in any way. Moreover, the formulas using these parameters may differ from those explicitly disclosed in this disclosure. The various channels (e.g., PUCCH, PDCCH, etc.) and information elements may be identified by any suitable names, and therefore the various names assigned to these various channels and information elements are not limiting in any way.

[0158] As used in this disclosure, the terms "determining" and "determining" may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, search, inquiry (e.g., searching in a table, database, or other data structure), ascertaining, and the like. "Determining" and "determining" may also include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in a memory), and the like. In addition, "judgment" and "decision" can include considering resolving, selecting, choosing, establishing, comparing, etc., to be a "judgment" or "decision." In other words, "judgment" and "decision" can include considering some action to be a "judgment" or "decision." In addition, "judgment" can be interpreted as "assuming," "expecting," "considering," etc.

[0159] As used in this disclosure, the phrase "based on" does not mean "based only on," unless expressly stated otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0160] When designations such as "first," "second," and the like are used in this disclosure, any reference to an element does not generally limit the quantity or order of those elements. These designations may be used herein as a convenient way to distinguish between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed therein or that the first element must precede the second element in some way.

[0161] The "means" in the configuration of each of the above devices may be replaced with "part," "circuit," "device," etc.

[0162] To the extent that the terms "include," "including," and variations thereof are used herein in the specification or claims, these terms are intended to be inclusive, similar to the term "comprising." Further, the term "or" as used herein is not intended to be an exclusive or.

[0163] In this disclosure, where articles have been added due to translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural form.

[0164] In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "coupled" may also be interpreted in the same way as "different."

[0165] Throughout this disclosure, the plural is intended to be included unless the context clearly indicates the singular.

[0166] The sentence generation model generating device, the sentence generation model MD, and the sentence generation device of the present disclosure may have the following configuration. [1] A sentence generation model generation device that generates, in response to an input sentence in a first language, a sentence generation model that generates an output sentence in a second language different from the first language by machine learning, comprising: the sentence generation model is an encoder-decoder model including a neural network and configured with an encoder and a decoder, learning data used in the machine learning of the sentence generation model includes first data, constraint data, and second data; the first data includes a sequence of a plurality of words that constitute the input sentence; the second data includes an arrangement of a plurality of words constituting the output sentence corresponding to the input sentence, the constraint data includes one or more constraint words that are words identified from a sequence of words that constitute the output sentence; The encoder has a configuration capable of processing an input word sequence in parallel, The sentence generation model generating device comprises: an encoder input unit that inputs a vector obtained by inputting words included in the first data and the constraint data into an embedding dictionary, the embedding dictionary including associations between words in the first language and vectors and associations between words in the second language and vectors, and converts the input words into corresponding vectors; a decoder input unit that inputs at least the words constituting the second data to the decoder in accordance with an arrangement order; a learning unit that learns the sentence generation model based on an error for each word between the sequence of words output from the decoder and the sequence of words included in the second data; a model output unit that outputs a sentence generation model trained by the training unit; A sentence generation model generating device comprising: [2] The embedding dictionary includes associations between subwords, which are obtained by dividing words into smaller units, and vectors; the encoder input unit inputs vectors obtained by inputting each word and each subword included in the first data and each constraint data into the embedding dictionary, to the encoder; A sentence generation model generating device according to [1]. [3] The sentence generation model is composed of a Transformer. A sentence generation model generating device according to [1] or [2]. [4] the constraint data maintains the order of the constraint terms in the constraint data in the order of the words in the output sentence; The sentence generation model generating device according to any one of [1] to [3]. [5] the constraint data is composed of an array of the constraint words and the replacement symbols in which words or word strings other than the constraint words in the array of words constituting the output sentence are replaced with predetermined replacement symbols; A sentence generation model generating device according to [4]. [6] a constraint data generating unit that generates the constraint data based on a corpus including a first sentence written in the first language and a second sentence written in a second language, the constraint data including the constraint words identified from an arrangement of words constituting the second sentence while maintaining an arrangement order of words in the second sentence, The sentence generation model generating device according to any one of [1] to [5]. [7] the constraint data generation unit replaces a word or a word string other than the word specified as the constraint word, in a word sequence constituting the second sentence, with a predetermined replacement symbol, to generate the constraint data consisting of the constraint word and the replacement symbol sequence. A sentence generation model generating device according to [6]. [8] A machine learning trained sentence generation model for causing a computer to function and generate an output sentence in a second language different from the first language in response to an input sentence in the first language, the machine learning trained sentence generation model comprising: the learning data used in the machine learning of the sentence generation model includes first data including an arrangement of a plurality of words constituting the input sentence, second data including an arrangement of a plurality of words constituting the output sentence corresponding to the input sentence, and constraint data; the constraint data includes one or more constraint words that are words identified from a sequence of words that constitute the output sentence; The sentence generation model is An encoder-decoder model including a neural network composed of an encoder and a decoder having a configuration capable of processing an input word sequence in parallel, A vector obtained by inputting words included in the first data and the constraint data into an embedding dictionary is input to the encoder; At least the words constituting the second data are input to the decoder in accordance with an arrangement order; The sentence generation model is constructed by machine learning in which a configuration of the sentence generation model is updated based on a word-by-word error between the word sequence output from the decoder and the word sequence included in the second data; the embedding dictionary includes associations between words and vectors in the first language and associations between words and vectors in the second language, and converts an input word into a corresponding vector; A trained sentence generation model. [9] A sentence generation device that generates an output sentence in a second language different from a first language in response to an input sentence in a first language by using a sentence generation model constructed by machine learning, the learning data used in the machine learning of the sentence generation model includes first data including an arrangement of a plurality of words constituting the input sentence, second data including an arrangement of a plurality of words constituting the output sentence corresponding to the input sentence, and constraint data; the constraint data includes one or more constraint words that are words identified from a sequence of words that constitute the output sentence; The sentence generation model is An encoder-decoder model including a neural network composed of an encoder and a decoder having a configuration capable of processing an input word sequence in parallel, A vector obtained by inputting words included in the first data and the constraint data into an embedding dictionary is input to the encoder; At least the words constituting the second data are input to the decoder in accordance with an arrangement order; The sentence generation model is constructed by machine learning in which a configuration of the sentence generation model is updated based on a word-by-word error between the word sequence output from the decoder and the word sequence included in the second data; the embedding dictionary includes associations between words and vectors in the first language and associations between words and vectors in the second language, and converts input words into corresponding vectors; The sentence generation device comprises: an input unit that inputs input data constituting the input sentence to the encoder in accordance with an arrangement order of words, the input unit inputting a vector obtained by inputting each word included in the input data to the embedding dictionary to the encoder; a constraint data input unit that inputs input constraint data, which includes input constraint words arbitrarily specified as words to be used in the output sentence while maintaining the order of the input constraint words in the output sentence, to the encoder, and inputs a vector obtained by inputting each word included in the input constraint data to the embedded dictionary to the encoder; a word input unit that, in each stage after the output from the encoder is input to the decoder, sequentially inputs the words output from the decoder in the previous stage to the decoder; an output unit that generates the output sentence by arranging the words sequentially output at each stage of the decoder, and outputs the generated output sentence; A sentence generation device comprising:

[10] The embedding dictionary includes associations between subwords, which are obtained by dividing words into smaller units, and vectors; the input unit inputs vectors obtained by inputting each word and the subword included in the input data into the embedding dictionary, to the encoder; the constraint data input unit inputs, to the encoder, vectors obtained by inputting each word and the subword included in the input constraint data into the embedded dictionary; [9] A sentence generation device according to the present invention. [Explanation of symbols]

[0167] 10... sentence generation model generating device, 11... constraint data generating unit, 12... encoder input unit, 13... decoder input unit, 14... learning unit, 15... model output unit, 20... sentence generating device, 21... input unit, 22... constraint data input unit, 23... word input unit, 24... output unit, 25... created sentence acquisition unit, 26... created sentence input unit, 27... created sentence evaluation unit, 30... model storage unit, 40... corpus storage unit, m11... constraint data generating module, m12... encoder input module, m13 ...decoder input module, m14...learning module, m15...model output module, M1, M2...recording medium, m21...input module, m22...constraint data input module, m23...word input module, m24...output module, m25...created sentence acquisition module, m26...created sentence input module, m27...created sentence evaluation module, MD, MD1, MD2...sentence generation model, P1...sentence generation model generation program, P2...sentence generation program.

Claims

1. A sentence generation model generation device that generates, in response to an input sentence in a first language, a sentence generation model that generates an output sentence in a second language different from the first language by machine learning, comprising: the sentence generation model is an encoder-decoder model including a neural network and configured with an encoder and a decoder, learning data used in the machine learning of the sentence generation model includes first data, constraint data, and second data; the first data includes a sequence of a plurality of words that constitute the input sentence; the second data includes an arrangement of a plurality of words constituting the output sentence corresponding to the input sentence; the constraint data includes one or more constraint words that are words identified from a sequence of words that constitute the output sentence; The encoder has a configuration capable of processing an input word sequence in parallel, The sentence generation model generating device comprises: an encoder input unit that inputs a vector obtained by inputting words included in the first data and the constraint data into an embedded dictionary, the embedded dictionary including associations between words in the first language and vectors and associations between words in the second language and vectors, and converts the input words into corresponding vectors; a decoder input unit that inputs at least the words constituting the second data to the decoder in accordance with an arrangement order; a learning unit that learns the sentence generation model based on an error for each word between a sequence of words output from the decoder and a sequence of words included in the second data; a model output unit that outputs a sentence generation model trained by the training unit; A sentence generation model generating device comprising:

2. The embedding dictionary includes associations between subwords, which are obtained by dividing words into smaller units, and vectors; the encoder input unit inputs vectors obtained by inputting each word and each subword included in the first data and each constraint data into the embedding dictionary, to the encoder; The sentence generation model generating device according to claim 1 .

3. The sentence generation model is composed of a Transformer. The sentence generation model generating device according to claim 1 .

4. the constraint data maintains the order of the constraint terms in the constraint data in the order of the words in the output sentence; The sentence generation model generating device according to claim 1 .

5. the constraint data is composed of an array of the constraint words and the replacement symbols in which words or word strings other than the constraint words in the array of words constituting the output sentence are replaced with predetermined replacement symbols; The sentence generation model generating device according to claim 4.

6. a constraint data generating unit that generates the constraint data based on a corpus including a first sentence written in the first language and a second sentence written in a second language, the constraint data including the constraint words identified from an arrangement of words constituting the second sentence while maintaining an arrangement order of words in the second sentence. The sentence generation model generating device according to claim 1 .

7. the constraint data generation unit replaces a word or a word string other than the word specified as the constraint word, in a word sequence constituting the second sentence, with a predetermined replacement symbol, to generate the constraint data consisting of the constraint word and the replacement symbol sequence. The sentence generation model generating device according to claim 6.

8. A machine learning trained sentence generation model for causing a computer to function and generate an output sentence in a second language different from a first language in response to an input sentence in the first language, the machine learning trained sentence generation model comprising: the learning data used in the machine learning of the sentence generation model includes first data including an arrangement of a plurality of words constituting the input sentence, second data including an arrangement of a plurality of words constituting the output sentence corresponding to the input sentence, and constraint data; the constraint data includes one or more constraint words that are words identified from a sequence of words that constitute the output sentence; The sentence generation model is An encoder-decoder model including a neural network composed of an encoder and a decoder having a configuration capable of processing an input word sequence in parallel, A vector obtained by inputting words included in the first data and the constraint data into an embedding dictionary is input to the encoder; At least the words constituting the second data are input to the decoder in accordance with an arrangement order; The sentence generation model is constructed by machine learning in which a configuration of the sentence generation model is updated based on an error for each word between the sequence of words output from the decoder and the sequence of words included in the second data; the embedding dictionary includes associations between words and vectors in the first language and associations between words and vectors in the second language, and converts an input word into a corresponding vector; A trained sentence generation model.

9. A sentence generation device that generates an output sentence in a second language different from a first language in response to an input sentence in a first language by using a sentence generation model constructed by machine learning, the learning data used in the machine learning of the sentence generation model includes first data including an arrangement of a plurality of words constituting the input sentence, second data including an arrangement of a plurality of words constituting the output sentence corresponding to the input sentence, and constraint data; the constraint data includes one or more constraint words that are words identified from a sequence of words that constitute the output sentence; The sentence generation model is An encoder-decoder model including a neural network composed of an encoder and a decoder having a configuration capable of processing an input word sequence in parallel, A vector obtained by inputting words included in the first data and the constraint data into an embedding dictionary is input to the encoder; At least the words constituting the second data are input to the decoder in accordance with an arrangement order; The sentence generation model is constructed by machine learning in which a configuration of the sentence generation model is updated based on an error for each word between the sequence of words output from the decoder and the sequence of words included in the second data; the embedding dictionary includes associations between words and vectors in the first language and associations between words and vectors in the second language, and converts an input word into a corresponding vector; The sentence generation device comprises: an input unit that inputs input data constituting the input sentence to the encoder in accordance with an arrangement order of words, the input unit inputting a vector obtained by inputting each word included in the input data to the embedding dictionary to the encoder; a constraint data input unit that inputs input constraint data, which includes input constraint words arbitrarily specified as words to be used in the output sentence while maintaining the order of the input constraint words in the output sentence, to the encoder, and inputs a vector obtained by inputting each word included in the input constraint data to the embedded dictionary to the encoder; a word input unit that, in each stage after the output from the encoder is input to the decoder, sequentially inputs the words output from the decoder in the previous stage to the decoder; an output unit that generates the output sentence by arranging the words sequentially output at each stage of the decoder, and outputs the generated output sentence; A sentence generation device comprising:

10. The embedding dictionary includes associations between subwords, which are obtained by dividing words into smaller units, and vectors; the input unit inputs vectors obtained by inputting each word and the subword included in the input data into the embedding dictionary, to the encoder; the constraint data input unit inputs, to the encoder, vectors obtained by inputting each word and the subword included in the input constraint data into the embedded dictionary; 10. A sentence generating apparatus according to claim 9.

Citation Information

Patent Citations

  • Generation device, learning device, generation method and program

    JP2020135457A