Language model learning device, interaction device, and trained language model

Abstract

To provide a language model learning device by which a large-scale language model can be learned with low computational cost independent of speech synthesis and speech recognition performance.SOLUTION: The language model learning device includes: conversion means for converting text of a natural language to output a symbol string of phonetic symbols; and learning means for performing learning of a language model using the text and the symbol string outputted by the conversion means.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] This invention relates to a technology for a human to interact with a machine using natural language, and particularly relates to a language model learning device, an interaction device, and a learned language model for learning a language model robust to errors in speech recognition.

Background Art

[0002] Recently, language models pre-trained using large-scale texts, such as BERT (Bidirectional Encoder Representation from Transformers), have attracted attention. These language models can be fine-tuned according to individual tasks after pre-training, and are evaluated to have high versatility and effectiveness, such as updating the highest performance in various language processing tasks.

[0003] On the other hand, speech recognition is an essential technology for a human to interact with a machine using natural language. However, in speech recognition, it is difficult to consider acoustically similar features, and there are limitations in robust language processing even when using the above-described language models. For example, if "morning (asa)" is misrecognized as "umbrella (kasa)" during speech recognition, the interaction between the human and the machine will not proceed smoothly.

[0004] Proposals for solving such problems are disclosed in Non-Patent Document 1 shown below. Non-Patent Document 1 is for pre-training a language model such as BERT used in speech recognition.

[0005] Referring to Figure 1, the language model learning system 50 disclosed in Non-Patent Literature 1 converts a reference sentence 60 used for learning into speech 64 using TEXT-TO-SPEECH (speech synthesis) 62. After adding synthesized noise 66 to this speech 64, environmental noise 68 is further added to the speech 64 to obtain noisy speech 70. The language model learning system 50 then converts this noisy speech 70 back into a speech-recognized sentence 74 using SPEECH-TO-TEXT 72 (speech recognition). The speech-recognized sentence 74 contains noise from TEXT-TO-SPEECH 62, synthesized noise 66, environmental noise 68, and noise from passing through SPEECH-TO-TEXT 72.

[0006] The language model learning system 50 further converts the speech recognition sentence 74 into a phoneme sequence 78 for the word sequence of the speech recognition sentence 74 using the LAS (Listen-Attend-Spell) model 76. This phoneme sequence 78 consists of phoneme symbols. The language model learning system 50 uses the word sequence of the speech recognition sentence 74 and this phoneme sequence 78 to perform pre-training 80 of the language model 82. In Non-Patent Literature 1, BERT is used as this language model 82, and the pre-trained language model 82 is called phonemeBERT. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] Mukuntha Narayanan Sundararaman, Ayush Kumar, Jithendra Vepa, Phoneme-BERT: Joint Language Modelling of Phoneme Sequence and ASR (Automatic Speech Recognition) Transcript, in Proceedings of Interspeech 2021 [Overview of the project] [Problems that the invention aims to solve]

[0008] However, the technology disclosed in Non-Patent Document 1 requires a series of speech processing steps, including speech synthesis and speech recognition, to create data for pre-training the language model 82. Generally, speech processing requires significantly higher computational costs compared to text-only language processing. It is known that billions of sentences are required for pre-training to improve performance in large-scale language models such as BERT. Therefore, it is practically difficult to apply the technology disclosed in Non-Patent Document 1 to the training of large-scale language models such as BERT.

[0009] Furthermore, the language model obtained using the technology disclosed in Non-Patent Document 1 has the problem of being highly dependent on the speech synthesizer and speech recognition system used to create the training data. Therefore, if one wishes to change the speech synthesizer or speech recognition system after the language model has been trained, the pre-training process must be redone. In addition, the performance of this language model is greatly affected by the performance of the speech synthesis and speech recognition systems used to create the training data.

[0010] Therefore, this invention aims to provide a language model learning device, a dialogue device, and a trained language model that are independent of speech synthesis and speech recognition performance and can train large-scale language models with low computational cost. [Means for solving the problem]

[0011] The language model learning device according to the first aspect of this invention includes a conversion means for converting natural language text and outputting a sequence of phonetic symbols, and a learning means for learning a language model using the text and the sequence of symbols output by the conversion means.

[0012] Preferably, the learning means includes a learning data creation means for creating learning data for a language model by combining text and a sequence of symbols output by a conversion means, and a pre-training means for pre-training the language model using the learning data.

[0013] More preferably, the language model learning device further includes noise addition means for generating a noisy symbol sequence by adding noise to a symbol sequence; learning data creation means for creating learning data for fine-tuning a language model pre-trained by a pre-training means using text, a symbol sequence, and a noisy symbol sequence; and fine-tuning means for performing fine-tuning of the pre-trained language model using the learning data.

[0014] More preferably, the language model includes a pre-trained language model, and the learning means includes noise addition means for generating a noisy symbol sequence by adding noise to a symbol sequence, learning data creation means for creating learning data for fine-tuning the pre-trained language model using text, a symbol sequence, and a noisy symbol sequence, and fine-tuning means for performing fine-tuning of the pre-trained language model using the learning data.

[0015] Preferably, the language model includes a pre-trained language model, and the learning means includes noise addition means for generating a noisy symbol sequence by adding noise to a symbol sequence, additional learning data creation means for creating additional learning data for further pre-training of the pre-trained language model using text, a symbol sequence, and a noisy symbol sequence, and additional pre-training means for performing additional pre-training on the pre-trained language model using the learning data.

[0016] The noise addition means may include substitution means for generating a new noisy symbol sequence by substituting a portion of the symbol sequence with one or more other phonetic symbols. The substitution means may include word substitution means for generating a new noisy symbol sequence by substituting each of the one or more phonetic symbols corresponding to one or more words randomly selected in a predetermined proportion from the words in the text with one or more phonetic symbols representing the pronunciation of another word having a similar pronunciation to the said word. The substitution means may also include symbol substitution means for generating a new noisy symbol sequence by substituting each of the one or more phonetic symbols randomly selected in a predetermined proportion from the phonetic symbols constituting the symbol sequence with another phonetic symbol having a similar pronunciation to the said phonetic symbol. The conversion means may include morphological analysis means for performing morphological analysis on the text and outputting a phonetic string corresponding to the text. The language model is a Japanese language model, and the morphological analysis means may include hiragana output means for performing morphological analysis on the text and outputting a hiragana string corresponding to the text as a phonetic string.

[0017] The dialogue device according to the second aspect of this invention is a dialogue device that performs dialogue with a user based on voice, and comprises a trained language model generated by machine learning using at least natural language text and a sequence of phonetic symbols converted from the text; a semantic interpretation module equipped with the trained language model and inputting the user's voice information; and an utterance / response module that inputs the user's voice information and performs dialogue with the user under the control of the semantic interpretation module.

[0018] The trained language model according to the third aspect of this invention is generated by machine learning using at least natural language text and a sequence of phonetic symbols converted from the text.

[0019] The computer program according to the fourth aspect of this invention causes the computer to function as a conversion means for converting text for speech recognition into a sequence of phonetic symbols, and as a learning means for learning a language model using the text and the sequence of symbols converted by the conversion means.

[0020] The above and other objects, features, aspects and advantages of the present invention will become apparent from the following detailed description of the present invention, which is to be understood in conjunction with the accompanying drawings.

Brief Description of the Drawings

[0021] [Figure 1] FIG. 1 is a diagram schematically showing the configuration of a language model learning system according to the prior art. [Figure 2] FIG. 2 is a block diagram schematically showing the configuration of a language model learning apparatus according to the first embodiment of the present invention. [Figure 3] FIG. 3 is a diagram schematically showing the configuration of learning data used in the learning of a language model according to the first embodiment of the present invention. [Figure 4] FIG. 4 is a block diagram schematically showing the procedure for learning a language model in the first embodiment of the present invention. [Figure 5] FIG. 5 is a schematic diagram for explaining the content of MLM (Masked Language Modeling) when learning a language model in the first embodiment of the present invention. [Figure 6] FIG. 6 is a block diagram showing the functional configuration of the noise addition unit 124 in the first embodiment of the present invention. [Figure 7] FIG. 7 is a diagram showing a configuration example of a noise addition dictionary used for noise addition in the first embodiment of the present invention. [Figure 8] FIG. 8 is a flowchart showing the control structure of a program for realizing noise addition in the first embodiment of the present invention. [Figure 9] FIG. 9 is a diagram showing a specific example of noise addition to learning data in the first embodiment of the present invention. [Figure 10] FIG. 10 is a functional block diagram of an interactive device using the learned language model according to the first embodiment of the present invention. [Figure 11]Figure 11 shows an example of a response in a YES / NO judgment task using a trained language model according to the first embodiment of this invention. [Figure 12] Figure 12 shows an example of the speech recognition result of a user's response in a YES / NO decision task using a language model in the first embodiment of this invention. [Figure 13] Figure 13 is a diagram illustrating the configuration of the dataset used in the experiment relating to this invention. [Figure 14] Figure 14 is a table showing the settings for fine-tuning a language model trained using this invention for a YES / NO decision task. [Figure 15] Figure 15 is a table showing the experimental results of a YES / NO judgment task using the language model trained according to this invention. [Figure 16] Figure 16 is an external view of a computer system that realizes the language model learning device according to this invention. [Figure 17] Figure 17 is a block diagram showing the hardware configuration of the computer system shown in Figure 16. [Modes for carrying out the invention]

[0022] In the following descriptions and drawings, identical parts are assigned the same reference number. Therefore, detailed descriptions of them will not be repeated.

[0023] First Embodiment 1. Structure A. Overall configuration Figure 2 shows the overall configuration of the language model learning device 100 according to the first embodiment of the present invention in block diagram form. Referring to Figure 2, this language model learning device 100 is for pre-training a large-scale language model. This language model learning device 100 includes a pre-training text storage unit 110 for storing the original text of the pre-training text, and an additional pre-training text storage unit 111 for storing the original text of the additional pre-training text. Here, it is assumed that all of these learning texts are sentences consisting of sequences of Japanese words.

[0024] The language model learning device 100 further includes a morphological analysis dictionary 113 that is referenced when performing morphological analysis on text, and a morphological analysis unit 112 that performs the following processes: morphological analysis of each sentence of text stored in the pre-training text storage unit 110 while referring to the morphological analysis dictionary 113, converting them into reading sequences (sequences of phonetic symbols) consisting of hiragana, and outputting them as word sequence-reading sequence pairs; and performing the same process on text stored in the additional pre-training text storage unit 111 and outputting them as word sequence-reading sequence pairs.

[0025] The language model learning device 100 further includes a first storage unit 114 for storing word sequence / reading sequence pairs output by the morphological analysis unit 112 after processing text in the pre-training text storage unit 110, and a second storage unit 115 for storing word sequence / reading sequence pairs output by the morphological analysis unit 112 after processing text in the additional pre-training text storage unit 111.

[0026] The language model learning device 100 further includes a learning data generation unit 116 for generating learning data for pre-training a language model from word sequence / reading sequence pairs stored in the first storage unit 114, and a third storage unit 118 for storing the learning data generated by the learning data generation unit 116. The configuration of the learning data generation unit 116 will be described later.

[0027] The language model learning device 100 further includes a pre-training unit 120 for generating a pre-trained language model 122 by pre-training a large-scale language model using the training data stored in the third memory unit 118. As mentioned above, in this embodiment, BERT is used for the pre-trained language model 122.

[0028] The language model learning device 100 further includes a noise addition unit 124 for adding noise to each of the word sequence / reading sequence pairs stored in the second memory unit 115 and outputting them as noisy word sequence / hiragana sequence pairs, and a fourth memory unit 126 for storing the noisy word sequence / hiragana sequence pairs output by the noise addition unit 124 and the original text of the word sequence / hiragana sequence pairs before noise was added.

[0029] The language model learning device 100 further includes an additional pre-training learning data generation unit 128 for generating additional pre-training learning data from each of the word sequences and reading sequences stored in the fourth storage unit 126, and a fifth storage unit 130 for storing the learning data generated by the additional pre-training learning data generation unit 128.

[0030] The language model learning device 100 further includes an additional pre-training unit 132 for performing additional pre-training on the pre-trained language model 122 using the learning data stored in the fifth memory unit 130, and for generating an additional pre-trained language model 134.

[0031] Figure 3 shows a word sequence / reading sequence 140, which is an example of a word sequence / reading sequence stored in the first memory unit 114 shown in Figure 2. Referring to Figure 3, the word sequence / reading sequence 140 includes a word sequence and a reading sequence consisting of the readings of that word sequence. Each word and its reading are associated with each other.

[0032] B. Pre-learning Figure 4 shows the training procedure 150 for the pre-trained language model 122 and the additional pre-trained language model 134 shown in Figure 2. This procedure is common to both the initial pre-training and the additional pre-training. In Figure 4, the language models to be trained are represented by BERT 170 in order to commonly represent both the pre-trained language model 122 and the additional pre-trained language model 134.

[0033] Referring to Figure 4, in the learning procedure 150, the word sequence 160 from the word sequence / reading sequence 140, the concatenated string 164 obtained by concatenating the word sequence 160 and the reading sequence 162, and the concatenation of the reading sequence 162 in this order are used as the training data 166 for BERT 170. In the case of pre-training, this process is performed by the training data generation unit 116 shown in Figure 2, and in the case of additional pre-training, it is performed by the additional pre-training training data generation unit 128 shown in Figure 2. In the learning procedure 150, pre-training 168 is further performed on BERT 170 according to the normal procedure.

[0034] In this embodiment, pre-training involves both MLM and NSP (Next Sentence Prediction), which are well-known pre-training procedures for BERT. As shown in Figure 5, in this embodiment, BERT170 is trained by masking both the word sequence and the reading sequence in MLM226 and estimating the word or reading in the masked area. Alternatively, only the words or only the readings may be masked.

[0035] Specifically, in Figure 5, the training data 200 includes a word sequence and a reading sequence. During pre-training, for example, the 3rd, 6th, and 11th words in the word sequence are masked by masks 210, 212, and 214. Similarly, the readings in the reading sequence are masked by masks 220, 222, and 224. BERT170 is trained to estimate the original words 230, 232, and 234 and the original readings 240, 242, and 244 from this training data 200.

[0036] Figure 6 shows a block diagram of the noise addition unit 124. Referring to Figure 6, the noise addition unit 124 includes a noise addition dictionary 316. In this embodiment, the noise addition dictionary 316 is created using words whose frequency is above a certain value among the word positions used for pre-training. In this embodiment, the words registered in the noise addition dictionary 316 are words composed of kanji, hiragana, and katakana, and whose reading length is above a predetermined value (e.g., 2).

[0037] Figure 7 shows a portion of the noise-adding dictionary 316. Referring to Figure 7, the noise-adding dictionary 316 allows you to look up words based on their phonemes (pronunciation). That is, given a certain pronunciation (for example, "kasen"), words with that pronunciation (kasen, kasen, kasen, kasen, kasen, oligopoly, overhead wire, river) can be retrieved from the noise-adding dictionary 316.

[0038] Returning to Figure 6, the noise addition unit 124 further includes a word selection unit 314 that receives the word sequence 310 and selects words to be noise-added from it in a predetermined proportion, and a search unit 318 that, for each word selected by the word selection unit 314, extracts its reading from the reading sequence 312 and extracts all words corresponding to readings whose edit distance to that reading is 1 or 2 from the noise addition dictionary 316. In this case, depending on the reading, multiple words may be extracted from the noise addition dictionary 316 as shown in Figure 7.

[0039] The noise addition unit 124 further includes a replacement word determination unit 320 for selecting one of the words extracted by the search unit 318 when there are multiple words, and determining that the word to replace the first selected word is the replacement word, and a replacement unit 322 for replacing the first selected word and its reading with the word and its reading determined by the replacement word determination unit 320, according to the determination of the replacement word determination unit 320, and outputting it as training data 324.

[0040] Figure 8 is a flowchart showing the control structure of a program for implementing the noise addition unit 124 shown in Figure 6 using a computer. Referring to Figure 6, this program includes step 330, which performs the following learning data addition process 332 for each word sequence among all the learning data stored in the second storage unit 115 in Figure 2.

[0041] The training data addition process 332 includes step 340, which performs the following word replacement process 342 for all words in the word sequence being processed, and step 344, which adds the new data obtained in step 340 to the training data. The word replacement process 342 includes a step 350 which determines whether or not to replace the word being processed using noise and branches the control flow according to the result, and a step 352 which, if the determination in step 350 is affirmative, searches the noise addition dictionary 316 for words whose readings have an edit distance of 1 or 2 from the reading of the word being processed and retrieves them.

[0042] For example, suppose the word being processed is "public (koukai)". Then, in step 352, words with an edit distance of 1 or 2 from the reading "koukai" are searched for and extracted from the noise-adding dictionary 316. Here, for example, let's assume that "kouka" is a reading with an edit distance of 1 from "koukai", and "kougaku" and "saikai" are readings with an edit distance of 2. Then, as shown in Figure 8, 11 words with the reading "kouka" are extracted from the noise-adding dictionary 316. Similarly, 4 words with the reading "kougaku" and 6 words with the reading "saikai" are extracted from the noise-adding dictionary 316. Of course, the words shown in Figure 8 are just one example, and there may be many more readings to search for, in which case the number of extracted words will be even greater.

[0043] This program further includes step 354, which randomly selects one word from the one or more words extracted in step 352, and step 356, which uses the word selected in step 354 to replace the word being processed and the corresponding reading sequence in the word sequence being processed, thereby ending the word replacement process 342. If the determination in step 350 is negative, nothing is done to the word being processed in the word replacement process 342. That is, if the determination in step 350 is positive in the word replacement process 342, words and readings different from the original word and its reading will be added as noise to the word sequence being processed.

[0044] In Figure 8, "edit distance" is shown as a detail of the noise-adding dictionary 316, but edit distance is not included in the noise-adding dictionary 316. Edit distance is calculated according to the reading of the original word and the reading of each word in the noise-adding dictionary 316. In this embodiment, the edit distance between two strings means the minimum number of character insertion, deletion, and replacement operations required to convert one string to the other.

[0045] Figure 9 shows an example of a word sequence obtained by adding noise to a word sequence. The word sequence and reading pair 400 shown in the upper part of Figure 9 is the original word sequence. The reading and reading pair 402 shown in the lower part of Figure 9 is the word sequence with noise added.

[0046] In the example shown in Figure 9, the underlined portion of reading pair 400 is the word to be replaced and its reading. In reading pair 402, the double-lined portion is the replacement word and its reading. As can be seen from the example shown in Figure 9, the noisy data closely resembles the error-prone speech recognition results. In this embodiment, by replacing the reading corresponding to a reading with the reading of another word in this way, it is possible to create training data that contains errors similar to those of speech recognition misrecognition.

[0047] 2.Operation Referring to Figures 2 to 9, the language model learning device 100 having the above configuration operates as follows. The pre-training text storage unit 110 of the language model learning device 100 stores the original text for pre-training in advance. The additional pre-training text storage unit 111 also stores the original text for additional pre-training. Below, the operation of the language model learning device 100 during pre-training will be explained first, followed by the operation of the language model learning device 100 during additional pre-training.

[0048] A. Pre-learning During pre-training, the morphological analysis unit 112 performs the following processing on each sentence of the text stored in the additional training text storage unit 110. Specifically, the morphological analysis unit 112 performs morphological analysis on each sentence while referring to the morphological analysis dictionary 113, converts it into a reading sequence, and outputs it to the first storage unit 114 as a word sequence-reading sequence pair.

[0049] The learning data generation unit 116 divides each word sequence / hiragana pair stored in the first storage unit 114 into a word sequence 160 and a reading sequence 162, as shown in Figure 4. The learning data generation unit 116 further concatenates the word sequence 160 and the reading sequence 162 to create a concatenated string 164. The learning data generation unit 116 concatenates the word sequence 160, the concatenated string 164, and the reading sequence 162 in this order to generate learning data 166. At this time, tags indicating the beginning and end are attached to the beginning and end of the learning data 166, respectively. Tags indicating string boundaries are also inserted at the boundaries between the word sequence 160 and the concatenated string 164, and at the boundaries between the concatenated string 164 and the reading sequence 162. The learning data 166 is stored in the third storage unit 118 shown in Figure 2.

[0050] The pre-training unit 120 performs pre-training 168 of BERT using the pre-training data stored in the third memory unit 118. As a result, a pre-trained BERT 170 is obtained as the pre-trained language model 122 shown in Figure 3. Each parameter defined by the pre-trained language model 122 is stored in a predetermined memory device.

[0051] B. Additional pre-learning During additional pre-training, the language model learning device 100 operates as follows:

[0052] The morphological analysis unit 112 performs the following processing on each sentence of the text stored in the additional pre-training text storage unit 111. Specifically, the morphological analysis unit 112 performs morphological analysis on each sentence while referring to the morphological analysis dictionary 113, converts it into a reading sequence, and outputs it to the second storage unit 115 as a word sequence-reading sequence pair.

[0053] The noise addition unit 124 performs the following processing on each of the word sequence / reading sequence pairs stored in the second storage unit 115.

[0054] Referring to Figure 6, the word selection unit 314 of the noise addition unit 124 receives the word sequence 310 being processed and selects words to be noise-added from it in a predetermined proportion. For each word selected by the word selection unit 314, the search unit 318 extracts its reading from the reading sequence 312 and extracts all words corresponding to readings whose edit distance to that reading is 1 or 2 from the noise addition dictionary 316. As a result, one or more words are extracted from the noise addition dictionary 316.

[0055] The replacement word determination unit 320 of the noise addition unit 124 selects one word from the one or more words extracted by the search unit 318 for each word to be processed. In this embodiment, this selection is performed randomly. The replacement unit 322 replaces each word and its reading selected by the word selection unit 314 with the word and its reading determined by the replacement word determination unit 320, according to the determination of the replacement word determination unit 320, and outputs the original word sequence and reading sequence together as training data 324. This training data 324 is stored in the fourth storage unit 126 shown in Figure 2.

[0056] Referring to Figure 2, the additional pre-training data generation unit 128 divides each word sequence / reading sequence pair stored in the fourth storage unit 126 into a word sequence 160 and a reading sequence 162, as shown in Figure 4. The additional pre-training data generation unit 128 further concatenates the word sequence 160 and the reading sequence 162 to create a concatenated string 164. The additional pre-training data generation unit 128 concatenates these word sequences 160, concatenated string 164, and reading sequences 162 in this order to generate additional pre-training data 166. At this time, tags indicating the beginning and end are attached to the beginning and end of the training data 166, respectively, and tags indicating string boundaries are inserted at the boundaries between the word sequence 160 and the concatenated string 164, and at the boundaries between the concatenated string 164 and the reading sequence 162. The additional pre-training data 166 is stored in the fifth storage unit 130 shown in Figure 2.

[0057] The additional pre-training unit 132 performs additional pre-training on the pre-trained language model 122 using the training data for additional pre-training stored in the fifth storage unit 130. As a result, an additional pre-trained language model 134 is obtained. Each parameter defined by the additional pre-trained language model 134 is stored in a predetermined storage device.

[0058] In this way, the additional pre-trained language model 134 is generated. As will be discussed in relation to later experiments, it was confirmed that the additional pre-trained language model 134 obtained in this manner is robust against speech recognition errors.

[0059] 3. Variant A. First variation In the above embodiment, first, pre-training is performed on BERT to obtain a pre-trained language model 122. Then, noise is added to the additional pre-training text to obtain training data for additional pre-training. Additional pre-training of the pre-trained language model 122 is performed using this additional pre-training data. Noise is added in the initial pre-training. However, this invention is not limited to such embodiments. The entire pre-training process may be performed using training data with added noise. In this case, the additional pre-training text storage unit 111, morphological analysis unit 112, morphological analysis dictionary 113, noise addition unit 124, fourth storage unit 126, additional pre-training training data generation unit 128, and fifth storage unit 130 shown in Figure 2 may be used.

[0060] B. Second variation In the above embodiment, after initial pre-training, additional pre-training is performed using noisy training data. However, this invention is not limited to such embodiments. For example, if there is a language model consisting of BERT that has already been pre-trained using some data (pre-trained language model), additional pre-training may be performed only on that pre-trained language model using noisy training data. In this case as well, the same configuration as in the first modified example can be used.

[0061] C. Third Variation In the above embodiments, as well as the first and second modifications, noisy training data is used for pre-training. However, this invention is not limited to such embodiments. Instead of additional pre-training, training data with added noise, using the same method as in the first embodiment, may be used for fine-tuning to adapt the pre-trained language model to specific application examples. In this case, task-specific labels will be added to the training data. The following third modification relates to such fine-tuning.

[0062] Before describing the third modified example, we will briefly explain an example of applying the trained language model according to this embodiment. Figure 10 shows an outline of the assumed dialogue system 410. The dialogue system 410 shown in Figure 10 is a system that is assumed to engage in dialogue with a user for a predetermined purpose. For example, it is assumed that the speech response module 412 will ask the user questions and collect information about the user, such as their current situation and physical condition, through the interaction. In this case, the interaction with the user will be primarily voice-based, and the use of the additional trained language model 134 according to this embodiment will help improve the performance of the speech / dialogue module.

[0063] The speech response module 412 processes the basic utterance and response to the user input 414 (the user's utterance (speech information) which is recognized as speech, converted into text, and further converted into a reading sequence through morphological analysis), and outputs a speech response output 416. A semantic interpretation module 418 is also used to control the dialogue with higher accuracy. The semantic interpretation module 418 receives system internal information (the information used varies depending on the task, but it is information related to the context of the dialogue response) from the user input 414 and the speech response module 412, and is designed to enable not only standardized dialogue but also natural dialogue. In order to accurately interpret complex, non-standardized user input, various tasks are defined, and the additional pre-trained language model 134 is fine-tuned to suit each task. As a result, the semantic interpretation module 418 can infer the information necessary for the speech response module 412 to perform each task and output it to the speech response module 412. The speech response module 412 uses the output from the semantic interpretation module 418 to output a speech response output 416.

[0064] Possible tasks include, for example, YES / NO discrimination (a type of classification task that categorizes responses into multiple categories), personal attribute discrimination (a task that identifies information about whether a question about a person's preferences has been answered, and extracts keywords from the answers), and small talk (a task that detects user utterances appropriate for starting and ending small talk), as illustrated in the diagram. All of these tasks involve some form of inference based on the input. Then, the additional pre-trained language model 134 will be fine-tuned using training data appropriate for each task. Below, we will explain in more detail an example of applying the pre-trained language model to YES / NO discrimination as an example of a task.

[0065] For example, in a task that classifies answers to a question into multiple categories, the question and the expected answer candidates are treated as a pair of words, and their readings are treated as a reading sequence, forming the word-reading sequence pair in the embodiment described above. Training data is generated by attaching labels indicating the category of the answer candidate to this word-reading sequence pair. The learning process in this case is the same as in normal supervised learning.

[0066] Figure 11 shows an example of training data for fine-tuning in that case. This example is an illustration of what will be used in later experiments.

[0067] Referring to Figure 11, Example 450 is an example in which a robot asks an elderly person about their living conditions. Here, the robot is referred to as the "system." Generally, when a person asks an elderly person about their living conditions, there are questions that are expected to elicit a YES / NO response and questions that are expected to elicit a more open response. Here, we will consider a case where a question is asked that is expected to elicit a YES / NO response, and the response is classified into one of five categories, including YES / NO.

[0068] Let's consider the question, "It seems you eat with family or others at least once a week, but has that increased since last time?" If the response is "More often this month because of my grandchildren's events," as in response 460, the category should be YES. Response 462, "My daughter's family moved away, so I'm lonely," should be categorized as NO. Response 464, "Well, I wonder," is also possible. The category for response 464 should be "Unknown." Response 466, "I saw a strange comedian on TV the other day," is unrelated to the question, so the category should be "Other." Finally, if the response is "I don't have any family anymore," as in response 468, the question itself was inappropriate. Therefore, the category for response 468 should be "PresuppositionFailure."

[0069] The majority of responses fall into one of these five categories. Therefore, in this example, we can fine-tune the model by assigning labels corresponding to these five categories to the noisy training data.

[0070] In tasks like this, it is necessary to perform speech recognition on the other party's response. Using a finely tuned BERT model in this modified example is effective in correcting misrecognitions in this speech recognition. In the semantic interpretation module, the trained language model for YES / NO discrimination performs inference using the recognition result of the speech information (and the reading sequence after morphological analysis) which is the user input, and contextual information such as the question sentence issued by the system to obtain this user input. The probability that the user's response falls into one of the five categories described above is output for each category. The output of this trained language model for YES / NO discrimination (output of semantic interpretation module 408) is supplied to the utterance / response module 402, used for YES / NO determination of ambiguous user input, and reflected in subsequent utterances and responses. The trained language model for YES / NO discrimination described above can be obtained by fine-tuning BERT to suit YES / NO discrimination.

[0071] An example of noisy training data used in this embodiment is shown in Figure 12. Referring to Figure 12, this training data 500 uses the same system questions as those shown in Figure 11. Instead of response 460 in Figure 11, noisy response candidates 510, 512, 514, and 516 are used as response candidates. Response candidates 510, 512, 514, and 516 are obtained by adding 0% noise, 10% noise, 30% noise, and 50% noise to the user response 464 shown in Figure 11, respectively. In the case of training data with 10% noise, 10% of all words in the training data have been replaced with noise. The same approach applies to 30% noise and 50% noise.

[0072] As described later, a pre-trained language model can be obtained by using a BERT model that has been fine-tuned using this training data. This pre-trained language model enables robust speech recognition of user responses and has been shown to improve response classification accuracy. This pre-trained language model is obtained by fine-tuning a pre-trained BERT model to suit the task. Therefore, for any inference task that uses language, a high-performance pre-trained language model can be realized by fine-tuning the BERT model according to this embodiment using appropriate training data and using it for inference.

[0073] 4. Effects As will be described later, the above embodiment enables robust speech recognition. Moreover, only text processing is required to generate training data for pre-training. The computational cost is far lower compared to what is disclosed in Non-Patent Literature 1. Furthermore, the performance of the final obtained language model does not depend on the speech synthesizer or speech recognition system used for training. As a result, training can be performed at a low cost, and a highly accurate language model can be obtained. This language model is independent of the speech recognition system. Therefore, it also has the effect that there is no need for retraining regardless of what kind of speech recognition system is used in the task to which this language model is applied. Furthermore, by using a pre-trained language model, a robust pre-trained language model can be realized.

[0074] In the above embodiment, BERT LARGE is used to train BERT. However, as is clear from the above description, the present invention can be used not only for BERT LARGE but also for large-scale language models that use the same pre-training procedure as BERT. For example, it is known that BERT has a large configuration BERT LARGE and a small configuration BERT BASE. A language model showing high performance can also be obtained for BERT BASE using the same procedure as in the above embodiment. Although the overall configuration of BERT BASE is much smaller than that of BERT LARGE, it can sometimes achieve high performance comparable to BERT LARGE. Therefore, BERT BASE may be applicable to a different range of technologies than BERT LARGE. In both the case of BERT BASE and BERT LARGE, the models trained according to the above embodiment and each modification will be referred to as "Hiragana BERT" or, to save space, "HBERT" in this specification.

[0075] Experiment 2 A. Experiment Setup In the experiment, we adopted the task of classifying the system's responses to questions, as described in the third modification above, and performed fine-tuning of the Hiragana BERT for this purpose.

[0076] Figure 13 shows the statistics of the dataset used for fine-tuning the hiragana BERT used in the experiment. Referring to Figure 13, CData is noise-free data created manually. NData1, NData2, and NData3 are noisy data automatically generated based on CData, and added to the training data in a 1+1 format, with one noisy data and one original data. As explained in relation to the above embodiment, the noise is generated as a pseudo-speech recognition error by randomly selecting a word from the training data and replacing it with a word that has a similar pronunciation. In this experiment as well, the words to be replaced are limited to those with an edit distance of 1 or 2 from the pronunciation of the original word.

[0077] The difference between NData1, NData2, and NData3 lies in the probability of noise addition. NData1 adds noise to words with a 10% probability. In this dataset, the Word Error Rate (WER) was 9.7%. NData2 adds noise to words with a 30% probability. The WER for NData2 was 22.05. NData3 adds noise to words with a 50% probability. The WER for NData3 was 34.15%.

[0078] In Figure 13, the "TRAIN" column represents the number of sentences used for fine-tuning. "DEV" represents the number of sentences in the development data used for hyperparameter selection. "test" represents the number of sentences in the test data prepared in advance to check accuracy. "test.v8.0" is the actual dialogue data obtained in the demonstration experiment, and represents the number of sentences used to evaluate the final Hiragana BERT.

[0079] Figure 14 shows the statistical data of the training data used in the pre-training of the Hiragana BERT in the experiment.

[0080] Referring to Figure 14, two types of hiragana BERT were used in the experiment. Both hiragana BERT models are language models that were further trained according to the above embodiment, based on BERT LARGE, which underwent 1 million steps of pre-training using 2.2 billion causal relationship sentences extracted from Japanese texts collected from the internet as training data.

[0081] The first Hiragana BERT was trained using 18.4 million sentences obtained from Wikipedia on the internet, with a maximum input length of 768 words (word sequence + reading sequence), and further training was performed with 100,000 training steps and a batch size of 1024. In the following explanation, this first Hiragana BERT will be referred to as HBERT LARGE. Wiki,100k It is called that.

[0082] The second hiragana BERT was trained using 2.2 billion causal sentences, the same data used in the training of BERT LARGE, with additional training settings of a maximum length of 768, 200,000 training steps, and a batch size of 1024. In the following explanation, this first hiragana BERT will be referred to as BERT LARGE. Cs,200k It is called that.

[0083] The values ​​of these hyperparameters were evaluated using the average precision of the Hiragana BERT algorithm with the development data, and selected from the following options.

[0084] ·Learning rate (lr): {1e-5, 2e-5, 3e-5, 4e-5, 5e-5, 6e-5} • Number of epochs: {1, 2, 3, 4} Batch size: 256 ·Maximum length: 128 B. Experimental Results Figure 15 shows the experimental results. In the table shown in Figure 15, the leftmost column shows the names of the datasets used for fine-tuning and development. The second column shows the name of the model used and its training parameters. The third column shows the average precision of each model on the development data. The fourth column shows the average precision of each model on the test data. The fifth column shows the average precision of each model on the validation data (text.v8.0).

[0085] Of these results, the most important is the performance of each model on the empirical data (5th column). Focusing on this point, HBERT LARGE Wiki,100k It can be seen that showed the highest performance. In particular, HBERT LARGE, which was fine-tuned using the Ndata3 dataset with high noise probability, performed best. Wiki,100k It is noteworthy that the performance of the HBERT LARGE demonstrated the highest performance. In addition to this, Wiki,100k and HBERT LARGE Cs,200k In all cases, it was confirmed that the performance on the empirical data was higher than that of BERT LARGE before fine-tuning.

[0086] Third computer implementation Figure 16 is an external view of an example of a computer system that functions as the language model learning device 100 shown in Figure 2. Figure 17 is a hardware block diagram of the computer system shown in Figure 16. This computer connects to the other party's home computer, for example, via the internet, and operates to automatically interact with the other party using the screen, voice, and microphone. Alternatively, this computer can be used in connection with a robot that interacts with the other party. If a smaller computer is used, it can also be built into the robot that interacts with the other party.

[0087] Referring to Figure 16, this computer system 950 includes a computer 970 having a DVD (Digital Versatile Disc) drive 1002, and a keyboard 974, a mouse 976, and a monitor 972, all connected to the computer 970, for user interaction. Of course, these are just one example of a configuration for when user interaction is required, and any general hardware and software available for user interaction (e.g., touch panels, voice input, pointing devices in general) can be used.

[0088] Referring to Figure 17, the computer 970 includes, in addition to the DVD drive 1002, a CPU (Central Processing Unit) 990, a GPU (Graphics Processing Unit) 992, a bus 1010 connected to the CPU 990, GPU 992, and DVD drive 1002, a ROM (Read-Only Memory) 996 connected to the bus 1010 for storing the computer 970's boot-up program, etc., a RAM (Random Access Memory) 998 connected to the bus 1010 for storing program instructions, system programs, and work data, etc., and a non-volatile memory SSD (Solid State Drive) 1000 connected to the bus 1010. The SSD 1000 is for storing programs executed by the CPU 990 and GPU 992, as well as data used by programs executed by the CPU 990 and GPU 992. Computer 970 further includes a network interface 1008 that provides a connection to a network 986 enabling communication with other terminals, and a USB port 1006 that allows a USB memory 984 to be attached and detached, and provides communication between the USB memory 984 and various parts within Computer 970.

[0089] The computer 970 further includes an audio interface 1004 connected to the microphone 982 and speaker 980 and the bus 1010, which has the function of reading audio signals, video signals, and text data generated by the CPU 990 and stored in RAM 998 or SSD 1000 according to the instructions of the CPU 990, performing analog conversion and amplification processing to drive the speaker 980, and digitizing the analog audio signal from the microphone 982 and storing it in RAM 998 or SSD 1000 at any address specified by the CPU 990.

[0090] In the above embodiment, the programs that implement each function of the language model learning device 100, the program that implements Hiragana BERT, and their parameters are all stored in a storage medium of an external device (not shown) connected via network I / F 1008 and network 986, for example, as shown in Figure 17: SSD 1000, RAM 998, DVD 978, USB memory 984, or network I / F 1008 and network 986. Typically, these programs, data, and parameters are written to the SSD 1000 from an external source and loaded into the RAM 998 when executed by the computer 970.

[0091] The computer program for operating this computer system to realize the functions of the language model learning device 100 and its components shown in Figure 2 is stored on a DVD 978 inserted into the DVD drive 1002 and transferred from the DVD drive 1002 to the SSD 1000. Alternatively, these programs are stored on a USB memory 984, the USB memory 984 is inserted into the USB port 1006, and the programs are transferred to the SSD 1000. Alternatively, this program may be transmitted to a computer 970 via the network 986 and stored in the SSD 1000.

[0092] The program is loaded into RAM998 at runtime. Of course, the source program may be input using the keyboard 974, monitor 972, and mouse 976, and the compiled object program may be stored in SSD1000. If the program is written in a scripting language, the script may be input using the keyboard 974, etc., and stored in SSD1000. In the case of a program that runs on a virtual machine, the program that functions as a virtual machine must be installed on computer 970 in advance. Neural networks are used for speech recognition and speech synthesis, etc. A pre-trained network may be used, or training may be performed using the language model learning device 100.

[0093] The CPU990 reads the program from RAM998 according to the address indicated by an internal register called the program counter (not shown), interprets the instructions, reads the data necessary for executing the instructions from RAM998, SSD1000, or other devices according to the address specified by the instructions, and executes the processing specified by the instructions. The CPU990 stores the execution result data at an address specified by the program, such as RAM998, SSD1000, or a register within the CPU990. In embodiments using a robot, this is output from the computer as commands to the robot's actuators, audio signals, etc. At this time, the value of the program counter is also updated by the program. The computer program may be loaded directly into RAM998 from DVD978, USB memory 984, or via a network. In addition, some tasks (mainly numerical calculations) within the program executed by the CPU990 are dispatched to the GPU992 according to the instructions included in the program or according to the analysis results when the CPU990 executes the instructions.

[0094] The program that implements the functions of each part according to the above embodiment using the computer 970 includes a plurality of instructions written and arranged to operate the computer 970 to implement those functions. Some of the basic functions necessary to execute these instructions may be provided by the operating system (OS) running on the computer 970, a third-party program, modules of various toolkits installed on the computer 970, or the program execution environment. Therefore, this program does not necessarily have to include all the functions necessary to implement the system and method of this embodiment. This program only needs to include instructions that perform the operation of each of the above-mentioned devices and their components by statically linking or dynamically calling appropriate functions or modules in a controlled manner to obtain the desired result. The method of operating the computer 970 for this purpose is well known and will not be repeated here.

[0095] Alternatively, a computer may be controlled directly by a program without having an operating system installed.

[0096] Furthermore, the GPU992 is capable of parallel processing, allowing it to execute large amounts of calculations associated with machine learning concurrently, in parallel, or in a pipelined manner. For example, parallel computation elements discovered in the program during compilation, or during program execution, are dispatched from the CPU990 to the GPU992 as needed, executed, and the results are returned to the CPU990 either directly or via a predetermined address in RAM998, and assigned to a predetermined variable in the program.

[0097] 4. Further variations The above embodiment assumes Japanese as the target language. Hiragana, a phonetic script, is used as the phonetic symbol to be converted from kanji. However, this invention is not limited to such an embodiment. In the case of Japanese, other phonetic scripts such as katakana may be used as the phonetic symbol, or Romanization may be used. In any case, although some changes to the dictionary structure may be necessary, the procedures for pre-training, additional pre-training, and fine-tuning of the language model may be the same as in the above embodiment. Furthermore, it is also conceivable to use other phonetic scripts besides those mentioned above, such as phonetic symbols.

[0098] The same applies to languages ​​other than Japanese. For example, if there is a symbolic system (like phonetic symbols) that represents the pronunciation of a word with some kind of symbol, the above invention can be applied to any language using such a symbolic system. In this case, the present invention can be applied to both cases where one character (one symbol) represents one phoneme and cases where one syllable or one mora represents one phoneme.

[0099] Furthermore, in the above embodiment, as shown in Figure 8, for each word in the sequence of words being processed, it is first randomly determined whether or not to replace that word with noise. Then, the process of replacing noise is performed only for the words that are to be replaced. However, this invention is not limited to such embodiments. For example, the words to be replaced may be determined according to some method. Restrictions may be placed in some way on the words that may be replaced. The noise words to be replaced may be determined for all words, and then the words to be actually replaced with noise may be determined at the end. Also, the upper limit of the edit distance when selecting words with similar sounds is not limited to 2, but may be 1 or around 3. Depending on the language, this value may be even larger.

[0100] The embodiments disclosed herein are illustrative and not limited to those embodiments. The scope of the present invention is defined by the claims, with reference to the detailed description of the invention, and includes all modifications within the meaning and scope equivalent to the wording contained herein. [Explanation of symbols]

[0101] 100 Language Model Learning Devices 110 Pre-learning text storage unit 111 Additional pre-learning text storage unit 112 Morphological analysis section 113 Dictionary for Morphological Analysis 114 1st memory section 115 2nd memory section 116 Training Data Generation Unit 118 Third memory section 120 Pre-learning Section 122 Pre-trained language models 124 Noise Addition Section 126 4th memory section 128 Additional pre-training data generation unit 130 5th memory section 132 Additional Pre-Learning Section 134 Additional pre-trained language models 140 word sequences / reading sequences 150 Learning Steps 160, 310 word sequence 162, 312 Reading sequence 164 concatenated strings 166, 200, 324, 500 training data 168 Pre-learning 170 BERT 210, 212, 214, 220, 222, 224 Public Notices 226 MLM 230, 232 words 314 Word Selection Section 316 Dictionary for adding noise 318 Search Section 320 Replacement word determination unit 322 Substitution part 332 Processing to add training data 342 word replacement process 400, 402 Pair of reading sequences 410 Dialogue Systems 412 Speech Response Module 418 Semantic Interpretation Module

Claims

1. a conversion means for converting a natural language text to output a string of phonetic symbols; a learning means for learning a language model using the text and the symbol string output by the converting means;

2. The learning means a training data creation means for creating training data for the language model by combining the text and the symbol string output by the conversion means; 2. The language model training device according to claim 1, further comprising: pre-training means for pre-training said language model using said training data.

3. a noise adding means for adding noise to the symbol string to generate a noisy symbol string; training data creation means for creating training data for fine-tuning the language model pre-trained by the pre-training means, using the text, the symbol string, and the noisy symbol string; The language model training device according to claim 2 , further comprising: fine tuning means for fine tuning the pre-trained language model using the training data.

4. the language model includes a pre-trained language model; The learning means a noise adding means for adding noise to the symbol string to generate a noisy symbol string; training data creation means for creating training data for fine-tuning the pre-trained language model using the text, the symbol string, and the noisy symbol string; and fine tuning means for fine-tuning the pre-trained language model using the training data.

5. the language model includes a pre-trained language model; The learning means a noise adding means for adding noise to the symbol string to generate a noisy symbol string; additional training data creation means for creating training data for additional pre-training of the pre-trained language model using the text, the symbol string, and the noise-added symbol string; 2. The language model training device according to claim 1, further comprising: additional pre-training means for performing additional pre-training on the pre-trained language model using the training data.

6. An interactive device that conducts a dialogue with a user based on voice, a trained language model generated by machine learning using at least a natural language text and a string of phonetic symbols converted from the text; a semantic interpretation module having the trained language model and receiving the user's speech information; a speech and response module that receives voice information from the user and executes a dialogue with the user under the control of the semantic interpretation module.

7. A trained language model generated by machine learning using at least natural language text and a string of phonetic symbols converted from the text.

Images