Language processing method, language processing device and program

Abstract

In a language processing method according to one aspect of the present disclosure, a computer executes an error sentence creation procedure in which: an error dictionary in which a token sequence is associated with a plurality of first error token sequences, which respectively represent token sequences that are vocally close to the token sequence but portions of which are different from the token sequence, is used to replace a portion of an original sentence token sequence, which represents a token sequence in original text included in given text data, with the first error token sequences; and a second error token sequence, which represents a token sequence that is vocally close to the original text token sequence but a portion of which is different from the original text token sequence, is created as data for language model construction.

Description

[Technical Field]

[0001] The present disclosure relates to a language processing method, a language processing device, and a program. [Background technology]

[0002] In recent years, research on language models such as BERT (Bidirectional Encoder Representations from Transformers) has been actively conducted (see, for example, Non-Patent Document 1). A language model is a neural network model that obtains distributed representations of tokens. Since a language model inputs not a single token but the entire text in which a token is used, it is possible to obtain distributed representations that reflect the semantic relationships with other tokens in the text. Note that a token is a grouping of characters in a text into a unit that has some meaning, typically a word or an even smaller unit such as a subword. Furthermore, a distributed representation is a representation of a token as a vector (generally a high-dimensional real vector). In a distributed representation, tokens that are semantically similar correspond to similar vectors.

[0003] The step of learning the distributed representations using a language model is called pre-training. The pre-trained distributed representations can be used to solve various tasks such as text classification and question answering, and this step is called fine-tuning.

[0004] The language model described in Non-Patent Document 1 demonstrates high performance in each task in fine tuning by learning accurate distributed representations of each token through pre-training using a large-scale language resource. However, sufficient pre-training is required to achieve high performance in each task in fine tuning. Therefore, two tasks, a word fill-in task and a next sentence prediction task, are used in pre-training. The word fill-in task is a task in which an input token sequence is created by randomly sampling tokens from a token sequence and replacing them with mask tokens, replacing them with random tokens, or keeping the tokens as they are, and then predicting the original token sequence (correct token sequence) from this input token sequence.

[0005] For example, as shown in Figure 1, assume that there is an original sentence "It's good weather today." A new token sequence (input token sequence) "Today / [MASK] / good / fire engine / desu / " is obtained from the token sequence (correct token sequence) resulting from tokenizing this original sentence (where " / " indicates a break between tokens). At this time, this input token sequence is input to a language model, and the language model is trained so that it can predict the correct token sequence "Today / is / good / weather / desu / ." Since the language model described in Non-Patent Document 1 is implemented using a neural network, a general supervised neural network training method using the correct token sequence as a teacher label can be applied. Tokenizing refers to dividing text into token units. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding". Summary of the Invention [Problem to be solved by the invention]

[0007] However, the conventional language model described in Non-Patent Document 1 and the like has the following two problems (1) and (2).

[0008] (1) Consider the application of a conventional language model to a task that uses spoken utterances as input, such as summarizing call center conversations. Because the input for a conventional language model is text, the spoken utterance must be converted into text using speech recognition, which can lead to speech recognition errors. Therefore, accurately solving tasks such as dialogue summarization requires accurately understanding the content and intent of sentences containing speech recognition errors (error sentences). Meanwhile, while the input token sequence for the word fill-in-the-blank task can be considered an artificially generated error sentence, as described above, the phonetic connections of the original token sequence are not taken into account at all. As a result, this task is unable to address one type of speech recognition error, namely, "phonetically similar but semantically different errors." As a result, it is difficult to accurately summarize dialogues using speech recognition results. For example, in the example shown in Figure 1, the "weather" token is replaced with the "fire engine" token to create an error sentence. However, in actual speech recognition, the phonetically similar "turning point" token is likely to be more likely to appear as an error.

[0009] (2) To accurately understand the content and intent of a sentence containing a speech recognition error (an error sentence), both the error sentence and the correct sentence are required. In other words, paired data consisting of speech data and its transcription is required. However, paired data consisting of recorded human speech data and its transcription is extremely scarce compared to written text data, and the cost of recording speech and collecting transcription text is also very high. For this reason, it is difficult to build a language model that is robust to speech recognition errors using paired data consisting of speech data and transcription text. On the other hand, one possible method is to artificially create speech data by synthesizing the vast amount of written text data, and then perform speech recognition on the speech data to create error sentences. However, this method is impractical because it requires a large amount of time for both speech synthesis and speech recognition. Furthermore, it is too specialized for the speech synthesis and speech recognition used, making it difficult to build a general-purpose language model.

[0010] The present disclosure has been made in consideration of the above points, and provides a technology that enables a language model that reflects phonetic connections to be constructed in a short period of time. [Means for solving the problem]

[0011] A language processing method according to one aspect of the present disclosure includes an error sentence creation procedure performed by a computer, which uses an error dictionary in which a token sequence is associated with a plurality of first error token sequences, each representing a token sequence that is phonetically similar to the token sequence but partially different from the token sequence, to replace a portion of an original token sequence representing a token sequence of an original sentence included in given text data with the first error token sequence, and creates a second error token sequence representing a token sequence that is phonetically similar to the original token sequence but partially different from the original token sequence as data for building a language model. [Effects of the Invention]

[0012] A technology is provided that enables a language model that reflects phonetic connections to be constructed in a short period of time. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 10 is a diagram showing an example of conversion into an input token sequence in a word fill-in task. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a language processing device. [Figure 3] FIG. 2 is a diagram illustrating an example of a functional configuration of a language processing apparatus in an error dictionary creation phase and a language model training phase. [Figure 4] FIG. 10 is a diagram illustrating an example of an error dictionary. [Figure 5] 10 is a flowchart illustrating an example of a language model learning process. [Figure 6] 10 is a flowchart illustrating an example of an error dictionary creation process. [Figure 7] FIG. 10 is a diagram illustrating an example of a functional configuration of a language processing device in a phoneme conversion model learning phase. [Figure 8] 10 is a flowchart illustrating an example of a phoneme conversion model learning process. [Figure 9] FIG. 10 is a diagram illustrating an example of a functional configuration of a language processing device in a token conversion model learning phase. [Figure 10] 10 is a flowchart illustrating an example of a token conversion model learning process. DETAILED DESCRIPTION OF THE INVENTION

[0014] An embodiment of the present invention will be described below. In the following embodiment, a language processing device 10 capable of constructing a language model that reflects phonetic connections in a short time will be described. To achieve this, the language processing device 10 according to this embodiment artificially creates erroneous sentences using phonemes and constructs a language model that reflects phonetic connections using these erroneous sentences. Furthermore, when creating erroneous sentences, the language processing device 10 according to this embodiment creates the erroneous sentences by referring to a dictionary (hereinafter also referred to as an error dictionary) that associates token sequences (correct token sequences) with token sequences (error token sequences) that are phonetically similar to the token sequences but contain erroneous parts. This allows the language processing device 10 according to this embodiment to construct a language model that reflects phonetic connections in a short time (for example, a time approximately equivalent to the time required for pre-training the language model described in Non-Patent Document 1). A phoneme refers to the smallest unit of speech that humans can distinguish.

[0015] Here, the language processing device 10 according to this embodiment has an "error dictionary creation phase" for creating an error dictionary and a "language model training phase" for building a language model (pre-training) using this error dictionary.

[0016] The error dictionary is created by converting a token sequence into a phoneme sequence using a phoneme conversion model, then reconverting the phoneme sequence into a token sequence using a token conversion model, and then associating the original token sequence with the reconverted token sequence. For this reason, the language processing device 10 according to this embodiment has a "phoneme conversion model training phase" for building (training) a phoneme conversion model, and a "token conversion model training phase" for building (training) a token conversion model. Both the phoneme conversion model and the token conversion model are realized by a neural network that receives a sequence as input and outputs a sequence.

[0017] <Example of hardware configuration of language processing device 10> An example of the hardware configuration of the language processing device 10 according to this embodiment is shown in Fig. 2. As shown in Fig. 2, the language processing device 10 according to this embodiment includes an input device 101, a display device 102, an external I / F 103, a communication I / F 104, a RAM (Random Access Memory) 105, a ROM (Read Only Memory) 106, an auxiliary storage device 107, and a processor 108. Each of these pieces of hardware is connected to each other via a bus 109 so as to be able to communicate with each other.

[0018] The input device 101 is, for example, a keyboard, a mouse, a touch panel, a physical button, etc. The display device 102 is, for example, a display, a display panel, etc. Note that the language processing device 10 does not necessarily have to include at least one of the input device 101 and the display device 102, for example.

[0019] The external I / F 103 is an interface with an external device such as a recording medium 103a. The language processing device 10 can read from and write to the recording medium 103a via the external I / F 103. Examples of the recording medium 103a include a flexible disk, a CD (Compact Disc), a DVD (Digital Versatile Disk), an SD memory card (Secure Digital memory card), and a USB (Universal Serial Bus) memory card.

[0020] The communication I / F 104 is an interface for connecting the language processing device 10 to a communication network. The RAM 105 is a volatile semiconductor memory (storage device) that temporarily stores programs and data. The ROM 106 is a non-volatile semiconductor memory (storage device) that can store programs and data even when the power is turned off. The auxiliary storage device 107 is a storage device (storage device) such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a flash memory. The processor 108 is an arithmetic device such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit).

[0021] The language processing device 10 according to this embodiment has the hardware configuration shown in Fig. 2 and is therefore capable of implementing various processes described below. Note that the hardware configuration shown in Fig. 2 is merely an example, and the hardware configuration of the language processing device 10 is not limited to this. For example, the language processing device 10 may have multiple auxiliary storage devices 107 or multiple processors 108, may not have some of the hardware shown in the figure, or may have various hardware other than the hardware shown in the figure.

[0022] [Error dictionary creation phase and language model training phase] The error dictionary creation phase and the language model training phase will be described below. Note that in the error dictionary creation phase and the language model training phase, it is assumed that the model parameters of the phoneme conversion model and the token conversion model have both been trained.

[0023] <Example of functional configuration of the language processing device 10 in the error dictionary creation phase and the language model training phase> 3 shows an example of the functional configuration of the language processing device 10 in the error dictionary creation phase and the language model training phase. As shown in FIG. 3, the language processing device 10 in the error dictionary creation phase and the language model training phase includes an input data creation unit 201, an error sentence creation unit 202, a label creation unit 203, a language model unit 204, an update unit 205, a phoneme conversion unit 206, a token conversion unit 207, and an error dictionary creation unit 208. Each of these units is realized, for example, by processing executed by the processor 108 or the like of one or more programs installed in the language processing device 10. Furthermore, the language processing device 10 in the error dictionary creation phase and the language model training phase includes a language model parameter storage unit 209, a phoneme conversion model parameter storage unit 210, a token conversion model parameter storage unit 211, and an error dictionary storage unit 212. Each of these storage units is realized, for example, by the auxiliary storage device 107 or the like. At least one of these storage units may be realized by a storage device such as a database server connected to the language processing device 10 via a communication network.

[0024] The input data creation unit 201 creates original sentences from given text data in the language model training phase, and also creates a set of token sequences from given text data in the error dictionary creation phase.

[0025] In the language model training phase, the error sentence creation unit 202 creates an error token sequence from the original sentence using the error dictionary stored in the error dictionary storage unit 212. An error token sequence is a token sequence that is phonetically close to the token sequence representing the original sentence but contains an error (in other words, a token sequence that represents an error sentence in the original sentence). Specific examples of the error dictionary will be described later.

[0026] In the language model learning phase, the label creation unit 203 creates an input token sequence for the word fill-in task and a correct label from the erroneous token sequence. The correct label is information that indicates the correct answer as to how to correct errors in the input token sequence (including mask tokens) to obtain the token sequence of the original text.

[0027] In the language model learning phase, the language model unit 204 creates a predicted label from the input token sequence using the language model parameters stored in the language model parameter storage unit 209. The predicted label is information that predicts how errors in the input token sequence (including mask tokens) should be corrected to obtain the token sequence of the original text (correct token sequence). Here, the language model unit 204 is realized by a language model that uses a neural network (a neural network that receives a token sequence as input and outputs its distributed representation), such as BERT, as described in Non-Patent Document 1.

[0028] In the language model learning phase, the update unit 205 updates the language model parameters from the correct labels and the predicted labels.

[0029] In the error dictionary creation phase, the phoneme conversion unit 206 creates a set of phoneme sequences from a token sequence included in the token sequence set, using the phoneme conversion model parameters stored in the phoneme conversion model parameter storage unit 210. Here, the phoneme conversion unit 206 is realized by a phoneme conversion model using a neural network that receives a sequence as input and outputs a sequence. For example, the Transformer model described in Reference 1 can be used as the phoneme conversion model.

[0030] In the error dictionary creation phase, the token conversion unit 207 creates a set of token sequences from the set of phoneme sequences using the token conversion model parameters stored in the token conversion model parameter storage unit 211. This set of token sequences is a set of error token sequences of the original token sequence (i.e., the token sequence input to the phoneme conversion unit 206). Here, the token conversion unit 207 is realized by a token conversion model that uses a neural network that receives a sequence as input and outputs a sequence. For example, the Transformer model described in Reference 1 can be used as the token conversion model.

[0031] The error dictionary creation unit 208 creates an error dictionary that associates the token sequence input to the phoneme conversion unit 206 with the set of token sequences output from the token conversion unit 207, and stores the created error dictionary in the error dictionary storage unit 212. That is, the error dictionary creation unit 208 defines the token sequence input to the phoneme conversion unit 206 as a "correct token sequence" and each of the token sequences included in the set of token sequences output from the token conversion unit 207 as an "error token sequence," and creates and stores the error dictionary that associates the correct token sequence with each error token sequence.

[0032] The language model parameter storage unit 209 stores the parameters of the language model (language model parameters) that realize the language model unit 204. Note that in the language model learning phase, it is assumed that the language model parameters have not been learned.

[0033] The phoneme conversion model parameter storage unit 210 stores parameters of the phoneme conversion model (phoneme conversion model parameters) that realize the phoneme conversion unit 206. Note that in the error dictionary creation phase, it is assumed that the phoneme conversion model parameters have already been learned.

[0034] The token conversion model parameter storage unit 211 stores parameters of the token conversion model (token conversion model parameters) that realize the token conversion unit 207. Note that in the error dictionary creation phase, it is assumed that the token conversion model parameters have already been learned.

[0035] The error dictionary storage unit 212 stores an error dictionary. An example of the error dictionary is shown in FIG. 4. As shown in FIG. 4, the error dictionary is information in which a single token sequence (correct token sequence) is associated with a plurality of error token sequences related to that token sequence. In the example shown in FIG. 4, the token sequence "shi / te / i / ru" is associated with error token sequences such as "tte / i / ru / [terminal]," "shi / te / ire / [terminal]," "shi / te / i / re," and "shi / te / ire / re." Note that [terminal] represents the terminal of the token sequence.

[0036] <Language model learning process> The language model learning process executed in the language model learning phase will be described below with reference to FIG.

[0037] The input data creation unit 201 samples an original sentence from the given text data (step S101). In this way, an original sentence is created. Note that the text data may be, for example, text data acquired from a web page of a website such as an online encyclopedia.

[0038] Next, the error sentence creation unit 202 creates an error token sequence from the original sentence using the error dictionary stored in the error dictionary storage unit 212 (step S102). Specifically, the error sentence creation unit 202 creates an error token sequence by the following steps 1-1 to 1-5.

[0039] Step 1-1: First, the error sentence generator 202 tokenizes the original sentence to generate a token sequence.

[0040] Step 1-2: Next, the error sentence creation unit 202 samples a portion of the token sequence created in step 1-1. That is, the error sentence creation unit 202 samples a sequence of one or more consecutive tokens from the token sequence as a partial token sequence. The location from which the partial token sequence is sampled may be one location or multiple locations. However, if multiple locations are sampled, the sampling locations of each partial token sequence must not overlap. Furthermore, the sequence length of the sampled partial token sequence may be sampled from within a predetermined range, or may be a fixed sequence length. For simplicity, the following description will be based on a single sampling location. If there are multiple sampling locations, steps 1-3 and 1-4 are performed for each sampling location, and then step 1-5 is performed.

[0041] Step 1-3: Next, the error sentence creation unit 202 obtains a set of error token sequences from the error dictionary using the partial token sequence sampled in step 1-2. That is, the error sentence creation unit 202 searches the error dictionary using the partial token sequence sampled in step 1-2 as a "correct token sequence" to obtain a set of error token sequences corresponding to the correct token sequence.

[0042] Step 1-4: Next, the error sentence generating unit 202 samples one error token sequence from the set of error token sequences obtained in step 1-3.

[0043] Step 1-5: Then, the error sentence creation unit 202 replaces the partial token sequence in the original token sequence (i.e., the token sequence created in step 1-1) with the error token sequence obtained in step 1-4. As a result, a token sequence (an error token sequence representing an error sentence in the original sentence) in which a part (sampling part) of the token sequence of the original sentence has been replaced with an error token sequence is obtained.

[0044] Next, the label creating unit 203 creates an input token sequence and a correct label for the word fill-in-the-blank task from the error token sequence created in step S102 (step S103). Specifically, the label creating unit 203 creates the input token sequence and the correct label by the following steps 2-1 to 2-3.

[0045] Step 2-1: First, the label creating unit 203 randomly samples tokens from the error token sequence created in step S102.

[0046] Step 2-2: Next, the label creation unit 203 performs the word fill-in-the-blank task (MLM: Masked Language Model) described in Non-Patent Document 1 on the tokens sampled in Step 2-1. That is, the label creation unit 203 performs one of the following operations on the tokens sampled in Step 2-1: (a) replace with a mask token, (b) replace with a random token, or (c) keep the token as is. As a result, an input token sequence is obtained from the error token sequence.

[0047] Step 2-3: Then, the label creation unit 203 compares the token sequence of the original text with the input token sequence to create a correct label. The method for creating the correct label is not limited, and any method may be used. For example, if emphasis is placed on detecting and correcting erroneous tokens, a method may be used in which an operation label indicating which of the above operations (a) to (c) should be performed on which token to obtain the token sequence of the original text from the input token sequence is obtained, and then, using this operation label as a reference, a correct label is created for the erroneous token in the input token sequence to reproduce the token sequence of the original text. In this method, a correct label is assigned to a token to which an operation label indicating an operation other than the above operation (c) is assigned. The correct label is an element of the token vocabulary set and indicates which vocabulary token should be replaced with which erroneous token. Alternatively, for example, if emphasis is placed on correcting errors and obtaining fluent text, the token sequence of the original document may be used as the correct label.

[0048] Next, the language model unit 204 uses the language model parameters stored in the language model parameter storage unit 209 to generate a predicted label from the input token sequence generated in step S103 (step S104).

[0049] Then, the updating unit 205 updates the language model parameters using the correct labels created in step S103 and the predicted labels created in step S104 (step S105). This updating can be performed in the same manner as known supervised learning for neural network models. As a result, the language model parameters stored in the language model parameter storage unit 209 are learned, and as a result, a language model that reflects phonetic connections is constructed.

[0050] <Error dictionary creation process> The error dictionary creation process executed in the error dictionary creation phase will be described below with reference to FIG.

[0051] The input data creation unit 201 creates a token sequence set from given text data (step S201). That is, the input data creation unit 201 tokenizes each sentence in the given text data to create a token sequence set. For example, the input data creation unit 201 may create a token sequence set by aggregating the N-grams of each sentence in the given text data and tokenizing the top M sentences. Note that it is sufficient that the language type of the text data matches that of the text data given in the language model training process; however, in order to efficiently obtain an error token sequence in step S102 of FIG. 5, it is preferable that the text data be the same as the text data given in the language model training process.

[0052] The following steps S202 to S204 are repeatedly executed for each token sequence included in the token sequence set. The following describes the case where steps S202 to S204 are executed for a certain token sequence included in the token sequence set.

[0053] Next, the phoneme conversion unit 206 creates a phoneme sequence set from the token sequence using the phoneme conversion model parameters stored in the phoneme conversion model parameter storage unit 210 (step S202). That is, the phoneme conversion unit 206 converts the token sequence into a plurality of phoneme sequences, which are sequences of phonemes (sequences of pronunciations) that are appropriate when the token sequence is vocalized, and creates a phoneme sequence set consisting of some of these phoneme sequences. For example, the phoneme conversion unit 206 may create a phoneme sequence set by selecting the N-best of a plurality of phoneme sequences that are output when the token sequence is input to a phoneme conversion model.

[0054] Next, the token conversion unit 207 creates a token sequence set from the phoneme sequence set created in step S202, using the token conversion model parameters stored in the token conversion model parameter storage unit 211 (step S203). That is, the token conversion unit 207 converts each phoneme sequence included in the phoneme sequence set into a token sequence that is pronounced the same as that phoneme sequence, and creates a token sequence set consisting of these multiple token sequences. This results in a set of token sequences (error token sequences) that are phonetically close to the original token sequence (i.e., the token sequence input to the phoneme conversion unit 206 in step S202) but contain erroneous parts.

[0055] In step S203, one phoneme sequence is converted into one token sequence. However, as in step S202, one phoneme sequence may be converted into multiple token sequences. In this case, only the N-best of these multiple token sequences may be included in the token sequence set.

[0056] Then, the error dictionary creation unit 208 creates an error dictionary by associating the token sequence (i.e., the token sequence input to the phoneme conversion unit 206 in step S202) with each token sequence included in the token sequence set obtained in step S203, and stores this error dictionary in the error dictionary storage unit 212 (step S204). That is, the error dictionary creation unit 208 creates an error dictionary with the token sequence as a "correct token sequence" and each token sequence included in the token sequence set obtained in step S203 as an "error token sequence," and stores the error dictionary in the error dictionary storage unit 212.

[0057] [Phoneme conversion model training phase] The phoneme conversion model training phase will be described below. Note that in the phoneme conversion model training phase, it is assumed that the phoneme conversion model parameters have not been trained.

[0058] <Example of functional configuration of the language processing device 10 in the phoneme conversion model learning phase> An example of the functional configuration of the language processing device 10 in the phoneme conversion model training phase is shown in Fig. 7. As shown in Fig. 7, the language processing device 10 in the phoneme conversion model training phase has an input data creation unit 201, an update unit 205, and a phoneme conversion unit 206. These units are realized, for example, by processing in which one or more programs installed in the language processing device 10 are executed by the processor 108 or the like. Furthermore, the language processing device 10 in the phoneme conversion model training phase has a phoneme conversion model parameter storage unit 210. The phoneme conversion model parameter storage unit 210 is realized, for example, by the auxiliary storage device 107 or the like. Note that the phoneme conversion model parameter storage unit 210 may also be realized by a storage device such as a database server connected to the language processing device 10 via a communication network.

[0059] In the phoneme conversion model learning phase, the input data creation unit 201 creates a token sequence and a correct phoneme sequence from given text data.

[0060] In the phoneme conversion model learning phase, the phoneme conversion unit 206 uses the phoneme conversion model parameters stored in the phoneme conversion model parameter storage unit 210 to create a predicted phoneme sequence that predicts the pronunciation of the token sequence from the token sequence.

[0061] The update unit 205 updates the phoneme conversion model parameters from the correct phoneme sequence and the predicted phoneme sequence in the phoneme conversion model learning phase.

[0062] The phoneme conversion model parameter storage unit 210 stores parameters of the phoneme conversion model (phoneme conversion model parameters) that realize the phoneme conversion unit 206. Note that in the phoneme conversion model learning phase, the phoneme conversion model parameters are not yet learned.

[0063] <Phoneme conversion model training process> The phoneme conversion model training process executed in the phoneme conversion model training phase will be described below with reference to FIG.

[0064] First, the input data creation unit 201 assigns readings to given text data (step S301). Any method may be used to assign readings to text data, but for example, readings may be assigned to text data using the method described in Reference 2. Note that if readings have already been assigned to given text data, this step is unnecessary.

[0065] The input data creation unit 201 tokenizes a certain text in the text data to create a token sequence, and also creates a reading (for example, a phoneme sequence label) assigned to the text as a correct phoneme sequence (step S302).

[0066] Next, the phoneme conversion unit 206 generates a predicted phoneme sequence from the token sequence using the phoneme conversion model parameters stored in the phoneme conversion model parameter storage unit 210 (step S303). For example, the phoneme conversion unit 206 converts the token sequence into a plurality of phoneme sequences, and then generates the most likely phoneme sequence from the plurality of phoneme sequences as the predicted phoneme sequence.

[0067] The updating unit 205 then updates the phoneme conversion model parameters using the correct phoneme sequence generated in step S302 and the predicted phoneme sequence generated in step S303 (step S304). This updating can be performed in the same manner as known supervised learning for neural network models. In this way, the phoneme conversion model parameters stored in the phoneme conversion model parameter storage unit 210 are learned.

[0068] [Token conversion model training phase] The token conversion model learning phase will be described below. Note that in the token conversion model learning phase, it is assumed that the token conversion model parameters have not been learned.

[0069] <Example of functional configuration of the language processing device 10 in the token conversion model learning phase> An example of the functional configuration of the language processing device 10 in the token conversion model learning phase is shown in Fig. 9. As shown in Fig. 9, the language processing device 10 in the token conversion model learning phase has an input data creation unit 201, an update unit 205, and a token conversion unit 207. These units are realized, for example, by processing in which one or more programs installed in the language processing device 10 are executed by the processor 108 or the like. Furthermore, the language processing device 10 in the token conversion model learning phase has a token conversion model parameter storage unit 211. The token conversion model parameter storage unit 211 is realized, for example, by the auxiliary storage device 107 or the like. Note that the token conversion model parameter storage unit 211 may also be realized by a storage device such as a database server connected to the language processing device 10 via a communication network.

[0070] In the token conversion model learning phase, the input data creation unit 201 creates a phoneme sequence and a correct token sequence from given text data.

[0071] In the token conversion model learning phase, the token conversion unit 207 uses the token conversion model parameters stored in the token conversion model parameter storage unit 211 to create a predicted token sequence by predicting, from a phoneme sequence, a token sequence that has the same pronunciation as the phoneme sequence.

[0072] In the token transformation model learning phase, the update unit 205 updates the token transformation model parameters from the correct token sequence and the predicted token sequence.

[0073] The token conversion model parameter storage unit 211 stores parameters of the token conversion model (token conversion model parameters) that realize the token conversion unit 207. Note that in the token conversion model learning phase, it is assumed that the token conversion model parameters have not been learned.

[0074] <Token conversion model learning process> The token conversion model learning process executed in the token conversion model learning phase will be described below with reference to FIG.

[0075] First, the input data creation unit 201 assigns readings to the given text data (step S401). Any method may be used to assign readings to the text data, but for example, readings may be assigned to the text data by the method described in Reference 2. Note that if readings have already been assigned to the given text data, this step is unnecessary.

[0076] The input data creation unit 201 tokenizes a certain text in the text data to create a correct token sequence, and also creates a phoneme sequence from the reading (for example, a phoneme sequence label) assigned to the text (step S402).

[0077] Next, the token conversion unit 207 generates a predicted token sequence from the phoneme sequence using the token conversion model parameters stored in the token conversion model parameter storage unit 211 (step S403).

[0078] Then, the update unit 205 updates the token transformation model parameters using the correct token sequence generated in step S302 and the predicted token sequence generated in step S303 (step S404). This update can be performed in the same manner as known supervised learning for neural network models. As a result, the token transformation model parameters stored in the token transformation model parameter storage unit 211 are learned.

[0079] [summary] As described above, the language processing device 10 according to this embodiment can use a pre-created error dictionary to create a token sequence that is phonetically similar to a certain token sequence but contains erroneous parts (error token sequence). This makes it possible to build a language model that reflects phonetic connections in approximately the same amount of time as with conventional techniques.

[0080] Furthermore, when creating an error dictionary, the language processing device 10 according to this embodiment converts a token sequence into a phoneme sequence, and then reconverts the phoneme sequence into a token sequence to create an error token sequence. This makes it possible to create a token sequence (error token sequence) that is phonetically similar to the original token sequence but includes erroneous parts, and therefore an error dictionary can be created by associating it with the original token sequence.

[0081] The language model constructed (pre-trained) by the language processing device 10 according to this embodiment can be fine-tuned for various tasks, particularly for solving tasks such as dialogue summarization with actual speech recognition results as input with high accuracy.

[0082] The present invention is not limited to the above-described specifically disclosed embodiments, and various modifications, changes, and combinations with known technologies are possible without departing from the scope of the claims.

[0083] [References] Reference 1: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin, "Attention Is All You Need", In Proceedings of the NIPS 2017, pp. 5998-6008, 2017. Reference 2: KyTea (Kyoto Text Analysis Toolkit), Internet<URL:http: / / www.phontron.com / kytea / index-ja.html> [Explanation of symbols]

[0084] 10 Language Processing Device 101 Input Device 102 Display device 103 External I / F 103a Recording media 104 Communication I / F 105 RAM 106 ROM 107 Auxiliary storage 108 processors 109 Bus 201 Input Data Creation Department 202 Error sentence creation section 203 Label Creation Department 204 Language Model Section 205 Update Department 206 Phoneme Conversion Unit 207 Token Conversion Unit 208 Error Dictionary Creation Department 209 Language model parameter storage unit 210 Phoneme conversion model parameter storage unit 211 Token conversion model parameter storage unit 212 Error Dictionary Storage Unit

Claims

1. an error sentence generation procedure for replacing a part of an original token sequence representing a token sequence of an original sentence included in given text data with the first error token sequence, using an error dictionary in which a token sequence is associated with a plurality of first error token sequences each representing a token sequence that is phonetically similar to the original token sequence but partially different from the original token sequence, and generating a second error token sequence representing a token sequence that is phonetically similar to the original token sequence but partially different from the original token sequence as data for building a language model; A language processing method executed by a computer.

2. a speech conversion procedure for generating, using a first token sequence representing a sentence included in given text data, a plurality of phoneme sequences each representing a phoneme sequence when the first token sequence is converted into speech, using a phoneme conversion model realized by a neural network; a token conversion step of using the plurality of phoneme sequences to create, for each of the plurality of phoneme sequences, a third erroneous token sequence representing a token sequence that has the same pronunciation as that represented by the phoneme sequence but is partially different, by a token conversion model realized by a neural network; an error dictionary creation step of creating the error dictionary by associating the first token sequence with a plurality of the third error token sequences; 2. The language processing method according to claim 1, wherein the method is executed by a computer.

3. The language processing method according to claim 2 , wherein each of the phoneme conversion model and the token conversion model is realized by a neural network that receives a sequence as input and outputs a sequence.

4. a label generation step of generating an input token sequence for a word fill-in task and a correct label for the input token sequence using the second error token sequence; a language model step of generating predicted labels representing a distributed representation of the input token sequence using a language model realized by a neural network using the input token sequence; an update procedure for updating parameters of the language model using the correct label and the predicted label; The language processing method according to any one of claims 1 to 3, wherein the method is executed by a computer.

5. an error sentence creation unit configured to use an error dictionary in which a token sequence is associated with a plurality of first error token sequences, each representing a token sequence that is phonetically similar to the token sequence but partially different from the token sequence, to replace a portion of an original token sequence representing a token sequence of an original sentence included in given text data with the first error token sequence, and to create a second error token sequence representing a token sequence that is phonetically similar to the original token sequence but partially different from the token sequence, as data for building a language model; A language processing device having:

6. an error sentence generation procedure for replacing a part of an original token sequence representing a token sequence of an original sentence included in given text data with the first error token sequence, using an error dictionary in which a token sequence is associated with a plurality of first error token sequences each representing a token sequence that is phonetically similar to the original token sequence but partially different from the original token sequence, and generating a second error token sequence representing a token sequence that is phonetically similar to the original token sequence but partially different from the original token sequence as data for building a language model; A program that causes a computer to execute the following.

Images