Language processing device, language processing method, and program

Abstract

The purpose of the present disclosure is to perform a training phase process so as to be able to perform language processing as accurately as possible even when input data contains errors that are phonologically similar but have different meanings in an inference phase.　To that end, the present disclosure is a language processing device 3 for performing language processing, the device having: an error generation unit 31 for generating an error sentence that corresponds to an original sentence on the basis of reading that corresponds to text data indicating the original sentence; a language model unit 33, which is a language model based on a neural network model, for generating a prediction sentence from the error sentence on the basis of a language model parameter of the language model; and, an update unit 34 for updating the language model parameter on the basis of a difference between the original sentence and the prediction sentence.

Description

[Technical Field]

[0001] The present disclosure relates to a language processing device, an image processing method, and a program. [Background technology]

[0002] In recent years, there have been many advancements in the field of Bidirectional Encoder Representations from Transformers (BERT). Research into word models has progressed (see Non-Patent Document 1). The language model here refers to a neural network model that obtains distributed representations of tokens, which represent single units of words contained in a text sentence. In this case, instead of inputting a single token, the entire text in which the token is used is input, thereby obtaining distributed representations that reflect the semantic relationships with other tokens in the text (a technique that represents words as high-dimensional real-number vectors, with words with similar meanings corresponding to similar vectors). The step of learning this distributed representation is called pre-training. In addition, pre-trained distributed representations can be used to solve various tasks such as text classification and question answering, and this step is called fine-tuning.

[0003] In the model of Non-Patent Document 1, high performance is achieved in each task in fine-tuning by learning accurate distributed representations of each token through pre-training using a large-scale language resource. It is demonstrating this.

[0004] However, in order to achieve high performance with fine-tuning, sufficient pre-training is required. Therefore, in pre-training, two tasks are used: a word fill-in task and a next sentence prediction task. The word fill-in task involves randomly sampling tokens from the erroneous token sequence c, replacing them with mask tokens, replacing them with random tokens, or leaving the tokens as they are, and then predicting the correct token.

[0005] For example, in the prior art, if there is an original sentence "Today is a good day," as shown in Figure 12, a new token sequence of an incorrect sentence indicating "Today is a good day." is obtained from the correct token sequence obtained by tokenizing it. (Note that " / " is a token separator.) (This represents the eyes.) This token sequence is input into a language model, and the language model is trained to be able to predict the correct token sequence, "Today is good weather." Note that, because the language model of the prior art is implemented using a neural network, it is possible to apply a general supervised neural network learning method in which the correct token sequence is used as a teacher label. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] BERT<(https: / / arxiv.org / abs / 1810.04805> Summary of the Invention [Problem to be solved by the invention]

[0007] However, when applying a conventional neural network model to a task such as summarizing a conversation in a call center by inputting speech utterances, the input is text data, so the speech utterance must be converted into text by speech recognition, which may result in speech recognition errors. Therefore, in order to accurately solve a task such as summarizing a conversation, it is necessary to accurately understand the content and intent of sentences containing speech recognition errors (error sentences).

[0008] Furthermore, while the input for the word fill-in-the-blank task can be considered artificially generated erroneous sentences as described above, the phonological connections of the erroneous token sequence c are not taken into consideration at all. This means that the technique is unable to address errors that are phonologically similar but semantically different, a common type of speech recognition error. As a result, accurate dialogue summarization using speech recognition results is not possible. For example, in Figure 12, an erroneous sentence is created by replacing the "weather" token with the "fire engine" token. However, in actual speech recognition, the phonologically similar "turning point" token is thought to be more likely to appear as an error.

[0009] The present invention has been made in consideration of the above points, and aims to perform processing in the training phase so that language processing can be performed as accurately as possible even if the input data in the inference phase contains errors that are phonetically similar but have different meanings. [Means for solving the problem]

[0010] In order to solve the above problem, the invention according to claim 1 is a language processing device that performs language processing, comprising an error generator that generates an error sentence corresponding to an original sentence based on a reading corresponding to text data indicating the original sentence. an error generator that generates the erroneous sentence by converting at least some of the first morphemes constituting a first morpheme sequence obtained by morphologically analyzing text data representing the original sentence into readings to obtain a second morpheme sequence, further morphologically analyzing a concatenated sentence formed by concatenating a plurality of adjacent second morphemes constituting the second morpheme sequence to obtain a third morpheme sequence, and converting at least some of the third morphemes constituting the third morpheme sequence into a predetermined standard notation; The language processing device is a language model based on a neural network model, and includes: a language model unit that generates a predicted sentence from the erroneous sentence based on language model parameters of the language model; and an update unit that updates the language model parameters based on a difference between the original sentence and the predicted sentence. [Effects of the Invention]

[0011] As described above, according to the present invention, even if the input data in the inference phase contains errors that are phonetically similar but have different meanings, it is possible to perform the training phase processing so as to perform language processing as accurately as possible. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a schematic diagram of a communication system according to an embodiment of the present invention; [Figure 2] FIG. 2 is a diagram illustrating the hardware configuration of a language processing device and a communication terminal. [Figure 3] 1 is a functional configuration diagram of a language processing apparatus according to an embodiment of the present invention; [Figure 4] 1 is a flowchart showing the processing executed by the language processing device in the training (learning) phase. [Figure 5] 10 is a flowchart showing a process in which an error generator generates an error sentence. [Figure 6] FIG. 10 is a conceptual diagram of a process in which an error generator generates an error sentence. [Figure 7] 10 is a flowchart showing a process in which a label creation unit creates a token string of an error sentence and a correct token string. [Figure 8] FIG. 10 is a conceptual diagram of a process in which a label creation unit creates a token string of an error sentence and a correct token string. [Figure 9] 10 is a flowchart showing an experimental process for verifying effectiveness. [Figure 10] FIG. 10 is a table showing other experimental conditions. [Figure 11] FIG. 10 is a table showing experimental results. [Figure 12] FIG. 1 is a conceptual diagram illustrating conventional language processing. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0014] [System configuration of the embodiment] First, an outline of the configuration of a communication system 1 according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a schematic diagram of a communication system according to an embodiment of the present invention.

[0015] 1, the communication system 1 of this embodiment is constructed by a language processing device 3 and a communication terminal 5. The communication terminal 5 is managed and used by a user Y.

[0016] Furthermore, the language processing device 3 and the communication terminal 5 can communicate with each other via a communication network 100 such as the Internet. The communication network 100 may be connected wirelessly or via a wired connection.

[0017] The language processing device 3 is configured by one or more computers. When the language processing device 3 is configured by multiple computers, it may be referred to as a "language processing device" or a "language processing system."

[0018] The language processing device 3 updates language model parameters of a neural network model for extracting features from text data representing the original sentence, based on the original sentence and the erroneous sentence corresponding to the original sentence. As the neural network model, for example, BERT (Bidirectional Encoder Representations from Transformers) is used. The language processing of this type involves executing a method for generating error sentences using the pronunciation of words in a sentence, and a pre-training method for a language model that is robust to speech recognition errors using this method.The language processing device 3 then outputs data indicating features extracted from the text data of the original sentence as result data.As an output method, the result data may be sent to the communication terminal 5, so that a table or the like related to the result data can be displayed or printed on the communication terminal 5, or the table or the like can be displayed on a display connected to the language processing device 3, or the table or the like can be printed on a printer or the like connected to the language processing device 3.

[0019] The communication terminal 5 is a computer, and while a notebook computer is shown as an example in FIG. 1, it is not limited to a notebook computer and may be a desktop computer. The communication terminal may also be a smartphone or a tablet terminal. In FIG. 1, a user Y is operating the communication terminal 5.

[0020] [Hardware configuration of language processing device and communication terminal] Next, the hardware configuration of the language processing device 3 and the communication terminal 5 will be described with reference to Fig. 2. Fig. 2 is a diagram showing the hardware configuration of the language processing device and the communication terminal.

[0021] 2, the language processing device 3 includes a processor 301, a memory 302, an auxiliary storage device 303, a connection device 304, a communication device 305, and a drive device 306. The hardware components constituting the language processing device 3 are connected to each other via a bus 307.

[0022] The processor 301 serves as a control unit that controls the entire language processing device 3, and includes various arithmetic devices such as a CPU (Central Processing Unit). The seed program is read and executed on the memory 302. The processor 301 may include a general-purpose computing on graphics processing unit (GPGPU).

[0023] The memory 302 has a main storage device such as a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The processor 301 and the memory 302 form a so-called computer, and the processor 301 executes various programs read onto the memory 302, thereby enabling the computer to realize various functions.

[0024] The auxiliary storage device 303 stores various programs and various information used when the processor 301 executes the various programs.

[0025] The connection device 304 is a connection device that connects the language processing device 3 to an external device (for example, a display device 310, an operation device 311).

[0026] The communication device 305 is a communication device for transmitting and receiving various types of information to and from other devices.

[0027] The drive device 306 is a device for loading a recording medium 330. The recording medium 330 here includes a CD-ROM (Compact Disc Read-Only Memory), a flexible disk, and the like. The recording medium 330 includes media that record information optically, electrically, or magnetically, such as a disk, a magneto-optical disk, etc. The recording medium 330 may also include semiconductor memory that records information electrically, such as a ROM (Read Only Memory), a flash memory, etc.

[0028] The various programs to be installed in the auxiliary storage device 303 are installed, for example, by setting the distributed recording medium 330 in the drive device 306 and reading the various programs recorded on the recording medium 330 by the drive device 306. Alternatively, the various programs to be installed in the auxiliary storage device 303 may be installed by being downloaded from a network via the communication device 305.

[0029] FIG. 2 also shows the hardware configuration of the communication terminal 5, but the only difference is that the reference numerals have changed from the 300s to the 500s, and the configurations are the same, so a description of these will be omitted.

[0030] [Functional configuration of language processing device] Next, the functional configuration of the language processing device will be described with reference to Fig. 3. Fig. 3 is a diagram showing the functional configuration of the language processing device according to the embodiment of the present invention.

[0031] 3, the language processing device 3 has an input unit 30, an error generation unit 31, a label creation unit 32, a language model unit 33, an update unit 34, and an output unit 39. Each of these units has a function realized by an instruction from the processor 301 in FIG. 2 based on a program.

[0032] 2 stores text data t and language model parameters f. The text data t is, for example, text data acquired from a web page and is used in the training phase. The language model parameters f are model parameters for machine learning such as BERT.

[0033] The input unit 30 inputs text data t from a web page or the like.

[0034] The error generator 31 performs processing such as converting a predetermined morpheme (first morpheme) constituting the text data representing the original sentence into a "reading," and converting a second morpheme based on the first morpheme converted into this "reading" into a predetermined standard notation, thereby generating an erroneous sentence. The detailed processing of the error generator 31 will be described later.

[0035] The label creation unit 32 creates a correct token sequence using comparison labels used when correcting the token sequence of the erroneous sentence to the token sequence of the original sentence. The detailed processing of the label creation unit 32 will be described later.

[0036] The language model unit 33 is a neural network model that obtains distributed representations of tokens, and for example, a model based on BERT, as disclosed in Non-Patent Document 1, can be used. In the training (learning) phase, the language model unit 33 acquires a token sequence c of an erroneous sentence from the label creation unit 32, and creates and outputs a predicted token sequence e using language model parameters f. In the inference phase, the language model unit 33 inputs an original sentence A, vectorizes the text pattern of the text data of the original sentence A, and extracts text features F.

[0037] The update unit 34 updates the language model parameters f based on the correct token sequence d obtained from the label creation unit 32 and the predicted token sequence e obtained from the language model unit 33. This update can be performed in the same way as in supervised learning of a normal neural network.

[0038] The output unit 39 acquires the feature F from the language model unit 33 and outputs it to the outside as result data.

[0039] The difference is that the error generation unit 31 does not handle tokens of text data but handles morphemes, whereas the label creation unit 32, language model unit 33, and update unit 34 handle tokens (and in some cases morphemes). The morphemes referred to here may be any unit suitable for assigning pronunciation. For example, in English, they are units of words. On the other hand, tokens may be any unit accepted by a neural network, including morphemes. Generally, subwords are often used.

[0040] The reason why the error generator 31 does not handle tokens is that, for example, a word with a single meaning, "representative," can be divided into "dai" and "hyo," making it inappropriate for processing that takes into account the reading, as in this embodiment. On the other hand, since a morpheme is a word with a single meaning, "representative," morphological analysis is performed to generate the reading.

[0041] [Processing or Operation of the Embodiment] Next, the processing or operation of this embodiment will be described in detail with reference to FIGS.

[0042] <Training (Learning) Phase> First, the processing in the training (learning) phase will be described with reference to Figure 4. Figure 4 is a flowchart showing the processing executed by the language processing device in the training (learning) phase.

[0043] First, the input unit 30 samples and inputs an original sentence a from the text data t (S10). The original sentence a does not necessarily have to be a complete sentence, and may include an incomplete character string such as "Prime Minister Osugi Yasuhito (representative of the Nationalist Party) is," as shown in FIG. 6(a).

[0044] Next, the error generator 31 generates an error sentence b based on the original sentence a of the text data t (S11).

[0045] (Generating an incorrect sentence) Here, the detailed processing of the error generator 31 will be explained using Fig. 5 and Fig. 6. Fig. 5 is a flowchart showing the processing by the error generator to generate an erroneous sentence. Fig. 6 is a conceptual diagram of the processing by the error generator to generate an erroneous sentence. Note that the erroneous sentences obtained by the series of operations (processing) shown in Fig. 5 are similar to errors in speech recognition in that they take into account how the sentence is pronounced.

[0046] First, as shown in Figures 6(a) and (b), the error generation unit 31 performs morphological analysis on text data representing original sentence a to generate a first morpheme string consisting of multiple morphemes (S111).

[0047] Next, the error generator 31 converts a randomly selected morpheme (an example of a first morpheme) from the first morpheme sequence into a reading (in the case of Japanese, this is "hiragana") (S112). For example, as shown in FIG. 6(b), the error generator 31 converts the randomly selected morphemes ("Oosugi", "Kokuminto", and "Daihyo") into "Oosugi", "Kokuminto", and "Daihyo", respectively, as shown in FIG. 6(c). The token sequence of the original sentence in this state is the second morpheme sequence.

[0048] Next, the error generator 31 concatenates all the morphemes including the morpheme "reading" and returns them to text data (S113), as shown in FIG. 6(d).

[0049] Next, the error generator 31 performs morphological analysis on the returned text data again (S114). For example, as shown in Fig. 6(e), the error generator 31 performs morphological analysis on the returned text data again to generate a third morpheme sequence.

[0050] Next, the error generator 31 converts morphemes (examples of second morphemes) that have standard notations into standard notations (S115). For example, as shown in Fig. 6(f), the error generator 31 generates a standard notation string by converting "kokumin" to "kokumin," "toudai" to "todai," and "hyo" to "hyo." Note that the standard notation is, for example, the first kanji or the like that appears corresponding to a hiragana character when the hiragana character is looked up in a Japanese dictionary.

[0051] Finally, the error generator 31 generates a final error sentence (error sentence b in this case) by concatenating all morphemes including the standard notation, as shown in FIG. 6(g) (S116).

[0052] In this way, the error generator 31 artificially generates an error sentence based on the "reading" (pronunciation) of the text.

[0053] Next, returning to FIG. 4, the label creation unit 32 creates a token string c of the error sentence and a correct token string d based on the original sentence a and the error sentence b (S12).

[0054] (Label creation) Here, detailed processing by the label creation unit 32 will be described with reference to Fig. 7 and Fig. 8. Fig. 7 is a flowchart showing processing by which the label creation unit creates a token sequence of an error sentence and a correct token sequence. Fig. 8 is a conceptual diagram of processing by which the label creation unit creates a token sequence of an error sentence and a correct token sequence.

[0055] First, the label creation unit 32 creates an original sentence token sequence g based on the original sentence a, and creates an error sentence token sequence c based on the error sentence b (S121). For example, as shown in Fig. 8(a), the label creation unit 32 tokenizes the original sentence a into the original sentence token sequence g using an appropriate tokenizer that breaks down the original sentence a into tokens. Similarly, the label creation unit 32 tokenizes the error sentence b into the error sentence token sequence c using an appropriate tokenizer.

[0056] Next, the label creation unit 32 compares the token sequence g of the original sentence with the token sequence c of the erroneous sentence to create a comparison label sequence h of each token (S122). For example, the label creation unit 32 uses the Gestalt Pattern Matching<https: / / www.drdobbs.com / database / pattern-matching-the-gestalt-approach / 184407970?pgno=5> ) method, the comparison label sequence h This method is shown in Figure 8(b).

[0057] As shown in Figure 8(b), in order to compare the token sequence g of the original sentence with the token sequence c of the erroneous sentence and correct the token sequence c of the erroneous sentence to the token sequence g of the original sentence, the label creation unit 32 creates each comparison label that indicates which token in the token sequence c of the erroneous sentence should be processed (deleted, replaced, inserted, or retained), and assigns it to the corresponding token.

[0058] The types of comparison labels that make up the comparison label string h include a deletion label D indicating deletion, a replacement label r indicating replacement, an insertion label I indicating insertion, and a retention label R indicating retention (or match). Note that insertion and deletion are expressed as "empty text" Since a substitution can be expressed as a replacement of a character, only the replacement label r and the retention label R may be used. Also, since a substitution can be expressed as a deletion and insertion, the replacement label r does not need to be used. Furthermore, retention means that the state is maintained, and the retention label R may be used when no label is assigned.

[0059] In Figure 8(b), the replacement label R is assigned to each of the tokens "o", "o", "sugi", "kokumin", "to", "dai", and "hyou", and the rest are assigned the retention label E. This means that by replacing "o", "o", and "sugi" with "Osugi", and "kokumin", "to", "dai", and "hyou" with "Kokuminto" and "dai", the token sequence c of the erroneous sentence can be corrected to the token sequence g of the original sentence.

[0060] If the processing history of the error generator 31 and the label creator 32 (which characters were converted to which hiragana and which kanji characters were converted back to) is stored, the label creator 32 may assign a comparative label based on the stored history information. In this case, it is not necessary to use the technique described in Reference 1.

[0061] Finally, the label creation unit 32 creates a correct token sequence d based on the token sequence g of the original sentence, the token sequence c of the error sentence, and the comparison label sequence h (S123). The requirement for this process is to refer to the comparison label sequence h and assign correct tokens to the erroneous (incorrect) tokens in the token sequence c of the error sentence so that the same sentence as the token sequence g of the original sentence can be reproduced. Tokens to which the retention label E is assigned as the comparison label are considered to be "correct tokens," so the label creation unit 32 does not use these incorrect tokens for training (learning).

[0062] There are several ways to create a correct token sequence, and two of them are explained below.

[0063] First, as a method for creating the correct token sequence d1 (first method), as shown in Figure 8(c), we use the method described in Reference 2 (WLM<<https: / / arxiv.org / pdf / 2011.01900.pdf> There is a method to assign labels as shown in Section 3 and Fig. 1. This (first method) is to assign labels to the token sequence of the error sentence. The method is to assign an insertion label I to unnecessary tokens and assign the missing tokens as labels to the missing parts of the input string. In the example of Figure 8(c), the label creation unit 32 assigns the token "Osugi" to the first "O" token, and assigns the insertion label I to the second "O" and "Sugi" tokens.

[0064] Furthermore, as a method (second method) for creating the correct token string d2, there is a method in which the token "Osugi" is added to each of the tokens "O", "O", and "Sugi" as shown in FIG. 8(d).

[0065] Next, returning to FIG. 4, the language model unit 33 uses the language model parameters f to generate a predicted token sequence e based on the token sequence c of the error sentence using a known method such as BERT (S13).

[0066] Next, the update unit 34 updates the language model parameters f based on the correct token sequence d and the predicted token sequence e using a known method such as BERT (S14).

[0067] This completes the training (learning) phase of processing.

[0068] <Inference phase> In the inference phase, the input unit 30 inputs text data (original sentence A) in which speech utterances related to the voice data have been converted into text by speech recognition, and as in the conventional case, the language model unit 33 vectorizes the text data representing the original sentence A using trained (learned) language model parameters f to generate features F. The output unit 39 then outputs the features as result data. These features as result data are then used for dialogue act estimation, etc.

[0069] The voice data input by the input unit 30 is an example of input data. Another example of input data is text data that contains characters that are phonetically similar but have different meanings. Such text data may be generated, for example, by incorrect keyboard input.

[0070] [Experimental Example] Next, an experimental example for verifying the effects of this embodiment will be described with reference to Fig. 9 to Fig. 11. Fig. 9 is a flowchart showing the experimental process for verifying the effects. Fig. 10 is a table showing other experimental conditions. Fig. 11 is a table showing the experimental results.

[0071] To verify the effectiveness of this embodiment, we pre-trained the model (BERT) shown in Non-Patent Document 1 (prior art) using this embodiment and fine-tuned it for three types of tasks: a dialogue act estimation task, a speech response selection task, and an extractive dialogue summarization task. However, in the pre-training step, we trained BERT in advance using a large amount of text data according to the method in Section 3.1 described in Non-Patent Document 1, and then performed the following experiment. In the second stage, a hyperparameter p is set, the correction task of this embodiment is performed with probability p, and the non-correction task is performed with probability 1-p. The Masked LM task of Task #1 in Section 3.1 described in Patent Document 1 was performed, and so on, with the tasks of this embodiment and those described in Non-Patent Document 1 being switched for each sample (see FIG. 9). Other experimental conditions are shown in FIG. 10, and the experimental results are shown in FIG. 11. As shown in FIG. 11, accuracy improved in the above three tasks, especially when speech recognition results were used as input, confirming the effectiveness of this embodiment.

[0072] Here, a specific experimental process will be described with reference to FIG.

[0073] First, the language model unit 33 initializes language model parameters with parameters of a language model previously trained with a large amount of text data (S101). Next, the input unit 30 samples a mini-batch from the training text data t (S102). Then, if a random number greater than or equal to 0 and less than 1 is less than p (S103; YES), the language model unit 33 updates the language model parameters f according to the above-described embodiment (S104). On the other hand, if a random number greater than or equal to 0 and less than 1 is less than p (S103; NO), the language model unit 33 updates the language model parameters f according to the above-described conventional technology (S105). Then, after the processing of steps S104 and S105, if it is not the last mini-batch (S106; NO), the process returns to the processing of step S102, and new sampling is performed. On the other hand, if it is the last mini-batch (S106; YES), the experiment ends.

[0074] [Major Effects of the Embodiments] As described above, according to this embodiment, the language processing device 3 can create a language model that reflects phonological connections by performing pre-training, such as artificially creating erroneous sentences based on the "reading" of text obtained through morphological analysis and correcting the erroneous sentences to restore the original text. In this way, the language processing device 3 can create erroneous sentences that are similar to errors in speech recognition by taking the "reading" of the text into consideration. Therefore, even when the input data in the inference phase is speech data, the language processing device 3 can perform processing in the training phase to perform language processing as accurately as possible. Furthermore, by comparing erroneous sentences with the correct original text and correcting the erroneous sentences, the language processing device 3 can identify parts that are phonetically similar but erroneous as words or tokens and learn error trends. Therefore, it can accurately solve (execute) tasks such as dialogue summarization, which use actual speech recognition results as input.

[0075] 〔supplement〕 The present invention is not limited to the above-described embodiment, and may have the following configurations or processes (operations).

[0076] The language processing device 3 can be realized by a computer and a program, but this program can also be recorded on a (non-transitory) recording medium or provided via the communication network 100.

[0077] [Additional notes] The above-described embodiment can also be expressed as the following invention.

[0078] [Additional Note 1] A language processing device having a language model based on a neural network model and a processor that performs language processing, The processor: generating an erroneous sentence corresponding to the original sentence based on a reading corresponding to text data indicating the original sentence; generating a predicted sentence from the erroneous sentence based on a language model parameter of the language model; updating the language model parameters based on the differences between the original sentence and the predicted sentence; Language processor.

[0079] [Additional note 2] The language processing device described in Appendix 1, wherein the processor converts a first morpheme, which is a predetermined morpheme that constitutes the text data representing the original sentence, into a second morpheme based on its reading, and generates the erroneous sentence by converting the second morpheme into a predetermined standard notation.

[0080] [Additional note 3] The language processing device described in Appendix 2, wherein the processor uses a morpheme randomly selected from a first morpheme sequence obtained by morphologically analyzing text data representing the original sentence as the second morpheme.

[0081] [Additional Note 4] 4. The language processing device according to claim 2 or 3, wherein the processor converts, among third morphemes obtained by concatenating multiple adjacent second morphemes and performing morphological analysis, the third morphemes having standard notations into the predetermined standard notation.

[0082] [Additional Note 5] 3. The language processing device according to claim 2, wherein converting the first morpheme based on its reading converts the first morpheme into hiragana if the original text is in Japanese.

[0083] [Additional Note 6] Item 1. The language processing device according to item 1, The processor: Dividing the error sentence and the original sentence into an error sentence token sequence and an original sentence token sequence in predetermined processing units, and creating a correct token sequence based on comparison information for correcting the error sentence token sequence to the original sentence token sequence; generating a predicted token sequence constituting the predicted sentence from the token sequence of the error sentence based on the language model parameters; updating the language model parameters based on the correct token sequence and the predicted token sequence; Language processor.

[0084] [Additional Note 7] A language processing method executed by a language processing device having a language model based on a neural network model, The language processing device comprises: generating an erroneous sentence corresponding to the original sentence based on a reading corresponding to text data indicating the original sentence; generating a predicted sentence from the erroneous sentence based on a language model parameter of the language model; updating the language model parameters based on the differences between the original sentence and the predicted sentence; Language processing methods.

[0085] [Additional Note 8] A non-transitory recording medium on which a program for causing a computer to execute the method described in appended paragraph 7 is recorded. [Explanation of symbols]

[0086] 1. Communication Systems 3 Language Processing Unit 5. Communication terminals 30 Input section 31 Error Generation Unit 32 Label Creation Department 33 Language Model Section 34 Update section 39 Output section

Claims

1. A language processing device that performs language processing, an error generator that generates an erroneous sentence corresponding to an original sentence based on a reading corresponding to the text data representing the original sentence, the error generator performing morphological analysis on the text data representing the original sentence to obtain a first morpheme sequence, converting at least some of the first morphemes constituting the first morpheme sequence into readings to obtain a second morpheme sequence, further performing morphological analysis on a concatenated sentence formed by concatenating a plurality of adjacent second morphemes constituting the second morpheme sequence to obtain a third morpheme sequence, and converting at least some of the third morphemes constituting the third morpheme sequence into a predetermined standard notation to generate the erroneous sentence; a language model unit that generates a predicted sentence from the erroneous sentence based on a language model parameter of the language model, the language model unit being based on a neural network model; an update unit that updates the language model parameters based on a difference between the original sentence and the predicted sentence; A language processing device having:

2. The language processing device according to claim 1 , wherein the error generator selects a morpheme randomly from a first morpheme sequence obtained by morphologically analyzing text data representing the original sentence, as the second morpheme.

3. The language processing device according to claim 1 , wherein converting the first morpheme based on its reading is converting the first morpheme into hiragana when the original text is in Japanese.

4. The language processing device according to claim 1, a label creation unit that divides the error sentence and the original sentence into predetermined processing units, each of which is a phrase or word, and creates an error sentence token string and an original sentence token string by treating each division unit as a token; creates a comparison label string in which a comparison label is assigned to each token based on a comparison result between the error sentence token string and the original sentence token string; and creates a correct token string by referring to the comparison label string and assigning, as correct tokens, tokens that correspond to the comparison labels among the tokens that make up the original sentence token string; the language model unit generates a predicted token sequence constituting the predicted sentence from the erroneous sentence token sequence based on the language model parameters; The language processing device, wherein the update unit updates the language model parameters based on the correct token sequence and the predicted token sequence.

5. A language processing method executed by a language processing device having a language model based on a neural network model, The language processing device comprises: In generating an erroneous sentence corresponding to an original sentence based on a reading corresponding to text data representing the original sentence, the method performs morphological analysis on the text data representing the original sentence to obtain a first morpheme sequence, converting at least some of the first morphemes constituting the first morpheme sequence into readings to obtain a second morpheme sequence, further performs morphological analysis on a concatenated sentence formed by concatenating a plurality of adjacent second morphemes constituting the second morpheme sequence to obtain a third morpheme sequence, and converts at least some of the third morphemes constituting the third morpheme sequence into a predetermined standard notation to generate the erroneous sentence, generating a predicted sentence from the erroneous sentence based on a language model parameter of the language model; updating the language model parameters based on the differences between the original sentence and the predicted sentence; Language processing methods.

6. A program causing a computer to execute the method according to claim 5.

Images