Language processing device, learning device, and their programs

Abstract

To provide a language processing device that can use both an accent dictionary and a learning-based language processing unit.SOLUTION: A language processor 1 includes: a proper noun conversion section 11 that converts proper nouns into reading kana and prosodic symbols surrounded by indicative symbols, using an accent dictionary 10; a language processing unit 12 that converts mixed kanji-kana sentences into intermediate language data, using a first neural network trained to convert the mixed kanji-kana sentences into the intermediate language data, excluding the parts surrounded by indicative symbols; and an accent-joining unit 13 that performs accent-joining of the intermediate language data, using a second neural network that has been learned to perform accent-joining.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a language processing device for speech synthesis, a learning device, and a program therefor. [Background technology]

[0002] Patent Document 1 proposes a sequence-to-sequence Japanese speech synthesis method that uses intermediate language data described in pronunciation and prosodic symbols. In this invention, the language processing unit that converts pronunciation and prosodic symbols from a mixed kanji-kana sentence is realized using an analytical method (for example, Non-Patent Document 1). Because this conventional technology is an analytical method, it is not possible to apply machine learning learning methods using training data, including automatic and manual methods. Therefore, a method has been proposed that realizes a learning-based language processing unit using a Transformer (for example, Non-Patent Document 2). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-34883 [Non-patent literature]

[0004] [Non-Patent Document 1] “OPEN JTalk”, [online], [searched November 2, 2021], Internet<URL:http: / / open-jtalk.sourceforge.net / > [Non-patent document 2] Naoto Kakegawa, Nao Hara, Masanobu Abe, Yusuke Ijima, "Estimating Kana Readings and Prosodic Symbol Sequences from Japanese Text Using Transformer", Autumn Meeting of the Acoustical Society of Japan, 3-2-17, pp. 829-832, Sep. 2020 Summary of the Invention [Problem to be solved by the invention]

[0005] In the method described in Non-Patent Document 2, the Transformer converts mixed kanji and kana sentences into reading kana and prosodic symbols on a one-to-one basis, which means that an accent dictionary cannot be used to convert proper nouns contained in mixed kanji and kana sentences, resulting in incorrect conversions.

[0006] The problems of the conventional technology will be specifically described with reference to FIG. In the example shown in Figure 9, the mixed kanji and kana sentence "Today's weather forecast for Tainai City, Niigata Prefecture" is converted into intermediate language data such as "KYO!UNO#NI^IGATAKE!N#TA!IUCHI#SHIN^NO#TE^ENKIYO!HOU#DES!SU=". Note that in Figure 9, the portions of the intermediate language data related to the explanation are indicated by dashed lines. For example, in weather information, place names such as "Tainai" are important words, so an accent dictionary must be used to accurately convert the place name "Tainai" into the pronunciation and prosodic symbols "TA!INAI." However, a language processing unit using a Transformer may erroneously convert the place name "Tainai" into the pronunciation and prosodic symbols "TA!IUCHI." Such incorrect conversion of intermediate language data is problematic because it is extremely time-consuming to correct after the fact and can cause misreadings when using a system that does not assume automated correction.

[0007] Therefore, an object of the present invention is to provide a language processing device, a learning device, and programs for them that can use both an accent dictionary and a learning-based language processing unit. [Means for solving the problem]

[0008] In order to solve the above problem, the language processing device of the present invention is a language processing device that converts mixed kanji and kana sentences into intermediate language data described in reading kana and prosodic symbols, and is configured to include a proper noun conversion unit, a language processing unit, and an accent combination unit.

[0009] According to this configuration, the proper noun conversion unit converts proper nouns contained in a mixed kanji-kana sentence into pronunciations and prosodic symbols surrounded by predetermined instruction symbols, using an accent dictionary in which pronunciations and prosodic symbols for each proper noun are registered in advance. In addition, the language processing unit converts the kanji and kana mixed sentences converted by the proper noun conversion unit into intermediate language data using a first neural network that has been trained to convert the kanji and kana mixed sentences into intermediate language data, excluding the parts surrounded by instruction symbols. The accent combining unit then performs accent combining on the intermediate language data converted by the language processing unit, using a second neural network that has been trained to combine accents between two consecutive accent phrases.

[0010] In addition, in order to solve the above problem, the learning device of the present invention is a learning device that trains a neural network for converting mixed kanji and kana sentences into intermediate language data described in reading kana and prosodic symbols, and is configured to include a language processing unit and an accent combination unit.

[0011] According to this configuration, the language processing unit trains the first neural network so as to convert a mixed kanji and kana sentence into intermediate language data, excluding the portion enclosed by predetermined instruction symbols. The accent combining unit also trains the second neural network to combine accents of two consecutive accent phrases in the intermediate language data.

[0012] In this way, the learning device trains the first neural network so that proper nouns registered in the accent dictionary are not converted into intermediate language data.The language processing device then uses this first neural network to convert mixed kanji and kana sentences into intermediate language data, excluding proper nouns converted using the accent dictionary, so that the accent dictionary and the learning-based language processing unit can be used together.

[0013] The present invention can also be realized by a program for causing a computer to function as the language processing device or learning device described above. [Effects of the Invention]

[0014] According to the present invention, since an accent dictionary and a learning-based language processing unit can be used in combination, erroneous conversion of proper nouns can be suppressed. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a block diagram showing a configuration of a language processing apparatus according to an embodiment. [Figure 2] FIG. 2 is an explanatory diagram illustrating an example of prosodic symbols in the embodiment. [Figure 3] FIG. 2 is an explanatory diagram illustrating a conversion mode of a language processing device in the embodiment. [Figure 4] FIG. 2 is an explanatory diagram illustrating learning of a first neural network in the embodiment. [Figure 5] FIG. 4 is an explanatory diagram illustrating learning of a second neural network in the embodiment. [Figure 6] 2 is a flowchart showing the operation of the language processing device of FIG. 1; [Figure 7] FIG. 10 is an explanatory diagram illustrating an example of an editing screen in the embodiment. [Figure 8] FIG. 2 is an explanatory diagram illustrating a language processing server in an embodiment. [Figure 9] FIG. 1 is an explanatory diagram illustrating a conventional technique. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the embodiments described below are intended to embody the technical concept of the present invention, and unless otherwise specified, the present invention is not limited to the following. Furthermore, the same means are given the same reference numerals, and their description may be omitted.

[0017] [Configuration of language processing device] The configuration of a language processing device 1 according to an embodiment will be described with reference to FIG. The language processing device 1 has two operating modes: a conversion mode and a learning mode. In the conversion mode, the language processing device 1 converts a mixed kanji and kana sentence into intermediate language data described in reading kana and prosodic symbols. In the learning mode, the language processing device (learning device) 1 trains a neural network for converting a mixed kanji and kana sentence into intermediate language data described in reading kana and prosodic symbols. Note that the language processing device 1 operating in the learning mode may be called a learning device. These operating modes can be manually switched by the user of the language processing device 1 using operating means such as a keyboard or mouse (not shown).

[0018] <Conversion mode> The configuration of the language processing device 1 will be explained below in the order of the conversion mode and the learning mode. As shown in FIG. 1, the language processing device 1 includes an accent dictionary 10, a proper noun conversion unit 11, a language processing unit 12, and an accent combination unit 13.

[0019] The accent dictionary 10 is a dictionary in which the pronunciation and prosodic symbols for each proper noun, such as a place name, a person's name, or a title, are registered in advance, and is referenced by the proper noun conversion unit 11, which will be described later. For example, the accent dictionary 10 registers the place name "Tainai" in association with its pronunciation and prosodic symbol "Tainai."

[0020] The accent dictionary 10 can be constructed by a known method. For example, proper nouns in a sentence are obtained by morphological analysis such as mecab or ginza, and the pronunciation and prosodic symbols of the obtained proper nouns are extracted from a known accent dictionary (e.g., Reference 1).

[0021] Reference 1: "NHK New Japanese Pronunciation and Accent Dictionary," [online], [Retrieved November 2, 2021]

[0022] Here, with reference to Figure 2, the prosodic symbols used in intermediate language data are explained (see, for example, Reference 2). As illustrated in Figure 2, prosodic information includes various types, such as specifying accent positions, specifying phrase / phrase divisions, specifying final intonation, and specifying pauses. Prosodic symbols that indicate accent position are the accent rise symbol "^" and the accent fall symbol "!". The accent rise symbol "^" indicates that the accent rises with the kana (mora) immediately following the symbol. The accent fall symbol "!" indicates that the accent fall with the kana (mora) immediately following the symbol. The accent rise symbol "^" and the accent fall symbol "!" are primary prosodic symbols that indicate accent. The prosodic symbol "#", which indicates the division of an accent phrase, is used to specify the division of a phrase / phrase. To specify the intonation at the end of a sentence, the prosodic symbol "=" which indicates the normal end of a sentence, the prosodic symbol "(" which indicates the end of a sentence with a noun, and the prosodic symbol "?" which indicates the end of a question are used. To specify a pause, the prosodic symbol "," which indicates a pause is used. The prosodic symbol "#" which indicates the end of an accent phrase, the prosodic symbol "=" which indicates the normal end of a sentence, the prosodic symbol "(", the prosodic symbol "?" which indicates the end of a question, and the prosodic symbol "," which indicates a pause are secondary prosodic symbols which indicate the end of reading. Note that these prosodic symbols are only examples, and other symbols may be used.

[0023] Reference 2: "Speech Input / Output Method Standardization Committee, JEITA Standard IT-4006, Symbols for Japanese Text-to-Speech Synthesis, Japan Electronics and Information Technology Industries Association, 2010, pp. 4-10"

[0024] The proper noun conversion unit 11 converts proper nouns contained in a kanji-kana mixed sentence into pronunciations and prosodic symbols surrounded by predetermined reference symbols, using an accent dictionary 10 in which pronunciations and prosodic symbols are registered in advance. These reference symbols are symbols that indicate to a first neural network, described below, the parts that should not be converted. For example, the reference symbols can be set arbitrarily other than the prosodic symbols shown in FIG. 2. In this embodiment, the reference symbols are set as a pair of "[" which indicates the beginning of a proper noun and "[]" which indicates the end of a proper noun. It goes without saying that the reference symbols are not limited to "[" and "[]".

[0025] As shown in FIG. 3, consider the case where a Japanese kanji-kana mixed sentence, "Today's weather forecast for Tainai City, Niigata Prefecture," is input. In this case, the proper noun conversion unit 11 performs morphological analysis (e.g., mecab, ginza) on the kanji-kana mixed sentence to extract proper nouns such as "Niigata" (Niigata) and "Tainai" (Tainai). The proper noun conversion unit 11 then determines whether the extracted proper noun is registered in the accent dictionary 10. If it is registered, the proper noun conversion unit 11 converts the proper noun into a reading kana and prosodic symbol enclosed in directional symbols. In the example of FIG. 3, the proper noun conversion unit 11 converts "Tainai" (Tainai), which is registered in the accent dictionary 10, into "[Tainai]" enclosed in "[" ([" that indicates the beginning of a proper noun) and ]" (] that indicates the end of a proper noun). Therefore, the proper noun conversion unit 11 outputs the kanji-kana mixed sentence, "Today's weather forecast for [Tainai] City, Niigata Prefecture," to the language processing unit 12. In Figure 3, the parts of the mixed kanji and kana sentences and intermediate language data that are relevant to the explanation are shown with dashed lines.

[0026] The language processing unit 12 converts the kanji-kana mixed sentence converted by the proper noun conversion unit 11 into intermediate language data using a first neural network that has been trained to convert the kanji-kana mixed sentence into intermediate language data, excluding the part enclosed by the reference symbols "[" and "]". This first neural network has been trained to convert the kanji-kana mixed sentence into intermediate language data, excluding the part enclosed by the reference symbols "[" and "]". In other words, the first neural network converts the part of the kanji-kana mixed sentence other than the proper nouns converted by the accent dictionary 10 into intermediate language data. For example, the above-mentioned Transformer can be cited as an example of the first neural network (Non-Patent Document 2).

[0027] As shown in Figure 3, consider the case where a mixed kanji and kana sentence, "Today's weather forecast for [ta!inai] city, Niigata prefecture." is input. In this case, the language processing unit 12 converts the part other than "[ta!inai]" enclosed by the directive symbols "[" and "]" into intermediate language data using the first neural network. Therefore, the language processing unit 12 outputs the intermediate language data "Kyou!uno#Ni^igatake!n#[ta!inai]shi^no#ten^kenkiyo!hou#de!su=" to the accent combination unit 13.

[0028] The accent combining unit 13 uses a second neural network that has been trained to combine accents between two consecutive accent phrases to perform accent combining on the intermediate language data converted by the language processing unit 12. For example, the second neural network can be the Transformer described above.

[0029] Here, incorrect accents on proper nouns can cause listeners a strong sense of discomfort. Therefore, we decided to identify the reading and accent of Japanese mixed kanji and kana sentences, and then learn in advance how accent phrases are combined, and reflect this in the intermediate language data. In other words, the second neural network reproduces the same accent combination that Tokyo dialect speakers naturally perform when speaking Japanese. For example, when a Tokyo dialect speaker reads the phrase "Tainai City," they can naturally pronounce it so that the accent kernel is placed on the "shi" part of "Tainai Shino." This is the result of combining the two accent phrases "Tainai" and "Shino," resulting in a single accent phrase, "Tainai Shino," with the accent on the "shi." The rules for such accent combinations have not generally been fully elucidated, making it difficult to formulate rules for them. Therefore, the accent combination unit 13 reproduces such rules using the second neural network.

[0030] Consider the segment "#[ta!inai]shi^no#" as shown in Figure 3. Ignoring the demonstrative symbols, this segment consists of a sequence of accent phrases "ta!inai" and "shi^no." The first accent phrase, "ta!inai," is Type 1, with the accent on the first mora, and the second accent phrase, "," is Type 2, with the accent on the second mora. When these accent phrases are combined, they form Type 5, with the accent on the fifth mora, as in "ta^inai shi!no." In this way, the accent phrase type indicates the mora on which the accent is located. Here, the accent combination unit 13 converts the segment "ta!inai shi^no" to "ta^inai shi!no" using a second neural network that has learned accent combination. Therefore, the accent combination unit 13 outputs the intermediate language data "kyo!uno#ni^igatake!n#[ta^inai]shi!no#ten^enkiyo!hou#de!su=".

[0031] In FIG. 3, the accent type is shown in parentheses for ease of understanding, but the accent attachment unit 13 does not actually need information indicating this accent type. Furthermore, there is a rule that a portion enclosed by the prosodic symbol "#" contains only one accent kernel. Therefore, the accent linking unit 13 may link accents for all portions enclosed by the prosodic symbol "#". Furthermore, when outputting the intermediate language data, the accent attachment unit 13 may delete the reference symbols "[" and "]" from the intermediate language data.

[0032] <Learning mode> Returning to FIG. 1, the learning mode of the language processing device 1 will be described. In this learning mode, the language processing device 1 performs learning on the first neural network used by the language processing unit 12 and the second neural network used by the accent attachment unit 13.

[0033] The proper noun conversion unit 11 converts proper nouns contained in the input kanji-kana mixed sentence into pronunciations and prosodic symbols surrounded by instruction symbols using the accent dictionary 10 in order to generate training data for the language processing unit 12. Note that the processing of the proper noun conversion unit 11 itself is the same as in the conversion mode, so a description thereof will be omitted.

[0034] The language processing unit 12 trains the first neural network to convert the kanji and kana mixed sentence into intermediate language data, excluding the part surrounded by predetermined instruction symbols. Here, the language processing unit 12 can use the kanji and kana mixed sentence input from the proper noun conversion unit 11 as training data. In the example of FIG. 4, the intermediate language data "KYO!UNO#NI^IGATAKE!N#[TA!INAI]SH^NO#TE^ENKIYO!HOU#DES!SU=" is the correct answer data, and the kanji and kana mixed sentence "Today's weather forecast for [TA!INAI] City, Niigata Prefecture." is the training data. Then, the language processing unit 12 trains the first neural network using this training data and correct answer data.

[0035] The accent combination unit 13 trains the second neural network to combine accents of two consecutive accent phrases in the intermediate language data. Here, the accent combination unit 13 can use the two consecutive accent phrases included in the intermediate language data input from the language processing unit 12 as training data. In the example of Figure 5, the two accent phrases "tai!inai" and "shi^no" are the training data, and the combined accent phrase "tai^inai shi!no" is the correct answer data. Then, the accent combination unit 13 trains the second neural network using this training data and correct answer data.

[0036] The learning of the first neural network and the second neural network itself is common, and therefore will not be described here. For example, the first neural network and the second neural network may be trained using a common neural network optimization method such as SGD (Stochastic Gradient Descent) or Adam. The learning termination condition may be, for example, when "the parameters have been updated a predetermined number of times" or "the error no longer decreases even after updating the parameters."

[0037] [Operation of the language processing device] The operation of the language processing device 1 will be described with reference to FIG. It is assumed that the accent dictionary 10 has already been registered, and the first and second neural networks have already been trained.

[0038] As shown in FIG. 6, in step S1, the proper noun conversion unit 11 uses the accent dictionary 10 to convert proper nouns contained in a mixed kanji-kana sentence into pronunciations and prosodic symbols enclosed in the reference symbols "[" and "]".

[0039] In step S2, the language processing unit 12 converts the kanji and kana mixed sentence converted in step S1 into intermediate language data using a first neural network that has been trained to convert the kanji and kana mixed sentence into intermediate language data, excluding the parts surrounded by the instruction symbols "[" and "]".

[0040] In step S3, the accent combining unit 13 performs accent combining on the intermediate language data converted in step S2, using a second neural network that has been trained to combine accents of two consecutive accent phrases.

[0041] [Actions and Effects] As described above, the language processing device 1 trains the first neural network so as not to convert proper nouns registered in the accent dictionary 10 into intermediate language data. Then, the language processing device 1 uses this first neural network to convert mixed kanji and kana sentences into intermediate language data, excluding proper nouns converted by the accent dictionary 10. This allows the language processing device 1 to use the accent dictionary 10 and the learning-based language processing unit 12 in combination, thereby suppressing erroneous conversion of proper nouns and generating high-quality intermediate language data. Furthermore, the language processing device 1 can generate high-quality intermediate language data by reflecting the regularity of accent binding in the intermediate language data. For example, if this intermediate language data is used in speech synthesis, natural synthetic speech can be generated.

[0042] (Example) Referring to FIG. 7, a method for generating training data and correct answer data for the first neural network will be described as an example. FIG. 7 illustrates an editing screen 100 of an editing device 2 (FIG. 8) that generates and modifies accents for mixed kanji and kana sentences (see, for example, Japanese Patent Application Laid-Open No. H09-171392). A document input screen 110 displays the mixed kanji and kana sentence input to the editing device 2. A reading / accent editing screen 120 displays accent information analyzed from the mixed kanji and kana sentence to correspond to the mixed kanji and kana sentence. This accent information includes symbols indicating the level of accent added to the reading of the mixed kanji and kana sentence. An operator may edit the accents to be appropriate using operating means such as a mouse or keyboard (not shown). Then, intermediate language data can be generated by performing morphological analysis on this accent information.

[0043] Here, the mixed kanji and kana sentences that are input to the editing device 2 can be used as training data. Also, the intermediate language data analyzed by the editing device 2 or the intermediate language data edited by the operator can be used as correct answer data. In this way, a large amount of high-quality training data and correct answer data can be obtained, allowing for highly accurate training of the first neural network.

[0044] As shown in Figure 8, the mechanism shown in Figure 7 can also be constructed on a network. Here, multiple operators perform editing work on the editing device 2. Furthermore, the language processing server 1B collects correct answer data and training data from the editing device 2 via a network and trains a first neural network. This language processing server 1B is the language processing device 1 of FIG. 1 run as a server on a network. The language processing server 1B then converts the data into intermediate language data using the trained first neural network, and outputs the converted intermediate language data to the speech synthesis server 3. Furthermore, the speech synthesis server 3 performs speech synthesis using the intermediate language data from the language processing server 1B (for example, JP 2020-34883 A).

[0045] Although the embodiments have been described in detail above, the present invention is not limited to the above-described embodiments, and includes design modifications and the like within the scope of the present invention.

[0046] In the above embodiment, the first neural network and the second neural network are described as Transformers, but are not limited to this. For example, the first neural network and the second neural network may be a Sequence-to-Sequence model, a Sequence-to-Sequence + Attention model, or a Recurrent Neural Network (RNN).

[0047] In the above-described embodiment, the language processing device and learning device are described as independent pieces of hardware, but the present invention is not limited to this. For example, the present invention can also be realized by a program that causes hardware resources such as a CPU, memory, and hard disk of a computer to function as the language processing device or learning device. This program may be distributed via a communication line or written to a recording medium such as a CD-ROM or flash memory. [Explanation of symbols]

[0048] 1 Language processing device (learning device) 1B Language Processing Server 2 Editing equipment 3. Speech synthesis server 10 Accent Dictionary 11 Proper noun conversion section 12 Language Processing Unit 13 Accent binding

Claims

1. A language processing device that converts a mixed kanji and kana sentence into intermediate language data described in reading kana and prosodic symbols, a proper noun conversion unit that converts a proper noun included in the kanji-kana mixed sentence into the pronunciation and the prosodic symbols surrounded by predetermined instruction symbols, using an accent dictionary in which the pronunciation and the prosodic symbols are registered in advance for each proper noun; a language processing unit that converts the kanji-kana mixed sentence converted by the proper noun conversion unit into the intermediate language data using a first neural network that has been trained to convert the kanji-kana mixed sentence into the intermediate language data, excluding the portion surrounded by the instruction symbols; an accent combining unit that combines accents of the intermediate language data converted by the language processing unit using a second neural network that has been trained to combine accents of two consecutive accent phrases; A language processing device comprising:

2. The language processing device according to claim 1 , wherein the first neural network is a Transformer.

3. 3. The language processing device according to claim 1, wherein the second neural network is a Transformer.

4. A learning device for learning a neural network for converting a mixed kanji and kana sentence into intermediate language data described in reading kana and prosodic symbols, comprising: a language processing unit that trains a first neural network to convert the mixed kanji and kana sentence into the intermediate language data, excluding a portion surrounded by a predetermined instruction symbol; an accent combining unit that trains a second neural network to combine accents of two consecutive accent phrases in the intermediate language data; A learning device comprising:

5. 5. The learning device according to claim 4, wherein the first neural network is a Transformer.

6. 6. The learning device according to claim 4, wherein the second neural network is a Transformer.

7. A program for causing a computer to function as the language processing device according to any one of claims 1 to 3.

8. A program for causing a computer to function as the learning device according to any one of claims 4 to 6.

Images