Speech synthesis device, speech synthesis method, and program

Abstract

In deep neural network (DNN) speech synthesis, the present invention allows synthesized speech to be adjusted more efficiently with respect to similar adjustment locations. A speech synthesis device of the present embodiment is provided with an encoder, a decoder, and an adjusting unit. The encoder, by means of a first neural network, converts attribute information per speech unit into an intermediate representation. The decoder, by means of a second neural network, generates acoustic-feature volumes from the intermediate representations. The adjusting unit utilizes, with at least the attribute information per speech unit being a key, an adjustment dictionary in which adjustment instructions for the acoustic-feature volumes are made values, to determine a segment to which an adjustment instruction will be applied according to the key, and adjusts the acoustic-feature volume in the determined segment on the basis of the adjustment instruction.

Description

Speech synthesis device, speech synthesis method and program

[0001] An embodiment of the present invention relates to a speech synthesis device, a speech synthesis method, and a program.

[0002] Recent speech synthesis technologies have achieved high-quality, human-like synthesized speech by using deep neural networks (DNNs). However, machine learning has its limitations, and it can sometimes synthesize speech with incorrect pronunciation or unnatural prosody, making it necessary to make adjustments to correct these issues in order to improve product quality.

[0003] In conventional HMM speech synthesis technology, adjustments can be made efficiently using, for example, a speech synthesis dictionary editing device disclosed in Patent Document 1. In addition, in DNN speech synthesis, for example, Patent Document 2 proposes a speech synthesis device that can obtain a high-quality synthetic speech signal when generating a synthetic speech signal in which the way a specific part of a text is read aloud is adjusted.

[0004] JP 2014-174278 A JP 2022-81691 A

[0005] M. Morise, F. Yokomori, and K. Ozawa, “WORLD: a vocoder-based high-quality speech synthesis system for real-time applications,” IEICE transactions on information and systems, vol. E99-D, no. 7, pp. 1877-1884, 2016. A. Tamamori, T. Hayashi, K. Kobayashi, K. Takeda and T. Toda, “Speaker-Dependent WaveNet Vocoder.Proc.” Proc. Interspeech 2017, pp. 1118-1122, 2017. A. Oord, O. Vinyls, and K. Kavukcuoglu, “Neural Discrete Representation Learning”, in Advances in Neural Information Processing Systems, vol. 30, 2017.

[0006] However, with conventional techniques, it has been difficult to adjust synthesized speech more efficiently for similar adjustment points in DNN speech synthesis.

[0007] A speech synthesis device according to an embodiment includes an encoder, a decoder, and an adjustment unit. The encoder converts attribute information of speech units into an intermediate representation using a first neural network. The decoder generates acoustic features from the intermediate representation using a second neural network. The adjustment unit uses an adjustment dictionary that uses at least the attribute information of the speech units as keys and adjustment instructions for the acoustic features as values ​​to determine a section to which the adjustment instructions are to be applied using the keys, and adjusts the acoustic features of the determined section based on the adjustment instructions.

[0008] FIG. 1 is a diagram illustrating an example of the functional configuration of a speech synthesis device according to a first embodiment. FIG. 2 is a diagram illustrating an example of the functional configuration of an adjustment unit according to the first embodiment. FIG. 3 is a flowchart illustrating an example of the overall flow of a speech synthesis method according to the first embodiment. FIG. 4 is a flowchart illustrating an example of a detailed flow of the duration adjustment process (step S4) according to the first embodiment. FIG. 5 is a diagram illustrating an example of duration adjustment according to the first embodiment. FIG. 6 is a flowchart illustrating an example of a detailed flow (in the case of a logarithmic fundamental frequency) of the acoustic feature adjustment process (step S6) according to the first embodiment. FIG. 7 is a diagram illustrating an example of adjustment of a logarithmic fundamental frequency according to the first embodiment. FIG. 8 is a diagram illustrating an example of the functional configuration of a speech synthesis device according to a second embodiment. FIG. 9 is a diagram illustrating an example of the functional configuration of an index acquisition unit according to the second embodiment. FIG. 10 is a diagram illustrating an example of the functional configuration of an adjustment unit according to the second embodiment. FIG. 11 is a flowchart illustrating an example of the overall flow of a speech synthesis method according to the second embodiment. FIG. 12 is a flowchart illustrating an example of a detailed flow of the cluster number acquisition process (step S23) according to the second embodiment. FIG. 13 is a flowchart illustrating an example of a detailed flow of the duration adjustment process (step S25) according to the second embodiment. FIG. 14 is a flowchart showing an example of a detailed flow (in the case of a logarithmic fundamental frequency) of the acoustic feature adjustment process (step S27) of the second embodiment. FIG. 15 is a diagram showing an example of the functional configuration of a speech synthesis device of the third embodiment. FIG. 16 is a diagram showing an example of the functional configuration of an acoustic feature decoder of the third embodiment. FIG. 17 is a diagram showing an example of the functional configuration of an adjustment unit of the third embodiment. FIG. 18 is a flowchart showing an example of the overall flow of a speech synthesis method of the third embodiment. FIG. 19 is a flowchart showing an example of a detailed flow of the intermediate representation sequence adjustment process (step S38) of the third embodiment. FIG. 20 is a diagram showing an example of the spectrum of synthetic speech when the intermediate representation sequence is adjusted by the adjustment process (step S38) of the third embodiment. FIG. 21 is a diagram showing an example of the hardware configuration of a speech synthesis device of the first to third embodiments.

[0009] Hereinafter, embodiments of a speech synthesis device, a speech synthesis method, and a program will be described in detail with reference to the accompanying drawings.

[0010] First Embodiment First, an overview of a speech synthesis device according to a first embodiment will be described.

[0011] 1 is a diagram showing an example of the functional configuration of a speech synthesizer 1 according to a first embodiment. The speech synthesizer 1 according to the first embodiment includes an analyzer 11, an encoder 12, a duration decoder 13, an acoustic feature decoder 14, a vocoder 15, and an adjuster 16.

[0012] In the speech synthesizer 1 of the first embodiment, the adjustment unit 16 adjusts the duration and each acoustic feature, and the vocoder 15 generates a speech waveform from the adjusted acoustic feature. The adjustment unit 16 is provided with pre-created adjustment dictionaries 161, 162, 163, 164, and 165 ( FIG. 2 ) of duration and each acoustic feature, and uses these dictionaries to adjust the duration and each acoustic feature. The keys of the entries in the adjustment dictionaries for duration and each acoustic feature are attribute information of speech units, and the values ​​are adjustment instructions. This allows the user to obtain appropriate synthesized speech without having to make the same adjustments repeatedly for similar problems.

[0013] Each functional block will be described in detail below.

[0014] The analysis unit 11 analyzes the input text and outputs attribute information for each speech unit. In Japanese, speech units include, for example, moras and phonemes. The attribute information is a collection of linguistic and phonetic information about the speech unit. For example, the attribute information includes the types of the preceding and following speech units, accent types and their relative positions within the accent phrase, and part-of-speech information.

[0015] The encoder 12 receives the vector representation of the attribute information of each speech unit and outputs a series of intermediate representations of the neural network (hereinafter referred to as the "intermediate representation series"). The intermediate representations are latent representations that contain information for ultimately obtaining a speech waveform, but are generally difficult for humans to interpret. In the first embodiment, there is a one-to-one correspondence between the attribute information of each speech unit and each intermediate representation.

[0016] The duration decoder 13 receives the intermediate representation sequence as input and outputs the duration (duration) using a neural network. The duration is the number of frames of the acoustic features corresponding to each speech unit. A frame is a waveform extraction unit when analyzing and synthesizing a speech waveform, and is either a fixed length or a length determined based on the pitch period.

[0017] The acoustic feature decoder 14 receives the intermediate representation sequence as input and outputs acoustic features based on the duration using a neural network. In the first embodiment, the acoustic features used are logarithmic fundamental frequency, energy, spectral features, voiced / unvoiced flags, and aperiodicity indices. The logarithmic fundamental frequency uses a value interpolated in unvoiced parts using values ​​of the preceding and following voiced parts. In the following description of the present specification, mel-linear spectral pairs are used as spectral features, but other spectral features, such as mel-cepstrum, mel-spectrogram, or intermediate representations of machine-learned spectral information, may also be used.

[0018] Note that the duration may be treated as one of the acoustic features, and the duration decoder 13 and the acoustic feature decoder 14 may be implemented by a single acoustic feature decoder.

[0019] The vocoder 15 generates a speech waveform from the acoustic features. The vocoder 15 generates the speech waveform using, for example, the signal processing method described in Non-Patent Document 1. Alternatively, for example, the vocoder 15 may generate the speech waveform using the neural network described in Non-Patent Document 2.

[0020] Next, the adjustment unit 16, which is a feature of the first embodiment, will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the functional configuration of the adjustment unit 16 of the first embodiment. The adjustment unit 16 includes a duration adjustment dictionary 161, a logarithmic fundamental frequency adjustment dictionary 162, an energy adjustment dictionary 163, a Mel-linear spectral pair adjustment dictionary 164, and an aperiodicity index adjustment dictionary 165.

[0021] The keys of the entries in the adjustment dictionary for duration and each acoustic feature are attribute information of speech units, and the values ​​are adjustment instructions. During speech synthesis, the adjustment unit 16 refers to the adjustment dictionary for duration and each acoustic feature, determines a section to which the adjustment instructions are to be applied based on the keys of each entry, and adjusts the acoustic features of the determined section based on the adjustment instructions.

[0022] <<Processing of Speech Synthesis Device 1>> Figure 3 is a flowchart showing an example of the overall flow of the speech synthesis method of the first embodiment. First, the analysis unit 11 analyzes the input text and outputs attribute information for each speech unit (step S1). For example, the analysis unit 11 performs morphological analysis on the input text to obtain linguistic information used for speech synthesis, such as reading information and accent information, and outputs attribute information for each speech unit from the obtained reading information and linguistic information. Alternatively, the analysis unit 11 may create attribute information for each speech unit from corrected reading and accent information corresponding to the input text that has been created separately.

[0023] Next, the encoder 12 generates an intermediate representation sequence from the vector representation of the attribute information of each speech unit (step S2), and the duration decoder 13 generates pre-adjustment durations from the intermediate representation sequence (step S3).

[0024] Next, the adjuster 16 adjusts the duration before adjustment obtained in step S3 based on the attribute information of each speech unit (step S4).

[0025] 4 is a flowchart showing an example of the detailed flow of the duration adjustment process (step S4) in the first embodiment. First, the adjuster 16 acquires attribute information of the speech unit at the beginning of the sentence (step S4-1), and searches the duration adjustment dictionary 161 for an entry whose attribute information of the speech unit that is the key matches (step S4-2).

[0026] If an entry is found (Yes in step S4-3), the adjustment unit 16 applies the adjustment instruction of the entry to the duration corresponding to the attribute information of the speech unit acquired in step S4-1 (step S4-4).If no entry is found (No in step S4-3), the process proceeds to step S4-5.

[0027] If adjustment is complete for all speech units (Yes in step S4-5), the process ends. If adjustment is not complete for all speech units (No in step S4-5), the adjuster 16 acquires attribute information for the next speech unit (step S4-6) and starts the process from step S4-2.

[0028] An example of the duration adjusted in step S4 is shown in FIG. 5. FIG. 5 is a diagram showing an example of duration adjustment in the first embodiment. FIG. 5 shows an example of duration adjustment when the speech unit is a phoneme and the sentence "Hello." is input. Since the input sentence contains an N (n) sandwiched between the vowel o (o) and the consonant n (na row), the first entry in the duration adjustment dictionary 161 is found in step S4-2. Therefore, in step S4-4, an adjustment instruction is applied to multiply the duration, which is the value of that entry, by 0.5. As a result, the duration is changed from 22 frames to 11 frames, which is 0.5 times that amount.

[0029] 3, the acoustic feature decoder 14 generates each pre-adjustment acoustic feature from the intermediate representation sequence based on the adjusted duration (step S5). Next, the adjustment unit 16 adjusts each acoustic feature based on the pre-adjustment acoustic feature and attribute information of each speech unit (step S6).

[0030] As an example, the details of the logarithmic fundamental frequency adjustment method are shown in Fig. 6. Fig. 6 is a flowchart showing a detailed example (in the case of logarithmic fundamental frequency) of the acoustic feature adjustment process (step S6) of the first embodiment. First, the adjustment unit 16 acquires attribute information of the speech unit at the beginning of the sentence (step S6-1), and searches the logarithmic fundamental frequency adjustment dictionary 162 for an entry whose attribute information of the speech unit that is the key matches (step S6-2).

[0031] If an entry is found (Yes in step S6-3), the adjustment unit 16 applies the adjustment instructions of that entry to the section corresponding to the attribute information of the speech unit acquired in step S6-1 (step S6-4).If no entry is found (No in step S6-3), the process proceeds to step S6-5.

[0032] If adjustment is complete for all speech units (Yes in step S6-5), the process ends. If adjustment is not complete for all speech units (No in step S6-5), the adjuster 16 acquires attribute information for the next speech unit (step S6-6) and starts the process from step S6-2.

[0033] The adjustment unit 16 also performs adjustment on other acoustic features other than the logarithmic fundamental frequency using the same process as in FIG.

[0034] An example of the logarithmic fundamental frequency adjusted in step S6 is shown in FIG. 7. FIG. 7 is a diagram showing an example of adjustment of the logarithmic fundamental frequency in the first embodiment. FIG. 7 shows an example of adjustment when the speech unit is a phoneme, the input sentence is "jikō.", and the analysis unit 11 analyzes the input sentence as having a 4-mora 3-type accent pattern. In this case, since the input sentence contains the vowel u (u) of the third mora "mu," the first entry in the logarithmic fundamental frequency adjustment dictionary 162 is found in step S6-2. Therefore, in step S6-4, an adjustment instruction is applied to increase the logarithmic fundamental frequency, which is the value of that entry, by 0.1. As a result, as shown in FIG. 7, the logarithmic fundamental frequency of the section corresponding to the vowel u (u) of "mu" is increased by 0.1.

[0035] 3 , finally, the vocoder 15 generates a speech waveform from the adjusted acoustic features obtained in step S6 (step S7). The speech waveform generated in step S7 can be used as desired by the user. For example, the speech waveform generated in step S7 can be played back by the user on a speech playback device (e.g., a speaker) external to the speech synthesizer 1, or can be stored in a storage device external to the speech synthesizer 1.

[0036] As described above, the speech synthesizer 1 of the first embodiment includes an adjustment unit 16 that adjusts the duration and each acoustic feature. The adjustment unit 16 includes a pre-created adjustment dictionary of durations and each acoustic feature, and adjusts the duration and each acoustic feature using the adjustment dictionary during speech synthesis. The keys of entries in the adjustment dictionary for durations and each acoustic feature are attribute information of speech units, and the values ​​are adjustment instructions. The speech synthesizer 1 adjusts the duration and each acoustic feature using the adjustment unit 16, and generates a speech waveform from the adjusted acoustic features using the vocoder 15, thereby enabling the user to obtain appropriate synthesized speech without having to make the same adjustments repeatedly to address similar problems.

[0037] <Details of Each Unit of the Speech Synthesizer 1> The adjustment instructions applied by the adjustment unit 16 are operations for correcting each problem. For example, problems are corrected by applying an operation to multiply the duration by a specified value, and an operation to add specified values ​​to the logarithmic fundamental frequency and energy. Furthermore, for example, the mel-linear spectrum pair and the aperiodicity index, which are multidimensional acoustic features, are corrected by replacing them with specified vectors. As a method for specifying a vector, a vector may be specified directly, or a list of replacement vectors may be created and its index may be specified. In the latter case, during adjustment, the adjustment unit 16 reads the vector of the corresponding index from the list of replacement vectors and replaces each acoustic feature at the adjustment location.

[0038] Each neural network used in the speech synthesizer 1 of the first embodiment is trained using a statistical method. During training, each neural network may be trained simultaneously. For example, the neural networks used in the encoder 12, the duration decoder 13, and the acoustic feature decoder 14 may be trained simultaneously. Furthermore, when a neural network is used in the vocoder 15, in addition to being trained using a statistical method as described above, that neural network may also be trained using a statistical method simultaneously with each of the other neural networks used in the speech synthesizer 1.

[0039] An example of a method for adding a new entry to the adjustment dictionaries 161, 162, 163, 164, and 165 for duration and each acoustic feature will be shown below. First, the speech synthesizer 1 of the first embodiment synthesizes speech from an arbitrary input text. Next, an adjuster of the speech synthesizer 1 (e.g., a developer at a vendor) listens to the synthesized speech obtained by the speech synthesizer 1 and checks for any problems.

[0040] If a problem is found, the adjuster identifies the location of the problem and the duration or acoustic feature that is causing the problem, and determines adjustment instructions to obtain appropriate synthesized speech.

[0041] Next, the adjuster determines attribute information from the attribute information of the speech unit at the location where the identified problem occurs that can be adjusted in step S4 or step S6 when a similar problem occurs, and that can appropriately determine the section to which the adjustment instruction should be applied so that the adjustment instruction is not applied in step S4 or step S6 when no problem occurs.

[0042] Then, in response to the adjuster's operational input, the speech synthesis device 1 of the first embodiment adds an entry to the adjustment dictionary corresponding to the duration or acoustic feature that was causing the problem, with the attribute information determined by the adjuster as the key and the adjustment instruction determined by the adjuster as the value.

[0043] <Effects of the speech synthesis device 1 of the first embodiment> In the speech synthesis device 1 that converts attribute information of speech units into an intermediate representation and generates acoustic features using an encoder-decoder type neural network, it is possible to provide a speech synthesis device 1 that does not require repeated adjustments to address problems that occur under the same conditions.

[0044] Specifically, as described above, in the speech synthesizer 1 of the first embodiment, the encoder 12 converts speech-unit attribute information into an intermediate representation using a first neural network. The decoders (the duration decoder 13 and the acoustic feature decoder 14 in the first embodiment) generate acoustic features from the intermediate representation using a second neural network. The adjuster 16 uses an adjustment dictionary (see FIG. 2 ) that uses at least speech-unit attribute information as keys and adjustment instructions for acoustic features as values ​​to determine a section to which adjustment instructions are to be applied using the keys, and adjusts the acoustic features of the determined section based on the adjustment instructions.

[0045] As a result, the speech synthesizer 1 of the first embodiment can adjust the synthesized speech more efficiently for similar adjustment points. For example, it becomes possible to obtain appropriate synthesized speech without the adjuster having to make the same adjustments multiple times for similar problems. In conventional DNN speech synthesis technology, the user had to input the amount of adjustment each time, which was time-consuming and required the user to repeatedly input the same amount of adjustment for similar adjustment points.

[0046] Second Embodiment Next, a second embodiment will be described. In the description of the second embodiment, the same description as in the first embodiment will be omitted, and only the differences from the first embodiment will be described.

[0047] Overview of the Speech Synthesizer 2 FIG. 8 is a diagram illustrating an example of the functional configuration of the speech synthesizer 2 according to the second embodiment. The speech synthesizer 2 according to the second embodiment includes an adjustment unit 27 that adjusts durations and acoustic features based on adjustment dictionaries 271, 272, 273, 274, and 275 (FIG. 10) for durations and acoustic features, using information identifying intermediate representations acquired by an index acquisition unit 26 (FIG. 9) as a key in addition to attribute information for speech units. This makes it possible to specify a section to which appropriate adjustments are applied without specifying detailed attribute information for speech units. The information identifying the intermediate representation is a number (an example of an index) obtained by a machine learning model that inputs the intermediate representation, specifically, a cluster number obtained when the intermediate representation is classified by a clustering model. Using cluster numbers improves key interpretability while maintaining a compact key.

[0048] Each functional block will be described in detail below.

[0049] The speech synthesizer 2 of the second embodiment differs from the speech synthesizer 1 of the first embodiment in that it includes an index acquisition unit 26. Similarly to the speech synthesizer 1 of the first embodiment, the speech synthesizer 2 of the second embodiment also includes an analysis unit 21, an encoder 22, a duration decoder 23, an acoustic feature decoder 24, a vocoder 25, and an adjustment unit 27.

[0050] Next, the index acquisition unit 26, which is one of the features of the second embodiment, will be described with reference to Fig. 9. Fig. 9 is a diagram showing an example of the functional configuration of the index acquisition unit 26 of the second embodiment. The index acquisition unit 26 outputs cluster numbers when each intermediate representation output from the encoder 22 is classified by a clustering model. The index acquisition unit 26 includes a list 261 of representative vectors of each cluster obtained from a clustering model trained in advance.

[0051] 10 is a diagram showing an example of the functional configuration of an adjustment unit according to the second embodiment. The adjustment unit 27 according to the second embodiment includes adjustment dictionaries 271, 272, 273, 274, and 275 for duration and each acoustic feature (logarithmic fundamental frequency, energy, Mel-linear spectral pair, and aperiodicity index). Unlike the speech synthesizer 1 according to the first embodiment, the keys of the entries in the adjustment dictionaries for duration and each acoustic feature are attribute information of speech units and cluster numbers of intermediate representations. The values ​​of the entries in the adjustment dictionaries 271, 272, 273, 274, and 275 for duration and each acoustic feature are adjustment instructions.

[0052] <Processing of Speech Synthesizer 2> Figure 11 is a flowchart showing an example of the overall flow of the speech synthesis method of the second embodiment. First, the analysis unit 21 analyzes the input text and outputs attribute information for each speech unit (step S21). Next, the encoder 22 generates an intermediate representation sequence from the attribute information for each speech unit (step S22). Then, the index acquisition unit 26 acquires cluster numbers from the intermediate representation sequence (step S23).

[0053] 12 is a flowchart showing an example of the detailed flow of the cluster number acquisition process (step S23) according to the second embodiment. First, the index acquisition unit 26 acquires an intermediate expression in a sentence (step S23-1), and searches the representative vector list 261 for each cluster to find a representative vector that is closest to the vector representing the intermediate expression (step S23-2). Then, the index acquisition unit 26 acquires the number of the cluster represented by the representative vector acquired in step S23-2 (step S23-3).

[0054] When the cluster numbers have been acquired for all intermediate representations (Yes in step S23-4), the process ends. If the cluster numbers have not been acquired for all intermediate representations (No in step S23-4), the index acquisition unit 26 acquires the next intermediate representation (step S23-5) and performs the process from step S23-2.

[0055] 11 , the duration decoder 23 receives the intermediate representation sequence and generates pre-adjustment durations (step S24). Next, the adjustment unit 27 adjusts the pre-adjustment durations obtained in step S24 based on the attribute information of each speech unit and the cluster number of each intermediate representation (step S25).

[0056] 13 is a flowchart showing an example of the detailed flow of the duration adjustment process (step S25) according to the second embodiment. First, the adjustment unit 27 acquires the attribute information of the speech unit at the beginning of the sentence and the cluster number of the intermediate representation (step S25-1), and searches the duration adjustment dictionary 271 for an entry whose key (the attribute information of the speech unit and the cluster number) matches (step S25-2). If an entry is found (Yes in step S25-3), the adjustment unit 27 applies the adjustment instruction of that entry to the duration corresponding to the attribute information of the speech unit acquired in step S25 (step S25-4). If no entry is found (No in step S25-3), the process proceeds to step S25-5.

[0057] If adjustment is complete for all speech units (Yes in step S25-5), the process ends. If adjustment is not complete for all speech units (No in step S25-5), the adjustment unit 27 acquires the attribute information of the next speech unit and the cluster number of the intermediate representation (step S25-6) and starts the process from step S25-2.

[0058] 11 , the acoustic feature decoder 24 generates each pre-adjustment acoustic feature from the intermediate representation sequence based on the adjusted duration (step S26). Next, the adjustment unit 27 adjusts each acoustic feature based on each pre-adjustment acoustic feature, attribute information of each speech unit, and cluster number of each intermediate representation (step S27).

[0059] As an example, the details of the logarithmic fundamental frequency adjustment method are shown in Fig. 14. Fig. 14 is a flowchart showing a detailed example (in the case of logarithmic fundamental frequency) of the acoustic feature adjustment process (step S27) according to the second embodiment. First, the adjustment unit 27 acquires attribute information of the speech unit at the beginning of the sentence and the cluster number of the intermediate representation (step S27-1), and searches the logarithmic fundamental frequency adjustment dictionary 272 for an entry whose key (attribute information of the speech unit and cluster number) matches (step S27-2).

[0060] If an entry is found (Yes in step S27-3), the adjustment unit 27 applies the adjustment instructions of that entry to the section corresponding to the attribute information of the speech unit and the cluster number of the intermediate representation acquired in step S27-1 (step S27-4).

[0061] If adjustment is complete for all speech units (Yes in step S27-5), the process ends. If adjustment is not complete for all speech units (No in step S27-5), the adjuster 27 acquires the attribute information and the cluster number of the intermediate representation of the next speech unit (step S27-6) and starts the process from step S27-2.

[0062] The adjustment unit 27 also adjusts other acoustic features than the logarithmic fundamental frequency using the same process as in FIG.

[0063] Returning to FIG. 11 again, finally, the vocoder 25 generates a speech waveform from the adjusted acoustic feature obtained in step S27 (step S28).

[0064] <Details of Each Unit of the Speech Synthesizer 2> The list 261 of representative vectors for each cluster provided in the index acquisition unit 26 is obtained by training a clustering model in advance. For example, the clustering model is a model trained using intermediate representations obtained from each sentence used in training after training of a neural network with an encoder-decoder structure used in the speech synthesizer 2 is completed. Alternatively, for example, a clustering model trained simultaneously with the neural network with an encoder-decoder structure by applying the training method shown in Non-Patent Document 3 may be used.

[0065] The criterion for "closeness" when the index acquisition unit 26 searches for a representative vector is the distance measure used when training the clustering model. For example, if the L2 norm is used to train the clustering model, the L2 norm is also used to search for a representative vector. Furthermore, for example, if the cosine similarity is used to train the clustering model, the cosine similarity is also used to search for a representative vector.

[0066] In the second embodiment, the adjustment dictionaries 271, 272, 273, 274, and 275 for duration and each acoustic feature use, as a key, not only the attribute information of each speech unit but also the cluster number obtained when the intermediate representation is clustered, thereby making it possible to specify appropriate adjustment application conditions without specifying the attribute information of each speech unit in detail.

[0067] According to paragraph

[0089] of Patent Document 2, for example, speech-unit attribute information is complex information that must be expressed using 312-dimensional binary values ​​and 13-dimensional numerical data. Therefore, it may be difficult to appropriately set the interval to which adjustment instructions are applied using speech-unit attribute information alone. However, the intermediate representations appropriately encode the speech-unit attribute information, and the representative vectors obtained by clustering these intermediate representations are considered to retain essential information about each intermediate representation. Therefore, by using the cluster number, it is possible to specify appropriate adjustment application conditions without specifying detailed attribute information.

[0068] Furthermore, the machine learning model used in the index acquisition unit 26 may be a decision tree model that receives attribute information of speech units as input and outputs intermediate representations. In this case, attribute information of speech units is input to the index acquisition unit 26, and the arrival leaf node number of the decision tree is used as information for identifying the intermediate representation instead of the cluster number. As in the case of using a clustering model, the decision tree model is obtained by, for example, completing training of a neural network with an encoder-decoder structure used in the speech synthesizer 2, and then training the intermediate representations obtained from each sentence used in training as training data.

[0069] Effect of the Speech Synthesizer 2 of the Second Embodiment As described above, in the speech synthesizer 2 of the second embodiment, the information for identifying an intermediate representation is an index obtained by using a machine learning model that classifies the intermediate representation or classifies attribute information of speech units. This improves the interpretability of the keys in the adjustment dictionaries 271, 272, 273, 274, and 275 ( FIG. 10 ) for duration and each acoustic feature compared to using the intermediate representation as is, for example.

[0070] That is, according to the speech synthesizer 2 of the second embodiment, it is possible to specify appropriate adjustment application conditions without specifying detailed attribute information of speech units. For example, by further using the above-mentioned cluster number (an example of information specifying an intermediate representation output from a neural network) as a key, the interpretability of the key can be improved while keeping the key compact. Also, by further using the above-mentioned leaf node number (an example of information specifying an intermediate representation output from a neural network) as a key, the interpretability of the key can be improved while keeping the key compact.

[0071] Third Embodiment Next, a third embodiment will be described. In the description of the third embodiment, the same description as in the first embodiment will be omitted, and only the differences from the first embodiment will be described.

[0072] <<Outline of Speech Synthesizer 3>> Fig. 15 is a diagram showing an example of the functional configuration of the speech synthesizer 3 of the third embodiment. In the speech synthesizer 3 of the third embodiment, the adjustment unit 36 ​​adjusts the intermediate representation in addition to the duration and acoustic features. The adjustment unit 36 ​​of the third embodiment includes an adjustment dictionary 366 of the intermediate representation (Fig. 17). The adjustment dictionary 366 of the intermediate representation uses attribute information of the speech unit and the type of acoustic feature to which the adjustment instruction should be applied as keys, and uses adjustment instructions for the intermediate representation as values. By adjusting an intermediate representation that causes a pronunciation error to one that is pronounced correctly, the pronunciation error can be adjusted efficiently.

[0073] In addition, by using the type of acoustic feature to which the adjustment instruction should be applied as a key for the intermediate representation adjustment dictionary 366, it is possible to determine whether to apply the adjustment instruction of each entry for each type of acoustic feature. Since the speech synthesizer 3 generates the duration and each acoustic feature from the same intermediate representation sequence output from the encoder 32, it is possible to reduce the impact on the duration or acoustic feature that is not related to the reading error by determining whether to apply the adjustment instruction of each entry for each type of acoustic feature.

[0074] Each functional block will be described in detail below.

[0075] Like the speech synthesizer 1 of the first embodiment, the speech synthesizer 3 of the third embodiment includes an analyzer 31, an encoder 32, a duration decoder 33, an acoustic feature decoder 34, a vocoder 35, and an adjuster 36. Each unit has the same function as the speech synthesizer 1 of the first embodiment.

[0076] Fig. 16 is a diagram showing an example of the functional configuration of the acoustic feature decoder 34 according to the third embodiment. As shown in Fig. 16, the acoustic feature decoder 34 according to the third embodiment outputs the logarithmic fundamental frequency and the energy using separate neural networks 341 and 342, respectively, and outputs the mel-linear spectrum pair, the voiced / unvoiced flag, and the aperiodicity index using the same neural network 343.

[0077] In the third embodiment, the types of acoustic features are assumed to be four: duration, logarithmic fundamental frequency, energy, and spectral feature. Here, the spectral feature as a type of acoustic feature collectively refers to three acoustic features: a mel-linear spectral pair, which is a spectral feature output using the neural network 343, as well as a voiced / unvoiced flag and an aperiodicity index. Note that the types of acoustic features may include at least one of duration, logarithmic fundamental frequency, energy, and spectral feature.

[0078] 17 is a diagram showing an example of the functional configuration of the adjustment unit 36 ​​of the third embodiment. The adjustment unit 36 ​​of the third embodiment includes an adjustment dictionary 366 of intermediate representations used to adjust the intermediate representations, and adjustment dictionaries 361, 362, 363, 364, and 365 for each acoustic feature. The keys of the adjustment dictionary 366 corresponding to an intermediate representation are attribute information of speech units and the names of target acoustic features, and the values ​​are adjustment instructions for replacing the intermediate representation with a specified vector.

[0079] 18 is a flowchart showing an example of the overall flow of the speech synthesis method according to the third embodiment. First, the analysis unit 31 generates attribute information for each speech unit from the input text, and the encoder 32 generates an intermediate representation sequence from the attribute information for each speech unit (step S31).

[0080] Next, the adjustment unit 36 ​​adjusts the intermediate representation sequence input to the duration decoder 33 (step S32). The duration decoder 33 then generates durations from the intermediate representation sequence obtained in step S32, and the adjustment unit 36 ​​adjusts the durations (step S33). Note that the detailed flow of the adjustment process in step S33 is the same as in the case of the speech synthesizer 1 of the first embodiment (see FIG. 4).

[0081] Next, the adjustment unit 36 ​​adjusts the intermediate representation sequence to be input to the neural network 341, which outputs a logarithmic fundamental frequency (step S34). The neural network 341 then receives the intermediate representation sequence obtained in step S34 and outputs a logarithmic fundamental frequency, and the adjustment unit 36 ​​adjusts the logarithmic fundamental frequency (step S35). Note that the detailed flow of the adjustment process in step S5 is also the same as in the case of the speech synthesizer 1 of the first embodiment (see FIG. 6).

[0082] Next, the adjustment unit 36 ​​adjusts the intermediate representation sequence to be input to the neural network 342 that outputs energy (step S36). The neural network 342 then receives the intermediate representation sequence obtained in step S36 and outputs energy, and the adjustment unit 36 ​​adjusts the energy (step S37). Note that the detailed flow of the adjustment process in step S37 is also the same as in the case of the speech synthesizer 1 of the first embodiment.

[0083] Next, the adjustment unit 36 ​​adjusts the intermediate representation sequence to be input to the neural network 343, which outputs the spectral features (step S38). The neural network 343 then receives the intermediate representation sequence obtained in step S38 and outputs the spectral features. The adjustment unit 36 ​​adjusts the spectral features by adjusting the mel-linear spectral pairs and the aperiodicity index of the spectral features (step S39). Note that the detailed flow of the adjustment process in step S39 is also the same as that in the speech synthesizer 1 of the first embodiment.

[0084] Finally, the vocoder 35 generates a speech waveform from the adjusted acoustic features obtained in the processes up to step S39 (step S40).

[0085] Fig. 19 shows details of the method for adjusting an intermediate representation series. Fig. 19 is a flowchart showing an example of the detailed flow of the adjustment process (step S38) of an intermediate representation series input to the neural network 343 that outputs spectral features according to the third embodiment. Fig. 19 shows the process in step S38 as an example, but similar processes are also performed in step S32 where the target acoustic feature is duration, step S34 where the target acoustic feature is logarithmic fundamental frequency, and step S36 where the target acoustic feature is energy.

[0086] First, the adjustment unit 36 ​​acquires attribute information of the speech unit at the beginning of the sentence (step S38-1), and searches the adjustment dictionary 366 of the intermediate representation for an entry whose attribute information of the speech unit matches and whose target acoustic feature is a spectral feature (step S38-2). If an entry is found (Yes in step S38-3), the adjustment unit 36 ​​applies the adjustment instruction of that entry to the intermediate representation corresponding to the attribute information of the speech unit (step S38-4). If no entry is found (No in step S38-3), the process proceeds to step S38-5.

[0087] If adjustment is complete for all speech units (Yes in step S38-5), the process ends. If adjustment is not complete for all speech units (No in step S38-5), the adjuster 36 acquires attribute information for the next speech unit (step S38-6) and starts the process from step S32-2.

[0088] FIG. 20 is a diagram showing an example of the spectrum of synthesized speech when the intermediate representation sequence is adjusted by the adjustment process (step S38) of the third embodiment. The example in FIG. 20 shows an example of the spectrum of synthesized speech when the speech unit is a phoneme and the sentence "Keshikata." is input. In this case, the vowel e (e) is sandwiched between the vowel i (i) and the consonant k (ka row), so when the intermediate representation adjustment dictionary 366 is searched, the first entry in the intermediate representation adjustment dictionary 366 is found. Therefore, in step S38-4, an adjustment instruction is applied to replace the intermediate representation, which is the value of that entry, with the vector [0.45, ..., 1.0e-3]. As a result, as shown in FIG. 20, the spectrum of the portion corresponding to the vowel e (e) has changed from before the adjustment.

[0089] In this way, in the third embodiment, mispronunciations can be efficiently adjusted by adjusting intermediate representations that cause mispronunciations into intermediate representations that produce correct pronunciations. Furthermore, by using the acoustic features to which adjustment instructions should be applied as keys for the intermediate representation adjustment dictionary 366, it is possible to determine whether to apply the adjustment instructions of each entry for each type of acoustic feature. Since the speech synthesizer 3 of the third embodiment outputs each acoustic feature from the same intermediate representation sequence output from the encoder 32, it is possible to reduce the impact on acoustic features that are not related to mispronunciations by determining whether to apply the adjustment instructions of each entry for each type of acoustic feature.

[0090] <Details of Each Unit of the Speech Synthesizer 3> The adjustment instruction in the intermediate representation adjustment dictionary 366 is, for example, an operation of replacing with a specified vector. As the specified vector, for example, for a sentence containing a speech unit with the same pronunciation and having no problems, a vector indicating the intermediate representation corresponding to that speech unit is used. Regarding the method of specifying the vector, while the vector is specified directly in FIG. 17, it is also possible to create a list of replacement vectors and specify the index of that list as the replacement target. In the latter case, during adjustment, the vector at the corresponding index is read from the list of replacement vectors, and the intermediate representation of the adjustment portion is replaced.

[0091] Effect of the Speech Synthesizer 3 of the Third Embodiment As described above, according to the speech synthesizer 3 of the third embodiment, a vector indicating an intermediate representation that causes a mispronunciation is replaced with a specified vector, thereby adjusting the intermediate representation to one that is pronounced correctly. This makes it possible to adjust mispronunciations more efficiently. Furthermore, by using acoustic features to which adjustment instructions should be applied as keys for the adjustment dictionary 366 of the intermediate representation, it is possible to determine whether or not to apply the adjustment instructions for each entry for each type of acoustic feature, and it is possible to reduce the impact on acoustic features that are not related to mispronunciations.

[0092] [Supplementary Notes on Each Example] The speech synthesizers 1, 2, and 3 of the first to third embodiments are configured to include one adjustment unit 16, 27, and 36, respectively, but may be configured to include multiple adjustment units. For example, they may be configured to include an adjustment unit corresponding to each of the duration, intermediate representation, and each acoustic feature.

[0093] Furthermore, although the speech synthesizers 1, 2, and 3 have been described as having a one-to-one correspondence between each speech unit and an intermediate representation, there may also be a one-to-multiple correspondence between each speech unit and an intermediate representation. In this case, the adjustment dictionaries 161, 267, and 361 may also use the number of the intermediate representation corresponding to the speech unit as a key. In this way, for example, when there is a one-to-two correspondence between each speech unit and an intermediate representation, it is possible to specify and adjust the acoustic features and intermediate representation corresponding to, for example, the first intermediate representation.

[0094] Finally, an example of the hardware configuration of the speech synthesis device 1 (2, 3) according to the first to third embodiments will be described.

[0095] 21 is a diagram showing an example of the hardware configuration of the speech synthesizer 1 (2, 3) according to the first to third embodiments. The speech synthesizer 1 (2, 3) includes a processor 91, a main storage device 92, an auxiliary storage device 93, a display device 94, an input device 95, and a communication device 96. The processor 91, the main storage device 92, the auxiliary storage device 93, the display device 94, the input device 95, and the communication device 96 are connected via a bus 97.

[0096] The speech synthesizer 1 (2, 3) may not have some of the above components. For example, if the speech synthesizer 1 (2, 3) can use the input function and display function of an external device, the speech synthesizer 1 (2, 3) may not have the display device 94 and the input device 95.

[0097] The processor 91 executes a program read from the auxiliary storage device 93 to the main storage device 92. The main storage device 92 is memory such as a ROM and a RAM. The auxiliary storage device 93 is a hard disk drive (HDD), a memory card, or the like.

[0098] The display device 94 is, for example, a liquid crystal display. The input device 95 is an interface for operating the speech synthesizer 1 (2, 3). The display device 94 and the input device 95 may be realized by a touch panel or the like having a display function and an input function. The communication device 96 is an interface for communicating with other devices.

[0099] Furthermore, for example, the program executed by the speech synthesizer 1 (2, 3) may be stored on a computer connected to a network such as the Internet, and may be provided by being downloaded via the network.

[0100] Alternatively, for example, the program executed by the speech synthesizer 1 (2, 3) may be provided via a network such as the Internet without being downloaded. Specifically, speech synthesis processing may be performed by a so-called ASP (Application Service Provider) type service, which realizes processing functions by issuing an execution instruction and obtaining the results, without transferring the program from a server computer.

[0101] Furthermore, for example, the program for the speech synthesizer 1 (2, 3) may be provided in a state where it is pre-installed in a ROM or the like.

[0102] The program executed by the speech synthesizer 1 (2, 3) has a modular structure that includes functions that can be realized by the program among the above-mentioned functional configurations. As for each function, the processor 91 reads the program from a storage medium and executes it, and the above-mentioned functional blocks are loaded onto the main memory device 92. In other words, the above-mentioned functional blocks are generated on the main memory device 92.

[0103] Note that some or all of the above-described functions may be realized by hardware such as an IC, rather than by software.

[0104] Furthermore, each function may be realized using multiple processors 91, in which case each processor 91 may realize one of the functions, or may realize two or more of the functions.

[0105] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims.

Claims

1. A speech synthesis device comprising: an encoder that converts attribute information of speech units into an intermediate representation using a first neural network; a decoder that generates acoustic features from the intermediate representation using a second neural network; and an adjustment unit that uses an adjustment dictionary that uses at least the attribute information of the speech units as keys and adjustment instructions for the acoustic features as values ​​to determine a section to which the adjustment instructions are to be applied using the keys, and adjusts the acoustic features of the determined section based on the adjustment instructions.

2. The speech synthesis device according to claim 1, wherein the keys of the training dictionary further include information specifying an intermediate representation obtained by the first neural network.

3. The speech synthesis device of claim 2, wherein the intermediate representation is a latent representation obtained by the encoder, and information identifying the intermediate representation is an index obtained using a machine learning model that classifies the intermediate representation or attribute information of the speech unit.

4. The speech synthesis device according to claim 3, wherein the machine learning model for classifying the intermediate representation is a clustering model, and the index is a cluster number when the intermediate representation is classified by the clustering model.

5. The speech synthesis device according to claim 3, wherein the machine learning model that classifies the attribute information of the speech units is a decision tree model, and the index is a leaf node number reached when the attribute information of the speech units is input into the decision tree model.

6. The speech synthesis device according to claim 1, wherein the adjustment instruction is an instruction to replace a vector representing the intermediate representation with a specified vector.

7. The speech synthesis device according to claim 6, wherein the instruction to replace with the specified vector is determined for each type of the acoustic feature.

8. The speech synthesis device according to claim 7, wherein the types of the acoustic features include at least one of duration, logarithmic fundamental frequency, energy, and spectral features of the speech unit.

9. The speech synthesis device according to claim 1, wherein the acoustic feature is a logarithmic fundamental frequency, and the adjustment instruction is an instruction to apply a specified operation to the logarithmic fundamental frequency.

10. The speech synthesis device according to claim 1, wherein the acoustic feature is a duration of the speech unit, and the adjustment instruction is an instruction to apply a specified operation to the duration.

11. The speech synthesis device according to claim 1, wherein the acoustic feature is energy, and the adjustment instruction is an instruction to apply a specified operation to the energy.

12. The speech synthesis device according to claim 1, wherein the acoustic features are spectral features, and the adjustment instruction is an instruction to apply a specified operation to the spectral features.

13. The speech synthesis device according to claim 1, wherein the acoustic feature is an aperiodicity index, and the adjustment instruction is an instruction to apply a specified operation to the aperiodicity index.

14. A speech synthesis method comprising: a step in which a speech synthesis device converts attribute information of speech units into an intermediate representation using a first neural network; a step in which the speech synthesis device generates acoustic features from the intermediate representation using a second neural network; and a step in which the speech synthesis device uses an adjustment dictionary that uses at least the attribute information of the speech units as keys and adjustment instructions for the acoustic features as values ​​to determine a section to which the adjustment instructions are to be applied using the keys, and adjusts the acoustic features of the determined section based on the adjustment instructions.

15. A program for causing a computer to function as: an encoder that converts attribute information of speech units into an intermediate representation using a first neural network; a decoder that generates acoustic features from the intermediate representation using a second neural network; and an adjustment unit that uses an adjustment dictionary that uses at least the attribute information of the speech units as keys and adjustment instructions for the acoustic features as values ​​to determine a section to which the adjustment instructions are to be applied using the keys, and adjusts the acoustic features of the determined section based on the adjustment instructions.

Images