Signal processing device and program

Abstract

To provide a signal processing device capable of constructing acoustic feature quantity estimation models for outputting high-quality synthetic speech.SOLUTION: An acoustic analysis unit obtains an acoustic feature quantity by performing acoustic analysis processing on speech signals. A linguistic analysis unit performs linguistic analysis processing on a text to obtain input labels that include information on Japanese reading syllabaries of the text and information on prosody corresponding to the text. An acoustic feature quantity estimation model learning unit executes machine learning of an acoustic feature quantity estimation model to obtain an acoustic feature quantity corresponding to input labels that include information on Japanese reading syllabaries and information on prosody, using the input labels output by the linguistic analysis unit and the acoustic feature quantity output by the acoustic analysis unit as data for learning. The linguistic analysis unit obtains a symbol representing relative sound pitch for a phrase in the text with respect to other adjacent phrases, based on a pitch obtained by the acoustic analysis unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a signal processing device and a program. [Background technology]

[0002] There is a conventional technique for constructing a statistical model for both text and its corresponding speech signal based on the text and its corresponding speech signal, and then using the constructed statistical model to obtain synthetic speech corresponding to any text.

[0003] For example, Patent Document 1 describes a method for constructing and utilizing a model relating to text and speech signals using deep learning (DNN, Deep Neural Network). In the method described in Patent Document 1, text and corresponding speech signals are prepared in advance. Input labels consisting of pronunciations and prosodic symbols are obtained for the text through linguistic analysis processing, and are manually corrected as appropriate to match the speech signals. Acoustic features are calculated for the speech signals through speech analysis processing. These input labels and acoustic features are then used to train an acoustic feature estimation model implemented by a neural network. Once this acoustic feature estimation model is constructed, acoustic features based on the acoustic feature estimation model can be estimated for any input text. That is, input labels corresponding to the input text are obtained by performing linguistic analysis processing on any input text. Acoustic features are estimated from these input labels and the acoustic feature estimation model. Once the acoustic features are estimated, speech signals corresponding to the acoustic features can be obtained through speech signal generation processing. That is, a synthetic speech signal for any text can be obtained.

[0004] Patent Document 1 describes the use of pronunciation and prosodic symbols as input labels corresponding to Japanese text expressed as a mixture of kanji and kana. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2020-034883 Summary of the Invention [Problem to be solved by the invention]

[0006] The above-mentioned conventional techniques have problems to be solved. The method described in Patent Document 1 describes that the prosodic symbols used as input labels include any one of a symbol specifying an accent position, a symbol specifying a boundary between phrases or sentences, a symbol specifying an intonation at the end of a sentence, and a symbol specifying a pause length.

[0007] (1) The prosodic symbols in the technology described in Patent Document 1 do not contain information about the relative pitch between adjacent phrases. In other words, such prosodic symbols have the problem that, when synthesizing speech, it is not possible to control the relative pitch between adjacent phrases.

[0008] (2) In the embodiment of Patent Document 1, it is stated that the symbols for specifying the division of phrases or phrases "may not be used." However, if the division of phrases or phrases is not specified, there is a problem that the correct accent cannot be reproduced.

[0009] (3) Patent Document 1 particularly describes that the prosodic symbol "$%" representing a pause is used to specify a pause. However, the technology described in Patent Document 1 uses only one type of prosodic symbol representing a pause. In other words, there is a problem in that this prosodic symbol cannot represent differences in pause length.

[0010] (4) Patent Document 1 particularly describes an embodiment in which prosodic symbols representing accent-raising positions and accent-falling positions are used as symbols for specifying the above-mentioned accent positions. However, when standard Japanese (also known as the Tokyo dialect) is used as a given, the accent-raising positions are automatically determined. In other words, using prosodic symbols representing accent-raising positions increases the redundancy of input labels. When an acoustic feature estimation model is trained using such redundant input labels, there is a problem in that the estimation accuracy of acoustic features decreases.

[0011] (5) Patent Document 1 describes an embodiment in which katakana is used as the reading kana used as input labels. However, the problem is that the set of all katakana characters contains too many symbols. In other words, when katakana is used, the representation space of adjacent characters is relatively sparse, which causes a problem in that the estimation accuracy of an acoustic feature estimation model trained using such input labels is low.

[0012] As described above, the conventional techniques have had the problem that when synthesizing speech based on arbitrary text, it is not possible to obtain synthesized speech based on detailed control.

[0013] The present invention has been made based on the above-mentioned problem recognition, and aims to provide a signal processing device and a program capable of constructing an acoustic feature estimation model for outputting high-quality synthetic speech, and a signal processing device and a program capable of estimating acoustic features for generating high-quality synthetic speech using such an acoustic feature estimation model. [Means for solving the problem]

[0014] [1] In order to solve the above problem, a signal processing device according to one aspect of the present invention includes: a training speech corpus storage unit that stores a training speech corpus including text and a speech signal corresponding to the text; an acoustic analysis unit that calculates acoustic features by performing acoustic analysis processing on the speech signal; a language analysis unit that calculates input labels including pronunciation information of the text and prosody information corresponding to the text by performing language analysis processing on the text; and an acoustic feature estimation model training unit that performs machine learning of an acoustic feature estimation model to calculate acoustic features corresponding to the input labels including pronunciation information and prosody information, using the input labels output by the language analysis unit and the acoustic features output by the acoustic analysis unit as training data, wherein the acoustic analysis unit calculates a pitch based on the speech signal, and the language analysis unit calculates a symbol representing the pitch of a phrase included in the text relative to adjacent phrases based on the pitch calculated by the acoustic analysis unit, and adds the symbol representing the relative pitch to the symbol representing the phrase boundary, and includes the symbol representing the relative pitch in the input label.

[0015] [2] In another aspect of the present invention, in the signal processing device described above, the acoustic analysis unit determines durations of pauses included in the speech signal by determining sections of phonemes included in the speech signal based on the speech signal, and the language analysis unit includes, in the input label, prosodic symbols for distinguishing between multiple types of pause durations based on the durations of the pauses determined by the acoustic analysis unit.

[0016] [3] In addition, according to one aspect of the present invention, in the signal processing device described above, the language analysis unit determines the input label that does not include a symbol indicating an accent rise position as prosodic information.

[0017] [4] In another aspect of the present invention, in the signal processing device described above, the language analysis unit obtains the input label as the reading information using a string of symbols representing phonemes.

[0018] [5] In addition, one aspect of the present invention is a signal processing device further comprising: a speech signal generation model training unit that performs machine learning of a speech signal generation model to obtain speech signals corresponding to acoustic features, using the acoustic features output by the acoustic analysis unit and the speech signals included in the training speech corpus as training data.

[0019] [6] Furthermore, a signal processing device according to one aspect of the present invention includes a language analysis unit that performs language analysis processing on an input text to determine an input label including information on the pronunciation of the text and information on prosody corresponding to the text, and an acoustic feature estimation unit that estimates acoustic features by inputting the input label determined by the language analysis unit into an acoustic feature estimation model trained by any one of the signal processing devices described above in [1] to [4].

[0020] [7] In addition, one aspect of the present invention is the above-mentioned signal processing device, further comprising an audio signal generation unit that generates an audio signal by inputting the acoustic features output by the acoustic feature estimation unit to an audio signal generation model trained by the signal processing device of [5] above.

[0021] [8] Another aspect of the present invention is a program for causing a computer to function as the signal processing device described above in [1] to [7]. [Effects of the Invention]

[0022] According to the present invention, it is possible to construct an acoustic feature estimation model for reproducing the relative pitch difference between adjacent phrases with high quality, and this acoustic feature estimation model can be used to estimate high-quality acoustic features. [Brief explanation of the drawings]

[0023] [Figure 1] 1 is a block diagram showing a schematic functional configuration of a signal processing device according to an embodiment of the present invention. [Figure 2]FIG. 10 is a schematic diagram (1 / 3) showing the format of a full-context label used by the accent estimation device according to the embodiment. [Figure 3] FIG. 2 is a schematic diagram (2 / 3) showing the format of a full-context label used by the accent estimation device according to the embodiment. [Figure 4] FIG. 3 is a schematic diagram (3 / 3) showing the format of a full-context label used by the accent estimation device according to the embodiment. [Figure 5] FIG. 10 is a schematic diagram (1 / 2) showing an example of a full-context label extracted by performing linguistic analysis processing on text in the embodiment. [Figure 6] FIG. 2 is a schematic diagram (2 / 2) showing an example of a full-context label extracted by performing linguistic analysis processing on text in the embodiment. [Figure 7] FIG. 2 is a schematic diagram showing a list of symbols used in input labels (reading kana and prosodic symbols) in the embodiment. [Figure 8] FIG. 2 is a block diagram showing a schematic functional configuration for a model learning device according to the embodiment to obtain an input label based on text and a speech signal. [Figure 9] 2 is a block diagram showing an example of the internal configuration of a signal processing device (or at least a part thereof) according to the embodiment. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0024] Next, an embodiment of the present invention will be described with reference to the drawings.

[0025] In this embodiment, text and a corresponding speech signal are prepared in advance. A linguistic analysis process is performed on this text to obtain an input label consisting of a phoneme sequence corresponding to the pronunciation and a prosodic symbol. Furthermore, a speech analysis process is performed on the speech signal to calculate acoustic features. Using these input labels and acoustic features, machine learning is performed on an acoustic feature estimation model for estimating acoustic features from the input labels. Meanwhile, using the acoustic features and the speech signal, machine learning is performed on a speech signal generation model for estimating speech signals from the acoustic features. Using these trained models, speech can be synthesized from any text. During synthesis, any text is input, and input labels are obtained by performing linguistic analysis on the text. Then, acoustic features are estimated using the input label and a pre-trained acoustic feature estimation model. Furthermore, a speech signal can be generated using the acoustic features obtained as a result of estimation and a pre-trained speech signal generation model. In other words, a synthetic speech signal corresponding to any text can be obtained.

[0026] The features of this embodiment are as follows. Note that only some of the features listed here may be implemented. One feature is that a phoneme symbol string is used as the pronunciation corresponding to the text. Another feature is that symbols representing accent rise positions are not used as prosodic symbols. Another feature is that symbols representing phrase delimiters, accent phrase delimiters, and sub-accent phrase delimiters are used as prosodic symbols, with symbols representing information about the relative pitch of adjacent phrases added. Another feature is that symbols representing information that can distinguish between multiple different pause lengths are used as prosodic symbols.

[0027] FIG. 1 is a block diagram showing a schematic functional configuration of a signal processing device according to this embodiment. As shown in the figure, the signal processing device 1 includes a speech synthesizer 2, a model learning device 3, an acoustic feature estimation model storage unit 14, and a speech signal generation model storage unit 15. The speech synthesizer 2 and the model learning device 3 can be implemented, for example, by a computer and a program. At least some of the functions of each device may be implemented as a dedicated electronic circuit rather than a program. Each device has a storage unit as needed. The acoustic feature estimation model storage unit 14 and the speech signal generation model storage unit 15 are implemented using a storage unit. The storage unit may store, for example, program variables, memory allocated by program execution, or data managed by a database management system. Non-volatile storage units such as a magnetic hard disk drive or a solid-state drive (SSD) may also be used as needed.

[0028] The speech synthesizer 2 generates synthetic speech based on unknown text using a trained acoustic feature estimation model and a speech signal generation model. The speech synthesizer 2 is also called a "signal processing device." Specifically, the speech synthesizer 2 reads and uses parameters of each model stored in the acoustic feature estimation model storage unit 14 and the speech signal generation model storage unit 15. A more detailed functional configuration of the speech synthesizer 2 will be described later.

[0029] The model learning device 3 performs machine learning of an acoustic feature estimation model and a speech signal generation model based on a training speech corpus. The model learning device 3 is also called a "signal processing device." As a result of the machine learning of each model, the model learning device 3 updates the parameters of each model stored in the acoustic feature estimation model storage unit 14 and the speech signal generation model storage unit 15. A more detailed functional configuration of the model learning device 3 will be described later.

[0030] The acoustic feature estimation model storage unit 14 stores internal parameters of the acoustic feature estimation model. The acoustic feature estimation model represents a statistical relationship between an input label based on text and acoustic features. The model parameters stored in the acoustic feature estimation model storage unit 14 are updated by a learning process performed by an acoustic feature estimation model learning unit 34, which will be described later. The model parameters stored in the acoustic feature estimation model storage unit 14 are also referenced during acoustic feature estimation processing performed by an acoustic feature estimation unit 24, which will be described later.

[0031] As acoustic feature estimation models, Tacotron2, TransformerTTS, FastSpeech, FastSpeech2, etc. can be used.

[0032] The speech signal generation model storage unit 15 stores internal parameters of the speech signal generation model. The speech signal generation model represents the statistical relationship between acoustic features and speech signals. The model parameters stored in the speech signal generation model storage unit 15 are updated by a learning process performed by a speech signal generation model learning unit 35, which will be described later. In addition, the model parameters stored in the speech signal generation model storage unit 15 are referenced during speech signal generation processing performed by a speech signal generation unit 25, which will be described later.

[0033] As a voice signal generation model, ParallelWaveGAN, MelGAN, FullBandMelGAN, MultiBandMelGAN, etc. can be used.

[0034] 1, the model training device 3 includes a training speech corpus storage unit 31, a language analysis unit 32, an acoustic analysis unit 33, an acoustic feature estimation model training unit 34, and a speech signal generation model training unit 35. The function of each unit is as follows.

[0035] The training speech corpus storage unit 31 stores a corpus to be used as training data. The training speech corpus is configured as a collection of pairs of text, including sentences, and speech signals corresponding to the text. The training speech corpus includes a large number of pairs. Here, the text is, for example, Japanese sentence text, but it may also be text of sentences in other languages. The speech signal may be, for example, monaural speech sampled at a sampling frequency of 22.050 kilohertz (kHz) and a quantization bit rate of 16 bits. However, speech signals of other forms (number of channels, sampling frequency, quantization bit rate) may also be used. Furthermore, analog speech signals may also be used. Note that the signal processing device 1 uses the entire training speech corpus in proportions, such as 80% for training a statistical model, 10% for parameter adjustment, and 10% for evaluation. The numerical proportions here may themselves be other values.

[0036] The language analysis unit 32 performs language analysis on each pair of text and speech signal contained in the speech corpus. Specifically, the language analysis unit 32 extracts full-context labels (context-dependent phoneme labels) through text language analysis processing. Full-context labels are context-dependent labels that contain information about phonemes and prosody. The extraction of full-context labels can be performed using existing technology. The language analysis unit 32 can be implemented, for example, using the language analysis processing function included in the speech synthesis processing toolkit of Open JTalk version 1.11 (released December 25, 2018). As of May 31, 2021, Open JTalk version 1.11 is available at the following URL: URL http: / / open-jtalk.sourceforge.net

[0037] The full context label contains information such as the two phonemes before and after the phoneme in question, part-of-speech information, the breath group position within the sentence, the accent phrase position within the breath group, and the mora position within the accent phrase. Using the part-of-speech information, breath group position information, and accent phrase position information contained in the full context label, it is possible to determine whether the current accent phrase division corresponds to a phrase division, accent phrase division, or sub-accent phrase division. Specific examples of full context labels will be explained in more detail later with reference to figures.

[0038] The language analysis unit 32 creates an input label based on the full context label and the pause length information and pitch information received from the acoustic analysis unit 33. In other words, the language analysis unit 32 performs a language analysis process on the text to obtain an input label that includes information on the pronunciation of the text and information on the prosody corresponding to the text.

[0039] The language analysis unit 32 also determines a symbol representing the relative pitch of a phrase included in the text relative to other adjacent phrases, based on the pitch determined by the acoustic analysis unit 33 (described later). The language analysis unit 32 adds the symbol representing the relative pitch to the symbol representing the phrase boundary, and includes the resulting symbol in the input label. Furthermore, the language analysis unit 32, as described later, includes in the input label a prosodic symbol for distinguishing between multiple types of pause duration, according to the pause duration in the speech determined by the acoustic analysis unit 33.

[0040] The language analysis unit 32 also determines an input label that does not include a symbol representing an accent rise position as prosodic information. The language analysis unit 32 also determines an input label that is a string of symbols representing phonemes as information on the pronunciation of the text, rather than a string of katakana or hiragana.

[0041] The acoustic analysis unit 33 performs acoustic analysis on each pair of text and speech signals contained in the speech corpus. Specifically, the acoustic analysis unit 33 obtains acoustic features by performing acoustic analysis processing on the speech signals. The acoustic analysis unit 33 also performs phoneme segmentation based on the central phonemes of the speech signals contained in the speech corpus and their full context labels. The phoneme segmentation process itself can be performed using existing technology. The acoustic analysis unit 33 can be implemented, for example, using the forced alignment technology included in the Hidden Markov Model Toolkit (HTK). As of May 31, 2021, the Hidden Markov Model Toolkit is available at the following URL: URL http: / / htk.eng.cam.ac.uk

[0042] Note that the phoneme segmentation by forced alignment may be manually checked or corrected. Through the phoneme segmentation process, the acoustic analysis unit 33 obtains phoneme section information for each phoneme in the speech signal. The phoneme section information is information expressed by the start position (absolute time) and end position (absolute time) of the phoneme. Furthermore, the acoustic analysis unit 33 calculates the duration of the pause t based on the phoneme section information for each pause. pau (unit: milliseconds (msec)). That is, the acoustic analysis unit 33 calculates the duration of a pause included in the audio signal by determining the section of the phoneme included in the audio signal based on the audio signal. The acoustic analysis unit 33 calculates this duration t pau The information on the time length t is passed to the language analysis unit 32. pau The values ​​of are classified by a threshold, and an input label according to the duration is assigned to each pose. pau If <100, the input label of the pose is set to "_0". Also, 100≦t pau If t<200, the input label of the pose is set to "_1". Also, if t<200, the input label of the pose is set to "_1". pauIf <500, the input label of the pose is set to "_2". Also, 500≦t pau If t<1000, the input label of the pose is "_3". pau In this case, the input label of the pause is set to "_4". Note that the language analysis unit 32 may determine the input label according to these pause lengths instead of the acoustic analysis unit 33. Note that the range of time length may be other than the values ​​exemplified here.

[0043] The acoustic analysis unit 33 extracts pitch on a frame-by-frame basis through acoustic analysis of the audio signal. For example, a frame may have a window width of 50 milliseconds and a frame shift width of 10 milliseconds. The frame window width and shift width may be values ​​other than those exemplified here. The pitch extraction process itself can be performed using existing technology. The pitch extraction function of the acoustic analysis unit 33 can be realized using, for example, REAPER (Robust Epoch And Pitch EstimatoR) technology. As of May 31, 2021, REAPER is available at the following URL: URL https: / / github.com / dtalkin / REAPER

[0044] For sections where pitch cannot be extracted, such as silent sections or voiceless sections, the acoustic analysis unit 33 interpolates or extrapolates based on the pitch value in an adjacent section where pitch extraction is possible.

[0045] The acoustic analysis unit 33 also obtains acoustic features of the audio signal, which may be, for example, mel spectrograms on a frame-by-frame basis.

[0046] The acoustic feature estimation model training unit 34 trains the acoustic feature estimation model. The acoustic feature estimation model training unit 34 uses, as training data, pairs of character strings consisting of pronunciations (phoneme label sequences) and prosodic symbols, and acoustic features (frame-by-frame mel spectrograms) calculated from a speech signal. This training data is obtained through the processing of the language analysis unit 32 and the acoustic analysis unit 33 described above. The acoustic feature estimation model training unit 34 inputs character strings consisting of pronunciations (phoneme label sequences) and prosodic symbols into the acoustic feature estimation model. The acoustic feature estimation model training unit 34 calculates acoustic features, which are outputs corresponding to these inputs, using the acoustic feature estimation model. The acoustic feature estimation model training unit 34 calculates the difference (loss) between the calculated acoustic features and the acoustic features included in the training data, and adjusts the internal parameters of the acoustic feature estimation model based on this difference. In other words, the acoustic feature estimation model training unit 34 updates the internal parameters stored in the acoustic feature estimation model storage unit 14. The acoustic feature estimation model learning unit 34 uses a sufficient amount of learning data to learn the acoustic feature estimation model.

[0047] In other words, the acoustic feature estimation model learning unit 34 performs machine learning of the acoustic feature estimation model to obtain acoustic features corresponding to input labels including pronunciation information and prosody information, using the input labels output by the language analysis unit 32 and the acoustic features output by the acoustic analysis unit 33 as learning data.

[0048] The acoustic feature estimation model can be trained using, for example, the ESPnet technology, which is available at the following URL: URL https: / / github.com / espnet / espnet

[0049] The speech signal generation model training unit 35 trains the speech signal generation model. The speech signal generation model training unit 35 uses pairs of acoustic features (frame-by-frame mel spectrograms) calculated from speech signals and speech signals included in the speech corpus as training data. The acoustic features are obtained through the processing of the acoustic analysis unit 33 described above. The speech signals are the speech signals themselves stored in the training speech corpus storage unit 31. The speech signal generation model training unit 35 inputs the acoustic features for each frame into the speech signal generation model. The speech signal generation model training unit 35 calculates a speech signal, which is an output corresponding to this input, using the speech signal generation model. The speech signal generation model training unit 35 calculates the difference between the calculated speech signal and the speech signal included in the training data, and adjusts the internal parameters of the speech signal generation model based on this difference. In other words, the speech signal generation model training unit 35 updates the internal parameters stored in the speech signal generation model storage unit 15. The voice signal generation model learning unit 35 uses a sufficient amount of learning data to learn the voice signal generation model.

[0050] In other words, the speech signal generation model learning unit 35 performs machine learning of the speech signal generation model to obtain speech signals corresponding to acoustic features, using the acoustic features output by the acoustic analysis unit 33 and the speech signals included in the training speech corpus as training data.

[0051] The speech signal generation model can be trained using, for example, Parallel WaveGAN technology. Parallel WaveGAN is available at the following URL, for example: URL https: / / github.com / kan-bayashi / ParallelWaveGAN

[0052] Next, we will explain the functions for processing to synthesize speech based on the trained model. As shown in Fig. 1, the speech synthesis device 2 includes an input text storage unit 21, a language analysis unit 22, an acoustic feature estimation unit 24, a speech signal generation unit 25, and a synthetic speech output unit 26. The functions of each unit are as follows.

[0053] The input text storage unit 21 at least temporarily stores input text. This text may be unknown text. The speech synthesizer 2 synthesizes speech corresponding to this text through the process described below.

[0054] The language analysis unit 22 performs a language analysis of the input text and obtains input labels to be input to the acoustic feature estimation model. The input labels are as described in the description of the model learning device 3. Specifically, the language analysis unit 22 obtains the morphemes that make up a phrase by performing morpheme analysis and dependency analysis while referring to dictionary data. The language analysis unit 22 also obtains the dependency structure of the phrases that make up a sentence. The language analysis unit 22 obtains the pronunciation and accent from the phrases and morphemes. The language analysis unit 22 obtains information on the pause length between adjacent phrases and the relative height between phrases based on the dependency structure. The language analysis unit 22 uses this information to create an input label. The language analysis unit 22 passes the created input label to the acoustic feature estimation unit 24.

[0055] The language analysis unit 22 can be realized using existing technology. To find the pause length between adjacent phrases, the language analysis unit 22 can use, for example, the method described in the following reference document. [References] Sato Naoko et al., Pause insertion processing using dependency analysis results in text-to-speech synthesis, Natural Language Processing, 1999, Vol. 6, No. 2, pp. 117-132, URL https: / / www.jstage.jst.go.jp / article / jnlp1994 / 6 / 2 / 6_2_117 / _pdf / -char / ja

[0056] Furthermore, the language analysis unit 22 can use, for example, a prosody estimation method, which is an existing technique, to determine the relative pitch between phrases.

[0057] Furthermore, the language analysis unit 22 can use, for example, MeCab (Yet Another Part-of-Speech and Morphological Analyzer) as a morphological analysis processing technique. Furthermore, the language analysis unit 22 can use, for example, CaboCha (Yet Another Japanese Dependency Structure Analyzer) as a dependency analysis processing technique. Furthermore, the language analysis unit 22 can use, for example, UniDic as dictionary data. MeCab, CaboCha, and UniDic listed here are each provided at the following URLs. URL http: / / taku910.github.io / mecab / URL https: / / taku910.github.io / cabocha / URL https: / / unidic.ninjal.ac.jp /

[0058] Furthermore, similar to the language analysis unit 32, the language analysis unit 22 can be realized using, for example, the language analysis processing function included in the speech synthesis processing toolkit of Open JTalk version 1.11 (released on December 25, 2018).

[0059] That is, the language analysis unit 22 performs a language analysis process on the input text to obtain an input label including information on the pronunciation of the text and information on the prosody corresponding to the text. This input label is data to be input to the acoustic feature estimation model.

[0060] The acoustic feature estimation unit 24 calculates (estimates) acoustic features by inputting the input labels (prosodic readings and prosodic symbols corresponding to the input text) obtained by the language analysis unit 22 into a trained acoustic feature estimation model. The acoustic feature estimation unit 24 passes the estimated acoustic features to the speech signal generation unit 25.

[0061] The speech signal generation unit 25 generates a speech signal by inputting the acoustic feature output by the acoustic feature estimation unit 24 into a trained speech signal generation model. The speech signal generation unit 25 passes the generated speech signal to the synthetic speech output unit 26.

[0062] The synthesized speech output unit 26 can output the synthesized speech passed from the speech signal generation unit 25 to the outside.

[0063] As described above, the speech synthesizer 2 can generate and output synthetic speech corresponding to an input text using a trained acoustic feature estimation model and speech signal generation model.

[0064] 2, 3, and 4 are schematic diagrams showing examples of the format of full context labels (context-dependent phoneme labels) used by the signal processing device 1. A column of full context labels is an input label that is input to a speech feature estimation model. A full context label is a phoneme label that depends on the context of a text. FIGS. 2 to 4 show one data structure of a full context label. As shown in FIGS. 2, 3, and 4, a full context label includes the data listed below.

[0065] The following p1 to p5 are phoneme information. In this way, the full context label includes identification information of phonemes within a range of ±2 of the current phoneme. p1: Identification information of the phoneme before the preceding phoneme p2: Pre-phoneme identification information p3: Identification information of the current phoneme (central phoneme from p1 to p5) p4: next phoneme identification information p5: Identification information of the phoneme following the next phoneme

[0066] A1 to A3 below are accent information. a1: The difference between the accent type (the mora position of the accent nucleus) and the position of the current mora identification information a2: Position of the current mora identification information in the current accent phrase (forward) a3: Position of current mora identification information in the current accent phrase (backward)

[0067] The following b1 to b3, c1 to c3, and d1 to d3 are part of speech information. b1: Part of speech of the previous word b2: Conjugated form of the previous word b3: Conjugation of the previous word

[0068] c1: Part of speech of the current word c2: Present conjugation forms c3: Present word conjugation type

[0069] d1: Part of speech of the next word d2: Conjugation of the following word d3: Conjugation type of the next word

[0070] The following e1 to e5, f1 to f8, and g1 to g5 are accent information. e1: number of moras in the previous accent phrase e2: Accent type in the previous accent phrase (mora position of the accent nucleus) e3: Whether the preceding accent phrase is an interrogative word or not e4:Undefined context e5: Whether a pause is inserted between the previous accent phrase and the current accent phrase

[0071] f1: Number of moras in the current accent phrase f2: Accent type in the current accent phrase (mora position of the accent nucleus) f3: Whether the current accent phrase is an interrogative word or not f4:Undefined context f5: Position of the current accent phrase identification information in the current breath paragraph (forward) f6: Position of the current accent phrase identification information in the current breath paragraph (backward) f7: Position of the current accent phrase identifier in the current breath sequence in mora units (forward) f8: Position of the current accent phrase identifier in the current breath sequence in mora units (backward)

[0072] g1: Number of moras in the next accent phrase g2: Accent type in the following accent phrase (mora position of the accent nucleus) g3: Is the following accent phrase an interrogative word? g4:Undefined context g5: Whether a pause is inserted between the next accent phrase and the current accent phrase

[0073] The following h1 to h2, i1 to i8, and j1 to j2 are breath-pan information. h1: Number of accent phrases in the previous breath h2: Number of moras in the previous breath

[0074] i1: Number of accent phrases in the current breath group i2: Number of moras in the current breath i3: Position of the current breath section identification information in breath section units (forward) i4: Position of the current breath section identification information in breath section units (backward) i5: Position of the accent phrase unit of the current breath paragraph identification information (forward) i6: Position of the accent phrase unit of the current breath paragraph identification information (backward) i7: Current breath paragraph identification information mora unit position (forward) i8: Current breath paragraph identification information mora unit position (backward)

[0075] j1: Number of accent phrases in the next breath j2: Number of moras in the next breath

[0076] The following k1 to k3 are total number information. k1: Total number of breaths in this utterance k2: Total number of accent phrases in this utterance k3: Total number of moras in this utterance

[0077] 5 and 6 are schematic diagrams showing examples of full-context labels extracted by the language analysis unit 32 or 22 through the language analysis of text. This example is a full-context label based on the text "Sunny, then cloudy." In this diagram, one line of full-context label data corresponds to one phoneme. Note that the data in each line is folded over in the diagram. FIG. 5 shows lines 1 to 8 of the data. FIG. 6 shows lines 9 to 15 of the data. The data in each line has the structure described in FIGS. 2, 3, and 4. The beginning of each line corresponds to items p1 to p5 described above. The portion beginning with " / A:" in each line corresponds to items a1 to a3 described above. The portion beginning with " / B:" in each line corresponds to items b1 to b3 described above. The portion beginning with " / C:" in each line corresponds to items c1 to c3 described above. Furthermore, the part of each line beginning with " / D:" corresponds to items d1 through d3 mentioned above. Furthermore, the part of each line beginning with " / E:" corresponds to items e1 through e5 mentioned above. Furthermore, the part of each line beginning with " / F:" ​​corresponds to items f1 through f8 mentioned above. Furthermore, the part of each line beginning with " / G:" corresponds to items g1 through g5 mentioned above. Furthermore, the part of each line beginning with " / H:" corresponds to items h1 through h2 mentioned above. Furthermore, the part of each line beginning with " / I:" corresponds to items i1 through i8 mentioned above. Furthermore, the part of each line beginning with " / J:" corresponds to items j1 through j2 mentioned above. Furthermore, the part of each line beginning with " / K:" corresponds to items k1 through k3 mentioned above.

[0078] That is, the full context label includes phoneme information, accent information, part of speech information, accent phrase information, breath group information, and total number information.

[0079] The language analysis unit 32 in the model learning device 3 performs language analysis on a certain text to obtain a full-context label such as those illustrated in FIGS. 5 and 6. The language analysis unit 32 also identifies the phonemes that make up each accent phrase included in the obtained full-context label. The location of each accent phrase in the speech signal can be identified based on the phoneme section information of each phoneme. The language analysis unit 32 calculates the average pitch of each accent phrase based on the pitch extracted (or interpolated) from the speech signal frame by frame and the location of each accent phrase in the speech signal. The language analysis unit 32 then compares the average pitch of a certain accent phrase with the average pitch of the accent phrase following that accent phrase to determine a label representing the relative pitch of the accent phrase as follows: If the average pitch of the next accent phrase is lower than that of the accent phrase, the relative pitch is set to "0." If the average pitch of the accent phrase and the next accent phrase are the same, the relative pitch is set to "1." If the average pitch of the next accent phrase is higher than that of the accent phrase, the relative pitch is set to "2."

[0080] When comparing the average pitch between accent phrases, the determination may be made based on whether the difference in average pitch exceeds a predetermined threshold (for example, ±10% of the average pitch value). That is, the language analysis unit 32 may make the determination as follows: If the average pitch of the next accent phrase is less than 90% of the average pitch of the current accent phrase, the relative pitch is set to "0." If the average pitch of the next accent phrase is 90% or more and 110% or less of the average pitch of the current accent phrase, the relative pitch is set to "1." If the average pitch of the next accent phrase is more than 110% of the average pitch of the current accent phrase, the relative pitch is set to "2."

[0081] As described above, the language analysis unit 32 can add, as a symbol, information about the relative pitch of a specific accent phrase with respect to adjacent accent phrases.

[0082] 7 is a schematic diagram showing a list of symbols used in the input label (reading kana and prosodic symbols) in this embodiment. As shown in the figure, the input label includes the following types: vowel, mora, geminate, consonant, accent nucleus, phrase boundary, relative pitch between phrases, pause, and end of sentence.

[0083] As shown in Fig. 7, the input labels of this embodiment use phoneme symbols (vowels, mora nasal sounds, double consonants, and consonants) as pronunciations, rather than Japanese hiragana or katakana. As a result, the input labels of this embodiment use a relatively smaller number of symbols than when katakana or the like is used. In other words, the input labels of this embodiment represent information in a denser representation space than when katakana or the like is used. Therefore, by using such input labels to train an acoustic feature estimation model, the estimation accuracy of acoustic features is improved.

[0084] Furthermore, the input label of this embodiment does not have a prosodic symbol indicating the position of an accent rise. In other words, since the input label of this embodiment does not have a prosodic symbol indicating the position of an accent rise, there is no (or relatively little) redundancy in the symbols indicating the position of the accent in the input label. In other words, with less redundancy, the acoustic feature estimation model can have information on the relationship between the prosodic symbol and the acoustic feature.

[0085] In addition, the input label of this embodiment always includes symbols that specify phrase boundaries (phrase boundaries, accent phrase boundaries, and subaccent boundaries). By using such input labels, it becomes possible to reproduce the correct accent when estimating acoustic features and synthesizing speech. Furthermore, the input label of this embodiment includes symbols (0, 1, 2) that represent the relative pitch of sounds between adjacent phrases. By using such input labels, it becomes possible to control the relative pitch of sounds between adjacent phrases when estimating acoustic features and synthesizing speech.

[0086] In addition, the input labels of this embodiment use multiple symbols (_0, _1, _2, _3, _4) corresponding to different pause lengths. By using such input labels, it becomes possible to represent different pause lengths when estimating acoustic features and synthesizing speech, which was not possible with conventional techniques.

[0087] 8 is a block diagram showing a schematic functional configuration for the model learning device 3 to determine an input label based on text and a speech signal. The language analysis unit 32 and acoustic analysis unit 33 shown here are as described with reference to FIG. 1. As shown in the figure, the language analysis unit 32 includes a full-context label extraction unit 321 and an input label creation unit 322.

[0088] The language analysis unit 32 is passed text and the analysis result by the acoustic analysis unit 33. The text is one piece of text included in the training speech corpus. The analysis result by the acoustic analysis unit 33 is the result of analyzing the speech signal corresponding to the text. Pause length information and pitch information on a frame-by-frame basis are passed from the acoustic analysis unit 33 to the language analysis unit 32. A full-context label extraction unit 321 in the language analysis unit 32 acquires the text. Furthermore, an input label creation unit 322 acquires the pause information and pitch information.

[0089] The full-context label extraction unit 321 performs language analysis processing based on the passed text to obtain a full-context label. An example of the passed text is "I covered New York for about a week." The full-context label extraction unit 321 passes the obtained full-context label to the input label creation unit 322.

[0090] The input label creation unit 322 creates an input label based on the full context label passed from the full context label extraction unit 321 and the pause information and pitch information passed from the acoustic analysis unit 33. The pronunciation information (expressed as a string of phoneme symbols) included in the full context label is passed on as is to the input label. The input label creation unit 322 also includes symbols for distinguishing the length of pauses in the input label based on pause information obtained as a result of acoustic analysis of the speech signal. The input label creation unit 322 also includes symbols representing the relative pitch between adjacent phrases in the input label based on pitch information obtained as a result of acoustic analysis of the speech signal.

[0091] As described above, in this embodiment, the input label contains information about the relative pitch of sounds between adjacent phrases (accent phrases). On the other hand, in the prior art (Patent Document 1), the input label does not contain information about the relative pitch of sounds between adjacent phrases. By using such input labels in this embodiment, it becomes possible to control the relative pitch of sounds between adjacent phrases when estimating acoustic features and synthesizing speech.

[0092] As described above, in this embodiment, the input label always includes a symbol specifying a phrase boundary. On the other hand, in the prior art (Patent Document 1), it is not necessary to use a symbol specifying a phrase or phrase boundary. By using such an input label in this embodiment, it becomes possible to reproduce the correct accent when estimating acoustic features and synthesizing speech.

[0093] As described above, in this embodiment, the input label uses a plurality of symbols (for example, five types) corresponding to differences in pause length. On the other hand, the conventional technology (Patent Document 1) uses only one type of prosodic symbol representing a pause. In this embodiment, when estimating acoustic features and synthesizing speech, it becomes possible to represent differences in pause length, which was not possible in the conventional technology.

[0094] Furthermore, as described above, in this embodiment, the input label does not have a prosodic symbol indicating an accent rise position. On the other hand, in the prior art (Patent Document 1), prosodic symbols indicating accent rise positions and prosodic symbols indicating accent fall positions are used as symbols to specify accent positions. In this embodiment, since the input label does not have a prosodic symbol indicating an accent rise position, there is no (or relatively little) redundancy in the symbols indicating accent positions in the input label. In other words, with less redundancy, the acoustic feature estimation model can have information on the relationship between prosodic symbols and acoustic features. In other words, according to this embodiment, the estimation accuracy of acoustic features is improved.

[0095] As described above, in this embodiment, the pronunciation in the input label is expressed using symbols corresponding to phonemes. On the other hand, in the prior art (Patent Document 1), the pronunciation is expressed using katakana. When the pronunciation is expressed using phoneme symbols as in this embodiment, the number of symbols used is relatively smaller than when it is expressed using katakana. In other words, the input label in this embodiment represents information in a denser representation space than in the prior art. That is, in this embodiment, by training the acoustic feature estimation model using such input labels, the effect of improving the estimation accuracy of acoustic features is obtained.

[0096] That is, in this embodiment, higher quality acoustic features can be estimated, and therefore high quality synthetic speech can be generated stably.

[0097] FIG. 9 is a block diagram showing an example of the internal configuration of the signal processing device 1 (at least a part thereof) in the above embodiment (including modifications). At least a part of the signal processing device 1 can be realized using a computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, etc., and a bus 906. The computer itself can be realized using existing technology. The central processing unit 901 executes instructions contained in a program read from the RAM 902, etc. In accordance with each instruction, the central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with external input / output devices, etc. The input / output devices 904 and 905 are input / output devices. Input / output devices 904 and 905 exchange data with the central processing unit 901 via an input / output port 903. A bus 906 is a common communication path used within the computer. For example, the central processing unit 901 reads and writes data from and to RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port via the bus 906.

[0098] At least some of the functions of the signal processing device 1 in the above-described embodiment can be realized by a computer and a program. In this case, the functions can be realized by recording a program for realizing the functions on a computer-readable recording medium and loading and executing the program recorded on the recording medium into a computer system. Note that the term "computer system" here includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, CD-ROMs, DVD-ROMs, and USB memory, as well as storage devices such as hard disks built into computer systems. In other words, a "computer-readable recording medium" may be a non-transitory computer-readable recording medium. Furthermore, the term "computer-readable recording medium" may also include media that temporarily and dynamically store programs, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, or media that store programs for a certain period of time, such as volatile memory within a computer system that serves as a server or client in such cases. The program may be designed to realize some of the functions described above, or may be capable of realizing the functions described above in combination with a program already stored in the computer system.

[0099] Although a number of embodiments have been described above, the present invention can also be embodied in the following modified examples.

[0100] [Variations] In this embodiment, as described above, the signal processing device 1 performs all of the following steps (1) to (4) in order to generate input labels for training a feature estimation model. As a variation, the signal processing device 1 may perform only at least some of steps (1) to (4) rather than all of these steps.

[0101] (1) The acoustic analysis unit 33 calculates the pitch for each frame based on the speech signal. The language analysis unit 32 calculates a symbol representing the relative pitch of each phrase included in the text relative to adjacent phrases based on the pitch calculated by the acoustic analysis unit 33, and adds the symbol representing the relative pitch to the symbol representing the phrase boundary, and includes the result in the input label. (2) The acoustic analysis unit 33 determines the duration of pauses included in the speech signal by determining the sections of phonemes included in the speech signal. Based on the determined durations of pauses, the language analysis unit 32 includes prosodic symbols for distinguishing the durations of pauses in the input label. (3) The language analysis unit 32 obtains an input label that does not include a symbol indicating the position of rising accent as prosodic information. (4) The language analysis unit 32 obtains an input label as pronunciation information for the text, which is a string of symbols representing phonemes.

[0102] As described above, according to this embodiment, it is possible to improve the estimation accuracy of the acoustic feature estimation model. By using acoustic features estimated using such a model, the synthesized speech generated by the speech signal generation model can be of high quality.

[0103] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Industrial Applicability]

[0104] The present invention can be used, for example, for speech synthesis in a variety of industries. The present invention can also be used for training an acoustic feature estimation model. However, the scope of use of the present invention is not limited to the examples given here. [Explanation of symbols]

[0105] 1. Signal Processing Device 2. Speech synthesis device (signal processing device) 3 Model learning device (signal processing device) 14 Acoustic feature estimation model memory unit 15. Speech signal generation model storage unit 21 Input text storage unit 22 Language Analysis Department 24 Acoustic feature estimation unit 25 Audio signal generator 26 Synthesized speech output unit 31 Training speech corpus storage unit 32 Language analysis department 33 Acoustic Analysis Department 34 Acoustic feature estimation model training unit 35 Speech signal generation model training unit 321 Full-context label extraction unit 322 Input Label Creation Unit 901 Central Processing Unit 902 RAM 903 Input / Output Ports 904,905 Input / Output Devices 906 Bus

Claims

1. a training speech corpus storage unit that stores a training speech corpus including text and speech signals corresponding to the text; an acoustic analysis unit that performs acoustic analysis processing on the speech signal to obtain acoustic features; a language analysis unit that performs a language analysis process on the text to obtain an input label including information on the pronunciation of the text and information on the prosody corresponding to the text; an acoustic feature estimation model training unit that executes machine learning of an acoustic feature estimation model for determining acoustic features corresponding to input labels including pronunciation information and prosody information, using the input labels output by the language analysis unit and the acoustic features output by the acoustic analysis unit as training data; Equipped with the acoustic analysis unit determines a pitch based on the audio signal; the language analysis unit determines a symbol representing a relative pitch of a phrase included in the text relative to other adjacent phrases based on the pitch determined by the acoustic analysis unit, and adds the symbol representing the relative pitch to the symbol representing the phrase boundary, and includes the symbol in the input label; Signal processing device.

2. the acoustic analysis unit determines a duration of a pause included in the speech signal by determining a section of a phoneme included in the speech signal based on the speech signal; the language analysis unit includes, in the input label, a prosodic symbol for distinguishing between multiple types of pause durations based on the pause durations calculated by the acoustic analysis unit. The signal processing device according to claim 1 .

3. the language analysis unit determines the input label that does not include a symbol indicating an accent rise position as prosodic information.

3. The signal processing device according to claim 1.

4. the language analysis unit obtains the input label as the reading information using a string of symbols representing phonemes; A signal processing device according to any one of claims 1 to 3.

5. a speech signal generation model training unit that executes machine learning of a speech signal generation model for determining speech signals corresponding to acoustic features, using the acoustic features output by the acoustic analysis unit and the speech signals included in the training speech corpus as training data; The signal processing device according to claim 1 , further comprising:

6. a language analysis unit that performs language analysis processing on input text, obtains symbols that represent the relative pitch of phrases included in the text relative to other adjacent phrases, and adds the symbols that represent the relative pitch to the symbols that represent the divisions of the phrases, and includes the resulting symbols in the input label, thereby obtaining an input label that includes information on the pronunciation of the text and information on the prosody corresponding to the text; an acoustic feature estimation unit that estimates acoustic features by inputting the input label determined by the language analysis unit into an acoustic feature estimation model trained by the signal processing device according to any one of claims 1 to 4; A signal processing device comprising:

7. a speech signal generation unit that generates a speech signal by inputting the acoustic feature output by the acoustic feature estimation unit into a speech signal generation model that has been trained by the signal processing device according to claim 5; The signal processing device of claim 6 further comprising:

8. Computer, A signal processing device according to any one of claims 1 to 7, A program to function as a

Images