Signal processing device and program

Abstract

To provide a signal processor for accurately estimating accent labels.SOLUTION: A signal processor is equipped with an acoustic analysis unit, a language analysis unit, and a label estimation unit. The acoustic analysis unit obtains frame-by-frame acoustic feature amounts for each predetermined unit of text, based on a voice signal corresponding to the predetermined unit of text. The language analysis unit obtains, based on the predetermined unit of text, the frame-by-frame language feature amounts for the text. The label estimation unit estimates a label of an accent corresponding to the text as an output from a statistical model by inputting the frame-by-frame acoustic and language feature amounts into the pre-trained statistical model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a signal processing device and a program. [Background technology]

[0002] To realize text-to-speech synthesis, which synthesizes speech for any text using statistical information, it is necessary to prepare a large amount of speech signal data, the text of the speech, readings, accents, and other information as advance preparations. Correctly assigning reading and accent information to a large amount of speech signal data requires the work of experts who can accurately understand and distinguish accents, such as announcers, which is costly. Therefore, technology to determine the accent type for speech signal data is in demand and is being researched.

[0003] The technology described in Non-Patent Document 1 is intended to identify which accent type given speech signal data corresponds to. Specifically, this technology determines in advance the pitch that represents each mora (consonant + vowel or vowel, geminated consonant, glottal stop) in an accent phrase. It also constructs a (number of moras-1)-dimensional feature vector from the difference between values ​​between adjacent moras. Then, using the feature vectors of accent phrases with the same accent type, it determines a normal distribution of the (number of moras-1)-dimensional feature vector for each accent type. Then, it uses these modeled normal distributions to identify which accent type the (number of moras-1)-dimensional feature vector calculated for the accent phrase to be determined corresponds to.

[0004] Patent Document 1 also describes a technique for determining the type of accent for a speech signal. The technique of Patent Document 1 aims to support text-to-speech synthesis. To this end, this technique divides speech corresponding to a target phrase into a plurality of accent phrases. Then, in the target accent phrase, the slope of the time change in pitch and the final pitch are obtained for each mora. Then, based on this information, this technique calculates the values ​​of four types of evaluation functions, High, Low, Up, and Down, for each mora. Then, the type of accent is determined by finding the combination that maximizes the sum of the evaluation function values ​​for all the moras that make up the accent phrase. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 4129989 [Non-patent literature]

[0006] [Non-Patent Document 1] Toshinori Ishii, Nobuaki Minematsu, Keiichi Hirose, Accent type determination of continuous Japanese speech considering pitch perception, Technical Report of the Institute of Electronics, Information and Communication Engineers, SP, Speech, 101(270), 23-30, August 23, 2001, Institute of Electronics, Information and Communication Engineers. Summary of the Invention [Problem to be solved by the invention]

[0007] However, the technology described in Non-Patent Document 1 uses the difference in pitch between moras, and is therefore only able to determine the accent type based on the difference values ​​of (the number of moras minus 1). Also, the technology described in Patent Document 1 classifies accent types as a combination of the calculation results of the evaluation function. All of these conventional methods are premised on pitch extraction, and there is a possibility that pitch extraction errors may affect the accuracy of recognition. Furthermore, all of these techniques are based on analysis of accent phrases, and do not take into account pitch fluctuations in the entire utterance or in phrases. Therefore, there is a problem in that the accent cannot be estimated with high accuracy. Thus, the conventional techniques have a problem in that the accent cannot be estimated with high accuracy from a speech signal.

[0008] The present invention has been made based on the above-mentioned problem recognition, and aims to provide a signal processing device and a program capable of estimating accent labels with high accuracy. [Means for solving the problem]

[0009] [1] In order to achieve the above object, a signal processing device according to one aspect of the present invention includes an acoustic analysis unit that calculates, for each predetermined unit of text, acoustic features on a frame-by-frame basis based on a speech signal corresponding to the predetermined unit of text; a language analysis unit that calculates linguistic features of the text on a frame-by-frame basis based on the predetermined unit of text; and a label estimation unit that inputs the frame-by-frame acoustic features and the linguistic features into a pre-trained statistical model and estimates an accent label corresponding to the text as an output from the statistical model.

[0010] [2] In one aspect of the present invention, the signal processing device further comprises a training acoustic analysis unit that calculates, for each training text of a predetermined unit, training acoustic features on a frame-by-frame basis based on a training speech signal corresponding to the training text of the predetermined unit; a training language analysis unit that calculates training language features on a frame-by-frame basis of the training text of the predetermined unit based on the training text of the predetermined unit; a correct label extraction unit that calculates a correct label of an accent corresponding to the text based on the training language features; and a model training unit that inputs the frame-by-frame training acoustic features and the training language features to a statistical model to obtain a training estimated label of an accent corresponding to the training text as an output from the statistical model, and trains the statistical model based on an error between the training estimated label and the correct label.

[0011] [3] Furthermore, a signal processing device according to one aspect of the present invention includes an acoustic analysis unit that calculates, for each predetermined unit of text, acoustic features on a frame-by-frame basis based on a speech signal corresponding to the predetermined unit of text; a language analysis unit that calculates linguistic features of the text on a frame-by-frame basis based on the predetermined unit of text; a correct label extraction unit that calculates a correct label of an accent corresponding to the text based on the linguistic features; and a model learning unit that inputs the frame-by-frame acoustic features and the linguistic features into a statistical model to obtain an estimated label of an accent corresponding to the text as an output from the statistical model, and learns the statistical model based on an error between the estimated label and the correct label.

[0012] [4] Moreover, according to one aspect of the present invention, in the above-mentioned signal processing device, the predetermined unit is an accent phrase unit.

[0013] [5] Also, one aspect of the present invention is that in the above-mentioned signal processing device, the linguistic features are a breath paragraph position in an utterance, an accent phrase position within the breath paragraph, a mora position within the accent phrase, and a frame position within the mora.

[0014] [6] In another aspect of the present invention, in the signal processing device described above, the statistical model is realized using a deep neural network.

[0015] [7] Moreover, according to one aspect of the present invention, in the above-mentioned signal processing device, the acoustic feature is a mel spectrogram.

[0016] [8] Another aspect of the present invention is a program for causing a computer to function as the signal processing device according to any one of [1] to [7] above. Effect of the Invention

[0017] According to the present invention, by using a statistical model trained on acoustic features and linguistic features on a frame-by-frame basis, it is possible to accurately estimate accent by taking into account the influence of utterances based on linguistic features as well. [Brief description of the drawings]

[0018] [Figure 1] 1 is a block diagram showing a schematic functional configuration of an accent estimation device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing a detailed functional configuration of a pre-processing unit (32) included in the accent estimation device according to the embodiment. [Diagram 3] FIG. 13 is a schematic diagram (1 / 3) showing a format of a full context label used by the accent estimation device according to the embodiment. [Figure 4] FIG. 2 is a schematic diagram (2 / 3) showing a format of a full context label used by the accent estimation device according to the embodiment. [Diagram 5] FIG. 3 is a schematic diagram (3 / 3) showing a format of a full context label used by the accent estimation device according to the embodiment. [Figure 6] FIG. 13 is a schematic diagram (1 / 2) showing an example of a full context label extracted by a language analysis unit in a preprocessing unit according to the embodiment performing language analysis processing on a text. [Figure 7] FIG. 2 is a schematic diagram (2 / 2) showing an example of a full context label extracted by a language analysis unit in a preprocessing unit according to the embodiment performing language analysis processing on a text. [Figure 8] 13 is a flowchart showing a processing procedure for a first correct label extraction unit and a second correct label extraction unit according to the embodiment to obtain a correct label for each accent phrase. [Figure 9] 13 is a flowchart showing a procedure of a process performed by a second correct label extraction unit according to the embodiment to create a tone for each accent phrase. [Figure 10] 13 is a schematic diagram showing rules for converting the tones of an accent phrase into numerical values ​​of correct labels in the accent estimation device of the embodiment. FIG. [Figure 11] 11 is a schematic diagram showing a process flow when a statistical model is learned by a model learning unit of the embodiment. FIG. [Figure 12] 11 is a schematic diagram showing a process flow for estimating a label by a label estimation unit according to the embodiment. FIG. [Figure 13] FIG. 2 is a block diagram showing an example of an internal configuration for realizing functions of the accent estimation device according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Next, an embodiment of the present invention will be described with reference to the drawings. First, some concepts that the present embodiment deals with will be described. An utterance is speech that corresponds to a set of sentences. A breath group is a unit of speech that is separated by a pause. A unit in a written sentence that is separated by a comma corresponds approximately to a breath group. An accent phrase is usually a unit of a Japanese word, or a word plus an attached word. A mora is a linguistic element consisting of one consonant and one vowel, or one vowel, mora, or mora. When a mora does not contain a yoon sound or a small vowel that is mainly used in foreign words, it is represented by one kana (such as katakana). When a mora contains a yoon sound or a small vowel that is mainly used in foreign words, it is represented by one kana plus a kana that represents the yoon sound. For example, "chi," "o," "sha," and "fi" are each units of a mora. A phoneme is a unit that is smaller than a mora and consists of only consonants or only vowels.

[0020] In this embodiment, in pre-learning, the accent estimation device divides the text for each utterance into accent phrase units and obtains acoustic features for each frame within an accent phrase from the speech signal. The accent estimation device also obtains linguistic features and correct accent labels for each frame within an accent phrase based on the text. The accent estimation device also learns a statistical model using the acoustic features and linguistic features for each frame within an accent phrase as input data so as to minimize the error between the estimated values ​​of the statistical model and the correct labels. In the label estimation process, the accent estimation device divides the text of each utterance into accent phrases and obtains acoustic features for each frame within the accent phrase from the speech signal. The accent estimation device also obtains linguistic features for each frame within the accent phrase based on the text. The accent estimation device then estimates accent labels by inputting the acoustic features and linguistic features for each frame within the accent phrase into a trained statistical model.

[0021] FIG. 1 is a block diagram showing a schematic functional configuration of an accent estimation device according to this embodiment. As shown in the figure, the accent estimation device 1 includes a model storage unit 11, an estimation speech corpus storage unit 21, a preprocessing unit 22, a label estimation unit 23, a training speech corpus storage unit 31, a preprocessing unit 32, and a model learning unit 33. These functions of the accent estimation device 1 can be realized by, for example, a computer and a program. Furthermore, each function has a storage means as necessary. The storage means is, for example, a variable in the program or a memory allocated by the execution of the program. Furthermore, non-volatile storage means such as a magnetic hard disk device or a solid state drive (SSD) may be used as necessary. Furthermore, at least a part of the functions of each functional unit may be realized as a dedicated electronic circuit rather than a program.

[0022] The accent estimation device 1 shown in the figure includes a function of constructing a statistical model (accent estimation model) by machine learning based on a training speech corpus. The accent estimation device 1 writes the constructed statistical model to a model storage unit 11. The accent estimation device also includes a function of estimating and outputting a label representing the accent of unknown text using the above statistical model. The unknown text is obtained from, for example, the estimation speech corpus.

[0023] Of the functions of the accent estimation device 1, only the function of estimating a label representing an accent of unknown text based on a statistical model constructed in advance may be implemented as the accent estimation device 2. In this case, the accent estimation device 2 is configured to include, among the functional units listed above, an estimation speech corpus storage unit 21, a preprocessing unit 22, and a label estimation unit 23. The accent estimation device 2 can perform label estimation processing by referring to the statistical model stored in the model storage unit 11. Learning of the statistical model at this time has been completed. Of the functions of the accent estimation device 1, only the function of constructing a statistical model may be implemented as the accent estimation model learning device 3. In this case, the accent estimation model learning device 3 is configured to include, among the functional units listed above, a learning speech corpus storage unit 31, a preprocessing unit 32, and a model learning unit 33. The accent estimation model learning device 3 writes information on the statistical model obtained by learning into the model storage unit 11. In other words, the accent estimation model learning device 3 can learn the statistical model. By using this statistical model, it becomes possible to estimate the accent label corresponding to the input data.

[0024] Each of the accent estimation device 1, the accent estimation device 2, and the accent estimation model learning device 3 may be called a “signal processing device.” The speech signal, text, sutra features, and language features described below are each a “signal” that is processed by the signal processing device.

[0025] The model storage unit 11 stores information about a statistical model. The statistical model is a model that can be machine-learned. The statistical model is realized, for example, by using a neural network. The statistical model may be realized, in particular, by using a deep neural network. However, the statistical model may be realized by a method other than a neural network. In either case, the statistical model outputs data (output data) that statistically corresponds to input data (input data). Specifically, the model storage unit 11 stores values ​​of internal parameters of the statistical model. Note that the machine learning method of the statistical model itself can be implemented by using existing technology.

[0026] The training speech corpus storage unit 31 stores a training speech corpus used to construct a statistical model. The training speech corpus contains a large number of pairs of speech signals and text. The speech signals are sampled, for example, at a sampling frequency of 16000 Hertz (Hz) and a conversion bit rate of 16 bits. However, the sampling frequency and the conversion bit rate do not need to be limited to the values ​​exemplified here. The text held in the training speech corpus corresponds to the speech signals and is text that has been created manually or appropriately using speech recognition processing or the like.

[0027] The preprocessing unit 32 processes pairs of speech signals and texts read from the training speech corpus storage unit 31, and outputs pairs of input data and correct labels for the training process. The preprocessing unit 32 divides the input data into accent phrase units, and outputs input data and correct labels for each accent phrase. The input data includes acoustic features and linguistic features for each frame for each accent phrase. The preprocessing unit 32 passes the pairs of input data and correct labels to the model training unit 33. The detailed functional configuration for the processing of the preprocessing unit 32 will be described later with reference to FIG. 2.

[0028] The model learning unit 33 performs machine learning processing using pairs of input data and correct labels passed from the preprocessing unit 32. Specifically, the model learning unit 33 uses a statistical model to estimate labels based on the input data, calculates the difference (loss) between the estimated labels (also called "estimated labels for learning") and the correct labels, and adjusts the internal parameters of the statistical model so as to minimize the loss. The model learning unit 33 performs processing to repeatedly adjust (update) the internal parameters for many pairs of input data and correct labels as learning data. The machine learning method itself used by the model learning unit 33 belongs to existing technology. For example, when the statistical model is realized by a neural network, the model learning unit 33 uses an error backpropagation method to learn the statistical model.

[0029] In other words, the model training unit 33 inputs frame-by-frame acoustic features (training acoustic features) and language features (training language features) to a statistical model, obtains estimated labels (training estimated labels) of accents corresponding to text (training text) as output from the statistical model, and trains the statistical model based on the error (loss) between the estimated labels (training estimated labels) and the correct labels. In other words, the model training unit 33 adjusts (updates) the internal parameters of the statistical model in the direction that reduces the error.

[0030] The estimation speech corpus storage unit 21 stores an estimation speech corpus including text that is a target for label estimation. The estimation speech corpus includes pairs of text and speech signals.

[0031] The preprocessing unit 22 performs preprocessing on the text read from the estimation speech corpus storage unit 21, and passes the result of the preprocessing to the label estimation unit 23 as input data.

[0032] The label estimation unit 23 uses a statistical model to estimate an accent label corresponding to the input data passed from the preprocessing unit 22. The label estimation unit 23 estimates the label by referring to the model storage unit 11. At that point in time, the model storage unit 11 has already stored information on the trained statistical model. Specifically, the label estimation unit 23 inputs frame-by-frame acoustic features and linguistic features to a pre-trained statistical model, thereby estimating an accent label corresponding to the text as an output from the statistical model.

[0033] 2 is a block diagram showing a more detailed functional configuration of the preprocessing unit 32 of the accent estimation device 1. As shown in the figure, the preprocessing unit 32 includes an acoustic analysis unit 351, a phoneme segmentation unit 352, a central phoneme extraction unit 353, a language analysis unit 354, a mora unit integration unit 361, an accent phrase unit integration unit 362, an accent phrase division unit 371, a first language feature extraction unit 381, a second language feature extraction unit 382, ​​a first correct label extraction unit 386, a second correct label extraction unit 387, and an acoustic feature / language feature integration unit 391.

[0034] The preprocessing unit 32 performs preprocessing for learning. The preprocessing unit 32 may be called a "learning preprocessing unit." The acoustic analysis unit 351 included in the preprocessing unit 32 may be called a "learning acoustic analysis unit." The voice signal that the acoustic analysis unit 351 analyzes may be called a "learning voice signal." The acoustic feature value determined by the acoustic analysis unit 351 may be called a "learning acoustic feature value." The language analysis unit 354 included in the preprocessing unit 32 may be called a "learning language analysis unit." The text that the language analysis unit 354 analyzes may be called a "learning text." The language feature value determined by the language analysis unit 354 may be called a "learning language feature value."

[0035] The acoustic analysis unit 351 determines frame-by-frame acoustic features for each predetermined unit of text based on the speech signal corresponding to the predetermined unit of text. In other words, the training acoustic analysis unit determines frame-by-frame training acoustic features for each predetermined unit of training text based on the training speech signal corresponding to the predetermined unit of training text. Here, the "predetermined unit of text" is, for example, text in units of accent phrases.

[0036] The phoneme segmentation unit 352 performs phoneme segmentation on the speech signal. That is, the phoneme segmentation unit 352 identifies a section (a start position and an end position) for each phoneme in the speech signal.

[0037] The central phoneme extraction unit 353 extracts a central phoneme based on the language analysis result (full context label).

[0038] The language analysis unit 354 obtains language features for each frame of the text based on the text in the above-mentioned predetermined unit. In other words, based on the training text in the predetermined unit, the language analysis unit 354 obtains training language features for each frame of the training text. Examples of the language features obtained by the language analysis unit 354 are the breath group position in the utterance, the accent phrase position in the breath group, the mora position in the accent phrase, and the frame position in the mora.

[0039] The mora unit integration unit 361 integrates the frame-by-frame acoustic feature data output by the acoustic analysis unit 351 into mora-by-mora acoustic feature data.

[0040] The accent phrase unit integration section 362 integrates the above acoustic feature data into acoustic feature data for each accent phrase.

[0041] The accent phrase dividing unit 371 divides the full context label, which is the result of the above-mentioned language analysis process, into accent phrase units.

[0042] The first language feature extraction unit 381 extracts language features on a frame-by-frame basis within a mora.

[0043] The second language feature extraction unit 382 extracts language features for each frame in an accent phrase.

[0044] The first correct label extraction unit 386 extracts a correct label for each frame in a mora.

[0045] The second correct label extraction unit 387 extracts a correct label for each frame in the accent phrase.

[0046] The first correct label extraction unit 386 and the second correct label extraction unit 387 may be collectively referred to simply as a “correct label extraction unit.” In other words, the correct label extraction unit obtains a correct label of an accent corresponding to a text based on a language feature (a learning language feature).

[0047] The acoustic feature / language feature integrating unit 391 integrates the acoustic feature data output by the accent phrase unit integrating unit 362 and the language feature data output by the second language feature extracting unit 382. The data integrated by the acoustic feature / language feature integrating unit 391 becomes input data to the statistical model.

[0048] Figures 3, 4 and 5 are schematic diagrams showing the format of a full context label (context-dependent phoneme label) used by the accent estimation device 1. The full context label is obtained as a result of analyzing a text, and is a phoneme label that depends on the context of the text. Figures 3 to 5 show one data structure of the full context label. Note that the method in which the accent estimation device 1 uses the full context label during processing will be described later. As shown in Figures 3, 4 and 5, the full context label includes the data listed below.

[0049] P1 to p5 below are phoneme information. p1: Identification information of the phoneme before the preceding phoneme p2: Front phoneme identification information p3: Identification information of the current phoneme (central phonemes from p1 to p5) p4: Subphoneme identification information p5: Identification information of the phoneme following the next phoneme

[0050] A1 to A3 below are accent information. a1: Difference between accent type (mora position of accent nucleus) and the position of current mora identification information a2: Position of the current mora identification information in the current accent phrase (forward) a3: Position of the current mora identification information in the current accent phrase (backwards)

[0051] The following b1 to b3, c1 to c3, and d1 to d3 are part-of-speech information. b1: Part of speech of the previous word b2: Conjugated form of the previous word b3: Conjugation of the previous word

[0052] c1: Part of speech of the current word c2: Present conjugation form c3: Present word conjugation form

[0053] d1: Part of speech of the next word d2: Conjugation of the following word d3: Conjugation type of the following word

[0054] The following e1 to e5, f1 to f8, and g1 to g5 are accent information. e1: number of moras in the previous accent phrase e2: Accent type in the previous accent phrase (mora position of the accent nucleus) e3: Whether the preceding accent phrase is an interrogative word or not e4:Undefined context e5: Whether a pause is inserted between the previous accent phrase and the current accent phrase

[0055] f1: Number of moras in the current accent phrase f2: Accent type in the current accent phrase (mora position of the accent nucleus) f3: Is the current accent phrase an interrogative word? f4: Undefined context f5: Position of the current accent phrase identification information in the current breath paragraph (forward) f6: Position of the current accent phrase identification information in the current breath paragraph in terms of accent phrases (backwards) f7: Position of the current accent phrase identification information in the current breath paragraph in mora units (forward) f8: Position of the current accent phrase identification information in the current breath paragraph in mora units (backwards)

[0056] g1: Number of moras in the next accent phrase g2: Accent type in the following accent phrase (mora position of the accent nucleus) g3: Is the following accent phrase an interrogative word? g4:Undefined context g5: Whether or not a pause is inserted between the next accent phrase and the current accent phrase

[0057] The following h1 to h2, i1 to i8, and j1 to j2 are breath data. h1: Number of accent phrases in the previous breath h2: Number of moras in the previous breath

[0058] i1: Number of accent phrases in the current breath i2: Number of moras in the current breath i3: Position of the current breath paragraph identification information in breath paragraph units (forward) i4: Position of the breath paragraph unit of the current breath paragraph identification information (backward) i5: Position of accent phrase unit of current breath paragraph identification information (forward) i6: Position of accent phrase unit of current breath paragraph identification information (backward) i7: Position of the current breath paragraph identification information in mora units (forward) i8: Position of the current breath paragraph identifier in mora units (backwards)

[0059] j1: Number of accent phrases in the next breath j2: Number of moras in the next breath

[0060] k1 to k3 below are total information. k1: The total number of breaths in this utterance k2: Total number of accent phrases in this utterance k3: Total number of moras in this utterance

[0061] 6 and 7 are schematic diagrams showing examples of full context labels extracted by the language analysis unit 354 performing language analysis processing on text. This example is a full context label based on the text "Sunny, then cloudy." In this figure, one line of full context label data corresponds to one phoneme. However, one line of data is folded back halfway. FIG. 6 shows the first to eighth lines of data. FIG. 7 shows the ninth to fifteenth lines of data. The beginning of each line corresponds to the above-mentioned items p1 to p5. Also, the part starting with " / A:" of each line corresponds to the above-mentioned items a1 to a3. Also, the part starting with " / B:" of each line corresponds to the above-mentioned items b1 to b3. Also, the part starting with " / C:" of each line corresponds to the above-mentioned items c1 to c3. Also, the part starting with " / D:" of each line corresponds to the above-mentioned items d1 to d3. Additionally, the part starting with " / E:" on each line corresponds to items e1 to e5 mentioned above. Additionally, the part starting with " / F:" ​​on each line corresponds to items f1 to f8 mentioned above. Additionally, the part starting with " / G:" on each line corresponds to items g1 to g5 mentioned above. Additionally, the part starting with " / H:" on each line corresponds to items h1 to h2 mentioned above. Additionally, the part starting with " / I:" on each line corresponds to items i1 to i8 mentioned above. Additionally, the part starting with " / J:" on each line corresponds to items j1 to j2 mentioned above. Additionally, the part starting with " / K:" on each line corresponds to items k1 to k3 mentioned above.

[0062] Next, the pre-learning process and the label estimation process will be described in detail.

[0063] In pre-learning, speech signals for each utterance and text corresponding to the speech signals are used as learning data. In the pre-learning process, frame-by-frame data is created within an accent phrase from the data for each utterance, and a statistical model for estimating correct accent labels is trained using the frame-by-frame data within the accent phrase. Specifically, the process is as follows.

[0064] The preprocessing unit 32 processes pairs of speech signals and texts read from the training speech corpus storage unit 31 .

[0065] The acoustic analysis unit 351 performs acoustic analysis of the audio signal. That is, the acoustic analysis unit 351 performs analysis of the acoustic features for each frame while shifting a predetermined window width. Specifically, the acoustic analysis unit 351 extracts acoustic features for each frame of, for example, 5 milliseconds (ms). However, the length of the frame may be other than 5 milliseconds. As the acoustic features, a mel spectrogram is used. The mel spectrogram is information of the audio signal in the frequency domain. The mel spectrogram is a spectrogram in which the frequency is converted to the mel scale. The mel spectrogram itself is an acoustic feature by an existing method. As an example, the mel spectrogram can be extracted using a function provided at the following URL. URL https: / / librosa.org / doc / latest / generated / librosa.feature.melspectrogram.html

[0066] In this embodiment, the acoustic analysis unit 351 extracts a mel spectrogram in the range of a minimum frequency of 80 Hz and a maximum frequency of 7600 Hz. The order is set to 80. The acoustic analysis unit 351 also standardizes the mel spectrogram for each utterance using the average value and standard deviation calculated for each dimension of the mel spectrogram in the sound section. The acoustic analysis unit 351 outputs the numerical value of this order 80 as an acoustic feature for each frame. It is expected that standardization for each utterance as described above can absorb variations between utterances and between speakers.

[0067] The language analysis unit 354 performs language analysis processing on the text corresponding to the speech signal to obtain a full context label that depends on the context. The language analysis processing includes morphological analysis and syntactic analysis processing of the text (sentence). The format of the full context label is as described in Figs. 3, 4, and 5. An example of an extracted full context label is as described in Figs. 6 and 7. The full context label itself shown in these figures is also used in existing methods. The language analysis unit 354 extracts the full context label from the text using, for example, the analysis technology of a speech synthesis processing toolkit provided at the following URL. URL http: / / open-jtalk.sourceforge.net

[0068] As described below, the full context label includes information such as the breath group position within the sentence in which the phoneme is included, the accent phrase position within the breath group, and the mora position within the accent phrase.

[0069] The central phoneme extraction unit 353 extracts a central phoneme from the full context label output by the language analysis unit 354 .

[0070] The phoneme segmentation unit 352 performs phoneme segmentation based on the speech signal and the central phoneme of the full context label extracted by the central phoneme extraction unit 353. The phoneme segmentation unit 352 obtains phoneme period information for each phoneme included in the speech signal by the phoneme segmentation process. The phoneme period information is information on the start time and end time of a phoneme. In other words, the phoneme segmentation unit 352 identifies the period of the phoneme in the speech signal. The phoneme segmentation unit 352 performs phoneme segmentation using, for example, the forced alignment technology of the speech recognition processing toolkit provided at the following URL. URL http: / / htk.eng.cam.ac.uk

[0071] In addition to the phoneme segmentation process by the phoneme segmentation unit 352, the segmentation may be manually checked and corrected.

[0072] On the other hand, the accent phrase dividing unit 371 divides the full context label output by the language analysis unit 354 into accent phrase units. The accent phrase dividing unit 371 passes information on the full context labels divided into accent phrases to the accent phrase unit integration unit 362.

[0073] The mora-unit integration unit 361 integrates, on a mora-by-mora basis, the acoustic feature data output by the acoustic analysis unit 351. Specifically, the mora-unit integration unit 361 determines acoustic features on a mora-by-mora basis using the central phoneme of the full context label extracted by the central phoneme extraction unit 353 and phoneme section information resulting from the processing by the phoneme segmentation unit 352. The acoustic features in a mora are a collection of acoustic features on a frame-by-frame basis within that mora.

[0074] The accent phrase unit integration unit 362 integrates the mora-based acoustic feature data output by the mora-based integration unit 361 into accent phrase units. Specifically, the accent phrase unit integration unit 362 integrates the mora-based acoustic feature data using the accent phrase information contained in the full context label. The acoustic feature data in an accent phrase is a collection of mora-based acoustic feature data included in the accent phrase.

[0075] Meanwhile, the first language feature extraction unit 381 and the second language feature extraction unit 382 obtain language features from the full context label output by the language analysis unit 354. The first language feature extraction unit 381 outputs frame-by-frame language features in a mora. The second language feature extraction unit 382 outputs frame-by-frame language features in an accent phrase. The first language feature extraction unit 381 assigns the following four pieces of information to each frame in each mora in the target accent phrase: (1) A number that indicates the position of the breath in the sentence (2) A number indicating the position of the accent phrase within the breath. (3) A number representing the position of the mora in the accent phrase (4) A number representing the position of the frame in the mora

[0076] Specifically, the above four pieces of information are as follows: The numerical value in (1) above is a value obtained by dividing the position of the breath group in question in the sentence by the total number of breath groups in the sentence. When expressed by a formula using symbols in the full context label format of Figs. 3 to 5, the breath group position BG (breath group) is BG = i3 / k1. The numerical value in (2) above is a value obtained by dividing the position of the accent phrase in question in the breath group by the total number of accent phrases in the breath group. When expressed by a formula using symbols in the full context label format, the accent phrase position AP (accent phrase) is AP = f5 / i1. The numerical value in (3) above is a value obtained by dividing the position of the mora in question in the accent phrase by the total number of moras in the accent phrase. When expressed by a formula using symbols in the full context label format, the mora position MR (mora) is MR = a2 / f1. The numerical value in (4) above is the value obtained by dividing the frame position in the mora by the total number of frames in the mora. That is, the frame position FR(frame) can be expressed as the formula: frame i / frame_mora. Note that, here, frame i is the frame position (integer value) in the mora, and frame_mora is the total number of frames in the mora. These fourth-order values ​​are used as frame-by-frame linguistic features.

[0077] The first language feature extraction unit 381 extracts and outputs a set of language features for each frame in a mora. That is, the language features in a mora are obtained by repeating the language features for each frame in the mora the number of times for the total number of frames in the mora.

[0078] The second language feature extraction unit 382 extracts and outputs a set of language features for each frame in an accent phrase. In other words, the language features in an accent phrase are a combination of all the language features in the moras in the accent phrase.

[0079] Then, the acoustic feature language feature integration unit 391 integrates the acoustic features output by the accent phrase unit integration unit 362 and the language features output by the second language feature extraction unit 382 to obtain input data for training the model.

[0080] Furthermore, the first correct label extraction unit 386 and the second correct label extraction unit 387 extract correct labels based on the full context label by the following process. Japanese accent can be simulated by expressing the pitch of each mora as a high or low tone, either high (H) or low (L). In one accent phrase, there can be up to one mora position where the pitch of the voice falls. In addition, standard Japanese (also called Tokyo dialect) is characterized by a low tone in the first mora and a high tone in the second mora, except for cases where the tone of the first mora in an accent phrase is high and the tone falls in the second mora (high-headed type).

[0081] 8 is a flowchart showing the procedure of the process for the first correct label extraction unit 386 and the second correct label extraction unit 387 in the preprocessing unit 32 to obtain a correct label for each accent phrase. As a prerequisite, the language analysis unit 354 has already obtained a full context label by analyzing the text. Below, the process of obtaining a correct label will be explained according to this flowchart.

[0082] In step S11, the first correct label extraction unit 386 converts the text into a pronunciation for each mora based on the full context label obtained by the language analysis unit 354. For example, if the original text is "I twisted all reality in my direction," the pronunciation "A / ra / y / ru / gen / n / ji / tsu / o / su / be / te / ji / bu / n / no / ho / o / e / ne / ji / ma / ge / ta / no / da" is obtained by the process in this step. In this pronunciation, a slash ( / ) indicates a mora separator.

[0083] Next, in step S12, the second correct label extraction unit 387 integrates the above pronunciation of the mora unit into pronunciation of the accent phrase unit with reference to the full context label. By the processing of this step, the pronunciation of the above example is converted into pronunciation of the accent phrase unit "arayuru / genjitsuo / subete / jibunnohooe / nejimagetanoda". Here, the slash ( / ) represents the division of the accent phrase.

[0084] Next, in step S13, the second correct label extraction unit 387 creates tones for accent phrases corresponding to the pronunciation of the accent phrases by referring to the full context label. Specifically, the second correct label extraction unit 387 creates tones for accent phrases from the information on the mora position of the accent nucleus in the full context label. The accent nucleus is the mora immediately before the pitch drops. Specifically, the values ​​of the mora position in the accent phrase (a2 represented by a symbol in the form of a full context label), the number of moras in the accent phrase (f1 represented by a symbol in the form of a full context label), and the accent type in the accent phrase (f2 represented by a symbol in the form of a full context label) are observed. When f2=1, it indicates a high-headed type, and the tone at the mora position a2=1 is H, and the tones at other positions are L. When f2=f1, it indicates a flat type or a high-tailed type, and the tone at a2=1 is L, and the tones at other positions are H. Cases other than the two mentioned above represent a middle-high type, with the tone for a2=1 and a2>f2 being L, and tones in other positions being H. Details of the processing in this step will be explained later with reference to FIG. 9. By processing in this step, tones for accent phrase units, "LHHL / LHHHH / HLL / LHHHHLL / LHHLLLL", corresponding to the pronunciation of the accent phrase units in the above example, are obtained. The slash ( / ) here also represents a separator between accent phrases. Furthermore, L corresponds to low, and H corresponds to high.

[0085] Next, in step S14, the second correct label extraction unit 387 converts the tone of each accent phrase into a correct label of the accent. In this embodiment, the accent correct label is a sequence of three numbers, 0, 1, and 2, corresponding to the tone of each accent phrase. The meaning of these numbers is as follows. That is, the number 2 corresponds to a case where the pitch of the voice falls in the mora following the mora in the accent phrase. In other words, when the tone changes from H to L between adjacent moras, the part corresponding to the H corresponds to 2. The number 1 corresponds to a case where the pitch of the voice rises in the mora following the mora in the accent phrase. In other words, when the tone changes from L to H between adjacent moras, the part corresponding to the L corresponds to 1. The number 0 corresponds to a case where the pitch of the voice does not change in the mora following the mora in the accent phrase. In other words, when the tone changes to LL or HH between adjacent moras, the part corresponding to the mora before that corresponds to 0. The number corresponding to the last mora in the accent phrase is also 0. This step of processing obtains the correct label for each accent phrase, "1020 / 10000 / 200 / 1000200 / 1020000," based on the tones of each accent phrase in the above example. The slash ( / ) here also indicates the division of the accent phrase. Each number here corresponds to a mora.

[0086] That is, the number "2" contained in the correct label corresponds to the mora immediately before a pitch drop, the number "1" contained in the correct label corresponds to the mora immediately before a pitch rise, and the number "0" contained in the correct label corresponds to a mora where the pitch does not change or is the last mora in an accent phrase.

[0087] In this step, the second correct label extraction unit 387 further assigns the above-mentioned mora numerical value (correct label) to each frame in the mora. In other words, the second correct label extraction unit 387 copies the numerical value (0, 1, or 2) corresponding to the correct label of a certain mora so as to repeat it for the number of frames contained in the mora. In this way, the second correct label extraction unit 387 generates a string of numerical values ​​for each frame in the accent phrase. In other words, this numerical value of degree 1 is the correct label for each frame in the accent phrase.

[0088] Through the above-described processing, the acoustic feature / language feature integration unit 391 of the preprocessing unit 32 outputs data in which the acoustic feature and the language feature are integrated. The second correct label extraction unit 387 outputs a correct label related to accent. The pair of both data is used by the model learning unit 33 to learn a statistical model.

[0089] Fig. 9 is a flowchart showing the details of the process of step S13 shown in Fig. 8. That is, the flowchart shows the procedure of the process in which the second correct label extraction unit 387 creates a tone for each accent phrase. The following will be described with reference to this flowchart.

[0090] In step S21, the second correct label extraction unit 387 judges whether or not f2=1. That is, it judges whether or not the value of the accent type in the accent phrase is 1. If f2=1 (step S21: YES), the process proceeds to the judgment in step S22. Otherwise (step S21: NO), the process proceeds to the judgment in step S25.

[0091] In step S22, the second correct label extraction unit 387 judges whether a2=1. That is, it judges whether the mora position in the accent phrase is 1. If a2=1 (step S22: YES), the tone is set to "H" (step S24) and the process of this flowchart ends. Otherwise (step S22: NO), the tone is set to "L" (step S23) and the process of this flowchart ends.

[0092] If the process proceeds to step S25, in this step the second correct label extraction unit 387 judges whether or not f2=f1. That is, it judges whether or not the value of the accent type in the accent phrase is equal to the number of moras in the accent phrase. If f2=f1 (step S25: YES), the process proceeds to the judgment in step S26. Otherwise (step S25: NO), the process proceeds to the judgment in step S29.

[0093] In step S26, the second correct label extraction unit 387 judges whether a2=1. That is, it judges whether the mora position in the accent phrase is 1. If a2=1 (step S26: YES), the tone is set to "L" (step S28) and the process of this flowchart ends. Otherwise (step S26: NO), the tone is set to "H" (step S27) and the process of this flowchart ends.

[0094] If the process proceeds to step S29, in that step the second correct label extraction unit 387 determines whether the condition (a2=1 or a2>f2) is met. a2=1 corresponds to the case where the mora position in the accent phrase is 1. a2>f2 corresponds to the case where the mora position in the accent phrase is greater than the accent type in the accent phrase. If (a2=1 or a2>f2) is met (step S29: YES), the tone is set to "L" (step S31) and the processing of this flowchart is terminated. Otherwise (step S29: NO), the tone is set to "H" (step S30) and the processing of this flowchart is terminated.

[0095] As described above, the second correct label extraction unit 387 creates tones for each accent phrase through the process described in this flowchart.

[0096] Fig. 10 is a schematic diagram showing the rules for converting the tones of an accent phrase into the numerical values ​​of the correct labels in the process of step S14 shown in Fig. 8. As shown in the figure, the second correct label extraction unit 387 determines the numerical value of the correct label corresponding to the earlier of two consecutive moras in the tone of the accent phrase, depending on the tone pattern of the two moras. However, for a mora that ends an accent phrase (pattern (E) or (F) in Fig. 10), the second correct label extraction unit 387 determines the numerical value of the correct label corresponding to the mora, regardless of the tone of the mora.

[0097] Pattern (A) in Figure 10 is a case where the tone of two consecutive moras changes from "H" to "L", and the correct label value corresponding to the preceding mora in this case is "2". Pattern (B) is a case where the tone of two consecutive moras changes from "H" to "L", and the correct label value corresponding to the preceding mora in this case is "1". Pattern (C) is a case where the tone of two consecutive moras goes from "H" to "H", and the correct label value corresponding to the preceding mora in this case is "0". Pattern (D) is a case where the tone of two consecutive moras goes from "L" to "L", and the correct label value corresponding to the preceding mora in this case is "0". In other words, patterns (C) and (D) are cases where the tone of two consecutive moras continues without changing. Patterns (E) and (F) correspond to the last mora in an accent phrase. In this case, regardless of whether the tone of the mora is "H" or "L", the numerical value of the correct label corresponding to the mora is "0".

[0098] The model learning unit 33 learns a statistical model using input data (learning data combining acoustic features and linguistic features on a frame-by-frame basis) in units of accent phrases passed from the preprocessing unit 32 and correct labels. The statistical model is realized, for example, using a DNN (deep neural network). Specifically, the model learning unit estimates a label based on the above input data, finds the difference between the estimated label and the correct label, and updates the internal parameters of the statistical model by an error backpropagation method based on the difference.

[0099] The statistical models may be constructed by distinguishing between a model for the voice of a male speaker and a model for the voice of a female speaker. Alternatively, a gender-independent model may be constructed as the statistical model without distinguishing between male and female speakers. When training a model for a male speaker, the model training unit 33 uses training data for a male speaker. When training a model for a female speaker, the model training unit 33 uses training data for a female speaker. When training a gender-independent model, the model training unit 33 uses a mixture of training data for both male and female speakers. Also, a statistical model for a specific individual speaker may be constructed as the statistical model.

[0100] The input data in the training data is, for example, 84th-order data combining 80th-order acoustic features and 4th-order linguistic features. As the input data, for example, data on hundreds of thousands of accent phrases is used. The input data is standardized by calculating the average value and standard deviation for each of the 84 dimensions. The data on the correct answer labels may be used as is without standardization.

[0101] As described above, for example, hundreds of thousands of pieces of learning data are rearranged in a random order and divided into training data, validation data, and evaluation data in a ratio of, for example, 8:1:1 for use. The learning process by the model learning unit 33 is performed in mini-batch units. The mini-batch size handled at one time is, for example, 32 accent phrases. For each mini-batch, the suffixes of shorter accent phrases are padded with zeros to match the accent phrase with the longest number of frames (number of frames: frame_max) and used for learning.

[0102] The process itself in which the model learning unit 33 learns the statistical model can be performed using existing technology. As an example, TensorFlow, an open source software machine learning toolkit, can be used. TensorFlow is described at the following URL: URL: https: / / tensorflow.org

[0103] 11 is a schematic diagram showing a process flow when the model learning unit 33 learns a statistical model. As shown in the figure, the statistical model is configured to include an input layer 411, a masking layer 412, a bidirectional LSTM 413, a layer normalization layer 414, a dropout layer 415, a fully connected layer 416, and a softmax function 417. Note that LSTM is an abbreviation for "Long short-term memory." Bidirectional LSTM may be written as "BLSTM."

[0104] During learning, the model learning unit 33 inputs a tensor [32, frame_max, 84] of 84th-order input data, which is adjusted to the longest number of frames per mini-batch, frame_max, to the input layer with a mini-batch size of 32. The input data is processed by the input layer 411 and the masking layer 412. The masking layer 412 outputs a tensor [32, frame_max, 84] having masking information that ignores the zero-padded portion. The masking information propagates to layers after the bidirectional LSTM 413, enabling calculations that ignore the zero-padded portion. The model learning unit 33 passes the output tensor [32, frame_max, 84] from the masking layer 412 through a bidirectional LSTM layer 413 (adding the forward and backward outputs) with 64 units × 6 layers, and outputs a tensor [32, frame_max, 64]. In the bidirectional LSTM layer 413, a 12-order regularization is applied to the kernel at 0.001 to prevent overlearning. Then, in order to further improve the generalization performance, the model learning unit 33 passes the output tensor [32,frame_max,64] from the 6-layer bidirectional LSTM layer 413 through a normalization layer 414 and a dropout layer 415 with a drop rate of 0.5 to output the tensor [32,frame_max,64]. Then, the model learning unit 33 passes the output tensor [32,frame_max,64] from the dropout layer 415 through a fully connected layer 416 with 3 units to output the tensor [32,frame_max,3]. Based on the output tensor [32,frame_max,3] from the fully connected layer 416, the model learning unit 33 performs the calculation of the softmax function 417 while ignoring values ​​masked by masking for each frame. The model learning unit 33 outputs the tensor [32,frame_max,3] of the estimated value, which is the output of the model.

[0105] The model learning unit 33 calculates a loss function (difference, loss) using the correct label [32, frame_max, 1] output from the preprocessing unit 32 and the above estimated value [32, frame_max, 3]. Specifically, the model learning unit 33 obtains a mask value [32, frame_max, 1] for ignoring the zero-padded portion of the correct label, and calculates the loss using the labels in multi-class classification with the correct label [32, frame_max, 1] and the estimated value [32, frame_max, 3]. To calculate the loss, the model learning unit 33 calculates the SparseCategoricalCrossentropy() function (sparse categorical cross entropy) and obtains the calculation result [32, frame_max]. Then, the model learning unit 33 expands the dimension to [32, frame_max, 1] using the expand_dims() function. The model learning unit 33 takes the logical product (logical_and() function) of that value and the mask value [32,frame_max,1] obtained above, and calculates the average (reduce_mean() function) of the obtained value as the loss. Note that the reduce_mean() function is a function that takes the average of a specific dimension of a tensor.

[0106] The model learning unit 33 calculates the training loss of the model for each batch of training data, obtains the gradient of the model weights with respect to the training loss, and uses an optimizer to update the model weights based on the gradient. In this case, the model learning unit 33 uses SGD (Stochastic Gradient Descent) with learning_rate=0.00001, decay=1e-6, and momentum=0.9 as the optimizer.

[0107] The model learning unit 33 updates the retained model when the training loss falls below the minimum training loss up to the previous epoch. The model learning unit 33 performs learning for up to 1000 epochs, with one epoch being the process of completing calculations for all mini-batches of training data. The model learning unit 33 obtains the validation loss and validation accuracy of the model using validation data for each epoch, and if the validation loss does not improve for 50 consecutive epochs, it early stops learning. The model learning unit 33 adopts the retained model when the validation loss is minimized in this way.

[0108] The model learning unit 33 writes the parameter information of the adopted model into the model storage unit 11.

[0109] Next, the label estimation process will be described in detail. In the label estimation process, the data on which the estimation is based includes a speech signal for each utterance and a transcribed text corresponding to the speech signal. In this embodiment, the estimation speech corpus storage unit 21 holds the above-mentioned pairs of speech signal and text.

[0110] The preprocessing unit 22 has a configuration similar to that of the preprocessing unit 32 described above. However, the preprocessing unit 22 does not need to have a configuration for extracting a correct label. That is, the preprocessing unit 22 does not need to have the functions of the first correct label extraction unit 386 and the second correct label extraction unit 387 of the preprocessing unit 32 shown in FIG. 2. The preprocessing unit 22 creates and outputs input data consisting of acoustic features and language features in units of frames for each accent phrase based on the text and speech signal for each utterance, similar to the preprocessing unit 32. Since the preprocessing unit 32 has already been described with reference to FIG. 2, a detailed description of the procedure by which the preprocessing unit 22 creates input data consisting of acoustic features and language features will be omitted here.

[0111] The label estimation unit 23 estimates a label for an accent based on the input data output by the pre-processing unit 22.

[0112] Fig. 12 is a schematic diagram showing the flow of processing in which the label estimation unit 23 estimates a label. As shown in the figure, the configuration of the statistical model at this time is the same as the configuration (reference numerals 411 to 417) shown in Fig. 11. However, at the time of estimation, the dropout layer 415 is ignored. At the time of label estimation processing, the statistical model has already been trained. In other words, the values ​​of the parameters (weights) of the statistical model have already been determined. At the time of label estimation processing by the label estimation unit 23, no adjustment of parameters based on the correct label is performed.

[0113] The input data that the label estimation unit 23 receives from the preprocessing unit 22 is a tensor [1, frame, 84] of input data with a mini-batch size of 1, the number of frames being frame, and the degree being 84. The label estimation unit 23 inputs the input data from the input layer 411. The processing in each layer from the input layer 411 to the softmax function 417 has already been described as the processing for model learning. The label estimation unit 23 obtains an estimated value [1, frame, 3] with a mini-batch size of 1, the number of frames being frame, and the degree being 3, using this trained model. For each frame of the obtained estimated value [1, frame, 3], the label estimation unit 23 sets the maximum index (0, 1, or 2) among the three-degree values ​​as the numerical value of the estimated label for each frame. Since one mora includes many frames, the label estimation unit 23 determines the estimated label for each mora by taking a majority vote of the values ​​(0, 1, or 2) of the estimated labels for each frame in the mora.

[0114] The accent can be identified by the value of the estimated label for each mora in the accent phrase. When generating speech using synthetic speech, the pitch can be adjusted by the value of the estimated label for each mora. If the estimated label for each mora is "1", the pitch of the voice rises in the mora following that mora. If the estimated label for each mora is "2", the pitch of the voice falls in the mora following that mora. If the estimated label for each mora is "0", there will be no situation in which the pitch changes in the mora following that mora (including cases where there is no next mora in the accent phrase).

[0115] FIG. 13 is a block diagram showing an example of an internal configuration for realizing the functions of the accent estimation device 1. The accent estimation device 1 (or at least a part thereof) can be realized using a computer. As shown in the figure, the computer is configured to include a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, etc., and a bus 906. The computer itself can be realized using existing technology. The central processing unit 901 executes instructions included in a program read from the RAM 902, etc. According to each instruction, the central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic operations and logical operations. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. It should be noted that RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with an external input / output device, etc. The input / output devices 904 and 905 are input / output devices. Input / output devices 904 and 905 exchange data with the central processing unit 901 via an input / output port 903. A bus 906 is a common communication path used inside the computer. For example, the central processing unit 901 reads and writes data from and to a RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port via the bus 906.

[0116] At least some of the functions of the accent estimation device 1 in the above-mentioned embodiment can be realized by a computer. In that case, a program for realizing the functions may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the functions. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, CD-ROMs, DVD-ROMs, and USB memories, and storage devices such as hard disks built into a computer system. In other words, the term "computer-readable recording medium" may be a non-transitory computer-readable recording medium. The term "computer-readable recording medium" may also include a medium that temporarily and dynamically holds a program, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in such a case. The above-mentioned program may be a program for realizing some of the above-mentioned functions, or may be a program that can be realized in combination with a program already recorded in the computer system.

[0117] As explained above, in this embodiment, by using a statistical model trained with acoustic features and linguistic features on a frame-by-frame basis within an accent phrase, it is possible to accurately estimate an accent by taking into account the influence of an utterance, such as the breath paragraph position within an utterance, the accent phrase position within a breath paragraph, the mora position within an accent phrase, and the frame position within a mora. Also, in this embodiment, the statistical model can be trained based on the acoustic features and linguistic features on a frame-by-frame basis within an accent phrase, and the correct label.

[0118] The accent estimation device 1 or 2 and the accent estimation model learning device 3 may be implemented by the following modified example. In the above embodiment, a statistical model is learned based on text and speech signals with accent phrases as units. Also, assuming that a learned statistical model exists, a process is performed to estimate an accent label based on text and speech signals with accent phrases as units. In a modified example, an accent label may be estimated or a statistical model may be learned based on text and speech signals of other predetermined units, not limited to accent phrase units. Also, in a modified example, information on the position of a predetermined unit (not limited to accent phrase units, any unit) in an utterance, information on the position of a mora, or information on the position of a frame is used as a language feature.

[0119] In the above embodiment, a mel spectrogram is used as an acoustic feature. As a modification, the accent estimation device 1 or 2 and the accent estimation model learning device 3 may be implemented using an acoustic feature other than a mel spectrogram.

[0120] The above describes in detail the embodiments and modified examples of the present invention with reference to the drawings, but the specific configurations are not limited to these embodiments, and designs that do not deviate from the gist of the present invention are also included.

[0121] [Evaluation of the Examples] The evaluation of the examples will be described below. In the examples, the accent estimation model learning device 3 learned an unspecified speaker statistical model for female voices and an unspecified speaker statistical model for male voices. As the learning data for the unspecified speaker statistical model for female voices, 289,197 accent phrases extracted from 37,226 utterances by eight female speakers were used. As the learning data for the unspecified speaker statistical model for male voices, 196,956 accent phrases extracted from 39,065 utterances by nine male speakers were used.

[0122] Furthermore, the label estimation was evaluated using the trained speaker-independent statistical model for female voices and speaker-independent statistical model for male voices. The evaluation data used was data that was not used for training the models. As a result, the accuracy of the estimated accent labels was as follows. That is, the accuracy of the speaker-independent statistical model for female voices was 98.98%. The accuracy of the speaker-independent statistical model for male voices was 99.28%. In this way, this embodiment was able to estimate accent labels with high accuracy. [Industrial Applicability]

[0123] The present invention can be used, for example, to estimate the correct accent label for a given accent phrase. Furthermore, the present invention can be used, for example, in speech synthesis. However, the scope of the present invention is not limited to the examples given here. [Explanation of symbols]

[0124] 1,2 Accent estimation device (signal processing device) 3. Accent estimation model learning device (signal processing device) 11 Model storage section 21 Estimation speech corpus memory unit 22 Pretreatment section 23 Label Estimation Unit 31 Training speech corpus storage unit 32 Pretreatment section 33 Model Learning Department 351 Acoustic Analysis Unit (Learning Acoustic Analysis Unit) 352 Phoneme Segmentation Unit 353 Central phoneme extraction part 354 Language Analysis Unit (Learning Language Analysis Unit) 361 Mora Unit Integration Section 362 Accent phrase unit integration section 371 Accent Phrase Segmentation 381 First Language Feature Extraction Unit 382 Second Language Feature Extraction Unit 386 First correct label extraction unit (correct label extraction unit) 387 Second correct label extraction unit (correct label extraction unit) 391 Acoustic Feature and Language Feature Integration Unit 901 Central Processing Unit 902 RAM 903 Input / Output Ports 904,905 Input / Output Devices 906 Bus

Claims

1. an acoustic analysis unit that calculates, for each predetermined unit of text, an acoustic feature quantity for each frame based on a speech signal corresponding to the predetermined unit of text; a language analysis unit that calculates language features of the text in units of frames based on the text in the predetermined units; a label estimation unit that estimates an accent label corresponding to the text as an output from a pre-trained statistical model by inputting the frame-by-frame acoustic features and the linguistic features into the pre-trained statistical model; A signal processing device comprising:

2. a training acoustic analysis unit that calculates training acoustic features in units of frames for each training text of a predetermined unit based on a training speech signal corresponding to the training text of the predetermined unit; a training language analysis unit that calculates training language features of the training text for each frame based on the training text for each predetermined unit; a correct label extraction unit for determining a correct label of an accent corresponding to the text based on the training language features; a model training unit that inputs the training acoustic features and the training linguistic features for each frame into a statistical model to obtain training estimated labels of accents corresponding to the training text as an output from the statistical model, and trains the statistical model based on an error between the training estimated labels and the ground truth labels; The signal processing device according to claim 1 , further comprising:

3. an acoustic analysis unit that calculates, for each predetermined unit of text, an acoustic feature quantity for each frame based on a speech signal corresponding to the predetermined unit of text; a language analysis unit that calculates language features of the text in units of frames based on the text in the predetermined units; a correct label extraction unit for determining a correct label of an accent corresponding to the text based on the linguistic feature; a model learning unit that inputs the frame-by-frame acoustic features and the linguistic features into a statistical model to obtain an estimated label of an accent corresponding to the text as an output from the statistical model, and learns the statistical model based on an error between the estimated label and the ground truth label; A signal processing device comprising:

4. The predetermined unit is an accent phrase unit. A signal processing device according to any one of claims 1 to 3.

5. The linguistic feature amounts are a breath group position in an utterance, an accent phrase position in the breath group, a mora position in the accent phrase, and a frame position in the mora. The signal processing device according to claim 4.

6. The statistical model is realized using a deep neural network. A signal processing device according to any one of claims 1 to 5.

7. The acoustic feature is a mel spectrogram. A signal processing device according to any one of claims 1 to 6.

8. Computer, A signal processing device according to any one of claims 1 to 7. A program to function as a

Images