Speech synthesis processing device, learning method, and acoustic feature generation device

Abstract

To provide a voice synthesis processing device which performs voice synthesis processing capable of performing stable high-precision speaking speed conversion while taking the elastic modulus of a phoneme into consideration, a learning method for performing learning processing thereof, and an acoustic feature amount generation device.SOLUTION: A voice synthesis processing device 100 comprises a noise addition processing unit 3A which calculates loss by adding Gaussian noise n to an aligned target phoneme continuation length to be estimated in learning processing. There is not the problem that an elastic modulus of a phoneme cannot be reproduced in the voice synthesis processing since a dispersion value of the phoneme continuation length becomes "0" by setting the dispersion value of the phoneme continuation length to be estimated to a finite value.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This invention relates to speech synthesis processing technology, and more particularly to speech synthesis processing technology capable of speech rate conversion. [Background technology]

[0002] In the human speech mechanism, it is known that some phonemes are more prone to stretching or contracting in length when speaking at high or low speeds, while others are less prone to stretching or contracting. Therefore, when performing speech speed conversion using text-to-speech synthesis technology, simply stretching or contracting the length of all phonemes uniformly results in unnatural-sounding synthesized speech. Conventional speech synthesis technology using HMM (hidden Markov model) estimates not only the average length of each phoneme but also its variance as an elastic modulus, enabling stretching or contraction that considers the stretching or contracting rate of each phoneme's length during speech speed conversion (see, for example, Non-Patent Document 1). However, speech synthesis technology using HMM has suffered from low model accuracy, resulting in issues with sound quality.

[0003] On the other hand, in order to achieve high-quality speech synthesis, attempts have been made to perform speech rate conversion using speech synthesis technology that uses neural network models (see, for example, Non-Patent Document 2) and speech synthesis technology that uses speech synthesis models equipped with an attention mechanism (see, for example, Non-Patent Document 3). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] T. Yoshimura, K. Tokuda, T. Masuko, T. Kobayashi, and T. Kitamura, "Duration modeling for HMM-based speech synthesis," in Proc. ICSLP, Nov. 1998. [Non-Patent Document 2] I. Elias, H. Zen, J. Shen, Y. Zhang, Y. Jia, RJ Skerry-Ryan, and Y. Wu, "Parallel Tacotron 2: A non-autoregressive neural TTS model with differentiable duration modeling," in Proc. Interspeech, Aug. 2021, pp. 141-145. [Non-Patent Document 3] J.-S. Bae, H. Bae, Y.-S. Joo, J. Lee, G.-H. Lee, and H.-Y. Cho, "Speaking speed control of end-to-end speech synthesis using sentence-level conditioning," in Proc. Interspeech, Oct. 2020, pp. 4402-4406. [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] However, while state-of-the-art speech synthesis technology using neural network models produces high-quality synthesized (generated) speech, it suffers from a problem where the accuracy of speech synthesis deteriorates because it uniformly stretches the phoneme length of all phonemes during speech rate conversion.

[0006] In response to this, attempts have been made to solve the above problem by inputting a speech rate ratio, which is the ratio of the number of phonemes in the text to the number of frames to be output, into a speech synthesis model equipped with an attention mechanism (attention mechanism type speech synthesis model). However, attention mechanism type speech synthesis models have a critical problem in that if the prediction by the attention mechanism fails, phonemes will not be spoken, or speech will stop midway.

[0007] Therefore, in view of the above problems, the present invention aims to realize a speech synthesis processing device that performs stable and highly accurate speech rate conversion while taking into account the elastic modulus of phonemes. [Means for solving the problem]

[0008] The first invention for solving the above problems is a speech synthesis processing device comprising an encoder, a duration estimation processing unit, a decoder, and a vocoder.

[0009] The encoder takes phoneme sequence data, which is a sequence of phonemes that make up text data, as input, performs encoding on the phoneme sequence data, and outputs the encoded data.

[0010] The duration estimation processing unit obtains the first data by performing a convolution operation on the data output from the encoder, and then estimates the duration of each phoneme included in the phoneme sequence data from the first data. The duration estimation processing unit also outputs the first data and the duration data of each phoneme included in the phoneme sequence data. The decoder obtains acoustic features by performing a decoding process using the data output from the encoder and the data output from the duration estimation processing unit.

[0011] The continuation length estimation processing unit comprises a mapping transformation layer, a range restriction processing unit, an activation processing unit, and a continuation length acquisition unit.

[0012] The mapping transformation layer obtains data from the first data set that corresponds to the estimated mean and estimated variance of the duration of each phoneme included in the phoneme sequence data.

[0013] The range restriction processing unit obtains the processed data as range-restricted variance data by executing a variance range restriction process that restricts the range of the estimated variance values ​​of the phoneme duration obtained by the mapping transformation layer.

[0014] The activation processing unit applies an activation function to the estimated average value data of phoneme durations obtained by the mapping transformation layer, and obtains the processed data as the estimated average data.

[0015] The continuation length acquisition unit obtains the estimated phoneme continuation length based on range-restricted variance data and estimated average data.

[0016] A vocoder generates an audio waveform based on acoustic features from a decoder.

[0017] Furthermore, the encoder, duration estimation processing unit, and decoder can each be equipped with a learning model that can be trained by updating its parameters, or a pre-trained model of the learning model. Based on data obtained by adding Gaussian noise to the correct duration data of each phoneme, range-restricted variance data, and estimated average data, a loss for phoneme duration is calculated. By updating the parameters of the encoder, duration estimation processing unit, and decoder based on the calculated loss, a learning process is executed, and the pre-trained model obtained through this learning process can be equipped.

[0018] In this speech synthesis processing device, during the learning process, the loss is calculated by adding Gaussian noise n to the aligned target phoneme duration to be estimated, so that the variance of the estimated phoneme duration can be a finite value.

[0019] Therefore, in this speech synthesis processing device, if training is performed without adding Gaussian noise, the variance of the phoneme duration becomes "0," and the problem of not being able to reproduce the elastic modulus of the phoneme (the rate at which the phoneme expands and contracts according to the speech speed) during prediction processing (speech synthesis processing) does not occur. In other words, since this speech synthesis processing device performs training using phoneme durations with Gaussian noise added, the variance of the phoneme duration can be made a finite value, and as a result, the elastic modulus of the phoneme (the rate at which the phoneme expands and contracts according to the speech speed) can be reproduced during prediction processing (speech synthesis processing).

[0020] That is, in this speech synthesis processing device, stable and highly accurate speech synthesis processing can be performed while considering the elastic modulus of phonemes.

[0021] The second invention is the first invention, wherein the phoneme sequence data consists of the first to the K-th phonemes which are K (K: natural number) phonemes, and the range-restricted variance data of the estimated average value of the duration of the k-th phoneme is σ min , max , min , 2 , 2 , 2 2 (k: natural number, 1 ≤ k ≤ K), and the estimated average data of the duration of the k-th phoneme is ξ k (k: natural number, 1 ≤ k ≤ K), then The range restriction processing unit inputs the upper limit value V max 2 and the lower limit value V min 2 and executes range restriction processing so that the range of the estimated variance value of the phoneme duration satisfies V min 2 ≤ σ k 2 ≤ V max 2 .

[0022] And, when the Gaussian noise added to the correct data of the duration of each phoneme is n, the Gaussian noise n is n ~ N(0, V min 2 ) N(μ, σ 2 ): Gaussian distribution with mean μ and variance σ 2 is set to satisfy.

[0023] As a result, in this speech synthesis processing device, the Gaussian noise n added to the correct data of the duration of each phoneme becomes noise following N(0, V min 2 ), and furthermore, the variance value of the phoneme duration is set between the lower limit value V min 2 and the upper limit value V max 2Since the learning process can be performed within the range between these two values, the variance of the estimated phoneme duration can be set to a predetermined range (from the lower limit to the upper limit) during the prediction process (speech synthesis). In other words, in this speech synthesis processing device, the upper limit of the variance of the phoneme duration is V. max 2 Since this can also be set, only the variance value of the duration of phonemes in silent parts will be a large value, and problems such as the expansion and contraction of only the silent parts during speech rate conversion (when speech rate conversion is performed and speech is synthesized) can be resolved. In other words, in this speech synthesis processing device, the variance range restriction processing unit of the duration estimation processing unit sets the upper limit V of the variance value of phoneme duration. max 2 Since this can also be set, the variance value of the phoneme duration in silent parts can also be set to the upper limit V. max 2 The following can be achieved, and as a result, for example, silent parts can be appropriately played back (speech synthesis) during speech rate conversion (when speech rate conversion is performed and speech is synthesized).

[0024] In other words, this speech synthesis processing device can perform stable and highly accurate speech synthesis processing while taking into account the elastic modulus of phonemes.

[0025] The third invention is the first or second invention, wherein the learning process is performed by updating the parameters of the encoder, the duration estimation processing unit, and the decoder based on an integrated loss obtained by integrating a loss for acoustic features, which is defined based on the norm of the error between the acoustic features acquired by the decoder and the ground truth acoustic features, which are acoustic features corresponding to the phoneme sequence data, and a loss for phoneme duration.

[0026] As a result, this speech synthesis processing device can perform the learning process more appropriately by considering losses related to acoustic features and losses related to phoneme duration.

[0027] The fourth invention is any of the first to third inventions, wherein the phoneme sequence data consists of the first to kth phonemes, which are K (K: a natural number) phonemes, and the range-restricted variance data of the estimated mean duration of the kth phoneme is σk 2 Let k be a natural number, 1 ≤ k ≤ K, and let ξ be the estimated average data of the duration of the k-th phoneme. k Let k be a natural number, 1 ≤ k ≤ K, and let T be the number of frames in a given period. The continuous length acquisition unit is,

number

[0028] The decoder obtains estimated acoustic feature data by performing a decoding process based on the estimated duration data of phonemes estimated by the duration acquisition unit and the first data.

[0029] As a result, in this speech synthesis processing device, the elastic modulus ρ of a phoneme G By taking these factors into consideration, stable and highly accurate speech synthesis processing can be performed. Furthermore, by adjusting the data T, the elastic modulus ρ of a phoneme at a given speech rate can be determined. G This allows us to obtain the data T. In other words, by adjusting the data T, we can perform speech synthesis processing at a predetermined speaking speed.

[0030] The fifth invention is the fourth invention, wherein the decoder is a learnable functional unit and includes a learnable upsampling layer that performs upsampling processing on first data based on the estimated duration data of phonemes estimated by the duration acquisition unit.

[0031] In this speech synthesis processing device, the weighting coefficients in the learnable upsampling layer can be real-valued, allowing for weighted addition processing. Therefore, it can handle cases where the estimated duration data of phonemes are real-valued. In other words, the data T that identifies the elastic modulus of the phoneme, which defines the estimated duration data of the phonemes, can be real-valued. As a result, this speech synthesis processing device can control the speech rate using real-valued values.

[0032] The sixth invention is a learning method for performing learning processing on a speech synthesis processing device which is one of the first to fifth inventions, comprising an encoding step, an estimated mean-variance data acquisition step, a noise addition step, a loss calculation step, and a parameter update step.

[0033] The encoding step takes phoneme sequence data, which is a sequence of phonemes that make up text data, as input, performs encoding on the phoneme sequence data, and outputs the data after encoding.

[0034] The estimated mean-variance data acquisition step obtains data corresponding to the estimated mean and estimated variance of the duration length of each phoneme included in the phoneme sequence data by performing a convolution operation on the data obtained in the encoding step.

[0035] The noise addition step involves adding Gaussian noise to the ground truth data of the duration of each phoneme to obtain the noise-added data.

[0036] The loss calculation step calculates the loss for phoneme duration based on the estimated mean, estimated variance, and noise-added data.

[0037] The parameter update step updates the parameters of the encoder, duration estimation unit, and decoder based on the loss calculated in the loss calculation step.

[0038] In this learning method, the loss is calculated by adding Gaussian noise n to the aligned target phoneme duration to be estimated, so the variance of the estimated phoneme duration can be made finite.

[0039] Therefore, with this learning method, if training is performed without adding Gaussian noise, the variance of the phoneme duration becomes "0," and the problem of not being able to reproduce the elastic modulus of the phoneme (the rate at which the phoneme expands and contracts according to the speech speed) during prediction processing (speech synthesis processing) does not occur. In other words, with this learning method, since the learning process is performed using phoneme durations with Gaussian noise added, the variance of the phoneme duration can be made a finite value, and as a result, a learning model that can reproduce the elastic modulus of the phoneme (the rate at which the phoneme expands and contracts according to the speech speed) can be constructed during prediction processing (speech synthesis processing).

[0040] In other words, this learning method allows you to obtain a trained model (a trained model for speech synthesis) that performs stable and highly accurate speech synthesis processing while taking into account the elastic modulus of phonemes.

[0041] The seventh invention is an acoustic feature generation device comprising an encoder, a duration estimation processing unit, and a decoder.

[0042] The encoder takes phoneme sequence data, which is a sequence of phonemes that make up text data, as input, performs encoding on the phoneme sequence data, and outputs the encoded data.

[0043] The continuation length estimation processing unit obtains first data by performing a convolution operation on the data output from the encoder, and performs a process to estimate the continuation length of each phoneme included in the phoneme sequence data from the first data, and outputs the first data and data on the continuation length of each phoneme included in the phoneme sequence data.

[0044] The decoder obtains acoustic features by performing a decoding process using the data output from the encoder and the data output from the duration estimation processing unit.

[0045] The continuation length estimation processing unit comprises a mapping transformation layer, a range restriction processing unit, an activation processing unit, and a continuation length acquisition unit.

[0046] The mapping transformation layer obtains data from the first data set that corresponds to the estimated mean and estimated variance of the duration of each phoneme included in the phoneme sequence data.

[0047] The range restriction processing unit obtains the processed data as range-restricted variance data by executing a variance range restriction process that restricts the range of the estimated variance values ​​of the phoneme duration obtained by the mapping transformation layer.

[0048] The activation processing unit applies an activation function to the estimated average value data of phoneme durations obtained by the mapping transformation layer, and obtains the processed data as the estimated average data.

[0049] The continuation length acquisition unit obtains the estimated phoneme continuation length based on range-restricted variance data and estimated average data.

[0050] The acoustic feature generator calculates a loss for phoneme duration based on data obtained by adding Gaussian noise to the correct duration data for each phoneme, range-restricted variance data, and estimated mean data. Based on the calculated loss, it performs a learning process by updating the parameters of the encoder, duration estimation processing unit, and decoder, and the trained model obtained through this learning process is incorporated into the system.

[0051] This allows the acoustic feature generator to acquire acoustic features necessary for stable and highly accurate speech synthesis processing, while taking into account the elastic modulus of phonemes. [Effects of the Invention]

[0052] According to the present invention, it is possible to realize a speech synthesis processing device that performs stable and highly accurate speech rate conversion while taking into account the elastic modulus of phonemes. [Brief explanation of the drawing]

[0053] [Figure 1] A schematic diagram of the speech synthesis processing device 100 according to the first embodiment. [Figure 2]A schematic diagram of the encoder 2 of the speech synthesis processing device 100 according to the first embodiment. [Figure 3] A schematic diagram of the speech synthesis processing device 100 according to the first embodiment, showing the duration estimation processing unit 3, the noise addition processing unit 3A, and the phoneme duration loss acquisition unit 3B. [Figure 4] A schematic diagram of the decoder 4 of the speech synthesis processing device 100 according to the first embodiment. [Figure 5] A graph showing a histogram of the speaking speed in the learning set and the average speaking speed in the test set. [Figure 6] A diagram showing the audio comparison results (spectrogram). [Figure 7] A diagram showing the results of the sound quality evaluation. [Figure 8] A diagram showing the CPU bus configuration. [Modes for carrying out the invention]

[0054] [First Embodiment] The first embodiment will be described below with reference to the drawings.

[0055] <1.1: Configuration of the speech synthesis processing device> Figure 1 is a schematic diagram of the speech synthesis processing device 100 according to the first embodiment.

[0056] Figure 2 is a schematic diagram of the encoder 2 of the speech synthesis processing device 100 according to the first embodiment.

[0057] Figure 3 is a schematic diagram of the speech synthesis processing device 100 according to the first embodiment, showing the duration estimation processing unit 3, the noise addition processing unit 3A, and the phoneme duration loss acquisition unit 3B.

[0058] Figure 4 is a schematic diagram of the decoder 4 of the speech synthesis processing device 100 according to the first embodiment.

[0059] As shown in Figure 1, the speech synthesis processing device 100 comprises a text analysis unit 1, an encoder 2, a duration estimation processing unit 3, a noise addition processing unit 3A, a phoneme duration loss acquisition unit 3B, a decoder 4, an acoustic feature loss acquisition unit 5, an integrated loss acquisition unit 6, a loss evaluation unit 7, and a vocoder 8.

[0060] The text analysis unit 1 receives text data Din as input, performs text analysis on the text data, and obtains phoneme sequence data, which is sequence data of phonemes that make up the text data, and accent label sequence data, which is sequence data of accent labels of phonemes that make up the text data. The text analysis unit 1 then outputs the data containing the obtained phoneme sequence data and accent label sequence data as data D0 to the encoder 2. The phoneme sequence data is, for example, sequence data of phoneme indices that identify phonemes. The accent label sequence data is, for example, sequence data of data indicating whether a phoneme is an accented phoneme or not, using "1" or "0".

[0061] As shown in Figure 2, the encoder 2 comprises a first embedding layer 21, a second embedding layer 22, an integration unit 23, and a machine learning model unit 24.

[0062] The first embedding layer 21 receives phoneme sequence data (referred to as data D0_phnm) contained in data D0 output from the text analysis unit 1, performs an embedding process (a process to acquire embedded representation data) on the data D0_phnm, and outputs the embedded representation data acquired by this process as data D01_emb to the integration unit 23.

[0063] The second embedding layer 22 receives accent label sequence data (referred to as data D0_accL) contained in data D0 output from the text analysis unit 1, performs an embedding process (a process to acquire embedded representation data) on the data D0_accL, and outputs the embedded representation data acquired by this process as data D02_emb to the integration unit 23.

[0064] The integration unit 23 receives the data D01_emb output from the first embedding layer 21 and the data D02_emb output from the second embedding layer 22, integrates (for example, concatenates) both data, and outputs the integrated data as data D03 to the machine learning model unit 24.

[0065] The machine learning model unit 24 is a functional unit that includes, for example, a machine learning model using a neural network (for example, a machine learning model using Transformer). The machine learning model unit 24 receives data D03 output from the integration unit 23, processes the data D03 using the machine learning model, and obtains the processing result (data output from the machine learning model when data D03 is input to the machine learning model) as data D1. The machine learning model unit 24 then outputs the obtained data D1 to the continuation length estimation processing unit 3.

[0066] As shown in Figure 3, the continuation length estimation processing unit 3 comprises a convolutional layer 31, a mapping transformation layer 32, a data partitioning unit 33, a sigmoid transformation unit 34, an activation processing unit 35, a dispersion range restriction processing unit 36, a continuation length acquisition unit 37, and a selector SEL1.

[0067] The convolutional layer 31 receives data D1 output from encoder 2, performs a convolution operation on data D1 using a kernel of a predetermined size (for example, LConv (Lightweight Convolution) operation using a 3x1 kernel), and obtains the data after the convolution operation as data D2. The convolutional layer 31 outputs the obtained data D2 to decoder 4 and mapping transformation layer 32.

[0068] The mapping transformation layer 32 receives data D2 output from the convolutional layer 21, performs a mapping transformation on data D2 (for example, processing by a fully connected layer capable of dimensionality reduction), and obtains data D3. The mapping transformation layer 32 then outputs the obtained data D3 to the data partitioning unit 33. The mapping transformation layer 32 performs a mapping transformation on high-dimensional data (for example, 512-dimensional data) to 2-dimensional data, and data D3 is 2-dimensional data (data corresponding to the mean and variance values ​​of the phoneme duration).

[0069] The data splitting unit 33 receives the data D3 (two-dimensional data) output from the mapping transformation layer 32 and splits the data D3 into two one-dimensional data. The data splitting unit 33 outputs one of the split data as data D31 (data corresponding to the variance value of the phoneme duration) to the sigmoid transformation unit 34, and outputs the other split data as data D32 (data corresponding to the average value of the phoneme duration) to the activation processing unit 35.

[0070] The sigmoid transformation unit 34 receives the data D31 output from the data partitioning unit 33, performs a sigmoid transformation on the data 31 (a process of inputting the data D31 into a sigmoid function and obtaining the output value), and obtains the transformed data as data D4. The sigmoid transformation unit 34 then outputs the obtained data D4 to the distribution range restriction processing unit 36.

[0071] The activation processing unit 35 receives the data D31 output from the data splitting unit 33, performs processing on the data 31 using an activation function (for example, processing using the Softplus function), and obtains the processed data as data D_avg. The activation processing unit 35 then outputs the obtained data D_avg to the phoneme duration loss acquisition unit 3B and the duration acquisition unit 37.

[0072] The dispersion range regulation processing unit 36 ​​takes the data D4 output from the sigmoid conversion unit 34 and the lower limit V of the dispersion value of the phoneme duration and duration, respectively. min 2 And the upper limit V of the variance of the phoneme duration. max2 Enter the following. Note that the lower limit of the variance value of the phoneme duration is V. min 2 , and the upper limit V of the variance of the phoneme duration. max 2 This is to be output from a control unit (not shown) that controls each functional part of the speech synthesis processing device 100. The dispersion range restriction processing unit 36 ​​controls the lower limit V of the dispersion value of the phoneme duration length. min 2 , and the upper limit V of the variance of the phoneme duration. max 2 Using this, the distribution range restriction process is performed on data D4, and the processed data is obtained as data D_var. The distribution range restriction processing unit 36 ​​then outputs the obtained data D_var to the phoneme duration loss acquisition unit 3B and the duration acquisition unit 37.

[0073] During the prediction processing of the speech synthesis processing device 100, the duration acquisition unit 37 receives data D_var output from the dispersion range restriction processing unit 36, data D_avg output from the activation processing unit 35, and data T that specifies the target value of the phoneme duration during playback (speech synthesis output) (for example, the number of frames corresponding to a predetermined period). Data T is output from the control unit (not shown). The duration acquisition unit 37 then uses data D_var, data D_avg, and data T to perform a process to acquire (estimate) the duration of the phoneme, and outputs the data after this process as data D52 to selector SEL1.

[0074] Selector SEL1 is a 2-input, 1-output selector that can select and output either of two inputs using a switching signal sel1. Selector SEL1 receives data D_correct_dur and data D52 output from the duration acquisition unit 37, and a switching signal sel1 output from the control unit (not shown). Selector SEL1 (1) when the switching signal sel1 is "1" (during the learning process of the speech synthesis processing device 100), selects data D_correct_dur and outputs the selected data as data D_dur to the decoder 4, and (2) when the switching signal sel1 is "0" (during the prediction process of the speech synthesis processing device 100), selects data D52 and outputs the selected data as data D_dur to the decoder 4.

[0075] The noise addition processing unit 3A, during the learning process of the speech synthesis processing unit 100, uses data D_correct_dur, which is the correct data (training data) for the duration of phonemes, and the lower limit V of the variance value of the duration of phonemes. min 2 Enter the following: Data D_correct_dur and the lower limit V of the variance of the phoneme duration. min 2 This is output from the control unit (not shown) that controls each functional part of the speech synthesis processing device 100. The noise addition processing unit 3A performs a noise addition process on the data D_correct_dur and outputs the processed data as data D_ns to the phoneme duration loss acquisition unit 3B.

[0076] The phoneme duration loss acquisition unit 3B receives, during the learning process of the speech synthesis processing device 100, the data D_var output from the dispersion range restriction processing unit 36, the data D_avg output from the activation processing unit 35 (data D_var and data D_avg are collectively called data D_VA), and the data D_ns output from the noise addition processing unit 3A. The phoneme duration loss acquisition unit 3B uses data D_var, data D_avg, and data D_ns to obtain a loss value L based on the loss function for the duration of the phoneme. dur,ML The phoneme duration loss acquisition unit 3B then obtains (calculates) the acquired loss value L.dur,ML This is output to the integrated loss acquisition unit 6.

[0077] As shown in Figure 4, the decoder 4 comprises a learnable upsampling layer 41 and M convolutional layers, specifically the first convolutional layer 42-1 to the Mth convolutional layer 42-M, which are M (M: a natural number) convolutional layers.

[0078] As shown in Figure 4, the learnable upsampling layer 41 comprises an upsampling layer 411, a position embedding layer 412, an adder add1, and a weighting layer 413.

[0079] The upsampling layer 411 receives data D2 and data D_dur output from the duration estimation processing unit 3. Based on data D_dur (phoneme duration data), the upsampling layer 411 performs an upsampling process on data D2 so that the length (size) of the output data matches the length (size) of the frame sequence of the spectral data, which is the target data. The upsampling layer 411 then obtains the data after the upsampling process as data DD3. The upsampling layer 411 then outputs the obtained data DD3 to the adder add1.

[0080] The position embedding layer 412 receives the data D_dur output from the continuation length estimation processing unit 3, performs a position embedding process (a process to acquire embedded representation data of position information) on the data D_dur, and outputs the position embedding representation data acquired by this process as data D_pos to the adder add1.

[0081] The adder unit add1 receives the data DD3 output from the upsampling layer 411 and the data D_pos output from the position embedding layer 412, and performs a process of adding the two data. Then, the adder unit add1 outputs the data after the addition process as data DD4 to the weighting layer 413.

[0082] The weighting layer 413 receives the data DD4 output from the addition unit add1, performs a weighting process (for example, a process by one or more hidden layers of a neural network) on the data DD4, and obtains the processed data DD5. Then, the weighting layer 413 outputs the obtained data DD5 to the first convolutional layer 42-1.

[0083] The first convolutional layer 42-1 receives the data DD5 output from the weighting layer 413, performs a convolution process (for example, LConv (Lightweight Convolution) process using a kernel of a predetermined size, such as a 17×1 kernel) on the data DD5, and obtains the data after the convolution process as data D_sp1. Then, during the learning process of the speech synthesis processing device 100, the first convolutional layer 42-1 outputs the obtained data D_sp1 to the second convolutional layer 42-2 and the acoustic feature amount loss acquisition unit 5. Also, during the prediction process of the speech synthesis processing device 100, the first convolutional layer 42-1 outputs the obtained data D_sp1 to the second convolutional layer 42-2.

[0084] The k-th convolutional layer 42-k (k: natural number, 2≦k<M) receives the data D_sp k-1 output from the (k-1)-th convolutional layer 42-k-1, k-1 performs a convolution process (for example, LConv (Lightweight Convolution) process using a kernel of a predetermined size, such as a 17×1 kernel) on the data D_sp k and obtains the data after the convolution process as data D_sp k . Then, during the learning process of the speech synthesis processing device 100, the k-th convolutional layer 42-k outputs the obtained data D_sp k to the (k + 1)-th convolutional layer 42-k+1 and the acoustic feature amount loss acquisition unit 5. Also, during the prediction process of the speech synthesis processing device 100, the k-th convolutional layer 42-k outputs the obtained data D_sp

[0085] The Mth convolutional layer 42-M (M: natural number) outputs data D_sp from the M-1st convolutional layer 42-M-1. M-1 Enter the data D_sp M-1 For this, a convolution operation is performed using a kernel of a predetermined size (for example, LConv (Lightweight Convolution) operation using a 17x1 kernel), and the data after the convolution operation is obtained as data D_sp M It is obtained as follows. Then, the Mth convolutional layer 42-M processes the acquired data D_sp during the learning process of the speech synthesis processing device 100. M The acquired data D_sp is output to the acoustic feature loss acquisition unit 5. In addition, the Mth convolutional layer 42-M processes the acquired data D_sp during the prediction processing of the speech synthesis processing device 100. M This is output to vocoder 8 as data Do_sp.

[0086] Note that the output data D_sp of the first convolutional layer 42-1 to the Mth convolutional layer 42-M are as follows: i (i: natural number, 1 ≤ i ≤ M) is assumed to be data of the same dimension as the correct data (training data) D_correct_spec for acoustic features.

[0087] The acoustic feature loss acquisition unit 5, during the learning process of the speech synthesis processing device 100, acquires the output data D_sp of each of the first convolutional layers 42-1 to the Mth convolutional layer 42-M. i The unit receives M output data (D_sp) (where i is a natural number, 1 ≤ i ≤ M) and the correct acoustic feature data (training data) D_correct_spec as input. The correct acoustic feature data (training data) D_correct_spec is output from the control unit (not shown). The acoustic feature loss acquisition unit 5 receives the output data D_sp from each of the first convolutional layers 42-1 to the Mth convolutional layer 42-M. i Using the correct data (training data) D_correct_spec for acoustic features, the loss value L is calculated using the loss function for acoustic features. SP The acoustic feature loss acquisition unit 5 then obtains (calculates) the acquired loss value L.SP This is output to the integrated loss acquisition unit 6.

[0088] The integrated loss acquisition unit 6, during the learning process of the speech synthesis processing device 100, outputs the loss value L from the acoustic feature loss acquisition unit 5. SP The loss value L is output from the phoneme duration loss acquisition unit 3B. dur,ML Enter the following, and the loss value L SP And the loss value L dur,ML The integrated loss obtained by combining the two is acquired, and the acquired integrated loss data is output to the loss evaluation unit 7 as data L.

[0089] The loss evaluation unit 7 receives data L (integrated loss data) output from the integrated loss acquisition unit 6 during the learning process of the speech synthesis processing device 100, and determines whether or not to terminate the learning process based on the data L (integrated loss data).

[0090] During the prediction processing of the speech synthesis processing device 100, the vocoder 8 receives the data Do_sp output from the decoder 4, performs vocoder processing on the data Do_sp, obtains the corresponding audio waveform data Dout, and outputs it. The vocoder 8 can be implemented, for example, by a vocoder using a neural network (neural vocoder).

[0091] <1.2: Operation of the speech synthesis processing device> The operation of the speech synthesis processing device 100 configured as described above will be explained below. In the following, the operation of the speech synthesis processing device 100 will be explained in two parts: (1) learning process and (2) prediction process (speech synthesis process).

[0092] (1.2.1: Learning Process) First, let's explain the learning process of the speech synthesis processing device 100.

[0093] Text data Din is input to the text analysis unit 1, and the text analysis unit 1 performs text analysis on the text data Din, thereby obtaining phoneme sequence data, which is the sequence data of phonemes that make up the text data Din, and accent label sequence data, which is the sequence data of accent labels of phonemes that make up the text data.

[0094] The data, including the phoneme sequence data and accent label sequence data acquired by the text analysis unit 1, is output to the encoder 2 as data D0. The phoneme sequence data is, for example, a sequence of phoneme indices that identify phonemes. The accent label sequence data is, for example, a sequence of data indicating whether a phoneme is an accented phoneme or not, using "1" or "0".

[0095] The first embedding layer 21 of the encoder 2 receives the phoneme sequence data D0_phnm contained in the data D0 output from the text analysis unit 1, performs an embedding process (a process to acquire embedded representation data) on the data D0_phnm, and outputs the embedded representation data acquired by this process as data D01_emb to the integration unit 23.

[0096] The second embedding layer 22 of encoder 2 receives accent label data D0_accL included in data D0 output from text analysis unit 1, performs embedding processing (process to acquire embedded representation data) on data D0_accL, and outputs the embedded representation data acquired by this process as data D02_emb to integration unit 23.

[0097] The integration unit 23 receives the data D01_emb output from the first embedding layer 21 and the data D02_emb output from the second embedding layer 22, integrates (for example, concatenates) both data, and outputs the integrated data as data D03 to the machine learning model unit 24.

[0098] The machine learning model unit 24 (for example, a functional unit including a machine learning model using a Transformer) receives the data D03 output from the integration unit 23, processes the data D03 using the machine learning model, and obtains the processing result (data output from the machine learning model when data D03 is input to the machine learning model) as data D1. The machine learning model unit 24 then outputs the obtained data D1 to the continuation length estimation processing unit 3.

[0099] The convolutional layer 31 of the continuation length estimation processing unit 3 receives data D1 output from the encoder 2, performs a convolution operation on the data D1 using a kernel of a predetermined size (for example, LConv (Lightweight Convolution) operation using a 3x1 kernel), and obtains the data after the convolution operation as data D2. The convolutional layer 31 then outputs the obtained data D2 to the decoder 4 and the mapping transformation layer 32.

[0100] The mapping transformation layer 32 performs a mapping transformation (for example, processing by a fully connected layer capable of dimensionality reduction) on the data D2 output from the convolutional layer 21 and obtains data D3. The mapping transformation layer 32 then outputs the obtained data D3 to the data partitioning unit 33. The mapping transformation layer 32 performs a mapping transformation on high-dimensional data (for example, 512-dimensional data) to 2-dimensional data, and data D3 is 2-dimensional data (data corresponding to the mean and variance values ​​of the phoneme duration).

[0101] The data splitting unit 33 splits the data D3 (two-dimensional data) output from the mapping transformation layer 32 into two one-dimensional data. The data splitting unit 33 outputs one of the split data as data D31 (data corresponding to the variance value of the phoneme duration) to the sigmoid transformation unit 34, and outputs the other split data as data D32 (data corresponding to the average value of the phoneme duration) to the activation processing unit 35.

[0102] The sigmoid conversion unit 34 performs sigmoid conversion (a process of inputting the data D31 to the sigmoid function and obtaining an output value on the data D31 output from the data division unit 33), and obtains the data after the sigmoid conversion as data D4. Then, the sigmoid conversion unit 34 outputs the acquired data D4 to the dispersion range regulation processing unit 36.

[0103] The activation processing unit 35 performs processing by an activation function (for example, processing by the Softplus function) on the data D31 output from the data division unit 33, and obtains the processed data as data D_avg. Then, the activation processing unit 35 outputs the acquired data D_avg to the phoneme duration loss acquisition unit 3B.

[0104] The dispersion range regulation processing unit 36 receives the data D4 output from the sigmoid conversion unit 34, the lower limit value V of the dispersion value of the phoneme duration, min 2 and the upper limit value V of the dispersion value of the phoneme duration. max 2 Note that the lower limit value V of the dispersion value of the phoneme duration, min 2 and the upper limit value V of the dispersion value of the phoneme duration max 2 are output from a control unit (not shown) that controls each functional unit of the voice synthesis processing device 100. The dispersion range regulation processing unit 36 uses the lower limit value V of the dispersion value of the phoneme duration min 2 and the upper limit value V of the dispersion value of the phoneme duration max 2 to perform dispersion range regulation processing on the data D4. Specifically, the dispersion range regulation processing unit 36 D_var = D4 × (V max 2 - V min 2 ) + V min 2 D4: Output data of the sigmoid conversion unit 34 (D4: real number, 0 ≤ D4 ≤ 1) Execute the process corresponding to it and obtain the data D_var after the dispersion range control process. As a result, the data D_var (dispersion value of phoneme duration) after the dispersion range control process satisfies V min 2 ≦D_var≦V max 2 and is regulated within a range that satisfies

[0105] Then, the dispersion range control unit 36 outputs the data D_var obtained as described above to the phoneme duration loss acquisition unit 3B.

[0106] The noise addition processing unit 3A inputs the correct data (teacher data) D_correct_dur of the phoneme duration output from the control unit (not shown) and the lower limit value V of the dispersion value of the phoneme duration output from the control unit (not shown). min 2 The noise addition processing unit 3A executes a process of adding noise to the data D_correct_dur. Specifically, the noise addition processing unit 3A adds Gaussian noise n to the correct data d k of the duration of the k-th phoneme (time duration), that is, D_ns=d k +n n~N(0,V min 2 ) d k : Correct data of the duration of the k-th phoneme (time duration) N(μ,σ 2 ): Gaussian distribution with mean μ and variance σ 2 By executing the process corresponding to the data D_ns (=d k +n) of the duration of the k-th phoneme with Gaussian noise added is obtained.

[0107] Note that the phoneme sequence data obtained by the processing of the text analysis unit 1 from the input data Din to the speech synthesis processing device 100 is composed of K phonemes, and the duration of the k-th phoneme is represented as d k (k: integer, 1≦k≦K).

[0108] The phoneme duration loss acquisition unit 3B receives data D_var output from the dispersion range restriction processing unit 36, data D_avg output from the activation processing unit 35, and data D_ns output from the noise addition processing unit 3A. The phoneme duration loss acquisition unit 3B uses data D_var, data D_avg, and data D_ns to obtain a loss value L based on the loss function for the duration of the phoneme. dur,ML The phoneme duration loss acquisition unit 3B obtains (calculates) the following. Specifically, the phoneme duration loss acquisition unit 3B

number

[0109] Since the loss for phoneme duration can be obtained (estimated) using a loss function for maximum likelihood estimation of a normal distribution, the phoneme duration loss acquisition unit 3B obtains the loss value for phoneme duration by performing a process corresponding to the above formula.

[0110] Then, the phoneme duration loss acquisition unit 3B receives the loss value L obtained by the above process. dur,ML This is output to the integrated loss acquisition unit 6.

[0111] Selector SEL1 selects the data D_correct_dur (correct signal for phoneme duration) output from the control unit (not shown) and outputs the selected data as data D_dur to decoder 4. During the learning process of the speech synthesis processing device 100, the control unit (not shown) outputs the signal value of the switching signal sel1 as "1" to selector SEL1, and selector SEL1 selects the data D_correct_dur according to the switching signal and outputs the selected data as data D_dur to decoder 4.

[0112] The upsampling layer 411 of the decoder 4 receives data D2 and data D_dur output from the duration estimation processing unit 3. Based on data D_dur (= D_correct_dur (correct phoneme duration data)), the upsampling layer 411 performs an upsampling process on data D2 so that the length (size) of the output data matches the length (size) of the frame sequence of the target spectral data, and obtains the upsampled data as data DD3. The upsampling layer 411 then outputs the obtained data DD3 to the adder add1.

[0113] The position embedding layer 412 receives the data D_dur output from the duration length estimation processing unit 3 and performs a position embedding process (a process to acquire embedded representation data of position information) on the said data D_dur. For example, the position embedding layer 412, (1) Frame position within a phoneme (within the duration of the phoneme being processed), (2) Phoneme duration, and, (3) The position within a frame of the data to be processed (this position is identified, for example, by the fractional progression (for example, 1 / 10, 2 / 10, ..., 10 / 10 (when the denominator is "10"))). Obtain the embedded representation data (position embedded representation data) for (for example, by performing processing using position encoding as used in a transformer to obtain the position embedded representation data), and output the obtained position embedded representation data as data D_pos to the adder add1.

[0114] The adder add1 inputs the data DD3 output from the upsampling layer 411 and the data D_pos output from the position embedding layer 412, and performs a process of adding the two data. Then, the adder add1 outputs the data after the addition process as data DD4 to the weighting layer 413.

[0115] The weighting layer 413 inputs the data DD4 output from the adder add1, and performs a weighting process (for example, a process by one or more hidden layers of a neural network) on the data DD4 to obtain the processed data DD5. Then, the weighting layer 413 outputs the obtained data DD5 to the first convolutional layer 42-1.

[0116] The first convolutional layer 42-1 performs a convolution process (for example, LConv (Lightweight Convolution) process using a kernel of size 17×1) on the data DD5 output from the weighting layer 413, and obtains the data after the convolution process as data D_sp1. Then, the first convolutional layer 42-1 outputs the obtained data D_sp1 to the second convolutional layer 42-2 and the acoustic feature amount loss acquisition unit 5.

[0117] The k-th convolutional layer 42-k (k: natural number, 2≤k<M) performs a convolution process (for example, LConv (Lightweight Convolution) process using a kernel of size 17×1) on the data D_sp k-1 output from the (k-1)-th convolutional layer 42-k-1, and obtains the data after the convolution process as data D_sp k Then, the k-th convolutional layer 42-k outputs the obtained data D_sp kThis is output to the k+1th convolutional layer 42-k+1 and the acoustic feature loss acquisition unit 5.

[0118] The Mth convolutional layer 42-M (M: natural number) outputs data D_sp from the M-1st convolutional layer 42-M-1. M-1 For this, a convolution operation is performed using a kernel of a predetermined size (for example, LConv (Lightweight Convolution) operation using a 17x1 kernel), and the data after the convolution operation is obtained as data D_sp M It is obtained as follows. Then, the Mth convolutional layer 42-M processes the obtained data D_sp M This is output to the acoustic feature loss acquisition unit 5.

[0119] Note that the output data D_sp of the first convolutional layer 42-1 to the Mth convolutional layer 42-M are as follows: i (i: natural number, 1 ≤ i ≤ M) is assumed to be data of the same dimension as the correct data (training data) D_correct_spec for acoustic features.

[0120] The acoustic feature loss acquisition unit 5 obtains the output data D_sp from each of the first convolutional layers 42-1 to the Mth convolutional layer 42-M. i The unit receives M output data (D_sp) (where i is a natural number, 1 ≤ i ≤ M) and the correct acoustic feature data (training data) D_correct_spec as input. The correct acoustic feature data (training data) D_correct_spec is output from the control unit (not shown). The acoustic feature loss acquisition unit 5 receives the output data D_sp from each of the first convolutional layers 42-1 to the Mth convolutional layer 42-M. i Using the correct data (training data) D_correct_spec for acoustic features, the loss value L is calculated using the loss function for acoustic features. SPi It obtains (calculates) the acoustic feature loss acquisition unit 5, L SPi =Norm L1 (D_sp i -D_correct_spec) Norm L1(): Function to obtain the L1 norm By performing a process equivalent to this, the output data D_sp of the first convolutional layer 42-1 to the Mth convolutional layer 42-M is obtained. i Then, the loss between the acoustic feature and the correct data (training data) D_correct_spec is obtained (calculated). Furthermore, the acoustic feature loss acquisition unit 5,

number

[0121] Then, the acoustic feature loss acquisition unit 5 acquires the loss value L SP This is output to the integrated loss acquisition unit 6. In this embodiment, the acoustic features are Mel spectrograms, and the correct data (training data) D_correct_spec for the acoustic features are the acoustic features (Mel spectrograms) corresponding to the input data Din to the speech synthesis processing device 100.

[0122] The integrated loss acquisition unit 6 receives the loss value L output from the acoustic feature loss acquisition unit 5. SP The loss value L is output from the phoneme duration loss acquisition unit 3B. dur,ML Enter the following, and the loss value L SP And the loss value L dur,ML The integrated loss is obtained by combining the and . Specifically, the integrated loss acquisition unit 6,

number

[0123] The integrated loss acquisition unit 6 then outputs the acquired integrated loss data as data L to the loss evaluation unit 7.

[0124] The loss evaluation unit 7 receives the data L (integrated loss data) output from the integrated loss acquisition unit 6 and determines whether or not to terminate the learning process based on the data L (integrated loss data). Specifically, the loss evaluation unit 7 evaluates the integrated loss value L and determines whether or not it is necessary to continue the learning process. If the integrated loss value L is (1) not within a predetermined range, or (2) the amount of change in the integrated loss value L is not within a predetermined range, the loss evaluation unit 7 determines that it is necessary to continue the learning process in the speech synthesis processing device 100.

[0125] If it is determined that the learning process needs to be continued, the speech synthesis processing device 100 uses the integrated loss L as the loss value (error) and performs the learning process by backpropagating the error using the backpropagation method (propagating the error in the direction of the dotted arrow in Figure 1), and updating the weight coefficients (updating the parameters) in the learnable functional part of the speech synthesis processing device 100.

[0126] The above process is repeated until (1) the integrated loss value L falls within a predetermined range, or (2) the amount of change in the integrated loss value L falls within a predetermined range. In such cases, the loss evaluation unit 7 determines that it is not necessary to continue the learning process and terminates the learning process.

[0127] Then, in the speech synthesis processing device 100, the weight coefficients (parameters) in the learnable functional unit of the speech synthesis processing device 100, after the learning process has been completed, are fixed as optimization weight coefficients (optimization parameters). As a result, a trained model is constructed in the speech synthesis processing device 100.

[0128] (1.2.2: Predictive Processing (Speech Synthesis Processing)) Next, the prediction process (speech synthesis process) of the speech synthesis processing device 100 will be described.

[0129] The prediction process (speech synthesis process) of the speech synthesis processing device 100 is executed in the speech synthesis processing device 100 after the trained model has been constructed (the training process has been completed).

[0130] First, the text data Din to be synthesized is input to the text analysis unit 1. The text analysis unit 1 performs text analysis on the text data Din and obtains phoneme sequence data, which is the sequence data of phonemes that make up the text data Din, and accent label sequence data, which is the sequence data of accent labels of phonemes that make up the text data.

[0131] The data, including the phoneme sequence data and accent label sequence data acquired by the text analysis unit 1, is output to the encoder 2 as data D0.

[0132] In encoder 2, the same process as described in the learning process is performed, and data D1 is output from the machine learning model unit 24 of encoder 2 to the duration length estimation processing unit 3.

[0133] In the convolutional layer 31, mapping transformation layer 32, data partitioning unit 33, and sigmoid transformation unit 34 of the continuation length estimation processing unit 3, the same processing as during the learning process is performed.

[0134] The activation processing unit 35 performs processing on the data D31 output from the data splitting unit 33 using an activation function (for example, processing using the Softplus function), and obtains the processed data as data D_avg. The activation processing unit 35 then outputs the obtained data D_avg to the continuation length acquisition unit 37.

[0135] The dispersion range regulation processing unit 36 ​​takes the data D4 output from the sigmoid conversion unit 34 and the lower limit V of the dispersion value of the phoneme duration and duration, respectively. min 2 And the upper limit V of the variance of the phoneme duration. max 2 Enter the following. Note that the lower limit of the variance value of the phoneme duration is V. min2 , and the upper limit V of the variance of the phoneme duration. max 2 This is output from a control unit (not shown) that controls each functional part of the speech synthesis processing device 100. The dispersion range restriction processing unit 36 ​​sets the lower limit V of the dispersion value of the phoneme duration length. min 2 , and the upper limit V of the variance of the phoneme duration. max 2 Using this, the distribution range restriction process is performed on data D4. Specifically, the distribution range restriction processing unit 36 ​​performs the following: D_var=D4×(V max 2 -V min 2 )+V min 2 D4: Output data of the sigmoid transformer 34 (D4: real number, 0 ≤ D4 ≤ 1) The equivalent process is executed to obtain the data D_var after the distribution range restriction process. As a result, the data D_var (dispersion value of phoneme duration) after the distribution range restriction process is, V min 2 ≤D_var≦V max 2 It is restricted to the extent that it satisfies the following conditions.

[0136] The distribution range restriction processing unit 36 ​​then outputs the data D_var obtained as described above to the continuation length acquisition unit 37.

[0137] The continuation length acquisition unit 37 receives data D_var output from the dispersion range restriction processing unit 36, data D_avg output from the activation processing unit 35, and data T that specifies the target value of the phoneme continuation length during playback (speech synthesis output) (for example, the number of frames corresponding to a predetermined period). Data T is output from the control unit (not shown). Then, the continuation length acquisition unit 37 uses data D_var, data D_avg, and data T to perform a process to acquire (estimate) the phoneme continuation length. Specifically, the continuation length acquisition unit 37 performs the following:

number

[0138] Furthermore, by changing the data T, which consists of a number of frames corresponding to a predetermined period, the speech rate of the synthesized speech signal can be changed.

[0139] Then, the continuation length acquisition unit 37 obtains the continuation length (predicted continuation length) d of the acquired k-th phoneme. k The data containing this will be output to selector SEL1 as data D52.

[0140] The control unit (not shown) outputs the signal value of the switching signal sel1 as "0" to the selector SEL1. The selector SEL1 then selects the data D52 output from the continuation length acquisition unit 37 according to the switching signal sel1 and outputs it as data D_dur to the decoder 4.

[0141] Decoder 4 receives data D2 and data D_dur output from duration estimation processing unit 3 and performs the same processing as during the training process.

[0142] Then, decoder 4 outputs the processed data (output data from the Mth convolutional layer 42-M of decoder 4) to vocoder 8 as data Do_sp (acoustic feature (mel spectrogram) data).

[0143] The vocoder 8 receives the data Do_sp output from the decoder 4, performs vocoder processing on the data Do_sp, and obtains and outputs the audio waveform data Dout corresponding to the data Do_sp. This makes it possible to obtain (output) audio waveform data (synthesized audio waveform data) corresponding to the input data Din.

[0144] Summary As described above, in the speech synthesis processing device 100, during the learning process, the aligned target phoneme duration (d k ), mean 0, variance V min 2 Gaussian noise n(n~N(0,V) min 2 )) Add the loss (loss L for phoneme duration) dur,ML Since it calculates the phoneme duration (d k The lower limit of the variance of ) is V min 2 This can be done. In other words, the speech synthesis processing device 100 has a loss L for phoneme duration. dur,ML Since this is calculated by the loss function (equation 2) (the loss function for maximum likelihood estimation of the normal distribution), (d k +n-ξ k ) 2 The learning process proceeds in such a way that the value of d becomes smaller. k +n (n: Gaussian noise) is ξ k Learning progresses so that the phoneme duration (d k ) is a value with variability (the lower limit of the variance of phoneme duration is V min 2 The data will converge to a value that takes place within that data distribution.

[0145] Therefore, in the speech synthesis processing device 100, if learning is performed without adding Gaussian noise, the variance value of the phoneme duration becomes "0," and the problem of not being able to reproduce the elastic modulus of the phoneme (the rate of expansion and contraction of the phoneme that changes according to the speech speed) during prediction processing (speech synthesis processing) does not occur. In other words, since the speech synthesis processing device 100 performs learning processing using phoneme durations to which Gaussian noise has been added, the variance value of the phoneme duration can be made a finite value, and as a result, the elastic modulus of the phoneme (the rate of expansion and contraction of the phoneme that changes according to the speech speed) can be reproduced during prediction processing (speech synthesis processing).

[0146] Furthermore, in the speech synthesis processing device 100, the variance range restriction processing device 36 of the duration length estimation processing device 3 sets an upper limit value V of the variance value of the phoneme duration length. max 2 Since this can also be set, the problem that only the variance value of the duration of phonemes in silent parts becomes large, and that only the silent parts become compressed or compressed during speech speed conversion (when speech speed conversion is performed and speech is synthesized), can be resolved. In other words, in the speech synthesis processing device 100, the variance range restriction processing device 36 of the duration estimation processing device 3 sets the upper limit value V of the variance value of phoneme duration. max 2 Since this can also be set, the variance value of the phoneme duration in silent parts can also be set to the upper limit V. max 2 The following can be achieved, and as a result, silent parts can be appropriately played back (speech synthesis) during speech rate conversion (when speech rate conversion is performed and speech is synthesized).

[0147] Furthermore, in the speech synthesis processing device 100, the learnable upsampling layer 41 of the decoder 4 performs upsampling processing according to the phoneme duration data D_dur, and the weighting layer performs weighted addition processing (parameter updateable processing) using learnable real number weighting coefficients, so that the speech rate conversion speed can be a real number. For example, in the speech synthesis processing device 100, during the prediction processing, by setting the data T of the number of frames for a predetermined time to a real number such as "4.7" (4.7 frames), the elastic modulus ρ of the phoneme duration can be calculated using (Equation 5). k This is obtained, and furthermore, the phoneme duration d kThe phoneme duration d is obtained. k By performing processing by decoder 4 and vocoder 8, it is possible to obtain speech synthesis waveform data in which the speech rate conversion speed is expressed as a real value.

[0148] Furthermore, since the speech synthesis processing device 100 does not use an attention mechanism, it can perform stable prediction processing (speech synthesis processing) without phenomena such as speech being interrupted or stopped midway.

[0149] As described above, the speech synthesis processing device 100 can perform speech synthesis processing that enables stable and highly accurate speech rate conversion while taking into account the elastic modulus of phonemes.

[0150] <<Experimental Results>> To demonstrate the effectiveness of the present invention (speech synthesis processing device 100), an evaluation experiment was conducted on speech rate converted speech. The training set consisted only of speech data spoken at normal speed. The test set included speech data spoken at normal speed, as well as speech data actually spoken at high and low speeds. Then, the speech synthesized with speech rate converted by neural speech synthesis (speech synthesis by speech synthesis processing device 100) was compared with the speech of the test data. The histogram of the speech rate of the training set and the average speech rate of the test set are shown in Figure 5. In the comparative experiment, the vocoder 8 of the speech synthesis processing device 100 adopted the Multi-stream HiFI-GAN described in the following document (Document A). (Reference A): T. Okamoto, T. Toda, and H. Kawai, "Multi-stream HiFi-GAN with data-driven waveform decomposition," in Proc. ASRU, Dec. 2021, (accepted, in press). Figure 6 shows the spectrograms of the original sound and the speech after speech rate conversion of the test set. Figure 6(a) is the spectrogram of the original sound at high speed. Figure 6(b) is the spectrogram of the original sound at normal speed. Figure 6(c) is the spectrogram of the slow speech (speech acquired by speech synthesis processing) using the conventional method (the method of Non-Patent Literature 2 (Parallel Tacotoron 2)). Figure 6(d) is the spectrogram of the original sound at low speed. Figure 6(e) is the slow speech (speech acquired by speech synthesis processing in the speech synthesis processing device 100) using the present invention (speech synthesis processing device 100).

[0151] As shown in Figure 6, the conventional method (the method described in Non-Patent Literature 2 (Parallel Tacotoron 2)) uniformly expands and contracts the phoneme duration regardless of the phoneme type, and therefore fails to reproduce short pauses that occur only under low-speed conditions (when playing back speech at low speeds). However, the present invention (speech synthesis processing device 100) performs speech synthesis processing considering the elastic modulus of each phoneme, and thus it can be confirmed that short pauses are reproduced.

[0152] Figure 7 shows the sound quality evaluation results. Figure 7(A) shows the sound quality evaluation results at normal speed, Figure 7(B) shows the sound quality evaluation results at high speed (when the audio playback period is approximately 0.78 times that of normal speed), and Figure 7(C) shows the sound quality evaluation results at low speed (when the audio playback period is approximately 1.42 times that of normal speed). The codes (a) to (g) used to identify the data in the graphs of Figure 7 are as follows. (a) Data about the original sound (b) Data on speech obtained by speech synthesis using analysis methods (c) Data on speech acquired by speech synthesis using AR Tacotron2 with an attention mechanism capable of speech rate conversion (allowing input of playback speed). (d) Data on speech acquired using a Transformer-type TTS capable of speech rate conversion (allowing input of playback speed). (e) Data on speech acquired by speech synthesis using Parallel Tacotron2 (f) Data on speech obtained using speech synthesis with Parallel Tacotron2, which is capable of speech rate conversion (allows input of playback speed). (g) Data regarding the speech obtained by speech synthesis using the present invention (speech synthesis processing device 100) As shown in Figure 7, it can be confirmed that the present invention (speech synthesis processing device 100) enables speech rate conversion with significantly improved sound quality under both high-speed and low-speed conditions. In particular, under high-speed conditions (when playing back speech at high speed), the speech synthesized by the present invention has a MOS value (MOS: Mean Opinion Score) that is similar to that of the original sound. This suggests that the speech synthesized by the present invention at high speed may be easier to understand than actual tongue twisters (tongue twisters actually spoken by people).

[0153] As described above, the effectiveness of the present invention can be confirmed through comparative experiments.

[0154] [Other embodiments] In the above embodiment, it is assumed that the machine learning model unit 24 of the encoder 2 in the speech synthesis processing device 100 includes a transformer model (transformer-type neural network model), but it is not limited to this, and for example, the encoder 2 may be an encoder implemented using an RNN (Recurrent Neural Network).

[0155] Furthermore, the encoder 2 of the speech synthesis processing device 100 shown in the above embodiment is just one example, and an encoder with a different configuration may be used.

[0156] Furthermore, the loss function shown in the above embodiment as the loss function for performing the learning process of the speech synthesis processing device 100 is just one example, and other loss functions may be used to perform the learning process of the speech synthesis processing device 100.

[0157] Furthermore, in the speech synthesis processing device 100 described in the above embodiment, each block may be individually integrated into a single chip using a semiconductor device such as an LSI, or it may be integrated into a single chip including part or all of the blocks.

[0158] Although we have used the term LSI here, depending on the degree of integration, they may also be called IC, system LSI, super LSI, or ultra LSI.

[0159] Furthermore, the method of integrated circuit implementation is not limited to LSIs; it may also be implemented using dedicated circuits or general-purpose processors. FPGAs (Field Programmable Gate Arrays), which can be programmed after LSI manufacturing, or reconfigurable processors, which allow for the reconfiguration of the connections and settings of the circuit cells within the LSI, may also be used.

[0160] Furthermore, some or all of the processing of each functional block in each of the above embodiments may be implemented by a program. And some or all of the processing of each functional block in each of the above embodiments is performed by the central processing unit (CPU) in a computer. The programs for each of these processes are stored in a storage device such as a hard disk or ROM, and are read from the ROM or RAM and executed.

[0161] Furthermore, each of the processes in the above embodiment may be implemented by hardware, or by software (including cases where it is implemented together with an OS (operating system), middleware, or a predetermined library). It may also be implemented by a combination of software and hardware processing.

[0162] For example, when each functional part of the above embodiment is implemented by software, the hardware configuration shown in Figure 8 (for example, a hardware configuration in which a CPU, GPU, ROM, RAM, input unit, output unit, communication unit, storage unit (for example, a storage unit implemented by an HDD, SSD, etc.), and an external media drive are connected by a bus) may be used to implement each functional part by software processing.

[0163] Furthermore, when each of the functional units of the above embodiment is implemented by software, the software may be implemented using a single computer having the hardware configuration shown in Figure 8, or it may be implemented using distributed processing with multiple computers.

[0164] Furthermore, the execution order of the processing method in the above embodiments (including modified examples) is not necessarily limited to the description of the above embodiments, and the execution order can be changed without departing from the spirit of the invention.

[0165] A computer program that causes a computer to execute the method described above, and a computer-readable recording medium on which such program is recorded, are included in the scope of the present invention. Examples of computer-readable recording media include flexible disks, hard disks, CD-ROMs, MOs, DVDs, DVD-ROMs, DVD-RAMs, high-capacity DVDs, next-generation DVDs, and semiconductor memory.

[0166] The above-mentioned computer program is not limited to one recorded on the above-mentioned recording medium, but may also be transmitted via telecommunications lines, wireless or wired communication lines, networks such as the Internet, etc.

[0167] Furthermore, in this specification and in the claims, "optimization" means bringing something to its best state, and the parameters for "optimizing" a system (model) are the parameters at which the value of the system's objective function becomes the optimal value. The "optimal value" is the maximum value when the system performs better as the value of the system's objective function increases, and the minimum value when the system performs better as the value of the system's objective function decreases. The "optimal value" may also be an extremum. The "optimal value" may also allow for a predetermined error (measurement error, quantization error, etc.) and may be a value that falls within a predetermined range (a range that can be considered to have converged sufficiently).

[0168] It should be noted that the specific configuration of the present invention is not limited to the embodiments (including modified examples) described above, and various changes and modifications are possible without departing from the spirit of the invention. [Explanation of Symbols]

[0169] 100 Speech synthesis processing device 2. Encoder Multistream Generation Unit 3. Continuation length estimation processing unit 4 Decoders

Claims

1. An encoder that takes phoneme sequence data, which is a sequence of phonemes that make up text data, as input, performs encoding on the phoneme sequence data, and outputs the encoded data, A convolution processing unit that obtains first data by performing a convolution process on the data output from the encoder, and performs a process to estimate the duration of each phoneme included in the phoneme sequence data from the first data, the convolution processing unit that outputs the first data and data on the duration of each phoneme included in the phoneme sequence data, A decoder that obtains acoustic features by performing a decoding process using the first data and the data output from the duration estimation processing unit, A vocoder that generates an audio waveform based on acoustic features from the decoder, Equipped with, The aforementioned duration length estimation processing unit, A mapping transformation layer that obtains data corresponding to the estimated mean value and estimated variance of the duration of each phoneme included in the phoneme sequence data from the first data, A range restriction processing unit obtains the processed data as range-restricted variance data by performing a variance range restriction process that restricts the range of the estimated variance value of the phoneme duration obtained by the mapping transformation layer, An activation processing unit performs an activation function on the estimated average value data of phoneme durations obtained by the mapping transformation layer, and obtains the processed data as estimated average data. A duration acquisition unit that acquires the estimated phoneme duration based on the range-restricted dispersion data and the estimated average data, It has, The encoder, the duration estimation processing unit, and the decoder can each be equipped with a learning model that can be trained by updating its parameters, or a pre-trained model of the learning model. Based on data obtained by adding Gaussian noise to the correct duration data of each phoneme, the range-restricted variance data, and the estimated average data, a loss for phoneme duration is calculated. Based on the calculated loss, the parameters of the encoder, the duration estimation processing unit, and the decoder are updated, thereby executing a learning process, and the trained model obtained through this learning process can be incorporated. Speech synthesis processing device.

2. The aforementioned phoneme sequence data consists of K phonemes (K: a natural number), from the first to the kth phoneme, and the range-restricted variance data of the estimated mean duration of the kth phoneme is σ ｋ ２ Let k be a natural number, 1 ≤ k ≤ K, and the estimated average data of the duration of the k-th phoneme be ξ ｋ Let k be a natural number, 1 ≤ k ≤ K. The range regulation processing unit uses the upper limit value V ｍａｘ ２ and the lower limit value V ｍｉｎ ２ as inputs, and executes the range regulation processing so that the range of the estimated variance value of the phoneme duration satisfies V ｍｉｎ ２ ≦ σ ｋ ２ ≦ V ｍａｘ ２ ​ If n is the Gaussian noise added to the correct data for the duration of each phoneme, then the Gaussian noise n is: n~N(0,V ｍｉｎ ２ ) N(μ, σ) ２ ): mean μ, variance σ ２ Gaussian distribution Set to satisfy, The speech synthesis processing apparatus according to claim 1.

3. The learning process is executed by updating the parameters of the encoder, the duration estimation processing unit, and the decoder based on an integrated loss obtained by integrating the loss for the acoustic features, which is defined based on the norm of the error between the acoustic features acquired by the decoder and the ground truth acoustic features, which are acoustic features corresponding to the phoneme sequence data, and the loss for the phoneme duration. The speech synthesis processing apparatus according to claim 1 or 2.

4. The aforementioned phoneme sequence data consists of K phonemes (K: a natural number), from the first to the kth phoneme, and the range-restricted variance data of the estimated mean duration of the kth phoneme is σ ｋ ２ Let k be a natural number, 1 ≤ k ≤ K, and the estimated average data of the duration of the k-th phoneme be ξ ｋ Let k be a natural number, 1 ≤ k ≤ K, and let T be the number of frames in a given period. The aforementioned continuation length acquisition unit, By performing a process equivalent to this, the estimated duration d of the k-th phoneme is determined. ｋ Obtain, The decoder obtains estimated acoustic feature data by performing a decoding process based on the estimated duration data of the phoneme estimated by the duration acquisition unit and the first data. A speech synthesis processing device according to any one of claims 1 to 3.

5. The decoder is a learnable functional unit and includes a learnable upsampling layer that performs upsampling processing on the first data based on the estimated duration data of phonemes estimated by the duration acquisition unit. The speech synthesis processing apparatus according to claim 4.

6. A learning method for performing learning processing on a speech synthesis processing device according to any one of claims 1 to 5, An encoding step that takes phoneme sequence data, which is a sequence of phonemes that make up text data, as input, performs an encoding process on the phoneme sequence data, and outputs the data after the encoding process, An estimated mean-variance data acquisition step is performed on the data obtained in the encoding step to obtain data corresponding to the estimated mean and estimated variance of the duration length of each phoneme included in the phoneme sequence data. A noise addition step involves adding Gaussian noise to the correct data of the duration of each phoneme to obtain noisy data, A loss calculation step that calculates the loss for phoneme duration based on the estimated mean value, the estimated variance value, and the noise-added data, A parameter update step in which the parameters of the encoder, the duration estimation processing unit, and the decoder are updated based on the loss calculated in the loss calculation step, A learning method that includes [the following features].

7. An encoder that takes phoneme sequence data, which is a sequence of phonemes that make up text data, as input, performs encoding on the phoneme sequence data, and outputs the encoded data, A convolution processing unit that obtains first data by performing a convolution process on the data output from the encoder, and performs a process to estimate the duration of each phoneme included in the phoneme sequence data from the first data, the convolution processing unit that outputs the first data and data on the duration of each phoneme included in the phoneme sequence data, A decoder that obtains acoustic features by performing a decoding process using the first data and the data output from the duration estimation processing unit, Equipped with, The aforementioned duration length estimation processing unit, A mapping transformation layer that obtains data corresponding to the estimated mean value and estimated variance of the duration of each phoneme included in the phoneme sequence data from the first data, A range restriction processing unit obtains the processed data as range-restricted variance data by performing a variance range restriction process that restricts the range of the estimated variance value of the phoneme duration obtained by the mapping transformation layer, An activation processing unit performs an activation function on the estimated average value data of phoneme durations obtained by the mapping transformation layer, and obtains the processed data as estimated average data. A duration acquisition unit that acquires the estimated phoneme duration based on the range-restricted dispersion data and the estimated average data, It has, Based on data obtained by adding Gaussian noise to the correct duration data of each phoneme, the range-restricted variance data, and the estimated average data, a loss for phoneme duration is calculated, and based on the calculated loss, the parameters of the encoder, the duration estimation processing unit, and the decoder are updated to perform a learning process, and the trained model obtained through this learning process is installed. Acoustic feature generation device.

Images