GENERATION METHOD, GENERATION PROGRAM, AND GENERATION DEVICE

Abstract

This generation device generates intermediate representation information of a subband signal on the basis of the acoustic feature amount of a speech waveform, generates simultaneously a plurality of subband signals corresponding to a plurality of different times and a plurality of different bands by inputting the intermediate representation information to a plurality of probability distribution generation models that correspond to the number of channels of the subband signal and the number of samples to be generated simultaneously, and output information relating to the subband signals corresponding to the times and bands respectively assigned thereto, and generates a speech waveform on the basis of the plurality of subband signals.

Description

[Technical field]

[0001] The present invention relates to a generation method, a generation program, and a generation device. [Background technology]

[0002] In speech synthesis, the module that converts acoustic features such as spectrum and pitch, which indicates the pitch of the voice, into a speech waveform is called a vocoder. There are two main ways to implement a vocoder. The first is a signal processing method, and well-known methods include STRAIGHT and WORLD (see, for example, Non-Patent Documents 1 and 2). These methods use a mathematical model to represent the conversion from acoustic features to a speech waveform, so they do not require learning and have high processing speeds, but the quality of the analyzed and resynthesized speech is inferior to natural speech.

[0003] Another method that has been proposed is a neural network technique (neural vocoder) typified by WaveNet (see, for example, Patent Document 1). This technique can synthesize voice with a quality comparable to natural voice, but it operates slower than a signal processing vocoder due to the large amount of calculation required. Normally, a neural network must be forward propagated once to predict one voice sample, so real-time operation is difficult when implemented as is.

[0004] There are two main approaches to reduce the amount of calculations in neural vocoders and to operate them in real time, especially on a CPU. One is to reduce the calculation cost per forward propagation of a neural network, and examples of this include WaveRNN (for example, Patent Document 2), which replaces the huge convolutional neural network (CNN: Convolutional Neural Network) used in WaveNet with a small recurrent neural network (RNN: Recurrent Neural Network), and LPCNet (for example, Non-Patent Document 3), which utilizes linear predictive analysis (LPC), a knowledge of signal processing, in the process of generating voice waveforms.

[0005] The other is a method for reducing the number of times of forward propagation itself. It has been disclosed that the number of times of forward propagation can be reduced by changing the prediction target from the speech waveform to a short sequence divided into bands using a pseudo-quadrature mirror filter (Pseudo-QMF) called a subband signal (for example, Non-Patent Document 4). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] International Publication No. 2018 / 048934 [Patent Document 2] International Publication No. 2019 / 155054 [Non-patent literature]

[0007] [Non-Patent Document 1] Hideki Kawahara, Ikuyo Masuda-Katsuuse and Alain de Cheveigne, “Restructuring speech representations using a pitch-adaptive time-frequency smoothing and an instantaneous-frequency-based F0 extraction: Possible role of a repetitive structure in sounds,” Speech Communication, vol. 27, no. 3-4, pp. 187-207, 1999. [Non-Patent Document 2] Masanori Morise, Fumiya Yokomori, Kenji Ozawa, “WORLD: a Vocoder-Based High-Quality Speech Synthesis System for Real-Time Applications,” IEICE transactions on information and systems, vol. E99-D, no. 7, pp. 1877-1884, 2016. [Non-Patent Document 3] Jean-Marc Valin and Jan Skoglund, “LPCNET: IMPROVING NEURAL SPEECH SYNTHESIS THROUGH LINEAR PREDICTION,” Proc. ICASSP, 2019, pp. 5891-5895 [Non-Patent Document 4] Chengzhu Yu, Heng Lu, Na Hu, Meng Yu, Chao Weng, Kun Xu, Peng Liu, Deyi Tuo, Shiyin Kang, Guangzhi Lei, Dan Su, Dong Yu, “DurIAN: Duration Informed Attention Network for Speech Synthesis”, Proc. INTERSPEECH 2020, pp. 2027-2031, 2020 Summary of the Invention [Problem to be solved by the invention]

[0008] Non-Patent Document 4 improves on Reference Document 2, using one sample of a subband signal instead of one sample of a speech waveform to predict the subband signal at the next time. In this case, if the subbands are expressed using B channels, the sequence length can be shortened to 1 / B, and the number of net propagations can be reduced to 1 / B. However, because samples are generated one by one based on past samples using an autoregressive structure such as an RNN, it is not possible to increase the speed by more than a maximum of B times.

[0009] The present invention has been made in consideration of the above, and has an object to provide a generation method, a generation program, and a generation device that can generate a speech waveform from acoustic features at high speed. [Means for solving the problem]

[0010] In order to solve the above-mentioned problems and achieve the objective, the computer includes an intermediate representation generation step of generating intermediate representation information of a subband signal based on acoustic features of an audio waveform, a subband signal generation step of simultaneously generating a plurality of subband signals corresponding to different times and different bands by inputting the intermediate representation information into a plurality of probability distribution generation models according to the number of channels of the subband signal and the number of samples to be simultaneously generated, the plurality of probability distribution generation models outputting information of the subband signal corresponding to the time and band assigned to each of the plurality of probability distribution generation models, and an audio waveform generation step of generating an audio waveform based on the plurality of subband signals. Effect of the Invention

[0011] According to the present invention, a speech waveform can be generated from acoustic features at high speed. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a functional block diagram of a generating device according to the first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a subband signal. [Diagram 3] FIG. 3 is a diagram for explaining the learning unit according to the first embodiment. [Figure 4] FIG. 4 is a diagram for explaining the generating unit according to the first embodiment. [Diagram 5] FIG. 5 is a flowchart of a process performed by the generating device according to the first embodiment during learning. [Figure 6] FIG. 6 is a flowchart of a process performed by the generating device according to the first embodiment at the time of generation. [Figure 7]FIG. 7 is a functional block diagram of a generating device according to the second embodiment. As shown in FIG. [Figure 8] FIG. 8 is a diagram for explaining the learning unit according to the second embodiment. [Figure 9] FIG. 9 is a diagram illustrating a generating unit according to the second embodiment. [Figure 10] FIG. 10 is a flowchart of a process performed by the generating device according to the second embodiment during learning. [Figure 11] FIG. 11 is a flowchart of a process performed by the generating device according to the second embodiment at the time of generation. [Figure 12] FIG. 12 is a functional block diagram of a generating device according to the third embodiment. As shown in FIG. [Figure 13] FIG. 13 is a diagram illustrating the learning unit according to the third embodiment. [Figure 14] FIG. 14 is a diagram illustrating a generating unit according to the third embodiment. [Figure 15] FIG. 15 is a flowchart of a process performed by the generating device according to the third embodiment during learning. [Figure 16] FIG. 16 is a flowchart of a process performed by the generating device according to the third embodiment at the time of generation. [Figure 17] FIG. 17 is a diagram illustrating an example of a computer that executes a generation program. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, embodiments of the generating method, generating program, and generating device disclosed in the present application will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to these embodiments. EXAMPLES

[0014] First, a configuration example of a generating device according to the present embodiment 1 will be described. Fig. 1 is a functional block diagram showing a configuration of a generating device according to the present embodiment 1. As shown in Fig. 1, the generating device 100 includes a communication control unit 110, an input unit 120, an output unit 130, a storage unit 140, and a control unit 150.

[0015] The communication control unit 110 is realized by a NIC (Network Interface Card) or the like, and controls communication between the control unit 150 and external devices via electric communication lines such as a LAN (Local Area Network) or the Internet.

[0016] The input unit 120 is realized using input devices such as a keyboard and a mouse, and inputs various instruction information such as a command to start processing to the control unit 150 in response to an input operation by an operator.

[0017] The output unit 130 is an output device that outputs information acquired from the control unit 150, and is realized by a display device such as a liquid crystal display, a printing device such as a printer, or the like.

[0018] The storage unit 140 has training speech waveform data 141, a first intermediate representation model 10, a second intermediate representation model 11, and a probability distribution generation model 12. The storage unit 140 is realized by a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk.

[0019] The training speech waveform data 141 includes data of a plurality of speech waveforms for training. The training speech waveform data 141 is used when training the first intermediate representation model 10, the second intermediate representation model 11, and the probability distribution generation model 12.

[0020] The first intermediate representation model 10 is a model that outputs an intermediate representation m of an acoustic feature when the acoustic feature calculated from a speech waveform is input. The first intermediate representation model 10 corresponds to, for example, a Neural Network (NN), a CNN, an RNN, a WaveRNN, etc. In the following description, the intermediate representation m of the acoustic feature is appropriately written as "intermediate representation m".

[0021] The acoustic features correspond to spectral information such as a spectrogram of a speech waveform and a mel-cepstrum, and prosodic information such as a fundamental frequency and a pitch frequency.

[0022] The intermediate representation of the acoustic features is information in which the sequence length of the acoustic features is expanded to be the same as the number of speech samples. As a method of realizing the first intermediate representation model 10 that generates the intermediate representation m, there is a method of arranging and expanding the vector of the acoustic features corresponding to one frame by the number of samples. In addition, when the first intermediate representation model 10 is realized by WaveRNN, a method of expanding while converting the features using one-dimensional CNN or two-dimensional CNN is adopted in order to take into account the continuity between previous and next frames.

[0023] The second intermediate representation model 11 is configured to generate an intermediate representation x of a subband signal when the intermediate representation m is input. t As will be described later, there are a plurality of second intermediate representation models 11. In the following description, the intermediate representation x t Then, we call it "intermediate representation x t " Intermediate representation x t The t in is the time index. For example, x t=-1、0 means that it is a vector that contains information on the time t=-1 and t=0. Here, the intermediate representation x t=-1,0 is the intermediate representation x t This information corresponds to the initial value of

[0024] FIG. 2 is a diagram showing an example of subband signals. The vertical axis of FIG. 2 corresponds to the Amplitude Response, and the horizontal axis corresponds to the Normalized Frequency. FIG. 2 shows a case where four subband signals sub1, sub2, sub3, and sub4 are generated by filtering an audio signal (full band signal). Subband signal sub1 is a low-frequency subband signal. Subband signal sub2 is a low- to mid-frequency subband signal. Subband signal sub3 is a mid- to high-frequency subband signal. Subband signal sub4 is a high-frequency subband signal.

[0025] The probability distribution model 12 is an intermediate representation x t Given the input, the probability distribution of the subband signal y t , b As will be described later, there are multiple probability distribution generation models 12. t , b where t is the time index and b is the channel index of the subband signal. t , b is the probability of generating a speech waveform sample value. When the speech waveform is expressed as a discrete value such as a μ-law signal, the probability distribution y t , b If the speech waveform is expressed as a continuous value, the probability distribution y t , b A Gaussian distribution, a beta distribution, or a mixed logistic distribution can be used as

[0026] 1, the control unit 150 includes a learning unit 151 and a generating unit 152. The control unit 150 corresponds to a CPU (Central Processing Unit) or the like.

[0027] The learning unit 151 uses the learning speech waveform data 141 to execute learning of the first intermediate representation model 10 , the second intermediate representation model 11 , and the probability distribution generation model 12 .

[0028] The generation unit 152 generates a speech waveform from acoustic features by using the trained first intermediate representation model 10, the second intermediate representation model 11, and the probability distribution generation model 12. The generation unit 152 may obtain information on the acoustic features from the input unit 120, or may obtain the information from an external device via the communication control unit 110.

[0029] Next, the learning unit 151 shown in Fig. 1 will be specifically described. Fig. 3 is a diagram for explaining the learning unit according to the first embodiment. As shown in Fig. 3, the learning unit 151 includes an acoustic feature calculation unit 20, a first intermediate representation model 10, a plurality of second intermediate representation models 11-n (n=1 to T / B-1), a plurality of probability distribution generation models 12-tb (t=1, , T / B, b=1, , B), a sampling unit 21-tb, a subband signal calculation unit 30, a loss calculation unit 31, and a model learning unit 32. Here, the sequence length of the voice is T, the number of channels of the subband signal is B, and the number of samples of the subband signal is N (=2).

[0030] The learning unit 151 inputs the speech waveform D1 stored in the learning speech waveform data 141 to the acoustic feature calculation unit 20. The acoustic feature calculation unit 20 calculates an acoustic feature D2 based on the speech waveform D1. The acoustic feature calculation unit 20 inputs the acoustic feature D2 to the first intermediate representation model 10.

[0031] When an acoustic feature D2 is input, the first intermediate representation model 10 outputs an intermediate representation m of the acoustic feature. The first intermediate representation model 10 outputs the intermediate representation m to the second intermediate representation models 11-1 to 11-n. The first intermediate representation model 10 outputs gradient information to the model learning unit 32.

[0032] When an intermediate representation m is input, each of the second intermediate representation models 11-1 to 11-n generates an intermediate representation x t For example, the second intermediate representation model 11-1 outputs the intermediate representation x t=-1,0 The second intermediate representation models 11-2 to 11-n calculate the intermediate representations x t When calculating the subband signal s t、b We further use the subband signal s t、b In the above, t is the time index and b is the channel index of the subband signal.

[0033] The second intermediate representation model 11-2 is the intermediate representation x t=1,2The second intermediate representation model 11-3 calculates the intermediate representation x t=3,4 and outputs it to probability distribution generation models 12-5-1 to 12-6-B (not shown).

[0034] In the following description, when there is no need to distinguish between the second intermediate representation models 11-1 to 11-n, they will be collectively referred to as the second intermediate representation model 11. The second intermediate representation model 11 outputs gradient information to the model learning unit 32.

[0035] Each probability distribution model 12-tb is an intermediate representation x t Given the input, the probability distribution y t , b For example, the probability distribution generation model 12-1-1 outputs the probability distribution y t=1 , b=1 Although not shown in the figure, the probability distribution generation models 12-1-2 to 12-1-(B-1) output the probability distribution y t , b to the subordinate sampling units 21-1-2 to 21-1-(B-1). t=1 , b=B to the sampling unit 21-1-B.

[0036] The probability distribution generation model 12-2-1 generates the probability distribution y t=2 , b=1 Although not shown in the figure, the probability distribution generation models 12-2-2 to 12-2-(B-1) output the probability distribution y t , b to the subordinate sampling units 21-2-2 to 21-2-(B-1). t=2 , b=B to the sampling unit 21-2-B.

[0037] The probability distribution generation model 12-3-1 generates the probability distribution y t=3 , b=1To the sampling unit 21-3-1, the probability distribution generation models 12-3-2 to 12-3-(B-1) output the probability distribution y t , b to the subordinate sampling units 21-3-2 to 21-3-(B-1). t=3 , b=B to the sampling unit 21-3-B.

[0038] The probability distribution generation model 12-4-1 generates the probability distribution y t=4 , b=1 Although not shown in the figure, the probability distribution generation models 12-4-2 to 12-4-(B-1) output the probability distribution y t , b to the subordinate sampling units 21-4-2 to 21-4-(B-1). t=3 , b=B to the sampling unit 21-4-B.

[0039] We will not explain the other probability distribution generation models, but each intermediate representation x t Given the input, the probability distribution y t , b to the subordinate sampling unit. In the following description, when there is no need to distinguish between the probability distribution generation models 12-tb, they will be collectively referred to as the probability distribution generation model 12. The probability distribution generation model 12 outputs gradient information to the model learning unit 32. The probability distribution generation model 12 outputs the probability distribution y t=1~T / B、b=1~B to the loss calculation unit 31.

[0040] Each sampling unit 21-tb detects a probability distribution y t , b When the input is a subband signal s t、b It should be noted that, when each probability distribution generation model 12-tb directly predicts a subband signal, the learning unit 151 does not need to use each sampling unit 21-tb.

[0041] The sampling unit 21-1-1 samples the subband signal s t=1、b=1 Although not shown in the figure, the sampling units 21-1-2 to 21-1-(B-1) calculate the corresponding subband signal s t、b The sampling unit 21-1-B calculates the subband signal s t=1、b=B Calculate.

[0042] The sampling unit 21-2-1 samples the subband signal s t=2、b=1 Although not shown in the figure, the sampling units 21-2-2 to 21-2-(B-1) calculate the corresponding sub-band signal s t、b The sampling unit 21-2-B calculates the subband signal s t=2、b=B Calculate.

[0043] As described above, the subband signals s t=1、b=1~B The sub-band signals s t=2、b=1~B is calculated. t=1、b=1~B , subband signal s t=2、b=1~B is input to the second intermediate representation model 11-2.

[0044] That is, a set of the probability distribution generation models 12-1-1 to 12-1-B and the sampling units 21-1-1 to 21-1-B, and a set of the probability distribution generation models 12-2-1 to 12-2-B and the sampling units 21-2-1 to 21-2-B generate one intermediate representation x t=-1,0 Then, the subband signals (s t=1、b=1~B , s t=2、b=1~B ) is generated.

[0045] The sampling unit 21-3-1 samples the subband signal s t=3、b=1 Although not shown in the figure, the sampling units 21-3-2 to 21-3-(B-1) calculate the corresponding sub-band signal s t、b The sampling unit 21-3-B calculates the subband signal s t=3、b=B Calculate.

[0046] The sampling unit 21-4-1 samples the subband signal s t=4、b=1 Although not shown in the figure, the sampling units 21-4-2 to 21-4-(B-1) calculate the corresponding sub-band signal s t、b The sampling unit 21-4-B calculates the sub-band signal s t=4、b=B Calculate.

[0047] As described above, the sub-band signals s t=3、b=1~B The sub-band signals s t=4、b=1~B is calculated. t=3、b=1~B , subband signal s t=4、b=1~B is input to the second intermediate representation model 11-3.

[0048] That is, a set of the probability distribution generation models 12-3-1 to 12-3-B and the sampling units 21-3-1 to 21-3-B, and a set of the probability distribution generation models 12-4-1 to 12-4-B and the sampling units 21-4-1 to 21-4-B generate one intermediate representation x t=1,2 Then, the subband signals (s t=3、b=1~B , s t=4、b=1~B ) is generated.

[0049] The explanation of the other sampling parts will be omitted, but each of them has a probability distribution y t , b When the input is a subband signal s t、b The subband signal s is calculated from each sampling unit 21-tb. t=1~T / B、b=1~B will be output.

[0050] Next, the subband signal calculation unit 30 will be described. The subband signal calculation unit 30 calculates a subband signal s t=1~T / B、b=1~BA quadrature mirror filter (QMF) or a pseudo-quadrature mirror filter (PQMF) can be used as the subband signal calculation unit 30. By using a quadrature mirror filter or a pseudo-quadrature mirror filter, the input speech waveform D1 is converted into a subband signal with a sequence length shortened to 1 / B.

[0051] The subband signal calculation unit 30 calculates the subband signal s t=1~T / B、b=1~B to the loss calculation unit 31.

[0052] The loss calculation unit 31 calculates the subband signal s t=1~T / B、b=1~B and the probability distribution y t=1~T / B、b=1~B For example, the loss value D3 is calculated based on the probability distribution y t=1~T / B、b=1~B and the subband signal s t=1~T / B、b=1~B This value indicates the error between the

[0053] When a categorical distribution is used as the probability distribution, the loss calculation unit 31 calculates the loss value D3 using cross entropy. When a Gaussian distribution, a beta distribution, or a mixed logistic distribution is used as the probability distribution, the loss calculation unit 31 calculates the loss value D3 using negative log likelihood.

[0054] The loss calculation unit 31 outputs the loss value D3 to the model learning unit 32.

[0055] The model learning unit 32 acquires the loss value D3, gradient information of the first intermediate representation model 10, gradient information of the second intermediate representation model 11, and gradient information of the probability distribution generation model 12, and performs machine learning on the parameters of the first intermediate representation model 10, the parameters of the second intermediate representation model 11, and the parameters of the probability distribution generation model 12 so as to reduce the loss value D3. For example, the model learning unit 32 performs machine learning by utilizing an error backpropagation algorithm.

[0056] Each processing unit included in the learning unit 151 repeatedly executes the above-mentioned process for multiple speech waveforms D1 included in the training speech waveform data 141, thereby obtaining a learned first intermediate representation model 10, a second intermediate representation model 11, and a probability distribution generation model 12.

[0057] Next, the generation unit 152 shown in Fig. 1 will be specifically described. Fig. 4 is a diagram for explaining the generation unit according to the first embodiment. As shown in Fig. 4, the generation unit 152 has a first intermediate representation model 10, a plurality of second intermediate representation models 11-n (n=1 to T / B-1), a plurality of probability distribution generation models 12-tb (t=1, , T / B, b=1, , B), a sampling unit 21-tb, and a voice waveform conversion unit 40. Here, the sequence length of the voice is T, the number of channels of the subband signal is B, and the number of samples of the subband signal is N (=2).

[0058] 3, when the second intermediate representation models 11-1 to 11-n are not distinguished from one another, they are collectively referred to as the second intermediate representation model 11. When the probability distribution generation model 12-tb is not distinguished from one another, they are collectively referred to as the probability distribution generation model 12.

[0059] The first intermediate representation model 10 , the second intermediate representation model 11 , and the probability distribution generation model 12 are assumed to have been trained by the training unit 151 .

[0060] The generation unit 152 inputs the acoustic feature D10 to the first intermediate representation model 10. When the acoustic feature D10 is input, the first intermediate representation model 10 outputs an intermediate representation m of the acoustic feature. The first intermediate representation model 10 outputs the intermediate representation m to the second intermediate representation models 11-1 to 11-n.

[0061] The second intermediate representation model 11, when the intermediate representation m is input, generates an intermediate representation x t to the probability distribution generation model 12. The explanation regarding the second intermediate representation model 11 is similar to the explanation regarding each of the second intermediate representation models 11-1 to 11-n given in FIG.

[0062] The probability distribution model 12 is an intermediate representation x t Given the input, the probability distribution y t , b to each sampling unit 21-tb. The description of the probability distribution generation model 12 is the same as the description of each probability distribution generation model 12-tb given in FIG.

[0063] The sampling unit 21-tb calculates the probability distribution y t , b When the input is a subband signal s t、b Each sampling unit 21-tb calculates the subband signal s t=1~T / B、b=1~B is output to the voice waveform conversion unit 40.

[0064] The speech waveform converter 40 converts the subband signal s t=1~T / B、b=1~B Based on this, the audio waveform z t=1~T The audio waveform conversion unit 40 converts the subband signals into audio waveforms by inverse conversion of a quadrature mirror filter or a pseudo quadrature mirror filter.

[0065] Next, an example of a processing procedure of the generating device 100 according to the embodiment 1 will be described. Fig. 5 is a flowchart showing a processing procedure during learning of the generating device according to the embodiment 1. As shown in Fig. 5, the learning unit 151 of the generating device 100 acquires a voice waveform from the learning voice waveform data 141 (step S101).

[0066] The acoustic feature calculation unit 20 of the learning unit 151 calculates the acoustic feature from the speech waveform (step S102). The learning unit 151 inputs the acoustic feature to the first intermediate representation model 10 and calculates an intermediate representation m of the acoustic feature (step S103).

[0067] The learning unit 151 inputs the intermediate representation m of the acoustic feature quantity to the second intermediate representation model 11 and obtains the intermediate representation x of the subband signal. t (Step S104). The learning unit 151 calculates the probability distribution y t、b Calculate the probability distribution y t、b Based on the subband signal st、b is calculated (step S105).

[0068] Meanwhile, the subband signal calculation unit 30 of the learning unit 151 calculates a subband signal based on the voice waveform (step S106).

[0069] The loss calculation unit 31 of the learning unit 151 calculates the subband signal s t=1~T / B、b=1~B and the probability distribution y t=1~T / B、b=1~B Based on this, the loss value is calculated (step S107).

[0070] The model learning unit 32 of the learning unit 151 learns parameters of the first intermediate representation model 10, the second intermediate representation model 11, and the probability distribution generation model 12 based on error backpropagation (step S108).

[0071] If the learning unit 151 continues the learning (step S109, Yes), the learning unit 151 proceeds to step S101. On the other hand, if the learning unit 151 does not continue the learning (step S109, No), the learning unit 151 ends the process.

[0072] Fig. 6 is a flowchart showing a processing procedure during generation by the generating device according to the embodiment 1. As shown in Fig. 6, the generating unit 152 of the generating device 100 acquires acoustic features (step S201). The generating unit 152 inputs the acoustic features to the first intermediate representation model 10 and calculates an intermediate representation m of the acoustic features (step S202).

[0073] The generation unit 152 inputs the intermediate representation m of the acoustic feature quantity to the second intermediate representation model 11 and generates an intermediate representation x of the subband signal. t (Step S203). The generation unit 152 calculates the probability distribution y t、b Calculate the probability distribution y t、b Based on the subband signal s t、b is calculated (step S204).

[0074] The speech waveform conversion unit 40 of the generation unit 152 converts the subband signal s t=1~T / B、b=1~B Based on this, the audio waveform zt=1~T (Step S205). The generation unit 152 calculates the voice waveform z t=1~T (Step S206).

[0075] If the generation unit 152 continues the process (Yes at step S207), the process proceeds to step S201. If the generation unit 152 does not continue the process (No at step S207), the generation unit 152 ends the process.

[0076] Next, a description will be given of the effects of the generating device 100 according to the first embodiment. The generating device 100 can speed up the process of generating a speech waveform from acoustic features by up to B×N times by simultaneously generating N samples of B subband signals in one forward propagation.

[0077] For example, a set of the probability distribution generation models 12-1-1 to 12-1-B and the sampling units 21-1-1 to 21-1-B, and a set of the probability distribution generation models 12-2-1 to 12-2-B and the sampling units 21-2-1 to 21-2-B generate one intermediate representation x t=-1,0 Then, the subband signals (s t=1、b=1~B , s t=2、b=1~B ) is generated.

[0078] In addition, by accelerating the process of generating speech waveforms from acoustic features, the number of forward propagations of the neural vocoder can be reduced, improving the learning and inference speeds. This is also effective in expanding the range of devices that can be equipped with neural vocoders and real-time operation at high sampling frequencies. EXAMPLES

[0079] First, a configuration example of a generating device according to the present embodiment 2 will be described. Fig. 7 is a functional block diagram showing a configuration of a generating device according to the present embodiment 2. As shown in Fig. 7, the generating device 200 includes a communication control unit 210, an input unit 220, an output unit 230, a storage unit 240, and a control unit 250.

[0080] The explanations regarding the communication control unit 210, the input unit 220, and the output unit 230 are similar to the explanations regarding the communication control unit 110, the input unit 120, and the output unit 130 described in the first embodiment.

[0081] The storage unit 240 has training speech waveform data 141, a first intermediate representation model 10, a second intermediate representation model 11, and a joint probability distribution generation model 13. The storage unit 240 is realized by a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk.

[0082] The training speech waveform data 141, the first intermediate representation model 10, and the second intermediate representation model 11 are similar to those described in the first embodiment.

[0083] The joint probability distribution generation model 13 is a model that simultaneously estimates the subband signals of channels b=1 to B. From the joint probability obtained from the joint probability distribution generation model 13, the dimension corresponding to each subband signal is set as a probability distribution.

[0084] Here, the probability distribution y t=τ、b is expressed as equation (1).

[0085]

number

[0086] In the second embodiment, the joint probability distribution model 13 simultaneously estimates the subband signals of channels b=1 to B, and therefore the joint probability distribution model 13 outputs a probability distribution y t=τ、b=1~B is expressed as in equation (2). This makes it possible to estimate a probability distribution that takes into account the relationship between subbands.

[0087]

number

[0088] 7, the control unit 250 includes a learning unit 251 and a generating unit 252. The control unit 250 corresponds to a CPU or the like.

[0089] The learning unit 251 uses the learning speech waveform data 141 to execute learning of the first intermediate representation model 10 , the second intermediate representation model 11 , and the joint probability distribution generation model 13 .

[0090] The generation unit 252 generates a speech waveform from the acoustic features by using the trained first intermediate representation model 10, the second intermediate representation model 11, and the joint probability distribution generation model 13. The generation unit 252 may obtain information on the acoustic features from the input unit 220, or may obtain the information from an external device via the communication control unit 210.

[0091] Next, the learning unit 251 shown in Fig. 7 will be specifically described. Fig. 8 is a diagram for explaining the learning unit according to the second embodiment. As shown in Fig. 8, the learning unit 251 includes an acoustic feature calculation unit 20, a first intermediate representation model 10, a plurality of second intermediate representation models 11-n (n = 1 to T / B-1), a plurality of joint probability distribution generation models 13-t (t = 1,..., T / B), a sampling unit 21-tb, a subband signal calculation unit 30, a loss calculation unit 31, and a model learning unit 32. Here, the sequence length of the voice is T, the number of channels of the subband signal is B, and the number of samples of the subband signal is N (=2).

[0092] The learning unit 251 inputs the speech waveform D1 stored in the learning speech waveform data 141 to the acoustic feature calculation unit 20. The acoustic feature calculation unit 20 calculates an acoustic feature D2 based on the speech waveform D1. The acoustic feature calculation unit 20 inputs the acoustic feature D2 to the first intermediate representation model 10.

[0093] When an acoustic feature D2 is input, the first intermediate representation model 10 outputs an intermediate representation m of the acoustic feature. The first intermediate representation model 10 outputs the intermediate representation m to the second intermediate representation models 11-1 to 11-n. The first intermediate representation model 10 outputs gradient information to the model learning unit 32.

[0094] When an intermediate representation m is input, each of the second intermediate representation models 11-1 to 11-n generates an intermediate representation x t For example, the second intermediate representation model 11-1 outputs the intermediate representation x t=-1,0 The second intermediate representation models 11-2 to 11-n calculate the intermediate representations x t When calculating the subband signal s t、b is further used.

[0095] The second intermediate representation model 11-2 is the intermediate representation x t=1,2 The second intermediate representation model 11-3 calculates the intermediate representation x t=3,4 and outputs it to joint probability distribution generation models 12-5 and 12-6 (not shown).

[0096] In the following description, when the second intermediate representation models 11-1 to 11-n are not distinguished from one another, they will be collectively referred to as the second intermediate representation model 11.

[0097] Each joint probability distribution generation model 13-t is an intermediate representation x t Given the input, the probability distribution y t , b=1~B For example, the joint probability distribution generation model 13-1 outputs the probability distribution y t=1 , b=1~B to the sampling units 21-1-1 to 21-1-B.

[0098] The joint probability distribution generation model 13-1 may output the probability distribution for each channel to the sampling units 21-1-1 to 21-1-B, respectively, or the sampling units 21-1-1 to 21-1-B may sample and acquire the corresponding probability distribution from the joint probability distribution. For example, if a multivariate distribution is used, it is possible to sample the corresponding probability distribution from the joint probability distribution. The same applies to the other joint probability distribution generation models described below.

[0099] The joint probability distribution generation model 13-2 is a probability distribution y t=2 , b=1~B to the sampling units 21-2-1 to 21-2-B.

[0100] The joint probability distribution generation model 13-3 is a probability distribution y t=3 , b=1~B to the sampling units 21-3-1 to 21-3-B.

[0101] The joint probability distribution generation model 13-4 is a probability distribution y t=4 , b=1~B to the sampling units 21-4-1 to 21-4-B.

[0102] We will not explain the other joint probability distribution generation models, but each intermediate representation x t Given the input, the probability distribution y t , b=1~B to the subordinate sampling unit. In the following description, when there is no need to distinguish between the individual joint probability distribution generation models 13-t, they will be collectively referred to as the joint probability distribution generation model 13. The joint probability distribution generation model 13 outputs gradient information to the model learning unit 32. The joint probability distribution generation model 13 outputs the probability distribution y t=1~T / B、b=1~B to the loss calculation unit 31.

[0103] Each sampling unit 21-tb detects a probability distribution y t , b When the input is a subband signal s t、b It should be noted that, when each joint probability distribution generation model 13 directly predicts a subband signal, the learning unit 251 does not need to use each sampling unit 21-tb.

[0104] Other explanations regarding each sampling unit 21-tb are similar to those regarding each sampling unit 21-tb described in the first embodiment.

[0105] The subband signal calculation unit 30 calculates a subband signal s t=1~T / B、b=1~BThe subband signal calculation unit 30 calculates the subband signal s t=1~T / B、b=1~B to the loss calculation unit 31. Other details regarding the subband signal calculation unit 30 are the same as those in the first embodiment.

[0106] The loss calculation unit 31 calculates the subband signal s t=1~T / B、b=1~B and the probability distribution y t=1~T / B、b=1~B The loss calculation unit 31 calculates a loss value D3 based on the above. The loss calculation unit 31 outputs the loss value D3 to the model learning unit 32. Other explanations regarding the loss calculation unit 31 are the same as those in the first embodiment.

[0107] The model learning unit 32 acquires the loss value D3, gradient information of the first intermediate representation model 10, gradient information of the second intermediate representation model 11, and gradient information of the joint probability distribution generation model 13, and performs machine learning on the parameters of the first intermediate representation model 10, the parameters of the second intermediate representation model 11, and the parameters of the joint probability distribution generation model 13 so as to reduce the loss value D3. For example, the model learning unit 32 performs machine learning by utilizing an error backpropagation algorithm.

[0108] Each processing unit included in the learning unit 251 repeatedly executes the above-mentioned process for multiple speech waveforms D1 included in the training speech waveform data 141, thereby obtaining a learned first intermediate representation model 10, a second intermediate representation model 11, and a joint probability distribution generation model 13.

[0109] Next, the generation unit 252 shown in Fig. 7 will be specifically described. Fig. 9 is a diagram for explaining the generation unit according to the second embodiment. As shown in Fig. 9, the generation unit 252 has a first intermediate representation model 10, a plurality of second intermediate representation models 11-n (n = 1 to T / B-1), a plurality of joint probability distribution generation models 13-t (t = 1,..., T / B), a sampling unit 21-tb, and a voice waveform conversion unit 40. Here, the sequence length of the voice is T, the number of channels of the subband signal is B, and the number of samples of the subband signal is N (= 2).

[0110] 8, when the second intermediate representation models 11-1 to 11-n are not distinguished from one another, they are collectively referred to as the second intermediate representation models 11. When the joint probability distribution generation model 13-t is not distinguished from one another, they are collectively referred to as the joint probability distribution generation model 13.

[0111] The first intermediate representation model 10, the second intermediate representation model 11, and the joint probability distribution generation model 13 are assumed to have been trained by the training unit 251.

[0112] The generation unit 252 inputs the acoustic feature D10 to the first intermediate representation model 10. When the acoustic feature D10 is input, the first intermediate representation model 10 outputs an intermediate representation m of the acoustic feature. The first intermediate representation model 10 outputs the intermediate representation m to the second intermediate representation models 11-1 to 11-n.

[0113] The second intermediate representation model 11 is configured to generate an intermediate representation x of a subband signal when the intermediate representation m is input. t to the joint probability distribution generation model 13. The explanation regarding the second intermediate representation model 11 is similar to the explanation regarding each of the second intermediate representation models 11-1 to 11-n given in FIG.

[0114] The joint probability distribution generation model 13 is an intermediate representation x t Given the input, the probability distribution y t , b=1~B to each sampling unit 21-tb. The explanation regarding the joint probability distribution generation model 13 is the same as the explanation regarding each joint probability distribution generation model 12-t given in FIG.

[0115] The sampling unit 21-tb calculates the probability distribution y t , b When the input is a subband signal s t、b Each sampling unit 21-tb calculates the subband signal s t=1~T / B、b=1~B is output to the voice waveform conversion unit 40.

[0116] The speech waveform converter 40 converts the subband signal s t=1~T / B、b=1~B Based on this, the audio waveform zt=1~T The audio waveform conversion unit 40 converts the subband signals into audio waveforms by inverse conversion of a quadrature mirror filter or a pseudo quadrature mirror filter.

[0117] Next, an example of a processing procedure of the generating device 200 according to the embodiment 2 will be described. Fig. 10 is a flowchart showing a processing procedure during learning of the generating device according to the embodiment 2. As shown in Fig. 10, the learning unit 251 of the generating device 200 acquires a voice waveform from the learning voice waveform data 141 (step S301).

[0118] The acoustic feature calculation unit 20 of the learning unit 251 calculates the acoustic feature from the speech waveform (step S302). The learning unit 251 inputs the acoustic feature to the first intermediate representation model 10 and calculates an intermediate representation m of the acoustic feature (step S303).

[0119] The learning unit 251 inputs the intermediate representation m of the acoustic feature quantity to the second intermediate representation model 11 and obtains the intermediate representation x of the subband signal. t (Step S304). The learning unit 251 calculates the probability distribution y t、b Calculate the probability distribution y t、b Based on the subband signal s t、b is calculated (step S305).

[0120] On the other hand, the subband signal calculation unit 30 of the learning unit 251 calculates a subband signal based on the voice waveform (step S306).

[0121] The loss calculation unit 31 of the learning unit 251 calculates the subband signal s t=1~T / B、b=1~B and the probability distribution y t=1~T / B、b=1~B Based on this, the loss value is calculated (step S307).

[0122] The model learning unit 32 of the learning unit 251 learns parameters of the first intermediate representation model 10, the second intermediate representation model 11, and the joint probability distribution generation model 13 based on error backpropagation (step S308).

[0123] If the learning unit 251 continues the learning (step S309, Yes), the process proceeds to step S301. On the other hand, if the learning unit 251 does not continue the learning (step S309, No), the process ends.

[0124] Fig. 11 is a flowchart showing a processing procedure during generation by the generating device according to the embodiment 2. As shown in Fig. 11, the generating unit 252 of the generating device 200 acquires an acoustic feature (step S401). The generating unit 252 inputs the acoustic feature to the first intermediate representation model 10 and calculates an intermediate representation m of the acoustic feature (step S402).

[0125] The generation unit 252 inputs the intermediate representation m of the acoustic feature quantity to the second intermediate representation model 11 and generates an intermediate representation x of the subband signal. t (Step S403). The generation unit 252 calculates the probability distribution y t、b Calculate the probability distribution y t、b Based on the subband signal s t、b is calculated (step S404).

[0126] The speech waveform converter 40 of the generator 252 converts the subband signal s t=1~T / B、b=1~B Based on this, the audio waveform z t=1~T (Step S405). The generation unit 252 calculates the voice waveform z t=1~T is output (step S406).

[0127] If the generation unit 252 continues the process (step S407, Yes), the generation unit 252 proceeds to step S401. If the generation unit 252 does not continue the process (step S407, No), the generation unit 252 ends the process.

[0128] Next, the effect of the generating device 200 according to the second embodiment will be described. The generating device 200 can speed up the process of generating a speech waveform from an acoustic feature by up to B×N times by simultaneously generating B subband signals by one forward propagation. In addition, the generating device 200 can estimate a probability distribution that takes into account the relationship between subbands by using a joint probability distribution generation model 13. Since the probability distribution calculation is performed simultaneously for multiple bands, the number of times that a matrix operation instruction is called can be reduced to 1 / B compared to the first embodiment, and the efficiency in terms of processing can be improved. EXAMPLES

[0129] First, a configuration example of a generating device according to the present embodiment 3 will be described. Fig. 12 is a functional block diagram showing a configuration of a generating device according to the present embodiment 3. As shown in Fig. 12, the generating device 300 includes a communication control unit 310, an input unit 320, an output unit 330, a storage unit 340, and a control unit 350.

[0130] The explanations regarding the communication control unit 310, the input unit 320, and the output unit 330 are similar to the explanations regarding the communication control unit 110, the input unit 120, and the output unit 130 described in the first embodiment.

[0131] The storage unit 340 has training speech waveform data 141, a first intermediate representation model 10, a second intermediate representation model 11, and a joint probability distribution generation model 14. The storage unit 340 is realized by a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk.

[0132] The training speech waveform data 141, the first intermediate representation model 10, and the second intermediate representation model 11 are similar to those described in the first embodiment.

[0133] The joint probability distribution generation model 14 is a model that simultaneously estimates the subband signals of channels b=1 to B at each time t=τ, τ+1. From the joint probability obtained from the joint probability distribution generation model 14, the dimension corresponding to each subband signal and each time is set as a probability distribution.

[0134] Here, the probability distributions output from the joint probability distribution 13 models that simultaneously generate N samples described in the second embodiment are respectively expressed as in formula (3). For example, when N=2, the joint probability distribution generation models 13-1 and 13-2 are used to estimate subband signals of channels b=1 to B at each time t=τ, τ+1.

[0135]

number

[0136] On the other hand, since the joint probability distribution generation model 14 of the present embodiment 3 simultaneously estimates the subband signals at each time and channel b=1 to B, the probability distribution output from the joint probability distribution generation model 14 is expressed as in formula (4). This makes it possible to estimate a probability distribution that takes into account the relationship between multiple subbands.

[0137]

number

[0138] 12, the control unit 350 includes a learning unit 351 and a generating unit 352. The control unit 350 corresponds to a CPU or the like.

[0139] The learning unit 351 uses the learning speech waveform data 141 to execute learning of the first intermediate representation model 10, the second intermediate representation model 11, and the joint probability distribution generation model 14.

[0140] The generation unit 352 generates a speech waveform from the acoustic features by using the trained first intermediate representation model 10, the second intermediate representation model 11, and the joint probability distribution generation model 14. The generation unit 352 may obtain information on the acoustic features from the input unit 320, or may obtain the information from an external device via the communication control unit 310.

[0141] Next, the learning unit 351 shown in Fig. 12 will be specifically described. Fig. 13 is a diagram for explaining the learning unit according to the third embodiment. As shown in Fig. 13, the learning unit 351 includes an acoustic feature calculation unit 20, a first intermediate representation model 10, a plurality of second intermediate representation models 11-n (n = 1 to T / B-1), a plurality of joint probability distribution generation models 14-t (t = 1,..., T / 2B), a sampling unit 21-tb, a subband signal calculation unit 30, a loss calculation unit 31, and a model learning unit 32. Here, the sequence length of the voice is T, the number of channels of the subband signal is B, and the number of samples of the subband signal is N (=2).

[0142] The learning unit 351 inputs the speech waveform D1 stored in the learning speech waveform data 141 to the acoustic feature calculation unit 20. The acoustic feature calculation unit 20 calculates an acoustic feature D2 based on the speech waveform D1. The acoustic feature calculation unit 20 inputs the acoustic feature D2 to the first intermediate representation model 10.

[0143] When an acoustic feature D2 is input, the first intermediate representation model 10 outputs an intermediate representation m of the acoustic feature. The first intermediate representation model 10 outputs the intermediate representation m to the second intermediate representation models 11-1 to 11-n. The first intermediate representation model 10 outputs gradient information to the model learning unit 32.

[0144] When an intermediate representation m is input, each of the second intermediate representation models 11-1 to 11-n generates an intermediate representation x t For example, the second intermediate representation model 11-1 outputs the intermediate representation x t=-1,0 The second intermediate representation models 11-2 to 11-n calculate the intermediate representations x tWhen calculating the subband signal s t、b is further used.

[0145] The second intermediate representation model 11-2 is the intermediate representation x t=1,2 The second intermediate representation model 11-3 calculates the intermediate representation x t=3,4 and outputs it to a joint probability distribution generation model 14-3 (not shown).

[0146] In the following description, when the second intermediate representation models 11-1 to 11-n are not distinguished from one another, they will be collectively referred to as the second intermediate representation model 11.

[0147] Each joint probability distribution generation model 14-t is an intermediate representation x t Given the input, the probability distribution y t=t、t+1 , b=1~B For example, the joint probability distribution generation model 14-1 outputs the probability distribution y t=1 , b=1~B to the sampling units 21-1-1 to 21-1-B, and the probability distribution y t=2 , b=1~B to the sampling units 21-2-1 to 21-2-B.

[0148] The joint probability distribution generation model 14-1 may output the probability distribution for each time and each channel to the sampling units 21-1-1 to 21-1-B and 21-2-1 to 21-2-B, respectively, or the sampling units 21-1-1 to 21-1-B and 21-2-1 to 21-2-B may sample and acquire the corresponding probability distribution from the joint probability distribution. For example, if a multivariate distribution is used, it is possible to sample the corresponding probability distribution from the joint probability distribution. The same applies to other joint probability distribution generation models described below.

[0149] The joint probability distribution generation model 14-2 is a probability distribution y t=3 , b=1~B to the sampling units 21-3-1 to 21-3-B, and the probability distribution y t=4 , b=1~Bto the sampling units 21-4-1 to 21-4-B.

[0150] We will not explain the other joint probability distribution generation models, but each intermediate representation x t Given the input, the probability distribution y t、t+1 , b=1~B to the subordinate sampling unit. In the following description, when there is no need to distinguish between the individual joint probability distribution generation models 14, they will be collectively referred to as joint probability distribution generation models 14. The joint probability distribution generation model 14 outputs gradient information to the model learning unit 32. The joint probability distribution generation model 14 outputs the probability distribution y t=1~T / B、b=1~B to the loss calculation unit 31.

[0151] Each sampling unit 21-tb detects a probability distribution y t , b When the input is a subband signal s t、b It should be noted that, when each joint probability distribution generation model 14 directly predicts a subband signal, the learning unit 351 does not need to use each sampling unit 21-tb.

[0152] Other explanations regarding each sampling unit 21-tb are similar to those regarding each sampling unit 21-tb described in the first embodiment.

[0153] The subband signal calculation unit 30 calculates a subband signal s t=1~T / B、b=1~B The subband signal calculation unit 30 calculates the subband signal s t=1~T / B、b=1~B to the loss calculation unit 31. Other details regarding the subband signal calculation unit 30 are the same as those in the first embodiment.

[0154] The loss calculation unit 31 calculates the subband signal s t=1~T / B、b=1~B and the probability distribution y t=1~T / B、b=1~B The loss calculation unit 31 calculates a loss value D3 based on the above. The loss calculation unit 31 outputs the loss value D3 to the model learning unit 32. Other explanations regarding the loss calculation unit 31 are the same as those in the first embodiment.

[0155] The model learning unit 32 acquires the loss value D3, gradient information of the first intermediate representation model 10, gradient information of the second intermediate representation model 11, and gradient information of the joint probability distribution generation model 14, and performs machine learning on the parameters of the first intermediate representation model 10, the parameters of the second intermediate representation model 11, and the parameters of the joint probability distribution generation model 14 so as to reduce the loss value D3. For example, the model learning unit 32 performs machine learning by utilizing an error backpropagation algorithm.

[0156] Each processing unit included in the learning unit 351 repeatedly executes the above-mentioned process for multiple speech waveforms D1 included in the training speech waveform data 141, thereby obtaining a learned first intermediate representation model 10, a second intermediate representation model 11, and a joint probability distribution generation model 14.

[0157] Next, the generation unit 352 shown in Fig. 12 will be specifically described. Fig. 14 is a diagram for explaining the generation unit according to the third embodiment. As shown in Fig. 14, the generation unit 352 has a first intermediate representation model 10, a plurality of second intermediate representation models 11-n (n = 1 to T / B-1), a plurality of joint probability distribution generation models 14-t (t = 1,..., T / 2B), a sampling unit 21-tb, and a voice waveform conversion unit 40. Here, the sequence length of the voice is T, the number of channels of the subband signal is B, and the number of samples of the subband signal is N (=2).

[0158] 13, when the second intermediate representation models 11-1 to 11-n are not distinguished from one another, they are collectively referred to as the second intermediate representation models 11. When the joint probability distribution generation model 14-t is not distinguished from one another, they are collectively referred to as the joint probability distribution generation model 14.

[0159] The first intermediate representation model 10, the second intermediate representation model 11, and the joint probability distribution generation model 14 are assumed to have been trained by the training unit 351.

[0160] The generation unit 352 inputs the acoustic feature D10 to the first intermediate representation model 10. When the acoustic feature D10 is input, the first intermediate representation model 10 outputs an intermediate representation m of the acoustic feature. The first intermediate representation model 10 outputs the intermediate representation m to the second intermediate representation models 11-1 to 11-n.

[0161] The second intermediate representation model 11 is configured to generate an intermediate representation x of a subband signal when the intermediate representation m is input. t to the joint probability distribution generation model 14. The explanation regarding the second intermediate representation model 11 is similar to the explanation regarding each of the second intermediate representation models 11-1 to 11-n given in FIG.

[0162] The joint probability distribution generation model 14 is an intermediate representation x t Given the input, the probability distribution y t、t+1 , b=1~B to each sampling unit 21-tb. The explanation regarding the joint probability distribution generation model 14 is the same as the explanation regarding each joint probability distribution generation model 12-t given in FIG.

[0163] The sampling unit 21-tb calculates the probability distribution y t , b When the input is a subband signal s t、b Each sampling unit 21-tb calculates the subband signal s t=1~T / B、b=1~B is output to the voice waveform conversion unit 40.

[0164] The speech waveform converter 40 converts the subband signal s t=1~T / B、b=1~B Based on this, the audio waveform z t=1~T The audio waveform conversion unit 40 converts the subband signals into audio waveforms by inverse conversion of a quadrature mirror filter or a pseudo quadrature mirror filter.

[0165] Next, an example of a processing procedure of the generating device 300 according to the present embodiment 3 will be described. Fig. 15 is a flowchart showing a processing procedure during learning of the generating device according to the present embodiment 3. As shown in Fig. 15, the learning unit 351 of the generating device 300 acquires a voice waveform from the learning voice waveform data 141 (step S501).

[0166] The acoustic feature calculation unit 20 of the learning unit 351 calculates the acoustic feature from the speech waveform (step S502). The learning unit 351 inputs the acoustic feature to the first intermediate representation model 10 and calculates an intermediate representation m of the acoustic feature (step S503).

[0167] The learning unit 351 inputs the intermediate representation m of the acoustic feature quantity to the second intermediate representation model 11 and obtains the intermediate representation x of the subband signal. t (Step S504). The learning unit 351 calculates the probability distribution y t、t+1、b Calculate the probability distribution y t、t+1、b Based on the subband signal s t、b is calculated (step S505).

[0168] On the other hand, the subband signal calculation unit 30 of the learning unit 351 calculates a subband signal based on the speech waveform (step S506).

[0169] The loss calculation unit 31 of the learning unit 351 calculates the subband signal s t=1~T / B、b=1~B and the probability distribution y t=1~T / B、b=1~B Based on this, the loss value is calculated (step S507).

[0170] The model learning unit 32 of the learning unit 351 learns parameters of the first intermediate representation model 10, the second intermediate representation model 11, and the joint probability distribution generation model 14 based on error backpropagation (step S508).

[0171] If the learning unit 351 continues the learning (step S509, Yes), the process proceeds to step S501. On the other hand, if the learning unit 351 does not continue the learning (step S509, No), the process ends.

[0172] Fig. 16 is a flowchart showing a processing procedure during generation by the generating device according to the third embodiment. As shown in Fig. 16, the generating unit 352 of the generating device 300 acquires acoustic features (step S601). The generating unit 352 inputs the acoustic features to the first intermediate representation model 10 and calculates an intermediate representation m of the acoustic features (step S602).

[0173] The generation unit 352 inputs the intermediate representation m of the acoustic feature quantity to the second intermediate representation model 11 and generates an intermediate representation x of the subband signal. t (Step S603). The generation unit 352 calculates the probability distribution y t、t+1、b Calculate the probability distribution y t、t+1、b Based on the subband signal s t、b is calculated (step S604).

[0174] The speech waveform converter 40 of the generator 352 converts the subband signal s t=1~T / B、b=1~B Based on this, the audio waveform z t=1~T (Step S605). The generation unit 352 calculates the voice waveform z t=1~T is output (step S606).

[0175] If the generation unit 352 continues the process (step S607, Yes), the process proceeds to step S601. If the generation unit 352 does not continue the process (step S607, No), the generation unit 352 ends the process.

[0176] Next, the effects of the generating device 300 according to the third embodiment will be described. The generating device 300 simultaneously generates N samples of B subband signals for each time in one forward propagation. This can speed up the process of generating a speech waveform from speech features, and can reduce the number of calls of a matrix calculation instruction to a minimum of 1 / (B×N) times compared to the first embodiment, which is also effective in terms of processing.

[0177] Next, an example of a computer that executes the generation program will be described. Fig. 17 is a diagram showing an example of a computer that executes the generation program. The computer 1000 has, for example, a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. Each of these units is connected by a bus 1080.

[0178] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores, for example, a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1031. The disk drive interface 1040 is connected to a disk drive 1041. A removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1041. The serial port interface 1050 is connected to, for example, a mouse 1051 and a keyboard 1052. The video adapter 1060 is connected to, for example, a display 1061.

[0179] Here, the hard disk drive 1031 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. Each piece of information described in the above embodiments is stored in the hard disk drive 1031 or memory 1010, for example.

[0180] The generating program is stored in the hard disk drive 1031, for example, as a program module 1093 in which instructions to be executed by the computer 1000 are written. Specifically, the hard disk drive 1031 stores the program module 1093 in which each process to be executed by the generating device 100 (200, 300) described in the above embodiment is written.

[0181] Furthermore, data used for information processing by the generation program is stored as program data 1094, for example, in the hard disk drive 1031. Then, the CPU 1020 reads out the program module 1093 and the program data 1094 stored in the hard disk drive 1031 into the RAM 1012 as necessary, and executes each of the above-mentioned procedures.

[0182] Note that the program module 1093 and program data 1094 relating to the generation program are not limited to being stored in the hard disk drive 1031, and may be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1041 or the like. Alternatively, the program module 1093 and program data 1094 relating to the generation program may be stored in another computer connected via a network such as a LAN or a WAN (Wide Area Network), and read by the CPU 1020 via the network interface 1070.

[0183] Although the embodiment to which the invention made by the present inventor is applied has been described above, the present invention is not limited by the description and drawings which form a part of the disclosure of the present invention according to the present embodiment. In other words, other embodiments, examples, operation techniques, etc. made by those skilled in the art based on the present embodiment are all included in the scope of the present invention. [Explanation of symbols]

[0184] 100,200,300 generator 110, 210, 310 Communication control unit 120,220,320 Input section 130,230,330 Output section 140,240,340 storage section 150,250,350 Control section 151,251,351 Learning Department 152,252,352 generation part

Claims

1. an intermediate representation generating step of generating intermediate representation information of a subband signal based on acoustic features of a speech waveform; a subband signal generation process for simultaneously generating a plurality of subband signals corresponding to a plurality of different times and a plurality of different bands by inputting the intermediate representation information into a plurality of probability distribution generation models corresponding to the number of channels of the subband signals and the number of samples to be simultaneously generated, the plurality of probability distribution generation models outputting information of the subband signals corresponding to the times and bands respectively assigned thereto; a speech waveform generating step of generating a speech waveform based on the plurality of subband signals; A generating method comprising:

2. The generation method according to claim 1, further comprising a conversion step of converting the acoustic features of the speech waveform into intermediate representation information of the acoustic features using a first intermediate representation model that outputs intermediate representation information of the acoustic features when the acoustic features are input.

3. The generation method described in claim 2, characterized in that the intermediate representation generation step generates intermediate representation information of the subband signal using a second intermediate representation model that outputs intermediate representation information of the subband signal when intermediate representation information of the acoustic feature is input.

4. The generation method according to claim 3, further comprising a learning process for calculating a loss value based on a plurality of subband signals calculated from a speech waveform and a plurality of subband signals generated by the subband signal generation process, and for performing learning of at least one model among the first intermediate representation model, the second intermediate representation model, and the plurality of probability distribution generation models based on the loss value.

5. The generation method according to claim 1, characterized in that the subband signal generation process simultaneously generates the multiple subband signals using a joint probability distribution generation model that simultaneously outputs information on subband signals corresponding to multiple time periods and multiple bands from one model.

6. an intermediate representation generating step of generating intermediate representation information of a subband signal based on acoustic features of a speech waveform; a subband signal generation step of simultaneously generating a plurality of subband signals corresponding to a plurality of different times and a plurality of different bands by inputting the intermediate representation information into a plurality of probability distribution generation models corresponding to the number of channels of the subband signals and the number of samples to be simultaneously generated, the plurality of probability distribution generation models outputting information of subband signals corresponding to times and bands respectively assigned thereto; a speech waveform generating step of generating a speech waveform based on the plurality of subband signals; A generating program for causing a computer to execute the above.

7. a generation unit that generates intermediate representation information of a subband signal based on acoustic features of a speech waveform, and simultaneously generates a plurality of subband signals corresponding to different times and different bands by inputting the intermediate representation information into a plurality of probability distribution generation models corresponding to the number of channels of the subband signals and the number of samples to be simultaneously generated, the plurality of probability distribution generation models outputting information of the subband signal corresponding to a time and a band respectively assigned thereto; a voice waveform generating unit that generates a voice waveform based on the plurality of subband signals; A generating device comprising:

Images