Audio waveform generation method, audio waveform generation device, and program

Abstract

A speech waveform generation method according to one aspect of the present disclosure causes a computer to execute: a forming step for, using a block size to be utilized for pruning of a weight matrix of a neural network that implements a model for generating a speech waveform, forming the weight matrix such that the weight matrix is configured by a block of the block size unit; a regularization term calculation step for, using the formed weight matrix, calculating a regularization term that, when a weight becomes zero by regularization, promotes all of weights within the block in which the weight is included collectively becoming zero; a training step for, using given training data and the regularization term, training the neural network that implements the model; and a pruning step for, using the block size, pruning the weight matrix of the neural network that implements the trained model into the block size unit.

Description

[Technical Field]

[0001] The present disclosure relates to a voice waveform generation method, a voice waveform generation device, and a program. [Background technology]

[0002] In the field of speech synthesis, a module that converts acoustic features, such as spectrum and pitch, into a speech waveform is called a vocoder. There are two main types of vocoder implementations. One is a signal processing-based approach, with well-known methods such as STRAIGHT and WORLD (Non-Patent Documents 1 and 2). These methods use mathematical models to represent the conversion from acoustic features to speech waveforms, eliminating the need for training and enabling high processing speeds. However, the quality of the analytically resynthesized speech is inferior to that of natural speech. The other is a neural vocoder, which utilizes neural networks, such as WaveNet (Non-Patent Document 3). While this approach can synthesize speech with quality comparable to natural speech, it requires a large amount of computation and therefore operates slower than vocoders that use signal processing. Typically, a single forward propagation through a neural network is required to predict one speech sample, making real-time operation difficult when implemented as is.

[0003] There are two main approaches to reducing the computational complexity of neural vocoders, particularly for real-time operation on a central processing unit (CPU). One is to reduce the computational cost per forward propagation of a neural network. Examples of this approach include WaveRNN (Non-Patent Document 4), which replaces the large convolutional neural network (CNN) used in WaveNet with a small recurrent neural network (RNN), and LPCNet (Non-Patent Document 5), which utilizes linear predictive coding (LPC), a signal processing technique, in the speech waveform generation process. The other is to reduce the number of forward propagation passes themselves. One example of this approach is a method that reduces the number of forward propagation passes by changing the prediction target from speech waveforms to short sequences (called subband signals) obtained by band division using a quadrature mirror filter (QMF) (Non-Patent Document 6).

[0004] In methods adopting the first approach (Non-Patent Documents 4 and 5), it has been suggested that simply reducing the model size is insufficient for real-time operation, and pruning is used to further reduce the amount of computation. Pruning is a technique that replaces some weights with zero in the matrix operations that account for most of the neural network's calculations, thereby omitting those operations and thereby increasing the speed. There is a trade-off between inference speed and the quality of synthesized speech, due to the difference in calculation results before and after pruning. To reduce the difference in calculation results before and after pruning, regularization methods such as lasso and group lasso are generally used to increase the number of elements close to zero during training. For example, Non-Patent Document 7, although for image recognition applications rather than speech waveform generation, verified the accuracy and speed when group lasso was applied to several patterns of structures related to CNN filters, channels, etc. It is known that combining regularization and pruning in this way makes it possible to increase speed while suppressing quality degradation caused by pruning. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Hideki Kawahara, Ikuyo Masuda-Katsuuse and Alain de Cheveigne, "Restructuring speech representations using a pitch-adaptive time frequency smoothing and an instantaneous-frequency-based F0 extraction: Possible role of a repetitive structure in sounds," Speech Communication, vol. 27, no. 3-4, pp. 187-207, 1999. [Non-patent document 2] Masanori Morise, Fumiya Yokomori, Kenji Ozawa, "WORLD: a vocoder-based high-quality speech synthesis system for real-time applications," IEICE transactions on information and systems, vol. E99-D, no. 7, pp. 1877-1884, 2016.

Table 3

Fashion 4

Wood 5

Wood 6

[0006] The regularization used in Non-Patent Document 7 uses a group lasso to bring the weights of multiple elements closer to 0. Therefore, unlike the lasso, it has the advantage that non-zero elements tend to be consecutive, and can be calculated efficiently using hardware SIMD (single instruction multiple data) instructions. However, to make the most of SIMD performance, pruning must be performed under the constraints that (1) the sequence of non-zero elements is consecutive, and (2) the number of elements is stored in the SIMD register at one time.

[0007] However, since Group Lasso does not have any explicit constraints regarding (2) above, it is difficult to control the continuity of non-zero elements while taking SIMD into consideration. For this reason, for example, when pruning, the weights of elements that should not be zero may be set to zero, leading to a deterioration in quality.

[0008] The present disclosure has been made in consideration of the above points, and provides a technology that can efficiently generate audio waveforms while suppressing quality degradation. [Means for solving the problem]

[0009] A speech waveform generation method according to one aspect of the present disclosure includes a computer-implemented process: a shaping step of shaping a weight matrix using a block size used to prune the weight matrix of a neural network that realizes a model for generating speech waveforms so that the weight matrix is ​​composed of blocks in units of the block size; a regularization term calculation step of using the shaped weight matrix to calculate a regularization term that promotes all weights in a block containing a weight to become zero collectively when the weight becomes zero due to regularization; a training step of training a neural network that realizes the model using given training data and the regularization term; and a pruning step of pruning the weight matrix of the neural network that realizes the trained model to units of the block size using the block size. [Effects of the Invention]

[0010] A technology is provided that can efficiently generate audio waveforms while suppressing quality degradation. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 10 is a diagram for explaining a conventional example of a voice waveform generation model learning processing unit. [Figure 2] FIG. 1 is a diagram for explaining a conventional example of a voice waveform generation processing unit. [Figure 3] 1 is a diagram illustrating an example of a hardware configuration of a voice waveform generating device according to a first embodiment. [Figure 4] 1 is a diagram illustrating an example of a functional configuration of a voice waveform generating device according to a first embodiment. [Figure 5] FIG. 3 is a diagram for explaining a voice waveform generation model learning processing unit in the first embodiment. [Figure 6] FIG. 3 is a diagram for explaining a voice waveform generation processing unit in the first embodiment. [Figure 7] 10 is a flowchart illustrating an example of a voice waveform generation model learning process according to the first embodiment. [Figure 8] 10 is a flowchart illustrating an example of a voice waveform generation process according to the first embodiment. [Figure 9] FIG. 10 is a diagram illustrating an example of a functional configuration of a voice waveform generating device according to a second embodiment. [Figure 10] FIG. 10 is a diagram illustrating a subband signal generation model learning processing unit according to the second embodiment. [Figure 11] FIG. 10 is a diagram for explaining a voice waveform generation processing unit in the second embodiment. [Figure 12] 10 is a flowchart illustrating an example of a subband signal generation model learning process in the second embodiment. [Figure 13] 10 is a flowchart illustrating an example of a voice waveform generation process according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] An embodiment of the present invention will be described below, which is a voice waveform generation device 10 that can efficiently generate a voice waveform while suppressing quality degradation.

[0013] [Conventional example] Before describing a specific example of this embodiment, a conventional method for generating a voice waveform will be described.

[0014] As one of the conventional methods, a speech waveform generation model learning processor and a speech waveform generation processor in which calculation of a regularization term is added to the method described in Non-Patent Document 3 or 4 will be described.

[0015] Conventional example of speech waveform generation model learning processing section A conventional example of a speech waveform generation model training processor is shown in Figure 1. As shown in Figure 1, the conventional speech waveform generation model training processor includes a speech waveform generation unit, a generation error calculation unit, a regularization term calculation unit, an error weighting unit, a speech waveform generation model training unit, and a pruning unit. Hereinafter, the acoustic feature and the sequence length of the speech waveform are defined as N and T, respectively. Also, the acoustic feature x1,...,x N x n=1,···,N Similarly, the speech waveforms y1, ,y T y t=1,···,T It is abbreviated as:

[0016] At this time, the conventional speech waveform generation model training processing unit creates a pruned speech waveform generation model by repeating the following steps S11 to S16 for the speech waveforms and acoustic features included in all training data. Note that the training data is data represented as pairs of speech waveforms and their acoustic features.

[0017] S11: The speech waveform generator generates the acoustic feature x included in the training data. n=1,···,N and the speech waveform generation model as input, the generated speech waveform

[0018]

number

[0019] If the output of a speech waveform generation model is a speech waveform, due to the nature of speech waveform samples having zero crossings, the model will predict values ​​near zero, which is the average value, and this can easily lead to breakdowns such as silence. For this reason, instead of predicting and outputting a speech waveform, a method is generally used in which a probability distribution is predicted and output, and the speech waveform is obtained by sampling from that probability distribution. If the speech waveform is expressed as discrete values ​​using bit representation, a categorical distribution can be used, and if it is expressed as continuous values, a Gaussian distribution, beta distribution, or mixed logistic distribution can be used.

[0020] S12: Next, the generation error calculation unit calculates the speech waveform y t=1,···,T and the generated speech waveform ^y t=1,···,T The generated speech waveform ^y is used as input and the generated error is output. t=1,···,T Or the probability distribution of its generator is t=1,···,T For example, if a categorical distribution is used as the probability distribution, the cross-entropy error is used as the generation error, and if a continuous distribution such as a Gaussian distribution or a beta distribution is used as the probability distribution, the negative log-likelihood is used as the generation error.

[0021] S13: Meanwhile, the regularization term calculation unit receives the speech waveform generation model as input and outputs a regularization term. The regularization term evaluates how many non-zero elements there are in the weights of the neural network that realizes the speech waveform model. For example, lasso, group lasso, etc. are used as a method for calculating the regularization term.

[0022] S14: Next, the error weighting unit receives the generation error and the regularization term as inputs and outputs a weighted error. Because the generation error and the regularization term often have significantly different ranges of values ​​that they can take, the weighted sum of the two is calculated to balance this and used as the weighted error.

[0023] S15: The voice waveform generation model learning unit receives the weighting error and the voice waveform generation model as input and outputs the learned voice waveform generation model. In the neural network learning framework, it is possible to efficiently update the weights (hereinafter also referred to as weight coefficients) of the neural network using the error backpropagation method.

[0024] S16: The pruning unit receives the trained speech waveform generation model and the pruning block size as input, and outputs the pruned speech waveform generation model. The block size is a hyperparameter for parallel execution of matrix multiplication and accumulation operations using SIMD. This block size is used to divide the weight matrix, which is made up of the neural network's weight coefficients, into blocks of a certain size, and pruning replaces several blocks collectively with zeros. Since the majority of neural network operations are multiplication and accumulation operations with the weight matrix, generally, areas with small absolute values ​​of the weight coefficients are pruned to reduce prediction errors.

[0025] By repeating the above steps S11 to S16 for the speech waveforms and acoustic features of all the training data, a pruned speech waveform generation model is obtained.

[0026] Conventional example of audio waveform generation processing section A conventional example of a voice waveform generation processing unit is shown in Fig. 2. As shown in Fig. 2, the conventional voice waveform generation processing unit includes a voice waveform generation unit.

[0027] At this time, the conventional speech waveform generation processing unit generates a speech waveform by the following S21. Note that the speech waveform generation unit included in the speech waveform generation processing unit is realized by the same processing as the speech waveform generation unit included in the speech waveform generation model learning processing unit.

[0028] S21: The speech waveform generator generates the acoustic feature x n=1,···,N and the pruned speech waveform generation model as input, the generated speech waveform ^y t=1,···,T At this time, the speech waveform generator omits calculations for the pruned parts and executes parallel calculations using SIMD with the block size at the time of pruning. This enables high-speed speech waveform generation.

[0029] [Example 1] Example 1 of this embodiment will be described below.

[0030] <Hardware Configuration of the Voice Waveform Generation Device 10 (First Embodiment)> An example of the hardware configuration of the speech waveform generation device 10 in the first embodiment is shown in Fig. 3. As shown in Fig. 3, the speech waveform generation device 10 in the first embodiment includes an input device 101, a display device 102, an external I / F 103, a communication I / F 104, a RAM (Random Access Memory) 105, a ROM (Read Only Memory) 106, an auxiliary storage device 107, and a processor 108. Each of these pieces of hardware is connected to each other via a bus 109 so as to be able to communicate with each other.

[0031] The input device 101 is, for example, a keyboard, a mouse, a touch panel, a physical button, etc. The display device 102 is, for example, a display, a display panel, etc. Note that the audio waveform generation device 10 does not necessarily have to include at least one of the input device 101 and the display device 102, for example.

[0032] The external I / F 103 is an interface with an external device such as a recording medium 103a. The audio waveform generation device 10 can read from and write to the recording medium 103a via the external I / F 103. Examples of the recording medium 103a include a flexible disk, a CD (Compact Disc), a DVD (Digital Versatile Disk), an SD memory card (Secure Digital memory card), and a USB (Universal Serial Bus) memory card.

[0033] The communication I / F 104 is an interface for connecting the voice waveform generation device 10 to a communication network. The RAM 105 is a volatile semiconductor memory (storage device) that temporarily stores programs and data. The ROM 106 is a non-volatile semiconductor memory (storage device) that can store programs and data even when the power is turned off. The auxiliary storage device 107 is a storage device (storage device) such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a flash memory. The processor 108 is an arithmetic device such as a CPU. It is assumed that the processor 108 is an arithmetic device that can execute parallel arithmetic processing using SIMD instructions.

[0034] The speech waveform generation device 10 in the first embodiment has the hardware configuration shown in Fig. 3, and is thereby able to realize various processes described below. Note that the hardware configuration shown in Fig. 1 is an example, and the hardware configuration of the speech waveform generation device 10 is not limited to this. For example, the speech waveform generation device 10 may have multiple auxiliary storage devices 107 or multiple processors 108, may not have some of the hardware shown in the figure, or may have various hardware other than the hardware shown in the figure.

[0035] <Functional configuration of the voice waveform generation device 10 (first embodiment)> An example of the functional configuration of the speech waveform generation device 10 in the first embodiment is shown in Fig. 4. As shown in Fig. 4, the speech waveform generation device 10 in this embodiment has a speech waveform generation model learning processing unit 210 and a speech waveform generation processing unit 220. These units are realized, for example, by processing in which one or more programs installed in the speech waveform generation device 10 are executed by the processor 108. The speech waveform generation device 10 in this embodiment also has a storage unit 230. The storage unit 230 is realized, for example, by the auxiliary storage device 107.

[0036] The speech waveform generation model learning processing unit 210 executes a speech waveform generation model learning process for learning a speech waveform generation model. An example of a detailed functional configuration of the speech waveform generation model learning processing unit 210 is shown in Fig. 5. As shown in Fig. 5, the speech waveform generation model learning processing unit 210 in this embodiment includes a speech waveform generation unit 211, a generation error calculation unit 212, a weighting matrix shaping unit 213, a regularization term calculation unit 214, an error weighting unit 215, a speech waveform generation model learning unit 216, and a pruning unit 217. Note that the speech waveform generation unit 211, the generation error calculation unit 212, the error weighting unit 215, the speech waveform generation model learning unit 216, and the pruning unit 217 are similar to the speech waveform generation unit, the generation error calculation unit, the error weighting unit, the speech waveform generation model learning unit, and the pruning unit in the conventional example, respectively. That is, this embodiment differs from the conventional example in that the voice waveform generation model learning processing unit 210 has a weight matrix forming unit 213 and in the method of calculating the regularization term by the regularization term calculation unit 214.

[0037] The speech waveform generation unit 211 generates the acoustic feature x included in the training data. n=1,···,N and the speech waveform generation model as input, the generated speech waveform ^y t=1,···,T Output.

[0038] The generation error calculation unit 212 calculates the speech waveform y t=1,···,T and the generated speech waveform ^y t=1,···,T It takes input and outputs the generated error.

[0039] The weight matrix shaping unit 213 receives the voice waveform generation model and the block size of pruning as input, and outputs a shaped weight matrix. The shaped weight matrix is ​​a weight matrix of a neural network that realizes the voice waveform model, shaped using the block size. For example, the weight matrix is

[0040]

number

[0041]

number

[0042] In this way, the weight matrix forming unit 213 changes the output dimension of the weight matrix to D out / D block Then, the number of dimensions D block Add the dimension of D block The weight coefficients are shaped so that they form a single unit (block unit).

[0043] The regularization term calculation unit 214 receives the shaped weight matrix (tensor) as input and outputs the regularization term calculated as follows:

[0044]

number

[0045]

number

[0046] In the above, the weight matrix is ​​assumed to be a normal matrix (in other words, a second-order tensor), but it can also be applied to a weight matrix that is a third-order or higher tensor. If the weight matrix is ​​a third-order or higher tensor, the last output dimension of the weight matrix is ​​set to D out / D block Then, the number of dimensions D block Add the dimension of D block The weight matrix after shaping should be created so that the weight coefficients are grouped together (block unit). Also, for the regularization term, the added dimension (number of dimensions D block We can sum the square roots of the sums of squares in the other dimensions.

[0047] The error weighting unit 215 receives the generation error and the regularization term as input, and outputs a weighted error.

[0048] The voice waveform generation model learning unit 216 receives the weighted error and the voice waveform generation model as input, and outputs the learned voice waveform generation model.

[0049] Pruning unit 217 receives the trained speech waveform generation model and the block size for pruning as input, and outputs the pruned speech waveform generation model.

[0050] The voice waveform generation processing unit 220 executes voice waveform generation processing to generate a voice waveform. An example of the detailed functional configuration of the voice waveform generation processing unit 220 is shown in Fig. 6. As shown in Fig. 6, the voice waveform generation processing unit 220 in this embodiment includes a voice waveform generation unit 221. Note that the voice waveform generation unit 221 is similar to the voice waveform generation unit in the conventional example.

[0051] The speech waveform generation unit 221 generates an acoustic feature x n=1,···,N and the pruned speech waveform generation model as input, the generated speech waveform ^y t=1,···,T Output.

[0052] The storage unit 230 stores various data (for example, a speech waveform generation model, training data used when training the speech waveform generation model, acoustic features used when generating speech waveforms, hyperparameters such as block size, etc.).

[0053] <Speech waveform generation model learning process (Example 1)> The speech waveform generation model training process in this embodiment will be described below with reference to Fig. 7. Steps S101 to S107 in Fig. 7 are repeatedly executed for the speech waveforms and acoustic features included in all training data. Below, steps S101 to S107 related to the speech waveforms and acoustic features included in certain training data will be described.

[0054] The speech waveform generation unit 211 of the speech waveform generation model learning processing unit 210 calculates the acoustic feature x included in the learning data. n=1,···,N and the speech waveform generation model as input, the generated speech waveform ^y t=1,···,T is output (step S101).

[0055] Next, the generation error calculation unit 212 of the speech waveform generation model learning processing unit 210 calculates the speech waveform y t=1,···,T and the generated speech waveform ^y t=1,···,T and are input, and the generated error is output (step S102).

[0056] Next, weight matrix shaping unit 213 of speech waveform generation model learning processing unit 210 receives the speech waveform generation model and the pruning block size as input, and outputs a shaped weight matrix (step S103). At this time, for example, if the weight matrix is ​​a normal matrix (in other words, a second-order tensor), weight matrix shaping unit 213 creates a shaped weight matrix using Equation 3 above.

[0057] Next, the regularization term calculation unit 214 of the speech waveform generation model learning processing unit 210 receives the shaped weight matrix (tensor) as input and outputs a regularization term (step S104). At this time, if the shaped weight matrix is ​​expressed by the above equation 3, the regularization term calculation unit 214 calculates the regularization term using the above equation 4.

[0058] Next, the error weighting unit 215 of the voice waveform generation model learning processing unit 210 receives the generation error and the regularization term as input, and outputs a weighted error (step S105).

[0059] Next, the voice waveform generation model learning unit 216 of the voice waveform generation model learning processing unit 210 receives the weighted error and the voice waveform generation model as input, and outputs the voice waveform generation model after learning (step S106).

[0060] Then, the pruning unit 217 of the speech waveform generation model learning processing unit 210 receives the learned speech waveform generation model and the pruning block size as input, and outputs the pruned speech waveform generation model (step S107).

[0061] The above steps S101 to S107 are repeatedly executed for the speech waveforms and acoustic features contained in all the training data, thereby obtaining a pruned speech waveform generation model.

[0062] <Audio waveform generation process (Example 1)> The voice waveform generation process in this embodiment will be described below with reference to FIG.

[0063] The speech waveform generation unit 221 of the speech waveform generation processing unit 220 generates an acoustic feature x n=1,···,N and the pruned speech waveform generation model as input, the generated speech waveform ^y t=1,···,T (Step S201). At this time, the speech waveform generation unit 221 omits calculations of the pruned part and executes parallel calculations using SIMD with the block size at the time of pruning. This allows for high-speed generation of speech waveforms.

[0064] [Example 2] Example 2 of this embodiment will be described below. In this example, a subband signal generation model is trained instead of a speech waveform generation model, and a subband signal generated by the subband signal generation model is converted into a speech waveform. Note that in Example 2, differences from Example 1 will be described, and a description of components that may be the same as Example 1 will be omitted.

[0065] <Functional configuration of the voice waveform generation device 10 (Example 2)> An example of the functional configuration of the speech waveform generation device 10 in Example 2 is shown in Fig. 9. As shown in Fig. 9, the speech waveform generation device 10 in this example has a subband signal generation model learning processing unit 310 and a speech waveform generation processing unit 320. These units are realized, for example, by processing in which one or more programs installed in the speech waveform generation device 10 are executed by the processor 108. The speech waveform generation device 10 in this example also has a storage unit 330. The storage unit 330 is realized, for example, by the auxiliary storage device 107.

[0066] The subband signal generation model training processor 310 executes a subband signal generation model training process for training a subband signal generation model. An example of a detailed functional configuration of the subband signal generation model training processor 310 is shown in FIG. 10. As shown in FIG. 10, the subband signal generation model training processor 310 includes a transform unit 311, a subband signal generator 312, a generation error calculator 313, a weighting matrix shaping unit 314, a regularization term calculator 315, an error weighting unit 316, a subband signal generation model training unit 317, and a pruning unit 318.

[0067] The conversion unit 311 converts the speech waveform y t=1,···,T is used as input, and the subband signal

[0068]

number

[0069] The subband signal generator 312 generates the acoustic feature x n=1,···,N and the subband signal generation model are used as inputs to generate the subband signal

[0070]

number

[0071] The generation error calculation unit 313 receives the subband signals output from the conversion unit 311 and the generated subband signals output from the subband signal generation unit 312 as input, and outputs a generation error. For example, the generation error calculation unit 313 may calculate the generation error between the subband signal and the generated subband signal for each band b=1, . . . , B, or may calculate the generation error based on a joint distribution or the like in order to explicitly model the relationship between bands. For example, if the subband signals are continuous values, the generation error calculation unit 313 can calculate the generation error taking into account the relationship between all bands using a multivariate Gaussian distribution or the like.

[0072] The weight matrix forming unit 314 receives the subband signal generation model and the pruning block size as input, and outputs a formed weight matrix using the same method as in Example 1. However, the weight matrix used is the weight matrix of the neural network that realizes the subband signal generation model.

[0073] The regularization term calculation unit 315 receives the shaped weight matrix (tensor) as input and outputs the regularization term using the same method as in the first embodiment.

[0074] The error weighting unit 316 receives the generation error and the regularization term as input, and outputs a weighted error using the same method as in the first embodiment.

[0075] The subband signal generation model learning unit 317 receives the weighting error and the subband signal generation model as input, and outputs the learned subband signal generation model using the same method as in Example 1. Since the subband signal generation model is realized using a neural network, it is possible to learn the subband signal generation model using the same method as in Example 1.

[0076] The pruning unit 318 receives the trained subband signal generation model and the block size for pruning as input, and outputs the pruned subband signal generation model using the same method as in the first embodiment.

[0077] The audio waveform generation processing unit 320 executes audio waveform generation processing to generate audio waveforms. An example of a detailed functional configuration of the audio waveform generation processing unit 320 is shown in Fig. 11. As shown in Fig. 11, the audio waveform generation processing unit 320 in this embodiment includes a subband signal generation unit 321 and a conversion unit 322.

[0078] The subband signal generator 321 generates an acoustic feature x n=1,···,N and the pruned speech waveform generation model as input, the generated subband signal

[0079]

number

[0080] The conversion unit 322 receives the generated subband signals output from the subband signal generation unit 321 and converts them into the generated speech waveform ^y t=1,···,T The transform unit 322 can transform the generated subband signals into generated speech waveforms using, for example, a quadrature mirror filter (QMF) or a pseudo quadrature mirror filter (Pseudo QMF), which is the inverse transform of the transform unit 311.

[0081] The storage unit 330 stores various data (e.g., subband signal generation models, training data used when training the subband signal generation models, acoustic features used when generating speech waveforms, hyperparameters such as block sizes, etc.).

[0082] <Subband signal generation model learning process (Example 2)> The subband signal generation model training process in this embodiment will be described below with reference to Fig. 12. Steps S301 to S308 in Fig. 12 are repeatedly executed for the speech waveforms and acoustic features included in all training data. Steps S301 to S308 related to the speech waveforms and acoustic features included in certain training data will be described below.

[0083] The conversion unit 311 of the subband signal generation model learning processing unit 310 converts the speech waveform y t=1,···,T is used as input, and the subband signal

[0084]

number

[0085] Next, the subband signal generation unit 312 of the subband signal generation model learning processing unit 310 calculates the acoustic feature x n=1,···,N and the subband signal generation model are used as inputs to generate the subband signal

[0086]

number

[0087] Next, the generation error calculation unit 313 of the subband signal generation model learning processing unit 310 receives the subband signal output in the above step S301 and the generated subband signal output in the above step S302 as input, and outputs a generation error (step S303).

[0088] Next, the weight matrix forming unit 314 of the subband signal generation model learning processing unit 310 receives the subband signal generation model and the pruning block size as input, and outputs a formed weight matrix (step S304).

[0089] Next, the regularization term calculation unit 315 of the subband signal generation model learning processing unit 310 receives the shaped weight matrix (tensor) as input and outputs a regularization term (step S305).

[0090] Next, the error weighting unit 316 of the subband signal generation model learning processing unit 310 receives the generation error and the regularization term as input, and outputs a weighted error (step S306).

[0091] Next, the subband signal generation model learning unit 317 of the subband signal generation model learning processing unit 310 receives the weighted error and the subband signal generation model as input, and outputs the learned subband signal generation model (step S307).

[0092] Then, the pruning unit 318 of the subband signal generation model learning processing unit 310 receives the learned subband signal generation model and the pruning block size as input, and outputs the pruned subband signal generation model (step S308).

[0093] The above steps S301 to S308 are repeatedly executed for the speech waveforms and acoustic features included in all the training data, thereby obtaining a pruned subband signal generation model.

[0094] This allows pruning while explicitly specifying the block size for pruning, even in training of the subband signal generation model. Note that Non-Patent Document 6 places emphasis on the quality of synthesized speech and does not perform pruning.

[0095] In a subband signal generation model, it is necessary to appropriately model the relationships between multiple subband signals. Therefore, if regularization is not performed or if pruning is performed after regularization without considering the block size of pruning, the relationships between the subband signals will be disrupted, which will likely lead to a breakdown in speech waveform generation. In contrast, in this embodiment, pruning can be performed while maintaining the expressive power of the model, thereby reducing or avoiding this problem.

[0096] <Audio waveform generation process (Example 2)> The voice waveform generation process in this embodiment will be described below with reference to FIG.

[0097] The subband signal generator 321 of the speech waveform generator 320 generates an acoustic feature x n=1,···,N and the pruned speech waveform generation model as input, the generated subband signal

[0098]

number

[0099] Then, the conversion unit 322 of the speech waveform generation processing unit 320 receives the generated subband signals output in step S401 and converts them into the generated speech waveform ^y t=1,···,T is output (step S402).

[0100] [summary] The speech waveform generation device 10 in the first and second embodiments described above shapes a weighting matrix using the pruning block size during model training for generating synthetic speech, and calculates the regularization term using this shaped weighting matrix. In this way, the speech waveform generation device 10 in the first and second embodiments explicitly uses the pruning block size in calculating the regularization term. This causes the distribution of weighting coefficients at the time of regularization to be aligned continuously with the pruning block size, making it possible to explicitly separate portions that can be pruned from portions that must not be pruned. This makes it possible to efficiently reduce non-zero elements while preventing degradation of the quality of the synthetic speech, and as a result, it is possible to efficiently generate a speech waveform for synthetic speech with little degradation in quality.

[0101] The present invention is not limited to the above-described specifically disclosed embodiments, and various modifications, changes, and combinations with known technologies are possible without departing from the scope of the claims. [Explanation of symbols]

[0102] 10. Audio waveform generator 101 Input Device 102 Display device 103 External I / F 103a Recording media 104 Communication I / F 105 RAM 106 ROM 107 Auxiliary storage 108 processors 109 Bus 210 Speech waveform generation model learning processing unit 211 Audio waveform generator 212 Generation error calculation section 213 Weight matrix forming section 214 Regularization term calculation part 215 Error Weighting Unit 216 Voice waveform generation model training unit 217 Pruning Department 220 Audio waveform generation processing unit 221 Audio waveform generator 230 Storage section 310 Subband signal generation model learning processing unit 311 Conversion Unit 312 Subband signal generator 313 Generation error calculation section 314 Weight matrix forming section 315 Regularization term calculation part 316 Error Weighting Unit 317 Subband signal generation model training unit 318 Pruning Department 320 Audio waveform generation processing unit 321 Subband signal generator 322 Conversion Unit 330 Storage section

Claims

1. a shaping step of shaping the weight matrix using a block size used for pruning a weight matrix of a neural network that realizes a model for generating a speech waveform, so that the weight matrix is ​​composed of blocks of the block size unit; a regularization term calculation step of calculating a regularization term using the shaped weight matrix to promote all weights in a block including a weight to become zero when the weight becomes zero due to regularization; a learning procedure for learning a neural network that realizes the model using given training data and the regularization term; a pruning step of pruning a weight matrix of a neural network that realizes the trained model into units of the block size using the block size; The computer performs the audio waveform generation method.

2. The speech waveform generation method according to claim 1 , further comprising the step of generating the speech waveform using the pruned model and given acoustic features.

3. 2. The audio waveform generating method according to claim 1, wherein the block size is a size that allows blocks of the block size to be stored in an SIMD register.

4. 4. The speech waveform generation method according to claim 1, wherein the model is either a neural network that takes speech features as input and outputs a speech waveform, or a neural network that takes speech features as input and outputs subband signals obtained by band-dividing a speech waveform.

5. a shaping unit configured to shape the weighting matrix using a block size used for pruning a weighting matrix of a neural network that realizes a model for generating a speech waveform, so that the weighting matrix is ​​formed in blocks of the block size unit; a regularization term calculation unit configured to calculate a regularization term that, when a weight becomes zero due to regularization, promotes all weights in a block including the weight to become zero together, using the shaped weight matrix; and a learning unit configured to learn a neural network that realizes the model using given training data and the regularization term; a pruning unit configured to prune a weight matrix of a neural network that realizes the trained model in units of the block size using the block size; An audio waveform generating device having:

6. a shaping step of shaping the weight matrix using a block size used for pruning a weight matrix of a neural network that realizes a model for generating a speech waveform, so that the weight matrix is ​​composed of blocks of the block size unit; a regularization term calculation step of calculating a regularization term using the shaped weight matrix to promote all weights in a block including a weight to become zero when the weight becomes zero due to regularization; a learning procedure for learning a neural network that realizes the model using given training data and the regularization term; a pruning step of pruning a weight matrix of a neural network that realizes the trained model into units of the block size using the block size; A program that causes a computer to execute the following.

Images