Speech synthesis training device, speech synthesis device, speech synthesis training method, speech synthesis method, and program

Abstract

This speech synthesis training device includes: a speaker vector extraction unit that generates a speaker vector from a speech uttered by a speaker; and an update unit that updates, on the basis of an error between speech series information outputted by a speech synthesis model to which the speaker vector and text series information are inputted and speech series information as a correct answer corresponding to the text series information, a parameter of the speech synthesis model. The speech synthesis model includes: a parallel structure that generates an output corresponding to an input by using, among a plurality of modules having a parallel relationship, a part of the modules; and a route determination unit that determines the part of the modules among the plurality of modules on the basis of the speaker vector. Accordingly, speech synthesis with high naturalness and similarity is realized while suppressing an increase in the number of parameters.

Description

Speech synthesis training device, speech synthesis device, speech synthesis training method, speech synthesis method and program

[0001] The present invention relates to a voice synthesis training device, a voice synthesis device, a voice synthesis training method, a voice synthesis method, and a program.

[0002] In the field of speech synthesis, which takes text as input and generates speech that utters the contents of the text, DNN models have been proposed that use a single model to synthesize speech for multiple speakers from text and speaker information (for example, a method using Tacotron2 (Non-Patent Document 1) (Non-Patent Document 2) and a method using FastSpeech2 (Non-Patent Document 3) (Non-Patent Document 4)). Many of these methods condition the speech synthesis model with speaker vectors and train it with speech data from multiple speakers to obtain a model that can generate speech for multiple speakers. To extract speaker vectors, there are methods such as using x-vector (Non-Patent Document 5), which trains an extractor separately from the speech synthesis model, and a method using a speaker vector extractor based on a self-supervised learning model (SSL model), which trains a speech synthesis model and a speaker vector extractor simultaneously (Non-Patent Document 6).

[0003] The speaker vector extractor can extract speaker information even from speech that is not included in the training data, making it possible to generate speech for any speaker.

[0004] Shen, Jonathan, et al.、"Natural tts synthesis by conditioning wavenet on mel spectrogram predictions"、2018 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2018Cooper, Erica, et al.、"Zero-shot multi-speaker text-to-speech with state-of-the-art neural speaker embeddings"、ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020Ren, Yi, et al.、"FastSpeech 2: Fast and High-Quality End-to-End Text to Speech"、International Conference on Learning Representations. 2020Chien, Chung-Ming, et al.、"Investigating on incorporating pretrained and learnable speaker representations for multi-speaker multi-style text-to-speech"、ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021Snyder, David, et al.、"X-vectors: Robust dnn embeddings for speaker recognition"、2018 IEEE international conference on acoustics, speech and signal processing (ICASSP).IEEE, 2018Fujita, Kenichi, et al., "Zero-shot text-to-speech synthesis conditioned using self-supervised speech representation model", 2023 IEEE international conference on acoustics, speech and signal processing workshop (ICASSPW). IEEE, 2023.

[0005] However, if the number of parameters in the speech synthesis model is small, the generated speech may lose similarity to the target speaker and lose naturalness.

[0006] For example, when performing speech synthesis on devices such as smartphones or smartwatches, models with a large number of parameters cannot be used. Achieving high-quality speech synthesis with a model with a small number of parameters is important for the widespread use of speech synthesis technology.

[0007] The present invention has been made in view of the above points, and has as its object to realize speech synthesis with high naturalness and similarity while suppressing an increase in the number of parameters.

[0008] To solve the above problem, the speech synthesis training device includes a speaker vector extraction unit that generates a speaker vector from speech spoken by a speaker, and an update unit that updates the parameters of the speech synthesis model based on the error between speech sequence information output by the speech synthesis model to which the speaker vector and text sequence information are input and speech sequence information that is a correct answer corresponding to the text sequence information, and the speech synthesis model includes a parallel structure that generates an output in response to an input using some of a plurality of modules that have a parallel relationship, and a route determination unit that determines some of the plurality of modules based on the speaker vector.

[0009] It is possible to achieve speech synthesis with high naturalness and similarity while suppressing an increase in the number of parameters.

[0010] FIG. 1 is a diagram showing an example of the configuration of a speech synthesis model m1 in an embodiment of the present invention. FIG. 2 is a diagram showing an example of the configuration of a variance adapter 113. FIG. 3 is a diagram for explaining an example of the basic configuration of a speech feature predictor 30x. FIG. 4 is a diagram showing an example of the basic configuration of a speech synthesis model decoder 114. FIG. 5 is a diagram for explaining the parallel structure of a speech feature predictor 30x. FIG. 6 is a diagram for explaining the parallel structure of a speech synthesis model decoder 114. FIG. 7 is a diagram showing an example of the hardware configuration of a speech synthesis training device 10 in an embodiment of the present invention. FIG. 8 is a diagram showing an example of the functional configuration of a speech synthesis training device 10 in an embodiment of the present invention. FIG. 9 is a diagram showing an example of the functional configuration of a speech synthesis training device 10 in the case where a speaker vector extraction model m2 is also trained simultaneously in an embodiment of the present invention. FIG. 10 is a diagram showing an example of the functional configuration of a speech synthesis device 10a in an embodiment of the present invention.

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 shows an example of the configuration of a speech synthesis model m1 according to an embodiment of the present invention. The speech synthesis model m1 is a neural network that receives as input a speaker vector extracted from a reference target speaker's speech by a speaker vector extraction unit 12 and input text sequence information, and generates output speech sequence information, causing a computer to function as a speech synthesis unit 11. In the following description, the speech synthesis model m1 is equivalent to the speech synthesis unit 11. The output speech sequence information is a predicted value of the speech that will be obtained when a speaker associated with the input speaker vector speaks the input input text sequence information.

[0012] The speaker vector is a fixed-length vector that indicates characteristics of the speech of the speaker associated with the reference target speaker voice (hereinafter referred to as the "reference target speaker"), such as voice characteristics (voice quality, etc.) and speaking style (speaking slowly, rising pitch at the end of words).

[0013] The speaker vector extraction unit 12 is a machine learning model that extracts a speaker vector from a reference target speaker's speech. The reference target speaker's speech refers to the speech of a target speaker related to the output of the speech synthesis model m1. The speaker vector extraction unit 12 may use a speaker vector extractor that has been trained separately from the speech synthesis model m1, such as x-vector (Non-Patent Document 5), or may use a speaker vector extractor that has been trained together with the speech synthesis model m1, such as a method using a speaker vector extractor based on a self-supervised learning model (SSL model) (Non-Patent Document 6).

[0014] In FIG. 1, the voice synthesis model m1 (voice synthesis unit 11) includes a voice synthesis model encoder 111, an adding unit 112, a variance adapter 113, a voice synthesis model decoder 114, a route determining unit 115, and a route determining unit .

[0015] The speech synthesis model encoder 111 is a machine learning model that converts input text sequence information into post-encoder sequence information. This mainly converts information related to text. The post-encoder sequence information is a vector sequence in which vectors of a certain dimension (e.g., 256 dimensions) are arranged in a sequence equal to the length of the input text sequence information (e.g., the number of tokens).

[0016] The adding unit 112 adds (concatenates) the speaker vector to the encoded series information.

[0017] The variance adapter 113 predicts multiple types of speech information (information indicating speech characteristics or speech features) for the speech to be generated from information about text included in the post-encoder series information of the speaker-information-added series information and information about the speech of the speaker to be generated included in the speaker vector of the speaker-information-added series information. In this embodiment, an example will be described in which predictions are made for three types of speech information, including pitch, energy, and duration.

[0018] Fig. 2 is a diagram showing an example of the configuration of the variance adapter 113. As shown in Fig. 2, the variance adapter 113 includes three speech feature predictors 30x (x = a, b, or c), namely, a speech feature predictor 30a, a speech feature predictor 30b, and a speech feature predictor 30c.

[0019] FIG. 3 is a diagram illustrating an example of the basic configuration of a speech feature predictor 30x. While FIG. 3 shows the configuration of one speech feature predictor 30x, each of the three speech feature predictors 30x shown in FIG. 2 has the basic configuration shown in FIG. 3. As shown in FIG. 3, one speech feature predictor 30x includes one feature transformer 31x. The feature transformer 31x is a machine learning model that predicts speech information (pitch, energy, or duration) from series information after speaker information is added. For example, a normal MLP (Multilayer Perceptron) or the like can be used as the feature transformer 31x.

[0020] The variance adapter 113 concatenates or adds the predicted results (vectors) of pitch and energy to the speaker-information-added sequence information, and outputs the result as speaker-information / voice-information-added sequence information. Note that the duration is not used for concatenation or addition, but rather for converting the length of the speaker-information-added sequence information, which is input to the variance adapter 113 (output from the adding unit 112), into the sequence length of the voice. Generally, the lengths of the text sequence and the voice sequence are different (the number of frames of the generated voice is greater than the length of the text). Therefore, the variance adapter 113 estimates how many frames of voice each frame of text corresponds to using the voice feature predictor 30c, and performs conversion to match the length of the sequence output from the variance adapter 113 with the sequence length of the voice.

[0021] Returning to Figure 1, the speech synthesis model decoder 114 is a machine learning model that converts the speaker information and speech information-added sequence information into a predicted output speech sequence. The predicted output speech sequence is a predicted value of the output speech sequence (output value of the speech sequence). The conversion mainly involves speech-related information. The output speech sequence may be a vector that directly represents the speech waveform, or a vector that represents the speech waveform transformed by short-time Fourier transform, known as a spectrogram.

[0022] Fig. 4 is a diagram showing an example of the basic configuration of the speech synthesis model decoder 114. As shown in Fig. 4, the speech synthesis model decoder 114 includes, for example, multiple layers (L layers in Fig. 4) of feature quantity conversion units 41-i (j = 1 to L). As with FastSpeech2 (Non-Patent Document 3), a Transformer Encoder or the like can be used for the feature quantity conversion units 41-i.

[0023] 3 and 4, in this embodiment, each speech feature predictor 30x and speech synthesis model decoder 114 included in variance adapter 113 includes a parallel structure (a structure including modules (machine learning models) arranged in parallel) after feature conversion unit 31x or each feature conversion unit 41-i. Details of the parallel structure will be described later.

[0024] Returning to FIG. 2 , the route determination unit 115 is a machine learning model that determines, based on the speaker vector, some of the modules to be used for the parallel structure included in each speech feature predictor 30x. More specifically, the route determination unit 115 outputs, based on the speaker vector, a weight (probability) for each module included in the parallel structure included in each speech feature predictor 30x. In FIG. 2 , the route determination unit 115 is represented by one block, but a route determination unit 115 is provided for each parallel structure. Since one speech feature predictor 30x includes one parallel structure, there are three route determination units 115 for the variance adapter 113 for each speech feature predictor 30x.

[0025] The route determination unit 116 determines, based on the speaker vector, which module to use for each parallel structure included in the speech synthesis model decoder 114 for each feature transformation unit 41-i. More specifically, based on the speaker vector, the route determination unit 116 outputs a weight (probability) for each module included in each parallel structure for each feature transformation unit 41-i. Note that although the route determination unit 116 is represented by one block in FIG. 2, a route determination unit 116 is provided for each parallel structure. If the speech synthesis model decoder 114 includes L layers of feature transformation units 41-i, there are L route determination units 116 for the speech synthesis model decoder 114.

[0026] The parallel structure is configured by arranging in parallel multiple modules (machine learning models) each consisting of a small number of parameters called an adapter, which may be a structure combining a linear transformation and an activation function (see Reference 1, described below), for example.

[0027] 5 is a diagram for explaining the parallel structure of the speech feature predictor 30x, in which the same parts as those in FIG. 3 are given the same reference numerals and their explanations are omitted.

[0028] The speech feature predictor 30x of this embodiment includes, in the subsequent stage of a feature transformer 31x, a switch 32x, a plurality of (M) parallel units 33x-1 to 33x-M (hereinafter, when not distinguishing between them, referred to as "parallel units 33x"), and a parallel aggregation unit 34x. Here, the parallel unit 33x corresponds to the above-mentioned adapter. In other words, a parallel structure is formed by arranging the M parallel units 33x in parallel.

[0029] The switch 32x inputs the output value from the feature transform unit 31x as is to the parallel unit 33x determined to be used by the route determination unit 115x corresponding to the speech feature predictor 30x. The route determination unit 115x outputs weights (real numbers) whose total sum is 1 and whose individual weights are greater than 0 for a number (e.g., K) of parallel units 33x that are preset as hyperparameters, and outputs 0 as weights for the remaining M-K parallel units 33x. The value of K may be greater than or equal to 1 and less than K. For example, if the total number of parallel units 33x is 8 (M=8) and the preset number is 3, the route determination unit 115x outputs an array of weights in which only three parallel units 33x have values ​​greater than 0 (the sum is 1) and the other parallel units 33x have values ​​of 0. Furthermore, if the total number of parallel portions 33x is 8 (M=8) and the preset number is 1, the route determination unit 115x outputs an array of weights in which only one parallel portion 33x has a value of 1 and the other parallel portions 33x have values ​​of 0. In the above, "parallel portions 33x determined to be used by the route determination unit 115x" means parallel portions 33x whose weights output from the route determination unit 115x are greater than 0. Therefore, the route determination unit 115x determines to use the preset number of parallel portions 33x.

[0030] The parallel aggregation unit 34x calculates a weighted sum based on the weights output from the route determination unit 115x for the output values ​​from the parallel units 33x to which the output values ​​from the feature conversion unit 31x are input by the switch 32x (i.e., the parallel units 33x determined to be used by the route determination unit 115x), and outputs the weighted sum as prediction information of the audio features.

[0031] 6 is a diagram for explaining the parallel structure of the voice synthesis model decoder 114. In FIG. 6, the same parts as those in FIG. 4 are given the same reference numerals, and their explanation will be omitted.

[0032] The speech synthesis model decoder 114 of this embodiment includes, after each feature conversion unit 41-i, a switch 42-i, a plurality of (N) parallel units 43-i-1 to 43-i-N (hereinafter, when not distinguishing between them, referred to as "parallel units 43-i-j"), and a parallel aggregation unit 44-i. Here, the parallel units 43-i-j correspond to the above-mentioned adapters. In other words, a parallel structure is formed by arranging the N parallel units 43-i-j in parallel.

[0033] Furthermore, the speech synthesis model decoder 114 includes a parallel structure for each feature transform unit 41-i. Therefore, the speech synthesis model decoder 114 including the L-layer feature transform units 41-i includes L parallel structures. As is clear from FIG. 6 , a route determination unit 116 actually exists corresponding to each of the L parallel structures. That is, the route determination unit 116-i corresponds to the j-th parallel structure. The processing content executed by each parallel structure included in the speech synthesis model decoder 114 is the same as that of the parallel structure included in the speech feature predictor 30x. The output from the parallel aggregation unit 44-i in the final-stage parallel structure is predicted output speech sequence information as the output from the speech synthesis model m1 coder.

[0034] 5 and 6, the location where the parallel structure is inserted is not limited to a specific location as long as it is inside the speech feature predictor 30x or the speech synthesis model decoder 114. One parallel structure may be inserted for one feature transformer 31x or one feature transformer 41-i.

[0035] Next, we will explain the speech synthesis training device 10 that trains the above-mentioned speech synthesis model m1. Figure 7 is a diagram showing an example of the hardware configuration of the speech synthesis training device 10 in an embodiment of the present invention. The speech synthesis training device 10 in Figure 7 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a processor 104, and an interface device 105, all of which are connected to each other via a bus B.

[0036] The program that realizes the processing in the speech synthesis training device 10 is provided by a recording medium 101 such as a CD-ROM. When the recording medium 101 storing the program is set in the drive device 100, the program is installed from the recording medium 101 to the auxiliary storage device 102 via the drive device 100. However, the program does not necessarily have to be installed from the recording medium 101; it may be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program as well as necessary files, data, etc.

[0037] When an instruction to start the program is received, the memory device 103 reads and stores the program from the auxiliary storage device 102. The processor 104 is a CPU or a GPU (Graphics Processing Unit), or a CPU and a GPU, and executes functions related to the speech synthesis training device 10 in accordance with the program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network.

[0038] FIG. 8 is a diagram showing an example of the functional configuration of a speech synthesis training device 10 according to an embodiment of the present invention. In FIG. 8, the speech synthesis training device 10 includes a text preprocessing unit 13, a speech preprocessing unit 14, a speaker vector extraction unit 12, a speech synthesis unit 11, and an update unit 15. Each of these units is implemented by one or more programs installed in the speech synthesis training device 10, which are executed by the processor 104. For example, the speech synthesis unit 11 is implemented by a speech synthesis model m1, which is executed by the processor 104. Furthermore, the speaker vector extraction unit 12 is implemented by a speaker vector extraction model m2, which is executed by the processor 104. In other words, the speaker vector extraction model m2 is a machine learning model that causes a computer to function as the speaker vector extraction unit 12.

[0039] The voice synthesis training device 10 also uses a training data storage unit 16. The training data storage unit 16 can be realized using, for example, the auxiliary storage device 102 or a storage device connectable to the voice synthesis training device 10 via a network.

[0040] The training data storage unit 16 stores a set of training data for the voice synthesis model m1. Each training data is a pair of text data (hereinafter referred to as "training text data") and speech data (hereinafter referred to as "training speech data") obtained by speaking the contents of the text data.

[0041] The text preprocessing unit 13 converts the learning text data into text sequence information (input text sequence information). The input text sequence information is, for example, a language vector obtained by vectorizing text information.

[0042] The speech pre-processing unit 14 converts the training speech data into output speech sequence information (correct output speech sequence information). The output speech sequence information is, for example, a mel spectrogram.

[0043] The speaker vector extraction unit 12 generates a speaker vector from the correct output speech sequence information.

[0044] The speech synthesis unit 11 receives the speaker vector and input text sequence information based on the speech synthesis model m1 according to the learning results up to that point, and outputs output speech sequence information (predicted output speech sequence information) through the above-mentioned processing.

[0045] The update unit 15 receives the correct output speech sequence information and the predicted output speech sequence information, and updates the parameters of the speech synthesis model m1 so as to reduce the error between the correct output speech sequence information and the predicted output speech sequence information. In this way, the speech synthesis model m1 learns the correspondence between the input text sequence information and speaker vectors and the speech sequence information.

[0046] The parameters to be updated are those of the speech synthesis model encoder 111, variance adapter 113, speech synthesis model decoder 114, and route determiner. The parameters of the variance adapter 113 are those of the feature transformer 31x and parallel unit 33x of each speech feature predictor 30x. The parameters of the speech synthesis model decoder 114 are those of the feature transformer 41-i, parallel unit 43-i-j, and parallel aggregation unit 44-i. Furthermore, as shown in FIG. 9 , the update unit 15 may also update (train) the parameters of the speaker vector extraction model m2 simultaneously (in parallel) with training of the speech synthesis model m1 so as to reduce the error between the correct output speech sequence information and the predicted output speech sequence information.

[0047] In this way, each parallel structure is also trained. As a result, a speech synthesis model m1 is trained in which the parallel portion 33x and the parallel portion 43-ij to be used are switched depending on the speaker vector, and the appropriate parallel portion 33x and the parallel portion 43-ij are selected depending on the speaker vector. In other words, each module in the parallel structure specializes in speakers with different voice characteristics (similar modules are selected for speakers with similar voices, and different modules are selected for different speakers). By switching the route (module) for each speaker, speech synthesis with high naturalness and similarity is possible even with a small number of model parameters.

[0048] Although the present embodiment describes an example in which both variance adapter 113 and voice synthesis model decoder 114 have a parallel structure, only one of them may have a parallel structure. For example, only voice synthesis model decoder 114 may have a parallel structure, or only variance adapter 113 may have a parallel structure.

[0049] Next, a description will be given of a speech synthesis device 10a that generates speech corresponding to input text using the trained speech synthesis model m1. The hardware configuration of the speech synthesis device 10a may be the same as that shown in FIG.

[0050] Fig. 10 is a diagram showing an example of the functional configuration of a speech synthesis device 10a according to an embodiment of the present invention. In Fig. 10, the same components as those in Fig. 1 are designated by the same reference numerals. As shown in Fig. 10, the speech synthesis device 10a includes a speech synthesis unit 11 and a speaker vector extraction unit 12.

[0051] The speech synthesis device 10a executes the following processes in response to input of text (input text sequence information) and reference target speaker speech. (1) The speaker vector extraction unit 12 generates a speaker vector based on the reference target speaker speech. (2) The speech synthesis unit 11 executes the learned process of the speech synthesis model m1 based on the input text sequence information and the speaker vector of (1), and outputs predicted output speech sequence information. At this time, the parallel unit 33x and the parallel unit 43-i-j to be used are switched depending on the speaker vector.

[0052] During inference, the reference target speaker's speech does not have to be the speech that corresponds to the content of the input text. In this case, the speech feature predictor 30x corresponding to the duration estimates the duration from speech that is unrelated to the text. However, if the armband target speaker has a speaking tendency (for example, if they speak quickly or tend to extend their sentences), that tendency will be included as feature information in the speech even if the content of the text changes. Therefore, even if the speech content of the reference target speaker's speech is unrelated to the content of the input text, it is possible to estimate the duration to some extent because it can reflect the likeliness of the reference target speaker.

[0053] As described above, according to this embodiment, by switching between modules to be used from among a plurality of modules having a parallel relationship according to the characteristics of the speaker, it is possible to realize speech synthesis with high naturalness and similarity while suppressing an increase in the number of parameters.

[0054] [Reference 1] Houlsby, Neil, et al., "Parameter-efficient transfer learning for NLP," International Conference on Machine Learning. PMLR, 2019. While the embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the gist of the present invention as defined in the claims.

[0055] 10 Speech synthesis training device 10a Speech synthesis device 11 Speech synthesis unit 12 Speaker vector extraction unit 13 Text preprocessing unit 14 Speech preprocessing unit 15 Update unit 16 Training data storage unit 30a Speech feature predictor 30b Speech feature predictor 30c Speech feature predictor 30x Speech feature predictor 31x Feature conversion unit 32x Switch 33x Parallel unit 34x Parallel aggregation unit 41 Feature conversion unit 41-i Feature conversion unit 42-i Switch 43-i-j Parallel unit 44-i Parallel aggregation unit 100 Drive device 101 Recording medium 102 Auxiliary storage device 103 Memory device 104 Processor 105 Interface device 111 Speech synthesis model encoder 112 Addition unit 113 Variance adapter 114 Speech synthesis model decoder 115 Route determination unit 116 Route determination unit B Bus m1 Speech synthesis model m2 Speaker vector extraction model

Claims

1. A speech synthesis training device comprising: a speaker vector extraction unit that generates a speaker vector from speech spoken by a speaker; and an update unit that updates parameters of the speech synthesis model based on an error between speech sequence information output by the speech synthesis model to which the speaker vector and text sequence information are input and speech sequence information that is a correct answer corresponding to the text sequence information, wherein the speech synthesis model has a parallel structure that generates an output in response to an input using some of a plurality of modules that have a parallel relationship; and a route determination unit that determines some of the plurality of modules based on the speaker vector.

2. The speech synthesis training device according to claim 1, characterized in that the speech synthesis model includes: a speech feature predictor that generates speech information indicating speech features from a vector sequence generated from the text sequence information and the speaker vector; and a speech synthesis model decoder that generates the speech sequence information from the vector sequence, the speaker vector, and the speech information; and at least one of the speech feature predictor and the speech synthesis model decoder has the parallel structure.

3. The speech synthesis training device according to claim 1 or 2, characterized in that the route determination unit outputs a weight for each of the partial modules based on the speaker vector, and the parallel structure outputs a weighted sum based on the weights for the outputs from the partial modules.

4. A speech synthesis device comprising: a speaker vector extraction unit that generates a speaker vector from speech uttered by a speaker; and a speech synthesis unit that generates speech sequence information for input text sequence information and the speaker vector generated by the speaker vector extraction unit using a speech synthesis model that has been trained to understand the correspondence between text sequence information and speaker vectors and speech sequence information, wherein the speech synthesis model has a parallel structure that generates output for an input using some of a plurality of modules that have a parallel relationship; and a route determination unit that determines some of the plurality of modules based on the speaker vector.

5. A speech synthesis training method comprising: a speaker vector extraction procedure for generating a speaker vector from speech uttered by a speaker; and an update procedure for updating parameters of a speech synthesis model based on an error between speech sequence information output by a speech synthesis model to which the speaker vector and text sequence information have been input and speech sequence information serving as a correct answer corresponding to the text sequence information; wherein the speech synthesis model has a parallel structure that generates an output in response to an input using some of a plurality of modules having a parallel relationship; and a route determination unit that determines some of the plurality of modules based on the speaker vector.

6. A speech synthesis method comprising: a computer executing a speaker vector extraction procedure for generating a speaker vector from speech uttered by a speaker; and a speech synthesis procedure for generating speech sequence information for input text sequence information and the speaker vector generated by the speaker vector extraction procedure, using a speech synthesis model that has been trained to learn correspondences between text sequence information and speaker vectors and speech sequence information; wherein the speech synthesis model has a parallel structure that generates output for an input using some of a plurality of modules that have a parallel relationship; and a route determination unit that determines some of the plurality of modules based on the speaker vector.

7. A program causing a computer to execute the following steps: a speaker vector extraction procedure for generating a speaker vector from speech uttered by a speaker; and an update procedure for updating parameters of a speech synthesis model based on an error between speech sequence information output by a speech synthesis model to which the speaker vector and text sequence information have been input and speech sequence information as a correct answer corresponding to the text sequence information, wherein the speech synthesis model has a parallel structure that generates an output in response to an input using some of a plurality of modules having a parallel relationship; and a route determination unit that determines some of the plurality of modules based on the speaker vector.

8. A program that causes a computer to execute: a speaker vector extraction procedure that generates a speaker vector from speech uttered by a speaker; and a speech synthesis procedure that generates speech sequence information for input text sequence information and the speaker vector generated by the speaker vector extraction procedure, using a speech synthesis model that has been trained to understand the correspondence between text sequence information and speaker vectors and speech sequence information, wherein the speech synthesis model has a parallel structure that generates output for an input using some of a plurality of modules that have a parallel relationship; and a route determination unit that determines some of the plurality of modules based on the speaker vector.

Images