Speech synthesis training method, speech synthesis method, speech synthesis training device, speech synthesis device, and program

Abstract

According to the present invention, the reproducibility of speech synthesis in a case where noise is included in reference target speaker speech is improved by causing a computer to execute: a first speaker vector extraction procedure for, concerning speech data in which an utterance is recorded, generating a first speaker vector indicating a speaker's feature related to the utterance on the basis of a self-supervised learning model; a second speaker vector extraction procedure for, concerning noise-added speech data in which noise is added to the speech data, generating a second speaker vector indicating a speaker's feature related to the utterance on the basis of the self-supervised learning model and one or more conversion models that execute additional conversion processing in series or in parallel with a plurality of conversion processes in the self-supervised learning model; a text-to-speech conversion procedure for inputting vector information generated from text data indicating the content of the utterance and the first speaker vector or the second speaker vector, and outputting speech sequence information; a pre-update procedure for updating a parameter used by the first speaker vector extraction procedure and the text-to-speech conversion procedure such that, concerning the first speaker vector, the speech sequence information outputted by the text-to-speech conversion procedure and speech sequence information extracted from the speech data approach each other; and an additional update procedure for updating a parameter used by the second speaker vector extraction procedure, the text-to-speech conversion procedure, and the conversion model such that, concerning the second speaker vector, the speech sequence information outputted by the text-to-speech conversion procedure the parameter of which has been updated by the pre-update procedure and speech sequence information extracted from the speech data approach each other.

Description

Speech synthesis training method, speech synthesis method, speech synthesis training device, speech synthesis device and program

[0001] The present invention relates to a speech synthesis training method, a speech synthesis method, a speech synthesis training device, a speech synthesis device, and a program.

[0002] Currently, in the field of speech synthesis, DNN models have been proposed that use a single model to synthesize speech for multiple speakers from text and speaker information (for example, a method using Tacotron2 (Non-Patent Document 1) (Non-Patent Document 2) and a method using FastSpeech2 (Non-Patent Document 3) (Non-Patent Document 4)). In many of these methods, a model capable of generating speech for multiple speakers is obtained by conditioning the speech synthesis model with speaker vectors and training it with speech data from multiple speakers. To extract speaker vectors, there is a method that uses x-vector (Non-Patent Document 5), which trains an extraction unit separately from the speech synthesis model.

[0003] Shen, Jonathan, et al.、"Natural tts synthesis by conditioning wavenet on mel spectrogram predictions"、2018 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2018Cooper, Erica, et al、"Zero-shot multi-speaker text-to-speech with state-of-the-art neural speaker embeddings"、ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020Ren, Yi, et al.、"FastSpeech 2: Fast and High-Quality End-to-End Text to Speech"、International Conference on Learning Representations. 2020Chien, Chung-Ming, et al.、"Investigating on incorporating pretrained and learnable speaker representations for multi-speaker multi-style text-to-speech"、ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021Snyder, David, et al.、"X-vectors: Robust dnn embeddings for speaker recognition"、2018 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2018。

[0004] In multi-speaker speech synthesis, a speech synthesis model is conditioned using speaker vectors obtained from x-vectors or other sources, and trained on speech data from multiple speakers. The speaker vector extraction unit can extract speaker information even from speech not included in the training data, making it possible to generate speech from any speaker. However, if noise is added to the reference target speaker speech input to the speaker vector extraction unit, accurate speaker vector extraction cannot be performed, resulting in a decrease in the similarity of the generated speech to the target speaker and a decrease in quality.

[0005] For example, when generating the voice of someone who has lost their voice due to illness by referencing that person's past speech, the past speech was not necessarily recorded in a low-noise recording environment that is intended for speech synthesis, so it is important to be able to synthesize high-quality speech even when the reference target speaker's speech contains noise.

[0006] The present invention has been made in view of the above points, and has as its object to improve the reproducibility of speech synthesis when noise is included in the reference target speaker's speech.

[0007] Therefore, in order to solve the above problem, a first speaker vector extraction procedure is provided for generating, for audio data in which an utterance has been recorded, a first speaker vector indicating characteristics of a speaker related to the utterance based on a self-supervised learning model; a second speaker vector extraction procedure is provided for generating, for noise-added audio data in which noise has been added to the audio data, a second speaker vector indicating characteristics of a speaker related to the utterance based on the self-supervised learning model and one or more conversion models that perform additional conversion processes in series or in parallel to a plurality of conversion processes in the self-supervised learning model; and a method of extracting a second speaker vector from text data indicating the content of the utterance and the first speaker vector or the second speaker vector. a pre-update procedure for updating parameters used by the first speaker vector extraction procedure and the text-to-speech conversion procedure so that the speech sequence information output by the text-to-speech conversion procedure for the first speaker vector and the speech sequence information extracted from the speech data come closer to each other; and an additional update procedure for updating parameters used by the second speaker vector extraction procedure, the text-to-speech conversion procedure, and the conversion model so that the speech sequence information output by the text-to-speech conversion procedure for the second speaker vector, the parameters of which have been updated by the pre-update procedure, comes closer to the speech sequence information extracted from the speech data.

[0008] The reproducibility of speech synthesis can be improved when noise is included in the reference target speaker's speech.

[0009] FIG. 1 is a diagram illustrating an example of a hardware configuration of a speech synthesis device 10 according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an example of a functional configuration of the speech synthesis device 10 during pre-learning according to an embodiment of the present invention. FIG. 3 is a diagram illustrating a speech feature extraction unit 13. FIG. 4 is a diagram illustrating a speech synthesis model 14 during learning. FIG. 5 is a flowchart illustrating an example of a processing procedure for pre-learning. FIG. 6 is a diagram illustrating an example of a functional configuration of the speech synthesis device 10 during additional learning according to an embodiment of the present invention. FIG. 7 is a diagram illustrating an adapter-equipped speech feature extraction unit 13a. FIG. 8 is a flowchart illustrating an example of a processing procedure for additional learning. FIG. 9 is a diagram illustrating an example of a functional configuration of the speech synthesis device 10 during inference according to an embodiment of the present invention.

[0010] This embodiment discloses a technology related to a speech synthesis method using a speaker vector extraction unit based on a self-supervised learning (SSL) model acquired by self-supervised learning. In this method, a linguistic information vector is used as input to the speech synthesis model, and a reference target speaker speech is used as input to the speaker vector extraction unit to generate a target speaker speech corresponding to the linguistic information vector. This speaker vector extraction unit is trained simultaneously with the speech synthesis model. The reference target speaker speech refers to speech that is referenced as the speech of a target speaker in speech synthesis.

[0011] In this embodiment, in order to perform noise-robust speech synthesis, training is performed in two stages. In the first stage (pre-training), a speech synthesis model is trained in the same way as in existing methods. In the second stage (additional training), additional training is performed using the parameters of the model trained in the first stage as initial parameters. During the additional training, speech in which noise is probabilistically added to the reference target speaker speech is used. Furthermore, during the additional training, a trainable layer (model) consisting of a small number of parameters called an adapter is inserted into the middle of the self-supervised learning model so that the self-supervised learning model can domain adapt to speech with added noise. This makes the speaker vector extraction unit robust to noise even when noise is added to the speech, making it possible to generate high-quality speech.

[0012] An embodiment of the present invention will now be described with reference to the accompanying drawings. Fig. 1 is a diagram showing an example of the hardware configuration of a speech synthesizer 10 according to an embodiment of the present invention. The speech synthesizer 10 shown in Fig. 1 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a processor 104, and an interface device 105, all of which are interconnected via a bus B.

[0013] The program that realizes the processing in the speech synthesizer 10 is provided by a recording medium 101 such as a CD-ROM. When the recording medium 101 storing the program is set in the drive device 100, the program is installed from the recording medium 101 to the auxiliary storage device 102 via the drive device 100. However, the program does not necessarily have to be installed from the recording medium 101, but may be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program as well as necessary files, data, etc.

[0014] When an instruction to start the program is received, the memory device 103 reads and stores the program from the auxiliary storage device 102. The processor 104 is a CPU or a GPU (Graphics Processing Unit), or a CPU and a GPU, and executes functions related to the speech synthesizer 10 in accordance with the program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network.

[0015] FIG. 2 is a diagram showing an example of the functional configuration of the speech synthesizer 10 during pre-training according to an embodiment of the present invention. In FIG. 2, the speech synthesizer 10 includes a text pre-processing unit 11, a speech pre-processing unit 12, a speech feature extraction unit 13, a speech synthesis model 14, a speech synthesis unit 15, and a pre-update unit 16. Each of these units is implemented by a processor 104 executing one or more programs installed in the speech synthesizer 10. The speech synthesizer 10 also uses a paired data storage unit 121. The paired data storage unit 121 can be implemented using, for example, the auxiliary storage device 102 or a storage device connectable to the speech synthesizer 10 via a network.

[0016] The paired data storage unit 121 stores a set of paired data of text data and voice data as training data for the voice synthesis model 14. The text data is a character string indicating the content of an utterance. The voice data is voice data in which the utterance (voice) is recorded. In other words, the voice data is data in which the voice obtained by speaking the character string indicated by the text data is recorded, and is voice data that is the correct answer to the output from the voice synthesis model 14 to which the training data has been input. The voice data does not contain noise.

[0017] The text preprocessing unit 11 converts the text data in the pair data into input text sequence information, which is, for example, a language vector obtained by vectorizing text information.

[0018] The speech pre-processing unit 12 converts the speech data in the paired data into speech sequence information, which is information necessary (used for synthesis) for synthesizing the speech represented by the speech data. The speech sequence information is, for example, a mel spectrogram. The speech sequence information generated by the speech pre-processing unit 12 is treated as a correct answer for speech synthesis that the speech synthesis model 14 learns. Hereinafter, the speech sequence information generated by the speech pre-processing unit 12 will be referred to as "correct answer speech sequence information."

[0019] The speech feature extraction unit 13 extracts a series of features from speech data based on a self-supervised learning model 131 .

[0020] FIG. 3 is a diagram illustrating the speech feature extraction unit 13. As shown in FIG. 3, the speech feature extraction unit 13 includes a self-supervised learning model 131. The self-supervised learning model 131 is a model that, when performing learning to extract speech features, is trained using targets obtained from the speech data itself (e.g., estimation of the masked portion of a partially masked speech), rather than learning using manually assigned labels as the correct answer. Existing self-supervised learning models 131 include models such as Wav2vec2 (Reference 1) and HuBERT (Reference 2). For example, Wav2vec2 and HuBERT, which serve as self-supervised learning models 131, are composed of multiple Transformer layers.

[0021] In FIG. 3 , the self-supervised learning model 131 also has a structure in which multiple layers of structures, such as a one-layer CNN feature extraction unit and M-layer Transformers, are stacked. Each layer (CNN feature extraction unit and M-layer Transformers) includes multiple feature transformation units (such as a feed-forward network (FFN) module in a Transformer, a multi-head self-attention (MHSA) module, or a one-dimensional CNN layer module in the CNN feature transformation extraction unit) that transform input features. The speech feature extraction unit 13 can obtain each feature sequence output from each layer of the self-supervised learning model 131 to which speech data is input. The feature sequence is a hidden-layer / frame-level speaker vector sequence, which is a sequence containing as many feature sequences as the number of layers, each with a length corresponding to the sequence length of the speech data. For example, if the self-supervised learning model 131 is a neural network consisting of a one-layer CNN feature extraction unit and 12 layers of Transformers, 13 feature sequences, each with a length corresponding to the sequence length of the speech data, are obtained as the output of each layer.

[0022] The speech synthesis model 14 is a model (neural network) that receives the intermediate layer / frame-level speaker vector sequence extracted from speech data by the speech feature extraction unit 13 and input text sequence information, and generates (outputs) output speech sequence information. As shown in Figure 4, the speech synthesis model 14 includes a fixed-dimensional vector conversion unit 141 and a text-to-speech conversion unit 142.

[0023] The fixed-dimensional vector conversion unit 141 converts the intermediate-layer / frame-level speaker vector sequence, which is a feature sequence obtained for each layer and has a length corresponding to the sequence length of the speech data, into a speaker vector, which is a vector of fixed length, using trainable parameters. That is, the speaker vector is a fixed-length vector extracted from speech by converting the intermediate-layer / frame-level speaker vector sequence extracted from the speech feature extraction unit 13 using the fixed-dimensional vector conversion unit 141, and represents the features of the speaker. Because the speech data is converted into a speaker vector through the speech feature extraction unit 13 and the fixed-dimensional vector conversion unit 141, the speech feature extraction unit 13 and the fixed-dimensional vector conversion unit 141 will hereinafter be collectively referred to as the "speaker vector extraction unit 17."

[0024] For example, the fixed-dimension vector conversion unit 141 first multiplies the feature sequences obtained from each layer by weights calculated based on the importance of each layer, and then adds them together. As a result, for example, 12 sequences obtained from the speech feature extraction unit 13 including the 12-layer self-supervised learning model 131 become a sequence of a length corresponding to the sequence length of one piece of speech data. Next, the fixed-dimension vector conversion unit 141 aggregates the dimensions corresponding to the sequence length of the speech data into a single dimension. For example, the sequence can be converted into a fixed-length vector that does not correspond to the sequence length of the speech data by processing the sequence using LSTM (Reference 3) and obtaining the final output, or by calculating the average vector of the entire sequence. It has been suggested that each layer of a self-supervised learning model has different information about speech (Reference 4). Therefore, by obtaining a sequence with different weights for each speech feature, features specialized for each speech feature can be extracted.

[0025] The procedure for extracting a speaker vector from speech is shown below.

[0026] (1) Speech is input to the speech feature extraction unit 13 based on the self-supervised learning model 131. The speech feature extraction unit 13 obtains features of a sequence length corresponding to the length of the speech from each layer of the self-supervised learning model 131. For example, if the CNN feature extraction unit has one layer, the Transformer has M layers, and the number of frames in the sequence length is L, features of (1+M)×L×D dimensions are obtained as an intermediate layer / frame-level speaker vector sequence (D is the dimension for each frame of the sequence output from each layer).

[0027] (2) A speaker vector is obtained using the fixed-dimensional vector conversion unit 141. For example, in the case of a feature of (1+M)×L×D dimension, the fixed-dimensional vector conversion unit 141 converts the feature into a feature of 1×L×D dimension by taking a weighted sum for each layer. The fixed-dimensional vector conversion unit 141 further converts the 1×L×D dimension feature into a feature of a fixed-dimensional vector of 1×1×D by calculating the average vector of the entire sequence or using LSTM, for example. A speaker vector of any dimension can be obtained by converting this vector into an appropriate dimension D' using linear transformation, for example.

[0028] The text-to-speech converter 142 receives input text sequence information and a speaker vector, and converts the input information into output speech sequence information using trainable parameters. The text-to-speech converter 142 uses, for example, a model based on a non-autoregressive speech synthesis method based on FastSpeech2 or an autoregressive speech synthesis method based on Tacotron2.

[0029] The speech synthesis unit 15 generates output speech sequence information according to input text sequence information and intermediate layer / frame level speaker vector sequences using the speech synthesis model 14 .

[0030] The pre-update unit 16 updates the parameters used by the voice synthesis model 14 using a set of paired data in pre-learning.

[0031] FIG. 5 is a flowchart illustrating an example of a procedure for pre-learning.

[0032] In step S110, the text preprocessing unit 11 converts text data of a certain pair of data stored in the pair data storage unit 121 into input text sequence information.

[0033] Next, the speech pre-processing unit 12 converts the noise-free speech data of the paired data into correct output speech sequence information (S120).

[0034] Next, the speech feature extraction unit 13 extracts an intermediate layer / frame level speaker vector sequence from the speech data that does not contain noise of the paired data (S130).

[0035] Next, the speech synthesis unit 15 inputs the input text sequence information and the inter-layer / frame-level speaker vector sequence to the speech synthesis model 14, and obtains (predicted) output speech sequence information generated by the speech synthesis model 14 (S140).

[0036] Next, the advance update unit 16 uses the (predicted) output speech sequence and the correct output speech sequence information to train parameters used by the speech synthesis model 14 (the text-to-speech conversion unit 142 and the fixed-dimensional vector conversion unit 141) (S150). Specifically, the advance update unit 16 updates the parameters used by the text-to-speech conversion unit 142 and the fixed-dimensional vector conversion unit 141 so that the (predicted) output speech sequence information approaches the correct output speech sequence information. Here, the fixed-dimensional vector conversion unit 141 is a component of the speaker vector extraction unit 17. Therefore, updating the parameters of the fixed-dimensional vector conversion unit 141 can also be said to be updating the parameters of the speaker vector extraction unit 17. At this time, the parameters of the self-supervised learning model 131 may also be updated.

[0037] Steps S110 to S150 are executed for each paired data stored in the paired data storage unit 121 until it is determined that the (predicted) output speech sequence is sufficiently close to the correct output speech sequence information.

[0038] Next, additional learning will be described. Fig. 6 is a diagram showing an example of the functional configuration of the speech synthesis device 10 during additional learning according to an embodiment of the present invention. In Fig. 6, parts that are the same as or correspond to those in Fig. 2 are given the same reference numerals, and their description will be omitted. In Fig. 6, an adapter 132 for the speech feature extraction unit, an adding unit 18, and a noise sound data storage unit 122 have been added. In Fig. 6, the speech feature extraction unit 13 and the advance update unit 16 have also been replaced with an adapter-equipped speech feature extraction unit 13a and an additional update unit 19.

[0039] The adaptor-equipped speech feature extraction unit 13a is an extraction unit (FIG. 7) in which an adaptor 132 for speech feature extraction (hereinafter simply referred to as "adapter") is inserted into the self-supervised learning model 131 used in pre-training. Even with the adapter inserted, it is possible to extract intermediate layer / frame-level speaker vector sequences in the same way as the speech feature extraction unit 13 before the adapter is inserted.

[0040] The self-supervised learning model 131 extracts excellent features from speech. In this embodiment, in order to domain-adapt the features to inputs of different domains (e.g., speech with added noise) while preserving the features, a conversion model (e.g., a machine learning model having a structure combining a linear transformation and an activation function (Reference 5)) called an adapter, which performs a conversion process using a small number of learnable parameters, is inserted into the self-supervised learning model 131. As shown in FIG. 7 , the adapter is inserted in parallel (e.g., adapter1-1 in FIG. 7 ) or in series (e.g., adapter1-2 in FIG. 7 ) with the modules (feature conversion units) included in each layer (CNN feature extraction unit and Transformer) included in the self-supervised learning model 131. Therefore, by inserting the adapter, the adapter-equipped speech feature extraction unit 13a performs additional conversion processes in parallel or series with the multiple conversion processes performed by the self-supervised learning model 131. However, it is not necessary to insert an adapter into all modules (feature conversion units). During additional learning, the parameters of the self-supervised learning model 131 are fixed, while the parameters used by the adapter are trainable.

[0041] The noise sound data storage unit 122 stores noise sound data, which is a database containing a large amount of noise sound. To make the speech synthesis model 14 robust against all kinds of noise sound, it is desirable that the noise contained in one noise sound data be diverse. Furthermore, to make the speaker vector extraction unit 17 robust against noise not contained in the noise sound data, it is desirable that there are several thousand or more pieces of noise sound data. If the number of noise sound data is small, the speaker vector extraction unit 17 will over-learn the noise sound data, reducing robustness against unknown noise. An example of noise sound data is MUSAN (Reference 6).

[0042] The adding unit 18 generates noise-added audio data by adding noise selected from the noise sound data to the audio data of each paired data (each paired data used in the pre-learning) stored in the paired data storage unit 121 .

[0043] In the additional learning, the additional update unit 19 updates the parameters of the pre-trained voice synthesis model 14 using a set of pair data in which noise is added to voice data.

[0044] The voice feature extraction adapter 132, the addition unit 18, and the addition / update unit 19 are realized by the processor 104 executing one or more programs installed in the voice synthesizer 10. The noise sound data storage unit 122 can be realized using, for example, the auxiliary storage device 102 or a storage device connectable to the voice synthesizer 10 via a network.

[0045] 8 is a flowchart for explaining an example of a processing procedure for additional learning. In FIG. 8, the same steps as those in FIG. 5 are assigned the same step numbers, and their explanations will be omitted as appropriate.

[0046] 8, steps S130 and S150 are replaced with steps S130a and S150a, respectively, and step S125 is executed before step S130a.

[0047] In step S125, the adding unit 18 generates noise-added audio data by adding noise selected from the noise sound data to the audio data of the paired data processed in steps S110 and S120.

[0048] In step S130a, the adaptor-equipped speech feature extracting unit 13a extracts an intermediate layer / frame level speaker vector sequence from the noise-added speech data.

[0049] Step S140 is performed using the intermediate layer / frame level speaker vector sequence.

[0050] In step S150a, the additional update unit 19 updates the parameters used by the speech synthesis model 14 (the text-to-speech conversion unit 142 and the fixed-dimensional vector conversion unit 141) and the parameters used by the speech feature extraction adapter 132 included in the adapter-equipped speech feature extraction unit 13a so that the (predicted) output speech sequence information approaches the correct output speech sequence information. However, the parameters of the self-supervised learning model 131 included in the adapter-equipped speech feature extraction unit 13a are not updated.

[0051] Next, the inference process will be described. Fig. 9 is a diagram showing an example of the functional configuration of the speech synthesis device 10 during inference according to an embodiment of the present invention. In Fig. 9, the same components as those in Fig. 4 or Fig. 6 are designated by the same reference numerals, and their description will be omitted. However, Fig. 9 shows the configuration after additional training. In Fig. 9, the speaker vector extraction unit 17 in Fig. 4 has been replaced with an adapter-attached speaker vector extraction unit 17a. The adapter-attached speaker vector extraction unit 17a includes an adapter-attached speaker vector extraction unit 13a instead of the speech feature extraction unit 13.

[0052] During inference, speech data (reference target speaker speech) in which speech by a target speaker for the speech to be synthesized is recorded and input text sequence information obtained from text data indicating the speech content of the speech to be synthesized are input to the speech synthesizer 10. Note that the reference target speaker speech may contain noise.

[0053] The adapter-equipped speech feature extraction unit 13a inputs reference target speaker speech and generates an intermediate-layer / frame-level speaker vector sequence using a trained speech feature extraction unit adapter 132. The trained fixed-dimensional vector conversion unit 141 inputs the fixed-dimensional vector conversion unit 141 and outputs a speaker vector. The trained text-to-speech conversion unit 142 inputs the speaker vector and input text sequence information and outputs predicted output speech sequence information. The speech synthesized based on the predicted output speech sequence information is a speech in which the content of the input text is spoken in a speech that has the characteristics of the reference target speaker speech.

[0054] As described above, according to this embodiment, it is possible to improve the reproducibility of speech synthesis when noise is included in the reference target speaker's speech.

[0055] In this embodiment, the speech synthesis device 10 during pre-training and additional training is an example of a speech synthesis training device. The speaker vector extraction unit 17 is an example of a first speaker vector extraction unit. The adaptor-equipped speaker vector extraction unit 17a is an example of a second speaker vector extraction unit.

[0056] [Reference 1] Baevski, Alexei, et al., "wav2vec 2.0: A framework for self-supervised learning of speech representations," Advances in Neural Information Processing Systems 33 (2020): 12449-12460 [Reference 2] Hsu, Wei-Ning, et al., "Hubert: Self-supervised speech representation learning by masked prediction of hidden units," IEEE / ACM Transactions on Audio, Speech, and Language Processing 29 (2021): 3451-3460 [Reference 3] Greff, Klaus, et al., "LSTM: A search space odyssey," IEEE transactions on neural networks and learning systems 28.10 (2016): 2222-2232 [Reference 4] Chen, Sanyuan, et al., "Why does Self-Supervised Learning for Speech Recognition Benefit Speakers?" Recognition?", arXiv preprint arXiv:2204.12765 (2022) [Reference 5] Houlsby, Neil, et al., "Parameter-efficient transfer learning for NLP," International Conference on Machine Learning. PMLR, 2019 [Reference 6] Snyder, David, Guoguo Chen, and Daniel Povey., "Musan: A music, speech, and noise corpus," arXiv preprint arXiv:1510.08484 (2015) Although the embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the gist of the present invention as set forth in the claims.

[0057] 10 Speech synthesis device 11 Text pre-processing unit 12 Speech pre-processing unit 13 Speech feature extraction unit 13a Speech feature extraction unit with adapter 14 Speech synthesis model 15 Speech synthesis unit 16 Pre-update unit 17 Speaker vector extraction unit 17a Speaker vector extraction unit with adapter 18 Addition unit 19 Additional update unit 100 Drive device 101 Recording medium 102 Auxiliary storage device 103 Memory device 104 Processor 105 Interface device 121 Pair data storage unit 122 Noise sound data storage unit 131 Self-supervised learning model 132 Adapter for speech feature extraction unit 141 Fixed dimension vector conversion unit 142 Text-to-speech conversion unit B Bus

Claims

1. A first speaker vector extraction procedure for generating a first speaker vector indicating characteristics of a speaker related to an utterance based on a self-supervised learning model for voice data in which an utterance has been recorded; a second speaker vector extraction procedure for generating a second speaker vector indicating characteristics of a speaker related to the utterance for noise-added voice data in which noise has been added to the voice data based on the self-supervised learning model and one or more conversion models that perform additional conversion processes in series or in parallel to multiple conversion processes in the self-supervised learning model; a text-to-speech conversion procedure for inputting vector information generated from text data indicating the content of the utterance and the first speaker vector or the second speaker vector, and outputting speech sequence information; a pre-update procedure for updating parameters used by the first speaker vector extraction procedure and the text-to-speech conversion procedure so that the speech sequence information output by the text-to-speech conversion procedure for the first speaker vector and the speech sequence information extracted from the voice data become closer to each other; and an additional update step of updating parameters used by the second speaker vector extraction step, the text-to-speech conversion step, and the conversion model so that the speech sequence information output by the text-to-speech conversion step, whose parameters have been updated by the preliminary update step for the second speaker vector, approaches speech sequence information extracted from the speech data.

2. The speech synthesis training method according to claim 1, wherein the additional transformation process is a process that combines a linear transformation and an activation function.

3. A speech synthesis method characterized in that a computer executes the following steps: a second speaker vector extraction step in which parameters are updated by the addition and update step of the speech synthesis training method described in claim 1 or 2; and a text-to-speech conversion step in which parameters are updated by the addition and update step of the speech synthesis training method described in claim 1 or 2.

4. A first speaker vector extraction unit configured to generate, for voice data in which an utterance is recorded, a first speaker vector indicating characteristics of a speaker related to the utterance based on a self-supervised learning model; a second speaker vector extraction unit configured to generate, for noise-added voice data in which noise is added to the voice data, a second speaker vector indicating characteristics of a speaker related to the utterance based on the self-supervised learning model and one or more conversion models that perform additional conversion processes in series or in parallel to multiple conversion processes in the self-supervised learning model; a text-to-speech conversion unit configured to input vector information generated from text data indicating the content of the utterance and the first speaker vector or the second speaker vector, and output speech sequence information; a pre-update unit configured to update parameters used by the first speaker vector extraction unit and the text-to-speech conversion unit so that the speech sequence information output by the text-to-speech conversion unit for the first speaker vector and the speech sequence information extracted from the voice data become closer to each other; and an additional update unit configured to update parameters used by the second speaker vector extraction unit, the text-to-speech conversion unit, and the conversion model so that the speech sequence information output by the text-to-speech conversion unit, whose parameters have been updated by the pre-update unit for the second speaker vector, approaches speech sequence information extracted from the speech data.

5. A speech synthesis device comprising: a second speaker vector extraction unit whose parameters have been updated by the additional update unit of the speech synthesis training device according to claim 4; and a text-to-speech conversion unit whose parameters have been updated by the additional update unit of the speech synthesis training device according to claim 4.

6. A program for causing a computer to execute the speech synthesis training method according to claim 1 or 2.

7. A program for causing a computer to execute the speech synthesis method according to claim 3.

Images