Voice conversion training method and voice conversion method

Abstract

The present invention makes streamable voice conversion possible by: calculating an error between a first content feature amount that a first encoder outputs when a first speech waveform is input into the first encoder, which has been trained so as to receive as input a speech waveform related to an utterance of a certain speaker and output a content feature amount that is the feature amount of the content of the utterance, and a second content feature amount that a second encoder outputs when a spectrum obtained from the first speech waveform is input into the second encoder, which receives as input a spectrum obtained from a speech waveform and outputs a content feature amount; calculating an error between the first speech waveform and a speech waveform that a speech waveform decoder outputs when the second content feature amount and a speaker feature amount related to said certain speaker are input into the speech waveform decoder, which receives as input a content feature amount and a speaker feature amount and outputs a speech waveform related to a speaker indicated by the speaker feature amount; and training the second encoder and the speech waveform decoder such that said errors are reduced.

Description

Voice conversion training method and voice conversion method

[0001] The present invention relates to a voice conversion training method and a voice conversion method.

[0002] Voice conversion (hereafter referred to as "VC") is a technology that converts the speaker characteristics of the input voice to that of a target speaker while maintaining the speech content of the input voice. Statistical model-based voice conversion methods can be broadly divided into two types based on the method of providing training data: (1) parallel VC, in which the speech content of the source and target speakers is the same due to the correspondence between their respective speech features; and (2) non-parallel VC, in which the speech content of the two speakers can be different. The former method requires high costs for collecting parallel data and cannot handle large amounts of data, making it difficult to train models with diverse source and target speakers (e.g., any-to-any VC).

[0003] On the other hand, the latter non-parallel VC can learn from non-parallel data and handle large amounts of data from multiple speakers, making it promising for building VC models that can be converted between diverse speakers. Most of these methods have a content encoder that extracts the speech content (hereinafter referred to as "content") and a speech predictive decoder. During training, both the input to the content encoder and the output of the speech predictive decoder are set to the source speaker's speech, and training is performed to reconstruct the speech using a VC model. The speech predictive decoder predicts the speech waveform from the content and speaker information. During inference, by changing the input to the speech predictive decoder to the target speaker's speaker information, speech can be converted to the target speaker's speaker characteristics while maintaining the source speaker's content. However, if the speaker characteristics cannot be sufficiently removed when obtaining content from the source speaker's speech, the quality of the converted speech will deteriorate.

[0004] Non-Patent Documents 1 and 2 use a speech recognizer as a content encoder to prevent speaker characteristics from being included in the content. The text output by the speech recognizer is less speaker-dependent, so the problem of speaker characteristics being mixed into the content can be avoided. However, the performance of VC is significantly dependent on speech recognition performance, and separate transcribed text is also required to train the speech recognizer.

[0005] To eliminate the need for transcribed text in VC training, Non-Patent Documents 3 and 4 construct VC using a generative adversarial network or a variational autoencoder, respectively. However, whether or not speaker characteristics are included in the content depends heavily on the model size of the content encoder, and tuning is required. Furthermore, when unknown speech content or source speaker voice outside the training data is input, these methods are unable to clearly separate content and speaker characteristics, and performance is likely to deteriorate.

[0006] To address these challenges, the self-supervised learning representation (SSL) model is promising for improving robustness against unknown data. The SSL model is a general-purpose model that can be applied to various tasks such as speech recognition, speech synthesis, speaker recognition, and speech enhancement. It does not require speech annotation and can be trained from extremely large amounts of speech data, not limited to clean speech. Furthermore, these trained models are publicly available. Non-Patent Documents 5 and 6 utilize the SSL model as a content encoder to enable VC without retraining using unknown speaker data (zero-shot VC).

[0007] Y. Saito, Y. Ijima, K. Nishida and S. Takamichi、"Non-Parallel Voice Conversion Using Variational Autoencoders Conditioned by Phonetic Posteriorgrams and D-Vectors"、Proc. ICASSP, pp. 5274-5278, 2018S. Liu, Y. Cao, D. Wang, X. Wu, X. Liu and H. Meng、"Any-to-Many Voice Conversion with Location-Relative Sequence-to-Sequence Modeling"、IEEE. Trans. on Audio, Speech, and Language Processing, vol. 29, pp. 1717-1728, 2021H. Kameoka, T. Kaneko, K. Tanaka and N. Hojo、"StarGAN-VC: non-parallel many-to-many Voice Conversion Using Star Generative Adversarial Networks"、Proc. SLT, pp. 266-273, 2018K. Qian, Y. Zhang, S. Chang, X. Yang, M. Hasegawa-Johnson、"AutoVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss"、Proc. ICML, pp. 5210--5219, 2019B. van Niekerk, M. -A. Carbonneau, J. Zaidi, M. Baas, H. Seute and H. Kamper、"A Comparison of Discrete and Soft Speech Units for Improved Voice Conversion"、Proc. ICASSP, pp. 6562-6566, 2022J. Li, W. Tu and L.Xiao, "Freevc: Towards High-Quality Text-Free One-Shot Voice Conversion", Proc. ICASSP, pp. 1-5, 2023S. Takaki, T. Nakashika, X. Wang and J. Yamagishi, "STFT Spectral Loss for Training a Neural Speech Waveform Model", Proc. ICASSP, pp. 7065-7069, 2019.R. Yamamoto, E. Song and J. -M. Kim, "Parallel Wavegan: A Fast Waveform Generation Model Based on Generative Adversarial Networks with Multi-Resolution Spectrogram", Proc. ICASSP pp. 6199-6203, 2020.

[0008] VCs using SSL models such as those described in Non-Patent Documents 5 and 6 are primarily designed for offline operation due to their large non-autoregressive models, making it difficult to perform streaming operations (converting voice quality in parallel with speech), which are important in practical VC usage scenarios. Furthermore, since the input to a content encoder is a speech waveform, the computational complexity is large due to the large sequence length.

[0009] The present invention has been made in view of the above points, and has as its object to enable voice conversion that is capable of streaming operation.

[0010] To solve the above problem, a computer executes the following steps: a content prediction error calculation procedure for calculating a content prediction error, which is the error between a first content feature that a first encoder, which is trained to input a speech waveform related to a certain speaker's speech and output content features that are features of the content of the speech, outputs when it receives a first speech waveform as input, and a second content feature that a second encoder, which inputs a spectrum obtained from the speech waveform and outputs content features, outputs when it receives a spectrum obtained from the first speech waveform as input; a speech waveform prediction error calculation procedure for calculating a speech waveform prediction error, which is the error between a speech waveform that a speech waveform decoder, which inputs content features and speaker features and outputs a speech waveform related to the speaker indicated by the speaker features, outputs when it receives the second content feature and speaker features related to the certain speaker, and the first speech waveform; and a learning procedure for training the second encoder and the speech waveform decoder so that the content prediction error and the speech waveform prediction error are small.

[0011] It is possible to enable streaming voice conversion.

[0012] FIG. 1 is a diagram illustrating an example of the hardware configuration of a voice conversion device 10 according to a first embodiment. FIG. 2 is a diagram illustrating an example of the functional configuration of the voice conversion device 10 according to the first embodiment in the learning phase of a voice conversion model. FIG. 3 is a diagram illustrating an example of the functional configuration of the voice conversion device 10 according to the first embodiment in the inference phase of a voice conversion model. FIG. 4 is a diagram illustrating an example of the functional configuration of the voice conversion device 10 according to a second embodiment in the learning phase of a voice conversion model. FIG. 5 is a diagram illustrating an example of the functional configuration of the voice conversion device 10 according to a third embodiment in the learning phase of a voice conversion model. FIG. 6 is a diagram illustrating an example of the functional configuration of the voice conversion device 10 according to the third embodiment in the inference phase of a voice conversion model.

[0013] This embodiment discloses a learning and inference method for voice conversion (hereinafter referred to as "VC") that performs streaming operations (outputting the speech of a target speaker in parallel with the speech of a source speaker) while taking advantage of the advantages of self-supervised learning representation (SSL).

[0014] In the first embodiment, an SSL model is used as the teacher model for the content encoder, and a lightweight, streaming-capable content encoder is prepared as the student model. Learning is performed within the knowledge distillation framework. To achieve streaming, a non-autoregressive model is not used in the modules. To reduce the amount of computation, the input is frame-level acoustic features such as spectra instead of speech waveforms, which have very long sequences. The loss function for learning not only calculates the speech waveform prediction error (as used in Non-Patent Documents 7 and 8) used in Non-Patent Documents 6 and the like, but also calculates content prediction error to ensure that the speech content (hereinafter referred to as "content") predicted by the student model matches that of the teacher model.

[0015] In the second embodiment, in addition to the first embodiment, speaker-invariant learning is introduced to explicitly remove speaker characteristics that may be included in the content, and is added to the loss function of the first embodiment.

[0016] In the third embodiment, a learning and inference technique is disclosed for predicting the target speaker's prosody from the content and using this as conditioning for the speech waveform decoder in order to improve the reproducibility of prosody, such as voice pitch and volume.

[0017] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0018] [First embodiment] Fig. 1 is a diagram showing an example of the hardware configuration of a voice conversion device 10 according to the first embodiment. The voice conversion device 10 in Fig. 1 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a processor 104, and an interface device 105, which are all interconnected via a bus B.

[0019] The program that realizes the processing in the voice conversion device 10 is provided by a recording medium 101 such as a CD-ROM. When the recording medium 101 storing the program is set in the drive device 100, the program is installed from the recording medium 101 to the auxiliary storage device 102 via the drive device 100. However, the program does not necessarily have to be installed from the recording medium 101, but may be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program as well as necessary files, data, etc.

[0020] When an instruction to start the program is received, the memory device 103 reads out and stores the program from the auxiliary storage device 102. The processor 104 is a CPU or a GPU (Graphics Processing Unit), or a CPU and a GPU, and executes functions related to the voice conversion device 10 in accordance with the program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network.

[0021] FIG. 2 is a diagram showing an example of the functional configuration of the voice conversion device 10 in the learning phase of the voice conversion model according to the first embodiment.

[0022] In the voice conversion model training phase, the voice conversion device 10 has a teacher voice conversion model 11 and a student voice conversion model 12. The teacher voice conversion model 11 includes a content encoder (offline) 111. The student voice conversion model 12 includes a content encoder (streaming) 121 and a voice waveform decoder 122. The voice conversion device 10 also has a content prediction error calculation unit 13, a voice waveform prediction error calculation unit 14, and a training unit 15 for training the student voice conversion model 12. Each of these models and units is realized by the processor 104 executing one or more programs installed in the voice conversion device 10.

[0023] The flow of the voice conversion model training phase (voice conversion training method) will be described with reference to FIG.

[0024] The content encoder (offline) 111 of the teacher voice conversion model 11 receives the voice waveform of the source speaker and outputs (generates) content features (hereinafter referred to as "teacher content features").

[0025] Here, the content features are vector sequences obtained from the output of the final layer or intermediate layer of the SSL model. As in Non-Patent Document 5, discrete sequences obtained by discretizing these vector sequences using vector quantization may be used instead. The content encoder (offline) 111 is a machine learning model (e.g., a neural network) trained using a known method, self-supervised learning (SSL), to input a speech waveform related to a speaker's speech and output content features, which are features of the content of the speech. The content encoder (offline) 111 is trained to obtain highly accurate content features by referring to the entire sequence, including not only past but also future utterances. However, because it is necessary to refer to future information (speech content), it is not suitable for streaming operations.

[0026] Next, the content encoder (streaming) 121 of the student voice conversion model 12 inputs a spectrum obtained from the "source speaker's voice waveform" input to the content encoder (offline) 111 and outputs (generates) predicted content features (hereinafter referred to as "predicted content features"). In other words, the content encoder (streaming) 121 is a machine learning model that inputs a spectrum and outputs content features. By changing the input to the content encoder (streaming) 121 from a voice waveform to a spectrum, it is possible to reduce the sequence length and thereby reduce the amount of calculation. While a spectrum is used here, a mel spectrogram, which is expressed in low dimensions to match human hearing, or a mel cepstrum, which is specialized for spectral envelopes and can be expressed in lower dimensions, may also be used instead of a spectrogram obtained by a short-time Fourier transform (STFT). This also applies to the inference phase described below. Note that the model structure of the content encoder (streaming) 121 is preferably a lightweight model structure that does not use future information to enable streaming operation. For example, a unidirectional LSTM, a causal CNN, or a causal transformer can be used as the content encoder (streaming) 121.

[0027] Next, the content prediction error calculation unit 13 inputs the teacher content features and the predicted content features, and calculates a content prediction error (error of the predicted content features from the teacher content features), which is the error between the teacher content features and the predicted content features. This is to enable the content encoder (streaming) 121 to output content features similar to those of the content encoder (streaming) 121. In other words, this is to enable the content encoder (streaming) 121 to imitate the content encoder (offline) 111. By enabling such imitation, the content encoder (streaming) 121 can output content features similar to those of the content encoder (offline) 111 while still enabling streaming operations. When the content features are a vector sequence of continuous values, squared error or absolute error can be used for the content prediction error, and when they are a discrete sequence, cross entropy can be used.

[0028] Next, the speech waveform decoder 122 inputs the predicted content features and the speaker features of the source speaker, and outputs (generates) a predicted speech waveform of the source speaker. In other words, the speech waveform decoder 122 is a machine learning model (for example, a neural network) that inputs content features and speaker features and outputs a speech waveform related to the speaker indicated by the speaker features. The speaker features are information that indicates who the speaker is (identifies the speaker). In addition to one-hot vectors, embedding vectors that take continuous values, such as i-vectors and x-vectors, can be used as speaker features.

[0029] Next, the speech waveform prediction error calculation unit 14 inputs the source speaker's speech waveform and the predicted speech waveform, and calculates the speech waveform prediction error, which is the error between the source speaker's speech waveform and the predicted speech waveform. Here, the STFT loss or adversarial loss proposed in Non-Patent Document 7 or Non-Patent Document 8 can be used for the speech waveform prediction error. Furthermore, if the speech waveform is a discrete value such as μ-law, cross-entropy can also be used as the speech waveform prediction error.

[0030] Finally, the learning unit 15 receives the speech waveform prediction error, the content prediction error, and the student voice conversion model 12, and updates the model parameters of the student voice conversion model 12 so as to reduce the speech waveform prediction error and the content prediction error (i.e., learns the student voice conversion model 12). At this time, the learning unit 15 may update the model parameters of the content encoder (streaming) 121 and the speech waveform decoder 122 based on the sum of the speech waveform prediction error and the content prediction error, or may update the model parameters of the content encoder (streaming) 121 based only on the content prediction error. Furthermore, the learning unit 15 may update the model parameters of the speech waveform decoder 122 based only on the speech waveform prediction error.

[0031] By repeatedly performing the above procedure across multiple sets of training data (a set of the source speaker's voice waveform, the source speaker's spectrum, and the source speaker's speaker features), it is possible to train a student voice conversion model 12 that is guaranteed to produce the same content as the teacher voice conversion model 11.

[0032] Next, the inference phase will be described. Fig. 3 is a diagram showing an example of the functional configuration of the voice conversion device 10 in the inference phase of the voice conversion model according to the first embodiment. In Fig. 3, the same components as those in Fig. 2 are designated by the same reference numerals.

[0033] 3, the voice conversion device 10 in the inference phase only needs to have a student voice conversion model 12. The student voice conversion model 12 is composed of the same modules as during training (a content encoder (streaming) 121 and a speech waveform decoder 122). Unlike during training, the input to the speech waveform decoder 122 is replaced with the speaker features of the target speaker, so that the speaker characteristics of the target speaker are imparted to the content features.

[0034] The flow of the inference phase of the voice conversion model (voice conversion method) will be described with reference to FIG.

[0035] First, the content encoder (streaming) 121 of the student voice conversion model 12 inputs the spectrum of the source speaker and outputs predicted content features.

[0036] Next, the speech waveform decoder 122 receives the predicted content features and the speaker features of the target speaker, and outputs (generates) a predicted speech waveform for the target speaker. Note that the target speaker is a speaker whose voice quality is to be converted, and is a speaker different from the source speaker.

[0037] The rest of the process is the same as in the prior art, so a description thereof will be omitted.

[0038] As described above, according to the first embodiment, it is possible to train a VC model that operates in streaming mode while maintaining performance similar to that of the SSL model, which is a large non-autoregressive model that has difficulty in streaming operation when used in VC. In other words, it is possible to achieve streaming voice conversion while taking advantage of the advantages of VC based on the SSL model (teacher voice conversion model 11), such as high robustness to unknown data and learning without text. Furthermore, because the SSL model is not used during inference, expensive computational resources such as a GPU are not required, and it is possible to expand the scope of application while maintaining performance comparable to that of the teacher model.

[0039] [Second embodiment] Next, a second embodiment will be described. In the second embodiment, differences from the first embodiment will be described. Points not specifically mentioned in the second embodiment may be the same as those in the first embodiment.

[0040] 4 is a diagram showing an example of the functional configuration of the voice conversion device 10 in the learning phase of the voice conversion model according to the second embodiment. In Fig. 4, parts that are the same as or correspond to those in Fig. 2 are given the same reference numerals, and their explanation will be omitted as appropriate.

[0041] The voice conversion device 10 in Fig. 4 further includes a speaker invariant learning unit 16 and a speaker prediction error calculation unit 17 in addition to the configuration in Fig. 2. These units are realized by the processor 104 executing one or more programs installed in the voice conversion device 10.

[0042] Referring to FIG. 4, the differences from the first embodiment will be described regarding the flow of the learning phase (voice quality conversion learning method) of the voice quality conversion model in the second embodiment.

[0043] The speaker-invariant learner 16 takes as input the predicted content feature amount output by the content encoder (streaming) 121 and outputs the speaker posterior probability. That is, the speaker-invariant learner 16 is a machine learning model (for example, a neural network) that takes the content feature amount as input and outputs the speaker posterior probability. Here, the speaker posterior probability is the probability indicating to which speaker the predicted content feature amount is classified (a probability distribution (categorical distribution of speakers) with the speaker category as a random variable). The speaker-invariant learner 16 can be composed of a deep learning module with a gradient reversal module, similar to Non-Patent Document 9. In the present embodiment, blocks whose names end with "learner" (the speaker-invariant learner 16 and the prosody predictor 123 in the third embodiment described later) are machine learning models having weights that can be learned within the framework of deep learning.

[0044] Next, the speaker prediction error calculation unit 17 takes as input the speaker posterior probability and the speaker ID of the source speaker, and calculates the error of speaker prediction (hereinafter referred to as "speaker prediction error"). The speaker ID is an ID indicating which speaker in the learning data the spectrum of the source speaker is, and the speaker prediction error is obtained by the cross-entropy between the speaker posterior probability and the categorical distribution of the speaker with a probability of 100% for the speaker corresponding to the speaker ID (that is, the speaker posterior probability (probability distribution) indicating the speaker related to the speaker ID).

[0045] Finally, the training unit 15 receives the speaker prediction error in addition to the speech waveform prediction error and the content prediction error, and updates the model parameters of the student voice conversion model 12 and the speaker invariant learner 16 so as to reduce these errors (i.e., trains the student voice conversion model 12 and the speaker invariant learner 16). In this case, the training unit 15 may update the model parameters of the content encoder (streaming) 121, the speech waveform decoder 122, and the speaker invariant learner 16 based on the sum of the speech waveform prediction error, the content prediction error, and the speaker prediction error, or may update the model parameters of the content encoder (streaming) 121 based only on the content prediction error. Furthermore, the training unit 15 may update the model parameters of the speech waveform decoder 122 based only on the speech waveform prediction error, and update the model parameters of the speaker invariant learner 16 based only on the speaker prediction error.

[0046] The gradient reversal module included in the speaker-invariant learner 16 facilitates the independence of predicted content features from a particular speaker, which is expected to result in improved VC performance.

[0047] The inference phase may be the same as in the first embodiment.

[0048] As described above, according to the second embodiment, the following effects can be obtained.

[0049] Since predicted content features are obtained from speech, they contain speaker characteristics to a greater or lesser extent. When speaker characteristics are present, this can cause degradation of VC performance, such as a decrease in speaker reproducibility and unclear speech. In the second embodiment, the framework of speaker-invariant training (Non-Patent Document 9) is introduced into the training of the student voice conversion model 12, which can explicitly promote the removal of speaker characteristics from content features and contribute to reducing degradation of VC performance.

[0050] [Third Embodiment] Next, a third embodiment will be described. In the third embodiment, differences from the first embodiment will be described. Points not specifically mentioned in the third embodiment may be the same as those in the first embodiment.

[0051] 5 is a diagram showing an example of the functional configuration of the voice conversion device 10 in the learning phase of the voice conversion model according to the third embodiment. In Fig. 5, the same or corresponding parts as those in Fig. 2 are denoted by the same reference numerals, and their explanation will be omitted as appropriate.

[0052] The student voice conversion model 12 in FIG. 5 further includes a prosodic predictor 123 in addition to the configuration in FIG. 2, and the voice conversion device 10 in FIG.

[0053] With reference to FIG. 5, the flow of the voice conversion model training phase (voice conversion training method) in the third embodiment will be described, focusing on the differences from the first embodiment.

[0054] The prosody predictor 123 receives predicted content features and speaker features of the source speaker and outputs predicted prosody features (hereinafter referred to as "predicted prosody features"). That is, the prosody predictor 123 is a machine learning model (e.g., a neural network) that receives content features and speaker features and outputs prosody features related to the speaker indicated by the speaker features. Examples of prosody features include the fundamental frequency (F0) representing the pitch of a voice, a voiced / unvoiced flag, and instantaneous energy for suppressing loudness or extreme F0 changes. Alternatively, a vector connecting these may be used as the prosody feature. The prosody predictor 123 also needs to be configured with only modules capable of streaming (e.g., a causal CNN, a unidirectional RNN, an LSTM, etc.).

[0055] Next, the speech waveform decoder 122 inputs predicted prosodic features in addition to the predicted content features and speaker features of the source speaker input in the first embodiment, and outputs (generates) a predicted speech waveform of the source speaker. Therefore, the speech waveform prediction error calculation unit 14 in the third embodiment inputs the speech waveform of the source speaker and the predicted speech waveform output from the speech waveform decoder 122, and calculates a speech waveform prediction error, which is the error between the speech waveform of the source speaker and the predicted speech waveform.

[0056] Furthermore, the prosodic feature error calculation unit 18 receives the predicted prosodic features and the source speaker's prosodic features and calculates the prosodic feature error, which is the error relative to the source speaker's prosodic features. The source speaker's prosodic features are prosodic features extracted from the source speaker's speech waveform, and are taken as the correct answer. Here, as the prosodic feature error, if the prosodic features are continuous values ​​such as the aforementioned F0, the squared error or absolute error can be used, and if the prosodic features are quantized within a specific maximum / minimum value range determined according to the prosodic features, the cross-entropy can be used.

[0057] Finally, the training unit 15 receives the prosodic feature error in addition to the speech waveform prediction error, content prediction error, and student voice conversion model 12 input in the first embodiment, and updates the model parameters of the student voice conversion model 12 and the prosodic predictor 123 so as to reduce these errors (i.e., trains the student voice conversion model 12 and the prosodic predictor 123). In this case, the training unit 15 may update the model parameters of the content encoder (streaming) 121, speech waveform decoder 122, and prosodic predictor 123 based on the sum of the speech waveform prediction error, content prediction error, and prosodic feature error, or may update the model parameters of the content encoder (streaming) 121 based only on the content prediction error. Alternatively, the training unit 15 may update the model parameters of the speech waveform decoder 122 based only on the speech waveform prediction error, and update the model parameters of the prosodic predictor 123 based only on the prosodic feature error.

[0058] Next, the inference phase will be described. Fig. 6 is a diagram showing an example of the functional configuration of the voice conversion device 10 in the inference phase of the voice conversion model according to the third embodiment. In Fig. 6, the same components as those in Fig. 5 are assigned the same reference numerals.

[0059] With reference to FIG. 6, the flow of the inference phase (voice conversion method) of the voice conversion model in the third embodiment will be described, focusing on the differences from the first embodiment.

[0060] The prosody predictor 123 receives the predicted content features output by the content encoder (streaming) 121 and the speaker features of the target speaker, and outputs predicted prosody features.

[0061] The speech waveform decoder 122 receives the predicted content features, the speaker features of the target speaker, and the predicted prosodic features, and outputs (generates) a predicted speech waveform of the target speaker.

[0062] As described above, according to the third embodiment, the following effects can be obtained.

[0063] In both the first and second embodiments, the speech waveform decoder 122 receives only content and speaker features, which allows it to reproduce the speaker characteristics of the target speaker, but makes it difficult to reproduce prosody, such as voice pitch and volume. The third embodiment introduces a prosody predictor 123 to explicitly predict prosody from content features. The prosody predictor 123 trains using a loss function so that the predicted prosody features approach the correct prosody features. These prosody features are also input as conditioning for the speech waveform decoder 122, enabling it to train under constraints that are optimal in terms of prediction error in the speech waveform.

[0064] The third embodiment may be combined with the second embodiment in order to remove speaker characteristics from content features.

[0065] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0066] 10 Voice conversion device 11 Teacher voice conversion model 12 Student voice conversion model 13 Content prediction error calculation unit 14 Speech waveform prediction error calculation unit 15 Learning unit 16 Speaker invariant learning device 17 Speaker prediction error calculation unit 18 Prosodic feature error calculation unit 100 Drive device 101 Recording medium 102 Auxiliary storage device 103 Memory device 104 Processor 105 Interface device 111 Content encoder (offline) 121 Content encoder (streaming) 122 Speech waveform decoder 123 Prosodic predictor B Bus

Claims

1. A voice conversion training method characterized by being executed by a computer comprising: a content prediction error calculation procedure for calculating a content prediction error, which is the difference between a first content feature that a first encoder trained to input a speech waveform related to a certain speaker's speech and output content features that are features of the content of the speech outputs when it receives a first speech waveform as input, and a second content feature that a second encoder trained to input a spectrum obtained from the speech waveform and output content features outputs when it receives a spectrum obtained from the first speech waveform as input; a speech waveform prediction error calculation procedure for calculating a speech waveform prediction error, which is the difference between a speech waveform that a speech waveform decoder trained to input content features and output a speech waveform related to the speaker indicated by the speaker features outputs when it receives the second content feature and the speaker feature related to the certain speaker outputs the first speech waveform; and a training procedure for training the second encoder and the speech waveform decoder so as to reduce the content prediction error and the speech waveform prediction error.

2. A voice conversion training method according to claim 1, characterized in that a computer executes a speaker prediction error calculation procedure to calculate a speaker prediction error, which is the difference between the posterior probability output by a speaker-invariant learning device that takes content features as input and outputs the posterior probability related to the speaker classification associated with the content features when the second content features are input, and the posterior probability indicating that the speaker associated with the first voice waveform is correct, and the training procedure further trains the speaker-invariant learning device so that the speaker prediction error is also reduced.

3. A voice conversion training method according to claim 1 or 2, characterized in that a computer executes a prosodic feature error calculation procedure for calculating a prosodic feature error, which is the error between a first prosodic feature output by a prosodic predictor that receives content features and speaker features as input and outputs prosodic features related to a speaker indicated by the speaker features when the second content features and speaker features related to the certain speaker are input, and a prosodic feature error, which is the error between the first prosodic feature and a prosodic feature obtained from the first speech waveform; the speech waveform prediction error calculation procedure calculates a speech waveform prediction error, which is the error between the first speech waveform and a speech waveform output by the speech waveform decoder when the second content features, speaker features related to the certain speaker, and the first prosodic feature are input; and the training procedure further trains the prosodic predictor so that the prosodic feature error is also reduced.

4. A voice quality conversion method characterized by a computer executing the steps of: a first encoder that has been trained to input a speech waveform related to a certain speaker's speech and output content features that are features of the content of the speech, and a speech waveform decoder that has been trained to minimize the error between: a first content feature that a first encoder outputs when it receives as input a first speech waveform; and a second content feature that a second encoder that receives as input a spectrum obtained from a speech waveform and outputs content features when it receives as input a spectrum obtained from the first speech waveform; and an error between a speech waveform that a speech waveform decoder that receives as input content features and speaker features and outputs a speech waveform related to the speaker indicated by the speaker features outputs when it receives as input the second content feature and speaker features related to the certain speaker;

Images