Voice conversion learning device, voice conversion device, voice conversion method, and program

Abstract

One aspect of the present invention is a voice conversion learning device comprising: a filter that calculates a speaker vector using a speaker vector of an input speaker, a speaker vector of a target speaker, a first expansion / contraction coefficient which is information indicating a change in speech speed, and a second expansion / contraction coefficient which is information indicating a change in speech speed; an encoder that generates an encoded input feature quantity sequence using the speaker vector and a first input feature quantity sequence generated on the basis of the first expansion / contraction coefficient with respect to a voice signal for the input speaker; and a decoder that uses the encoded input feature quantity sequence to predict an output log-mel spectrogram and output the output log-mel spectrogram as predicted values.

Description

Voice conversion training device, voice conversion device, voice conversion method, and program

[0001] The present invention relates to a voice conversion training device, a voice conversion device, a voice conversion method, and a program technology.

[0002] Speech conversion is a technology that converts only the non-linguistic and paralinguistic information (speaker identity, speaking style, etc.) of input speech while preserving the linguistic information (sentence) of the input speech, and can be applied to speaker identity conversion, speech assistance, speech enhancement, accent conversion, etc. The problem of speech conversion can be formulated as a regression analysis problem that estimates a mapping function from the acoustic features of the source speech to the acoustic features of the target speech.

[0003] In recent years, sequence-to-sequence (S2S) learning approaches have demonstrated high effectiveness in tasks such as machine translation, speech recognition, and text-to-speech synthesis, and their application to speech conversion has also begun to be gradually addressed (see, for example, Non-Patent Document 1). Furthermore, in a speech conversion method using phoneme posterior probabilities (see, for example, Non-Patent Document 2), the average duration of each phoneme is calculated in advance for each speaker to change the speaking rate when converting speech from an input speaker to a target speaker. During testing, the phoneme durations of the input speech are manually corrected based on the pre-calculated durations. Furthermore, a speech synthesis method that uses representation learning within a semi-supervised learning framework has been proposed (see, for example, Non-Patent Document 3). Furthermore, a speech conversion method that acquires separable representations within a self-supervised learning framework has been proposed (see, for example, Non-Patent Document 4) to completely avoid annotation tasks such as tagging the training data.

[0004] Jing-Xuan Zhang, Zhen-Hua Ling, wt al., “Sequenceto-sequence acoustic modeling for voice conversion”, IEEE / ACM Transactions on Audio,Speech, and Language Processing, vol.27, no.3, pp.631-644, 2019Li-Juan Liu, Yan-Nian Chen, et al., “Non-parallel voice conversion with autoregressive conversion model and duration adjustment”, in Joint Workshop for the Blizzard Challenge and Voice Conversion Challenge, pp.126-130, 2020Raza Habib, Soroosh Mariooryad, et al., “Semi-supervised generative modeling for controllable speech synthesis,” in ICLR, 2020Chak Ho Chan, Kaizhi Qian,et al.,“SpeechSplit2.0: Unsupervised speech disentanglement for voice conversion without tuning autoencoder bottlenecks”, in ICASSP, pp.6332-6336, 2022

[0005] In speech conversion, a technology for converting speech into a speech pattern, in order to control only the speaking rate while preserving the voice quality, it is essential to decompose the speaker vector into a component related to the speaking rate and other components. However, in conventional technologies, obtaining such decomposed speaker representations requires either prior statistical calculation or annotation, or speech processing or reference speech during testing. In view of the above circumstances, the present invention aims to provide a technology that makes it possible to obtain speaker representations in which the speaker vector is decomposed into a component related to the speaking rate and other components, without requiring prior statistical calculation or annotation, or speech processing or reference speech during testing.

[0006] One aspect of the present invention is a voice conversion training device comprising: a filter that calculates a speaker vector using a speaker vector of an input speaker, a speaker vector of a target speaker, a first scaling factor that is information representing a speech rate change, and a second scaling factor that is information representing the speech rate change; a first input feature sequence that is generated for a speech signal of the input speaker based on the first scaling factor; an encoder that generates an encoded first input feature sequence using the speaker vector; and a decoder that predicts an output logarithmic mel spectrogram using the encoded input feature sequence and outputs the predicted value.

[0007] One aspect of the present invention is a voice conversion device comprising: the voice conversion training device described above; an acquisition unit that acquires the voice signal of the input speaker and the voice signal of the target speaker; a conversion unit that uses the trained encoder and decoder to convert at least one of the voice quality and speaking rate of the voice signal of the input speaker to generate the predicted value; and a vocoder that converts the predicted value into a voice signal.

[0008] One aspect of the present invention is a speech conversion method in which a filter generates an input speaker vector and a target speaker vector using a speaker vector of the input speaker, a speaker vector of the target speaker, a first scaling factor that is information representing a speech rate change, and a second scaling factor that is information representing the speech rate change, outputs the generated input speaker vector to the encoder, and outputs the generated target speaker vector to a decoder; the encoder generates an encoded input feature sequence using the input speaker vector and an input feature sequence that is generated for a speech signal of the input speaker based on the first scaling factor, and outputs the generated encoded input feature sequence to the decoder; the decoder generates a prediction from the encoded input feature sequence using the target speaker vector and the encoded input feature sequence, and outputs the generated prediction to the vocoder; and the trained vocoder converts the prediction into a speech signal.

[0009] One aspect of the present invention is a program for causing a computer to function as the voice conversion training device described above.

[0010] This invention makes it possible to obtain speaker representations in which speaker vectors are decomposed into components related to speaking rate and other components, without the need for prior statistical calculation or annotation, or for speech processing or reference speech during testing.

[0011] FIG. 1 is a diagram showing an example of the configuration of a speech conversion device according to an embodiment; FIG. 2 is a diagram showing an example of the function of a filter according to an embodiment; FIG. 3 is a diagram showing an example of processing during training according to an embodiment; FIG. 4 is a diagram showing an example of estimation processing in constant ratio conversion according to an embodiment; FIG. 5 is a diagram showing an example of estimation processing in vector exchange conversion according to an embodiment; FIG. 6 is a diagram showing example evaluation results of objective evaluations (DF and DRCC) regarding speech recognition error rate (CER) and speaking rate; and FIG. 7 is a diagram showing example subjective evaluation results of speaker similarity regarding sound quality and voice quality.

[0012] An embodiment of the present invention will be described in detail with reference to the drawings. Fig. 1 is a diagram showing an example configuration of a voice conversion device according to this embodiment. The voice conversion device 1 includes, for example, a voice conversion training device 2, an acquisition unit 3, a conversion unit 4, a vocoder 5, and a storage unit 6. The voice conversion training device 2 includes, for example, a training unit 21, an acquisition unit 22, a first expansion / contraction unit 23, a second expansion / contraction unit 24, a feature extraction unit 25, a filter 26, and a voice conversion model 27. The voice conversion model 27 includes, for example, an encoder 271 and a decoder 272.

[0013] In the speech conversion task, the speech conversion device 1 acquires and controls a separate expression related to speaking rate using a speaker index.

[0014] The voice conversion training device 2 trains the voice conversion model 27 .

[0015] The acquisition unit 3 acquires, during conversion and learning, the speech signal of the input speaker, the speech signal of the target speaker, and information indicating a speech rate change to be converted (a second scaling factor). The scaling factor is, for example, a speech rate magnification (relative value).

[0016] The conversion unit 4 converts the voice quality and speech rate of the speech signal of the input speaker using a trained speech conversion model, etc. As will be described later, the conversion unit 4 can convert the speech signal into a voice having, for example, the speech rate of the input speaker and the tone of voice of the target speaker by manipulating the speaker vector.

[0017] The vocoder 5 converts the output logarithmic mel spectrogram X^, which has been converted using the speech conversion model 27, into a speech signal. The vocoder 5 may be any neural vocoder conditioned with a Mel spectrogram as input. In this embodiment, the vocoder 5 uses HiFi-GAN (see Reference 1). In this embodiment, the V2 setting described in Reference 1 is applied, except for the 8 ms frame shift condition.

[0018] Reference 1: Jungil Kong, Jaehyeon Kim, et al., “HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis”, pp. 17022-17033, 2020

[0019] The storage unit 6 stores, for example, programs used for various processes of the speech conversion device 1, thresholds, setting values, programs, formulas, parameters, etc. used for processes during learning.

[0020] During training, the training unit 21 trains the voice conversion model 27. The training method will be described later. Training the voice conversion model 27 requires only voice data, and does not require annotation of the speaking rate.

[0021] During training, the acquisition unit 22 acquires the speech signal of the input speaker, the speech signal of the target speaker, a first scaling factor (speech rate change) representing the scaling factor of the speech signal of the input speaker, and a second scaling factor (speech rate change) representing the scaling factor of the speech signal of the target speaker. The acquisition unit 22 may also serve as the acquisition unit 3.

[0022] During training, the first scaling unit 23 receives a first scaling coefficient and the speech signal of the input speaker, and scales the speech signal of the input speaker based on the value of the first scaling coefficient to generate a first input feature sequence. Note that the first scaling unit 23 performs the scaling process before feature extraction. Note that the scaling method will be described later.

[0023] During training, the second scaling unit 24 receives the second scaling coefficient and the target speaker's speech signal, and scales the target speaker's speech signal based on the value of the second scaling coefficient to generate a first input feature sequence. Note that the second scaling unit 24 performs the scaling process before feature extraction. Note that the scaling method will be described later.

[0024] The feature extractor 25 extracts acoustic features from the audio signal. The acoustic features are, for example, logarithmic mel spectrograms.

[0025] The filter 26 is a conditional filtering mechanism. The filter 26 receives as input a speaker vector of an input speaker (e.g., a one-hot vector corresponding to an input speaker index), a speaker vector of a target speaker (e.g., a one-hot vector corresponding to a target speaker index), a first scaling factor, and a second scaling factor. The filter 26 calculates the speaker vector. An example configuration and processing of the filter 26 will be described later. A one-hot vector is a discrete vector whose value is either 0 or 1.

[0026] The encoder 271 is configured with, for example, three one-dimensional non-causal convolution layers including 512 filters with a kernel size of 5, a batch normalization layer followed by a rectified linear unit (ReLU) activation. The encoder 271 receives an input logarithmic mel-spectrogram X as a feature. ｓｒｃ and the speaker speech vector c ｓｒｃ The output of the final convolutional layer is passed to a single bidirectional long short-term memory (LSTM) layer containing 512 units, as described below, to generate the encoded input feature sequence Z. However, the unit size of the LSTM is changed from 1024 to 512, and the target speaker vector c ｔｇｔ The input and output signals will be explained later.

[0027] The decoder 272 includes an attention function and a post-net. The decoder 272 predicts the attention matrix A and the output logarithmic mel-spectrogram X̂. The processing will be described later.

[0028] The voice conversion device 1 and the voice conversion training device 2 are configured using a processor such as a CPU (Central Processing Unit) and memory. The voice conversion device 1 functions, for example, as an acquisition unit 3, a conversion unit 4, and a vocoder 5 when the processor executes a program. The voice conversion training device 2 functions, for example, as a training unit 21, an acquisition unit 22, a first expansion / contraction unit 23, a second expansion / contraction unit 24, a feature extraction unit 25, a filter 26, and a voice conversion model 27 when the processor executes a program. Note that all or part of the functions of the voice conversion device 1 and the voice conversion training device 2 may be realized using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The above programs may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, and semiconductor storage devices (e.g., solid-state drives (SSDs)), as well as storage devices such as hard disks and semiconductor storage devices built into computer systems. The above program may be transmitted via a telecommunications line.

[0029] (Functions and Processing of the Filter) Next, the functions and processing of the filter 26 will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of the functions of the filter of this embodiment. The filter 26 includes, for example, Tanh 261, Linear 262, Linear 263, Linear 264, a multiplication unit 265 (calculation unit), an addition unit 266 (calculation unit), and Concat 267.

[0030] Tanh 261 is a hyperbolic tangent function. Linear 262 to 264 are linear transformation layers. Concat 267 is a processing unit that performs vector concatenation.

[0031] The input to Tanh 261 is the speaker vector o ｓｒｃ and the target speaker's speaker vector o ｔｇｔand the first expansion coefficient α ｓｒｃ (scale factor) and the second expansion / contraction coefficient α ｔｇｔ (scale factor). Note that the subscript "src" is an abbreviation for "source," which is the input, and "tgt" is an abbreviation for "target," which is the transformation target. Tanh261 is 10 x 1024 (dimensions).

[0032] Linear 262 is 1024 x 3 (dimensions). Linear 263 is 1024 x 3 (dimensions). Linear 264 is 1024 x 253 (dimensions).

[0033] The speaker vector o of the input speaker ｓｒｃ and the target speaker's speaker vector o ｔｇｔ is passed through a DNN (Deep Neural Network) to become three components I to III. I. Three-dimensional speech rate component c that changes with data expansion and contraction ｖ，ｓｒｃ ・c ｖ，ｔｇｔ II. Three-dimensional speech rate component c that does not change due to data expansion / contraction ｃ，ｓｒｃ ・c ｃ，ｔｇｔ III. The remaining 253rd order components c that are not related to speaking rate ｒ，ｓｒｃ ・c ｒ，ｔｇｔ In addition, Linear 262 is c ｖ，ｓｒｃ ・c ｖ，ｔｇｔ and Linear263 outputs c ｃ，ｓｒｃ ・c ｃ，ｔｇｔ and Linear264 outputs c ｒ，ｓｒｃ ・c ｒ，ｔｇｔ That is, the three-dimensional variable component c that changes due to data expansion and contraction is output. ｖ，ｓｒｃ and C ｖ，ｔｇｔ , a three-dimensional constant component c that does not change due to data expansion and contraction ｃ，ｓｒｃ and C ｃ，ｔｇｔ , 253-dimensional residual component c ｒ，ｓｒｃ and cc ｒ，ｔｇｔ is obtained (step S11).

[0034] The multiplication unit 265 and the addition unit 266 calculate c as a three-dimensional vector representing the speaking rate of the speaker. ｓ，ｓｒｃ and C ｓ，ｔｇｔ are generated as shown in the following equations (1) and (2) (step S12).

[0035]

[0036]

[0037] Finally, Concat267 computes the speaking rate vector c ｓ，ｓｒｃ and C ｓ，ｔｇｔ , and the remaining 253-order components that are not related to speaking rate, ｒ，ｓｒｃ and C ｒ，ｔｇｔ By combining these, a 256-dimensional speaker (speaking) vector c is obtained as input to speech conversion or synthesis. ｓｒｃ and C ｔｇｔ is output (step S13).

[0038] In this embodiment, the second expansion / contraction coefficient α ｔｇｔ By changing the rate of speech of the converted voice, the speech rate of the converted voice is controlled. This conversion is called "constant rate conversion" in this embodiment. ｒ，ｓｒｃ and C ｒ，ｔｇｔ By using a vector combining these, conversion to a speech signal with the voice quality of the target speaker and the speaking rate of the input speaker is realized. In this embodiment, this conversion is called "vector exchange conversion."

[0039] (Data Warping) Next, an example of a data warping method used in this embodiment will be described. In this embodiment, the WSOLA method (Reference 2) is used as the data warping method. Note that the WSOLA method requires a warping coefficient (scale factor) but does not require pre-calculation of the speaking rate.

[0040] Reference 2: Werner Verhelst, Marc Roelands, “An overlap-add technique based on waveform similarity (WSOLA) for high quality timescale modification of speech”, in ICASSP, vol. 2, pp. 554-557, 1993

[0041] The speech conversion device 1 converts a time domain speech waveform Y ｓｒｃ and Y ｔｇｔis processed before extracting the logarithmic mel-spectrogram. The requirements for the short-time Fourier transform are, for example, a Hanning window, a 64 ms frame length, an 8 ms frame shift, and a 1024-point fast Fourier transform. Since shorter sequences make it easier to train each model, in this embodiment, the length of the sequence is shortened by creating a subframe containing two time frames.

[0042] The voice conversion device 1 uses a first expansion / contraction coefficient α ｓｒｃ and the second expansion coefficient α ｔｇｔ Using this, the audio waveform Y ｓｒｃ and Y ｔｇｔ the original length α ｓｒｃ and α ｔｇｔ The logarithmic mel spectrogram X used in the voice conversion model 27 is expanded or contracted by a factor of 1. ｓｒｃ and X ｔｇｔ is a feature that has already been scaled. In the evaluation described later, the first scale factor α ｓｒｃ and the second expansion coefficient α ｔｇｔ was sampled from a uniform distribution in the range of [0.8, 1.25], and data was stretched with a 50% probability during learning. ｓｒｃ The second stretchable portion 24 stretches by α ｔｇｔ In this embodiment, the range of the uniform distribution is determined, for example, from the fastest and slowest speakers in the training data.

[0043] (Voice Conversion) The feature sequence of the target voice, which is the input of the voice conversion model 27, is expressed as X ｓｒｃ = [x １，ｓｒｃ , ..., x ｉ，ｓｒｃ ] and X ｔｇｔ = [x １，ｔｇｔ , ..., x ｊ，ｔｇｔ The feature extraction unit 25 extracts an 80-dimensional logarithmic mel spectrogram X from the sound source and audio signal sampled at, for example, 16 kHz in the range of 80 to 7600 Hz. ｓｒｃ = [x １，ｓｒｃ , ..., x ｉ，ｓｒｃ ] and X ｔｇｔ = [x １，ｔｇｔ , ..., x ｊ，ｔｇｔ ] is extracted.

[0044] First input feature sequence X ｓｒｃ and the input speaker vector c ｓｒｃ is passed to the encoder 271. Then, the encoder 271 generates an encoded input feature sequence Z of the following equation (3). Note that in equation (3), f Ｅｎｃ (•) indicates encoder 271 processing.

[0045]

[0046] The decoder 272 receives the past target speaker feature sequence X ～ (the following equation (4)) and the target speaker vector c ｔｇｔ , the output X^ (the following equation (5)) is predicted from the encoded input feature sequence Z as shown in the following equation (6). Ｄｅｃ (•) indicates the processing of the encoder 271. Note that j is an integer equal to or greater than 2. Note that in equation (4), x0 indicates a zero vector called a start token.

[0047]

[0048]

[0049]

[0050] In equation (6), A is the feature sequence X of the input speech. ｓｒｃ and the feature sequence X of the target speech with different sequence lengths. ｔｇｔ Finally, the objective function L when training the entire model including conditional filtering is defined as follows:

[0051]

[0052] In equation (7), ∥·∥ is a norm and represents a distance in a metric space. The subscript number of ∥·∥ represents the value of the norm, and "2" is the 2-norm (Euclidean norm).

[0053] (Learning Process) Next, the learning process will be described. Fig. 3 is a diagram showing an example of the learning process of this embodiment. The voice conversion device 1 employs an S2S (Sequence To Sequence) (https: / / google.github.io / seq2seq / ) model architecture, which uses parameters of a source voice as input and generates parameters of a target voice as output. What is required for learning is a parallel corpus of paired input and output voices.

[0054] The input of the filter 26 is the speaker vector o ｓｒｃ and the target speaker's speaker vector o ｔｇｔ and the first expansion coefficient α ｓｒｃ and the second expansion coefficient α ｔｇｔ The filter 26 processes the input speaker vector c ｓｒｃ is output to the encoder 271, and the target speaker vector c ｔｇｔ is output to the decoder 272 (step S21).

[0055] The first expansion / contraction unit 23 receives the speech signal of the input speaker and the first expansion / contraction coefficient α ｓｒｃ The first expansion / contraction unit 23 expands the first input feature sequence X ｓｒｃ is output to the encoder 271 (step S22).

[0056] The encoder 271 receives the input speaker vector c output by the filter 26. ｓｒｃ and the first input feature sequence X output by the first expansion / contraction unit 23. ｓｒｃ The encoder 271 generates an encoded input feature sequence Z and outputs the generated encoded input feature sequence Z to the decoder 272 (step S23).

[0057] The decoder 272 receives the target speaker vector c output by the filter 26. ｔｇｔ The decoder 272 receives the coded input feature sequence Z output by the encoder 271. The decoder 272 predicts the output X^ from the coded input feature sequence Z (step S24).

[0058] The second expansion / contraction section 24 includes the speech signal of the target speaker and a second expansion / contraction coefficient α ｔｇｔThe second expansion / contraction unit 24 expands the second input feature sequence X ｔｇｔ is output (step S25).

[0059] The learning unit 21 compares the predicted value X^ output by the decoder 272 with the second input feature sequence X output by the second expansion / contraction unit 24. ｔｇｔ , the objective function L of the equation (7) is calculated, and the encoder 271 and the decoder 272 are trained so as to minimize the objective function (step S26).

[0060] (Estimation Process (Constant Ratio Conversion)) Next, the estimation process in constant ratio conversion will be described. Fig. 4 is a diagram showing an example of the estimation process in constant ratio conversion of this embodiment. Note that the example in Fig. 4 is an example in which the speech rate is controlled to be a constant multiple.

[0061] The input of the filter 26 is the speaker vector o ｓｒｃ and the target speaker's speaker vector o ｔｇｔ and the first expansion coefficient α ｓｒｃ (=1) and the second expansion coefficient α ｔｇｔ The filter 26 processes the input speaker vector c ｓｒｃ is output to the encoder 271, and the target speaker vector c ｔｇｔ is output to the decoder 272 (step S31).

[0062] The encoder 271 receives the input speaker vector c output by the filter 26. ｓｒｃ and the first input feature sequence X of the speech signal of the input speaker. ｓｒｃ The encoder 271 generates an encoded input feature sequence Z and outputs the generated encoded input feature sequence Z to the decoder 272 (step S32).

[0063] The decoder 272 receives the target speaker vector c output by the filter 26. ｔｇｔ The decoder 272 receives the coded input feature sequence Z output by the encoder 271. The decoder 272 predicts the output X^ from the coded input feature sequence Z and outputs the output X^ to the trained vocoder 5 (step S33).

[0064] The trained vocoder 5 converts the input output X^ into a voice signal (step S34).

[0065] (Estimation (Vector Exchange Transformation)) Next, the estimation process in vector exchange transformation will be described. FIG. 5 is a diagram showing an example of the estimation process in vector exchange transformation of this embodiment. Note that the example in FIG. 5 is a processing example in which converted speech having the voice quality of the target speaker and the speaking rate of the input speaker is realized by exchanging part of the speaker vectors.

[0066] The input of the filter 26 is the speaker vector o ｓｒ and the target speaker's speaker vector o ｔｇｔ and the first expansion coefficient α ｓｒｃ (=1) and the second expansion coefficient α ｔｇｔ The filter 26 processes the input speaker vector c ｓｒｃ to the encoder 271. The filter 26 also outputs the target speaker vector c ｔｇｔ A part of the input speaker vector c ｓｒｃ and output to the decoder 272.

[0067] The encoder 271 receives the input speaker vector c output by the filter 26. ｓｒｃ and the first input feature sequence X of the speech signal of the input speaker. ｓｒｃ The encoder 271 generates an encoded input feature sequence Z and outputs the generated encoded input feature sequence Z to the decoder 272.

[0068] The decoder 272 receives the target speaker vector c output by the filter 26. ｔｇｔ A part of the input speaker vector c ｓｒｃ and the encoded input feature sequence Z output by the encoder 271. The decoder 272 predicts the output X^ from the encoded input feature sequence Z and outputs the output X^ to the trained vocoder 5.

[0069] The vocoder 5 converts the input output X^ into a voice signal.

[0070] (Evaluation) Next, an example of the results of evaluating the speech conversion device 1 described above will be described. In the evaluation, 450 sentences out of 503 sentences in the Japanese speech dataset ATR phoneme-balanced sentences (see Reference 3) were used as training data and 53 sentences as evaluation data, and objective evaluations were performed on the speech recognition error rate and speaking rate, as well as listening evaluations on the speaker similarity of speech quality and voice quality. The speech sampling rate was 16 kHz. Note that the speech used had silent periods removed in advance.

[0071] The systems evaluated are as follows: Comparative example (Baseline): Converted speech using a speech conversion model equivalent to the method of this embodiment, without using the data expansion and conditional filtering mechanisms. Proposed-swap: In vector swap conversion using the method of this embodiment, the speech rate vector c of the input speaker is ｓ，ｓｒｃ and a vector other than the target speaker's speaking rate c ｒ，ｔｇｔ ・Proposed-0.8 to 1.2: In the fixed ratio conversion by the proposed method, the expansion coefficient α ｔｇｔ Converted audio when changing from 0.8 to 1.2.

[0072] Reference 3: Fadi Biadsy, Ron JWeiss, et al., “Parrotron: An end-to-end speech-to-speech conversion model and its applications to hearing-impaired speech and speech separation”, in INTERSPEECH, pp. 4115-4119, 2019

[0073] In the evaluation, the CER (Character Error Rate) was calculated and used as the speech recognition error rate. The lower the CER error rate, the better the quality. In addition, the DF (Duration Factor) and DRCC (Duration Ratio Correlation Coefficient) were calculated and used as indices related to the speech speed. In the fixed ratio conversion, the length of the converted speech is increased by a given expansion coefficient α ｔｇｔ For example, α ｔｇｔThe length of the converted audio is α ｔｇｔ = 1.0, and must be 0.8 times the length of the converted audio. Therefore, in this embodiment, DF is defined as in the following equation (8).

[0074]

[0075] In equation 98), N is the number of audio clips evaluated. ｔｇｔ In the vector conversion, first, two speech rate ratios DR1 and DR2 are calculated as shown in the following equations (9) and (10).

[0076]

[0077]

[0078] If the model learning is successful as intended, DR2 should have the same value as the reference DR1. In the evaluation, the DRCC, which is the correlation coefficient between DR1 and DR2, was calculated as the DRCC of the comparative example (Baseline) and Proposed-1.0. The closer the DRCC value is to 1, the higher the performance. In addition, for Proposed-swap, the DRCC was calculated and evaluated using DR2 in the following formula (11):

[0079]

[0080] If the method of this embodiment works as intended, the speech rate vector of the input speaker ｃｓ，ｓｒｃ The converted voice X^ ｖｓｃ is X ｓｒｃ The speech quality was evaluated using the MOS (Mean Opinion Score) Test. The MOS evaluation is a five-point scale, with 5 being the best and 1 being the worst. In other words, the higher the value, the better the speech quality.

[0081] The speaker similarity of voice quality was evaluated using a preference test (PT). The PT evaluation was on a four-point scale: 4: same (sure), 3: same (not sure), 2: different (not sure), and 1: different (sure). In other words, the higher the proportion of "same" values, the higher the similarity and the better the performance. Ten native Japanese speakers participated in each subjective evaluation. Each system was evaluated more than 275 times.

[0082] FIG. 6 shows example evaluation results of objective evaluations (DF and DRCC) regarding speech recognition error rate (CER) and speaking rate. FIG. 7 shows example subjective evaluation results of speaker similarity regarding sound quality and voice quality. In FIG. 7, the graph indicated by symbol g21 shows example subjective evaluation results of sound quality, and the graph indicated by symbol g22 shows example subjective evaluation results of voice and timbre similarity. Note that in symbol g2, "Re-synthesized" means analyzed and resynthesized speech. In addition, in the graph indicated by symbol g21, the horizontal axis shows items, and the vertical axis shows MOS evaluation scores. In symbol g22, the horizontal axis shows items, and the vertical axis shows subjective evaluation results (%).

[0083] As shown in Figures 6 and 7, a comparison of the evaluation example (Baseline) and Proposed-1.0 shows that there is no difference between the objective and subjective evaluation results, indicating that there is no degradation due to the introduction of data expansion / contraction and conditional filtering. Furthermore, even with the DRCC in which the denominator of Proposed-swap is changed, the correlation coefficient is 0.9 or higher, indicating that the speaking rate of Proposed-swap is that of the input speaker. Next, the speaker similarity of the voice quality of Proposed-swap is comparable to that of the evaluation example (Baseline), indicating that the voice quality is that of the target speaker.

[0084] The evaluation results showed that the proposed-swap method of this embodiment has the speaking rate of the input speaker and the voice quality of the target speaker. As described above, the method of this embodiment succeeded in decomposing the speaker vector into components related to speaking rate and other components, making it possible to control it. Note that the DFs of Proposed-0.8 and 0.9, 1.1, and 1.2 are the expansion coefficient αｔｇｔ Since this is equivalent to

[0085] The voice conversion learning device 2 configured in this way controls high-level attributes such as speaking rate while maintaining the speaker's voice quality without using prior statistical calculations, annotations, voice processing during testing, or reference voice, and therefore filters the voice or speaker vector to achieve the acquisition of separate representations of the speaker and voice.

[0086] Furthermore, the voice conversion training device 2 can acquire separate expressions relating to voice and speaker through vector exchange conversion processing.

[0087] Furthermore, by combining the conditional filtering method and the data augmentation method, the voice conversion training device 2 can eliminate the need for annotations regarding expressions to be acquired.

[0088] Furthermore, in the voice conversion training device 2, by combining the conditional filtering method and data reinforcement, it is possible to acquire expressions separated not only as one-dimensional scalars but also as multidimensional vectors.

[0089] (Modification) In the above example, the speech rate of the input voice is increased. However, the speech conversion device 1 may also decrease the speech rate of the input voice.

[0090] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention.

[0091] The present invention is applicable to devices and processes that perform, for example, gender conversion of a speaker's voice, speech support, voice enhancement, accent conversion, and the like.

[0092] 1... Voice conversion device, 2... Voice conversion training device, 3... Acquisition unit, 4... Conversion unit, 5... Vocoder, 6... Storage unit, 21... Learning unit, 22... Acquisition unit, 23... First expansion / contraction unit, 24... Second expansion / contraction unit, 25... Feature extraction unit, 26... Filter, 27... Voice conversion model, 271... Encoder, 272... Decoder

Claims

1. A voice conversion training device comprising: a filter that calculates a speaker vector using a speaker vector of an input speaker, a speaker vector of a target speaker, a first scaling factor that is information representing a speech rate change, and a second scaling factor that is information representing the speech rate change; an encoder that generates a first input feature sequence based on the first scaling factor for the speech signal of the input speaker and an encoded input feature sequence using the speaker vector; and a decoder that predicts an output log mel spectrogram using the encoded input feature sequence and outputs it as a predicted value.

2. A first scaling unit that uses the first scaling coefficient and the speech signal of the input speaker to scale the speech signal of the input speaker based on the value of the first scaling coefficient to generate the first input feature sequence; a second scaling unit that uses the second scaling coefficient and the speech signal of the target speaker to scale the speech signal of the target speaker based on the value of the second scaling coefficient to generate a second input feature sequence; and a learning unit that calculates an objective function for the predicted value and the second input feature sequence, and trains the encoder and the decoder so as to minimize the objective function.

3. The voice conversion training device of claim 1, wherein the filter comprises: a hyperbolic tangent function function having a shape conversion layer which receives as input the speaker vector of the input speaker, the speaker vector of the target speaker, the first scaling coefficient, and the second scaling coefficient, and outputs a three-dimensional speaking rate component which changes due to data scaling, a three-dimensional speaking rate component which does not change due to data scaling, and a residual which is the remaining component unrelated to speaking rate; a calculation unit which calculates a three-dimensional vector which represents the speaking rate of the speaker; and a vector connection unit which combines the three-dimensional vector which represents the speaking rate of the speaker with the residual to generate a speaker vector.

4. The voice conversion training device of claim 2, wherein the filter generates an input speaker vector and a target speaker vector using the speaker vector of the input speaker, the speaker vector of the target speaker, the first scaling factor, and the second scaling factor, outputs the generated input speaker vector to the encoder, and outputs the generated target speaker vector to the decoder.

5. A voice conversion device comprising: a voice conversion training device as described in claim 1; an acquisition unit that acquires the voice signal of the input speaker and the voice signal of the target speaker; a conversion unit that uses the trained encoder and decoder to convert at least one of the voice quality and speaking rate of the voice signal of the input speaker to generate the predicted value; and a vocoder that converts the predicted value into a voice signal.

6. The voice conversion device according to claim 5, wherein the filter generates an input speaker vector and a target speaker vector using a speaker vector of the input speaker, a speaker vector of the target speaker, the first scale factor, and the second scale factor, outputs the generated input speaker vector to the encoder, and outputs the generated target speaker vector to the decoder; the encoder generates an encoded input feature sequence using the input speaker vector and the first input feature sequence, and outputs the generated encoded input feature sequence to the decoder; the decoder generates a predicted value from the encoded input feature sequence using the target speaker vector and the encoded input feature sequence, and outputs the generated predicted value to the vocoder; and a trained vocoder converts the predicted value into a voice signal.

7. The voice conversion device according to claim 5, wherein the filter uses a speaker vector of the input speaker, a speaker vector of the target speaker, the first scaling factor, and the second scaling factor to generate an input speaker vector and a speaker vector in which a portion of the target speaker vector has been replaced with the input speaker vector, outputs the input speaker vector to the encoder, and outputs the speaker vector in which a portion of the target speaker vector has been replaced with the input speaker vector to the decoder; the encoder uses the input speaker vector and the first input feature sequence to generate an encoded input feature sequence, and outputs the generated encoded input feature sequence to the decoder; the decoder uses the input speaker vector in which a portion of the target speaker vector has been replaced with the input speaker vector and the encoded input feature sequence to generate a predicted value from the encoded input feature sequence, and outputs the generated predicted value to a trained vocoder; and the vocoder converts the predicted value into a voice signal.

8. A speech conversion method, comprising: a filter generating an input speaker vector and a target speaker vector using a speaker vector of the input speaker, a speaker vector of the target speaker, a first scaling factor that is information representing speech rate conversion, and a second scaling factor that is information representing the speech rate conversion, outputting the generated input speaker vector to the encoder, and outputting the generated target speaker vector to a decoder; the encoder generating an encoded input feature sequence using the input speaker vector and an input feature sequence generated for the speech signal of the input speaker based on the first scaling factor, and outputting the generated encoded input feature sequence to the decoder; the decoder generating a predicted value from the encoded input feature sequence using the target speaker vector and the encoded input feature sequence, and outputting the generated predicted value to the vocoder; and a trained vocoder converting the predicted value into a speech signal.

9. A program for causing a computer to function as the voice conversion training device according to claim 1.

Images