Voice property conversion device, machine learning method, voice property conversion method, and program

The voice quality conversion device employs an RNN-T model with attention mechanisms and feedback loops to address alignment failures in sequence-to-sequence models, resulting in robust and reliable voice conversion with maintained speech content and rates.

WO2024231982A1PCT designated stage expired Publication Date: 2024-11-14NT T INC

Patent Information

Application Number
PCT/JP2023/017294
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-08
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

Voice quality conversion technologies based on sequence-to-sequence models face alignment failures, leading to irregular speech rates and content disruptions, which impair the practicality of voice conversion applications.

Method used

A voice quality conversion device using a machine learning approach with a Recurrent Neural Network Transducer (RNN-T) model, incorporating an attention mechanism and feedback loops to ensure accurate alignment and spectral error minimization, thereby preventing alignment breakdowns and maintaining utterance content integrity.

Benefits of technology

The solution achieves robust voice quality conversion by preventing alignment failures, ensuring consistent speech rates and content preservation, and providing a comfortable user experience through streaming processing capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2023017294_14112024_PF_FP_ABST
    Figure JP2023017294_14112024_PF_FP_ABST
Patent Text Reader

Abstract

The present disclosure provides a voice property conversion device 10 for performing machine learning of a voice property conversion model in a learning phase, the voice property conversion model 13a includes an input spectrum encoder 131 that inputs an input spectrum as a spectrum of a source speaker, encodes the input spectrum, and outputs the encoded input spectrum, a decoder 133 that decodes the input spectrum input from the input spectrum encoder and outputs a predicted spectrum of a target speaker, and an output spectrum encoder 132 that inputs an output spectrum as the predicted spectrum of the target speaker output by the decoder, encodes the output spectrum, and outputs the encoded output spectrum. The decoder decodes the current input spectrum input from the input spectrum encoder in consideration of whether or not a spectrum of an utterance content corresponding to the input spectrum in the past can have been output on the basis of the output spectrum input from the output spectrum encoder.
Need to check novelty before this filing date? Find Prior Art

Description

Voice conversion device, machine learning method, voice conversion method, and program

[0001] The present disclosure relates to a model training technique for voice conversion that converts the speaker characteristics of a voice.

[0002] Voice conversion (VC) is a technology that converts the speaker identity of the input voice to that of the target speaker while preserving the speech content of the input voice. Statistical model-based voice conversion methods are broadly divided into two categories based on the training data provided. The first is parallel VC, in which the speech content of both the source and target speakers is the same due to the correspondence between their respective speech features. The second is non-parallel VC, in which the speech content of both speakers can be different. The first has been studied for a long time and has the disadvantage of high cost in collecting parallel data, but the advantage of being easier to model than non-parallel VC and of higher quality. The second is that machine learning is possible with non-parallel data, but has the disadvantage of difficulty in ensuring both the intended speech content and the speaker identity of the target speaker during inference.

[0003] Furthermore, the most common method for achieving the first type of parallel VC is to match the number of input and output frames in training data before modeling in order to solve the problem of different speaking rates even when the speech content of the input and output voices is the same. This method uses dynamic time warping (DTW) to stretch or shrink acoustically similar frames to match their lengths, but there is no guarantee that the acoustic features of the input and output will match the same phonemes after stretching or shrinking. Therefore, the accuracy of VC has the drawback of being heavily dependent on the accuracy of DTW.

[0004] To address this drawback, recent advances in sequence-to-sequence (also referred to as seq-to-seq) models have led to the proposal of methods capable of training parallel VC without DTW (Non-Patent Documents 1 and 2). These methods use an attention mechanism to represent the correspondence (alignment) between input and output speech and optimize the model, including the alignment along with the spectral generation error. Non-Patent Document 1 discloses a method that uses an Automatic Speech Recognition (ASR) model as a sequence-to-seq encoder to ensure the speech content while reducing speaker identity of the input speech, and uses its output posterior probability or intermediate output as input to the decoder. On the other hand, Non-Patent Document 2 discloses a method that directly represents the relationship between input and output speech using an attention mechanism. Both methods can achieve high-quality VC.

[0005] Songxiang Liu, Yuewen Cao, Disong Wang, Xixin Wu, Xunying Liu, Helen Meng, "Any-to-Many Voice Conversion with Location-Relative Sequence-to-Sequence Modeling," IEEE. Trans. on Audio, Speech, and Language Processing, vol. 29, pp. 1717-1728, 2021.Hirokazu Kameoka, Kou Tanaka, Damian Kwasny, Takuhiro Kaneko, Nobukatsu Hojo, "ConvS2S-VC: Fully Convolutional Sequence-to-Sequence Voice Conversion," IEEE. Trans. on Audio, Speech, and Language Processing, vol. 28, pp. 1849-1863, 2020.

[0006] However, seq-to-seq based VC alignment can fail if the attention mechanism does not work properly. In this case, not only does the speech rate become abnormally fast or slow, but the speech content also breaks down, with missing utterances and repeated sentences. Because maintaining the speech content of the input speech is essential for VC, this breakdown significantly impairs the practicality of the application.

[0007] The present disclosure has been made in consideration of the above circumstances, and aims to achieve robust voice conversion by suppressing breakdown in speech content.

[0008] In order to achieve the above object, the invention of claim 1 is a voice conversion device that performs machine learning to generate a voice conversion model in a learning phase, wherein the voice conversion model has: an input spectrum encoder that receives an input spectrum as a spectrum of a source speaker, encodes the input spectrum, and outputs the encoded spectrum; a decoder that decodes the input spectrum received from the input spectrum encoder and outputs a predicted spectrum of a target speaker; and an output spectrum encoder that receives an output spectrum as the predicted spectrum of the target speaker output by the decoder, encodes the output spectrum, and outputs the output spectrum; and the decoder decodes the current input spectrum received from the input spectrum encoder, taking into account whether it has been able to output a spectrum of speech content corresponding to the past input spectrum, based on the output spectrum received from the output spectrum encoder.

[0009] As described above, the present disclosure provides the effect of realizing robust voice conversion by suppressing breakdown in speech content.

[0010] FIG. 1 is a functional configuration diagram of a voice conversion device in a training phase according to the first embodiment. FIG. 2 is a conceptual diagram showing the processing contents of a phoneme label correspondence generation unit. FIG. 3 is a functional configuration diagram of a training unit. FIG. 4 is a functional configuration diagram of a voice conversion device in an inference phase according to the first embodiment. FIG. 5 is an electrical hardware configuration diagram of a voice conversion device. FIG. 6 is a flowchart showing processing in the training phase according to the first embodiment. FIG. 7 is a flowchart showing processing in the training phase according to the first embodiment. FIG. 8 is a flowchart showing processing in the inference phase according to the first embodiment. FIG. 9 is a diagram showing a comparison between results of voice conversion according to the first embodiment and results of voice conversion according to conventional technology. FIG. 10 is a diagram showing subjective evaluation scores on a five-point scale regarding voice naturalness and speaker similarity according to the first embodiment. FIG. 11 is a diagram showing subjective evaluation scores on a five-point scale regarding voice naturalness and speaker similarity according to the first embodiment. FIG. 12 is a functional configuration diagram of a voice conversion device in a training phase according to the second embodiment. FIG. 13 is a functional configuration diagram of a voice conversion device in an inference phase according to the second embodiment. 10 is a flowchart showing a process in a learning phase according to a second embodiment; FIG. 11 is a flowchart showing a process in a learning phase according to a second embodiment; FIG. 12 is a flowchart showing a process in an inference phase according to a second embodiment;

[0011] First Embodiment First, the first embodiment will be described with reference to the drawings.

[0012] This embodiment provides a model that is less likely to fail in alignment in sequence-to-sequence (also referred to as seq-to-seq) based voice conversion (VC) and its machine learning (also simply referred to as "training") method. This embodiment applies the feedback technology of recurrent neural network transducer (RNN-T) (see Reference 1), which is widely used in automatic speech recognition (ASR), to VC. While ASR outputs a character string from a trained model, VC outputs a predicted spectrum of a target speaker from a trained model.

[0013] (Reference 1) Alex Graves, "Sequence transduction with recurrent neural networks," Proc. ICML Representation Learning Workshop, 2012. Since ASR requires receiving speech and outputting the recognition result text in real time, it is highly desirable to ensure that alignment does not fail. RNN-T is a seq-to-seq model that is effective in ASR because it is less likely to fail alignment and is capable of streaming processing. This embodiment describes a method for realizing VC based on RNN-T to take advantage of this advantage. Based on this, Modifications 1 and 2 are also described.

[0014] The voice conversion device 10 is configured by one or more computers. When the voice conversion device 10 is configured by multiple computers, it may be referred to as a "voice conversion device" or a "voice conversion system."

[0015] [Functional Configuration of Voice Conversion Device] <Functional Configuration of Learning Phase> First, the functional configuration of the voice conversion device 10 in the learning phase will be described with reference to Fig. 1. Fig. 1 is a functional configuration diagram of the voice conversion device in the learning phase according to the first embodiment. In the learning phase, the voice conversion device 10 performs machine learning on a voice conversion model 13a.

[0016] 1, the voice conversion device 10 includes a spectrum calculation unit 12, a voice conversion model 13a, a learning unit 14, and a correspondence generation unit 15. These units each have a function realized by commands from the CPU 101 in FIG.

[0017] The spectrum calculation unit 12 calculates and outputs the spectrum of the source speaker based on the input voice of the source speaker. Note that if the voice conversion device 10 can acquire the spectrum of the source speaker from an external device, the spectrum calculation unit 12 is not necessary.

[0018] The voice conversion model 13a is a machine learning model based on a neural network, and is composed of an input spectrum encoder 131, an output spectrum encoder 132, a decoder 133, a transition probability prediction unit 134, and a spectrum prediction unit 135. These are modules capable of machine learning, each of which is composed of a neural network and has its own model parameters.

[0019] Of these, the input spectrum encoder 131 receives an input spectrum as the spectrum of the source speaker from the spectrum calculation unit 12, encodes this input spectrum, and outputs it.

[0020] The output spectrum encoder 132 receives the output spectrum as the predicted spectrum of the target speaker output by the spectrum prediction unit 135 (originally the decoder 133) as feedback, encodes this output spectrum, and outputs it.

[0021] The decoder 133 decodes the current input spectrum input from the input spectrum encoder 131, taking into consideration whether it is possible to output a spectrum of the speech content corresponding to the past input spectrum, based on the output spectrum input from the output spectrum encoder 132.

[0022] The transition probability prediction unit 134 predicts transition probabilities based on the input spectrum of the source speaker input from the decoder 133, and outputs the predicted transition probabilities to the training unit 14. The training unit 14 will be described later.

[0023] The spectrum prediction unit 135 generates and outputs a predicted spectrum of the target speaker based on the input spectrum of the source speaker input from the decoder 133. When the spectrum prediction unit 135 calculates the predicted spectrum of the target speaker at the next time (u+1), it inputs the spectrum at the current time (u) (predicted spectrum of the target speaker) as the output spectrum by feedback to the output spectrum encoder 132. The spectrum prediction unit 135 repeats this feedback process to generate predicted spectra of the target speaker for the number of frames of the input spectrum.

[0024] The decoder 133 may include a spectrum prediction unit 135 and a transition probability prediction unit 134. For example, if the spectrum of a speech is represented by an 80-dimensional vector sequence, the decoder 133 predicts 81 dimensions including a one-dimensional transition probability.

[0025] On the other hand, the correspondence generation unit 15 inputs the phoneme labels with time information of the source speaker ("alignment" in the narrow sense) and the phoneme labels with time information of the target speaker ("alignment" in the narrow sense), generates a correct alignment for machine learning that shows the correspondence between both phoneme labels, and outputs the correct alignment to the learning unit 14.

[0026] (Phoneme Label Correspondence) Here, the generation of phoneme label correspondence will be described with reference to FIG. 2. FIG. 2 is a conceptual diagram showing the processing content of the phoneme label correspondence generation unit. FIG. 2(a) shows the source speaker's phoneme label sl with time information when the source speaker's spectrum ss is generated from the source speaker's speech waveform sw. FIG. 2(b) shows the target speaker's phoneme label tl with time information when the target speaker's spectrum ts is generated from the target speaker's speech waveform tw. Here, the first half of the utterance content "Noroshiku" is shown. As shown in FIG. 2, the correspondence generation unit 15 generates and outputs a correct alignment indicating the correspondence of the phoneme label to the target speaker's phoneme label tl with time information from the source speaker's phoneme label sl with time information. This correspondence will be described in more detail later.

[0027] (Functional Configuration of Learning Unit) Next, the functional configuration of the learning unit 14 will be described with reference to Fig. 3. Fig. 3 is a diagram showing the functional configuration of the learning unit.

[0028] As shown in FIG. 3, the learning unit 14 includes an alignment calculation unit 141 , an alignment mask unit 142 , a spectral weighting unit 143 , a spectral error calculation unit 144 , and a model optimization unit 145 .

[0029] Of these, the alignment calculation unit 141 calculates and outputs a predicted alignment based on the predicted transition probability input from the transition probability prediction unit 134 .

[0030] The alignment masking unit 142 generates and outputs a masked predicted alignment based on the predicted alignment input from the alignment calculation unit 141 and the correct alignment input from the correspondence generation unit 15 described above.

[0031] The spectral weighting unit 143 generates and outputs a weighted predicted spectrum of the target speaker based on the masked predicted alignment input from the alignment masking unit 142 and the predicted spectrum of the target speaker input from the above-mentioned spectral prediction unit 135.

[0032] The spectral error calculation unit 144 calculates the error between the weighted predicted spectrum of the target speaker input from the spectral weighting unit 143 and the correct spectrum of the target speaker, and outputs the spectral error.

[0033] The model optimization unit 145 updates each model parameter of the voice conversion model 13a (the input spectrum encoder 131, the output spectrum encoder 132, the decoder 133, the spectrum prediction unit 135, and the transition probability prediction unit 134) by error backpropagation so as to minimize the spectrum error input from the spectrum error calculation unit 144. When the model optimization unit 145 treats the spectrum as a continuous value as it is, mean absolute error, mean squared error, etc. can be used as the spectrum error. When the model optimization unit 145 treats the spectrum as a discrete representation by vector quantization, etc., cross entropy may also be used.

[0034] The voice conversion device 10 refines the voice conversion model by repeatedly applying the processing flow configured as above to the training data.

[0035] In the explanation so far, the acoustic features used for input and output have been referred to as spectra, but it is also possible to use spectrograms obtained by short-time Fourier transforms, or mel-spectrograms, which are low-dimensional representations of spectrograms adapted to human hearing. It is also possible to use mel-cepstrum, which is often used in speech synthesis, or a vector that concatenates the fundamental frequency, which is the pitch of the voice.

[0036] <Background Technology Leading to Present Embodiment, Detailed Technology of Present Embodiment> Next, the background technology leading to the present embodiment and detailed technology of the present embodiment will be described.

[0037] In the case of automatic speech recognition (ASR), since the target is a discrete symbol such as a phoneme, the transition probability of an RNN-T can be optimized by minimizing cross-entropy by adding dwell and transition symbols in addition to these symbols and minimizing the cross-entropy of the predictor for whether to transition to the next phoneme. However, in the case of voice conversion (VC), the output is a continuous spectrum, and the scales of the output values ​​of the transition probability and dwell probability predictor (134) and the spectral predictor (135) are significantly different, making it difficult to balance the training of the two. To solve this problem, Reference 2 proposes a lazy forward algorithm for applying RNN-T to speech synthesis tasks where the output is a spectrum, similar to VC.

[0038] (Reference 2) Jiawei Chen, Xu Tan, Yichong Leng, Jin Xu, Guihua Wen, Tao Qin, Tie-Yan Liu, "Speech-T: Transducer for text to speech and beyond," Proc. NeurIPS, vol.34, pp. 6621-6633, 2021. This replaces the transition probability used in the probability calculation of the input / output trellis with the value obtained from the stay probability φ, as shown in the following (Equation 1).

[0039] Here, t and u are the time indices of the input text and output spectrum, respectively, and α(t,u) is the forward probability (= predicted alignment) on the trellis for the t-th input text and the output spectrum of frame u. The transition probability is given by 1-φ(t,u-1). The recurrence formula for this α is calculated for each lattice of the trellis, and the loss function is defined as follows (Equation 2).

[0040] where:

[0041] are the ground truth spectrum and predicted spectrum of the target speaker at time u+1, respectively. By designing the loss function in this way, even regression tasks in which the model requires continuous values ​​can be realized within the RNN-T framework. Note that since calculating all trellis elements is not realistic from the perspective of computational complexity, the voice conversion device 10 masks the predicted alignment using the ground truth alignment and omits trellis calculations for impossible paths.

[0042] Furthermore, in this embodiment, focusing on the advantages of the Lazy Forward algorithm, parallel VC is realized by using the spectra of the source speaker and the target speaker as the input and output of Reference 2, respectively. However, unlike speech synthesis, the temporal correspondence between the input spectrum and the output spectrum is not explicitly given. In this embodiment, to realize VC using RNN-T, phoneme alignments of the source speaker and the target speaker are used, and a correct alignment is generated from the correspondence between the alignments. However, even if the phonemes are the same, their sequence lengths are not necessarily the same, so the correct alignment matrix will not be a square matrix and some ingenuity is required.

[0043] Therefore, the voice conversion device 10 of this embodiment has a correspondence generation unit 15 as a function that applies a constraint method (see Reference 3) that forces a non-square alignment matrix to be monotonic. The correspondence generation unit 15 is a means for providing the correct solution for the above α.

[0044] (Reference 3) Hideyuki Tachibana, Katsuya Uenoyama, Shunsuke Aihara, "Efficiently trainable text-to-speech system based on deep convolutional networks with guided attention," Proc. ICASSP, pp.4784-4788, 2018. The technique in Reference 3 designs the following penalty matrix, expressed between 0 and 1, which imposes a small penalty on monotonic alignments and increases the penalty as the alignment deviates from the monotonic alignment.

[0045] where T and U are the number of frames of the input spectrum and the output spectrum, respectively. In other words, when (Equation 5) is a unit matrix, the correct alignment is calculated as argmax for each row of (Equation 6).

[0046]

[0047] However, because this is an approximately calculated correct alignment, the correspondence generation unit 15 can stabilize the convergence of the alignment by considering not only the argmax frames of each row but also a predetermined number of frames before and after them (for example, one frame before and after) as correct when constructing the correspondence. This allows the correspondence generation unit 15 to generate a correct alignment even when, unlike automatic speech recognition (ASR), the speaking time of the source speaker and the speaking time of the target speaker differ, as in voice conversion (VC) (see FIG. 2).

[0048] <Functional Configuration of Inference Phase> Next, the functional configuration of the voice conversion device in the inference phase will be described with reference to Fig. 4. Fig. 4 is a diagram showing the functional configuration of the voice conversion device in the inference phase according to the first embodiment.

[0049] As shown in Fig. 4, the voice conversion device 10 has a spectrum calculation unit 12, a trained voice conversion model 13b, a speech waveform generation unit 16, and an end determination unit 17. These units have functions realized by commands from the CPU 101 in Fig. 5 based on a program. Note that functional components similar to those in the learning phase are denoted by the same reference numerals, and descriptions thereof will be omitted.

[0050] The voice conversion model 13b has the same functional configuration (modules) as the voice conversion model 13a in the learning phase.

[0051] The speech waveform generating unit 16 generates and outputs the speech of the target speaker based on the predicted spectrum of the target speaker input from the spectrum predicting unit 135 .

[0052] The termination determination unit 17 determines whether or not to terminate output of the predicted spectrum of the target speaker to the spectrum prediction unit 135 based on the predicted transition probabilities input from the transition probability prediction unit 134 (see Reference 1). Because the termination determination unit 17 is trained to transition at the timing when the phonemes of the input spectrum change, the termination determination unit 17 causes the spectrum prediction unit 135 to continue outputting the predicted spectrum of the target speaker until it has output a spectrum with the same number of phonemes as the input spectrum. However, unlike ASR, in VC it ​​is not possible to determine termination before the source speaker has finished speaking, so if the number of frames in the output spectrum is equal to or less than the number of frames in the input spectrum, termination will not occur.

[0053] [Hardware Configuration] Next, the electrical hardware configuration of the voice conversion device 10 will be described with reference to Fig. 5. Fig. 5 is a diagram showing the electrical hardware configuration of the voice conversion device.

[0054] As shown in FIG. 5, the voice conversion device 10 is a computer and includes a CPU 101, a ROM 102, a RAM 103, an SSD 104, an external device connection I / F (Interface) 105, a network I / F 106, a display 107, an input device 108, a media I / F 109, and a bus line 110.

[0055] Of these, the CPU 101 controls the overall operation of the voice conversion device 10. The ROM 102 stores programs such as IPL used to drive the CPU 101. The RAM 103 is used as a work area for the CPU 101.

[0056] The SSD 104 reads or writes various data under the control of the CPU 101. Note that instead of the SSD 104, a hard disk drive (HDD) may be used.

[0057] The external device connection I / F 105 is an interface for connecting various external devices, such as a display, a speaker, a keyboard, a mouse, a USB memory, and a printer.

[0058] The network I / F 106 is an interface for performing data communication via the communication network 100 .

[0059] The display 107 is a type of display means such as a liquid crystal display or organic electroluminescence (EL) display that displays various images.

[0060] The input device 108 is a keyboard, a pointing device, or the like, and is an example of an input means for accepting input operations such as characters, numbers, and various instructions.

[0061] The media I / F 109 controls reading and writing (storing) of data from and to a recording medium 109m such as a flash memory, etc. The recording medium 109m includes DVDs and Blu-ray Discs (registered trademarks).

[0062] The bus line 110 is an address bus, a data bus, etc. for electrically connecting the components such as the CPU 101 shown in FIG.

[0063] [Processing or Operation of First Embodiment] Next, the processing or operation of the voice conversion device 10 in the learning phase and inference phase will be described with reference to FIGS.

[0064] <Processing or Operation in Learning Phase> FIGS. 6 and 7 are flowcharts showing processing in the learning phase according to the first embodiment.

[0065] S110: First, as shown in FIG. 6, the spectrum calculation unit 12 calculates and outputs the spectrum of the source speaker based on the input speech of the source speaker.

[0066] S111: The input spectrum encoder 131 receives the current input spectrum as the spectrum of the source speaker from the spectrum calculation unit 12, encodes this input spectrum, and outputs it.

[0067] S112: The output spectrum encoder 132 receives, as feedback, the past output spectrum as the predicted spectrum of the target speaker output by the spectrum prediction unit 135 (originally the decoder 133), and encodes and outputs this output spectrum.

[0068] S113: The decoder 133 decodes the current input spectrum input from the input spectrum encoder 131, taking into consideration whether it is possible to output a spectrum of the speech content corresponding to the past input spectrum based on the output spectrum input from the output spectrum encoder 132.

[0069] S114: The transition probability prediction unit 134 predicts and outputs transition probabilities based on the current input spectrum of the source speaker input from the decoder 133. Furthermore, the spectrum prediction unit 135 generates and outputs a predicted spectrum of the target speaker based on the current input spectrum of the source speaker input from the decoder 133.

[0070] S115: As shown in FIG. 7, the alignment calculation unit 141 calculates and outputs a predicted alignment based on the predicted transition probabilities input from the transition probability prediction unit 134.

[0071] S116: The correspondence generation unit 15 receives the phoneme labels with time information of the source speaker and the phoneme labels with time information of the target speaker to generate a correct alignment, and outputs the correct alignment to the training unit 14.

[0072] S117: The alignment masking unit 142 generates and outputs a masked predicted alignment based on the predicted alignment input from the alignment calculation unit 141 and the correct alignment input from the correspondence generation unit 15 described above.

[0073] S118: The spectral weighting unit 143 generates and outputs a weighted predicted spectrum of the target speaker based on the masked predicted alignment input from the alignment masking unit 142 and the predicted spectrum of the target speaker (current input spectrum) input from the above-mentioned spectral prediction unit 135.

[0074] S119: The spectrum error calculation unit 144 calculates the error between the weighted predicted spectrum of the target speaker input from the spectrum weighting unit 143 and the correct spectrum of the target speaker, and outputs the spectrum error.

[0075] S120: The model optimization unit 145 updates each parameter of the voice conversion model 13a by backpropagation so as to minimize the spectral error input from the spectral error calculation unit 144.

[0076] The voice conversion device 10 repeats the above steps S110 to S120 to complete the machine learning of the voice conversion model 13a.

[0077] <Processing or Operation in Inference Phase> FIG. 8 is a flowchart showing processing in the inference phase according to the first embodiment.

[0078] S150: First, as shown in FIG. 8, the spectrum calculation unit 12 calculates and outputs the spectrum of the source speaker based on the input speech of the source speaker.

[0079] S151: The input spectrum encoder 131 receives the current input spectrum as the spectrum of the source speaker from the spectrum calculation unit 12, encodes this input spectrum, and outputs it.

[0080] S152: The output spectrum encoder 132 receives, as feedback, the past output spectrum as the predicted spectrum of the target speaker output by the spectrum prediction unit 135 (originally the decoder 133), and encodes and outputs this output spectrum.

[0081] S153: The decoder 133 decodes the current input spectrum input from the input spectrum encoder 131, taking into consideration whether it is possible to output a spectrum of the speech content corresponding to the past input spectrum based on the output spectrum input from the output spectrum encoder 132.

[0082] S154: The transition probability prediction unit 134 predicts and outputs transition probabilities based on the current input spectrum of the source speaker input from the decoder 133. Furthermore, the spectrum prediction unit 135 generates and outputs a predicted spectrum of the target speaker based on the current input spectrum of the source speaker input from the decoder 133.

[0083] S155: The speech waveform generating unit 16 generates and outputs the speech of the target speaker based on the predicted spectrum of the target speaker input from the spectrum predicting unit 135.

[0084] S156: The termination determination unit 17 determines whether or not to terminate the output of the predicted spectrum of the target speaker based on the predicted transition probabilities input from the transition probability prediction unit 134. If not (NO), the process returns to step S150. On the other hand, if terminated (YES), the inference phase process ends.

[0085] [Experimental Results] Next, experimental results of this embodiment and the prior art will be compared with each other using FIGS.

[0086] FIG. 9 is a diagram showing a comparison between the results of voice conversion according to the first embodiment and the results of voice conversion according to the prior art.

[0087] Figure 9 shows the Mel-Cepstrum Distortion (MCD) and Character Error Rate (CER) for each method. The "Method" column lists the speech input to the speech recognizer: speech recorded in a studio or the like (GT, RESYN), speech obtained using the conventional technique p1 described in Non-Patent Document 1 (BNE-S2SMoL-VC), speech obtained using the conventional technique p2 described in Non-Patent Document 2 (ConvS2S-VC), and speech obtained using the method e1 of this embodiment (VC-T). GT represents the case where the speech recorded in a studio or the like is input as is. RESYN represents the case where the speech waveform recorded in a studio or the like is first converted into a spectrum and then converted back into a speech waveform before input. For conventional technique p2 and this embodiment, the results are also shown for a case where the entire sequence of the source speaker's speech is input to the input spectrum encoder at once, taking future information into account (OFFLINE), and a practical use case where speech is input frame by frame without using future information for streaming operation (STREAM). To obtain these results, Reference 4 was used for the speech recognizer, and Reference 5 was used for the conversion from spectrum to speech waveform generation.

[0088] (Reference 4) Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, Michael Auli, "wav2vec 2.0: A framework for self-supervised learning of speech representations," Proc. NeurIPS, pp. 12449-12460, 2020. (Reference 5) Kong, J. Kim, and J. Bae, "HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis," Proc. NeurIPS, vol. 33, pp. 17–022–17–033, 2020. CER indicates the character error rate when speech is input to a speech recognizer; a lower value indicates better results. In other words, the closer the CER value is to that of GT, the more easily voice conversion can be performed without disrupting the speech content.

[0089] F2F indicates the case of speech obtained by voice conversion from a female voice to a female voice. M2F indicates the case of speech obtained by voice conversion from a male voice to a female voice. F2M indicates the case of speech obtained by voice conversion from a female voice to a male voice. In general, voice conversion between opposite-sex people is more difficult than voice conversion between same-sex people, and tends to result in a higher character error rate.

[0090] Note that the conventional techniques p1 and p2 and the method of this embodiment require restoring a speech waveform from a voice-converted spectrum. For this reason, the character error rate of the recorded speech (GT) as well as that of RESYN are shown for reference as the upper limit performance when restoring a speech waveform from a spectrum without using voice conversion.

[0091] As shown in Figure 9, the voice conversion method of this embodiment is superior to the conventional techniques p1 and p2 not only in same-sex voice conversion but also in opposite-sex voice conversion. In particular, the results of opposite-sex voice conversion using the conventional techniques p1 and p2 are extremely degraded, while no significant degradation is observed using this embodiment. The reason why this embodiment is superior is that, for example, when the speech content is "yoroshiku," the voice conversion device 10 can take into account both (1) the correspondence between the spectrum of the currently input "Y" and the spectrum of the "Y" it has output, and (2) the transition probability of the phoneme corresponding to the spectrum, such as from "Y" to "O."

[0092] 10 and 11 show subjective evaluation scores on a five-point scale for voice naturalness and speaker similarity, respectively. Because voice conversion involves spectral conversion, the scores are always lower than the evaluation results of RESYN, which does not involve conversion. FIG. 11 shows the evaluation results when comparing each method, excluding RESYN. As shown in the Avg. column in FIG. 11, the evaluation results tend to be worse in the case where no practical future information is used (STREAM) than in the case where operation is not possible until the end of the utterance (OFFLINE). However, the degree of degradation in this embodiment was less than that of conventional technology p2. Furthermore, statistical testing showed no significant difference between STREAM and OFFLINE in this embodiment, indicating no significant degradation.

[0093] [Major Effects of the First Embodiment] As described above, the first embodiment has the effect of realizing a VC that is robust against alignment breakdown by suppressing breakdowns in speech content. Furthermore, since it is desirable in VC to output converted speech immediately after inputting speech, RNN-T is also strong in streaming processing, making it possible to provide a VC with a comfortable response to the user.

[0094] Specifically, to avoid the problem of alignment failure, which was an issue with VCs of conventional inventions based on the seq-to-seq model, the voice conversion device 10 inputs not only the input spectrum but also an output spectrum based on the past (immediately preceding) input spectrum to the decoder via feedback. This allows the decoder 133 to consider whether it is able to output a spectrum of the speech content that corresponds to the input spectrum, which helps to avoid alignment failure.

[0095] Another reason why alignment failure can be avoided is that the voice conversion device 10 learns the transition probability to the next phoneme together with the spectral error. The voice conversion device 10 weights the predicted spectrum based on the ground truth alignment and the predicted alignment derived from the transition probability, and calculates the spectral error based on the trellis of the input and output frames. This makes the predicted alignment monotonic and robust to failure.

[0096] [Modification 1] Next, a description will be given of Modification 1 of the first embodiment. The difference from the first embodiment is that the speaker vector of the target speaker is input to the decoder 133.

[0097] 1, in the first modification, the decoder 133 receives the speaker vector of the target speaker as well as the input spectrum from the input spectrum encoder 131 and the output spectrum from the output spectrum encoder 132. Also, in the inference phase of FIG. 4, the decoder 133 performs the same processing as in the learning phase.

[0098] Here, speaker vectors can be obtained by converting speaker IDs into one-hot vectors or by passing speaker IDs through a trainable embedding layer. Also, vectors obtained as intermediate representations for speaker recognition tasks (see Reference 6) and vectors obtained from pre-trained models trained on huge data sets (see Reference 4) can also be used as speaker vectors. When using pre-trained models trained on a huge number of speakers for these other tasks, it is assumed that sufficient speaker representations have been acquired, making it possible to realize zero-shot VC, which converts voices of speakers not included in the VC training data as target speakers.

[0099] (Reference 6) David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey, Sanjeev Khudanpur, "X-vectors: Robust DNN embeddings for speaker recognition," Proc. ICASSP, pp. 5329-5333, 2018. In the first embodiment described above, the source speaker and target speaker were limited to a one-to-one VC. In contrast, Modification 1 uses the speaker vector of the target speaker as conditioning for the decoder 133, making it possible to expand the target speaker to multiple speakers, i.e., the relationship between the source speaker and the target speaker to one-to-many.

[0100] [Modification 2] Next, a description will be given of Modification 2 of the first embodiment. The difference from Modification 1 is that not only the spectrum of the source speaker but also the speaker vector of the source speaker is input to the input spectrum encoder 131.

[0101] 1, in Variation 2, the input spectrum encoder 131 inputs the speaker vector of the source speaker along with the spectrum of the source speaker. Furthermore, as in Variation 1, the decoder 133 inputs the speaker vector of the target speaker along with the input spectrum from the input spectrum encoder 131 and the output spectrum from the output spectrum encoder 132.

[0102] In the inference phase of FIG. 4, the input spectrum encoder 131 and the decoder 133 perform the same processing as in the learning phase.

[0103] In the above-mentioned variant 1, the target speaker can be multiple, but the source speaker is limited to one. In variant 2, the source speaker is expanded to multiple people so that more people can use VC. Specifically, the speaker vector described in variant 1 is also calculated for the source speaker, and this and the source speaker's spectrum are provided as conditioning for the input spectrum encoder 131. This makes it possible to expand the relationship between source speakers and target speakers to many-to-many.

[0104] In the explanation so far, the acoustic features used for input and output have been referred to as spectra, but it is also possible to use spectrograms obtained by short-time Fourier transforms, or mel-spectrograms, which are low-dimensional representations of spectrograms adapted to human hearing. It is also possible to use mel-cepstrum, which is often used in speech synthesis, or a vector that concatenates the fundamental frequency, which is the pitch of the voice.

[0105] Second Embodiment First, a second embodiment will be described with reference to the drawings. In the first embodiment, machine learning and inference are performed using a spectrum extracted from a speech waveform, but in the second embodiment, an example will be shown in which machine learning and inference are performed using the speech waveform as is.

[0106] [Functional Configuration of Voice Conversion Device] <Functional Configuration in Learning Phase> First, the functional configuration of the voice conversion device 20 in the learning phase will be described with reference to Fig. 12. Fig. 12 is a functional configuration diagram of the voice conversion device in the learning phase according to the second embodiment. In the learning phase, the voice conversion device 20 performs machine learning to generate a voice conversion model 23a. Note that the electrical hardware configuration of the voice conversion device 20 is the same as that of the voice conversion device 10 of the first embodiment (see Fig. 5), and therefore description thereof will be omitted.

[0107] 12, the voice conversion device 20 includes a voice conversion model 23a, a learning unit 24, and a correspondence generation unit 25. These units each have a function that is realized by commands from the CPU 101 in FIG. 5 based on a program.

[0108] The voice conversion model 23a is a machine learning model based on a neural network, and is composed of an input voice waveform encoder 231, an output voice waveform encoder 232, a decoder 233, a transition probability prediction unit 234, and a voice waveform prediction unit 235. These are modules capable of machine learning, each of which is composed of a neural network and has its own model parameters.

[0109] Of these, the input speech waveform encoder 231 receives an input speech waveform as the speech waveform of a source speaker from outside the voice conversion device 20, encodes this input speech waveform, and outputs it. During the learning phase, the input speech waveform encoder 231 can predict a speech waveform by having a DNN (Deep Neural Network) mask part of the input speech waveform and perform machine learning. As a result, the input speech waveform encoder 231, unlike the first embodiment, can input a speech waveform and output a speech waveform.

[0110] The output speech waveform encoder 232 receives, via feedback, the output speech waveform as the predicted speech waveform of the target speaker output by the speech waveform prediction unit 235 (originally the decoder 233), encodes this output speech waveform, and outputs it. Like the input speech waveform encoder 231, the output speech waveform encoder 232 can also predict the speech waveform using a DNN.

[0111] The decoder 233 decodes the current input audio waveform input from the input audio waveform encoder 231, taking into consideration whether it is possible to output an audio waveform of the speech content corresponding to the past input audio waveform, based on the output audio waveform input from the output audio waveform encoder 232.

[0112] The transition probability prediction unit 234 predicts transition probabilities based on the input speech waveform of the source speaker input from the decoder 233, and outputs the predicted transition probabilities to the training unit 24. The training unit 24 will be described later.

[0113] The speech waveform prediction unit 235 generates and outputs a predicted speech waveform of the target speaker based on the input speech waveform of the source speaker input from the decoder 233. When the speech waveform prediction unit 235 determines the predicted speech waveform of the target speaker at the next time (u+1), it inputs the speech waveform at the current time (u) (predicted speech waveform of the target speaker) as the output speech waveform by feedback to the output speech waveform encoder 232. The speech waveform prediction unit 235 repeats this feedback process to generate predicted speech waveforms of the target speaker for the number of frames of the input speech waveform.

[0114] The decoder 233 may include a speech waveform prediction unit 235 and a transition probability prediction unit 234. In this case, since the speech sequence is one-dimensional, the decoder 233 predicts two dimensions in total, including the one-dimensional transition probability.

[0115] On the other hand, the correspondence generation unit 25 performs the same processing as the correspondence generation unit 15 of the first embodiment, and therefore a description thereof will be omitted.

[0116] (Functional Configuration of Learning Unit) Next, the functional configuration of the learning unit 24 will be described with reference to Fig. 13. Fig. 13 is a diagram showing the functional configuration of the learning unit.

[0117] As shown in FIG. 13, the learning unit 24 includes an alignment calculation unit 241 , an alignment mask unit 242 , a voice waveform weighting unit 243 , a voice waveform reconstruction error calculation unit 244 , and a model optimization unit 245 .

[0118] Of these, the alignment calculation unit 241 calculates and outputs a predicted alignment based on the predicted transition probability input from the transition probability prediction unit 234 .

[0119] The alignment masking unit 242 generates and outputs a masked predicted alignment based on the predicted alignment input from the alignment calculation unit 241 and the correct alignment input from the correspondence generation unit 25 described above.

[0120] The speech waveform weighting unit 243 generates and outputs a weighted predicted speech waveform of the target speaker based on the masked predicted alignment input from the alignment masking unit 242 and the predicted speech waveform of the target speaker input from the above-mentioned speech waveform prediction unit 235.

[0121] The speech waveform reconstruction error calculation unit 244 calculates the error between the weighted predicted speech waveform of the target speaker input from the speech waveform weighting unit 243 and the correct speech waveform of the target speaker, and outputs the speech waveform reconstruction error.

[0122] The model optimization unit 245 updates each model parameter of the voice conversion model 23a (the input speech waveform encoder 231, the output speech waveform encoder 232, the decoder 233, the speech waveform prediction unit 235, and the transition probability prediction unit 234) by error backpropagation so as to minimize the reconstruction error of the speech waveform input from the speech waveform reconstruction error calculation unit 244. When the model optimization unit 245 treats the speech waveform as a continuous value as it is as the speech waveform error, mean absolute error, mean squared error, etc. can be used. When the model optimization unit 245 treats the speech waveform as a discrete representation by vector quantization, etc., cross entropy may also be used.

[0123] The voice conversion device 20 refines the voice conversion model by repeatedly applying the processing flow according to the above configuration to the training data.

[0124] <Functional Configuration of Inference Phase> Next, the functional configuration of the voice conversion device in the inference phase will be described with reference to Fig. 14. Fig. 14 is a functional configuration diagram of the voice conversion device in the inference phase according to the second embodiment.

[0125] As shown in Fig. 14, the voice conversion device 20 has a trained voice conversion model 23b and an end determination unit 27. These units are functions realized by commands from the CPU 101 in Fig. 5 based on a program. Note that functional components similar to those in the learning phase are denoted by the same reference numerals and will not be described again.

[0126] The voice conversion model 23b has the same functional configuration (modules) as the voice conversion model 23a in the learning phase.

[0127] The end determination unit 27 performs the same processing as the end determination unit 17 of the first embodiment.

[0128] [Processing or Operation of Second Embodiment] Next, the processing or operation of the voice conversion device 20 in the learning phase and inference phase will be described with reference to FIGS.

[0129] <Processing or Operation in Learning Phase> FIGS. 15 and 16 are flowcharts showing processing in the learning phase according to the first embodiment.

[0130] S211: The input speech waveform encoder 231 receives a current input speech waveform as the speech waveform of the source speaker from outside the voice quality conversion device 20, encodes this input speech waveform, and outputs it.

[0131] S212: The output speech waveform encoder 232 receives, via feedback, the past output speech waveform as the predicted speech waveform of the target speaker output by the speech waveform prediction unit 235 (originally the decoder 233), and encodes and outputs this output speech waveform.

[0132] S213: The decoder 233 decodes the current input audio waveform input from the input audio waveform encoder 231, taking into consideration whether it is possible to output an audio waveform of the speech content corresponding to the past input audio waveform based on the output audio waveform input from the output audio waveform encoder 232.

[0133] S214: The transition probability prediction unit 234 predicts and outputs a transition probability based on the current input speech waveform of the source speaker input from the decoder 233. In addition, the speech waveform prediction unit 235 generates and outputs a predicted speech waveform of the target speaker based on the current input speech waveform of the source speaker input from the decoder 233.

[0134] S215: As shown in FIG. 16, the alignment calculation unit 241 calculates and outputs a predicted alignment based on the predicted transition probabilities input from the transition probability prediction unit 234.

[0135] S216: The correspondence generation unit 25 receives the phoneme labels with time information of the source speaker and the phoneme labels with time information of the target speaker to generate a correct alignment, and outputs the correct alignment to the training unit 24.

[0136] S217: The alignment masking unit 242 generates and outputs a masked predicted alignment based on the predicted alignment input from the alignment calculation unit 241 and the ground truth alignment input from the correspondence generation unit 25 described above.

[0137] S218: The speech waveform weighting unit 243 generates and outputs a weighted predicted speech waveform of the target speaker based on the masked predicted alignment input from the alignment masking unit 242 and the predicted speech waveform of the target speaker (current input speech waveform) input from the above-mentioned speech waveform prediction unit 235.

[0138] S219: The speech waveform reconstruction error calculation unit 244 calculates the error between the weighted predicted speech waveform of the target speaker input from the speech waveform weighting unit 243 and the correct speech waveform of the target speaker, and outputs the speech waveform reconstruction error.

[0139] S220: The model optimization unit 245 updates each parameter of the voice conversion model 23a by error backpropagation so as to minimize the speech waveform error input from the speech waveform reconstruction error calculation unit 244.

[0140] The voice conversion device 20 repeats the above steps S211 to S220 to complete the machine learning of the voice conversion model 23a.

[0141] <Processing or Operation in Inference Phase> FIG. 17 is a flowchart showing processing in the inference phase according to the first embodiment.

[0142] S251: The input speech waveform encoder 231 receives a current input speech waveform as the speech waveform of the source speaker from outside the voice quality conversion device 20, encodes this input speech waveform, and outputs it.

[0143] S252: The output speech waveform encoder 232 receives, via feedback, the past output speech waveform as the predicted speech waveform of the target speaker output by the speech waveform prediction unit 235 (originally the decoder 233), and encodes and outputs this output speech waveform.

[0144] S253: The decoder 233 decodes the current input audio waveform input from the input audio waveform encoder 231, taking into consideration whether it is possible to output an audio waveform of the speech content corresponding to the past input audio waveform based on the output audio waveform input from the output audio waveform encoder 232.

[0145] S254: The transition probability prediction unit 234 predicts and outputs a transition probability based on the current input speech waveform of the source speaker input from the decoder 233. In addition, the speech waveform prediction unit 235 generates and outputs a predicted speech waveform of the target speaker based on the current input speech waveform of the source speaker input from the decoder 233.

[0146] S255: The termination determination unit 27 determines whether or not to terminate the output of the predicted speech waveform of the target speaker based on the predicted transition probabilities input from the transition probability prediction unit 234. If not (NO), the process returns to step S251. On the other hand, if it is (YES), the process of the inference phase ends.

[0147] [Major Effects of the Second Embodiment] As described above, the second embodiment provides the same effects as the first embodiment.

[0148] Furthermore, in the second embodiment, a spectrum is not generated from the audio waveform, and machine learning and inference are performed on the audio waveform as is, so there is no mismatch with the audio waveform generation means that can occur in the first embodiment, and the quality is superior.

[0149] Supplementary Note: The present invention is not limited to the above-described embodiment, and may have the following configurations or processes (operations), for example.

[0150] (1) The voice conversion device 10 can be realized by a computer and a program, but this program can also be recorded on a (non-temporary) recording medium or provided via a communication network 100 such as the Internet.

[0151] (2) The CPU 101 as a processor may be a single processor or multiple processors.

[0152] Supplementary Notes The above-described embodiment can also be expressed as the following invention.

[0153] [Appendix 1] A voice conversion device that machine-learns a voice conversion model in a learning phase, wherein the voice conversion model has: an input voice waveform encoder that inputs an input voice waveform as the voice waveform of a source speaker, encodes it, and outputs it; a decoder that decodes the input voice waveform input from the input voice waveform encoder and outputs a predicted voice waveform of a target speaker; and an output voice waveform encoder that inputs an output voice waveform as the predicted voice waveform of the target speaker output by the decoder, encodes it, and outputs it; wherein the decoder decodes the current input voice waveform input from the input voice waveform encoder, based on the output voice waveform input from the output voice waveform encoder, taking into account whether it is able to output a voice waveform of speech content corresponding to the past input voice waveform.

[0154] [Appendix 2] A voice conversion device that performs voice conversion using a machine-learned voice conversion model in an inference phase, wherein the machine-learned voice conversion model has: an input voice waveform encoder that inputs an input voice waveform as the voice waveform of a source speaker, encodes it, and outputs it; a decoder that decodes the input voice waveform input from the input voice waveform encoder and outputs a predicted voice waveform of a target speaker; and an output voice waveform encoder that inputs an output voice waveform as the predicted voice waveform of the target speaker output by the decoder, encodes it, and outputs it; wherein the decoder decodes the current input voice waveform input from the input voice waveform encoder, taking into account whether it is able to output a voice waveform of speech content corresponding to the past input voice waveform based on the output voice waveform input from the output voice waveform encoder.

[0155] 10 Voice conversion device 12 Spectrum calculation unit 13a Voice conversion model 13b (Trained) voice conversion model 14 Training unit 15 Correspondence generation unit 16 Speech waveform generation unit 17 Termination determination unit 23a Voice conversion model 23b (Machine learned) voice conversion model 24 Training unit 25 Correspondence generation unit 27 Termination determination unit 131 Input spectrum encoder 132 Output spectrum encoder 133 Decoder 134 Transition probability prediction unit 135 Spectral prediction unit 141 Alignment calculation unit 142 Alignment mask unit 143 Spectral weighting unit 144 Spectral error calculation unit 145 Model optimization unit 231 Input speech waveform encoder 232 Output speech waveform encoder 233 Decoder 234 Transition probability prediction unit 235 Speech waveform prediction unit 241 Alignment calculation unit 242 Alignment mask unit 243 Voice waveform weighting unit 244 Voice waveform reconstruction error calculation unit 245 Model optimization unit

Claims

1. A voice conversion device that machine-learns a voice conversion model in a learning phase, wherein the voice conversion model has: an input spectrum encoder that inputs an input spectrum as the spectrum of a source speaker, encodes and outputs it; a decoder that decodes the input spectrum input from the input spectrum encoder and outputs a predicted spectrum of a target speaker; and an output spectrum encoder that inputs an output spectrum as the predicted spectrum of the target speaker output by the decoder, encodes and outputs it; and the decoder decodes the current input spectrum input from the input spectrum encoder, taking into consideration whether it has been able to output a spectrum of speech content corresponding to the past input spectrum based on the output spectrum input from the output spectrum encoder.

2. A voice conversion device as described in claim 1, comprising: a correspondence generation unit that receives as input a phoneme label with time information of a source speaker and a phoneme label with time information of a target speaker, and generates and outputs a ground truth alignment that indicates the correspondence between the phoneme label with time information of the source speaker and the phoneme label with time information of the target speaker; and a learning unit that uses the ground truth alignment to machine-train the voice conversion model so as to minimize the error between the measured spectrum of the target speaker and the ground truth spectrum of the target speaker.

3. The voice quality conversion device of claim 2, wherein the correspondence generation unit generates the correct alignment by considering a frame related to the correspondence between the phoneme label with time information of the source speaker and the phoneme label with time information of the target speaker and a predetermined number of frames before and after the frame as correct when constructing the correspondence.

4. The voice conversion device according to claim 1, wherein the input spectrum encoder inputs and encodes a speaker vector of the source speaker together with the spectrum of the source speaker, and the decoder inputs and decodes a speaker vector of a target speaker together with the input spectrum from the input spectrum encoder and the output spectrum from the output spectrum encoder.

5. A voice conversion device that converts voice quality using a machine-learned voice conversion model in an inference phase, wherein the machine-learned voice conversion model has: an input spectrum encoder that inputs an input spectrum as a spectrum of a source speaker, encodes and outputs it; a decoder that decodes the input spectrum input from the input spectrum encoder and outputs a predicted spectrum of a target speaker; and an output spectrum encoder that inputs an output spectrum as the predicted spectrum of the target speaker output by the decoder, encodes and outputs it; and the decoder decodes the current input spectrum input from the input spectrum encoder, taking into consideration whether it has been able to output a spectrum of speech content corresponding to the past input spectrum based on the output spectrum input from the output spectrum encoder.

6. A machine learning method executed by a voice conversion device that machine-learns a voice conversion model in a learning phase, wherein the voice conversion model has: an input spectrum encoder that inputs an input spectrum as a spectrum of a source speaker, encodes and outputs it; a decoder that decodes the input spectrum input from the input spectrum encoder and outputs a predicted spectrum of a target speaker; and an output spectrum encoder that inputs an output spectrum as the predicted spectrum of the target speaker output by the decoder, encodes and outputs it; and the decoder decodes the current input spectrum input from the input spectrum encoder, taking into consideration whether it has been able to output a spectrum of speech content corresponding to the past input spectrum based on the output spectrum input from the output spectrum encoder.

7. A voice conversion method executed by a voice conversion device that converts voice quality using a machine-learned voice conversion model in an inference phase, wherein the machine-learned voice conversion model has: an input spectrum encoder that inputs an input spectrum as a spectrum of a source speaker, encodes and outputs it; a decoder that decodes the input spectrum input from the input spectrum encoder and outputs a predicted spectrum of a target speaker; and an output spectrum encoder that inputs an output spectrum as the predicted spectrum of the target speaker output by the decoder, encodes and outputs it; and the decoder decodes the current input spectrum input from the input spectrum encoder, taking into consideration whether it has been able to output a spectrum of speech content corresponding to the past input spectrum based on the output spectrum input from the output spectrum encoder.

8. A program for causing a computer to execute the method according to claim 6 or 7.

Citation Information

Patent Citations

  • Speech translation method and system using a multilingual text-to-speech synthesis model

    JP2022169714A

  • System and method for streaming end-to-end speech recognition with asynchronous decoder

    JP2023504219A

Cited By

  • Voice quality conversion learning device and voice quality conversion device

    WO2026190923A1