Conformer-based voice conversion model

Abstract

A method (600) for speech conversion includes receiving an input spectrogram (102) corresponding to an utterance (108) as an input to an encoder (210) of a speech conversion model (200), the encoder including a stack of self-attention blocks (400). The method further includes generating an encoded spectrogram (212) as an output from the encoder, and receiving the encoded spectrogram generated as an output from the encoder as an input to a spectrogram decoder (220) of the speech conversion model. The method further includes generating an output spectrogram (222) corresponding to a synthetic speech representation of the utterance as an output from the spectrogram decoder.

Description

Technical Field

[0001] The present disclosure relates to a conformer-based voice conversion model.

Background Art

[0002] A voice conversion model can be used to modify the voice of a source speaker into another form without changing the language information of the voice. For example, a voice conversion model can generate a transcript of a user's voice. Alternatively, a voice conversion model can convert a user's voice into a voice waveform of a voice in another language. Machine learning methods can be used to train a voice conversion model that accurately and efficiently converts voice into another form.

Summary of the Invention

Problems to be Solved by the Invention

[0003] Provide a conformer-based voice conversion model.

Means for Solving the Problems

[0004] One aspect of the present disclosure provides a voice conversion model including an encoder including a stack of self-attention blocks configured to encode an input spectrogram corresponding to an utterance. The voice conversion model further includes a spectrogram decoder configured to receive the encoded spectrogram from the encoder as an input. The spectrogram decoder is further configured to generate an output spectrogram corresponding to a synthetic voice representation of the utterance as an output.

[0005] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, an input spectrogram corresponding to an utterance is extracted from input speech spoken by a speaker associated with non-typical speech. In these implementations, the synthetic speech representation of the utterance includes a synthetic standard and fluent speech representation of the utterance. Further, the speech conversion model may include a word-piece decoder configured to receive an encoded spectrogram from an encoder as an input and generate a text representation corresponding to a transcription of the utterance as an output.

[0006] Further, the speech conversion model may include a phoneme decoder configured to receive an encoded spectrogram from the encoder as an input and generate a phoneme representation of the utterance as an output.

[0007] In some implementations, the stack of self-attention blocks includes a stack of conformer blocks, and each conformer block has a multi-head self-attention mechanism. In these implementations, the encoder may be further arranged in front of the stack of conformer blocks and further include a first subsampling layer configured to receive an input spectrogram, the first subsampling layer including a CNN (Convolutional Neural Network) layer, and then pooling is performed in the time direction to reduce the number of frames processed by the first conformer block in the stack of conformer blocks. Further, in these implementations, the encoder may include a second subsampling layer arranged between the first set of conformer blocks in the stack of conformer blocks and the last set of conformer blocks in the stack of conformer blocks, the second subsampling layer being configured to subsample the hidden representation output by the last conformer block in the first set of conformer blocks to reduce the number of frames processed by the last set of conformer blocks. In these implementations, the encoder may further include an upsampling layer arranged after the stack of conformer blocks, the upsampling layer including a single transposed CNN layer configured to upsample the hidden representation output by the last conformer block in the stack of conformer blocks, thereby increasing the number of frames processed by the mutual attention mechanism arranged between the encoder and the spectrogram decoder.

[0008] Furthermore, the voice conversion model may be trained using a two-step training process that includes a first training step of pre-training the voice conversion model on a plurality of spoken utterances by a typical speaker associated with a standard and fluent voice. Here, each spoken utterance is paired with a ground truth corresponding to the utterance and a synthesized standard and fluent voice representation. The two-step training process further includes a second training step of fine-tuning the parameters of the pre-trained voice conversion model based on a plurality of non-typical voice samples spoken by a speaker associated with non-typical voice.

[0009] In some implementations, the spectrogram decoder directly generates an output spectrogram from the encoded spectrogram without performing an intermediate text-to-voice conversion on the text representation corresponding to the transcription of the utterance.

[0010] Another aspect of the present disclosure provides a computer-implemented method for a voice conversion model. The computer-implemented method, when executed on data processing hardware, causes the data processing hardware to perform operations. The operations include receiving, as an input to an encoder of the voice conversion model, an input spectrogram corresponding to an utterance, the encoder including a stack of self-attention blocks. The operations further include generating, as an output from the encoder, an encoded spectrogram. The operations include receiving, as an input to a spectrogram decoder of the voice conversion model, the encoded spectrogram generated as an output from the encoder. The operations further include an output spectrogram generation step of generating, as an output from the spectrogram decoder, an output spectrogram corresponding to the synthesized voice representation of the utterance.

[0011] This aspect may include one or more of the following optional features. In some implementations, the input spectrogram corresponding to the utterance is extracted from the input speech spoken by a speaker associated with non-typical speech. In these implementations, the synthetic speech representation of the utterance includes the synthetic standard and fluent speech representation of the utterance.

[0012] In some implementations, the operation includes receiving, as an input to the word-piece decoder of the voice conversion model, the encoded spectrogram generated as an output from the encoder. These implementations further include generating, as an output from the word-piece decoder, a text representation corresponding to the transcription of the utterance. The operation may further include receiving, as an input to the phoneme decoder of the voice conversion model, the encoded spectrogram generated as an output from the encoder, and generating, as an output from the phoneme decoder, a phoneme representation of the utterance.

[0013] In some implementations, the stack of self-attention blocks includes a stack of conformer blocks, and each conformer block has a multi-head self-attention mechanism. In these implementations, the encoder may be further arranged in front of the stack of conformer blocks and may further include a first subsampling layer configured to receive an input spectrogram. The first subsampling layer includes a CNN (Convolutional Neural Network) layer, and then pooling is performed in the time direction to reduce the number of frames processed by the first conformer block in the stack of conformer blocks. Further, in these implementations, the encoder may include a second subsampling layer arranged between the first set of conformer blocks in the stack of conformer blocks and the last set of conformer blocks in the stack of conformer blocks. The second subsampling layer is configured to reduce the number of frames processed by the last set of conformer blocks by subsampling the hidden representation output by the last conformer block in the first set of conformer blocks. In these implementations, the encoder may further include an upsampling layer arranged after the stack of conformer blocks. The upsampling layer includes a single transposed CNN layer configured to upsample the hidden representation output by the last conformer block in the stack of conformer blocks, thereby increasing the number of frames processed by the mutual attention mechanism arranged between the encoder and the spectrogram decoder.

[0014] Furthermore, the voice conversion model may be trained using a two-step training process that includes a first training step of pre-training the voice conversion model on a plurality of spoken utterances by a typical speaker associated with a standard and fluent voice. Here, each spoken utterance is paired with a ground truth corresponding to the utterance and a synthesized standard and fluent voice representation. The two-step training process further includes a second training step of fine-tuning the parameters of the pre-trained voice conversion model based on a plurality of non-typical voice samples spoken by a speaker associated with non-typical voice.

[0015] In some implementations, the spectrogram decoder directly generates an output spectrogram from the encoded spectrogram without performing an intermediate text-to-voice conversion on the text representation corresponding to the transcription of the utterance.

[0016] Details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the following description. Other aspects, features, and advantages will become apparent from the description and drawings, as well as from the claims.

Brief Description of the Drawings

[0017]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

DETAILED DESCRIPTION

[0018] Like reference numerals in the various drawings indicate like elements. There is a growing interest in developing more comprehensive speech technologies, particularly speech technologies that can assist people with speech impairments. ASR (Automatic Speech Recognition) has made tremendous progress through the introduction of an E2E (end-to-end) deep learning-based model that recognizes speech from speakers with dysarthria or atypical speech patterns for conversion to an accurate transcription. For example, atypical speech patterns can include, but are not limited to, speech impaired due to physical or neurological conditions (e.g., a speaker with ALS (Amyotrophic Lateral Sclerosis) disease), speech with a strong accent, and speech with a hearing impairment. A voice conversion system can convert speech with an atypical speech pattern into a standard and fluent output voice by applying a similar deep learning-based model.

[0019] The matched training data distribution and test data distribution are known to provide optimal performance for training a voice conversion model. However, since the training data from speakers with speech disorders is insufficient, it can be difficult to train the model using current methods. Further, users with speech disorders may feel that it takes a great deal of effort to record enough data necessary to adequately train the model, and thus it is difficult to obtain such training data. The present disclosure provides an improvement to a voice conversion model having an encoder / decoder architecture. This improvement reduces the amount of training data required, facilitates the training of the voice conversion model, enables the model to scale to a large user population, and further enables handling of a wide range of non-typical voices. The present disclosure provides this improvement through an architectural change to the voice conversion model using subsampling of encoder activations and corresponding upsampling of typical encoder outputs. The present disclosure further provides a combination of many-to-one VC (voice conversion) and ASR in an integrated model that jointly decodes both voice and text during inference using a shared encoder architecture for the task.

[0020] As used herein, unless otherwise specified, the terms "speech conversion system" and "speech conversion model" refer to any combination of an ASR system / model that recognizes the input non - typical speech and converts it into the corresponding text (e.g., transcription) and / or a set of phonemes representing the non - typical speech, or a speech - to - speech conversion system / model that directly converts the input non - typical speech into a standard and fluent synthetic speech without performing speech recognition. In other words, the speech conversion system / model is configured to directly convert the input speech waveform or spectrogram corresponding to non - typical speech into an output speech waveform or spectrogram corresponding to standard and fluent speech without converting the input speech waveform into an intermediate representation (e.g., text or phonemes). As will become apparent, the speech conversion model, and the techniques for training the speech conversion model, enable users with non - typical speech to talk to and be understood by both other humans and speech interfaces (e.g., digital assistants) by enabling the recognition and / or reproduction of the user's intended speech. The examples herein show a speech conversion model that receives an input speech waveform or spectrogram corresponding to non - typical speech for conversion to an output speech waveform or spectrogram corresponding to standard and fluent speech, but the speech conversion model may be similarly adapted to perform other types of speech conversion tasks without departing from the scope of the present disclosure. For example, the speech conversion model may convert an input speech waveform or spectrogram corresponding to an utterance in a first language into an output speech waveform or spectrogram corresponding to a translation of the utterance in a different second language. The speech conversion model may similarly receive an utterance input from a user and output a synthetic speech that contains the same language content as the utterance input but has different speech characteristics of a target speaker.

[0021] FIG. 1 shows a voice conversion system 100 including a voice conversion model 200 and a vocoder 374. The voice conversion model 200 is configured to convert input voice data 102 corresponding to an utterance 108 spoken by a source speaker 104 associated with non-typical voice into output voice data 106 corresponding to a synthesized standard and fluent voice expression of the same utterance 114 spoken by a target speaker 104. As used herein, the input voice data 102 may include an input spectrogram corresponding to the utterance 108. As used herein, the output voice data 106 may include an output spectrogram 222 corresponding to a synthesized standard and fluent voice expression of the same utterance 114, or a time-domain voice waveform 376 converted from the output spectrogram 222 by the vocoder 375. Although not shown, an acoustic front end resident on the user device 110 may convert the time-domain voice waveform of the utterance 108 captured via the microphone of the user device 110 into the input spectrogram 102 or other types of voice data 102. In some implementations, the voice conversion model 200 converts the input voice data 102 corresponding to the utterance 108 into a text expression (e.g., grapheme, sub-word, or word) corresponding to a transcription 201 or phoneme representation 202 of the utterance 108. In some additional implementations, the voice conversion model 200 of the voice conversion system 100 is configured to directly convert the input voice data 102 (e.g., input spectrogram) into the output voice data 106 (e.g., output spectrogram 222) without performing voice recognition or requiring the generation of an intermediate discrete representation (e.g., text or phoneme) from the input voice data 102.

[0022] The voice conversion model 200 includes a spectrogram encoder 210 configured to encode an input spectrogram 102 into an encoded spectrogram 212 (e.g., a hidden feature representation including a series of vectors), and one or more decoders 220, 220a-c configured to decode the encoded spectrogram 212 into an output spectrogram 222 corresponding to a synthesized standard and fluent voice representation, a transcription 201, and / or a phoneme representation 202. The transcript 201 may include a standard and fluent transcription of the utterance 108 that can be understood by a human reader and / or by a downstream application (e.g., a digital assistant).

[0023] The encoder 210 may include a stack of multi-head attention blocks 400 (referred to herein as conformer blocks 400) that may include a conformer or a transformer. Each multi-head attention block 400 may include a multi-head attention mechanism 420 (FIG. 4). The conformer block 400 may be implemented by the encoder 210 to capture high-resolution spectral patterns (patters) of the input non-typical speech. For example, when the spectrogram encoder 210 receives the input speech data 102 of utterance 108, the spectrogram encoder 210 may use the conformer block 400 to process 10 milliseconds (ms) of speech samples of the input spectrogram 102 to generate an upsampled 40 ms encoded spectrogram 212. The upsampling process by the conformer block 400 of the encoder 210 will be described in more detail with reference to FIGS. 3 and 4 below. Next, the spectrogram decoder 220a may generate an output spectrogram 222 corresponding to a synthesized standard and fluent speech representation based on the upsampled encoded spectrogram 212 output from the spectrogram encoder 210. For example, the spectrogram decoder 220a may receive the upsampled 40 ms encoded spectrogram 212 representing 10 ms of speech samples of the input spectrogram 102 from the spectrogram encoder 210. Here, through the mutual attention mechanisms 231, 231a (FIGS. 2 and 3), the spectrogram decoder 220a may generate a 12.5 ms output spectrogram 222 corresponding to a synthesized standard and fluent speech representation of utterance 114 that includes words or parts of words intended as the 10 ms input speech data 102 but does not include the non-fluency of the non-typical speech.

[0024] In some examples, the voice conversion model 200 also includes a word-piece decoder 220b that decodes the encoded spectrogram 212 into a text representation, such as a transcription 201. For example, the word-piece decoder 220b may be trained to decode the encoded spectrogram 212 into corresponding word pieces that can form the transcription 201. In the illustrated example, the word-piece decoder 220b is used by the model 200, but the model 200 may alternatively use a grapheme decoder 220b or a word decoder 220b configured to decode the encoded spectrogram into graphemes or words. Additionally or alternatively, the voice conversion model 200 may include a phoneme decoder 220c that decodes the encoded spectrogram 212 into a phoneme representation 202 that includes phonemes indicative of a synthesized standard and fluent voice representation of the utterance 114. Thus, the spectrogram, word-piece, and phoneme decoders 220a-c may correspond to parallel decoding branches of the voice conversion model 200. Each of the decoding branches receives the upsampled encoded spectrogram 212 encoded by the spectrogram encoder 210 and outputs the corresponding one of the output spectrogram 222, the transcription 201, and the phoneme representation 202 in parallel. The vocoder 375 (also referred to as the synthesizer 375) of the voice conversion system 100 is configured to convert the output spectrogram 222 output by the spectrogram decoder 220a into a time-domain waveform 376 of synthesized standard and fluent voice of the same utterance 114 for audible output from another computing device 116. The time-domain voice waveform includes a voice waveform that defines the amplitude of the voice signal over time. The vocoder 375 may include a unit selection module or a WaveNet module for synthesizing the output spectrogram 222 into a time-domain waveform of standard and fluent voice. In some implementations, the synthesizer 375 includes a vocoder network, i.e., a neural vocoder that is separately trained and tuned on a mel-frequency spectrogram for conversion into a time-domain voice waveform.In some additional examples, the vocoder 375 includes a streaming vocoder 375, such as a streaming Griffin-Lim vocoder. An exemplary streaming vocoder is described in U.S. Provisional Application No. 63 / 312,195, filed Feb. 21, 2022, the entire contents of which are incorporated by reference.

[0025] In the example shown, the source speaker 104 is associated with non-typical speech, such as where the source speaker 104 speaks with a non-typical speech pattern that may be difficult to understand. Non-typical speech patterns can include, but are not limited to, speech impaired due to a physical or neurological condition (e.g., a speaker having ALS (amyotrophic lateral sclerosis) disease), speech with a strong accent, and speech with a hearing impairment. As an example, the source speaker 104 has ALS disease and is associated with non-typical speech due to the ALS disease. Thus, the voice conversion model 200 is trained to directly convert the input spectrogram 102 corresponding to the utterance 108 spoken by the source speaker 104 associated with ALS speech into an output spectrogram 222 corresponding to a synthesized standard and fluent speech representation of the same utterance 108. Thus, the synthesized standard and fluent speech representation provided by the output spectrogram 222 improves the intelligibility of the ALS speech spoken by the source speaker 104. Without departing from the scope of the present disclosure, the voice conversion model 200 may be trained as a multilingual voice conversion model that directly converts the input spectrogram 102 corresponding to the utterance 108 in a first language into an output spectrogram 222 corresponding to a synthesized speech representation of the utterance 108 in a second language that has the same voice as the source speaker but is different. Further, the model 20 may be trained to directly convert the input spectrogram 102 corresponding to the utterance 108 spoken by a source speaker having a first voice characteristic into an output spectrogram 222 corresponding to a synthesized speech representation of the same utterance 108 having a different voice characteristic corresponding to a target speaker.

[0026] The computing device 110 associated with the source speaker 104 captures the utterance 108 spoken by the source speaker 104 and may provide the corresponding input audio data 102 to the voice-to-voice conversion system 100 for conversion to either an output spectrogram 222, a transcription 201, or a phoneme representation 202. The computing device 110 may include, but is not limited to, a smartphone, a tablet, a desktop / laptop computer, a smart speaker, a smart display, a smart home appliance, an assistant-compatible wearable device (e.g., a smartwatch, smart headphones, smart glasses, etc.), or a vehicle infotainment system. Thereafter, the voice conversion system 100 may convert the output spectrogram 222 to a time-domain audio waveform 376 by using a vocoder 375. The time-domain audio waveform 376 may be audibly output from the computing device 110 or another computing device 116 as a synthesized, standard, and fluent speech utterance 114. Without departing from the scope of the present disclosure, the voice conversion system 100 may also provide a transcription 201 and / or a phoneme representation 202 corresponding to the synthesized, standard, and fluent speech representation of the same utterance 114 spoken by the source speaker 104 to another computing device 116 associated with the user 118. Thereby, the other computing device 116 may display the standard transcription 201 as an understandable representation of the utterance 108 spoken by the source speaker 104, and / or may use a TTS (text-to-speech) system that converts the transcription 201 or the phoneme representation 202 to synthesized, standard, and fluent speech. In this example, the source speaker 104 and the user 118 are having a conversation through their respective computing devices 110, 116 via a telephone or another type of voice communication protocol, e.g., Voice over Internet Protocol.Source speaker 104 and other users 118 may speak the same language, but since source speaker 104 has a medical condition (e.g., atypical speech), a strong accent, or atypical speech due to a different native language, it may be difficult for other users 118 to understand source speaker 104. Thus, source speaker 104 speaks in atypical speech (e.g., ALS speech) that may be difficult to understand, while other users 118 who hear a synthesized standard and fluent speech representation have an easier time understanding utterance 108 intended by source speaker 104. Put another way, the synthesized standard and fluent speech representation provides a more consistent rhythm that may be easier for another user to understand than the original utterance 108 spoken by the target speaker in atypical speech. In particular, the synthesized standard and fluent speech representation is spoken in the voice of source speaker 104. However, depending on the application, voice conversion system 100 may generate the synthesized standard and fluent speech in the voice of a target speaker having voice characteristics different from the source speaker.

[0027] In some additional examples, voice conversion system 100 sends output audio data 106 corresponding to the synthesized standard and fluent speech representation of the utterance spoken by source speaker 104 to an output audio device for audibly outputting the synthesized standard and fluent speech representation to the listener in the voice of source speaker 104. For example, source speaker 104 may be a psychology professor giving a lecture to students in a class, and the utterance spoken by source speaker 104 may include medical terms belonging to a particular domain, e.g., psychology. As will become apparent, voice-to-voice conversion model 200 is trained to learn the linguistic diversity associated with a particular domain as well as the acoustic diversity associated with a particular type of atypical speech associated with source speaker 104.

[0028] Alternatively, another computing device 116 may be associated with a downstream ASR system that functions as a front end for providing output audio data 106 corresponding to a synthesized standard and fluent speech representation by the speech conversion system 100 as an input to an ASR (Automatic Speech Recognition) system for conversion to recognized text. The recognized text may be presented to another user 118 and / or provided to an NLU (Natural Language Understanding) system for further processing. The functionality of the speech conversion system 100 may reside on any one or both of the remote server 112, computing devices 110, 116, or any combination of the remote server and computing devices 110, 116. The speech conversion system 100 may be distributed among multiple devices such that the speech conversion model 200 resides on one of the computing device 110 or the remote server 112, and the vocoder 375 resides on one of the remote server 112 or another computing device 116. In some implementations, the speech conversion model 200 continuously generates an output spectrogram 222 corresponding to a synthesized standard and fluent speech representation of the utterance when the source speaker 104 speaks the portion corresponding to the utterance as non-typical speech. By continuously generating the output spectrogram 222 corresponding to the synthesized standard and fluent speech representation of the portion of the utterance 108 spoken by the source speaker 104, the conversation between the source speaker 104 and the user 118 (or listener) can be paced more naturally. In some additional implementations, the speech conversion model 200 waits to determine / detect when the source speaker 104 stops speaking by using techniques such as voice activity detection, end point, end of query detection, etc. before converting the input audio data 102 corresponding to the utterance 108 with non-typical speech to the output spectrogram 222 corresponding to the synthesized standard and fluent speech representation of the same utterance 114.

[0029] FIG. 2 shows a schematic diagram of an exemplary voice conversion model 200 used by the voice conversion system 100 of FIG. 1. The voice conversion model 200 includes an encoder 210 and one or more decoders 220, 220a - c. The encoder 210 is configured to encode input voice data 102 into an encoded spectrogram 212. Here, the input voice data 102 includes a sequence of input spectrograms corresponding to utterances 108 spoken by the source speaker 104. In some embodiments, the encoder 210 includes a stack of conformable blocks 400. In these implementations, the encoder subsamples the input voice data 102 using convolutional layers and then processes the input voice data 102 using a stack of conformable blocks 400. Each conformable block 400 may include a feed - forward layer, a self - attention layer, a convolutional layer, and a second feed - forward layer. In some examples, the stack of conformable blocks 400 includes 17 conformable blocks each having 512 states, 8 attention heads, and a convolutional kernel size of 32×1. FIG. 4 is a schematic diagram of an exemplary conformable block. The encoder 210 may alternatively use a stack of transformer blocks or lightweight convolutional blocks instead of conformable blocks.

[0030] The spectrogram, phoneme, and word-piece decoders 220, 220a-c may each include a recurrent neural network-based architecture that receives the shared encoded spectrogram 212 output by the encoder 210. The spectrogram decoder 220a may include mutual attention mechanisms 231, 231a (also shown in FIG. 3) configured to receive the shared encoded spectrogram 212 from the encoder 210. The spectrogram decoder 220a may further process the shared encoded spectrogram 212 by using a plurality of LSTM (long short-term memory) layers 233, 233a and a plurality of convolutional layers 235. For example, the spectrogram decoder 220a may include five LSTM layers 233a and five transformation layers 235. The spectrogram decoder 220a may generate an output spectrogram 222. In some implementations, the spectrogram decoder 220a may directly generate the output spectrogram 222 from the encoded spectrogram 212 without performing intermediate text-to-speech conversion on the text representation corresponding to the utterance transcription.

[0031] In the illustrated example, the word-piece decoder 220b includes corresponding mutual attention mechanisms 231, 231b configured to receive the shared encoded spectrogram from the encoder, followed by two LSTM (long short-term memory) layers 233, 233b, and Softmax layers 245, 245a that output a text representation 201 corresponding to the utterance transcription.

[0032] Similar to the word-piece decoder 220b, the phoneme decoder 220c may include mutual attention mechanisms 231, 231c configured to receive the shared encoded spectrogram 212 from the encoder 210, followed by two LSTM (long short-term memory) layers 233, 233c, and Softmax layers 245, 245b that output a phoneme representation of the utterance 202.

[0033] FIG. 3 shows a schematic diagram 300 of an exemplary mixed frame rate processing scheme for improving the training time and inference time of the speech conversion model 200 of FIG. 1. The mixed frame rate processing scheme can improve the memory consumption and training speed of the encoder 220 in speech-to-speech processing (i.e., generating the output spectrogram 222 via the spectrogram decoder 220a). Different from other models such as ASR (Automatic Speech Recognition) or TTS (Text-to-Speech) where the predicted target or input sequence is text, the speech-to-speech conversion model uses acoustic frames as the input sequence while also outputting a sequence of acoustic frames. Since the number of output acoustic frames is much larger than the number of output text sequences, speech-to-speech conversion requires more calculations compared to ASR or TTS models. In some cases, the model complexity becomes a quadratic function based on the number of input frames due to the self-attention mechanism of the encoder 210. Furthermore, since the memory usage can be directly proportional to the length of the acoustic sequence, as a result, the batch size may become small and the training speed may become slow. The mixed frame rate processing scheme shown in FIG. 3 can significantly reduce the number of calculations and then improve the training.

[0034] In some implementations, the mixed frame rate processing scheme uses convolutional subsampling with a 3×3 kernel size and a 2×2 stride, resulting in a subsampling factor of 4. In these implementations, the transposed convolutional network includes one CNN (Convolutional Neural Network) layer with 512 channels, a filter size of 4, and a stride of 2 in the time direction. Further, the mixed frame rate scheme may include extracting 128-dimensional log mel spectrogram features from the input speech by using a 30 ms window and a 10 ms frame shift. The features may be provided to the encoder 210. In an exemplary implementation, the target of the spectrogram decoder 220a includes a 1025-dimensional STFT (Short-Time Fourier Transform) amplitude calculated using a frame length of 50 ms, a shift of 12.5 ms, and a 2048-point FFT.

[0035] The processing method may start when the spectrogram encoder 210 receives a 10 ms (millisecond) audio sample of the input spectrogram 102. The encoder 210 may first process the 10 ms audio sample using a first subsampling layer 305 that includes a plurality of CNN layers. When performing subsampling by the first subsampling layer 305, CNN layers are used, and then pooling is performed in the time direction to reduce the number of frames processed by the first conformable block in the stack of conformable blocks 400, 400a - b. The CNN may subsample 10 ms of audio into a 40 ms representation, which is then provided to the first set of conformable blocks 400a. The first set of conformable blocks 400a may provide it to the second subsampling layer 315 after processing the 40 ms representation. The second subsampling layer 315 may be arranged between the first set of conformable blocks 400a and the last set of conformable blocks 400b. In some examples, the first set of conformable blocks 400a may include 4 conformable blocks and the last set of conformable blocks 400b may include 13 conformable blocks such that the total number of conformable blocks in the encoder 210 is 17. Here, the second subsampling layer 315 may be configured to reduce the number of frames processed by the last set of conformable blocks 400b by subsampling the hidden representation 308 output by the last conformable block in the first set of conformable blocks 400a. For example, the second subsampling layer 315 may be configured to subsample the 40 ms hidden representation 308 output by the first set of conformable blocks 400a into a corresponding 80 ms representation 318. At the end of the last conformable block of the last set of conformable blocks 400b, the encoder 210 upsamples the 80 ms hidden representation 322 using an upsampling layer 325.The upsampling layer 325 may include a single transposed CNN layer configured to increase the number of frames of the encoded spectrogram 212 by upsampling the 80 ms hidden representation 322 output by the last conformable block of the last set of conformable blocks 400b to the corresponding 40 ms representation of the encoded spectrogram 212.

[0036] The encoded spectrogram 212 may be received by a mutual attention mechanism 231a disposed between the encoder 210 and the spectrogram decoder 220a. In some implementations, the mutual attention mechanism 231a is included in the spectrogram decoder 220a. The spectrogram decoder 220a may use the mutual attention mechanism 231a to reduce the 40 ms representation of the encoded spectrogram 212 to a 25 ms representation. The 25 ms representation may then be provided to the LSTM 233a. The output of the LSTM 233a may be reduced by a reduction factor 335. The spectrogram decoder 220a may output the resulting output spectrogram 222 at a final size of 12.5 ms. The output spectrogram 222 may be provided to a vocoder 375 (FIG. 1) for conversion to the corresponding time-domain audio waveform of the synthesized speech.

[0037] The above examples are not intended to be limiting. The encoder 210 may receive an audio sample of any suitable length for processing. The encoder 210 may then generate an encoded spectrogram 212 that may have any suitable length by processing, subsampling, or upsampling the audio sample. Similarly, the decoder 220a may generate an output spectrogram 222 of a suitable length by processing the encoded spectrogram 212.

[0038] In the experiment, when the same encoder frame shift is given, the hybrid frame rate method allows different realizations with different subsampling and upsampling settings. For example, an increase in subsampling generally results in improved training but causes a regression of the spectrogram WER that is more difficult to recover through upsampling. The information loss may be evaluated based on the sparsity of the feedforward neural network weight matrix in the last conformable block of the last set of conformable blocks of the encoder 210. The CPV (cumulative variance ratio) may be calculated by the following formula.

[0039]

Equation

[0040] Here, s i is the i-th singular value of the matrix, k is the number of singular values to consider, and D is the size of the feedforward matrix (D = 512). For any given k, a larger CPV indicates that the network can learn the structure of data with a sparsity index of k. A small value of k indicates a sparse matrix structure.

[0041] FIG. 4 shows an example of a conformer block 400 in a stack of conformer layers of the encoder 210. The conformer block 400 includes a first half feed-forward layer 410, a second half feed-forward layer 440, a multi-head self-attention block 420 and a convolutional layer 430 arranged between the first half feed-forward layer 410 and the second half feed-forward layer 440, and a concatenation operator 405. The first half feed-forward layer 410 processes input audio data 102 including an input mel spectrogram sequence. Subsequently, the multi-head self-attention block 420 receives the input audio data 102 concatenated with the output of the first half feed-forward layer 410. Briefly speaking, the role of the multi-head self-attention block 420 is to separately summarize the context of noise for each input frame to be emphasized. The convolutional layer 430 subsamples the output of the multi-head self-attention block 420 concatenated with the output of the first half feed-forward layer 410. Then, the second half feed-forward layer 440 receives the concatenation of the output of the convolutional layer 430 and the multi-head self-attention block 420. The layernorm module 450 processes the output from the second half feed-forward layer 440. Mathematically, the conformer block 400 generates an output feature y as follows by transforming the input feature x using the modulation feature m.

[0042]

Number

[0043] FIG. 5 shows a training process 500 for the voice conversion model 200. In some implementations, the process 500 employs a two-step training technique. First, the voice conversion model 200 is pre-trained on typical voices from a large pool of speakers to obtain a many-to-one voice conversion model 200, which consequently becomes an ASR / conversion-based model independent of the speaker. The target voice used for training may be a voice synthesized from a given speech-based reference transcript reflecting typical voices. For individualization, any parameters of the basic model may be fine-tuned to the voice from a single input speaker (e.g., a speaker with hearing impairment) to obtain a one-to-one voice conversion model (and speaker-dependent ASR) model from an atypical voice to a typical voice.

[0044] Referring to FIG. 5, the process 500 first pre-trains the voice conversion model 200 using pre-training data 505. Pre-training of the model is a technique used to initialize the model. The model may then be further fine-tuned based on additional training data 510. With respect to the voice conversion model 200, pre-training may include starting the voice conversion model 200 with pre-training data 505 that includes a plurality of spoken utterances by typical speakers associated with standard and fluent voices. The pre-training data 505 may further include ground truths corresponding to the spoken utterances, the spoken utterances paired with synthesized standard and fluent voice representations.

[0045] Next, process 500 may fine-tune the parameters of the pre-trained voice conversion model 200 for non-typical voices. The training process may include training either the encoder 210 or the decoder 220, 220a - c separately or jointly in any suitable combination. Process 500 includes supplying a training input 510 to the voice conversion model 200. In some implementations, the training input 510 includes a plurality of non-typical voice samples spoken by one or more speakers associated with the non-typical voice. Further, the training input 510 may be labeled using a label 520 indicating a target output associated with the training input 510. Upon receiving the training input 510, the voice conversion model 200 may generate an output 515 (e.g., transcript 201, phoneme representation 202, output spectrogram 222). The voice conversion model 200 may process the training input 510 in the manner described with respect to any of FIGS. 2 - 4 or in any other suitable manner for voice conversion.

[0046] In some implementations, the output 515 is used by a loss function 530 to generate a loss 540. That is, the loss function 530 generates the loss 540 by comparing the output 515 and the label 520. The loss 540 indicates a mismatch between the label 520 (i.e., the target output) and the output 515. The loss function 350 may implement any suitable technique for determining a loss such as regression loss, mean squared error, mean squared logarithmic error, mean absolute error, binary classification, binary cross-entropy, hinge loss, multi-class loss, etc. The loss 540 may then be supplied directly to the voice conversion model 200. Here, the voice conversion model 200 processes the loss 540 and adjusts one or more parameters of the voice conversion model 200 to compensate for the loss 540.

[0047] FIG. 6 is a flowchart of an exemplary configuration of the operations of a method 600 that a computer performs to execute voice conversion. Method 600 may be executed by various elements of the exemplary voice conversion system 100 of FIG. 1, for example. In operation 610, the method includes receiving an input spectrogram 102 corresponding to utterance 108 as an input to an encoder 210 of a voice conversion model 200. The encoder 210 includes a stack of self-attention blocks 400. In operation 620, method 600 includes generating an encoded spectrogram 212 as an output from the encoder 210. In operation 630, method 600 includes receiving the encoded spectrogram 212 generated as an output from the encoder 210 as an input to a spectrogram decoder 200a of a voice conversion model 220. In operation 640, method 600 includes generating an output spectrogram 222 corresponding to a synthesized standard and fluent voice representation of the same utterance 114 as an output from the spectrogram decoder 220a.

[0048] A software application (i.e., a software resource) can refer to computer software that causes a computing device to perform a task. In some examples, a software application may be referred to as an “application,” an “app,” or a “program.” Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and game applications.

[0049] A non-transitory memory may be a physical device used to temporarily or persistently store a program (e.g., a sequence of instructions) or data (e.g., program state information) for use by a computing device. The non-transitory memory may be a volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and ROM (Read-Only Memory) / PROM (Programmable Read-Only Memory) / EPROM (Erasable Programmable Read-Only Memory) / EEPROM (Electrically Erasable Programmable Read-Only Memory) (typically used for firmware such as a boot program). Examples of volatile memory include, but are not limited to, RAM (Random Access Memory), DRAM (Dynamic Random Access Memory), SRAM (Static Random Access Memory), PCM (Phase Change Memory), and disks or tapes.

[0050] FIG. 7 is a schematic diagram of an exemplary computing device 700 that may be used to implement the systems and methods described herein. Computing device 700 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown herein, their connections and relationships, and their functions are exemplary only and are not intended to limit implementations of the invention described and / or claimed herein.

[0051] The computing device 700 includes a processor 710, a memory 720, a storage device 730, a high-speed interface / controller 740 connected to the memory 720 and the high-speed expansion port 750, and a low-speed interface / controller 760 connected to the low-speed bus 770 and the storage device 730. Each of the components 710, 720, 730, 740, 750, and 760 is interconnected using various buses and can be implemented on a common motherboard or in other ways as needed. The processor 710 can process instructions for execution within the computing device 700, including instructions stored in the memory 720 or the storage device 730 to display graphical information for a GUI (Graphical User Interface) on an external input / output device such as a display 780 connected to the high-speed interface 740. In other implementations, multiple memories and memory types, as well as multiple processors and / or multiple buses, may be used as needed. Also, multiple computing devices 700 may be connected, and each device may provide a portion of the required operations, for example, as a server bank, a group of blade servers, or a multiprocessor system. The processor 710 may be referred to as data processing hardware 710 that resides on any one or both of the remote server 112, the computing devices 110, 116, or any combination of the remote server and the computing devices 110, 116. The memory 710 may be referred to as memory hardware 720 that resides on any one or both of the remote server 112, the computing devices 110, 116, or any combination of the remote server and the computing devices 110, 116.

[0052] Memory 720 stores information non - transiently within computing device 700. Memory 720 may be a computer - readable medium, a volatile memory unit(s), or a non - volatile memory unit(s). The non - transient memory 720 may be a physical device used to store a program (e.g., a sequence of instructions) or data (e.g., program state information) temporarily or persistently for use by computing device 700. Examples of non - volatile memory include, but are not limited to, flash memory and ROM (Read - Only Memory) / PROM (Programmable Read - Only Memory) / EPROM (Erasable Programmable Read - Only Memory) / EEPROM (Electrically Erasable Programmable Read - Only Memory) (e.g., typically used for firmware such as a boot program). Examples of volatile memory include, but are not limited to, RAM (Random Access Memory), DRAM (Dynamic Random Access Memory), SRAM (Static Random Access Memory), PCM (Phase - Change Memory), and disk or tape.

[0053] Storage device 730 can provide mass storage to computing device 700. In some implementations, storage device 730 is a computer - readable medium. In various different implementations, storage device 730 may be an array of devices including a floppy (registered trademark) disk device, a hard disk device, an optical disk device, a tape device, a flash memory or other similar solid - state memory device, or a device in a storage area network or other configuration. In additional implementations, a computer program product is tangibly embodied as an information carrier. The computer program product includes instructions that, when executed, perform one or more of the methods as described above. The information carrier is a computer - readable medium or a machine - readable medium such as memory 720, storage device 730, or memory on processor 710.

[0054] The high-speed controller 740 manages bandwidth-intensive operations for the computing device 700, and the low-speed controller 760 manages lower-bandwidth-intensive operations. Such duty assignments are merely illustrative. In some implementations, the high-speed controller 740 is connected to a high-speed expansion port 750 that can accept a memory 720, a display 780 (e.g., via a graphics processor or accelerator), and various expansion cards (not shown). In some implementations, the low-speed controller 760 is connected to a storage device 730 and a low-speed expansion port 790. The low-speed expansion port 790, which may include various communication ports (e.g., USB, Bluetooth®, Ethernet®, wireless Ethernet), may be connected to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a network device such as a switch or router, for example, through a network adapter.

[0055] As shown in FIG. 7, the computing device 700 may be implemented in a plurality of different forms. For example, the computing device 700 may be implemented as a laptop computer 700b, as part of a rack server system 700c, or as a standard server 700a or multiple times in a group of such servers 700a.

[0056] Various implementations of the systems and techniques described herein may be realized in digital electronic circuits and / or optical circuits, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include an implementation in one or more computer programs executable and / or translatable on a programmable system including one or more programmable processors connected to transmit and receive data and instructions to and from a memory device, one or more input devices, and one or more output devices.

[0057] These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, device, and / or apparatus (e.g., magnetic disk, optical disk, memory, PLD (programmable logic device)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives the machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0058] The processes and logical flows described in this specification can be performed by one or more programmable processors, also referred to as data processing hardware, that execute one or more computer programs to operate on input data and generate output. The processes and logical flows can also be performed by special purpose logic circuitry, such as an FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit). Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. In general, a processor receives instructions and data from a read only memory or a random access memory or both. Essential elements of a computer are a processor that executes instructions and one or more memory devices for storing instructions and data. In general, a computer also includes, or is operatively coupled to receive data from, or transfer data to, or both, one or more mass storage devices for storing data, such as magnetic, magneto - optical disks, or optical disks. However, a computer need not have such devices. Computer - readable media suitable for storing computer program instructions and data include all forms of non - volatile memory, media, and memory devices, including by way of example semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices, magnetic disks, such as internal hard disks or removable disks, magneto - optical disks, and CDROM and DVD - ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0059] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube), an LCD (liquid crystal display) monitor, or a touch screen, and optionally a keyboard and a pointing device, such as a mouse or a trackball, by which the user can provide input to the computer. Other types of devices may be used to provide interaction with the user. For example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and the input from the user may be captured in any form including acoustic input, voice input, or tactile input. Further, the computer can interact with the user by sending a document to a device used by the user and receiving the document from the device, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0060] Although a number of implementations have been described, it will be understood that various changes may be made without departing from the spirit and scope of the present disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A voice conversion model (200) for operating a computer, comprising: An encoder (210) including a stack of self-attention blocks (400), the encoder (210) being configured to encode an input spectrogram (102) corresponding to an utterance (108); A spectrogram decoder (220a), Receiving the encoded spectrogram (212) from the encoder (210) as an input, A spectrogram decoder (220a) configured to generate an output spectrogram (222) corresponding to the synthetic speech representation of the utterance (108) as an output, The stack of self-attention blocks (400) includes a stack of conformer blocks (400), and each conformer block includes a multi-head self-attention mechanism (420), The encoder (210) is arranged in front of the stack of conformer blocks (400) and further includes a first subsampling layer (305) configured to receive the input spectrogram (102). The first subsampling layer (305) includes a CNN (Convolutional Neural Network) layer, and pooling is performed in the time direction after the CNN layer, thereby reducing the number of frames processed by the first conformer block in the stack of conformer blocks (400). Voice conversion model (200).

2. The input spectrogram (102) corresponding to the utterance (108) is extracted from input speech spoken by a speaker (104) associated with non-typical speech, The synthetic speech representation of the utterance (108) includes the synthesized standard and fluent speech representation of the utterance. The voice conversion model (200) according to claim 1.

3. A word-piece decoder (220b), Receiving the encoded spectrogram (212) from the encoder (210) as an input, The voice conversion model (200) according to claim 1 or 2, further comprising a word-piece decoder (220b) configured to generate a text representation corresponding to the transcription (201) of the utterance (108) as an output.

4. A phoneme decoder (220c), Receiving the encoded spectrogram (212) from the encoder (210) as an input, The speech conversion model (200) according to any one of claims 1 to 3, further comprising a phoneme decoder (220c) configured to generate the phoneme representation (202) of the speech (108) as an output.

5. The encoder (210) further includes a second subsampling layer (315) disposed between a first set of conformable blocks (400a) within the stack of the conformable blocks (400) and a last set of conformable blocks (400b) within the stack of the conformable blocks (400), and the second subsampling layer (315) subsamples the hidden representation output by the last conformable block within the first set of conformable blocks (400a) to reduce the number of frames processed by the last set of conformable blocks (400b). The speech conversion model (200) according to any one of claims 1 to 4, which is configured as such.

6. The encoder (210) further includes an upsampling layer (325) disposed after the stack of the conformable blocks (400), and the upsampling layer (325) includes a single transposed CNN layer configured to upsample the hidden representation output by the last conformable block within the stack of the conformable blocks (400), thereby increasing the number of frames processed by an inter-attention mechanism (231a) disposed between the encoder (210) and the spectrogram decoder (220a). The speech conversion model (200) according to claim 5.

7. The spectrogram decoder (220a) directly generates the output spectrogram (222) from the encoded spectrogram (212) without performing intermediate text-to-speech conversion on the text representation corresponding to the transcription (201) of the speech (108). The speech conversion model (200) according to any one of claims 1 to 6.

8. A method (600) implemented by a computer that, when executed on data processing hardware (710), causes the data processing hardware (710) to perform operations, the operations being A step of receiving an input spectrogram (102) corresponding to an utterance (108) as an input to an encoder (210) of an acoustic conversion model (200), wherein the encoder (210) includes a stack of self-attention blocks (400), the step and, A step of generating an encoded spectrogram (212) as an output from the encoder (210); A step of receiving, as an input to a spectrogram decoder (200a) of the acoustic conversion model (200), the encoded spectrogram (212) generated as an output from the encoder (210); An output spectrogram generation step of generating an output spectrogram (222) corresponding to a synthetic speech representation of the utterance (108) as an output from the spectrogram decoder (220a), and including, The stack of self-attention blocks (400) of the encoder (210) includes a stack of conformer blocks (400), and each conformer block includes a multi-head self-attention mechanism (420), The encoder (210) is arranged in front of the stack of the conformer blocks (400) and further includes a first subsampling layer (305) configured to receive the input spectrogram (102), and the first subsampling layer (305) includes a CNN (Convolutional Neural Network) layer, and pooling is performed in the time direction after the CNN layer, thereby reducing the number of frames processed by the first conformer block in the stack of the conformer blocks (400). A method (600) implemented by a computer.

9. The input spectrogram (102) corresponding to the utterance (108) is extracted from input speech spoken by a speaker (104) associated with non-typical speech, The method (600) according to claim 8, wherein the synthetic speech representation of the utterance includes a synthetic standard and fluent speech representation of the utterance.

10. The operation is A step of receiving, as an input to a word-piece decoder (220b) of the acoustic conversion model (200), the encoded spectrogram (212) generated as an output from the encoder (210); The method (600) according to claim 8 or 9, further comprising the step of generating a text representation corresponding to the transcription (201) of the utterance (108) as an output from the word piece decoder (220b).

11. The operation is receiving, as an input to the phoneme decoder (220c) of the voice conversion model (200), the encoded spectrogram (212) generated as an output from the encoder (210); The method (600) according to any one of claims 8 to 10, further comprising the step of generating a phoneme representation (2020) of the utterance (108) as an output from the phoneme decoder (220c).

12. The encoder (210) further includes a second subsampling layer (315) disposed between a first set of conformable blocks (400a) within the stack of the conformable block (400) and a last set of conformable blocks (400b) within the stack of the conformable block (400), and the second subsampling layer (315) subsamples the hidden representation output by the last conformable block within the first set of conformable blocks (400a) to reduce the number of frames processed by the last set of conformable blocks (400b). The method (600) according to any one of claims 8 to 11, configured as such.

13. The encoder (210) further includes an upsampling layer (325) disposed after the stack of the conformable block (400), and the upsampling layer (325) includes a single transposed CNN layer configured to upsample the hidden representation output by the last conformable block within the stack of the conformable block (400), thereby increasing the number of frames processed by the mutual attention mechanism (231a) disposed between the encoder (210) and the spectrogram decoder (220a). The method (600) according to claim 12.

14. A method (600) implemented by a computer that, when executed on data processing hardware (710), causes the data processing hardware (710) to perform an operation, the operation being Receiving, as an input to an encoder (210) of a voice conversion model (200), an input spectrogram (102) corresponding to an utterance (108), wherein the encoder (210) includes a stack of self-attention blocks (400), the step of, Generating, as an output from the encoder (210), an encoded spectrogram (212), the step of, Receiving, as an input to a spectrogram decoder (200a) of the voice conversion model (200), the encoded spectrogram (212) generated as an output from the encoder (210), the step of, Generating, as an output from the spectrogram decoder (220a), an output spectrogram (222) corresponding to a synthetic voice representation of the utterance (108), the output spectrogram generation step, including, The voice conversion model (200) is trained using a two-step training process (500), the two-step training process including, A first training step of pre-training the voice conversion model (200) on a plurality of spoken utterances by a typical speaker associated with standard and fluent speech, each spoken utterance being paired with a ground truth corresponding to the utterance, which is a synthetic standard and fluent voice representation, the first training step of, A second training step of fine-tuning the parameters of the pre-trained voice conversion model based on a plurality of non-typical voice samples spoken by a speaker associated with non-typical voice, the method (600).

15. The output spectrogram generation step includes directly generating the output spectrogram (222) from the input spectrogram (212) without performing an intermediate text-to-voice conversion on a text representation corresponding to a transcription (201) of the utterance (108), the method (600) according to any one of claims 8 to 14.

Images