Speech conversion model based on convolutional enhancement transform neural network
By improving the encoder/decoder structure and using a two-step training process, the problem of insufficient training data for atypical speech in speech conversion models is solved, and efficient conversion of atypical speech into standard and fluent speech is achieved, which is applicable to various speech impairment situations.
Patent Information
- Application Number
- CN202280033462.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2022-02-21
- Filing Date
- 2022-03-16
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2042-03-16
AI Technical Summary
Existing technologies struggle to effectively train speech conversion models, especially for speakers with speech impairments. The lack of sufficient training data leads to inadequate model performance, making it difficult to convert atypical speech into standard, fluent speech.
An improved model with an encoder/decoder architecture is adopted, using a stack of Conformer blocks and a hybrid frame rate processing scheme. It combines subsampling of encoder activations and upsampling of decoder, and trains the model through a two-step training process, including pre-training and fine-tuning, to achieve robust conversion of atypical speech.
It accelerates the training of speech conversion models, improves robustness to atypical speech, and can convert atypical speech into more understandable, standard, and fluent speech. It is applicable to a variety of atypical speech patterns, including ALS speech and heavily accented speech.
Smart Images

Figure CN117396958B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to a speech conversion model based on a convolutional augmentation transform neural network (Conformer). Background Technology
[0002] Speech conversion models can be used to modify a source speaker's speech into another form without altering the speech's linguistic information. For example, a speech conversion model can generate a copy of a user's speech. Alternatively, a speech conversion model can convert a user's speech into an audio waveform of speech in another language. Machine learning methods can be used to accurately train speech conversion models and efficiently convert speech into another form. Summary of the Invention
[0003] One aspect of the present invention provides a speech conversion model including an encoder comprising a stack of self-attention blocks, the encoder being configured to encode an input spectrogram corresponding to a utterance. The speech conversion model further includes a spectrogram decoder configured to receive the encoded spectrogram as input from the encoder. The spectrogram decoder is configured to generate an output spectrogram as output corresponding to a synthesized speech representation of the utterance.
[0004] Implementations of this disclosure may include one or more of the following optional features. In some implementations, the input spectrogram corresponding to the utterance is extracted from input speech spoken by a speaker associated with atypical speech. In these implementations, the synthesized speech representation of the utterance includes a synthesized canonical fluent speech representation of the utterance.
[0005] Furthermore, the speech conversion model may include a field decoder configured to receive the encoded spectrogram from the encoder as input and generate a text representation corresponding to the transcription of the utterance as output. Additionally, the speech conversion model may include a phoneme decoder configured to receive the encoded spectrogram from the encoder as input and generate a phoneme representation of the utterance as output.
[0006] In some implementations, the stack of self-attention blocks includes a stack of Conformer blocks, each with a multi-head self-attention mechanism. In these implementations, the encoder may further include a first subsampling layer positioned before the stack of Conformer blocks and configured to receive the input spectrogram. This first subsampling layer comprises a convolutional neural network (CNN) layer followed by temporal pooling to reduce the number of frames processed by the initial Conformer blocks in the stack of Conformer blocks. Furthermore, in these implementations, the encoder may include a second subsampling layer positioned between the initial set of Conformer blocks in the stack of Conformer blocks and the final set of Conformer blocks in the stack of Conformer blocks. This second subsampling layer is configured to subsample the hidden representation output by the last Conformer block in the initial set of Conformer blocks to reduce the number of frames processed by the final set of Conformer blocks. In these implementations, the encoder may also include an upsampling layer disposed after the stack of the Conformer blocks, the upsampling layer comprising a single transposed CNN layer configured to upsample the hidden representation output by the final Conformer block in the stack of the Conformer blocks to increase the number of frames processed by the cross-attention mechanism disposed between the encoder and the spectrogram decoder.
[0007] Furthermore, a two-step training process can be used to train the speech conversion model, which includes a first training step pre-training the model on multiple spoken utterances by a typical speaker associated with typical fluent speech. Here, each spoken utterance is paired with its corresponding ground truth synthesized canonical fluent speech representation. The two-step training process also includes a second training step, which fine-tunes the parameters of the pre-trained speech conversion model based on multiple atypical speech samples spoken by a speaker associated with atypical speech.
[0008] In some implementations, the spectrogram decoder generates the output spectrogram directly from the encoded spectrogram without performing any intermediate text-to-speech conversion on the text representation corresponding to the transcription of the utterance.
[0009] Another aspect of this disclosure provides a method for a computer implementation of a speech conversion model. When executed on data processing hardware, the data processing hardware performs the following operations: receiving an input spectrogram corresponding to a utterance as input to an encoder of the speech conversion model, the encoder including a stack of self-attention blocks. The operation further includes generating an encoded spectrogram as the output of the encoder. This operation includes receiving the encoded spectrogram, generated as the output of the encoder, as input to a spectrogram decoder of the speech conversion model. This operation further includes generating an output spectrogram corresponding to a synthesized speech representation of the utterance as the output of the spectrogram decoder.
[0010] This aspect may include one or more of the following optional features. In some implementations, the input spectrogram corresponding to the utterance is extracted from input speech spoken by a speaker associated with atypical speech. In these implementations, the synthesized speech representation of the utterance includes a synthesized canonical fluent speech representation of the utterance.
[0011] In some embodiments, the operation includes receiving the encoded spectrogram, generated as the output of the encoder, as input to a field decoder of the speech conversion model. These embodiments also include generating a text representation corresponding to the transcription of the utterance as the output of the field decoder. The operation may further include receiving the encoded spectrogram, generated as the output of the encoder, as input to a phoneme decoder of the speech conversion model, and generating a phoneme representation of the utterance as the output of the phoneme decoder.
[0012] In some embodiments, the stack of self-attention blocks includes a stack of Conformer blocks, each Conformer block having a multi-head self-attention mechanism. In these embodiments, the encoder further includes a first subsampling layer disposed before the stack of Conformer blocks and configured to receive the input spectrogram. The first subsampling layer includes a convolutional neural network (CNN) layer followed by temporal pooling to reduce the number of frames processed by the initial Conformer blocks in the stack of Conformer blocks. Furthermore, in these embodiments, the encoder further includes a second subsampling layer disposed between the initial set of Conformer blocks in the stack of Conformer blocks and the final set of Conformer blocks in the stack of Conformer blocks. The second subsampling layer is configured to subsample the hidden representation output by the last Conformer block in the initial set of Conformer blocks to reduce the number of frames processed by the final set of Conformer blocks. In these embodiments, the encoder further includes an upsampling layer disposed after the stack of the Conformer blocks, the upsampling layer comprising a single transposed CNN layer configured to upsample the hidden representation output by the final Conformer block in the stack of the Conformer blocks to increase the number of frames processed by the cross-attention mechanism disposed between the encoder and the spectrogram decoder.
[0013] Furthermore, a two-step training process can be used to train the speech conversion model, which includes a first training step by pre-training the model on multiple spoken utterances by a typical speaker associated with typical fluent speech. Here, each spoken utterance is paired with its corresponding ground truth synthesized canonical fluent speech representation. The two-step training process also includes a second training step, which fine-tunes the parameters of the pre-trained speech conversion model based on multiple atypical speech samples spoken by a speaker associated with atypical speech.
[0014] In some implementations, the spectrogram decoder (220a) generates the output spectrogram directly from the encoded spectrogram (212) without performing any intermediate text-to-speech conversion on the text representation corresponding to the transcription of the utterance.
[0015] Details of one or more implementations of this disclosure are set forth in the accompanying drawings and the following description. Other aspects, features, and advantages will be apparent from the specification, the drawings, and the claims. Attached Figure Description
[0016] Figure 1 This is a schematic diagram of an example speech conversion system that includes a speech conversion model.
[0017] Figure 2 This is a schematic diagram of a speech conversion model.
[0018] Figure 3 This is a schematic diagram of an exemplary hybrid frame rate processing scheme for accelerating the training and inference time of speech conversion models.
[0019] Figure 4 This is a schematic diagram of an exemplary Conformer block.
[0020] Figure 5 This is a schematic diagram of an example training scheme for a speech conversion model.
[0021] Figure 6 This is a flowchart illustrating an example of the operations used to perform the speech conversion method.
[0022] Figure 7 This is a schematic diagram of an exemplary computing device that can be used to implement the systems and methods described herein.
[0023] In the various figures, the same reference numerals denote the same elements. Detailed Implementation
[0024] There is a growing focus on developing more inclusive speech technologies, particularly those that can assist people with speech impairments. With the introduction of end-to-end (E2E) deep learning-based models, Automatic Speech Recognition (ASR) has made significant progress in recognizing speech from speakers with dysarthric or atypical speech patterns for accurate transcription. Atypical speech patterns can include, but are not limited to, impaired speech due to physical or neurological conditions (e.g., speakers with amyotrophic lateral sclerosis (ALS), accented speech, and deaf speech. Speech conversion systems can apply similar deep learning-based models to convert speech with atypical speech patterns into typical, fluent output speech.
[0025] It is known that matching training and testing data distributions yield optimal performance for training speech conversion models. However, training models using current methods is challenging due to insufficient training data from speakers with speech impairments. Furthermore, such training data is difficult to obtain, as users with speech impairments may find it difficult to record enough data to adequately train the model. This disclosure presents improvements to speech conversion models with encoder / decoder structures. These improvements require less training data, accelerate speech conversion model training, allow the model to scale to large user sets, and are robust to a variety of atypical speech types. These improvements are provided by structurally modifying the speech conversion model using subsampling of encoder activations and corresponding upsampling of typical encoder outputs. This disclosure also provides a combination of many-to-one speech conversion (VC) and ASR in a unified model, using a shared encoder architecture for the task during inference to jointly decode speech and text.
[0026] As used herein, and unless otherwise stated, the terms “speech-to-speech system” and “speech-to-speech model” can refer to any combination of ASR systems / models in which input atypical speech is recognized and converted into corresponding text (e.g., transcription) and / or a set of phonemes representing the atypical speech, or speech-to-speech systems / models in which input atypical speech is directly converted into canonical fluent synthetic speech without performing speech recognition. In other words, a speech-to-speech system / model is configured to directly convert an input audio waveform or spectrogram corresponding to atypical speech into an output audio waveform or spectrogram corresponding to canonical fluent speech, without converting the input audio waveform into an intermediate representation (e.g., text or phonemes). It is evident that speech-to-speech models, and the techniques used to train them, will enable users with atypical speech to speak to other people and voice interfaces (e.g., digital assistants) by recognizing and / or reproducing the user's expected speech, and to be understood by them. While the examples described herein depict a speech conversion model that receives input audio waveforms or spectrograms corresponding to atypical speech and converts them into output audio waveforms or spectrograms corresponding to standard fluent speech, the speech conversion model can be similarly adapted to perform other types of speech conversion tasks without departing from the scope of this disclosure. For example, the speech conversion model can convert input audio waveforms or spectrograms corresponding to utterances in a first language into output audio waveforms or spectrograms corresponding to translations of utterances in different second languages. Similarly, the speech conversion model can receive spoken input from a user and output synthesized speech containing the same linguistic content as the spoken input but with different speech features of the target speaker.
[0027] Figure 1A speech conversion system 100 including a speech conversion model 200 and a vocoder 374 is shown. The speech conversion model 200 is configured to convert input audio data 102 corresponding to utterance 108 spoken by a source speaker 104 associated with atypical speech into output audio data 106 corresponding to a synthesized canonical fluent speech representation of the same utterance 114 spoken by a target speaker 104. As used herein, the input audio data 102 may include an input spectrogram corresponding to utterance 108. As used herein, the output audio data 106 may include an output spectrogram 222 corresponding to a synthesized canonical fluent speech representation of the same utterance 114, or a time-domain audio waveform 376 converted from the output spectrogram 222 by the vocoder 375. Although not shown, an acoustic front end residing on user equipment 110 may convert the time-domain audio waveform of utterance 108 captured via a microphone of user equipment 110 into input spectrogram 102 or other types of audio data 102. In some implementations, the speech conversion model 200 converts the input audio data 102 corresponding to utterance 108 into a text representation (e.g., grapheme, wordpiece, or word) corresponding to utterance 108, either a transcription 201 or a phoneme representation 202. In other implementations, the speech conversion model 200 of the speech conversion system 100 is configured to directly convert the input audio data 102 (e.g., input spectrogram) into output audio data 106 (e.g., output spectrogram 222) without performing speech recognition, or without requiring the generation of any intermediate discrete representation (e.g., text or phonemes) from the input audio data 102.
[0028] The speech conversion model 200 includes a spectrogram encoder 210 and one or more decoders 220, 220ac. The spectrogram encoder 210 is configured to encode an input spectrogram 102 into an encoded spectrogram 212 (e.g., including a hidden feature representation comprising a series of vectors). The one or more decoders 220, 220ac are configured to decode the encoded spectrogram 212 into an output spectrogram 222 corresponding to a synthesized canonical fluent speech representation, a transcription 201, and / or a phoneme representation 202. The transcription 201 may include a canonical fluent transcription of utterance 108, which can be understood by a human reader and / or by downstream applications (e.g., a digital assistant).
[0029] Encoder 210 may include a stack of multi-head attention blocks 400 (here referred to as Conformer blocks 400), which may include a Conformer or a Transformer (transformer neural network). Each multi-head attention block 400 may include a multi-head attention mechanism 420. Figure 4The Conformer block 400 can be implemented by the encoder 210 to capture a fine-grained spectrogram pattern of incoming atypical speech. For example, when the spectrogram encoder 210 receives input audio data 102 of utterance 108, the spectrogram encoder 210 can use the Conformer block 400 to process 10-millisecond (ms) speech samples of the input spectrogram 102 to produce an upsampled 40-ms encoded spectrogram 212. See below for reference. Figure 3 and 4 The upsampling process of the Conformer block 400 of encoder 210 is discussed in more detail. Next, spectrogram decoder 220a can generate an output spectrogram 222 corresponding to the synthesized canonical fluent speech representation based on the upsampled encoded spectrogram 212 output from spectrogram encoder 210. For example, spectrogram decoder 220a can receive from spectrogram encoder 210 an upsampled 40ms encoded spectrogram 212 representing a 10ms speech sample of input spectrogram 102. Here, through cross-attention mechanisms 231, 231a (… Figure 2 and 3 The spectrogram decoder 220a can generate a 12.5ms output spectrogram 222, which corresponds to a synthesized canonical fluent speech representation of utterance 114, including the expected words or words as part of the 10ms input audio data 102, but excluding non-fluent parts of atypical speech.
[0030] In some examples, the speech conversion model 200 also includes a word piece decoder 220b that decodes the encoded spectrogram 212 into a text representation, such as a transcription 201. For example, the word piece decoder 220b can be trained to decode the encoded spectrogram 212 into corresponding words that can form the transcription 201. While the example shown uses the word piece decoder 220b, the model 200 may also use a grapheme decoder 220b or a word decoder 220b, which are configured to decode the encoded spectrogram into graphemes or words, respectively. Additionally or optionally, the speech conversion model 200 may also include a phoneme decoder 220c that decodes the encoded spectrogram 212 into a phoneme representation 202, which includes phonemes indicating a canonical fluent speech representation of the synthesis of utterance 114. Therefore, the spectrogram, field, and phoneme decoders 220a-c may correspond to the parallel decoding branches of the speech conversion model 200, each receiving an upsampled encoded spectrogram 212 encoded by the spectrogram encoder 210 and transmitting in parallel the corresponding in the output spectrogram 222, transcription 201, and phoneme representation 202. The vocoder 375 of the speech conversion system 100 (which may also be interchangeably referred to as synthesizer 375) is configured to convert the output spectrogram 222 transmitted by the spectrogram decoder 220a into a time-domain waveform 376 of synthesized canonical fluent speech of the same utterance 114, for audible output from another computing device 116. The time-domain audio waveform includes an audio waveform that defines the amplitude of the audio signal as it changes over time. The vocoder 375 may include a unit selection module or a WaveNet module for synthesizing the output spectrogram 222 into a time-domain waveform of synthesized typical fluent speech. In some embodiments, synthesizer 375 includes a vocoder network, i.e., a neural vocoder, which is trained individually and tuned based on a mel frequency spectrogram for conversion into a time-domain audio waveform. In some additional examples, vocoder 375 includes a streaming vocoder 375, such as a streaming Griffin-Lim vocoder. An exemplary streaming vocoder is described in U.S. Provisional Application 63 / 312,195, filed February 21, 2022, the contents of which are incorporated herein by reference in their entirety.
[0031] In the example shown, source speaker 104 is associated with atypical speech, meaning that source speaker 104 speaks in an atypical speech pattern that may be difficult to understand. Atypical speech patterns may include, but are not limited to, impaired speech caused by physical or neurological conditions (e.g., a speaker with amyotrophic lateral sclerosis (ALS)), severely aggravated speech, and deaf speech. As an example, source speaker 104 has ALS and is associated with atypical speech caused by ALS. Therefore, speech conversion model 200 is trained to directly convert the input spectrogram 102 corresponding to utterance 108 spoken by source speaker 104 associated with ALS speech into an output spectrogram 222 corresponding to a synthesized standard fluent speech representation of the same utterance 108. Thus, the synthesized standard fluent speech representation provided by output spectrogram 222 improves the intelligibility of the ALS speech spoken by source speaker 104. Without departing from the scope of this disclosure, speech conversion model 200 can be trained as a multilingual speech conversion model to directly convert input spectrogram 102 corresponding to utterances 108 spoken in a first language into output spectrogram 222 corresponding to synthesized speech representations of those utterances 108, which are identical to the speech of the source speaker but spoken in a different second language. Additionally, model 20 can be trained to directly convert input spectrogram 102 corresponding to utterances 108 spoken by a source speaker with first speech features into output spectrogram 222 corresponding to synthesized speech representations of the same utterances 108 with different speech features corresponding to the target speaker.
[0032] The computing device 110 associated with the source speaker 104 can capture the utterance 108 spoken by the source speaker 104 and provide the corresponding input audio data 102 to the speech-to-speech conversion system 100 to convert it into any one of the output spectrogram 222, transcription 201, or phoneme representation 202. The computing device 110 may include, but is not limited to, smartphones, tablets, desktop / laptop computers, smart speakers, smart displays, smart devices, wearable devices with assistant support (e.g., smartwatches, smart headphones, smart glasses, etc.), or vehicle infotainment systems. The speech conversion system 100 can then use a vocoder 375 to convert the output spectrogram 222 into a time-domain audio waveform 376, which can be audibly output from the computing device 110 or another computing device 116 as utterance 114 of synthesized, typically fluent speech. Without departing from the scope of this disclosure, the speech conversion system 100 may also provide to another computing device 116 associated with user 118 a transcription 201 and / or phoneme representation 202 corresponding to a synthesized canonical fluent speech representation of the same utterance 114 spoken by source speaker 104, whereby the other computing device 116 can display the canonical transcription 201 as an intelligible representation of the utterance 108 spoken by source speaker 104. Alternatively, a text-to-speech (TTS) system may be used to convert the transcription 201 or phoneme representation 202 into synthesized canonical fluent speech. In this example, source speaker 104 and user 118 speak to each other via their respective computing devices 110, 116 (e.g., via telephone call or other types of voice communication protocols, such as voice over the Internet Protocol). Although source speaker 104 and other users 118 may speak the same language, other users 118 may have difficulty understanding source speaker 104 because source speaker 104 has atypical speech due to medical conditions (e.g., atypical speech), accents, or a different native language. Therefore, although the source speaker 104 speaks in atypical speech that may be difficult to understand (e.g., ALS speech), another user 118 who hears the synthesized canonical fluent speech representation will more easily understand the utterance 108 expected by the source speaker 104. In other words, the synthesized canonical fluent speech representation provides a more consistent rhythm and may be easier for other users to understand compared to the original utterance 108 spoken by the target speaker in atypical speech. It is worth noting that the synthesized canonical fluent speech representation is within the speech of the source speaker 104. However, depending on the application, the speech conversion system 100 can generate synthesized canonical fluent speech within the speech of a target speaker who has different speech characteristics from the source speaker.
[0033] In some other examples, the speech-to-speech system 100 transmits output audio data 106 corresponding to a synthesized, canonical, fluent speech representation of the utterance spoken by the source speaker 104 to an output audio device, so as to audibly output the synthesized, canonical, fluent speech representation of the source speaker 104's speech to an audience. For example, the source speaker 104 could be a psychology professor giving lectures to students, where the utterance spoken by the source speaker 104 includes medical terminology belonging to a specific, specialized field, such as psychology. As will become apparent, the speech-to-speech model 200 is trained to learn the linguistic diversity associated with a specific field, as well as the acoustic diversity associated with a specific type of atypical speech associated with the source speaker 104.
[0034] Optionally, another computing device 116 may be associated with a downstream automatic speech recognition (ASR) system, in which the speech conversion system 100 acts as a front end to provide the ASR system with output audio data 106 corresponding to a synthesized, canonical, fluent speech representation as input for conversion into recognized text. The recognized text may be presented to other users 118 and / or provided to a natural language understanding (NLP) system for further processing. The functionality of the speech conversion system 100 may reside on a remote server 112, or on one or both of the computing devices 110 and 116, or on any combination of a remote server and computing devices 110 and 116. The speech conversion system 100 may be distributed across multiple devices, such that the speech conversion model 200 resides on one of the computing devices 110 or the remote server 112, and the vocoder 375 resides on one of the remote server 112 or another computing device 116. In some implementations, as the source speaker 104 pronounces a portion of the utterance in atypical speech, the speech conversion model 200 continuously generates an output spectrogram 222 corresponding to a synthesized, canonical, fluent speech representation of the utterance. By continuously generating output spectrograms 222 corresponding to synthesized, canonical, fluent speech representations of portions of the utterance 108 spoken by the source speaker 104, the rhythm of the dialogue between the source speaker 104 and the user 118 (or audience) can be more natural. In some other implementations, the speech conversion model 200 waits to determine / detect when the source speaker 104 stops speaking, using techniques such as speech activity detection, end-pointing, query end detection, etc., and before converting the corresponding input audio data 102 of the utterance 108 with atypical speech into a corresponding output spectrogram 222 of a synthesized, canonical, fluent speech representation of the same utterance 114.
[0035] Figure 2 It shows the result of Figure 1A schematic diagram of an example speech conversion model 200 used in the speech conversion system 100 is provided. The speech conversion model 200 includes an encoder 210 and one or more decoders 220, 220ac. The encoder 210 is configured to encode input audio data 102 into an encoded spectrogram 212. Here, the input audio data 102 includes a sequence of input spectrograms corresponding to the utterance 108 spoken by the source speaker 104. In some embodiments, the encoder 210 includes a stack of Conformer blocks 400. In these embodiments, the encoder subsamples the input audio data 102 using convolutional layers and then processes the input audio data 102 using the stack of Conformer blocks 400. Each Conformer block 400 may include a feedforward layer, a self-attention layer, a convolutional layer, and a second feedforward layer. In some examples, the stack of Conformer blocks 400 includes 17 Conformer blocks, each with 512 states, 8 attention heads, and a 32×1 convolutional kernel size. Figure 4 A schematic diagram of an exemplary Conformer block is provided. The encoder 210 can instead use a stack of Transformer blocks or lightweight convolution blocks instead of the Conformer block.
[0036] The spectrogram, phoneme, and field decoders 220 and 220ac may each include a recurrent neural network-based architecture, each receiving a shared coded spectrogram 212 output by the encoder 210. The spectrogram decoder 220a may include cross-attention mechanisms 231 and 231a (also...). Figure 3 (As shown in the figure), it is configured to receive a shared encoded spectrogram 212 from encoder 210. Spectrogram decoder 220a may further process the shared encoded spectrogram 212 using multiple long-short-time memory (LSTM) layers 233, 233a and multiple convolutional layers 235. For example, spectrogram decoder 220a may include five (5) LSTM layers 233a and five (5) transformation layers 235. Spectrogram decoder 220a may generate an output spectrogram 222. In some embodiments, spectrogram decoder 220a may generate the output spectrogram 222 directly from the encoded spectrogram 212 without performing any intermediate text-to-speech conversion on the text representation corresponding to the transcription of the speech.
[0037] In the example shown, the field decoder 220b includes corresponding cross-attention mechanisms 231, 231b, which are configured to receive a shared encoded spectrogram from the encoder, followed by two long short-term memory (LSTM) layers 233, 233b and a softmax layer 245, 245a, whose outputs correspond to the text representation 201 of the phonated transcription.
[0038] Similar to the field decoder 220b, the phoneme decoder 220c may also include cross-attention mechanisms 231, 231c, which are configured to receive a shared encoded spectrogram 212 from the encoder 210, followed by two long short-term memory (LSTM) layers 233, 233c and a softmax layer 245, 245b for the phoneme representation of the output utterance 202.
[0039] Figure 3 It shows the method for acceleration Figure 1 A schematic diagram 300 illustrates an exemplary hybrid frame rate processing scheme for the training and inference time of a speech-to-speech model 200. The hybrid frame rate processing scheme can improve the memory consumption and training speed of the encoder 220 in speech-to-speech processing (i.e., generating the output spectrogram 222 via spectrogram decoder 220a). Unlike other models, such as Automatic Speech Recognition (ASR) or Text-to-Speech (TTS), where the predicted target or input sequence is text, speech-to-speech models use sound frames as the input sequence and also output a sequence of sound frames. Since the number of output sound frames is much larger than the number of output text sequences, speech-to-speech requires increased computation compared to ASR or TTS models. In some cases, due to the self-attention mechanism of encoder 210, the model complexity becomes a quadratic function based on the number of input frames. Furthermore, memory usage can be proportional to the length of the acoustic sequence, resulting in smaller batch sizes and slower training speeds. Figure 3 As shown, the hybrid frame rate processing scheme can significantly reduce the number of computations and subsequently speed up training.
[0040] In some implementations, the hybrid frame rate processing scheme uses convolutional subsampling with a 3×3 kernel size and a 2×2 stride, resulting in a subsampling factor of 4. In these implementations, the transposed convolutional network comprises a single layer of a convolutional neural network (CNN) with 512 channels, a filter size of 4, and a time stride of 2. Furthermore, the hybrid frame rate scheme may include extracting 128-dim log-mel spectrogram features from the input speech using a 30ms window and a 10ms frame shift, which can be provided to encoder 210. In one example implementation, the spectrogram decoder 220a targets a 1025-dim short-time Fourier transform (STFT) amplitude, calculated using a 50ms frame length, a 12.5ms shift, and a 2048-point FFT.
[0041] The processing scheme can begin by receiving a 10-millisecond (ms) speech sample from the input spectrogram 102 via spectrogram encoder 210. Encoder 210 can initially process the 10ms speech sample using a first subsampling layer 305 comprising multiple CNN layers. Subsampling performed by the first subsampling layer 305 is implemented using CNN layers and subsequently pooled temporally to reduce the number of frames processed by the initial Conformer blocks in the stack of Conformer blocks 400, 400ab. The CNN can subsample the 10ms speech into a 40ms representation, which is then provided to the initial set 400a of the Conformer blocks. The initial set 400a of the Conformer blocks can process the 40ms representation, which is then provided to a second subsampling layer 315. The second subsampling layer 315 can be positioned between the initial set 400a and the final set 400b of the Conformer blocks. In some examples, the initial set 400a of Conformer blocks includes four Conformer blocks, and the final set 400b of Conformer blocks includes 13 Conformer blocks, making the total number of Conformer blocks in encoder 210 17. Here, the second subsampling layer 315 can be configured to subsample the hidden representation 308 output by the last Conformer block in the initial set 400a of Conformer blocks to reduce the number of frames processed by the final set 400b of Conformer blocks. For example, the second subsampling layer 315 can be configured to subsample the 40ms hidden representation 308 output by the initial set 400a of Conformer blocks into a corresponding 80ms representation 318. At the end of the last Conformer block in the final set 400b of Conformer blocks, encoder 210 uses an upsampling layer 325 to upsample the 80ms hidden representation 322. The upsampling layer 325 may include a single transposed CNN layer configured to upsample the 80ms hidden representation 322 output by the final Conformer block of the final set 400b of Conformer blocks to a corresponding 40ms representation of the encoded spectrogram 212, thereby increasing the number of frames of the encoded spectrogram 212.
[0042] The encoded spectrogram 212 can be received by a cross-attention mechanism 231a disposed between the encoder 210 and the spectrogram decoder 220a. In some embodiments, the cross-attention mechanism 231a is included in the spectrogram decoder 220a. The spectrogram decoder 220a can use the cross-attention mechanism 231a to reduce the 40ms representation of the encoded spectrogram 212 to a 25ms representation, which can then be provided to the LSTM 233a. The output of the LSTM 233a can be reduced by a reduction factor 335, and the spectrogram decoder 220a can output the resulting output spectrogram 222 with a final size of 12.5ms. The output spectrogram 222 can be provided to the vocoder 375. Figure 1 ), used to convert the corresponding time-domain audio waveform into synthesized speech.
[0043] The above example is not limiting. Encoder 210 can receive speech samples of any appropriate length for processing. Encoder 210 can then process, subsample, or upsample the speech samples to produce an encoded spectrogram 212, which can be of any appropriate length. Similarly, decoder 220a can then process the encoded spectrogram 212 to produce an output spectrogram 222 of appropriate length.
[0044] In the experiments, given the same encoder frame shift, the hybrid frame rate scheme achieved different results with different subsampling and upsampling settings. For example, increased subsampling typically leads to accelerated training but causes regressions in the spectrogram that are difficult to recover through upsampling. Information loss can be evaluated based on the sparsity of the feedforward neural network weighting matrix in the final Conformer block of the final set 400b of the encoder 210 Conformer blocks. The cumulative change ratio (CPV) can be calculated using the following formula:
[0045] Among them, s i Let be the i-th singular value of the matrix, k be the number of singular values we consider, and D be the size of the feedforward matrix (D = 512). For any given k, a larger CPV indicates that the network can learn a data structure with a sparsity exponent k. A smaller k value indicates a sparse matrix structure.
[0046] Figure 4An example of a Conformer block 400 from the stack of Conformer layers of encoder 210 is provided. Conformer block 400 includes a first half-feedforward layer 410, a second half-feedforward layer 440, and a multi-head self-attention block 420 and a convolutional layer 430 disposed between the first and second half-feedforward layers 410 and 440, as well as a cascaded arithmetic unit 405. The first half-feedforward layer 410 processes input audio data 102 comprising an input mel spectrogram sequence. Subsequently, the multi-head self-attention block 420 receives the input audio data 102 concatenated with the output of the first half-feedforward layer 410. Intuitively, the role of the multi-head self-attention block 420 is to summarize the noise context for each input frame to be enhanced. The convolutional layer 430 subsamples the output of the multi-head self-attention block 420 concatenated with the output of the first half-feedforward layer 410. Subsequently, the second half-feedforward layer 440 receives the concatenation of the output of the convolutional layer 430 and the multi-head self-attention block 420. The layer normalization module 450 processes the output from the second half-feedforward layer 440. Mathematically, the Conformer block 400 transforms the input feature x using the modulation feature m to produce the output feature y, as follows: x″=x′+MHCA(x′,n′) x″′=x′⊙r(x″)+h(x″) x″″=x′+MHCA(x′,x″′)
[0047] Figure 5 A training process 500 for the speech conversion model 200 is illustrated. In some embodiments, process 500 employs a two-step training technique. First, the speech conversion model 200 is pre-trained on typical speech from a large pool of speakers to obtain a many-to-one speech conversion model 200, thereby obtaining a speaker-independent ASR / conversion base model. The target speech used for training can be speech synthesized from a reference copy of a predetermined speech reflecting typical speech. For personalization, any parameters of the base model can be fine-tuned to speech from a single input speaker (e.g., a deaf speaker) to obtain a one-to-one speech conversion model from atypical speech to typical speech (speaker-related ASR).
[0048] Reference Figure 5The process 500 begins by pre-training the speech conversion model 200 using pre-training data 505. Pre-training is a technique used to initialize the model, which can then be further fine-tuned based on additional training data 510. For the speech conversion model 200, pre-training may include launching the speech conversion model 200 with pre-training data 505, which includes multiple spoken utterances by a typical speaker associated with typical fluent speech. The pre-training data 505 may also include spoken utterances paired with corresponding ground-truth synthesized canonical fluent speech representations of the spoken utterances.
[0049] Then, process 500 can fine-tune the parameters of the pre-trained speech conversion model 200 for atypical speech. The training process may include training either the encoder 210 or the decoder 220, 220ac individually or in any suitable combination. Process 500 includes feeding training input 510 to the speech conversion model 200. In some embodiments, training input 510 includes multiple atypical speech samples spoken by one or more speakers associated with atypical speech. Furthermore, training input 510 may be labeled using a label 520 indicating the target output associated with training input 510. Upon receiving training input 510, speech conversion model 200 may generate output 515 (e.g., copy 201, phoneme representation 202, output spectrogram 222). Speech conversion model 200 may be configured to... Figure 2-4 The training input 510 is processed in any of the ways described in the document or in any other appropriate way for speech conversion.
[0050] In some implementations, output 515 is used by loss function 530 to generate loss 540. That is, loss function 530 compares output 515 and label 520 to produce loss 540, where loss 540 indicates the difference between label 520 (i.e., the target output) and output 515. Loss function 350 can implement any suitable technique to determine the loss, such as regression loss, mean squared error, mean squared logarithmic error, mean squared absolute error, binary classification, binary cross-entropy, hinge loss, multi-class loss, etc. Loss 540 can then be directly fed into speech conversion model 200. Here, speech conversion model 200 processes loss 540 and adjusts one or more parameters of speech conversion model 200 to resolve loss 540.
[0051] Figure 6 This is a flowchart illustrating an exemplary arrangement of the operation of a computer-implemented method 600 for performing speech conversion. For example, it can be provided by... Figure 1Various elements of an example speech conversion system 100 are used to perform method 600. In operation 610, the method includes receiving an input spectrogram 102 corresponding to utterance 108 as input to an encoder 210 of a speech conversion model 200, the encoder 210 including a stack of self-attention blocks 400. In operation 620, method 600 includes generating an encoded spectrogram 212 as output of encoder 210. In operation 630, method 600 includes receiving the encoded spectrogram 212 as output from encoder 210 as input to spectrogram decoder 220a of speech conversion model 200. In operation 640, method 600 includes generating an output spectrogram 222 corresponding to a synthesized canonical fluent speech representation of the same utterance 114 as output from spectrogram decoder 220a.
[0052] A software application (i.e., a software resource) can refer to computer software that instructs a computing device to perform a task. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.
[0053] Non-transitory memory can be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) used by a computing device. Non-transitory memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include (but are not limited to) flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used in firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.
[0054] Figure 7 This is a schematic diagram of an exemplary computing device 700 that can be used to implement the systems and methods described herein. The computing device 700 represents various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the embodiments of the invention described and / or claimed herein.
[0055] Computing device 700 includes a processor 710, a memory 720, a storage device 730, a high-speed interface / controller 740 connected to the memory 720 and a high-speed expansion port 750, and a low-speed interface / controller 760 connected to a low-speed bus 770 and the storage device 730. Each of components 710, 720, 730, 740, 750, and 760 is interconnected using various buses and may be suitably mounted on a common motherboard or otherwise mounted. Processor 710 can process instructions for execution within computing device 700, including instructions stored in memory 720 or on storage device 730, to display graphical information of a graphical user interface (GUI) on an external input / output device (e.g., a display 780 coupled to high-speed interface 740). In other embodiments, multiple processors and / or multiple buses, as well as multiple memories and memory types, may be suitably used. Furthermore, multiple computing devices 700 may be connected, with each device providing a portion of the necessary operation (e.g., as a server library, blade server group, or multiprocessor system). The processor 710 may be referred to as data processing hardware 710 residing on remote server 112, one or both of computing devices 110 and 116, or any combination of remote server and computing devices 110 and 116. The memory 710 may be referred to as memory hardware 720 residing on remote server 112, one or both of computing devices 110 and 116, or any combination of remote server and computing devices 110 and 116.
[0056] Memory 720 stores information non-transitory within computing device 700. Memory 720 may be a computer-readable medium, a volatile memory cell, or a non-volatile memory cell. Non-transitory memory 720 may be a physical device for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) used by computing device 700. Examples of non-volatile memory include (but are not limited to) flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.
[0057] Storage device 730 provides mass storage for computing device 700. In some embodiments, storage device 730 is a computer-readable medium. In various embodiments, storage device 730 may be a floppy disk device, hard disk device, optical disk device, magnetic tape device, flash memory or other similar solid-state storage device, or device array, including devices in a storage area network or other configurations. In other embodiments, a computer program product is tangibly contained in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as memory 720, storage device 730, or memory on processor 710.
[0058] High-speed controller 740 manages bandwidth-intensive operations of computing device 700, while low-speed controller 760 manages less bandwidth-intensive operations. This allocation of responsibilities is merely exemplary. In some embodiments, high-speed controller 740 is coupled to memory 720, display 780 (e.g., via a graphics processor or accelerator), and high-speed expansion port 750, which accepts various expansion cards (not shown). In some embodiments, low-speed controller 760 is coupled to storage device 730 and low-speed expansion port 790. Low-speed expansion port 790, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), may be coupled to one or more input / output devices, such as keyboards, pointing devices, scanners, or networking devices, such as switches or routers, for example, via a network adapter.
[0059] The computing device 700 can be implemented in a variety of different forms, as shown in the figure. For example, it can be implemented as a standard server 700a or multiple times in a set of such servers 700a, a laptop computer 700b, or part of a rack server system 700c.
[0060] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system, the programmable system including at least one programmable processor, which may be dedicated or general-purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transmit data and instructions to the storage system, at least one input device, and at least one output device.
[0061] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.
[0062] The processes and logical flows described in this specification can be executed by one or more programmable processors (also known as data processing hardware), executing one or more computer programs to perform functions by manipulating input data and generating output. The processes and logical flows can also be executed by special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). For example, processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, as well as any one or more processors of any type of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices for receiving or transferring data to one or more mass storage devices used to store data, such as magnetic disks, magneto-optical disks, or optical disks. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; CD-ROMs and DVD-ROMs. Processors and memory can be supplemented by or integrated into dedicated logic circuits.
[0063] To provide interaction with the user, one or more aspects of this disclosure can be implemented on a computer with a display device, such as a CRT (cathode ray tube), an LCD (liquid crystal display) monitor, or a touchscreen for displaying information to the user, and optionally also includes a keyboard and a pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's client device in response to a request received from a web browser.
[0064] Various embodiments have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other embodiments are also within the scope of the appended claims.
Claims
1. A speech conversion system, characterized in that, include: An encoder (210) configured to encode an input spectrogram (102) corresponding to a speech (108), the input spectrogram being extracted from input speech spoken by a source speaker associated with atypical speech, and comprising a sequence of spectrogram frames of length equal to a first value, the encoder comprising: The first subsampling layer is configured to subsample the spectrogram frame sequence to increase the length of the spectrogram frame sequence to a second value greater than the first value; A stack of self-attention blocks, including an initial set and a final set of self-attention blocks, wherein the initial set of self-attention blocks is configured to process a sequence of spectrogram frames of subsampled data output from the first subsampling layer to output a first hidden representation of length equal to the second value; and A second subsampling layer is configured to subsample the first hidden representation output from the initial set of the self-attention blocks to increase the length of the first hidden representation to a third value greater than the second value, wherein the final set of the self-attention blocks is configured to process the subsampled first hidden representation output from the initial set of the self-attention blocks to output a second hidden representation with a length equal to the third value; and The spectrogram decoder (220a) is configured to: The encoder (210) receives the encoded spectrogram (212) as input; and An output spectrogram (222) corresponding to the synthesized speech representation of the utterance (108) is generated as output, wherein the synthesized speech representation of the utterance (108) includes the synthesized normalized fluent speech representation of the utterance in the voice of the source speaker.
2. The speech conversion system according to claim 1, characterized in that, It also includes a field decoder (220b), which is configured to: The encoded spectrogram (212) is received as input from the encoder (210); and The text representation corresponding to the transcription (201) of the utterance (108) is generated as the output.
3. The speech conversion system according to claim 2, characterized in that, It also includes a phoneme decoder (220c), which is configured to: The encoded spectrogram (212) is received as input from the encoder (210); and The phoneme representation (202) of the utterance (108) is generated as output.
4. The speech conversion system according to claim 3, characterized in that, The stack (400) of self-attention blocks includes a stack (400) of Conformer blocks, each Conformer block having a multi-head self-attention mechanism (420).
5. The speech conversion system according to claim 4, characterized in that, The first subsampling layer (305) is set before the stack of the self-attention block, and the first subsampling layer (305) includes a convolutional neural network (CNN) layer followed by temporal pooling.
6. The speech conversion system according to claim 5, characterized in that, The second sub-sampling layer (315) is positioned between the initial set of self-attention blocks in the stack of self-attention blocks and the final set of self-attention blocks in the stack of self-attention blocks.
7. The speech conversion system according to claim 6, characterized in that, The encoder (210) also includes an upsampling layer (325) disposed after the stack of the self-attention blocks, the upsampling layer (325) comprising a single transposed CNN layer configured to upsample the hidden representation output by the final set of the self-attention blocks in the stack of the self-attention blocks to increase the number of frames processed by the cross-attention mechanism (231a) disposed between the encoder (210) and the spectrogram decoder (220a).
8. The speech conversion system according to claim 7, characterized in that, The speech conversion system is trained by a two-step training process (500), which includes: The first training step involves pre-training the speech conversion system on multiple spoken utterances by a typical speaker associated with typical fluent speech, with each spoken utterance paired with its corresponding ground truth synthesized canonical fluent speech representation; and The second training step involves fine-tuning the parameters of the pre-trained speech conversion system based on multiple atypical speech samples spoken by atypical speakers associated with atypical speech.
9. The speech conversion system according to any one of claims 1-8, characterized in that, The spectrogram decoder (220a) generates the output spectrogram (222) directly from the encoded spectrogram (212) without performing any intermediate text-to-speech conversion on the text representation corresponding to the transcription (201) of the utterance (108).
10. A method (600) for speech conversion, characterized in that, When the method is executed on the data processing hardware (710), the data processing hardware (710) is caused to perform the following operations: The input spectrogram (102) corresponding to the speech (108) is received as input to the encoder (210) of the speech conversion system. The input spectrogram is extracted from the input speech spoken by the source speaker associated with atypical speech and includes a sequence of spectrogram frames of length equal to the first value. The first subsampling layer of the encoder is used to subsample the spectrogram frame sequence to increase the length of the spectrogram frame sequence to a second value greater than the first value; Using the initial set of self-attention blocks of the stack of the encoder's self-attention blocks, process the spectrogram frame sequence of subsampled subsampled output by the first subsampled layer to output a first hidden representation with a length equal to the second value; Using the second subsampling layer of the encoder, the first hidden representation output from the initial set of self-attention blocks is subsampled to increase the length of the first hidden representation to a third value greater than the second value; Using the final set of the self-attention blocks of the stack of the encoder, the first hidden representation of the subsampled output by the initial set of the self-attention blocks is processed to output a second hidden representation of length equal to the third value; and Based on the second hidden representation, an encoded spectrogram (212) is generated as the output of the encoder (210); The encoded spectrogram (212), which is generated as the output of the encoder (210), is received as the input to the spectrogram decoder (220a) of the speech conversion system; as well as An output spectrogram (222) corresponding to the synthesized speech representation of the utterance (108) is generated as the output of the spectrogram decoder (220a), wherein the synthesized speech representation of the utterance (108) includes a synthesized canonical fluent speech representation of the utterance in the voice of the source speaker.
11. The method (600) according to claim 10, characterized in that, The operation also includes: The encoded spectrogram (212), generated as the output of the encoder (210), is received as the input to the field decoder (220b) of the speech conversion system; and A text representation corresponding to the transcription (201) of the utterance (108) is generated as the output of the field decoder (220b).
12. The method (600) according to claim 11, characterized in that, The operation also includes: The encoded spectrogram (212), generated as the output of the encoder (210), is received as the input to the phoneme decoder (220c) of the speech conversion system; and A phoneme representation (202) of the utterance (108) is generated as the output of the phoneme decoder (220c).
13. The method (600) according to claim 12, characterized in that, The stack (400) of the self-attention blocks of the encoder (210) includes a stack (400) of Conformer blocks, each Conformer block having a multi-head self-attention mechanism (420).
14. The method (600) according to claim 13, characterized in that, The first subsampling layer (305) is set before the stack of the self-attention block, and the first subsampling layer (305) includes a convolutional neural network (CNN) layer followed by temporal pooling.
15. The method (600) according to claim 14, characterized in that, The second sub-sampling layer (315) is positioned between the initial set of self-attention blocks in the stack of self-attention blocks and the final set of self-attention blocks in the stack of self-attention blocks.
16. The method (600) according to claim 15, characterized in that, The encoder (210) further includes an upsampling layer (325) disposed after the stack of the self-attention blocks, the upsampling layer (325) comprising a single transposed CNN layer configured to upsample the hidden representation output by the final set of the self-attention blocks in the stack of the self-attention blocks to increase the number of frames processed by the cross-attention mechanism (231a) disposed between the encoder (210) and the spectrogram decoder (220a).
17. The method (600) according to claim 16, characterized in that, The speech conversion system is trained using a two-step training process (500), the two-step training process (500) including: The first training step involves pre-training the speech conversion system on multiple spoken utterances by a typical speaker associated with typical fluent speech, with each spoken utterance paired with its corresponding ground truth synthesized canonical fluent speech representation; and The second training step involves fine-tuning the parameters of the pre-trained speech conversion system based on multiple atypical speech samples spoken by atypical speakers associated with atypical speech.
18. The method (600) according to any one of claims 10-17, characterized in that, The spectrogram decoder (220a) generates the output spectrogram (222) directly from the encoded spectrogram (212) without performing any intermediate text-to-speech conversion on the text representation corresponding to the transcription (201) of the utterance (108).
Citation Information
Patent Citations
Systems and methods for neural voice cloning with a few samples
US20190251952A1
Singing voice separation with deep u-net convolutional networks
US20200043516A1
