Synthetic data augmentation using speech-to-speech and speech recognition models

Abstract

A method (380) for training a speech conversion model (300) includes obtaining multiple transcripts (302) from a set of spoken training utterances (305) and multiple non-spoken training text utterances. Each spoken training utterance is spoken by a target speaker (104) associated with atypical speech and includes a corresponding transcript paired with a corresponding non-synthetic speech representation (304). The method also includes using the set of spoken training utterances to tune a TTS model (210) to synthesize speech in the target speaker's voice and capture atypical speech. For each non-spoken training text utterance, the method further includes generating a synthetic speech representation (306) as output from the tuned TTS model, which includes the target speaker's voice and captures atypical speech. The method also includes training the speech conversion model based on the synthetic speech representation.

Description

Technical Field

[0001] This disclosure relates to synthetic data augmentation using speech conversion and speech recognition models. Background Technology

[0002] Automatic speech recognition (ASR), the process of acquiring audio input and transcribing it into text, has become a crucial technology widely used in mobile devices and other devices. Typically, ASR attempts to provide an accurate transcription of what a person says by acquiring audio input (e.g., spoken words) and transcribing it into text.

[0003] One challenge in developing deep learning-based speech conversion and ASR models is that the parameters of these models tend to overfit the training data, making it difficult to generalize unseen data when the training data is not extensive enough. While training speech conversion and ASR models on larger training datasets improves accuracy, there is a lack of sufficient training data that includes domain-specific (i.e., linguistic diversity) uttered by speakers in atypical speech patterns (i.e., acoustic diversity) to achieve acceptable accuracy with these models. Summary of the Invention

[0004] One aspect of this disclosure provides a method for training a speech conversion model, the speech conversion model being personalized for a target speaker associated with atypical speech. The method includes obtaining a plurality of training text utterances by data processing hardware. A first portion of the plurality of training text utterances includes a plurality of transcriptions from a set of spoken training utterances. Each spoken training utterance is spoken by a target speaker associated with atypical speech and includes a corresponding transcription paired with a corresponding non-synthetic speech representation of the corresponding spoken training utterance. A second portion of the plurality of training text utterances includes a plurality of non-spoken training text utterances associated with a specific domain in which the speech conversion model is trained to learn. Each non-spoken training text utterance is not paired with any corresponding spoken utterance. The method also includes using the set of spoken training utterances by data processing hardware to adjust a text-to-speech (TTS) model to synthesize speech in the target speaker's voice and capture the atypical speech associated with the target speaker. For each of the plurality of non-spoken training text utterances, the method includes generating a synthetic speech representation of the corresponding non-spoken training text utterance as output from the adjusted TTS model by data processing hardware. The synthetic speech representation includes the target speaker's voice and captures the atypical speech associated with the target speaker. The method also includes training a speech conversion model by data processing hardware based on synthetic speech representations generated by the adjusted TTS model for each of the multiple non-verbal training text utterances.

[0005] Implementations of this disclosure may include one or more of the following optional features. In some implementations, for each synthesized speech representation generated by the adjusted TTS model, the method further includes: generating a corresponding audio waveform of synthesized standard fluent speech in the target speaker's voice as output from a speech conversion model using data processing hardware; generating a text representation of the corresponding audio waveform of the synthesized standard fluent speech generated as output from the speech conversion model using data processing hardware, as output from a text decoder; and determining a word error rate loss associated with the corresponding synthesized speech representation using data processing hardware. The word error rate loss is based on the text representation generated as output from the text decoder for the corresponding audio waveform of the synthesized standard fluent speech and the corresponding non-spoken training text utterance. In these implementations, the method also includes identifying a filtered set of synthesized speech representations by data processing hardware. Each filtered set of synthesized speech representations corresponds to a specific one of the synthesized speech representations generated as output from the speech conversion model that has a word error rate loss satisfying a word error rate loss threshold. In these implementations, training the speech conversion model based on the synthesized speech representation generated by the adjusted TTS model for each non-spoken text utterance in a plurality of non-spoken text utterances includes training the speech conversion model on the filtered set of synthesized speech representations. The speech conversion model is not trained on any of the synthesized speech representations generated as output from the speech conversion model, which have a word error rate loss that does not meet the word error rate loss threshold.

[0006] In some examples, the method further includes, when the speech conversion model has not been previously trained to convert the audio waveform of an input utterance spoken by a speaker with atypical speech of the same type as the atypical speech associated with the target speaker, the data processing hardware uses the set of spoken training utterances to adjust the speech conversion model to convert the audio waveform of the input utterance spoken by the target speaker with atypical speech into an audio waveform of synthesized standard fluent speech. Here, generating the corresponding audio waveform of synthesized standard fluent speech includes generating the corresponding audio waveform of synthesized standard fluent speech in the target speaker's speech as the output from the adjusted speech conversion model. In some examples, the text decoder resides on the speech conversion model. In other examples, the text decoder resides on a reference automatic speech recognition model separate from the speech conversion model.

[0007] In some implementations, the speech conversion model includes an end-to-end neural network configured to directly convert an input audio waveform into a corresponding output audio waveform. In these implementations, after training the speech conversion model, the method may further include receiving an input audio waveform at data processing hardware, the input audio waveform corresponding to utterances spoken by a target speaker associated with atypical speech; and the data processing hardware using the trained speech conversion model to convert the input audio waveform corresponding to the utterances spoken by the target speaker associated with atypical speech into an output audio waveform corresponding to a synthesized standard fluent speech representation of the utterances spoken by the target speaker.

[0008] In other embodiments, the speech conversion model includes an automatic speech recognition model configured to convert speech into corresponding text. In these embodiments, after training the speech conversion model, the method may further include receiving audio data corresponding to utterances spoken by a target speaker associated with atypical speech by data processing hardware; and converting the audio data corresponding to the utterances spoken by the target speaker associated with atypical speech by the data processing hardware using the trained speech conversion model into a standard text representation of the utterances spoken by the target speaker.

[0009] At least a portion of the multiple non-verbal training text utterances in the second part of the multiple training text utterances may include manually written text for specific phrases related to a specific domain. Optionally, the method may include performing a non-verbal text selection process by data processing hardware to obtain non-verbal training text utterances in the second part of the multiple training text utterances. The text selection process is configured to obtain a corpus of non-verbal text utterances. For each non-verbal text utterance in the corpus of non-verbal text utterances, the text selection process is configured to determine a first probability associated with the non-verbal text utterance appearing in a domain-specific language model, and to determine a second probability associated with the non-verbal text utterance appearing in a background language model. The background language model is trained on each non-verbal text utterance in the corpus of non-verbal text utterances. For each non-verbal text utterance in the corpus of non-verbal text utterances, the text selection process is also configured to determine a score based on the first probability, the second probability, and the number of words appearing in the corresponding non-verbal text utterance. Finally, the text selection process is configured to select the non-verbal text utterance with the best score N from the corpus of non-verbal text utterances as the non-verbal training text utterances in the second part of the multiple training text utterances.

[0010] In some implementations, the TTS model includes a pre-trained reference TTS model, which includes an encoder part and a decoder part. In these implementations, tuning the TTS model includes adjusting the pre-trained reference TTS model by tuning the parameters of the decoder part while keeping the parameters of the encoder part fixed.

[0011] Another aspect of this disclosure provides a system for training a speech conversion model that is personalized for a target speaker associated with atypical speech. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. These operations include obtaining a plurality of training text utterances. A first portion of the plurality of training text utterances includes a plurality of transcriptions from a set of spoken training utterances. Each spoken training utterance is spoken by a target speaker associated with atypical speech and includes a corresponding transcription paired with a corresponding non-synthesized speech representation of the corresponding spoken training utterance. A second portion of the plurality of training text utterances includes a plurality of non-spoken training text utterances associated with a specific domain in which the speech conversion model is trained to learn. Each non-spoken training text utterance is not paired with any corresponding spoken utterance. These operations also include using the set of spoken training utterances to adjust a text-to-speech (TTS) model to synthesize speech in the target speaker's speech and capture the atypical speech associated with the target speaker. For each of the plurality of non-spoken training text utterances, these operations include generating a synthesized speech representation of the corresponding non-spoken training text utterance as output from the adjusted TTS model. The synthesized speech representation includes the target speaker's voice and captures atypical speech associated with the target speaker. These operations also include training the speech-to-speech model based on the synthesized speech representation generated by the adjusted TTS model for each of the multiple non-verbal training text utterances.

[0012] Implementations of this disclosure may include one or more of the following optional features. In some implementations, for each synthesized speech representation generated by the adjusted TTS model, these operations further include: generating a corresponding audio waveform of synthesized standard fluent speech in the target speaker's voice as output from the speech conversion model; generating a text representation of the corresponding audio waveform of the synthesized standard fluent speech generated as output from the speech conversion model, as output from the text decoder; and determining a word error rate loss associated with the corresponding synthesized speech representation. The word error rate loss is based on the text representation generated as output from the text decoder for the corresponding audio waveform of the synthesized standard fluent speech and the corresponding non-spoken training text utterance. In these implementations, these operations also include identifying a filtered set of synthesized speech representations. Each filtered set of synthesized speech representations corresponds to a specific synthesized speech representation generated as output from the speech conversion model that has a word error rate loss that satisfies the word error rate loss threshold. In these implementations, training the speech conversion model based on the synthesized speech representation generated by the adjusted TTS model for each non-spoken text utterance in a plurality of non-spoken text utterances includes training the speech conversion model on the filtered set of synthesized speech representations. The speech conversion model is not trained on any of the synthesized speech representations generated as output from the speech conversion model, which have a word error rate loss that does not meet the word error rate loss threshold.

[0013] In some examples, these operations further include, when the speech conversion model has not been previously trained to convert the audio waveform of an input utterance spoken by a speaker with atypical speech of the same type as the atypical speech associated with the target speaker, adjusting the speech conversion model using the set of spoken training utterances to convert the audio waveform of the input utterance spoken by the target speaker with atypical speech into an audio waveform of synthesized standard fluent speech. Here, generating the corresponding audio waveform of synthesized standard fluent speech includes generating the corresponding audio waveform of synthesized standard fluent speech in the target speaker's speech as the output from the adjusted speech conversion model. In some examples, the text decoder resides on the speech conversion model. In other examples, the text decoder resides on a reference automatic speech recognition model separate from the speech conversion model.

[0014] In some implementations, the speech conversion model includes an end-to-end neural network configured to directly convert an input audio waveform into a corresponding output audio waveform. In these implementations, after training the speech conversion model, the operations may further include: receiving an input audio waveform corresponding to utterances spoken by a target speaker associated with atypical speech; and using the trained speech conversion model to convert the input audio waveform corresponding to the utterances spoken by the target speaker associated with atypical speech into an output audio waveform corresponding to a synthesized standard fluent speech representation of the utterances spoken by the target speaker.

[0015] In other embodiments, the speech conversion model includes an automatic speech recognition model configured to convert speech into corresponding text. In these embodiments, after training the speech conversion model, the method may further include: receiving audio data corresponding to utterances spoken by a target speaker associated with atypical speech; and using the trained speech conversion model to convert the audio data corresponding to the utterances spoken by the target speaker associated with atypical speech into a standard text representation of the utterances spoken by the target speaker.

[0016] At least a portion of the multiple non-verbal training text discourses in the second part of the multiple training text discourses may include manually written text for specific phrases related to a specific domain. Optionally, these operations may include performing a non-verbal text selection process to obtain non-verbal training text discourses in the second part of the multiple training text discourses. The text selection process is configured to obtain a corpus of non-verbal text discourses. For each non-verbal text discourse in the corpus of non-verbal text discourses, the text selection process is configured to determine a first probability associated with the non-verbal text discourse appearing in a domain-specific language model, and to determine a second probability associated with the non-verbal text discourse appearing in a background language model. The background language model is trained on each non-verbal text discourse in the corpus of non-verbal text discourses. For each non-verbal text discourse in the corpus of non-verbal text discourses, the text selection process is also configured to determine a score based on the first probability, the second probability, and the number of words appearing in the corresponding non-verbal text discourse. Finally, the text selection process is configured to select the non-verbal text discourse with the best score N from the corpus of non-verbal text discourses as the non-verbal training text discourse in the second part of the multiple training text discourses.

[0017] In some implementations, the TTS model includes a pre-trained reference TTS model, which includes an encoder part and a decoder part. In these implementations, tuning the TTS model includes adjusting the pre-trained reference TTS model by tuning the parameters of the decoder part while keeping the parameters of the encoder part fixed.

[0018] Details of one or more embodiments of this disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the specification, the drawings, and the claims. Attached Figure Description

[0019] Figure 1A This is a schematic diagram of an example speech conversion system that includes a speech-to-speech conversion model.

[0020] Figure 1B This is a schematic diagram of an example speech conversion system that includes a speech-to-text conversion model.

[0021] Figure 2A This is a diagram illustrating the personalized seed data collection phase used to obtain a set of oral training utterances spoken by the target speaker.

[0022] Figure 2B This is a diagram illustrating the data generation stage used to obtain multiple non-verbal training text utterances.

[0023] Figure 2C This is a diagram illustrating the adjustment phase used to adjust the reference text-to-speech (TTS) model and the reference speech-to-speech conversion model.

[0024] Figure 2D This is a schematic diagram of the verification and filtering stages of the filtered synthesized speech representation set used to identify the output of the adjusted TTS model.

[0025] Figure 2E This is a schematic diagram of the final training stage used to train the speech conversion model.

[0026] Figure 3 This is a flowchart illustrating an example of the operational setup for training a speech conversion model.

[0027] Figure 4 This is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0028] The same reference numerals in the various figures indicate the same elements. Detailed Implementation

[0029] Significant progress has been made in Automatic Speech Recognition (ASR), which introduces end-to-end (E2E) deep learning-based models to recognize speech from speakers with atypical speech patterns for accurate transcription. Atypical speech patterns can include, but are not limited to, speech impairments due to physical or neurological conditions (e.g., speakers with amyotrophic lateral sclerosis (ALS)), heavily accented speech, and sign language. Speech-to-speech systems can apply similar deep learning-based models to convert speech with atypical speech patterns into standard, fluent output speech. As used herein and unless otherwise specified, the terms “speech conversion system” and “speech conversion model” can refer to an ASR system / model in which input atypical speech is recognized and converted into corresponding text (e.g., transcription), or a speech-to-speech system / model in which input atypical speech is directly converted into standard, fluent synthetic speech without performing speech recognition. In other words, the speech-to-speech system / model is configured to convert an input audio waveform or spectrogram corresponding to atypical speech into an output audio waveform or spectrogram corresponding to standard fluent speech, without converting the input audio waveform into an intermediate representation (e.g., text or phonemes). As will become apparent, the speech conversion model, and the techniques used to train the speech conversion model, will enable users with atypical speech to converse with other human and voice interfaces (e.g., digital assistants) and be understood by other human and voice interfaces by recognizing and / or reproducing the user's expected speech.

[0030] One challenge in developing deep learning-based speech conversion and ASR models is that the parameters of these models tend to overfit the training data, making it difficult to generalize unseen data when the training data is not extensive enough. While training speech conversion and ASR models on larger training datasets improves accuracy, there is a lack of training data that provides both linguistic and acoustic diversity for target speakers with atypical speech and in a specific target domain. For example, to obtain training data with sufficient acoustic diversity, target speakers with atypical speech would have to record hours of spoken utterance, each utterance paired with a corresponding transcription. Furthermore, obtaining sufficient linguistic diversity for a specific target domain would require the utterance recorded from the speaker to include terms associated with that domain. For example, a target domain could include, but is not limited to, occupational fields (e.g., retinal specialist), educational disciplines (e.g., psychology lecturer), music, navigation, or weather. A target domain could similarly be a personalized domain associated with the target speaker, where specific terms associated with the personalized domain could include family member names, contact names, music artists / albums / songs from the music library associated with the target speaker, etc.

[0031] Text-to-speech (TTS) or speech synthesis systems have successfully applied Seq2Seq models to obtain state-of-the-art, natural-sounding synthesized speech that is indistinguishable from human speech to the human ear. Advantageously, non-spoken text utterances or plain text data can be readily and inexpensively obtained to generate synthesized speech for training and improving speech-to-speech models. For example, not only can non-spoken text utterances be used to increase the size of the training dataset, but they can also increase the linguistic diversity in the training data without the difficulty of obtaining transcribed speech (e.g., audio of human speech and its corresponding transcription).

[0032] The implementation scheme described herein relates to improving training data for training a speech conversion model (i.e., ASR or speech-to-speech conversion) personalized for a target speaker with atypical speech and for the target speaker's specific domain of interest. Specifically, the implementation scheme includes: sampling initial personalized seed data corresponding to transcribed acoustic data of recorded utterances spoken by a target speaker with atypical speech; and using the sampled seed data to adjust / tune a baseline text-to-speech (TTS) model. Here, "baseline TTS model" simply refers to a reference / existing TTS model previously trained to convert input text into synthesized standard speech in the speech of one or more predefined speakers. Here, the baseline TTS model is tuned / adjusted from the personalized seed data sampled from the target speaker to convert the input text into output synthesized speech in the target speaker's speech and with the target speaker's atypical speech patterns. The pre-trained baseline TTS model includes an encoder part and a decoder part, wherein adjusting the TTS model may include tuning / retraining the parameters of the decoder part while keeping the parameters of the encoder part fixed. By adjusting the TTS model in this way using personalized seed data, the adjusted TTS model can be used to convert text utterances, including terms or phrases associated with a specific domain, into synthetic training utterances that include the target speaker's voice and have associated atypical speech patterns of the target speaker. As will become apparent, the adjusted TTS model can generate a large number of synthetic training utterances for a specific domain and with atypical speech patterns in the target speaker's voice (i.e., synthetic voice), thereby increasing both linguistic and acoustic diversity when training the speech conversion model.

[0033] Synthetic training utterances and corresponding transcriptions generated by the adjusted TTS model are used to adjust / tune the baseline speech-to-speech model. Here, "baseline speech-to-speech model" refers to a reference / existing ASR model pre-trained on a general corpus of transcribed acoustic data to recognize typical / standard speech, or a reference / existing speech-to-speech model trained to map input audio waveforms (or spectrograms) from each of multiple utterances from a corpus spanning various speakers and recording conditions to corresponding output audio waveforms (or spectrograms) in the speech from a predefined standard speaker. Therefore, the synthetic training utterances provide sufficient linguistic and acoustic diversity to adjust / tune the general speech-to-speech model to recognize atypical speech spoken by the target speaker and for a specific domain, and / or to convert said atypical speech into standard text and / or standard fluent synthetic speech. In these embodiments, sampled seed data corresponding to the transcribed acoustic data of the recorded utterances spoken by the target speaker can be further used to adjust / tune the baseline speech-to-speech model. In other implementations, a combination of synthetic training utterances generated by the adjusted TTS model and sampled seed data is used to train the speech-to-speech model from scratch.

[0034] Figure 1AThe illustration shows speech conversion models 300 and 300a, which are configured to convert input audio data 102 corresponding to utterance 108 spoken by a target speaker 104 associated with atypical speech into output audio data 106 corresponding to a synthesized standard fluent speech representation of the same utterance 114 spoken by the target speaker 104. The associated speech conversion model 300 of the speech conversion system 100a includes a speech-to-speech (S2S) conversion model 300a, which is configured to directly convert input audio data 102 (e.g., input spectrogram) into output audio data 106 (e.g., output spectrogram) without performing speech recognition, or otherwise without generating any intermediate discrete representations (e.g., text or phonemes) from the input audio data 102. The S2S conversion model 300a includes: a spectrogram encoder 310 configured to encode input audio data 102 into a hidden feature representation (e.g., a series of vectors); and a spectrogram decoder 320 configured to decode the hidden representation into output audio data 106 corresponding to a synthesized standard fluent speech representation. For example, when the spectrogram encoder 310 receives input audio data 102 of utterance 108, it can process five frames of the audio and convert these five frames into ten vectors. These vectors are not transcriptions of the frames of audio data 102, but rather mathematical representations of the frames. Furthermore, the spectrogram decoder 320 can generate output audio data 106 corresponding to a synthesized standard fluent speech representation based on the vectors received from the spectrogram encoder 310. For example, the spectrogram decoder 320 can receive ten vectors representing five frames of audio from the spectrogram encoder 310. Here, the spectrogram decoder 320 can generate five frames of output audio data 106 corresponding to a synthesized standard fluent speech representation of utterance 114, which includes the expected words or word portions of the five frames of input audio data 102, but without the non-fluency of atypical speech.

[0035] In some examples, the S2S transformation model 300a also includes a text decoder ( Figure 2D250, the text decoder decodes the hidden features into text representations, such as phonemes or epigraphs. In these examples, the spectrogram decoder 320 and the text decoder 250 may correspond to parallel decoding branches of the S2S conversion model 300a, each branch receiving the hidden representation encoded by the spectrogram encoder 310 and emitting their respective output audio data 106 or text representations in parallel. The S2S conversion system 100a may further include a synthesizer 375 to synthesize the output audio data 106 into a time-domain waveform for audible output as the same utterance 114 of fluent standard fluent speech. The time-domain audio waveform includes an audio waveform that defines the amplitude of the audio signal over time. The synthesizer 375 may include a unit selection module or a WaveNet module for synthesizing the output audio data 106 into a time-domain waveform of synthesized standard fluent speech. In some embodiments, the synthesizer 375 includes a vocoder network, i.e., a neural vocoder, which is trained separately and tuned based on a Mel-frequency spectrogram to convert it into a time-domain audio waveform.

[0036] In the illustrated example, the target speaker 104 is associated with atypical speech, meaning that the target speaker 104 speaks in an atypical speech pattern that may be difficult to understand. Atypical speech patterns can include, but are not limited to, speech impairments due to physical or neurological conditions (e.g., a speaker with amyotrophic lateral sclerosis (ALS)), speech with a heavy accent, and sign language. For the example, the target speaker 104 has ALS and is associated with atypical speech due to ALS. Accordingly, the speech-to-speech conversion system 100a is trained to directly convert input audio data 102 corresponding to utterance 108 spoken by the target speaker 104 associated with ALS speech into output audio data 106 representing a synthesized standard fluent speech corresponding to the same utterance 108. Therefore, the synthesized standard fluent speech representation provided by the output audio data 106 improves the intelligibility of the ALS speech spoken by the target speaker 104. Without departing from the scope of this disclosure, an S2S conversion model 300a can be trained to directly convert input audio data 102 corresponding to utterance 108 associated with atypical speech in a first language into output audio data 106 corresponding to a synthesized standard fluent speech representation of the same utterance 108 in the same speech but in a different second language.

[0037] A computing device 110 associated with the target speaker 104 can capture the utterance 108 spoken by the target speaker 104 and transmit the corresponding input audio data 102 to a speech-to-speech conversion system 100a to convert it into output audio data 106. The speech-to-speech conversion system 100a can then transmit the output audio data 106, corresponding to a synthesized standard fluent speech representation of the same utterance 114 spoken by the target speaker 104, to another computing device 116 associated with the user 118, whereby the other computing device 116 audibly outputs a synthesized standard fluent speech representation of the utterance 108 spoken by the target speaker 104. In this example, the target speaker 104 and the user 118 converse with each other via their respective computing devices 110, 116, such as through a telephone call or other types of voice communication protocols (e.g., Internet Voice Protocol). Although the target speaker 104 and the other user 118 can speak the same language, the other user 118 may have difficulty understanding the target speaker 104 because the target speaker 104 has atypical speech due to ALS. Therefore, when the target speaker 104 speaks in atypical speech that may be difficult to understand (e.g., ALS speech), other users 118 who hear the synthesized standard fluent speech representation will find it easier to understand the utterance 108 intended by the target speaker 104. In other words, the synthesized standard fluent speech representation provides a more consistent rhythm that is easier for another user to understand than the original utterance 108 spoken by the target speaker in atypical speech. It is worth noting that the synthesized standard fluent speech representation is present in the speech of the target speaker 104.

[0038] In some other examples, the S2S conversion system 100a may alternatively transmit output audio data 106, corresponding to a synthesized standard fluent speech representation of the utterance spoken by the target speaker 104, to an output audio device to audibly output the synthesized standard fluent speech representation of the target speaker 104's speech to an audience. For example, the target speaker 104 may be a psychology professor giving lectures to a class of students, where the utterance spoken by the target speaker 104 includes medical terminology belonging to a specific domain (e.g., psychology). As will become apparent, the speech-to-speech conversion model 300a is trained to learn the linguistic diversity associated with a specific domain, as well as the acoustic diversity associated with a specific type of atypical speech of the target speaker 104.

[0039] Alternatively, the other computing device 116 can be associated with a downstream Automatic Speech Recognition (ASR) system, where the speech-to-speech conversion system 100a serves as a front end, providing output audio data 106 corresponding to a synthetic standard fluent speech representation as input to the ASR system for conversion into recognized text. The recognized text can be presented to other users 118 and / or provided to a Natural Language Understanding (NLU) system for further processing. The functionality of the speech-to-speech conversion system 100a can reside on a remote server 112, any one or both of the computing devices 110 and 116, or any combination of a remote server and computing devices 110 and 116. In some embodiments, as the target speaker 104 speaks the corresponding portion of a utterance as atypical speech, the S2S conversion model 300a continuously generates output audio data 106 corresponding to a synthetic standard fluent speech representation of the utterance. By continuously generating output audio data 106 corresponding to the synthesized standard fluent speech representation of portions of utterance 108 spoken by the target speaker 104, the rhythm of the conversation between the target speaker 104 and the user 118 (or listener) can be set more naturally. In some additional embodiments, the S2S conversation model 300a uses techniques such as voice activity detection, end indication, query end detection, and waits to determine / detect when the target speaker 104 stops speaking before converting the corresponding input audio data 102 of utterance 108 with atypical speech into the corresponding output audio data 106 of the synthesized standard fluent speech representation of the same utterance 114.

[0040] Now for reference Figure 1B In some embodiments, speech conversion systems 100, 100b include a speech-to-text conversion system 100b configured to convert input audio data 102 corresponding to utterance 108 spoken by a target speaker 104 associated with atypical speech into a standard text representation (i.e., transcription) 120 of the utterance 108. Figure 1A Similar to the S2S conversion system 100a, the speech-to-text conversion system 100b is configured not only to recognize specific types of atypical speech (e.g., ALS speech) associated with the target speaker 104, but also to recognize specific words and / or phrases associated with specific domains. These specific words and / or phrases may include proper nouns or other terms that are generally absent or insufficiently represented in the general training corpus used to train the general / baseline speech-to-text system.

[0041] Therefore, the speech-to-text conversion system 100b can correspond to a personalized automatic speech recognition (ASR) system for the target speaker 104, which can identify specific types of atypical speech patterns and language information of a specific domain of the target speaker to convert them into a corresponding standard text representation 120, which captures the intent of the original utterance 108 spoken by the target speaker 104 associated with atypical speech. Another user 118 ( Figure 1A A standard text representation 120 of utterance 108 can be obtained. In some configurations, the standard text representation 120 output from system 100b is processed, for example, by a natural language understanding (NLU) module on user device 110 or remote server 112 to execute user commands. Alternatively, a text-to-speech system (e.g., executing on any combination of user device 110 or remote server 112) can convert the transcription into synthesized speech for audible output to another device. The functionality of the speech-to-text conversion system 100b can reside on remote server 112, either or both of computing device 110, or any combination of remote server 112 and computing device 110.

[0042] The speech conversion model 300 associated with the speech-to-text conversion system 100b may include a speech-to-text conversion model 300b (interchangeably referred to as an automatic speech recognition (ASR) model 300b) configured to perform speech recognition on atypical speech utterance 108 by converting input audio data 102 into a standard text representation (i.e., transcription) 120 of utterance 108. The S2S conversion model 300a includes: an encoder 350 configured to encode the input audio data 102 into a hidden feature representation (e.g., a series of vectors); and a text decoder 250 configured to decode the hidden representation into standard transcription 120. For example, when the text encoder 350 receives input audio data 102 of utterance 108, the encoder 350 may process five frames of the audio and convert these five frames into ten vectors. These vectors are not transcriptions of the frames of audio data 102, but rather mathematical representations of the frames of audio data 102. Furthermore, the text decoder 250 can generate a standard transcription 120 based on the vectors received from the encoder 350. For example, the text decoder 250 can generate a series of words corresponding to the standard transcription 120 of utterance 180, which includes expected words or word portions from five frames of the input audio data 102. Without departing from the scope of this disclosure, an ASR model 300b can be trained to convert input audio data 102 corresponding to utterance 108 in a first language associated with atypical speech into a corresponding standard transcription of utterance 108 in a second language.

[0043] refer to Figure 1Aand 1B The speech conversion system 100 executes a training process 200, which is configured to train a speech conversion model 300, i.e. Figure 1A S2S conversion model 300a and / or Figure 1B The speech-to-text conversion model 300b. The following will refer to... Figures 2A to 2E In more detail, the training process 200 includes a personalized seed data collection phase 200a. Figure 2A ), Data generation stage 200b ( Figure 2B ), Adjustment phase 200c ( Figure 2C ), verification and filtering phase 200d ( Figure 2D ) and the final training phase 200e ( Figure 2E ).

[0044] refer to Figure 2A The personalized seed data collection phase 200a of the training process 200 includes obtaining a set of spoken training utterances 305, 305a-n associated with atypical speech by the target speaker 104. Here, each spoken training utterance 305 is spoken by the target speaker 104 and includes a corresponding transcription 302a, which is paired with a corresponding non-synthetic speech representation 304 for the corresponding spoken training utterance 305. Thus, the non-synthetic speech representation 304 is in the speech of the target speaker 104 and includes atypical speech patterns for the type of atypical speech (e.g., ALS speech) associated with the target speaker 104. The transcriptions 302a in the set of spoken training utterances 305 can form the first part of a plurality of training text utterances 302. Each transcription 302a can be a standard transcription in the native language of the target speaker 104. In some examples, some or all of the spoken training utterances 305 include words and / or phrases associated with a specific domain in which the speech conversion model 300 is trained to learn.

[0045] In some implementations, the personalized seed data collection phase 200a prompts the target speaker 104 to say each spoken training utterance 305 and records the utterance to obtain a corresponding non-synthetic speech representation 304. Each non-synthetic speech representation 304 obtained for the target speaker 104 can be paired with a corresponding transcription 302a of the spoken training utterance 305. Thus, each spoken training utterance 305 includes acoustic data 302a, 304 of a manually transcribed utterance by the target speaker 104. In the illustrated example, the personalized seed data collection phase 200a prompts the user to say each spoken training utterance 305 by displaying the corresponding transcription 302a on a graphical user interface of a computing device 110 associated with the target speaker 104. This can include individual prompts for each spoken training utterance, or it can include prompting the target speaker to say any number of consecutive spoken training utterances at once. Alternatively or additionally, the computing device 110 can audibly output prompts to cause the target speaker to say each training utterance (e.g., “Please say the following phrase”). The set of spoken training utterances 305 can be stored overlaid on memory hardware 420 ( Figure 4 The data is stored in 202 on the data storage device. In some examples, the personalized seed data collection phase 200a collects approximately five hundred (500) verbal training utterances 305.

[0046] refer to Figure 2B The data generation phase 200b of the training process 200 includes obtaining multiple non-verbal training text utterances 302b associated with a specific domain in which the speech conversion model 300 is being trained to learn. For example, the target speaker 104 could be a psychology professor, such that the specific domain includes psychology terminology from university-level psychology courses. Each non-verbal training text utterance 302b is not paired with any corresponding spoken utterance. The multiple non-verbal training text utterances 302b can form a second part of multiple training text utterances 302.

[0047] In some implementations, the data generation phase 200b is configured to select non-spoken training text utterances 302b from a corpus 402 of non-spoken texts. The corpus 402 of non-spoken texts comprises a large number of non-spoken training text utterances 302b from across a wide range of domains, and includes a much greater linguistic diversity than the specific domain in which the speech conversion model 300 is being trained to learn. As previously mentioned, the set of spoken training utterances 305 can be domain-specific because they are associated with a specific domain. The corpus 402 of non-spoken texts can be stored in the same or different data storage 202 as the spoken training utterances 305. The corpus 402 of non-spoken texts can be dynamically modified to incorporate new non-spoken training text utterances 302b. Simply using all the non-verbal training text utterances 302b in the non-verbal text corpus 402 is not feasible for the following reasons: i) for each sentence, the speech modality requires much more memory to encode than the text, making it impractical to transform all the text in the corpus 402; ii) the transformation between the speech and text modalities in TTS inference and speech-to-text model training also requires a lot of computation; and iii) the huge difference between the transcriptions 302a in the verbal training utterances 305 and the non-verbal training text utterances 302b in the non-verbal text corpus 402 requires intelligent strategies to balance their contributions.

[0048] Data generation phase 200b aims to select a subset of available non-verbal training text discourses 302b from the non-verbal text corpus 402 as references for the following text. Figure 2D and 2E A more detailed description of the TTS synthesis data. In other words, the data generation phase 200b aims to improve the matching between a selected subset of available non-verbal training text utterances 302b and the targeted specific domain (e.g., psychological terminology), which in turn reduces the computational resources required to utilize large amounts of non-domain-specific data. Thus, the data generation phase 200b reduces computational and memory costs by selecting the best-matching non-verbal training text utterances 302b for the specific domain that the speech conversion model 300 is being trained to learn.

[0049] In some examples, by simply providing a domain identifier (not shown) associated with a specific domain as input to a background language model (LM) 406 previously trained on the entire non-spoken text corpus 402, the data generation phase 200b selects from the corpus 402 a subset of available non-spoken training text utterances 302b that best match the specific domain. As previously mentioned, the non-spoken text corpus 402 spans multiple different domains. In these examples, the background LM 406 may include a maximum entropy (MaxEnt LM) capable of optionally accepting a domain identifier as input, as described in U.S. Patent No. 9,842,592, filed February 12, 2014, the contents of which are incorporated herein by reference in their entirety. Here, the domain identifier associated with a specific domain may allow the MaxEnt LM to output from the corpus 402 a subset of available non-spoken training text utterances 302b that may include words and / or phrases related to the specific domain. In some configurations, instead of evaluating the probability of words, the statistical language model operates in reverse mode to randomly generate text phrases that match the statistical distribution of words associated with a specific domain.

[0050] In additional examples and such Figure 2A As depicted, the data generation phase 200b performs a non-verbal text selection process using transcriptions 302a from a set of spoken training utterances 305 obtained from the target speaker 104 to select a subset of available non-verbal training utterances 302b that best matches a specific domain from the corpus 402. Here, the spoken training utterances 305 spoken by the target speaker 104 include words, phrases, and / or other terms related to the specific domain. Alternatively, in addition to or instead of transcriptions 302a from the set of spoken training utterances 305, different sets of transcribed utterances related to the specific domain can be used to select the non-verbal training utterances 302b. This provides the advantage that not all spoken training utterances 305 are required to belong to a specific domain.

[0051] During the first stage (STAGE A) of the non-verbal text selection process, the data generation stage 200b constructs two language models 404 and 406 to achieve contrastive selection of non-verbal training text utterances 302b. Here, the domain-specific language model (LM) 404 is trained on each transcription 302a in the set of spoken training utterances 305. It is assumed that the set of spoken training utterances 305 belongs to the specific domain in which the speech conversion model 300 is trained. On the other hand, the background LM 406 is trained on each non-verbal training text utterance 302b in the entire non-verbal text corpus 402. As previously mentioned, the non-verbal text corpus 402 spans multiple different domains. In some examples, the first stage uses n-gram language model training to construct the two language models 404 and 406. In other examples, the first stage uses neural network language model training to construct the two language models 404 and 406.

[0052] During the second stage (STAGE B) of the non-verbal text selection process, the data generation stage 200b uses two contrastive LMs 404 and 406 to determine the first probability associated with each word in the non-verbal training text utterance 302b appearing in a specific domain LM404. And determine the second probability associated with each word in the non-verbal training text utterance 302b appearing in the background LM 406. To evaluate each non-spoken training text utterance 302b in the non-spoken text utterance corpus 402. Subsequently, for each non-spoken training text utterance 302b in the non-spoken text utterance corpus 402, process 200 determines a score S at scorer 408 based on a first probability, a second probability, and the number of words #(w) appearing in the corresponding non-spoken text utterance 302b. For example, the score S for each non-spoken training text utterance 302b can be calculated as follows.

[0053]

[0054] After determining the scores, the data generation process 200b selects non-verbal training text utterances 302b with the best scores S for N, because these non-verbal training text utterances 302a best match the specific domain. The text corpus 402 may include billions of text utterances 302b. Instead of or in addition to selection from the available text corpus 402, the non-verbal training text utterances 302b may include handwritten text not generated from LM, targeting specific phrases / inappropriate nouns (e.g., family member names, contact names, games, etc.), and / or may be derived from specific topics of interest using a topic classifier associated with the specific domain. The non-verbal training text utterances 302b generated during the data generation phase 200b may include hundreds of utterances, and thus far exceed the number of verbal training utterances 305 collected from speakers. As will become apparent, the content of the non-verbal training text discourse 302b increases the linguistic diversity of the speech conversion model 300 as it is trained to learn the specific domain of language, while the corresponding synthesized speech representation generated from the non-verbal training text discourse 302b increases the acoustic diversity of atypical speech that the speech conversion model 300 is converting.

[0055] refer to Figure 2C The adjustment phase 200c of the training process 200 includes the use of... Figure 2A The personalized seed data collection phase 200a collects a set of spoken training utterances 305 to adjust both the text-to-speech (TTS) model 210 and the reference S2S conversion model 301 to synthesize speech in the speech of the target speaker 104 and capture atypical speech (e.g., ALS speech) associated with the target speaker 104. The adjustment phase 200c can... Figure 2B The data generation phase can occur before, after, or simultaneously with phase 200b.

[0056] The TTS model 210 can be pre-trained on input text to generate synthesized standard fluent speech from the speech of one or more predefined speakers. Therefore, terrestrial real speech samples can be obtained from speakers with typical speech patterns to train the TTS model 210.

[0057] Similarly, the reference S2S conversion model 301 is pre-trained on input audio data corresponding to a large number of utterances spoken by various speakers to produce corresponding output audio data, which captures the same content in the speech of a single predefined speaker. It is noteworthy that utterances from various speakers can include typical speech patterns, various types of atypical speech patterns (e.g., accents across different dialects, irregular speech across different neural states), and background noise. For example, the reference S2S conversion model 301 can include an end-to-end trained speech-to-speech conversion model, which can... https: / / arxiv.org / pdf / 1904.04169.pdf This is derived from and incorporated herein by reference to Parrotron: An End-to-End Speech-to-Speech Conversion Model and its Applications to Hearing-Impaired Speech and Speech Separation. Referring to S2S conversion model 301, arbitrary speech can be normalized using sequence-to-sequence, potentially including background noise, and the same content is generated in the speech of a single predefined target speaker. The source speech can come from any speaker or accent and can contain complex prosodic patterns, defects, atypical speech, and background noise, all of which are removed through the normalization process when the first audio data is converted into clean second audio data with a fixed accent and consistent pronunciation and prosody. In other words, the system can be used to project away all non-linguistic information, including speaker characteristics, and retain only what is said, not who, how, or where it is said.

[0058] Since TTS model 210 is pre-trained to generate synthesized standard fluent speech from speech other than that of target speaker 104, and reference S2S conversion model 301 is pre-trained on utterances from various speakers associated with both typical speech and various types of atypical speech, the adjustment phase 200c aims to adapt models 210 and 301 to both the speech of target speaker 104 and the specific type of atypical speech (ALS speech) associated with target speaker 104. In some examples, however, when the reference S2S conversion model is pre-trained to convert input audio data associated with the specific type of atypical speech associated with target speaker 104, the adjustment phase 200c abandons adjusting the reference S2S conversion model 301, because model 301 has already been trained to convert the same type of atypical speech associated with target speaker 104 into standard fluent speech.

[0059] The adjustment phase 200c adjusts the TTS model 210 to convert the transcriptions 302a in the set of spoken training utterances 305 into corresponding synthesized speech 306 in the speech of the target speaker 104, capturing atypical speech associated with the target speaker 104. In some embodiments, the TTS model 210 includes an encoder 312 and a decoder 314 that cooperate to process the transcriptions 302a to adjust the TTS model 210 to generate a time-domain audio waveform of the synthesized speech 306 in the speech of the target speaker 104, capturing atypical speech associated with the target speaker 104. A time-domain audio waveform is an audio waveform that defines the amplitude of an audio signal over time.

[0060] Encoder 212 may be an encoder neural network 212 configured to receive transcription 304 as a character sequence and generate a fixed-length context vector 213 for each Mel-frequency spectrogram that decoder 214 will later generate. Since the TTS model 210 is tuned to produce synthesized speech that captures atypical speech in the speech of target speaker 104, the tuning phase 200c may include tuning / retraining the parameters of decoder 214 while keeping the parameters of encoder 212 fixed. Decoder 214 may be a neural network configured to receive the fixed-length context vectors 213 generated by encoder neural network 212 as input and generate corresponding frames of Mel-frequency spectrograms as output for each fixed-length context vector 213. A Mel-frequency spectrogram is a frequency domain representation of sound. Mel-frequency spectrograms emphasize lower frequencies, which are crucial for speech intelligibility, while neglecting higher frequencies, which consist primarily of fricatives and other noise bursts and generally do not require high-fidelity modeling. The synthesized speech 306 may include a synthesized speech representation associated with a Mel-frequency spectrogram output from the decoder 214, or the synthesized speech 306 may be a time-domain audio waveform generated by a vocoder (not shown) based on the Mel-frequency spectrogram output from the decoder 214. The decoder 214 may include a post-network that can adapt to the target speaker 104 by refining the acoustic characteristics of the Mel-frequency spectrogram generated by the decoder to better match the speech and atypical speech associated with the target speaker 104.

[0061] The adjustment phase 200c adjusts the reference S2S conversion model 301 to convert the non-synthetic speech representation 304 from the set of spoken training utterances 305 into synthetic standard fluent speech in the speech of the target speaker 104. As previously mentioned, the non-synthetic speech representation 304 is associated with the utterance spoken by the target speaker 104 and thus captures atypical speech associated with the target speaker 104. Here, the adjustment phase 200c can use the corresponding transcription 302a as the ground truth for the spectrogram decoder 320 to accurately decode / produce the synthetic standard fluent speech 316, which conveys the intended content of the input non-synthetic speech representation 304.

[0062] refer to Figure 2D For in Figure 2BDuring the data generation phase 200b, for each of the multiple non-verbal training text utterances 302b obtained, the verification and filtering phase 200d generates a synthesized speech representation 306 corresponding to the non-verbal training text utterance 302b as output from the adjusted TTS model 210. Since the verbal training utterances 305 are used to adjust the TTS model 210, each synthesized speech representation 306 includes the target speaker's voice and captures atypical speech associated with the target speaker 104. Therefore, each synthesized speech representation 306 output from the adjusted TTS model 210 is paired with a corresponding one of the multiple non-verbal training text utterances 302b.

[0063] In the example shown, the verification and filtering phase 200d further uses the adjusted S2S conversion model 301 to generate a corresponding audio waveform of the synthesized standard fluent speech 316 in the speech of the target speaker 104 for each synthesized speech representation 306 output from the adjusted TTS model 210, and then uses the text decoder 250 to generate a text representation 318 for the corresponding audio waveform of the synthesized standard fluent speech 316 generated as the output from the adjusted S2S conversion model 301. As previously mentioned, if the reference S2S speech conversion model 301 was previously trained to convert the same type of atypical speech, the reference S2S speech conversion model 301 does not need to (but can) be adjusted before generating the audio waveform of the synthesized standard fluent speech from the synthesized speech representation 306. In some examples, the text decoder 250 resides on an adjusted S2S speech conversion model 301, which outputs a text representation 318 from the text decoder 250 and, in parallel, the corresponding audio waveform of synthesized standard fluent speech 316 from the spectrogram decoder 320. In other examples, the text decoder 250 resides on a reference automatic speech recognition model separate from the speech conversion model.

[0064] Subsequently, for each synthesized speech representation 306 output from the adjusted TTS model 210, the verification and filtering stage 200d applies a supervised loss term module 340 to determine a word error rate loss 342 associated with the corresponding synthesized speech representation 306. Specifically, the word error rate loss 342 is based on a text representation 318 output from the text decoder 250 for the synthesized standard fluent speech 306 and the corresponding non-spoken training text utterance 302b. Notably, the non-spoken training text utterance 302b is used as input to the adjusted TTS model 210 to be transformed into the resulting synthesized speech representation 306 and as ground truth for verifying the corresponding text representation 318 output from the decoder 250. In the example shown, the verification and filtering stage 200d verifies each synthesized speech representation 306 output from the adjusted TTS model 210 by determining whether the corresponding word error rate loss 342 meets a word error rate loss threshold. When the corresponding word error rate loss 342 meets the word error rate loss threshold, the corresponding synthesized speech representation 306 is stored in the filtered synthesized speech representation set 306A for use in training the speech conversion model 300. When the corresponding word error rate loss 342 fails to meet the word error rate loss threshold, the corresponding synthesized speech representation 306 is discarded and not used to train the speech conversion model 300.

[0065] In the example shown, the supervised loss module 340 determines the word error rate loss 342 based on the number of misidentified words between the corresponding text representation 318 and the corresponding non-spoken training text utterance 302b, which serves as the ground truth. For example, a word error rate loss of 60% indicates that 40% of the words in the corresponding text representation 318 are misidentified by the text decoder 250 from the corresponding synthesized standard fluent speech 316. The word error rate loss threshold can be set to any value and can be adjusted as needed. In one example, a word error rate loss threshold of 70% indicates that at least 70% of the words in the corresponding text representation 318 must be accurately identified so that the corresponding synthesized speech representation 306 meets the word error rate loss threshold and is therefore accepted into the filtered set of synthesized speech representations 306A. The value set of word error rate loss 342 and word error rate loss threshold is used as a proxy to only identify synthesized speech representations 306 suitable for training the speech conversion model 300 (i.e., the filtered set of synthesized speech representations 306A) and discard any synthesized speech representations 306 that are not suitable for training. The discarded synthesized speech representation 306 indicates that the adjusted S2S conversion model 301 cannot accurately convert the input audio waveform (or spectrogram) into intelligible synthesized standard fluent speech 316 because the text decoder 250 produces the corresponding text representation 318 with a word error rate loss 342 that fails to meet the word error rate loss threshold. Simply put, the discarded synthesized speech representation 306 is associated with a small percentage of the total number of synthesized speech representations 306 generated by the adjusted TTS model 210, indicating that it is difficult to understand or falls far short of ideal in terms of content quality and style.

[0066] While it should be understood that the steps of using the adjusted S2S conversion model 301 to generate synthetic standard fluent speech 316 from each atypical synthetic speech representation 306 and applying speech recognition by the text encoder 250 to the synthetic standard fluent speech 316 to generate a text representation 318 help validate the synthetic speech representation 306 used to train the speech conversion model 300, these steps can be optional. For example, the adjusted TTS model 210 can be used to generate a corresponding synthetic speech representation 306 for each unpaired unspoken training text utterance 302b, and all synthetic speech representations 306 can be used to train the speech conversion model 300 without applying the adjusted S2S conversion model 301, the text decoder 250, and the supervised loss term module 340 to identify the filtered set of synthetic speech representations 306A.

[0067] Now for reference Figure 2EThe final training phase 200e includes training the speech conversion model 300 based on the synthesized speech representation 306 generated by the adjusted TTS model 210 for each of the multiple non-spoken training text utterances 302b. More specifically, the final training phase 200e of the training process 200 trains the speech conversion model 300 on a filtered set of synthesized speech representations 306A, rather than on... Figure 2D The word error rate loss 342 discussed in the verification and filtering phase 200d is any one of the synthesized speech representations 306 that are discarded because they do not meet the word error rate loss threshold, and is used to train the speech conversion model 300.

[0068] In some implementations, training the speech conversion model 300 includes training an S2S speech conversion model 300a personalized for the target speaker 104 to directly convert an input audio waveform associated with the speaker's atypical speech into a corresponding output waveform of standard fluent speech in the speaker's voice. The number of synthetic speech representations 306A in the filtered set of synthetic speech representations 306A provides sufficient acoustic diversity for training the S2S speech conversion model 300a to learn both the voice of the target speaker 104 and the types of atypical speech associated with the target speaker 104. Furthermore, since each synthetic speech representation 306 used to train the S2S conversion model 300a is associated with a specific domain, the number of synthetic speech representations 306A in the filtered set of synthetic speech representations 306A also provides sufficient linguistic diversity for training the S2S speech conversion model 300a to learn specific words, names, phrases, or other terms associated with a specific domain that may exist in the atypical speech utterance 108 spoken by the target speaker 104.

[0069] In some examples, training the S2S conversion model 300a involves adjusting the reference S2S conversion model 301 on a filtered set of synthesized speech representations 306A. Figure 2C Each filtered synthesized speech representation is paired with a corresponding one in the non-verbal training text utterance 302b. As previously mentioned, the reference S2S conversion model 301 was previously trained using utterances spoken by various different speakers with different speaking styles and tones to produce standard fluent speech in the predefined speaker's tone. Therefore, the reference S2S conversion model 301 is tuned on the filtered set of synthesized speech representations 306A to provide a trained S2S conversion model 300a personalized for the target speaker 104 to convert any input audio waveform (e.g., ) corresponding to utterances spoken by the target speaker 104 associated with a particular type of atypical speech. Figure 1A The input audio data 102) is converted into an output audio waveform (e.g.,) corresponding to the synthesized standard fluent speech representation 316 of the same utterance spoken by the target speaker. Figure 1A The output audio data (106). In these examples, it is possible to obtain the audio data from... Figure 2A During the personalized seed data collection phase 200a, the non-synthetic speech representation 304 of the oral training utterances 305 collected from the target speaker 104 is further adjusted / tuned on the trained S2S speech conversion model 300a.

[0070] In other examples, training the S2S conversion model 300a involves training the S2S conversion model 300a from scratch. Training the S2S conversion model 300a from scratch may include training on a mixture of filtered synthetic speech representations set 306A and non-synthetic speech representations 304 from a set of spoken training utterances 305, with each filtered synthetic speech representation paired with a corresponding one in the non-spoken training text utterances 302b, and each non-synthetic speech representation paired with a corresponding transcription 302a. Here, the model 300 can be trained in parallel on corresponding batches of non-synthetic speech representations 304 and synthetic speech representations 306, such that the model 300a outputs corresponding synthetic standard fluent speech representations 316 in parallel.

[0071] When training the S2S conversion model 300a, whether by adjusting the reference S2S conversion model 301 or starting from scratch, the final training phase 200e of the training process 200 can use a stochastic optimization algorithm (such as stochastic gradient descent) to train model 300a via backpropagation. For example, an automatic speech recognizer (e.g., Figure 2D The text decoder 250 can generate a text representation 318 for each corresponding synthetic standard fluent speech representation 316 output from the S2S conversion model 300a, which can be compared with corresponding training text utterances 302a, 302b used as ground truth to obtain a word error rate loss 342. Figure 2D While the automatic speech recognizer can be a standalone automatic speech recognizer, it can also include a text decoder of the S2S conversion model 300a, which produces a text representation 318 in parallel with a corresponding synthesized standard fluent speech representation 316 produced by the spectrogram decoder 320. Regardless of where the resulting text representation 318 is generated, a stochastic optimization algorithm can use a word error rate loss 342 to define a corresponding loss function (e.g., a cross-entropy loss function) based on the difference between the actual output (e.g., text representation 318) and the target output (transcribed and non-spoken training text utterances 302a, 302b). For example, the loss function can be computed for each batch of training examples and then differentiated relative to each weight in model 300a.

[0072] In some additional embodiments, training the speech conversion model 300 may additionally or alternatively include training a speech-to-text conversion model 300b (interchangeably referred to as the “ASR model”) personalized for the target speaker 104 to convert input audio waveforms associated with the speaker’s atypical speech into corresponding text, which includes a standard transcription 120 of the utterance spoken by the target speaker 104. The number of synthetic speech representations 306A in the filtered set of synthetic speech representations 306A provides sufficient acoustic diversity for training the ASR model 300b to learn to recognize utterances 108 spoken with the type of atypical speech associated with the target speaker 104. Furthermore, since each synthetic speech representation 306 used to train the ASR model 300b is associated with a specific domain, the number of synthetic speech representations 306A in the filtered set of synthetic speech representations 306A also provides sufficient linguistic diversity for training the ASR model 300b to learn to recognize specific words, names, phrases, or other terms associated with a specific domain that may be present in the atypical speech utterances 108 spoken by the target speaker 104.

[0073] Similar to the S2S speech conversion model 300a, training the ASR model 300b may include adjusting a reference ASR model 300b, which has previously been trained on a general corpus of training utterances spoken by various speakers with different speaking styles. Here, the reference ASR model 300b may be adjusted on a filtered set of synthesized speech representations 306A, each synthesized speech representation paired with a corresponding one in the non-spoken training text utterances 302b, and then the reference ASR model 300b may be adjusted from... Figure 2A During the personalized seed data collection phase 200a, further adjustments / tuning are made on the non-synthetic speech representations 304 of the oral training utterances 305 collected from the target speaker 104. On the other hand, an ASR model 300b can be trained de novo using a mixture of filtered synthetic speech representations 306A and non-synthetic speech representations 304 from the oral training utterances 305, with each filtered synthetic speech representation paired with a corresponding one in the non-spoken training text utterances 302b, and each non-synthetic speech representation paired with a corresponding transcription 302a.

[0074] In other examples, training the S2S conversion model 300a involves training the S2S conversion model 300a from scratch. Training the S2S conversion model 300a from scratch may include training on a mixture of filtered synthetic speech representations set 306A and non-synthetic speech representations 304 from a set of spoken training utterances 305, with each filtered synthetic speech representation paired with a corresponding one in the non-spoken training text utterances 302b, and each non-synthetic speech representation paired with a corresponding transcription 302a. Here, the model 300 can be trained in parallel on corresponding batches of non-synthetic speech representations 304 and synthetic speech representations 306, such that the model 300a outputs corresponding synthetic standard fluent speech representations 316 in parallel.

[0075] The final training phase 200e of the training process 200 can use stochastic optimization algorithms (such as stochastic gradient descent) to train the ASR model 300b via backpropagation. The details above regarding training the S2S transformation model 300a discuss the use of stochastic optimization algorithms to train the ASR model 300b.

[0076] Figure 3 A flowchart illustrating an example setup for training a speech conversion model 380, personalized for a target speaker associated with atypical speech, is provided. See also... Figure 1A , 1B and Figures 2A to 2E The training process is described in a flowchart (200). Data processing hardware (410). Figure 4 ) can be executed by memory hardware 420 (which communicates with data processing hardware 410). Figure 4 The instructions on the computing device 400 are used to execute the operation of method 380. Data processing hardware 410 and memory hardware 420 can reside on the computing device 400. Figure 4 On, such as Figure 1A and 1B On a remote server 112 and / or user computing device 110. At operation 382, ​​method 380 obtains a plurality of training text utterances 302a, 302b. Here, the first portion of the plurality of training text utterances includes a plurality of transcriptions 302a from a set of spoken training utterances 305, 305a-n. Each spoken training utterance 305 is spoken by a target speaker 104 associated with atypical speech and includes a corresponding transcription 302a, which is paired with a corresponding non-synthetic speech representation 304 of the corresponding spoken training utterance 305. Figure 2A During the personalized seed data collection phase 200a, a set of spoken training utterances 305 is obtained. A second part of the multiple training text utterances includes multiple non-spoken training text utterances 302b associated with a specific domain in which the speech conversion model 300 is trained to learn. Each non-spoken training text utterance 302b is not paired with any corresponding spoken utterance. [The last part, "can be...", appears to be incomplete and requires further context.] Figure 2B During the data generation phase 200b, multiple non-verbal training text discourses 302b were obtained.

[0077] At operation 384, method 380 includes using a set of spoken training utterances 305 to adjust the text-to-speech (TTS) model 210 to synthesize speech within the target speaker's utterance and capturing atypical speech associated with the target speaker. (See reference) Figure 2C The adjustment phase 200c describes the details of adjusting the TTS model 210.

[0078] At operation 386, for each of the plurality of non-verbal training text utterances 302b, method 380 further includes generating a synthetic speech representation 306 corresponding to the non-verbal training text utterance 302b as output from the adjusted TTS model 210. Here, each synthetic speech representation 306 output from the adjusted TTS model 210 is within the speech of the target speaker 104 and captures atypical speech associated with the target speaker 104. At operation 388, method 380 further includes training a speech conversion model 300 based on the synthetic speech representations 306 generated by the adjusted TTS model 210 for each of the plurality of non-verbal training text utterances 302b. Training the speech conversion model 300 includes training at least one of an S2S conversion model 300a or a speech-to-text (e.g., ASR) model 300b.

[0079] A software application (i.e., a software resource) can refer to computer software that instructs a computing device to perform a task. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and game applications.

[0080] Non-transitory memory can be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by a computing device. Non-transitory memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used in firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0081] Figure 4 This is a schematic diagram of an example computing device 400 that can be used to implement the systems and methods described in this document. The computing device 400 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit the implementation of the inventions described and / or claimed in this document.

[0082] Computing device 400 includes a processor 410, memory 420, storage device 430, a high-speed interface / controller 440 connected to memory 420 and high-speed expansion port 450, and a low-speed interface / controller 460 connected to low-speed bus 470 and storage device 430. Each of components 410, 420, 430, 440, 450, and 460 is interconnected using various buses and may be mounted on a common motherboard or otherwise. Processor 410 can process instructions for execution within computing device 400, including instructions stored in memory 420 or on storage device 430 for displaying graphical information of a graphical user interface (GUI) on external input / output devices, such as a display 480 coupled to high-speed interface 440. In other embodiments, multiple processors and / or multiple buses may be used with multiple memories and various types of memory, depending on the situation. Moreover, multiple computing devices 400 may be connected, with each device providing a portion of the necessary operation (e.g., as a server group, blade server cluster, or multiprocessor system).

[0083] Memory 420 stores information non-temporarily within computing device 400. Memory 420 may be a computer-readable medium, a volatile memory cell, or a non-volatile memory cell. Non-temporarily stored memory 420 may be a physical means for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) for use by computing device 400. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used in firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0084] Storage device 430 provides mass storage for computing device 400. In some embodiments, storage device 430 is a computer-readable medium. In various embodiments, storage device 430 may be a floppy disk device, hard disk device, optical disk device, magnetic tape device, flash memory or other similar solid-state storage device, or an array of devices, including devices in a storage area network or other configuration. In additional embodiments, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as memory 420, storage device 430, or memory on processor 410.

[0085] High-speed controller 440 manages bandwidth-intensive operations of computing device 400, while low-speed controller 460 manages lower bandwidth-intensive operations. This allocation of responsibilities is merely exemplary. In some embodiments, high-speed controller 440 is coupled to memory 420, display 480 (e.g., via a graphics processor or accelerator), and high-speed expansion port 450 which can accept various expansion cards (not shown). In some embodiments, low-speed controller 460 is coupled to storage device 430 and low-speed expansion port 490. Low-speed expansion port 490, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), may be coupled to one or more input / output devices, such as keyboards, pointing devices, scanners, or networking devices such as switches or routers, for example, via a network adapter.

[0086] As shown in the figure, the computing device 400 can be implemented in a variety of different forms. For example, the computing device can be implemented as a standard server 400a, or multiple times in a group of such servers 400a, as a laptop computer 400b, or as part of a rack server system 400c.

[0087] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be dedicated or general-purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transfer data and instructions to the storage system, at least one input device, and at least one output device.

[0088] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages, and / or in assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transitory computer-readable medium, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0089] The processes and logic flows described in this specification can be executed by one or more programmable processors, also known as data processing hardware, which execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by dedicated logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). For example, processors suitable for executing computer programs include both general-purpose microprocessors and dedicated microprocessors, as well as any one or more processors of any kind of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or operatively coupled to receive data from or transfer data to said one or more mass storage devices, or both. However, a computer does not need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0090] To provide interaction with a user, one or more aspects of this disclosure can be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, or a touchscreen, to display information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input generated from the user can be received in any form, including sound, speech, or tactile input. Additionally, a computer can interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on the user's client device in response to a request received from a web browser.

[0091] Several embodiments have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other embodiments are within the scope of the appended claims.

Claims

1. A method for training a speech conversion model, the method comprising: Multiple training text utterances were obtained from the data processing hardware, among which: The first portion of the plurality of training text utterances includes multiple transcriptions from a set of spoken training utterances, each spoken training utterance being spoken by a target speaker associated with atypical speech and including a corresponding transcription paired with a corresponding non-synthetic speech representation of the corresponding spoken training utterance, wherein the non-synthetic speech representation is situated within the speech of the target speaker and includes atypical speech patterns for the atypical speech associated with the target speaker; and The second part of the plurality of training text utterances includes a plurality of non-verbal training text utterances associated with a specific domain in which the speech conversion model is trained to learn, each non-verbal training text utterance not paired with any corresponding verbal utterance; The data processing hardware uses the spoken training utterance set to adjust the text-to-speech model to synthesize speech in the voice of the target speaker and capture the atypical speech associated with the target speaker; For each of the plurality of non-verbal training text utterances, the data processing hardware generates a synthesized speech representation of the corresponding non-verbal training text utterance as the output from the adjusted text-to-speech model. The synthesized speech representation includes the target speaker's voice and captures the atypical speech associated with the target speaker. The speech conversion model is trained by the data processing hardware based on the synthesized speech representation generated by the adjusted text-to-speech model for each of the plurality of non-verbal training text utterances. The speech conversion model includes an end-to-end neural network configured to directly convert an input audio waveform into a corresponding output audio waveform.

2. The method according to claim 1, further comprising: For each synthesized speech representation generated by the adjusted text-to-speech model: The data processing hardware generates the corresponding audio waveform of the synthesized standard fluent speech in the target speaker's voice, as the output from the speech conversion model; The data processing hardware generates a text representation of the corresponding audio waveform of the synthesized standard fluent speech generated as the output of the speech conversion model, as the output of the text decoder; as well as The text representation generated by the data processing hardware based on the corresponding audio waveform for synthesized standard fluent speech as the output from the text decoder, and the corresponding non-spoken training text utterance, are used to determine the word error rate loss associated with the corresponding synthesized speech representation; as well as The data processing hardware identifies a set of filtered synthetic speech representations, each filtered synthetic speech representation corresponding to one of the synthetic speech representations generated as output from the speech conversion model, which has a word error rate loss that satisfies a word error rate loss threshold. The speech conversion model is trained on the filtered set of synthesized speech representations, but not on any of the synthesized speech representations generated as output from the speech conversion model that have a word error rate loss that does not meet the word error rate loss threshold.

3. The method of claim 2, further comprising, when the speech conversion model has not previously been trained to convert the audio waveform of an input utterance spoken by a speaker having the same type of atypical speech as the atypical speech associated with the target speaker: The data processing hardware uses the spoken training utterance set to adjust the speech conversion model to convert the audio waveform of the input utterance spoken by the target speaker with atypical speech into the audio waveform of synthesized standard fluent speech. The corresponding audio waveform for generating synthesized standard fluent speech includes the corresponding audio waveform for generating synthesized standard fluent speech in the voice of the target speaker as the output from the adjusted speech conversion model.

4. The method of claim 2, wherein the text decoder resides on the speech conversion model.

5. The method of claim 2, wherein the text decoder resides on a reference automatic speech recognition model separate from the speech conversion model.

6. The method of claim 1, further comprising, after training the speech conversion model: The data processing hardware receives an input audio waveform corresponding to a utterance spoken by the target speaker associated with atypical speech; and The data processing hardware uses a trained speech conversion model to convert the input audio waveform, corresponding to the utterance spoken by the target speaker associated with atypical speech, into an output audio waveform that corresponds to a synthesized standard fluent speech representation of the utterance spoken by the target speaker.

7. The method of claim 1, wherein the speech conversion model further comprises an automatic speech recognition model configured to convert speech into corresponding text.

8. The method of claim 7, further comprising, after training the speech conversion model: Audio data corresponding to utterances by the target speaker associated with atypical speech is received at the data processing hardware; and The data processing hardware uses a trained speech conversion model to convert the audio data corresponding to the utterance spoken by the target speaker associated with atypical speech into a transcription of the utterance spoken by the target speaker.

9. The method of claim 1, wherein at least a portion of the plurality of non-verbal training text utterances in the second portion of the plurality of training text utterances comprises manually written text for specific phrases associated with the particular domain.

10. The method of claim 1, further comprising performing a non-verbal text selection process by the data processing hardware to obtain the non-verbal training text utterances in the second portion of the plurality of training text utterances, wherein the text selection process is configured to: Obtain a corpus of non-spoken text discourse; For each non-spoken text utterance in the corpus: Determine a first probability associated with the occurrence of the non-verbal text utterance in a domain-specific language model; Determine a second probability associated with the occurrence of the non-spoken text utterance in a background language model, the background language model being trained on each non-spoken text utterance in the corpus of non-spoken text utterances; and The score is determined based on the first probability, the second probability, and the number of words appearing in the corresponding non-spoken text utterance; and The non-verbal text discourse with the best score N from the corpus of non-verbal text discourses is selected as the non-verbal training text discourse in the second part of the plurality of training text discourses.

11. The method according to any one of claims 1 to 10, wherein: The text-to-speech model includes a pre-trained reference text-to-speech model, which includes an encoder part and a decoder part. as well as Adjusting the text-to-speech model involves tuning the pre-trained reference text-to-speech model by keeping the parameters of the encoder part fixed while tuning the parameters of the decoder part.

12. A system for training a speech conversion model, comprising: Data processing hardware; as well as Memory hardware communicating with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations, the operations including: Multiple training text utterances were obtained, among which: The first portion of the plurality of training text utterances includes multiple transcriptions from a set of spoken training utterances, each spoken training utterance being spoken by a target speaker associated with atypical speech and including a corresponding transcription paired with a corresponding non-synthetic speech representation of the corresponding spoken training utterance, wherein the non-synthetic speech representation is situated within the speech of the target speaker and includes atypical speech patterns for the atypical speech associated with the target speaker; and The second part of the plurality of training text utterances includes a plurality of non-verbal training text utterances associated with a specific domain in which the speech conversion model is trained to learn, each non-verbal training text utterance not paired with any corresponding verbal utterance; The text-to-speech model is adjusted using the oral training utterance set to synthesize speech in the voice of the target speaker and capture the atypical speech associated with the target speaker; For each of the plurality of non-verbal training text utterances, a synthesized speech representation of the corresponding non-verbal training text utterance is generated as the output from the adjusted text-to-speech model, the synthesized speech representation including the target speaker's voice and capturing the atypical speech associated with the target speaker; and The speech conversion model is trained based on the synthesized speech representation generated by the adjusted text-to-speech model for each of the plurality of non-verbal training text utterances, wherein the speech conversion model includes an end-to-end neural network configured to directly convert an input audio waveform into a corresponding output audio waveform.

13. The system of claim 12, wherein the operation further comprises: For each synthesized speech representation generated by the adjusted text-to-speech model: Generate the corresponding audio waveform of the synthesized standard fluent speech in the target speaker's voice as the output from the speech conversion model; A text representation is generated from the corresponding audio waveform of the synthesized standard fluent speech generated as the output of the speech conversion model, as the output of the text decoder; as well as The word error rate loss associated with the corresponding synthesized speech representation is determined based on the text representation generated by the corresponding audio waveform of synthesized standard fluent speech as the output of the text decoder and the corresponding non-verbal training text utterance; as well as The set of filtered synthetic speech representations is identified, each filtered synthetic speech representation corresponding to one of the synthetic speech representations generated as the output from the speech conversion model, which has a word error rate loss that satisfies a word error rate loss threshold. The speech conversion model is trained on the filtered set of synthesized speech representations, but not on any of the synthesized speech representations generated as output from the speech conversion model that have a word error rate loss that does not meet the word error rate loss threshold.

14. The system of claim 13, wherein the operation further comprises when the speech conversion model has not been previously trained to convert the audio waveform of an input utterance spoken by a speaker having the same type of atypical speech as the atypical speech associated with the target speaker: The speech conversion model is tuned using the spoken training utterance set to convert the audio waveform of the input utterance spoken by the target speaker with atypical speech into the audio waveform of synthesized standard fluent speech. The corresponding audio waveform for generating synthesized standard fluent speech includes the corresponding audio waveform for generating synthesized standard fluent speech in the voice of the target speaker as the output from the adjusted speech conversion model.

15. The system of claim 13, wherein the text decoder resides on the speech conversion model.

16. The system of claim 13, wherein the text decoder resides on a reference automatic speech recognition model separate from the speech conversion model.

17. The system of claim 12, wherein the operation further comprises, after training the speech conversion model: Receive an input audio waveform corresponding to a utterance spoken by the target speaker associated with atypical speech; and The input audio waveform, corresponding to the utterance spoken by the target speaker associated with atypical speech, is converted into an output audio waveform corresponding to a synthetic standard fluent speech representation of the utterance spoken by the target speaker using a trained speech conversion model.

18. The system of claim 12, wherein the speech conversion model further comprises an automatic speech recognition model configured to convert speech into corresponding text.

19. The system of claim 18, wherein the operation further comprises, after training the speech conversion model: Receive audio data corresponding to utterances spoken by the target speaker associated with atypical speech; and The audio data corresponding to the utterance spoken by the target speaker associated with atypical speech is converted into a transcription of the utterance spoken by the target speaker using a trained speech conversion model.

20. The system of claim 12, wherein at least a portion of the plurality of non-verbal training text utterances in the second portion of the plurality of training text utterances comprises manually written text for specific phrases associated with the particular domain.

21. The system of claim 12, wherein the operation further comprises performing a non-verbal text selection process to obtain the non-verbal training text utterances in the second portion of the plurality of training text utterances, wherein the text selection process is configured to: Obtain a corpus of non-spoken text discourse; For each non-spoken text utterance in the corpus: Determine a first probability associated with the occurrence of the non-verbal text utterance in a domain-specific language model; Determine a second probability associated with the occurrence of the non-spoken text utterance in a background language model, the background language model being trained on each non-spoken text utterance in the corpus of non-spoken text utterances; and The score is determined based on the first probability, the second probability, and the number of words appearing in the corresponding non-spoken text utterance; and The non-verbal text discourse with the best score N from the corpus of non-verbal text discourses is selected as the non-verbal training text discourse in the second part of the plurality of training text discourses.

22. The system according to any one of claims 12 to 21, wherein: The text-to-speech model includes a pre-trained reference text-to-speech model, which includes an encoder part and a decoder part. as well as Adjusting the text-to-speech model involves tuning the pre-trained reference text-to-speech model by keeping the parameters of the encoder part fixed while tuning the parameters of the decoder part.

Images