Speaker Embedding for Improved Automatic Speech Recognition

Abstract

The method (400) includes receiving a reference audio signal corresponding to a reference speech spoken by a target speaker having an atypical speech, and generating a speaker embedding (350) for the target speaker by a speaker embedding network (250) configured to receive the reference audio signal as an input. The speaker embedding conveys speaker characteristics of the target speaker. The method also includes receiving a speech conversion request including input audio data (102) corresponding to an utterance spoken by the target speaker associated with the atypical speech. The method also includes using the speaker embedding generated for the target speaker by the speaker embedding network to bias a speech conversion model (210) to convert the input audio data corresponding to the utterance spoken by the target speaker associated with the atypical speech into an output canonical representation of the utterance spoken by the target speaker.

Description

[Technical field]

[0001] This disclosure relates to using speaker embeddings to improve automatic speech recognition. [Background technology]

[0002] Automatic speech recognition (ASR), the process of taking audio input and transcribing it into text, is an important technology used in mobile and other devices. In general, automatic speech recognition attempts to provide an accurate transcription of what a person said by taking audio input (e.g., a spoken utterance) and transcribing that audio input into text.

[0003] One of the challenges in developing deep learning-based speech conversion and ASR models is that the parameters of these models tend to overfit to the training data, making it difficult to generalize to unseen data when the training data is not extensive enough. Although accuracy can be improved by training speech conversion and ASR models on larger training datasets, there is a lack of sufficient training data that includes utterances targeting a specific domain (i.e., linguistic diversity) spoken by speakers with atypical speech patterns (i.e., acoustic diversity) to achieve acceptable accuracy with these models. Summary of the Invention

[0004] One aspect of the disclosure provides a computer-implemented method of using speaker embedding to personalize a speech conversion model for a target speaker associated with atypical speech. When executed on data processing hardware, the computer-implemented method causes the data processing hardware to perform operations including receiving a reference audio signal corresponding to a reference speech spoken by a target speaker having atypical speech, and generating a speaker embedding for the target speaker by a speaker embedding network configured to receive the reference audio signal as an input. The speaker embedding conveys speaker characteristics of the target speaker. The operations also include receiving a speech conversion request including input audio data corresponding to an utterance spoken by the target speaker associated with atypical speech. The operations also include biasing a speech conversion model to convert the input audio data corresponding to the utterance spoken by the target speaker associated with atypical speech into an output canonical representation of the utterance spoken by the target speaker using the speaker embedding generated for the target speaker by the speaker embedding network.

[0005] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, the speech conversion model includes a speech-to-speech conversion model configured to directly convert an input spectrogram or audio waveform to an output spectrogram or audio waveform. In these implementations, biasing the speech conversion model includes biasing the speech-to-speech conversion model to convert, using speaker embeddings, input audio data corresponding to an utterance spoken by a target speaker associated with an atypical speech to output audio data corresponding to a synthesized canonical fluent speech representation of the utterance spoken by the target speaker, where the output canonical representation includes the synthesized canonical fluent speech representation.

[0006] In some examples, the speech conversion model includes an automatic speech recognition model configured to convert speech to text. In these examples, biasing the speech conversion model includes biasing the automatic speech recognition model using speaker embeddings to convert input audio data corresponding to utterances spoken by a target speaker associated with an atypical speech into a canonical text representation of the utterances spoken by the target speaker, where the output canonical representation includes the canonical text representation.

[0007] In some implementations, the operations also include using a speaker embedding network to determine a personalization embedding for the target speaker based on the speaker embedding, where the personalization embedding corresponds to each style cluster of the speaker embedding extracted from the training utterances spoken by the training speaker that conveys similar speaker characteristics as conveyed by the speaker embedding. Here, biasing the speech conversion model includes using the personalization embedding determined for the target speaker to bias the speech conversion model for a type of atypical speech associated with the target speaker. In these implementations, biasing the speech conversion model using the personalization embedding determined for the target speaker may include providing the personalization embedding as a side input to the speech conversion model to bias the speech conversion model for a type of atypical speech associated with the target speaker. Optionally, biasing the speech conversion model using the personalization embedding determined for the target speaker may include providing the personalization embedding as a side input to a sub-model that biases the speech conversion model for a type of atypical speech associated with the target speaker. The operations may also include selecting a sub-model from a plurality of different sub-models using the personalization embeddings determined for the target speaker prior to biasing the voice conversion model, where each sub-model of the plurality of different sub-models may be trained for a different type or severity of atypical speech.

[0008] In some examples, the training process simultaneously trains the speech conversion model end-to-end with the speaker embedding network by obtaining multiple sets of spoken training utterances and simultaneously training the speech conversion model and the speaker embedding model on the multiple sets of spoken training utterances, each set of spoken training utterances including audio data characterizing training utterances spoken by a different training speaker and including atypical speech patterns for a type of atypical speech associated with each training speaker, and a canonical transcription of the training utterances spoken by each training speaker.

[0009] The speaker embedding network may include a reference encoder and a style attention module, where the reference encoder is configured to extract a respective speaker embedding for each training speaker from each set of spoken training utterances. Training the speech conversion model and the speaker embedding network may include training the style attention module to learn how to group the speaker embeddings into style clusters, where each style cluster may represent a respective cluster of speaker embeddings extracted from training utterances spoken by training speakers with similar speaker characteristics and may be mapped to a respective personalization embedding representing a respective type of atypical speech. Training the speech conversion model for each corresponding set of the multiple sets of spoken training utterances includes biasing the speech conversion model for the corresponding set of spoken training utterances using each personalization embedding mapped to a style cluster that includes the respective speaker embeddings extracted from the corresponding set of spoken training utterances.

[0010] In some implementations, simultaneously training the speech conversion model and the speaker embedding model further includes training a sub-model on the multiple sets of training utterances to learn how to bias the speech conversion model, where parameters of the speech conversion model are frozen while training the sub-model on the speech embedding network and the multiple sets of training utterances. In some examples, one or more of the multiple sets of training utterances include atypical speech patterns for each type of atypical speech associated with each of the different training speakers that are different from each type of atypical speech associated with each of the different training speakers.

[0011] Another aspect of the present disclosure provides a system for training a speech conversion model and a speaker personalization network for a target speaker associated with an atypical speech. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving a reference audio signal corresponding to a reference speech spoken by a target speaker having an atypical speech, and generating a speaker embedding for the target speaker by a speaker embedding network configured to receive the reference audio signal as an input. The speaker embedding conveys speaker characteristics of the target speaker. The operations also include receiving a speech conversion request including input audio data corresponding to an utterance spoken by the target speaker associated with the atypical speech. The operations also include biasing the speech conversion model to convert the input audio data corresponding to the utterance spoken by the target speaker associated with the atypical speech into an output canonical representation of the utterance spoken by the target speaker using the speaker embedding generated for the target speaker by the speaker embedding network.

[0012] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, the speech conversion model includes a speech-to-speech conversion model configured to directly convert an input spectrogram or audio waveform to an output spectrogram or audio waveform. In these implementations, biasing the speech conversion model includes biasing the speech-to-speech conversion model to convert, using speaker embeddings, input audio data corresponding to an utterance spoken by a target speaker associated with an atypical speech to output audio data corresponding to a synthesized canonical fluent speech representation of the utterance spoken by the target speaker, where the output canonical representation includes the synthesized canonical fluent speech representation.

[0013] In some examples, the speech conversion model includes an automatic speech recognition model configured to convert speech to text. In these examples, biasing the speech conversion model includes biasing the automatic speech recognition model using speaker embeddings to convert input audio data corresponding to utterances spoken by a target speaker associated with an atypical speech into a canonical text representation of the utterances spoken by the target speaker, where the output canonical representation includes the canonical text representation.

[0014] In some implementations, the operations also include using a speaker embedding network to determine a personalization embedding for the target speaker based on the speaker embedding, where the personalization embedding corresponds to each style cluster of the speaker embedding extracted from the training utterances spoken by the training speaker that conveys similar speaker characteristics as conveyed by the speaker embedding. Here, biasing the speech conversion model includes using the personalization embedding determined for the target speaker to bias the speech conversion model for a type of atypical speech associated with the target speaker. In these implementations, biasing the speech conversion model using the personalization embedding determined for the target speaker may include providing the personalization embedding as a side input to the speech conversion model to bias the speech conversion model for a type of atypical speech associated with the target speaker. Optionally, biasing the speech conversion model using the personalization embedding determined for the target speaker may include providing the personalization embedding as a side input to a sub-model that biases the speech conversion model for a type of atypical speech associated with the target speaker. The operations may also include selecting a sub-model from a plurality of different sub-models using the personalization embeddings determined for the target speaker prior to biasing the voice conversion model, where each sub-model of the plurality of different sub-models may be trained for a different type or severity of atypical speech.

[0015] In some examples, the training process simultaneously trains the speech conversion model end-to-end with the speaker embedding network by obtaining multiple sets of spoken training utterances and simultaneously training the speech conversion model and the speaker embedding model on the multiple sets of spoken training utterances, each set of spoken training utterances including audio data characterizing training utterances spoken by a different training speaker and including atypical speech patterns for a type of atypical speech associated with each training speaker, and a canonical transcription of the training utterances spoken by each training speaker.

[0016] The speaker embedding network may include a reference encoder and a style attention module, where the reference encoder is configured to extract a respective speaker embedding for each training speaker from each set of spoken training utterances. Training the speech conversion model and the speaker embedding network may include training the style attention module to learn how to group the speaker embeddings into style clusters, where each style cluster may represent a respective cluster of speaker embeddings extracted from training utterances spoken by training speakers with similar speaker characteristics and may be mapped to a respective personalization embedding representing a respective type of atypical speech. For each corresponding set of the multiple sets of spoken training utterances, training the speech conversion model includes biasing the speech conversion model for the corresponding set of spoken training utterances using each personalization embedding mapped to a style cluster that includes a respective speaker embedding extracted from the corresponding set of spoken training utterances.

[0017] In some implementations, simultaneously training the speech conversion model and the speaker embedding model further includes training a sub-model on the multiple sets of training utterances to learn how to bias the speech conversion model, where parameters of the speech conversion model are frozen while training the sub-model on the speech embedding network and the multiple sets of training utterances. In some examples, one or more of the multiple sets of training utterances each include atypical speech patterns for each type of atypical speech associated with each different training speaker that are different from each type of atypical speech associated with each different training speaker.

[0018] The details of one or more embodiments of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0019] [Figure 1A] FIG. 1 is a schematic diagram of an exemplary voice conversion system including a voice-to-speech conversion model. [Figure 1B] FIG. 1 is a schematic diagram of an exemplary speech-to-text system including a speech-to-text model. [Diagram 2] FIG. 1 is a schematic diagram of an exemplary training process for simultaneously training a speaker embedding network and an end-to-end voice conversion model. [Diagram 3] FIG. 1 is a schematic diagram of an example deterministic reference encoder for extracting speaker embeddings. [Figure 4] 1 is a flowchart of an exemplary arrangement of method operations using speaker embedding to personalize a voice conversion model for a target speaker having an atypical voice. [Diagram 5] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Like reference symbols in the various drawings indicate like elements. Automatic speech recognition (ASR) systems are often optimized to work best with speakers who have canonical speech patterns. Unfortunately, these systems can perform poorly when tested with speakers associated with atypical and heavily accented speech. The poor performance with these speakers is due to the fact that the majority of the speech data used to train these ASR systems comes from speakers who have typical speech patterns and speak with a canonical accent, resulting in the ASR system being biased toward this accent and performing poorly with other accents.

[0021] ASR systems have made significant advances with the introduction of end-to-end (E2E) deep learning-based models to recognize the speech of speakers with atypical speech patterns for conversion into accurate transcripts. For example, atypical speech patterns may include, but are not limited to, speech disorders resulting from physical or neurological conditions (e.g., speakers with amyotrophic lateral sclerosis (ALS) disease), heavily accented speech, and deaf speech. Similar deep learning-based models can be applied to speech-to-speech conversion systems to convert speech with atypical speech patterns into canonical fluent output speech. As used herein, and unless otherwise specified, the terms "speech conversion system" and "speech conversion model" can refer to either an ASR system / model in which input atypical speech is recognized and converted into corresponding text (e.g., transcription), or a speech-to-speech conversion system / model in which input atypical speech is directly converted into canonical fluent synthetic speech without performing speech recognition. In other words, the speech-to-speech conversion system / model is configured to convert an input audio waveform or spectrogram corresponding to atypical speech into an output audio waveform or spectrogram corresponding to canonical fluent speech without converting the input audio waveform into an intermediate representation (e.g., text or phonemes). As will become apparent, speech-to-speech conversion models, and techniques for training speech-to-speech conversion models, enable users with atypical speech to speak to and be understood by both other humans and speech interfaces (e.g., digital assistants) by enabling them to recognize and / or reproduce the user's intended speech.

[0022] One of the challenges in developing deep learning-based speech conversion and ASR models is that the parameters of these models tend to overfit the training data, which makes it difficult to generalize to unseen data when the training data is not extensive enough. Although training speech conversion and ASR models on larger training datasets improves accuracy, there is a lack of training data that provides sufficient acoustic diversity for personalization towards target speakers with atypical voices. For example, to achieve training data with sufficient acoustic diversity, target speakers with atypical voices need to record several hours of spoken utterances, each paired with a corresponding transcription.

[0023] One approach to recognizing speech from speakers who exhibit atypical or heavily accented speech is to create a personalized speech conversion model for each speaker, which has the potential to dramatically improve speech recognition for each individual speaker. Although speaker-personalized speech conversion models are generally effective, a drawback of this approach is scalability due to the need to train and maintain a separate model for each speaker. Another drawback of speaker-personalized speech conversion models is that they usually need to run on dedicated computing resources, e.g., tensor processing units (TPUs), graphical processing units (GPUs), etc., which are not readily available when provided on a server or only available on high-end user computing devices (e.g., smartphones).

[0024] Embodiments herein are directed to building a speech conversion system that adapts to atypical speech patterns and performs equally well in recognizing / converting speech from people who speak with different accents, local languages, dialects, or different ways of speaking. Specifically, embodiments are directed to providing a personalizable speech conversion system that simultaneously trains speaker embedding networks and speech conversion models to share training data among speakers with similar acoustic-phonetic characteristics and enables implicit clustering of speaker embeddings.

[0025] 1A and 1B show a speech conversion system 100, 100a-b including a speech conversion model 210 and a speaker embedding network 250. FIG. 1A shows a speech-to-speech (S2S) conversion model 210, 210a configured to directly convert input audio data 102 (e.g., input spectrogram or input audio waveform) corresponding to an utterance 108 spoken by a target speaker 104 associated with an atypical speech to output audio data 106 (e.g., output spectrogram or output audio waveform) corresponding to a synthesized canonical fluent speech representation of the same utterance 114 spoken by the target speaker 104. Notably, the S2S conversion model 210a is configured to directly convert the input audio data 102 to the output audio data 106 without performing speech recognition or otherwise requiring the generation of any intermediate discrete representations (e.g., text or phonemes) from the input audio data 102. The S2S conversion model 210a includes an audio encoder 212 configured to encode the input audio data 102 into a hidden feature representation (e.g., a series of vectors) and a speech decoder 220 configured to decode the hidden representation into output audio data 106 corresponding to a synthesized canonical fluent speech representation. For example, when the audio encoder 210 receives the input audio data 102 of an utterance 108, the audio encoder 210 may process five frames of audio and convert the five frames of audio into ten vectors. The vectors are not transcriptions of the frames of audio data 102, but rather mathematical representations of the frames of audio data 102. The speech decoder 220 may then generate the output audio data 106 corresponding to the synthesized canonical fluent speech representation based on the vectors received from the audio encoder 210. For example, the speech decoder 220 may receive ten vectors representing the five frames of audio from the audio encoder 210.Here, the speech decoder 220 may generate five frames of output audio data 106 corresponding to a synthesized canonical fluent speech representation of the utterance 114 that includes intended words or parts of words as the five frames of input audio data 102, but without the atypical speech disfluencies.

[0026] The S2S conversion system 100a may further include a synthesizer 275 that synthesizes the output audio data 106 into a time-domain waveform for audible output as the same utterance 114 of the fluent canonical fluent speech. The time-domain audio waveform includes an audio waveform that defines the amplitude of the audio signal over time. The synthesizer 275 may include a unit selection module or a WaveNet module for synthesizing the output audio data 106 into a time-domain waveform of the synthesized canonical fluent speech. In some implementations, the synthesizer 275 includes a vocoder network, i.e., a neural vocoder, that is separately trained and conditioned on the mel-frequency spectrogram for conversion to a time-domain audio waveform.

[0027] In the illustrated example, the target speaker 104 is associated with an atypical voice, such that the target speaker 104 speaks with an atypical voice pattern that may be difficult to understand. The atypical voice pattern may include, but is not limited to, speech disorders resulting from physical or neurological conditions (e.g., a speaker with amyotrophic lateral sclerosis (ALS) disease), heavily accented voice, and deaf voice. The speaker embedding network 250 generates a speaker embedding 350 that represents the voice of the target speaker 104 and the speaker characteristics of the target speaker 104. The speaker characteristics may convey the accent (e.g., French accent) and voice pattern associated with the target speaker 104. Here, the voice pattern conveyed by the speaker embedding 350 may include a typical voice pattern or one of a variety of different types of atypical voice patterns (e.g., heavily accented across different dialects and / or irregular voice across different neurological conditions). Notably, and as described in more detail below with reference to FIG. 2, the speaker embedding network 250 may be trained simultaneously with the S2S conversion model 210a and / or with a sub-model 280 (FIG. 2) that is responsible for influencing the S2S conversion model 210a towards the target speaker 104 during inference. By training the speaker embedding network 250 and the S2S conversion model 210 or sub-model 280 simultaneously, the speaker embedding network 250 may generate a more optimal speaker embedding 350 to assist the S2S conversion model 210a in converting atypical speech patterns into canonical speech. In contrast, separately trained speaker embedding models, such as those implemented to generate speaker identification embeddings used for voice authentication / verification, are typically trained to recognize the speaker's voice and ignore any accent / speaker characteristics.

[0028] In some implementations, the S2S conversion model 210a receives the speaker embeddings 350 generated by the speaker embedding network 250 as a side input for biasing / influencing the S2S conversion model 210a in directly converting the input audio data 102 corresponding to an utterance 108 spoken by a target speaker 104 associated with an atypical voice to the output audio data 106 corresponding to a synthesized canonical fluent phonetic representation of the same utterance 108. In other implementations, the speaker embedding network 250 further generates a personalization embedding 251 based on the speaker embeddings 350 and provides the personalization embeddings 251 as a side input to the sub-model 280 or the S2S conversion model 210a for biasing / influencing the S2S conversion model 210a in directly converting the utterance 108 of the atypical voice to a synthesized canonical fluent phonetic representation of the same utterance 108. As described in more detail below with reference to FIG. 2, the personalization embeddings 251 may indicate each cluster of speaker embeddings 350 extracted from training utterances 305 (FIG. 2) spoken by a speaker with similar speaker characteristics (i.e., accent and speech pattern) as the target speaker 104. In this manner, the personalization embeddings 251 may correspond to context vectors for parameterizing / personalizing the sub-models 280 and / or the S2S conversion model 210a to improve speech conversion performance on the type of atypical speech associated with the target speaker 104. For example, the personalization embeddings 251 may indicate the type of atypical speech (e.g., ALS speech) associated with the target speaker 104, or may even provide more fine-grained information, such as a severity level of the type of atypical speech.

[0029] The speech-to-speech conversion system 100a is trained to directly convert input audio data 102 corresponding to an utterance 108 spoken by a target speaker 104 associated with an atypical speech into output audio data 106 corresponding to a synthesized canonical fluent speech representation of the same utterance 108. Thus, the synthesized canonical fluent speech representation provided by the output audio data 106 improves the intelligibility of the atypical speech (e.g., heavily accented or ALS speech) spoken by the target speaker 104. Without departing from the scope of this disclosure, the speech-to-speech conversion system 100a may be trained to directly convert input audio data 102 corresponding to an utterance 108 associated with an atypical speech of a first language into output audio data 106 corresponding to a synthesized canonical fluent speech representation of the same utterance 108 of the same voice, but in a different second language.

[0030] A computing device 110 associated with a target speaker 104 can capture an utterance 108 spoken by the target speaker 104 and transmit the corresponding input audio data 102 to the speech-to-speech conversion system 100a for conversion to output audio data 106. The speech-to-speech conversion system 100a can then transmit output audio data 106 corresponding to a synthesized canonical fluent speech representation of the same utterance 114 spoken by the target speaker 104 to another computing device 116 associated with a user 118, such that the other computing device 116 audibly outputs the synthesized canonical fluent speech representation of the utterance 108 spoken by the target speaker 104. In this example, the target speaker 104 and the user 118 are speaking to each other via their respective computing devices 110, 116, such as by telephone or other type of voice communication protocol, such as, for example, Voice over Internet Protocol. Although the target speaker 104 and the other users 118 may speak the same language, it may be difficult for the other users 118 to understand the target speaker 104 due to the target speaker 104 having an atypical voice due to ALS disease. Thus, while the target speaker 104 speaks with an atypical voice (e.g., ALS voice) that is difficult to understand, the other users 118 who listen to the synthesized canonical fluent voice representation will more easily understand the utterance 108 intended by the target speaker 104. In other words, the synthesized canonical fluent voice representation provides a consistent rhythm and may be easier for other users to understand than the original utterance 108 spoken by the target speaker with an atypical voice. It is worth noting that the synthesized canonical fluent voice representation is the voice of the target speaker 104.

[0031] In some other examples, the S2S conversion system 100a passes output audio data 106 corresponding to the synthesized canonical fluent speech representation of the utterance spoken by the target speaker 104 to an output audio device for audibly outputting the synthesized canonical fluent speech representation to the audience in the speech of the target speaker 104. For example, if the target speaker 104 is a psychology professor and is presenting a protest to a class of students, the utterance spoken by the target speaker 104 may include medical terms belonging to a particular field, e.g., psychology. As will become apparent, the S2S conversion system 100a is trained to learn the acoustic diversity associated with a particular type of atypical speech associated with the speaker who spoke the target utterance, along with the linguistic diversity from the linguistic content present in the training utterances.

[0032] Alternatively, the other computing device 116 can be associated with a downstream automatic speech recognition (ASR) system in which the speech-to-speech conversion system 100a serves as a front end to provide output audio data 106 corresponding to the synthesized canonical fluent speech representation as input to the ASR system for conversion to recognized text. The recognized text can be presented to another user 118 and / or provided to a natural language understanding (NLU) system for further processing. The functionality of the speech-to-speech conversion system 100a can reside on either or both of the remote server 112, the computing devices 110, 116, or any combination of the remote server and the computing devices 110, 116. In some implementations, the S2S conversion model 210a continuously generates output audio data 106 corresponding to the synthesized canonical fluent speech representation of the utterance when the target speaker 104 speaks the corresponding portion of the utterance as an atypical voice. By continuously generating output audio data 106 corresponding to synthesized canonical fluent speech representations of portions of utterances 108 spoken by the target speaker 104, the conversation between the target speaker 104 and the user 118 (or audience) may be at a more natural pace. In some additional implementations, the S2S conversion model 210a waits to determine / detect when the target speaker 104 stops speaking using techniques such as voice activity detection, end pointing, end of query detection, and then converts the corresponding input audio data 102 of the utterance 108 with atypical speech into corresponding output audio data 106 corresponding to synthesized canonical fluent speech representations of the same utterance 114.

[0033] 1B, in some implementations, the speech conversion system 100, 100b includes a speech-to-text conversion system 100b configured to convert (i.e., transcribe) input audio data 102 corresponding to an utterance 108 spoken by a target speaker 104 associated with an atypical voice into a canonical text representation 120 of the utterance 108. Thus, the speech-to-text conversion system 100b may correspond to a personalized automatic speech recognition (ASR) system for the target speaker 104 that can recognize a particular type of atypical speech pattern and linguistic information of the target speaker to convert the original utterance 108 spoken by the target speaker 104 associated with the atypical voice into a corresponding canonical text representation 120 that captures the intent of the original utterance 108 spoken by the target speaker 104 associated with the atypical voice. Another user 118 (FIG. 1A) may obtain the canonical text representation 120 of the utterance 108. In some configurations, the canonical text representation 120 output from system 100b is processed by a natural language understanding (NLU) module running, for example, on the user device 110 or the remote server 112 to execute user commands. Additionally or alternatively, a text-to-speech system (e.g., running on any combination of the user device 110 or the remote server 112) can convert the transcript into synthesized speech for audible output by another device. The functionality of speech-to-text system 100b can reside on the remote server 112, on either or both of the computing devices 110, or on any combination of the remote server 112 and the computing devices 110.

[0034] The speech-to-text system 100b associated with the speech conversion model 210 may include a speech-to-text model 210b (interchangeably referred to as an automatic speech recognition (ASR) model 210b) configured to perform speech recognition on the atypical speech utterance 108 by converting the input audio data 102 into a canonical text representation (i.e., a transcription) 120 of the utterance 108. The ASR model 210b includes an identical or different audio encoder 210 configured to encode the input audio data 102 into a hidden feature representation (e.g., a series of vectors) and a text decoder 225 configured to decode the hidden representation into the canonical transcription 120. For example, when the audio encoder 212 receives the input audio data 102 of the utterance 108, the encoder 212 may process five frames of audio and convert the five frames of audio into ten vectors. The vectors are not transcriptions of the frames of audio data 102, but rather mathematical representations of the frames of audio data 102. The text decoder 225 may then generate a canonical transcription 120 based on the vectors received from the encoder 212. For example, the text decoder 225 may generate a series of words corresponding to the canonical transcription 120 of an utterance 180 that includes an intended word or portion of a word in five frames of the input audio data 102. Without departing from the scope of this disclosure, the speech-to-text conversion system 100b may be trained to convert input audio data 102 corresponding to an utterance 108 associated with an atypical speech in a first language into a corresponding canonical transcription of the utterance 108 in a different second language.

[0035] 1A, 1B, and 2, the speech conversion system 100 executes a training process 200 configured to train a speech conversion model 210, i.e., the S2S conversion model 210a of FIG. 1A and / or the speech-to-text conversion model 210b of FIG. 1B, end-to-end in parallel with a speaker embedding network 250 to improve the performance of the speech conversion system 100 in recognizing and / or converting various types of atypical speech that are typically underrepresented in typical training data.

[0036] 2 illustrates a training process 200 that obtains a plurality of sets of spoken training utterances 305, 305a-n, each set including audio data 304 characterizing the training utterances 305 spoken by each training speaker. The set of spoken training utterances 305 may be stored in a data store 202 residing in memory hardware 520 (FIG. 5) of the user device 110 or the remote server 112. Here, the training utterances 305 in each set, spoken by each training speaker, include corresponding transcriptions 302 paired with audio data 304 characterizing the training utterances. In this manner, the spoken training utterances 305 in each set include manually transcribed audio data 302, 304, spoken by each training speaker. The audio data 304 corresponds to a phonetic representation of a corresponding training utterance 305 and conveys the voice and speaker characteristics (e.g., accent and / or speech pattern) of the respective training speaker speaking the training utterance 305 in each set of spoken training utterances 305. Examples include audio data 304 corresponding to non-synthetic speech representations (e.g., human-derived speech), although some of the training utterances 305 may include audio data 304 corresponding to synthesized / synthesized speech representations targeting particular voices and speaking patterns.

[0037] Implementations herein are directed to a plurality of sets of spoken training utterances 305, each including a respective atypical speech pattern of a type of atypical speech associated with a respective training speaker who spoke the training utterances 305. For example, one or more of the sets of spoken training utterances 305 can be spoken by a training speaker exhibiting an atypical speech pattern of ALS speech, one or more other sets of spoken training utterances 305 can be spoken by a training speaker exhibiting an atypical speech pattern of dysarthria, and one or more other sets of spoken training utterances 305 can be spoken by a training speaker exhibiting an atypical speech pattern of a heavily accented speech in a given dialect (e.g., French accent). Each transcription 302 can be a canonical transcription in the native language of the target speaker 104. In some examples, some or all of the spoken training utterances 305 include words and / or phrases related to the particular domain that the speech conversion model 210 is trained to learn.

[0038] The speech embedding network 250 includes a reference encoder 300 and a style attention module 254. During the training process 200, the reference encoder 300 receives as input audio data 304 characterizing training utterances 305 spoken by each training speaker associated with each set of training utterances 305 and generates respective speaker embeddings (Z E ) 350 as output, where the speaker embeddings 350 output from the reference encoder 300 represent the voice of each training speaker and the speaker characteristics of each training speaker. As previously mentioned, the speaker characteristics may convey the accent (e.g., French accent) and speech patterns associated with the training speakers. Here, the speech patterns conveyed by the speaker embeddings 350 may include typical speech patterns or one of a variety of different types of atypical speech patterns (e.g., heavy accents across different dialects, irregular speech across different neurological conditions).

[0039] Referring to FIG. 3, in some implementations, an exemplary reference encoder receives a reference audio signal 312 (e.g., training audio data 304 or input audio data 102 corresponding to a reference speech spoken by a target speaker 104) and derives a speaker embedding Z from the reference audio signal 312. E 350. The reference audio signal 312 may be of sufficient duration to capture various acoustic-phonetic features associated with a speaker. In some examples, the reference audio signal 312 comprises a duration of about one minute. E 350 can capture speaker characteristics of the reference audio signal 312 that convey accents and speech patterns associated with the reference audio signal 312 .

[0040] In the illustrated example, the reference audio signal 312 has a length L R and Dimension D R The spectrogram slices associated with the reference audio signal 312 may exhibit a Mel-warped spectrum. In some examples, the reference encoder 300 includes a 6-layer convolutional layer network 314, each layer including a 3×3 filter with a stride of 2×2, SAME padding, and ReLU activation. Batch normalization is applied to all layers, and the number of filters at each layer doubles with half the downsampling rate (32, 32, 64, 128, 128). A recurrent neural network (RNN) 310 with bidirectional long short-term memory (LSTM) layers receives the output 316 from the last convolutional layer to provide an output 322 that is applied to a fully connected layer 320 to generate the speaker embedding Z E This is followed by an activation function 330 that outputs 350. Instead of an RNN, the reference encoder 300 may incorporate a single 128-wide gated recurrent unit (GRU-RNN) layer that receives the output 316 from the last convolutional layer and outputs a 128-dimensional output 322 that is applied to a fully connected layer 320.

[0041] Referring back to the training process 200 of FIG. 2, the style attention module 254 receives the speaker embeddings 350 output from the reference encoder 300 and uses the speaker embeddings 350 as query vectors to learn similarities between the speaker embeddings 350. More specifically, the training process 200 trains the style attention module 254 to learn how to group the speaker embeddings 350 into style clusters 260, 260a-n, each representing a respective cluster of speaker embeddings 350 extracted from training utterances 305 spoken by a training speaker having similar characteristics (i.e., accent and speech pattern). For example, a first style cluster 260a may include speaker embeddings 350 extracted from training utterances 305 spoken by a training speaker associated with an ALS voice, and a second style cluster 260a may include speaker embeddings 350 extracted from training utterances 305 spoken by a training speaker associated with a Down Syndrome voice.

[0042] Furthermore, each style cluster 260 into which the style attention module 254 learns to group similar speaker embeddings 350 need not represent each type of atypical speech that is different from the types of atypical speech represented by other style clusters. That is, one style cluster 260 may include a group of similar speaker embeddings 350 extracted from training utterances 305 spoken by a training speaker associated with a first severity level of a type of atypical speech, while another style cluster 260 may include other clusters of speaker embeddings 350 extracted from training utterances 305 spoken by other training speakers associated with a second severity level of the same type of atypical speech. In this way, the style attention module 254 learns how to group similar speaker embeddings 350 into clusters 260 that represent different types (and / or severity levels) of atypical speech (e.g., heavy accents across different dialects and / or irregular speech across different neurological conditions). The attention module 254 may be viewed as a mechanism configured to map the new speaker embeddings 350 to respective style clusters 260 having speaker embeddings that convey similar speaker characteristics.

[0043] During the training process 200, the style attention module 254 can learn to generate a personalization embedding (h) 251 for each speaker embedding 350 based on the corresponding style cluster 260 in which each speaker embedding 250 is grouped. In some configurations, the style attention module 254 receives speaker embeddings 350 extracted from a set of training utterances 305 spoken by each speaker and outputs a set of combination weights representing the contribution of each style cluster 260 to the corresponding speaker embedding 350. In these configurations, the weighted sum of the style clusters 260 corresponds to the personalization embedding 251 that is input to the sub-models 280 and / or the speech conversion model 210 for conditioning at every time step.

[0044] In some implementations, the personalization embeddings 251 generated for each speaker embedding 350 extracted during the training process 200 correspond to context vectors that parameterize the speech conversion model 210 as additional side inputs leading to an implicit classifier for each training speaker while simultaneously training the speech conversion model 210 on the training utterances 305 spoken by the training speakers. Thus, the speech conversion model 210 can receive the personalization embeddings 251 (or speaker embeddings 350) as inputs for each training speaker when training the speech conversion model 210 on a set of training utterances 305 spoken by the training speakers. For example, the personalization embeddings 251 can indicate that the training speakers are grouped into style clusters 260 associated with heavy French accent speech, thereby enabling the speech conversion model 210 to adapt to improve its performance in recognizing / converting training utterances 305 conveying a heavy French accent.

[0045] In additional implementations, the personalization embeddings 251 (or speaker embeddings 350) are instead provided as side inputs to the sub-model 280 to train the sub-model 280 on training utterances 305 spoken by training speakers. In these implementations, the speech conversion model 210 includes a base model associated with an off-the-shelf pre-trained speech conversion model 210. For example, the pre-trained speech conversion model 210 may include a state-of-the-art model trained on a general corpus of training utterances spoken by a variety of different speakers with typical speech patterns, and the sub-model 280 can be trained on the training utterances 305 to learn how to bias itself against the pre-trained speech conversion model 210 to improve its performance in recognizing / converting training utterances 305 associated with atypical speech. Training the sub-model 280 as a modular extension for biasing the pre-trained base speech conversion model 210 advantageously aids during inference when the reference speech sample is not obtained from the target speaker by allowing the pre-trained base speech conversion model 210 to process the input utterance regardless of speaker characteristics. For example, the target speaker speaking the initial input utterance for conversion / recognition by the speech conversion system 100 may not be of sufficient duration for the speech embedding network 250 to extract the speaker embeddings 350 and / or personalization embeddings 251 therefrom. However, once audio data conveying the input utterance of sufficient duration is available to the speaker embedding network 250 for extracting the speaker embeddings 350 and / or personalization embeddings 251 therefrom, the speaker embedding network 250 can provide the speaker embeddings 350 and / or personalization embeddings 251 as side inputs to the sub-model 280 for biasing the speech conversion model 210.

[0046] In some examples, the sub-model 210 is placed in a neural network layer of the audio encoder 212 or between two neural network layers of the audio encoder 212. In other examples, the sub-model 210 is placed in a neural network layer of the decoders 220, 225 or between two neural network layers of the decoders 220, 225. In some additional examples, the first sub-model 210 is placed in the audio encoder 212 and the second sub-model 210 is placed in the decoders 220, 225 of the pre-trained speech conversion model 210. During the training process 200, parameters of the pre-trained speech conversion model 310 may be frozen while training the speech embedding network 250 and the sub-model(s) 280 on the training utterances 305.

[0047] In some implementations, a single sub-model 280 is trained to bias / influence the speech conversion model 210 across a diverse set of different types of atypical speech. During inference, the reference encoder 300 receives as input a reference audio signal 312 corresponding to a reference speech spoken by a given target speaker 104 associated with an atypical speech, and generates as output a speaker embedding 350 for the target speaker 104 conveying speaker characteristics (e.g., type of atypical speech) associated with the target speaker 104. Using the techniques described above, the style attention mechanism 254 can focus on the speaker embedding 350 to determine a personalization embedding 251 for the target speaker 104. The personalization embedding 251 can function as a one-hot vector that adapts the speech conversion model 210 to a particular one of the types of atypical speech. In additional implementations, multiple sub-models 280 are trained to bias / influence the speech conversion model 210 for each type of atypical speech, respectively. That is, a separate sub-model 280 may be trained for each type of atypical speech identified by the speaker embedding network 250 from multiple sets of training utterances 305 during the training process 200. In this manner, each style cluster 260 of the speaker embeddings 350 extracted during the training process 200 may be mapped to a respective personalization embedding 251 representing each type of atypical speech (and optionally a severity level of the type of atypical speech). Thus, the personalization embeddings 251 output by the speaker embedding network 250 during inference may act as a one-hot vector to select an appropriate sub-model 280 (i.e., a sub-model corresponding to a type of atypical speech associated with the source speaker 104) from multiple different sub-models.

[0048] 4 provides a flowchart of an exemplary arrangement of operations of a method 400 of using speaker embedding 350 to personalize a voice conversion model 210 for a target speaker 104 having an atypical voice. Data processing hardware 510 (FIG. 5) can perform the operations of the method 400 by executing instructions stored in memory hardware 520 (FIG. 5) in communication with the data processing hardware 510. The data processing hardware 510 and the memory hardware 520 may reside in a computing device 500 (FIG. 5), such as the remote server 112 and / or the user computing device 110 of FIGS. 1A and 1B. At operation 402, the method 400 includes receiving a reference audio signal 312 corresponding to a reference voice spoken by the target speaker 104 in an atypical voice.

[0049] At operation 404, the method 400 includes generating a speaker embedding 350 for the target speaker 104 by a speaker embedding network 250 configured to receive the reference audio signal 312 as an input, where the speaker embedding 350 conveys speaker characteristics of the target speaker 104. The speaker characteristics may convey an accent (e.g., French accent) and a speech pattern associated with the target speaker 104. For example, the speech pattern conveyed by the speaker embedding 350 may include a typical speech pattern or one of a variety of different types of atypical speech patterns (e.g., heavy accents across different dialects, irregular speech across different neurological conditions).

[0050] At operation 406, the method 400 includes receiving a speech conversion request including input audio data 102 corresponding to an utterance 108 spoken by a target speaker 104 associated with an atypical speech. At operation 408, the method 400 biases the speech conversion model 210 using the speaker embeddings 350 generated by the speaker embedding network 250 for the target speaker 104 to convert the input audio data 102 corresponding to the utterance 108 spoken by the target speaker 104 associated with the atypical speech into an output canonical representation 106, 120 of the utterance spoken by the target speaker.

[0051] In some examples, the speech conversion model 210 includes a speech-to-speech conversion model 210 configured to directly convert an input spectrogram or audio waveform to an output spectrogram or audio waveform. In these examples, the output canonical representation includes output audio data corresponding to a synthesized canonical fluent speech representation of the utterance spoken by the target speaker.

[0052] In other examples, the speech conversion model 210 includes an automatic speech recognition model 210b configured to convert speech to text. In these examples, the output canonical representations include canonical text representations of the utterances 108 spoken by the target speakers 104.

[0053] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," an "app," or a "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0054] Non-transient memory may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by a computing device. Non-transient memory may be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0055] 5 is a schematic diagram of an exemplary computing device 500 that can be used to implement the systems and methods described herein. Computing device 500 is intended to represent various types of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown, their connections and relationships, and their functionality are for illustrative purposes only and are not intended to limit the implementation of the invention described and / or claimed in this document.

[0056] Computing device 500 includes a processor 510, memory 520, storage device 530, a high-speed interface / controller 540 that connects to memory 520 and high-speed expansion port 550, and a low-speed interface / controller 560 that connects to low-speed bus 570 and storage device 530. Each component 510, 520, 530, 540, 550, and 560 are interconnected using various buses and may be mounted on a common motherboard or may exist in other manners as needed. Processor (i.e., data processing hardware) 510 may process instructions for execution within computing device 500, including instructions stored in memory 520 or storage device 530, and display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 580 connected to high-speed interface 540. In other implementations, multiple processors and / or more buses may be used along with multiple memories and types of memories, as needed. Also, multiple computing devices 500 may be connected, with each device performing some of the required operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system). The data processing hardware 510 and memory hardware 520 may be present in the remote server 112, in either or both of the computing devices, or in any combination of the remote server 112 and the computing device 110 of Figure 1A and the computing device 116 of Figure 1B.

[0057] The memory (i.e., memory hardware) 520 stores information non-transiently within the computing device 500. The memory 520 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transient memory 520 may be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by the computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0058] The storage device 530 can provide mass storage for the computing device 500. In some embodiments, the storage device 530 is a computer-readable medium. In various different implementations, the storage device 530 can be a device array, including a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. In additional embodiments, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or a machine-readable medium, such as the memory 520, the storage device 530, or a memory on the processor 510.

[0059] The high-speed controller 540 manages bandwidth-intensive operations of the computing device 500, while the low-speed controller 560 manages low-bandwidth-intensive operations. Such an allocation of roles is merely an example. In some implementations, the high-speed controller 540 is coupled to memory 520 (e.g., via a graphics processor or accelerator), to a display 580, and to a high-speed expansion port 550 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 560 is coupled to a storage device 530 and to a low-speed expansion port 590. The low-speed expansion port 590, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be connected, for example, via a network adapter to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a network device such as a switch or router.

[0060] The computing device 500 can be implemented in many different forms, as shown in the figure: for example, it may be implemented as a standard server 500a, or multiple times within a group of such servers 500a, as a laptop computer 500b, or as part of a rack server system 500c.

[0061] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may be specialized or general purpose and may include implementations in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, at least one input device, and at least one output device coupled to receive data and instructions from and transmit data and instructions to the storage system.

[0062] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in a high-level procedural and / or object-oriented programming language, and / or assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0063] The processes and logic flows described herein can be performed by one or more programmable processors, also referred to as data processing hardware, executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for executing computer programs include, for example, both general purpose and special purpose microprocessors, as well as any one or more processors of any type of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively connected to receive data from them, transmit data to them, or both. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all types of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0064] To provide for interaction with a user, one or more aspects herein may be implemented on a computer having a display device, such as, for example, a CRT (cathode ray tube), LCD (liquid crystal display) monitor, touch screen, etc., for displaying information to a user, and optionally a keyboard and a pointing device, such as, for example, a mouse or trackball, by which a user can provide input to the computer. Other types of devices may also be used to provide for interaction with a user, for example, feedback provided to the user may be any form of sensory feedback, for example, visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, speech, or tactile input. Additionally, a computer may interact with a user by sending and receiving documents to a device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0065] Although several embodiments have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. When executed on data processing hardware (510), the data processing hardware (510) receiving a reference audio signal (312) corresponding to a reference speech spoken by a target speaker having an atypical speech; generating a speaker embedding (350) for the target speaker by a speaker embedding network (250) configured to receive as input the reference audio signal (312), the speaker embedding (350) conveying speaker characteristics of the target speaker; receiving a voice conversion request including input audio data (102) corresponding to an utterance spoken by the target speaker associated with the atypical voice; biasing a speech conversion model (210) using the speaker embeddings (350) generated for the target speaker by the speaker embedding network (250) to convert the input audio data (102) corresponding to the utterances spoken by the target speaker associated with atypical speech into an output canonical representation (114) of the utterances spoken by the target speaker; A computer-implemented method (400) for performing operations including:

2. the speech conversion model (210) comprises a speech-to-speech conversion model (210) configured to directly convert an input spectrogram or audio waveform to an output spectrogram or audio waveform; biasing the speech conversion model (210) includes biasing the speech-to-speech conversion model (210) using the speaker embeddings (350) to convert the input audio data (102) corresponding to the utterance spoken by the target speaker associated with an atypical speech into output audio data (106) corresponding to a synthesized canonical fluent speech representation (114) of the utterance spoken by the target speaker; The computer-implemented method (400) of claim 1 , wherein the output canonical representation comprises the synthesized canonical fluent speech representation (114).

3. the speech conversion model (210) comprises an automatic speech recognition model configured to convert speech to text; Biasing the speech conversion model (210) includes using the speaker embeddings (350) to bias the automatic speech recognition model to convert the input audio data (102) corresponding to the utterances spoken by the target speaker associated with atypical speech into a canonical text representation (120) of the utterances spoken by the target speaker; The computer-implemented method (400) of claim 1 , wherein the output canonical representation (114) comprises the canonical textual representation (120).

4. The operation, determining a personalization embedding (251) for the target speaker based on the speaker embedding (350) using the speaker embedding network (250), the personalization embedding (251) corresponding to style clusters associated with a plurality of speaker embeddings (350) extracted from a plurality of training utterances (305) spoken by a plurality of speakers conveying speaker characteristics similar to the speaker characteristics conveyed by the speaker embedding (350); 4. The computer-implemented method of claim 1, wherein biasing the voice conversion model comprises biasing the voice conversion model towards a type of atypical speech associated with the target speaker using the personalization embeddings determined for the target speaker.

5. 5. The computer-implemented method of claim 4, wherein biasing the speech conversion model (210) using the personalization embeddings (251) determined for the target speaker comprises providing the personalization embeddings (251) as side inputs to the speech conversion model (210) to bias the speech conversion model (210) with respect to the type of atypical speech associated with the target speaker.

6. 5. The computer-implemented method of claim 4, wherein biasing the speech conversion model (210) using the personalization embeddings (251) determined for the target speaker comprises providing the personalization embeddings (251) as side inputs to a sub-model that biases the speech conversion model (210) with respect to the type of atypical speech associated with the target speaker.

7. 7. The computer-implemented method of claim 6, further comprising: selecting the sub-model from a plurality of different sub-models using the personalization embeddings determined for the target speaker prior to biasing the voice conversion model, each sub-model of the plurality of different sub-models being trained for a different type or severity of atypical speech.

8. The training process (200) trains the speech conversion model (210) end-to-end with the speaker embedding network (250) simultaneously, obtaining a plurality of sets of spoken training utterances (305), each set of spoken training utterances (305) being spoken by a different training speaker; audio data characterizing the training utterances (305) including atypical speech patterns for atypical speech types associated with each of the training speakers; a canonical transcription (120) of the training utterances (305) spoken by each of the training speakers; training the speech conversion model (210) and the speaker embedding network (250) simultaneously on a plurality of sets of the spoken training utterances (305); The computer-implemented method (400) of any one of claims 1 to 7, wherein the method is performed by:

9. the speaker embedding network (250) includes a reference encoder (300) and a style attention module (254), the reference encoder (300) configured to extract, from each set of spoken training utterances (305), a respective speaker embedding (350) for each of the training speakers; Training the speech conversion model (210) and the speaker embedding network (250) comprises: training the style attention module (254) to learn how to group the speaker embeddings (350) into style clusters (260), each style cluster representing a respective cluster of speaker embeddings (350) extracted from a plurality of training utterances (305) spoken by a plurality of training speakers having similar speaker characteristics, each style cluster being mapped to a respective personalization embedding (251) representing a respective type of atypical speech; biasing the speech conversion model (210) with respect to the corresponding set of the plurality of sets of spoken training utterances (305) using the respective personalization embeddings (251) mapped to the style clusters including the respective speaker embeddings (350) extracted from the corresponding set of the spoken training utterances (305); The computer-implemented method (400) of claim 8, comprising:

10. training the speech conversion model (210) and the speaker embedding network (250) simultaneously further comprises training a sub-model with the plurality of sets of training utterances (305) to learn how to bias the speech conversion model (210); 10. The computer-implemented method (400) of claim 8 or 9, wherein parameters of the speech conversion model (210) are frozen during training of the speech embedding network (250) and the sub-models with the multiple sets of training utterances (305).

11. 11. The computer-implemented method (400) of claim 8, wherein one or more of the multiple sets of training utterances (305) include atypical speech patterns for each type of atypical speech associated with each different training speaker that are different from each type of atypical speech associated with each different training speaker.

12. Data processing hardware (510); memory hardware (520) in communication with said data processing hardware (510); Including, The memory hardware (520) stores instructions that, when executed by the data processing hardware (510), cause the data processing hardware (510) to: receiving a reference audio signal (312) corresponding to a reference speech spoken by a target speaker having an atypical speech; generating a speaker embedding (350) for the target speaker by a speaker embedding network (250) configured to receive as input the reference audio signal (312), the speaker embedding (350) conveying speaker characteristics of the target speaker; receiving a voice conversion request including input audio data (102) corresponding to an utterance spoken by the target speaker associated with the atypical voice; biasing a speech conversion model (210) using the speaker embeddings (350) generated by the speaker embedding network (250) for the target speaker to convert the input audio data (102) corresponding to the utterances spoken by the target speaker associated with atypical speech into an output canonical representation of the utterances spoken by the target speaker; A system (100) that performs operations including:

13. the speech conversion model (210) comprises a speech-to-speech conversion model (210) configured to directly convert an input spectrogram or audio waveform to an output spectrogram or audio waveform; biasing the speech conversion model (210) includes biasing the speech-to-speech conversion model (210) using the speaker embeddings (350) to convert the input audio data (102) corresponding to the utterance spoken by the target speaker associated with an atypical speech into output audio data (106) corresponding to a synthesized canonical fluent speech representation of the utterance spoken by the target speaker; The system (100) of claim 12, wherein the output canonical representation comprises the synthesized canonical fluent speech representation.

14. the speech conversion model (210) comprises an automatic speech recognition model configured to convert speech to text; Biasing the speech conversion model (210) includes biasing the automatic speech recognition model using the speaker embeddings (350) to convert the input audio data (102) corresponding to the utterances spoken by the target speaker associated with an atypical speech into a canonical text representation (120) of the utterances spoken by the target speaker; The system (100) of claim 12, wherein the output canonical representation comprises the canonical text representation (120).

15. The operation, determining a personalization embedding (251) for the target speaker based on the speaker embedding (350) using the speaker embedding network (250), the personalization embedding (251) corresponding to style clusters associated with a plurality of speaker embeddings (350) extracted from a plurality of training utterances (305) spoken by a plurality of speakers conveying speaker characteristics similar to the speaker characteristics conveyed by the speaker embedding (350); 15. The system (100) of claim 12, wherein biasing the voice conversion model (210) comprises biasing the voice conversion model (210) towards the type of atypical voice associated with the target speaker using the personalization embeddings (251) determined for the target speaker.

16. 16. The system of claim 15, wherein biasing the speech conversion model with the personalization embeddings determined for the target speaker comprises providing the personalization embeddings as a side input to the speech conversion model to bias the speech conversion model with respect to the type of atypical speech associated with the target speaker.

17. 16. The system of claim 15, wherein biasing the speech conversion model with the personalization embeddings determined for the target speaker comprises providing the personalization embeddings as a side input to a sub-model that biases the speech conversion model with respect to the type of atypical speech associated with the target speaker.

18. 20. The system of claim 17, wherein the operations further comprise: using the personalization embeddings determined for the target speaker prior to biasing the speech conversion model, selecting the sub-model from a plurality of different sub-models, each sub-model of the plurality of different sub-models trained for a different type or severity of atypical speech.

19. The training process (200) trains the speech conversion model (210) end-to-end with the speaker embedding network (250) simultaneously, obtaining a plurality of sets of spoken training utterances (305), each set of spoken training utterances (305) being spoken by a different training speaker; audio data characterizing the training utterances (305) including atypical speech patterns for atypical speech types associated with each of the training speakers; a canonical transcription (120) of the training utterances (305) spoken by each of the training speakers; training the speech conversion model (210) and the speaker embedding network (250) simultaneously on a plurality of sets of the spoken training utterances (305); The system (100) according to any one of claims 12 to 18, wherein the system (100) is performed by

20. the speaker embedding network (250) includes a reference encoder (300) and a style attention module (254), the reference encoder (300) configured to extract, from each set of spoken training utterances (305), a respective speaker embedding (350) for each of the training speakers; Training the speech conversion model (210) and the speaker embedding network (250) comprises: training the style attention module (254) to learn how to group the speaker embeddings (350) into style clusters (260), each style cluster representing a respective cluster of speaker embeddings (350) extracted from a plurality of training utterances (305) spoken by a plurality of training speakers having similar speaker characteristics, each style cluster being mapped to a respective personalization embedding (251) representing a respective type of atypical speech; biasing the speech conversion model (210) with respect to the corresponding set of the plurality of sets of spoken training utterances (305) using the respective personalization embeddings (251) mapped to the style clusters including the respective speaker embeddings (350) extracted from the corresponding set of the spoken training utterances (305); 20. The system (100) of claim 19, comprising:

21. training the speech conversion model (210) and the speaker embedding model simultaneously further comprises training a sub-model with the plurality of sets of training utterances (305) to learn how to bias the speech conversion model (210); 21. The system (100) of claim 19 or 20, wherein parameters of the speech conversion model (210) are frozen during training of the speech embedding network (250) and the sub-models with the multiple sets of training utterances (305).

22. 22. The system (100) of claim 19, wherein one or more sets among the plurality of sets of training utterances (305) each include atypical speech patterns for each type of atypical speech associated with each different training speaker that differ from each type of atypical speech associated with each different training speaker.

Images