Speaker embedding for improved automatic speech recognition

Abstract

A method (400) includes receiving a reference audio signal corresponding to a reference speech spoken by a target speaker having an atypical speech; and generating a speaker insert (350) for the target speaker by a speaker insert network (250) configured to receive the reference audio signal as input. The speaker embedding conveys speaker characteristics of the target speaker. The method further includes receiving a voice conversion request, the voice conversion request including input audio data corresponding to an utterance spoken by a target speaker (102), the target speaker associated with an atypical voice. The method further includes biasing a speech conversion model (210) to convert input audio data corresponding to the utterance spoken by the target speaker into an output specification representation of the utterance spoken by the target speaker using speaker embedding generated by the speaker embedding network for the target speaker, the target speaker is associated with an atypical speech.

Description

Technical Field

[0001] The present disclosure relates to using speaker embeddings for improved automatic speech recognition. Background Art

[0002] Automatic speech recognition (ASR), the process of taking audio input and transcribing it into text, has become an important technology used in mobile devices and other devices. Generally, automatic speech recognition attempts to provide an accurate transcription of what a person says by taking audio input (e.g., voice utterances) and transcribing the audio input into text.

[0003] One challenge in developing deep learning-based speech conversion and ASR models is that the parameters of these models tend to overfit the training data, resulting in difficulty generalizing to unseen data when the training data is not extensive enough. While training speech conversion and ASR models on larger training datasets improves accuracy, the lack of sufficient training data (including utterances for specific domains (i.e., linguistic diversity) spoken by speakers with atypical speech patterns (i.e., acoustic diversity)) prevents these models from achieving acceptable accuracy. Summary of the invention

[0004] One aspect of the present disclosure provides a computer-implemented method for personalizing a speech conversion model using speaker embeddings for a target speaker associated with an atypical speech. The computer-implemented method, when executed on data processing hardware, causes the data processing hardware to perform operations, the operations comprising: receiving a reference audio signal corresponding to a reference speech spoken by a target speaker with an atypical speech; and generating a speaker embedding for the target speaker by a speaker embedding network configured to receive the reference audio signal as input. The speaker embedding conveys speaker characteristics of the target speaker. The operation also includes receiving a speech conversion request, the speech conversion request including input audio data corresponding to an utterance spoken by a target speaker associated with an atypical speech. The operation also includes biasing the speech conversion model using a speaker embedding generated by the speaker embedding network for the target speaker to convert input audio data corresponding to an utterance spoken by the target speaker into an output canonical representation of the utterance spoken by the target speaker, the target speaker being associated with an atypical speech.

[0005] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the speech conversion model includes a speech-to-speech conversion model that is configured to directly convert an input spectrogram or audio waveform into an output spectrogram or audio waveform. In these implementations, the biased speech conversion model includes using speaker embeddings to bias the speech-to-speech conversion model to convert input audio data corresponding to an utterance spoken by a target speaker into output audio data corresponding to a synthesized canonical fluent speech representation of an utterance spoken by the target speaker, the target speaker being associated with an atypical speech. Here, the output canonical representation includes a synthesized canonical fluent speech representation.

[0006] In some examples, the speech conversion model includes an automated speech recognition model configured to convert speech to text. In these examples, biasing the speech conversion model includes biasing the automated speech recognition model using speaker embeddings to convert input audio data corresponding to an utterance spoken by a target speaker into a canonical text representation of the utterance spoken by the target speaker, the target speaker being associated with the atypical speech. Here, the output canonical representation includes the canonical text representation.

[0007] In some implementations, the operations further include determining a personalized embedding for a target speaker based on the speaker embedding using a speaker embedding network, wherein the personalized embedding corresponds to a corresponding style cluster of speaker embeddings extracted from training utterances spoken by the training speaker and conveying speaker characteristics similar to the speaker characteristics conveyed by the speaker embedding. Here, biasing the speech conversion model includes biasing the speech conversion model using the personalized embedding determined for the target speaker for a type of atypical speech associated with the target speaker. In these implementations, biasing the speech conversion model using the personalized embedding determined for the target speaker may include providing the personalized embedding as a side input to the speech conversion model for biasing the speech conversion model for the type of atypical speech associated with the target speaker. Optionally, biasing the speech conversion model using the personalized embedding determined for the target speaker may include providing the personalized embedding as a side input to a sub-model of the biased speech conversion model for the type of atypical speech associated with the target speaker. The operations may also include selecting a sub-model from a plurality of different sub-models using the personalized embedding determined for the target speaker before biasing the speech conversion model. Here, each of the multiple different sub-models can be trained for a different type or severity of atypical speech.

[0008] In some examples, the training process trains the speech conversion model and the speaker embedding network concurrently end-to-end by obtaining multiple sets of spoken training utterances and concurrently training the speech conversion model and the speaker embedding model on the multiple sets of spoken training utterances. Each set of spoken training utterances is spoken by a different respective training speaker and includes: audio data representing the training utterances, the audio data including atypical speech patterns of an atypical speech type associated with the respective training speaker; and canonical transcriptions of the training utterances spoken by the respective training speaker.

[0009] The speaker embedding network may include a reference encoder and a style attention module, wherein the reference encoder is configured to extract a corresponding speaker embedding from each set of spoken training utterances for a corresponding training speaker. Training the speech conversion model and the speaker embedding network may include: training the style attention module to learn how to group speaker embeddings into style clusters, wherein each style cluster may represent a corresponding cluster of speaker embeddings extracted from training utterances spoken by training speakers having similar speaker characteristics, and may be mapped to a corresponding personalized embedding representing a corresponding type of atypical speech. For each corresponding set in a plurality of sets of spoken training utterances, training the speech conversion model includes biasing the speech conversion model for the corresponding set of spoken training utterances using a corresponding personalized embedding mapped to a style cluster including the corresponding speaker embedding extracted from the corresponding set of spoken training utterances, wherein the style cluster includes the corresponding speaker embedding extracted from the corresponding set of spoken training utterances.

[0010] In some implementations, concurrently training the speech conversion model and the speaker embedding model further comprises training a sub-model on multiple sets of training utterances to learn how to bias the speech conversion model. Here, parameters of the speech conversion model are frozen while the speech embedding network and the sub-model are trained on multiple sets of training utterances. In some examples, one or more of the multiple sets of training utterances each include an atypical speech pattern of a corresponding atypical speech type associated with a different corresponding training speaker, the corresponding atypical speech type associated with the different corresponding training speaker being different from the corresponding atypical speech type associated with each other different corresponding training speaker.

[0011] Another aspect of the present disclosure provides a system for training a speech conversion model and a speaker personalization network for a target speaker associated with an atypical speech. The system includes data processing hardware and memory hardware, the memory hardware communicating with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include: receiving a reference audio signal corresponding to a reference speech spoken by a target speaker with an atypical speech; and generating a speaker embedding for the target speaker by a speaker embedding network configured to receive the reference audio signal as input. The speaker embedding conveys speaker characteristics of the target speaker. The operations also include receiving a speech conversion request, the speech conversion request including input audio data corresponding to an utterance spoken by a target speaker, the target speaker being associated with an atypical speech. The operations also include biasing the speech conversion model using a speaker embedding generated by the speaker embedding network for the target speaker to convert the input audio data corresponding to an utterance spoken by the target speaker into an output canonical representation of the utterance spoken by the target speaker, the target speaker being associated with an atypical speech.

[0012] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the speech conversion model includes a speech-to-speech conversion model that is configured to directly convert an input spectrogram or audio waveform into an output spectrogram or audio waveform. In these implementations, the biased speech conversion model includes using speaker embeddings to bias the speech-to-speech conversion model to convert input audio data corresponding to an utterance spoken by a target speaker into output audio data corresponding to a synthesized canonical fluent speech representation of an utterance spoken by the target speaker, the target speaker being associated with an atypical speech. Here, the output canonical representation includes a synthesized canonical fluent speech representation.

[0013] In some examples, the speech conversion model includes an automated speech recognition model configured to convert speech to text. In these examples, biasing the speech conversion model includes biasing the automated speech recognition model using speaker embeddings to convert input audio data corresponding to an utterance spoken by a target speaker into a canonical text representation of the utterance spoken by the target speaker, the target speaker being associated with the atypical speech. Here, the output canonical representation includes the canonical text representation.

[0014] In some implementations, the operations further include determining a personalized embedding for a target speaker based on the speaker embedding using a speaker embedding network, wherein the personalized embedding corresponds to a corresponding style cluster of speaker embeddings extracted from training utterances spoken by the training speaker that convey speaker characteristics similar to the speaker characteristics conveyed by the speaker embedding. Here, biasing the speech conversion model includes biasing the speech conversion model using the personalized embedding determined for the target speaker for a type of atypical speech associated with the target speaker. In these implementations, biasing the speech conversion model using the personalized embedding determined for the target speaker may include providing the personalized embedding as a side input to the speech conversion model for biasing the speech conversion model for the type of atypical speech associated with the target speaker. Optionally, biasing the speech conversion model using the personalized embedding determined for the target speaker may include providing the personalized embedding as a side input to a sub-model of the biased speech conversion model for the type of atypical speech associated with the target speaker. The operations may also include selecting a sub-model from a plurality of different sub-models using the personalized embedding determined for the target speaker before biasing the speech conversion model. Here, each of the multiple different sub-models can be trained for a different type or severity of atypical speech.

[0015] In some examples, the training process trains the speech conversion model and the speaker embedding network concurrently end-to-end by obtaining multiple sets of spoken training utterances and concurrently training the speech conversion model and the speaker embedding model on the multiple sets of spoken training utterances. Each set of spoken training utterances is spoken by a different respective training speaker and includes: audio data representing the training utterances, the audio data including atypical speech patterns of an atypical speech type associated with the respective training speaker; and canonical transcriptions of the training utterances spoken by the respective training speaker.

[0016] The speaker embedding network may include a reference encoder and a style attention module, wherein the reference encoder is configured to extract a corresponding speaker embedding from each set of spoken training utterances for a corresponding training speaker. Training the speech conversion model and the speaker embedding network may include: training the style attention module to learn how to group speaker embeddings into style clusters, wherein each style cluster may represent a corresponding cluster of speaker embeddings extracted from training utterances spoken by training speakers having similar speaker characteristics, and may be mapped to a corresponding personalized embedding representing a corresponding type of atypical speech. For each corresponding set in a plurality of sets of spoken training utterances, training the speech conversion model includes biasing the speech conversion model for the corresponding set of spoken training utterances using a corresponding personalized embedding mapped to a style cluster including the corresponding speaker embedding extracted from the corresponding set of spoken training utterances.

[0017] In some implementations, concurrently training the speech conversion model and the speaker embedding model further comprises training a sub-model on multiple sets of training utterances to learn how to bias the speech conversion model. Here, parameters of the speech conversion model are frozen while the speech embedding network and the sub-model are trained on multiple sets of training utterances. In some examples, one or more of the multiple sets of training utterances each include an atypical speech pattern of a corresponding atypical speech type associated with a different corresponding training speaker, the corresponding atypical speech type associated with the different corresponding training speaker being different from the corresponding atypical speech type associated with each other different corresponding training speaker.

[0018] The details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1A is a schematic diagram of an example speech conversion system including a speech-to-speech conversion model.

[0020] Figure 1B is a schematic diagram of an example speech conversion system including a speech-to-text conversion model.

[0021] Figure 2 is a diagram of an example training process for concurrently training a speaker embedding network and an end-to-end speech conversion model.

[0022] Figure 3 is a diagram of an example deterministic reference encoder for extracting speaker embeddings.

[0023] Figure 4 is a flow diagram of an example arrangement of operations of a method for personalizing a speech conversion model using speaker embeddings for a target speaker with atypical speech.

[0024] Figure 5 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0025] Like reference numbers in the various drawings indicate like elements. DETAILED DESCRIPTION

[0026] Automated speech recognition (ASR) systems are often optimized to best fit speakers with canonical speech patterns. Unfortunately, these systems can perform poorly when tested on speakers associated with atypical and heavily accented speech. The poor performance on these speakers is because the vast majority of the speech data used to train these ASR systems is derived from speakers speaking a canonical accent with a typical speech pattern, which causes the ASR system to favor this accent, resulting in poor performance on other accents.

[0027] ASR systems have made great progress with the introduction of end-to-end (E2E) deep learning-based models to recognize speech from speakers with atypical speech patterns for conversion into accurate transcriptions. For example, atypical speech patterns may include, but are not limited to, impaired speech, heavily accented speech, and deaf speech due to physical or neurological conditions (e.g., speakers with amyotrophic lateral sclerosis (ALS)). Similar deep learning-based models can be applied by a speech-to-speech conversion system to convert speech with atypical speech patterns into canonical fluent output speech. As used herein, and unless otherwise specified, the terms "speech conversion system" and "speech conversion model" may refer to an ASR system / model or a speech-to-speech conversion system / model, in which an input atypical speech is recognized and converted into corresponding text (e.g., transcription), in which an input atypical speech is directly converted into canonical fluent synthesized speech without performing speech recognition. In other words, the speech-to-speech conversion system / model is configured to directly convert an input audio waveform or spectrogram corresponding to an atypical speech into an output audio waveform or spectrogram corresponding to a canonical fluent speech without converting the input audio waveform into an intermediate representation (e.g., text or phonemes). As will become apparent, the speech conversion model and techniques for training the speech conversion model will enable users with atypical speech to speak to and be understood by other humans and speech interfaces (e.g., digital assistants) by enabling recognition and reproduction of the user's intended speech.

[0028] One challenge in developing deep learning-based speech conversion models and ASR models is that the parameters of these models tend to overfit the training data, resulting in difficulty generalizing to unseen data when the training data is not extensive enough. While training speech conversion models and ASR models on larger training datasets improves accuracy, there is a lack of training data that provides enough acoustic diversity to personalize for target speakers with atypical speech. For example, in order to obtain training data with sufficient acoustic diversity, a target speaker with atypical speech would have to record hours of spoken utterances, each of which is paired with a corresponding transcription.

[0029] One approach for recognizing speech from speakers who exhibit atypical or heavily accented speech is to create personalized speech conversion models (one per speaker) that have the potential to significantly improve speech recognition for individual speakers. While per-speaker personalized speech conversion models are generally effective, a disadvantage of this approach is scalability, since a separate model must be trained and maintained for each speaker. Another disadvantage of per-speaker personalized speech conversion models is that they typically need to be run on specialized computing resources (e.g., tensor processing units (TPUs), graphics processing units (GPUs), etc.), which are not readily available when served on servers, or are only available on high-end consumer computing devices (e.g., smartphones).

[0030] The implementation herein is directed to building a speech conversion system that adapts to atypical speaking patterns to perform equally well in recognizing / converting speech from people who speak with different accents, locals, dialects, or different ways of speaking. Specifically, the implementation is directed to concurrently training a speaker embedding network and a speech conversion model to provide a personalizable speech conversion system that shares training data across speakers with similar acoustic-phonetic properties and implements implicit clustering of speaker embeddings.

[0031] Figure 1A and Figure 1B A speech conversion system 100 , 100a - 100b is shown, which includes a speech conversion model 210 and a speaker embedding network 250 . Figure 1AA speech-to-speech (S2S) conversion model 210, 210a is shown that is configured to convert input audio data 102 (e.g., an input spectrogram or an input audio waveform) corresponding to an utterance 108 spoken by a target speaker 104 associated with an atypical voice directly into output audio data 106 (e.g., an output spectrogram or an output audio waveform) corresponding to a synthesized canonical fluent speech representation of the same utterance 114 spoken by the target speaker 104. Notably, the S2S conversion model 210a is configured to convert the input audio data 102 directly into the output audio data 106 without performing speech recognition or otherwise requiring the generation of any intermediate discrete representation (e.g., text or phonemes) from the input audio data 102. The S2S conversion model 210a includes: an audio encoder 212, which is configured to encode the input audio data 102 into a hidden feature representation (e.g., a series of vectors); and a speech decoder 220, which is configured to decode the hidden representation into output audio data 106 corresponding to the synthesized canonical fluent speech representation. For example, when the audio encoder 210 receives the input audio data 102 of the speech 108, the audio encoder 210 can process five frames of audio and convert the five frames of audio into ten vectors. The vectors are not transcriptions of the frames of the audio data 102, but mathematical representations of the frames of the audio data 102. In turn, the speech decoder 220 can generate output audio data 106 corresponding to the synthesized canonical fluent speech representation based on the vectors received from the audio encoder 210. For example, the speech decoder 220 can receive ten vectors representing five frames of audio from the audio encoder 210. Here, the speech decoder 220 may generate five frames of output audio data 106 corresponding to a synthesized canonical fluent speech representation of the utterance 114, wherein the output audio data includes the intended words or word parts of the five frames of the input audio data 102, but without the disfluencies of the atypical speech.

[0032] The S2S conversion system 100a may also include a synthesizer 275 for synthesizing the output audio data 106 into a time domain waveform to be audibly output as the same speech 114 of fluent canonical fluent speech. The time domain audio waveform includes an audio waveform that defines the amplitude of the audio signal over time. The synthesizer 275 may include a unit selection module or a WaveNet module for synthesizing the output audio data 106 into a time domain waveform of synthesized canonical fluent speech. In some implementations, the synthesizer 275 includes a vocoder network that is trained and adjusted separately on a Mel frequency spectrogram for conversion into a time domain audio waveform, i.e., a neural vocoder.

[0033] In the example shown, the target speaker 104 is associated with an atypical speech, such that the target speaker 104 speaks in an atypical speech pattern that may be difficult to understand. Atypical speech patterns may include, but are not limited to, impaired speech due to a physical or neurological condition (e.g., a speaker with amyotrophic lateral sclerosis (ALS)), heavily accented speech, and deaf speech. The speaker embedding network 250 generates a speaker embedding 350 that represents the speech of the target speaker 104 and speaker characteristics of the target speaker 104. The speaker characteristics may convey an accent (e.g., a French accent) and a speech pattern associated with the target speaker 104. Here, the speech pattern conveyed by the speaker embedding 350 may include a typical speech pattern or one of a plurality of different types of atypical speech patterns (e.g., heavy accents across different dialects and / or irregular speech across different neurological conditions). Notably, and as described below with reference to Figure 2 As described in more detail, the speaker embedding network 250 can be trained concurrently with the S2S conversion model 210a, or with and / or with a sub-model 280 ( Figure 2 ) are trained concurrently. By concurrently training the speaker embedding network 250 and the S2S conversion model 210 or sub-model 280, the speaker embedding network 250 is able to generate a speaker embedding 350 that is better for assisting the S2S conversion model 210a in converting atypical speech patterns into canonical speech. In contrast, separately trained speaker embedding models (such as those implemented to generate speaker discriminative embeddings for voice authentication / verification) are typically trained to recognize the speaker's voice and ignore any accent / speaker characteristics.

[0034] In some implementations, the S2S conversion model 210a receives the speaker embedding 350 generated by the speaker embedding network 250 as a side input for biasing / influencing the S2S conversion model 210a to directly convert the input audio data 102 corresponding to the utterance 108 spoken by the target speaker 104 associated with the atypical speech into the output audio data 106 corresponding to the synthesized canonical fluent speech representation of the same utterance 108. In other implementations, the speaker embedding network 250 also generates a personalized embedding 251 based on the speaker embedding 350 and provides the personalized embedding 251 as a side input to the sub-model 280 or the S2S conversion model 210a for biasing / influencing the S2S conversion model 210a to directly convert the utterance 108 of the atypical speech into the synthesized canonical fluent speech representation of the same utterance 108. As described below with reference to Figure 2Described in more detail, personalized embedding 251 may represent training utterance 305 ( Figure 2 ). Thus, personalized embedding 251 may correspond to a context vector used to parameterize / personalize sub-model 280 and / or S2S conversion model 210a to improve speech conversion performance on atypical speech types associated with target speaker 104. For example, personalized embedding 251 may indicate an atypical speech type (e.g., ALS speech) associated with target speaker 104, or may even provide more fine-grained information, such as a severity level of an atypical speech type.

[0035] The speech-to-speech conversion system 100a is accordingly trained to convert input audio data 102 corresponding to an utterance 108 spoken by a target speaker 104 associated with an atypical speech directly into output audio data 106 corresponding to a synthesized canonical fluent speech representation of the same utterance 108. Thus, the synthesized canonical fluent speech representation provided by the output audio data 106 improves the intelligibility of the atypical speech (e.g., heavily accented speech or ALS speech) spoken by the target speaker 104. Without departing from the scope of the present disclosure, the speech-to-speech conversion system 100a may be trained to convert input audio data 102 corresponding to an utterance 108 associated with an atypical speech in a first language directly into output audio data 106 corresponding to a synthesized canonical fluent speech representation of the same utterance 108 in the same voice but a different second language.

[0036] A computing device 110 associated with a target speaker 104 may capture an utterance 108 spoken by the target speaker 104 and send corresponding input audio data 102 to a speech-to-speech conversion system 100a for conversion into output audio data 106. Thereafter, the speech-to-speech conversion system 100a may send output audio data 106 corresponding to a synthesized canonical fluent speech representation of the same utterance 114 spoken by the target speaker 104 to another computing device 116 associated with a user 118, whereby the other computing device 116 audibly outputs a synthesized canonical fluent speech representation of the utterance 108 spoken by the target speaker 104. In this example, the target speaker 104 and the user 118 speak to each other via their respective computing devices 110, 116 (such as via a telephone call or other type of voice communication protocol, e.g., voice over internet protocol). Although the target speaker 104 and another user 118 may speak the same language, the other user 118 may have difficulty understanding the target speaker 104 because the target speaker 104 has an atypical voice due to the ALS disease. Therefore, although the target speaker 104 speaks an atypical voice (e.g., ALS voice) that may be difficult to understand, the other user 118 who hears the synthesized canonical fluent voice representation will more easily understand the speech 108 intended by the target speaker 104. In other words, the synthesized canonical fluent voice representation provides a more consistent intonation (cadence) that may be easier for another user to understand than the original speech 108 spoken by the target speaker with an atypical voice. It is noteworthy that the synthesized canonical fluent voice representation is the voice of the target speaker 104.

[0037] In some other examples, the S2S conversion system 100a instead transmits output audio data 106 corresponding to the synthesized canonical fluent speech representation of the utterance spoken by the target speaker 104 to an output audio device for audibly outputting the synthesized canonical fluent speech representation to a listener in the voice of the target speaker 104. For example, the target speaker 104 may be a psychology professor lecturing a class of students, where the utterance spoken by the target speaker 104 includes medical terms belonging to a particular specific field (e.g., psychology). As will become apparent, the S2S conversion system 100a is trained to learn the linguistic diversity from the linguistic content present in the training utterances as well as the acoustic diversity associated with the particular type of atypical speech associated with the speaker speaking the target utterance.

[0038] Alternatively, another computing device 116 can be associated with a downstream automated speech recognition (ASR) system, wherein the speech-to-speech conversion system 100a is used as a front end to provide output audio data 106 corresponding to the synthesized canonical fluent speech representation as an input of the ASR system for conversion into recognized text. The recognized text can be presented to another user 118 and / or can be provided to a natural language understanding (NLU) system for further processing. The functionality of the speech-to-speech conversion system 100a can reside on a remote server 112, on any one or two of the computing devices 110, 116, or on any combination of a remote server and a computing device 110, 116. In some implementations, when the target speaker 104 speaks the corresponding portion of the speech as an atypical speech, the S2S conversion model 210a continuously generates output audio data 106 corresponding to the synthesized canonical fluent speech representation of the speech. By continuously generating output audio data 106 corresponding to synthesized canonical fluent speech representations of portions of an utterance 108 spoken by a target speaker 104, a conversation between the target speaker 104 and a user 118 (or listener) may have a more natural rhythm. In some additional implementations, the S2S conversion model 210a uses techniques such as voice activity detection, end pointing, end of query detection, etc. and waits to determine / detect when the target speaker 104 stops speaking before converting the corresponding input audio data 102 of an utterance 108 having an atypical voice into the corresponding output audio data 106 corresponding to the synthesized canonical fluent speech representation of the same utterance 114.

[0039] Reference now Figure 1B In some implementations, the speech conversion system 100, 100b includes a speech-to-text conversion system 100b configured to convert input audio data 102 corresponding to an utterance 108 spoken by a target speaker 104 associated with an atypical voice into a canonical text representation (i.e., transcription) 120 of the utterance 108. Thus, the speech-to-text conversion system 100b may correspond to a personalized automated speech recognition (ASR) system for the target speaker 104 that may recognize a specific type of atypical speech pattern and language information of the target speaker to convert into a corresponding canonical text representation 120 that captures the intent of the original utterance 108 spoken by the target speaker 104 associated with the atypical voice. Another user 118 ( Figure 1A) can obtain a canonical text representation 120 of the utterance 108. In some configurations, the canonical text representation 120 output from the system 100b is processed by a natural language understanding (NLU) module, for example, executed on the user device 110 or the remote server 112, to execute the user command. Additionally or alternatively, a text-to-speech system (e.g., executed on any combination of the user device 110 or the remote server 112) can convert the transcription into synthesized speech for audible output by another device. The functionality of the speech-to-text conversion system 100b can reside on the remote server 112, on either or both of the computing devices 110, or on any combination of the remote server 112 and the computing device 110.

[0040] The speech conversion model 210 associated with the speech-to-text conversion system 100b may include a speech-to-text conversion model 210b (interchangeably referred to as an automated speech recognition (ASR) model 210b) configured to perform speech recognition on an utterance 108 of atypical speech by converting the input audio data 102 into a canonical text representation (i.e., transcription) 120 of the utterance 108. The ASR model 210b includes: the same or different audio encoder 210, the same or different audio encoder configured to encode the input audio data 102 into a hidden feature representation (e.g., a series of vectors); and a text decoder 225, the text decoder configured to decode the hidden representation into a canonical transcription 120. For example, when the audio encoder 212 receives the input audio data 102 of the utterance 108, the encoder 212 may process five frames of audio and convert the five frames of audio into ten vectors. The vectors are not transcriptions of the frames of audio data 102, but mathematical representations of the frames of audio data 102. In turn, the text decoder 225 can generate a canonical transcription 120 based on the vectors received from the encoder 212. For example, the text decoder 225 can generate a word sequence corresponding to the canonical transcription 120 of the utterance 180, the word sequence including the intended words or word parts in the five frames of the input audio data 102. Without departing from the scope of the present disclosure, the speech-to-text conversion system 100b can be trained to convert the input audio data 102 corresponding to the utterance 108 in a first language and associated with an atypical speech into a corresponding canonical transcription of the utterance 108 in a different second language.

[0041] refer to Figure 1A , Figure 1B and Figure 2 , the speech conversion system 100 performs a training process 200 configured to train a speech conversion model 210 concurrently with a speaker embedding network 250 end-to-end, i.e., Figure 1A The S2S conversion model 210a and / or Figure 1BThe speech-to-text conversion model 210b is used to improve the performance of the speech conversion system 100 in recognizing and / or converting different types of atypical speech that are typically under-represented in general training data.

[0042] Figure 2 The training process 200 is shown to obtain multiple sets of spoken training utterances 305, 305a to 305n, each set including audio data 304 representing a training utterance 305 spoken by a corresponding training speaker. The sets of spoken training utterances 305 can be stored in the memory hardware 520 (residing at the user device 110 or the remote server 112) Figure 5 ) in a data store 202 on a computer readable medium. Here, each set of training utterances 305 spoken by a corresponding training speaker includes a corresponding transcription 302 paired with audio data 304 representing the training utterance. Thus, each set of spoken training utterances 305 includes manually transcribed audio data 302, 304 spoken by the corresponding training speaker. The audio data 304 corresponds to a speech representation of the corresponding training utterance 305, which is the voice of the corresponding training speaker who spoke the training utterances 305 in each set of spoken training utterances 305 and conveys speaker characteristics (e.g., accent and / or speech pattern) of the corresponding training speaker. While the examples include audio data 304 corresponding to non-synthesized speech representations (e.g., speech originating from a human), some of the training utterances 305 may include audio data 304 corresponding to synthesized / synthesized speech representations for specific voices and speech patterns.

[0043] Implementations herein involve multiple sets of spoken training utterances 305, each set including an atypical speech pattern of an atypical speech type associated with a respective training speaker that spoke the training utterances 305. For example, one or more of the sets of spoken training utterances 305 may be spoken by a training speaker that exhibits an atypical speech pattern of ALS speech, one or more other sets of spoken training utterances 305 may be spoken by a training speaker that exhibits an atypical speech pattern of dysarthria, one or more other sets of spoken training utterances 305 may be spoken by a training speaker that exhibits an atypical speech pattern of heavily accented speech in a given dialect (e.g., a French accent), and so on. Each transcription 302 may be a canonical transcription of the native language of the target speaker 104. In some examples, some or all of the spoken training utterances 305 include words and / or phrases related to a specific domain that the speech conversion model 210 is trained to learn.

[0044] The speech embedding network 250 includes a reference encoder 300 and a style attention module 254. During the training process 200, the reference encoder 300 receives as input audio data 304 representing training utterances 305 spoken by a corresponding training speaker associated with each set of training utterances 305, and generates a corresponding speaker embedding (Z ) for the corresponding training speaker. E ) 350 as output. Here, the speaker embedding 350 output from the reference encoder 300 represents the speech of the corresponding training speaker and the speaker characteristics of the corresponding training speaker. As previously described, the speaker characteristics can convey the accent (e.g., French accent) and speech pattern associated with the training speaker. Here, the speech pattern conveyed by the speaker embedding 350 can include a typical speech pattern or one of a plurality of different types of atypical speech patterns (e.g., heavy accents across different dialects, unconventional speech across different neurological conditions).

[0045] refer to Figure 3 In some implementations, the example reference encoder is configured to receive a reference audio signal 312 (e.g., training audio data 304 or input audio data 102 corresponding to a reference speech spoken by a target speaker 104) and generate / predict a speaker embedding based on the reference audio signal 312. E 350. The reference audio signal 312 may have a duration sufficient to capture various acoustic-phonetic features associated with the speaker. In some examples, the reference audio signal 312 comprises a duration of approximately one (1) minute. Speaker Embedding Z E 350 may capture speaker characteristics of the reference audio signal 312 , which convey accents and speech patterns associated with the reference audio signal 312 .

[0046] In the example shown, the reference audio signal 312 may be represented as having a length L R and size D R Spectrogram slices. The spectrogram slices associated with the reference audio signal 312 may indicate a Mel-warped spectrum. In some examples, the reference encoder 300 includes a six-layer convolutional layer network 314, where each layer includes 3×3 filters with 2×2 stride, SAME padding, and ReLU activation. Batch normalization is applied to each layer, and the number of filters in each layer is doubled by half the sampling rate: 32, 32, 64, 128, 128. A recurrent neural network (RNN) 310 with a bidirectional long short-term memory (LSTM) layer receives the output 316 from the last convolutional layer to provide an output 322, which is applied to a fully connected layer 320, followed by an activation function 330, which outputs a speaker embedding Z E350. Instead of an RNN, the reference encoder 300 may incorporate a single 128-width gated recurrent unit (GRU-RNN) layer that receives the output 316 from the last convolutional layer and outputs a 128-dimensional output 322, which is applied to a fully connected layer 320.

[0047] Return to reference Figure 2 In the training process 200 of FIG. 2 , the style attention module 254 receives the speaker embeddings 350 output from the reference encoder 300 and uses the speaker embeddings 350 as query vectors to learn similarities between the speaker embeddings 350. More specifically, the training process 200 trains the style attention module 254 to learn how to group the speaker embeddings 350 into style clusters 260, 260a to 260n, each of which represents a corresponding cluster of speaker embeddings 350 extracted from training utterances 305 spoken by training speakers having similar characteristics (i.e., accents and speech patterns). For example, the first style cluster 260a may include speaker embeddings 350 extracted from training utterances 305 spoken by training speakers associated with ALS speech, and the second style cluster 260a may include speaker embeddings 350 extracted from training utterances 305 spoken by training speakers associated with Down syndrome speech.

[0048] Furthermore, each style cluster 260 into which the style attention module 254 learns to group similar speaker embeddings 350 need not represent a corresponding atypical speech type that is different from atypical speech types represented by other style clusters. That is, one style cluster 260 may include a grouping of similar speaker embeddings 350 extracted from training utterances 305 spoken by training speakers associated with a first severity level of one type of atypical speech, while another style cluster 260 may include another cluster of speaker embeddings 350 extracted from training utterances 305 spoken by other training speakers associated with a second severity level of the same type of atypical speech. Thus, the style attention module 254 learns how to group similar speaker embeddings 350 into clusters 260 representing different types (and / or severity levels) of atypical speech (e.g., heavy accents across different dialects and / or atypical speech across different neurological conditions). The attention module 254 may be viewed as a mechanism configured to map new speaker embeddings 350 to corresponding style clusters 260 having speaker embeddings that convey similar speaker characteristics.

[0049] During the training process 200, the style attention module 254 can learn to generate a personalized embedding (h) 251 for each respective speaker embedding 350 based on the corresponding style cluster 260 into which the respective speaker embedding 250 is grouped. In some configurations, the style attention module 254 receives speaker embeddings 350 extracted from a set of training utterances 305 spoken by the respective speaker and outputs a set of combined weights representing the contribution of each style cluster 260 to the corresponding speaker embedding 350. In these configurations, the weighted sum of the style clusters 260 corresponds to the personalized embedding 251 that is input to the sub-model 280 and / or the speech conversion model 210 for conditioning at each time step.

[0050] In some implementations, the personalized embedding 251 generated for each respective speaker embedding 350 extracted during the training process 200 corresponds to a context vector that parameterizes the speech conversion model 210 as an additional side input, thereby generating an implicit classifier for the respective training speaker when the speech conversion model 210 is concurrently trained on the training utterances 305 spoken by the training speaker. Thus, when the speech conversion model 210 is trained on a set of training utterances 305 spoken by the training speaker, the speech conversion model 210 can receive as input the personalized embedding 251 (or speaker embedding 350) for the respective training speaker. For example, the personalized embedding 251 can indicate that the training speaker is grouped into a style cluster 260 associated with a heavy French accent, thereby allowing the speech conversion model 210 to adapt to improve performance in recognizing / converting training utterances 305 conveying a heavy French accent.

[0051] In additional implementations, the personalized embedding 251 (or speaker embedding 350) is instead provided as a side input to the sub-model 280 for training the sub-model 280 on the training utterances 305 spoken by the training speaker. In these implementations, the speech conversion model 210 includes a base model associated with an off-the-shelf pre-trained speech conversion model 210. For example, the pre-trained speech conversion model 210 may include a prior art model trained on a general corpus of training utterances spoken by multiple different speakers with typical speech patterns, and the sub-model 280 may be trained on the training utterances 305 to learn how to bias the pre-trained speech conversion model 210 to improve performance in recognizing / converting the training utterances 305 associated with atypical speech. Training the sub-model 280 as a modular extension for biasing the pre-trained base speech conversion model 210 advantageously helps during inference when reference speech samples are not available from the target speaker by allowing the pre-trained base speech conversion model 210 to process input utterances regardless of speaker characteristics. For example, the target speaker uttering the initial input utterance for conversion / recognition by the speech conversion system 100 may not be of sufficient duration for the speech embedding network 250 to extract therefrom the speaker embedding and / or the personalized embedding 251. However, once sufficient duration of audio data conveying the input utterance is available for the speaker embedding network 250 to extract therefrom the speaker embedding 350 and / or the personalized embedding 251, the speaker embedding network 250 may provide the speaker embedding 350 and / or the personalized embedding 251 as a side input to the sub-model 280 for biasing the speech conversion model 210.

[0052] In some examples, the sub-model 210 is disposed in a neural network layer of the audio encoder 212 or between two neural network layers of the audio encoder 212. In other examples, the sub-model 210 is disposed in a neural network layer of the decoder 220, 225 or between two neural network layers of the decoder 220, 225. In some additional examples, the first sub-model 210 is disposed in the audio encoder 212 and the second sub-model 210 is disposed in the decoder 220, 225 of the pre-trained speech conversion model 210. During the training process 200, when the speech embedding network 250 and the sub-model 280 are trained on the training utterance 305, the parameters of the pre-trained speech conversion model 310 can be frozen.

[0053] In some implementations, a single sub-model 280 is trained to bias / influence the speech conversion model 210 on a diverse set of different types of atypical speech. During inference, the reference encoder 300 may receive as input a reference audio signal 312 corresponding to a reference speech spoken by a given target speaker 104 associated with an atypical speech, and generate as output a speaker embedding 350 for the target speaker 104, the speaker embedding conveying speaker characteristics associated with the target speaker 104 (e.g., the type of atypical speech). Using the techniques described above, the style attention mechanism 254 may pay attention to the speaker embedding 350 to determine a personalized embedding 251 for the target speaker 104. The personalized embedding 251 may act as a one-hot vector that adapts the speech conversion model 210 to a specific atypical speech type in the atypical speech type. In additional implementations, multiple sub-models 280 are each trained to bias / influence the speech conversion model 210 for the corresponding atypical speech type. That is, a separate sub-model 280 may be trained for each type of atypical speech identified by the speaker embedding network 250 from the plurality of sets of training utterances 305 during the training process 200. Thus, each style cluster 260 of the speaker embeddings 350 extracted during the training process 200 may be mapped to a corresponding personalized embedding 251 representing the corresponding type of atypical speech (and optionally the severity level of the type of atypical speech). Thus, the personalized embedding 251 output by the speaker embedding network 250 during inference may serve as a one-hot vector for selecting an appropriate sub-model 280 (i.e., a sub-model corresponding to the type of atypical speech associated with the source speaker 104) from among the plurality of different sub-models.

[0054] Figure 4 A flowchart of an example arrangement of operations of a method 400 for personalizing a speech conversion model 210 using a speaker embedding 350 for a target speaker 104 having an atypical voice is provided. Figure 5 ) may be executed by executing a memory hardware 520 ( Figure 5 ) to perform the operations of method 400. Data processing hardware 510 and memory hardware 520 may reside on computing device 500 ( Figure 5 )(such as Figure 1A and Figure 1B At operation 402, the method 400 includes receiving a reference audio signal 312 corresponding to a reference speech spoken by a target speaker 104 having an atypical speech.

[0055] At operation 404, the method 400 includes generating, by the speaker embedding network 250 configured to receive as input the reference audio signal 312, a speaker embedding 350 for the target speaker 104. Here, the speaker embedding 350 conveys speaker characteristics of the target speaker 104. The speaker characteristics may convey an accent (e.g., a French accent) and a speech pattern associated with the target speaker 104. For example, the speech pattern conveyed by the speaker embedding 350 may include a typical speech pattern or one of a plurality of different types of atypical speech patterns (e.g., heavy accents across different dialects, unconventional speech across different neurological conditions).

[0056] At operation 406, the method 400 includes receiving a speech conversion request including input audio data 102 corresponding to an utterance 108 spoken by a target speaker 104 associated with an atypical speech. At operation 408, the method 400 includes biasing the speech conversion model 210 using the speaker embedding 350 generated by the speaker embedding network 250 for the target speaker 104 to convert the input audio data 102 corresponding to the utterance 108 spoken by the target speaker 104 associated with the atypical speech into an output canonical representation 106, 120 of the utterance spoken by the target speaker.

[0057] In some examples, the speech conversion model 210 includes a speech-to-speech conversion model 210 configured to convert an input spectrogram or audio waveform directly into an output spectrogram or audio waveform. In these examples, the output canonical representation includes output audio data corresponding to a synthesized canonical fluent speech representation of an utterance spoken by a target speaker.

[0058] In other examples, the speech conversion model 210 includes an automated speech recognition model 210 b that is configured to convert speech to text. In these examples, the output canonical representation includes a canonical text representation of the utterance 108 spoken by the target speaker 104 .

[0059] A software application (i.e., software resource) may refer to computer software that enables a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0060] Non-transitory memory can be a physical device used to temporarily or persistently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by a computing device. Non-transitory memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0061] Figure 5 is a schematic diagram of an example computing device 500 that can be used to implement the systems and methods described in this document. Computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions are intended to be exemplary only, and are not intended to limit implementations of the inventions described and / or claimed in this document.

[0062] The computing device 500 includes a processor 510, a memory 520, a storage device 530, a high-speed interface / controller 540 connected to the memory 520 and a high-speed expansion port 550, and a low-speed interface / controller 560 connected to a low-speed bus 570 and the storage device 530. Each of the components 510, 520, 530, 540, 550, and 560 is interconnected using various buses and can be installed on a common motherboard or installed in other ways as appropriate. The processor (i.e., data processing hardware) 510 can process instructions for execution within the computing device 500, including instructions stored in the memory 520 or on the storage device 530, to display graphical information of a graphical user interface (GUI) on an external input / output device (such as a display 580 coupled to the high-speed interface 540). In other implementations, multiple processors and / or multiple buses and multiple memories and multiple types of memories can be used as appropriate. Furthermore, multiple computing devices 500 may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a blade server group, or a multi-processor system). Data processing hardware 510 and memory hardware 520 may reside on Figure 1A and Figure 1B On a remote server 112, any one or both of the computing devices, or any combination of the remote server 112 and the computing devices 110, 116.

[0063] Memory (i.e., memory hardware) 520 stores information non-temporarily within computing device 500. Memory 520 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. Non-temporary memory 520 may be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0064] Storage device 530 is capable of providing mass storage for computing device 500. In some implementations, storage device 530 is a computer-readable medium. In various implementations, storage device 530 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device or array of devices (including devices in a storage area network or other configuration). In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as memory 520, storage device 530, or memory on processor 510.

[0065] The high-speed controller 540 manages bandwidth-intensive operations of the computing device 500, while the low-speed controller 560 manages less bandwidth-intensive operations. Such a division of responsibilities is exemplary only. In some implementations, the high-speed controller 540 is coupled to the memory 520, the display 580 (e.g., through a graphics processor or accelerator), and the high-speed expansion port 550 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 560 is coupled to the storage device 530 and the low-speed expansion port 590. The low-speed expansion port 590, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device, such as a switch or a router, for example, through a network adapter.

[0066] As shown, computing device 500 can be implemented in a variety of different forms. For example, computing device can be implemented as a standard server 500a or multiple implementations in a group of such servers 500a, implemented as a laptop computer 500b, or implemented as part of a rack server system 500c.

[0067] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuit systems, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system that includes at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from and send data and instructions to a storage system, at least one input device, and at least one output device.

[0068] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0069] The processes and logic flows described in this specification can be performed by one or more programmable processors (also referred to as data processing hardware), which execute one or more computer programs to perform functions by operating on input data and generating outputs. The processes and logic flows can also be performed by a dedicated logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). For example, processors suitable for executing computer programs include both general-purpose microprocessors and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include one or more mass storage devices for storing data, such as a magnetic disk, a magneto-optical disk, or an optical disk, or be operably coupled to receive data from one or more mass storage devices or to transfer data to one or more mass storage devices or both. However, a computer does not have to have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and storage devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0070] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor) or a touch screen for displaying information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0071] A variety of implementations have been described. However, it should be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Therefore, other implementations are within the scope of the appended claims.

Claims

1. A computer-implemented method (400) which, when executed on data processing hardware (510), causes the data processing hardware to perform operations comprising: receiving a reference audio signal (312) corresponding to a reference speech spoken by a target speaker having an atypical speech; generating a speaker embedding (350) for the target speaker by a speaker embedding network (250) configured to receive the reference audio signal (312) as input, the speaker embedding (350) conveying speaker characteristics of the target speaker; receiving a speech conversion request including input audio data corresponding to an utterance spoken by the target speaker (102), the target speaker being associated with the atypical speech; as well as A speech conversion model (210) is biased using the speaker embedding (350) generated by the speaker embedding network (250) for the target speaker to convert the input audio data (102) corresponding to the utterance spoken by the target speaker into an output canonical representation (114) of the utterance spoken by the target speaker, the target speaker being associated with an atypical speech.

2. The computer-implemented method (400) of claim 1, wherein: The speech conversion model (210) comprises a speech-to-speech conversion model (210) configured to directly convert an input spectrogram or audio waveform into an output spectrogram or audio waveform; and Biasing the speech conversion model (210) includes biasing the speech-to-speech conversion model (210) using the speaker embedding (350) to convert the input audio data (102) corresponding to the utterance spoken by the target speaker into output audio data (106) corresponding to a synthesized canonical fluent speech representation (114) of the utterance spoken by the target speaker, the target speaker being associated with an atypical speech, wherein the output canonical representation includes the synthesized canonical fluent speech representation (114).

3. The computer-implemented method (400) of claim 1, wherein: The speech conversion model (210) includes an automated speech recognition model configured to convert speech into text; and Biasing the speech conversion model (210) includes biasing the automated speech recognition model using the speaker embedding (350) to convert the input audio data (102) corresponding to the utterance spoken by the target speaker into a canonical text representation (120) of the utterance spoken by the target speaker, the target speaker being associated with an atypical speech, wherein the output canonical representation (114) includes the canonical text representation (120).

4. The computer-implemented method (400) of any one of claims 1 to 3, wherein the operations further comprise: determining, using the speaker embedding network (250), a personalized embedding (251) for the target speaker based on the speaker embedding (350), the personalized embedding (251) corresponding to a respective style cluster of speaker embeddings (350) extracted from training utterances (305) spoken by the speaker and conveying speaker characteristics similar to the speaker characteristics conveyed by the speaker embedding (350), 1920 Wherein biasing the speech conversion model (210) includes biasing the speech conversion model (210) for the type of the atypical speech associated with the target speaker using the personalized embedding (251) determined for the target speaker.

5. The computer-implemented method (400) of claim 4, wherein biasing the speech conversion model (210) using the personalized embedding (251) determined for the target speaker comprises: The personalized embedding (251) is provided as a side input to the speech conversion model (210) for biasing the speech conversion model (210) for the type of the atypical speech associated with the target speaker.

6. The computer-implemented method (400) of claim 4, wherein biasing the speech conversion model (210) using the personalized embedding (251) determined for the target speaker comprises: The personalized embedding (251) is provided as a side input to a sub-model that biases the speech conversion model (210) for the type of the atypical speech associated with the target speaker.

7. The computer-implemented method (400) of claim 6, wherein the operation further comprises, prior to biasing the speech conversion model (210), using the personalized embedding (251) determined for the target speaker to select the sub-model from a plurality of different sub-models, each of the plurality of different sub-models being trained for a different type or severity of atypical speech.

8. The computer-implemented method (400) of any one of claims 1 to 7, wherein the training process (200) trains the speech conversion model (210) and the speaker embedding network (250) concurrently end-to-end by: A plurality of sets of spoken training utterances (305) are obtained, each set of spoken training utterances (305) being spoken by a different respective training speaker and comprising: audio data representing the training utterance (305), the audio data comprising an atypical speech pattern of an atypical speech type associated with the respective training speaker; as well as a canonical transcription (120) of the training utterance (305) spoken by the corresponding training speaker; and The speech conversion model (210) and the speaker embedding network (250) are trained concurrently on the plurality of sets of spoken training utterances (305).

9. The computer-implemented method (400) of claim 8, wherein: The speaker embedding network (250) includes a reference encoder (300) and a style attention module (254), the reference encoder (300) being configured to extract a corresponding speaker embedding (350) from each set of spoken training utterances (305) for the corresponding training speaker; as well as Training the speech conversion model (210) and the speaker embedding network (250) includes: training the style attention module (254) to learn how to group the speaker embeddings (350) into style clusters (260), each style cluster representing a corresponding cluster of speaker embeddings (350) extracted from the training utterances (305) spoken by training speakers having similar speaker characteristics, and each style cluster mapping to a corresponding personalized embedding (251) representing a corresponding type of atypical speech; and For each corresponding set of the plurality of sets of spoken training utterances (305), biasing the speech conversion model (210) for the corresponding set of spoken training utterances (305) using the corresponding personalized embedding (251) mapped to the style cluster, the style cluster comprising the corresponding speaker embedding (350) extracted from the corresponding set of spoken training utterances (305).

10. The computer-implemented method (400) of claim 8 or 9, wherein: Concurrently training the speech conversion model (210) and the speaker embedding network (250) further includes: training a sub-model on the plurality of sets of the training utterances (305) to learn how to bias the speech conversion model (210); and While training the speech embedding network (250) and the sub-models on the plurality of sets of training utterances (305), parameters of the speech conversion model (210) are frozen.

11. A computer-implemented method (400) as described in any one of claims 8 to 10, wherein one or more of the multiple sets of training utterances (305) each include an atypical speech pattern of a corresponding atypical speech type associated with the different corresponding training speakers, and the corresponding atypical speech type associated with the different corresponding training speakers is different from the corresponding atypical speech type associated with each other different corresponding training speaker.

12. A system (100), comprising: Data processing hardware (510); as well as Memory hardware (520) in communication with the data processing hardware (510), the memory hardware (520) storing instructions that, when executed on the data processing hardware (510), cause the data processing hardware (510) to perform operations, the operations comprising: receiving a reference audio signal (312) corresponding to a reference speech spoken by a target speaker having an atypical speech; generating a speaker embedding (350) for the target speaker by a speaker embedding network (250) configured to receive the reference audio signal (312) as input, the speaker embedding (350) conveying speaker characteristics of the target speaker; receiving a speech conversion request including input audio data corresponding to an utterance spoken by the target speaker (102), the target speaker being associated with the atypical speech; and A speech conversion model (210) is biased using the speaker embedding (350) generated by the speaker embedding network (250) for the target speaker to convert the input audio data (102) corresponding to the utterance spoken by the target speaker to an output canonical representation of the utterance spoken by the target speaker, the target speaker being associated with the atypical speech.

13. The system (100) of claim 12, wherein: The speech conversion model (210) comprises a speech-to-speech conversion model (210) configured to directly convert an input spectrogram or audio waveform into an output spectrogram or audio waveform; and Biasing the speech-to-speech conversion model (210) includes biasing the speech-to-speech conversion model (210) using the speaker embedding (350) to convert the input audio data (102) corresponding to the utterance spoken by the target speaker into output audio data (106) corresponding to a synthesized canonical fluent speech representation of the utterance spoken by the target speaker, the target speaker being associated with atypical speech, wherein the output canonical representation includes the synthesized canonical fluent speech representation.

14. The system (100) of claim 12, wherein: The speech conversion model (210) includes an automated speech recognition model configured to convert speech into text; and Biasing the speech conversion model (210) includes biasing the automated speech recognition model using the speaker embedding (350) to convert the input audio data (102) corresponding to the utterance spoken by the target speaker into a canonical text representation (120) of the utterance spoken by the target speaker, the target speaker being associated with an atypical speech, wherein the output canonical representation includes the canonical text representation (120).

15. The system (100) of any one of claims 12 to 14, wherein the operations further comprise: determining, using the speaker embedding network (250), a personalized embedding (251) for the target speaker based on the speaker embedding (350), the personalized embedding (251) corresponding to a respective style cluster of speaker embeddings (350) extracted from training utterances (305) spoken by the speaker that convey speaker characteristics similar to those conveyed by the speaker embedding (350), Wherein biasing the speech conversion model (210) includes biasing the speech conversion model (210) for the type of the atypical speech associated with the target speaker using the personalized embedding (251) determined for the target speaker.

16. The system (100) of claim 15, wherein biasing the speech conversion model (210) using the personalized embedding (251) determined for the target speaker comprises: The personalized embedding (251) is provided as a side input to the speech conversion model (210) for biasing the speech conversion model (210) for the type of the atypical speech associated with the target speaker.

17. The system (100) of claim 15, wherein biasing the speech conversion model (210) using the personalized embedding (251) determined for the target speaker comprises: The personalized embedding (251) is provided as a side input to a sub-model that biases the speech conversion model (210) for the type of the atypical speech associated with the target speaker.

18. The system (100) of claim 17, wherein the operation further comprises, prior to biasing the speech conversion model (210), using the personalized embedding (251) determined for the target speaker to select the sub-model from a plurality of different sub-models, each of the plurality of different sub-models being trained for a different type or severity of atypical speech.

19. The system (100) of any one of claims 12 to 18, wherein the training process (200) trains the speech conversion model (210) and the speaker embedding network (250) concurrently end-to-end by: A plurality of sets of spoken training utterances (305) are obtained, each set of spoken training utterances (305) being spoken by a different respective training speaker and comprising: audio data representing the training utterance (305), the audio data comprising an atypical speech pattern of an atypical speech type associated with the respective training speaker; as well as a canonical transcription (120) of the training utterance (305) spoken by the corresponding training speaker; and The speech conversion model (210) and the speaker embedding network (250) are trained concurrently on the plurality of sets of spoken training utterances (305).

20. The system (100) of claim 19, wherein: The speaker embedding network (250) includes a reference encoder (300) and a style attention module (254), the reference encoder (300) being configured to extract a corresponding speaker embedding (350) from each set of spoken training utterances (305) for the corresponding training speaker; as well as Training the speech conversion model (210) and the speaker embedding network (250) includes: training the style attention module (254) to learn how to group the speaker embeddings (350) into style clusters (260), each style cluster representing a corresponding cluster of speaker embeddings (350) extracted from the training utterances (305) spoken by training speakers having similar speaker characteristics, and each style cluster mapping to a corresponding personalized embedding (251) representing a corresponding type of atypical speech; and For each corresponding set of the plurality of sets of spoken training utterances (305), biasing the speech conversion model (210) for the corresponding set of spoken training utterances (305) using the corresponding personalized embedding (251) mapped to the style cluster, the style cluster comprising the corresponding speaker embedding (350) extracted from the corresponding set of spoken training utterances (305).

21. The system (100) of claim 19 or 20, wherein: Concurrently training the speech conversion model (210) and the speaker embedding model further comprises: training a sub-model on the plurality of sets of the training utterances (305) to learn how to bias the speech conversion model (210); and While training the speech embedding network (250) and the sub-models on the plurality of sets of training utterances (305), parameters of the speech conversion model (210) are frozen.

22. A system (100) as described in any one of claims 19 to 21, wherein one or more of the multiple sets of training utterances (305) each include an atypical speech pattern of a corresponding atypical speech type associated with the different corresponding training speakers, and the corresponding atypical speech type associated with the different corresponding training speakers is different from the corresponding atypical speech type associated with each other different corresponding training speaker.

Images