A scalable model specialization framework for personalizing speech models

Abstract

A method (600) for speech conversion includes obtaining a speech conversion model (200) configured to directly convert an input utterance into a corresponding output utterance of a synthetic voice. The method further includes receiving a speech conversion request (105) including input audio data (102) corresponding to an utterance (108) associated with an atypical speech and a speaker identifier (103) uniquely identifying a target speaker (104). The method includes activating a specific sub-model (215) to bias the speech conversion model to recognize a type of atypical speech associated with the target speaker identified by the speaker identifier using the speaker identifier. The method includes converting the input audio data into output audio data corresponding to a synthesized standard and fluent speech representation (224) of the utterance spoken by the target speaker using the speech conversion model biased by the activated specific sub-model.

Description

[Technical field]

[0001] The present disclosure relates to a scalable model specialization framework for personalizing speech models. [Background technology]

[0002] The speech conversion model can be used to change the speech of a source speaker into another form without changing the linguistic information of the speech. For example, the speech conversion model can generate a transcript of the user's speech. Alternatively, the speech conversion model can convert the user's speech into an audio waveform of the speech in another language. Machine learning methods can be used to train the speech conversion model to accurately and efficiently convert speech into another form. The speech conversion model is typically a large-scale model trained on a large dataset, and is therefore typically adapted to a large set of users. Summary of the Invention

[0003] One aspect of the disclosure provides a computer-implemented method for biasing a speech conversion model using a sub-model. The computer-implemented method, when executed by data processing hardware, causes the data processing hardware to perform operations including obtaining a speech conversion model configured to directly convert an input utterance of a human voice to a corresponding output utterance of a synthetic voice. The operations include receiving a speech conversion request including input audio data corresponding to an utterance spoken by a target speaker associated with an atypical speech and a speaker identifier that uniquely identifies the target speaker. The operations further include activating, using the speaker identifier, a particular sub-model to bias the speech conversion model to recognize a type of atypical speech associated with the target speaker identified by the speaker identifier. The operations also include converting, using the speech conversion model biased by the activated particular sub-model, the input audio data corresponding to the utterance spoken by the target speaker associated with the atypical speech to output audio data corresponding to a synthesized standard and fluent speech representation of the utterance spoken by the target speaker.

[0004] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, the speech conversion model is trained with generalized training data and is speaker and domain independent. In some implementations, the speech conversion model includes an encoder configured to encode input audio data corresponding to speech. In these implementations, the speech conversion model also includes a decoder configured to receive as an input the encoded audio data from the encoder and generate as an output output audio data corresponding to a synthesized standard fluent speech representation of the speech. In these implementations, the encoder may include a stack of self-attention blocks, each having a multi-head self-attention mechanism. In these implementations, the sub-model may include a stack of residual adapters disposed between each self-attention block in the stack of self-attention blocks of the encoder. Furthermore, in these implementations, each residual adapter may include a normalization layer followed by a feedforward layer with down-projection to the bottleneck dimension, a nonlinear activation, and another feedforward layer with up-projection. In these implementations, the speech-conversion model may further include a word fragment decoder configured to receive as input the encoded audio data from the encoder and generate as output a text representation corresponding to a transcription of the utterance. Alternatively, in these implementations, the speech-conversion model may further include a phoneme decoder configured to receive as input the encoded audio data from the encoder and generate as output a phoneme representation of the utterance.

[0005] The input audio data may include one of an input spectrogram or an input audio waveform, and the output audio data may include one of an output spectrogram or an output audio waveform. In some implementations, activating the particular sub-model to bias the speech conversion model includes selecting a particular sub-model associated with a type of atypical speech associated with the target speaker from among a plurality of sub-models each associated with a different type of atypical speech. Implementations further include loading the particular sub-model into the speech conversion model to bias the speech conversion model to recognize a type of atypical speech associated with the target speaker.

[0006] Another aspect of the disclosure provides a system for biasing a speech conversion model using a sub-model. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed by the data processing hardware, cause the data processing hardware to perform operations. The operations include obtaining a speech conversion model configured to directly convert an input utterance of a human voice to a corresponding output utterance of a synthetic voice. The operations include receiving a speech conversion request including input audio data corresponding to an utterance spoken by a target speaker associated with an atypical speech and a speaker identifier that uniquely identifies the target speaker. The operations further include activating, using the speaker identifier, a particular sub-model to bias the speech conversion model to recognize a type of atypical speech associated with the target speaker identified by the speaker identifier. The operations also include converting, using the speech conversion model biased by the activated particular sub-model, the input audio data corresponding to the utterance spoken by the target speaker associated with the atypical speech to output audio data corresponding to a synthesized standard and fluent speech representation of the utterance spoken by the target speaker.

[0007] This aspect may include one or more of the following optional features: In some implementations, the speech conversion model is trained with generalized training data and is speaker and domain independent. In some implementations, the speech conversion model includes an encoder configured to encode input audio data corresponding to speech. In these implementations, the speech conversion model also includes a decoder configured to receive as an input the encoded audio data from the encoder and generate as an output output audio data corresponding to a synthesized standard fluent speech representation of the speech. In these implementations, the encoder may include a stack of self-attention blocks, each having a multi-head self-attention mechanism. In these implementations, the sub-model may include a stack of residual adapters disposed between each self-attention block in the stack of self-attention blocks of the encoder. Furthermore, in these implementations, each residual adapter may include a normalization layer followed by a feedforward layer with downward projection to the bottleneck dimension, a nonlinear activation, and another feedforward layer with upward projection. In these implementations, the speech-conversion model may further include a word fragment decoder configured to receive as input the encoded audio data from the encoder and generate as output a text representation corresponding to a transcription of the utterance. Alternatively, in these implementations, the speech-conversion model may further include a phoneme decoder configured to receive as input the encoded audio data from the encoder and generate as output a phoneme representation of the utterance.

[0008] The input audio data may include one of an input spectrogram or an input audio waveform, and the output audio data may include one of an output spectrogram or an output audio waveform. In some implementations, activating the particular sub-model to bias the speech conversion model includes selecting a particular sub-model associated with a type of atypical speech associated with the target speaker from among a plurality of sub-models each associated with a different type of atypical speech. Implementations further include loading the particular sub-model into the speech conversion model to bias the speech conversion model to recognize a type of atypical speech associated with the target speaker.

[0009] One or more embodiments of the present disclosure are described in detail in the accompanying drawings and the following description. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0010] [Figure 1] 1 is a schematic diagram of an example system of a voice conversion system including a voice conversion model. [Figure 2A] FIG. 1 is a schematic diagram of a speech conversion model that uses sub-models to generate a speech representation. [Figure 2B] FIG. 2 is a schematic diagram of a speech conversion model that uses selected sub-models to generate a speech representation. [Figure 3A] FIG. 1 is a schematic diagram of a speech conversion model that uses sub-models as residual adapter layers to generate speech representations. [Figure 3B] FIG. 1 is a schematic diagram of a speech conversion model that uses sub-models as residual adapter layers placed between layers of an encoder to generate speech representations. [Figure 3C] FIG. 2 is a schematic diagram of a speech conversion model with various decoders. [Figure 4] FIG. 2 is a schematic diagram of an exemplary training scheme for a voice conversion model. [Diagram 5]FIG. 2 is a schematic diagram of an exemplary training scheme for sub-models of a voice conversion model. [Figure 6] 1 is a flowchart of an exemplary arrangement of method operations for performing voice conversion using a sub-model modified voice conversion model. [Figure 7] FIG. 1 is a schematic diagram of an example computing device that may be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Like reference symbols in the various drawings indicate like elements.

[0012] There is growing interest in developing more inclusive speech technologies, especially those that can help people with speech impairments. Automatic speech recognition (ASR) has made significant advances with the introduction of end-to-end (E2E) deep learning-based models to recognize speech from speakers with dysarthric or atypical speech patterns for conversion to accurate transcriptions. For example, atypical speech patterns may include, but are not limited to, impaired speech due to physical or neurological conditions (e.g., speakers with amyotrophic lateral sclerosis (ALS) disease), heavily accented speech, and hearing-impaired voices. Similar deep learning-based models can be applied by speech conversion systems to convert speech with atypical speech patterns into standard, fluent output speech.

[0013] Using machine learning techniques, speech-to-text models may be trained on large training datasets that contain audio samples of speech to generate robust models for speech recognition / conversion. Generally, these speech-to-text models are large because the more extensively the model is trained, the better it performs. However, there are drawbacks to using such large-scale models, such as a single model used for a wide variety of users with different characteristics. For example, a single speech-to-text model may be built for users with stereotypical speech patterns, even though some users may have atypical speech patterns. As a result, the speech-to-text model may not work as accurately for a particular group of users. Furthermore, it is difficult to retrain or update the model with different sizes, as it is computationally expensive. This may cause the speech-to-text model to become outdated and not perform well for new / emerging words / phrases (e.g., slang, new television programs).

[0014] Fine-tuning and fitting neural models has become a standard approach for model specialization or task customization. Substantial improvements have been shown for a wide range of downstream tasks by fine-tuning the entire model or a subset of the model's parameters. However, these techniques introduce scalability limitations for both training and inference when applied to a large number of tasks, domains, conditions, and / or users.

[0015] The embodiments herein are directed to integrating one or more sub-models with a base speech conversion model to personalize the base speech conversion model for converting speech in a specific domain (i.e., a specific user and / or type of speech associated with the user). The sub-models include a set of parameters that can be added to or replaced by a general base speech conversion model. The sub-models can be loaded / enabled / disabled as needed, allowing the use of the base speech conversion model without personalization, thereby eliminating issues that arise from changing larger models, such as concerns about catastrophic forgetting. In some examples, the base speech conversion model is trained, and then the parameters of the base speech conversion model are frozen during operation (i.e., estimation). In this way, the base speech conversion model is stable and can continue to function for general traffic or multiple users / domains. However, if speaker identification information is available, the base speech conversion model may activate the appropriate sub-model for a specific speaker and / or type of speech to perform speech conversion based on the type of the user's voice. One advantage of these embodiments includes the ability to train the sub-models individually without the need to train or retrain a large base speech conversion model. Furthermore, one or more sub-models can be dynamically loaded and replaced on a per-request basis in real-time (e.g., using a memory cache). The base speech conversion model remains unchanged whether or not a sub-model is used, eliminating concerns that the model will be compromised by usage and / or updates.

[0016] As used herein, and unless otherwise indicated, the terms "speech conversion system" and "speech conversion model" can refer to any combination of an ASR system / model in which input atypical speech is recognized and converted into corresponding text (e.g., transcription) and / or a set of phonemes representing the atypical speech, or a speech-to-speech conversion system / model in which input atypical speech is directly converted into standard, fluent, synthetic speech without performing speech recognition. In other words, the speech conversion system / model is configured to directly convert an input audio waveform or spectrogram corresponding to the atypical speech into an output audio waveform or spectrogram corresponding to standard, fluent speech without converting the input audio waveform into an intermediate representation (e.g., text or phonemes). As will become apparent, speech conversion models and techniques for training speech conversion models enable users with atypical speech to speak to and be understood by both other humans and speech interfaces (e.g., digital assistants) by enabling recognition and / or reproduction of the user's intended speech. Although examples herein show a speech conversion model that receives an input audio waveform or spectrogram corresponding to atypical speech and converts it to an output audio waveform or spectrogram corresponding to standard, fluent speech, the speech conversion model may be adapted to perform other types of speech conversion tasks as well without departing from the scope of the present disclosure. For example, the speech conversion model may convert an input audio waveform or spectrogram corresponding to an utterance in a first language to an output audio waveform or spectrogram corresponding to a translation of the utterance in a different, second language. The speech conversion model may similarly receive input spoken by a user and output a synthetic speech that includes the same linguistic content as the spoken input, but differs from the characteristics of the target speaker's voice.

[0017] FIG. 1 illustrates a voice conversion system 100 including an automatic voice conversion model 200 and a sub-model 215. The voice conversion model 200 is configured to process a voice conversion request 105 using the sub-model 215. The voice conversion request 105 includes input audio data 102 corresponding to an utterance 108 spoken by a target speaker 104 and captured by a user device 110. The voice conversion request 105 may further include a speaker identifier 103. Using the audio data 102 and the speaker identifier 103, the voice conversion model 200 and the sub-model 215 generate or predict an unbiased voice representation 222 or a biased voice representation 224. The biased voice representation 224 is more likely to include words or phrases associated with a particular user and / or a particular voice type, which biases the voice conversion model 200 using the sub-model 215 (e.g., based on the speaker identifier 103). In some implementations, when the target speaker 104 is identified as having an atypical voice, the voice representation 224 is a synthesized standard fluent voice representation 224 of the utterance 108 spoken by the target speaker 104. In some examples, the input audio data 102 includes an input spectrogram corresponding to the utterance 108. The speaker identifier 103 may provide an indication of a particular user and / or a particular domain (i.e., voice type) among multiple different domains, each of which implies or represents a respective bias context of the utterance 108.

[0018] Although not shown, an acoustic front end present on the user device 110 can convert a time-domain audio waveform of the speech 108 captured via a microphone of the user device 110 into an input spectrogram 102, or other type or format of audio data 102. Additionally, the front end device may be configured to determine or obtain data representative of speaker identifiers 103 influencing the speech 108 and / or other relevant information corresponding to the target speaker 104 and / or the client device 110.

[0019] A user device 110 associated with a target speaker 104 may capture utterances 108 spoken by the target speaker 104 and provide corresponding input audio data 102 to the speech conversion system 100 as part of a speech conversion request 105. Additionally, the user device 110 may determine a speaker identifier 103 to include in the speech conversion request 105. The user device 110 may include, but is not limited to, a smartphone, a tablet, a desktop / laptop computer, a smart speaker, a smart display, a smart home appliance, an assistant-enabled wearable device (e.g., a smart watch, smart headphones, smart glasses, etc.), or a vehicle infotainment system. Alternatively, the remote server 112 may process the audio data 102 and any other additional data or metadata from the user device 110 to determine the speaker identifier 103.

[0020] The voice conversion system 100 may be distributed across multiple devices such that the voice conversion model 200 resides on one of the user devices 110 or on a remote system 150 (also referred to herein as a cloud computing environment) that communicates with the user devices 110 via the network 140. The remote system 150 may be a single computer, multiple computers, or a distributed system (e.g., a cloud environment) having scalable / elastic resources 152 including computing resources 154 (e.g., data processing hardware) and / or storage resources 156 (e.g., memory hardware). A data store 158 (i.e., a remote storage device) may be overlaid on the storage resources 146, allowing for scalable use of the storage resources 146 by one or more user devices 110 or computing resources 154. The voice conversion model 200 and sub-models 215 may be executed on the remote system 150 or the user device 110. The sub-models 215 may be stored locally on the user device 110, or on a remote system (eg, in the data store 158), or some combination in between.

[0021] The voice conversion system 100 implements a dynamic model that includes two or more constituent parts or models, including a general base model (e.g., a voice conversion model 200) and “swappable” sub-models 215, to generate biased voice representations 224 based on received voice conversion requests 105. The voice conversion model 200 may be trained on a large voice dataset. Once trained, the voice conversion model 200 may be frozen so that parameters of the voice conversion model 200 remain constant during operation. The voice conversion model 200 may be updated, retrained, or replaced as needed or when additional training data becomes available. In some implementations, a single sub-model 215 is used to bias the voice conversion model 200 by activating certain parameters based on the speaker identifier 103 of the voice conversion request 105. In other implementations, the system 100 includes multiple sub-models 215, each sub-model 215 corresponding to a particular domain (i.e., a particular user and / or a particular voice type). In these implementations, a particular sub-model 215 is selected / activated based on the speaker identifier 103. For example, if the speaker identifier 103 indicates that the target speaker 104 has a speech type associated with atypical speech (e.g., stuttering), the speech conversion model 200 may activate the sub-model 215 associated with that speech type. If the speech conversion request 105 does not have a speaker identifier 103, or if a sub-model 215 corresponding to the speaker identifier 103 does not exist or is under development, in some examples, the speech conversion model 200 generates an unbiased speech representation 222. That is, the unbiased speech representation 222 is generated solely by the speech conversion model 200 and is not acted upon or influenced by any of the sub-models 215.

[0022] The speaker identifier 103 may be based on any signal or data that can be used to improve the accuracy of the biased speech representation 224. The speaker identifier 103 may be based on information related to the target speaker 104. For example, the target speaker 104 has a particular dialect, native tone, mannerisms, speech patterns, hesitations, etc. Thus, the system 100 may generate and train a sub-model 215 corresponding to the target speaker, which biases / individuates the speech conversion model 200 to make predictions that are specifically suited to the target speaker 104. In some implementations, the speaker identifier 103 includes a one-hot vector, and the system uses the speaker identifier 103 to select an appropriate sub-model 215 (i.e., the sub-model 215 corresponding to the target speaker 104) from multiple different sub-models 215.

[0023] In some implementations, the speaker identifier 103 is based on information associated with the user device 110. For example, the user device 110 is associated with a particular user (i.e., the target speaker 104), and therefore any utterance 108 associated with the user device 110 is sent as a voice conversion request 105 that includes a speaker identifier 103 indicative of the particular user. Additionally, the user device 110 may include a smart device with sensors such as a GPS, an accelerometer, a gyroscope, a microphone, a proximity sensor, a camera, etc. Alternatively, the voice conversion model 200 may determine the speaker identifier 103 associated with a voice type by identifying a particular voice type based on an analysis of the audio input 102.

[0024] The speaker identifier 103 may indicate a domain associated with the user device 110, as inferred from one of the sensors. For example, the speaker identifier 103 may indicate the geographic location of the user device 110, as inferred from GPS data (with the user 104 explicitly consenting to sharing the user's geographic location, which may be overridden at any time). Here, the speaker identifier 103 may correspond to a more geographic location (e.g., a city such as Chicago) or a more specific location (e.g., a gym). In either case, the sub-model 215 may bias the biased speech representation 224 towards a particular domain (i.e., speech type) based on the location. Specifically, a speaker identifier 103 identifying the location Chicago may activate a sub-model 215 trained with data from users from that city, state, and / or region, and the sub-model 215 may be biased based on the user's speech habits or other characteristics from that area and named entities specific to that region (e.g., restaurants, sports teams, street names, etc.). As a result, the sub-model 215 can bias the speech conversion model 200 to generate a biased speech representation 224 that is biased against predictions that fit the domain. For example, the biased speech representation 224 can increase the likelihood that the speech conversion model 200 predicts that the utterance 108 contains references to restaurants or streets in Chicago. Similarly, a speaker identifier 103 indicating that the source speaker 104 is at a gym may activate a sub-model 215 trained based on speech from a user exercising or in a similar position. Here, the speech may be affected by breathing disturbances or directed to a particular word or phrase (e.g., voice instructions for operating a music player on a smart device). Thus, the sub-model 215 can incorporate these contextual elements when processing the audio data 102 to generate the biased speech representation 224.

[0025] In some implementations, the speaker identifier 103 indicates that multiple domains are applicable to the utterance 108. In this scenario, a single sub-model 215 may bias the speech conversion model 200 to generate a biased speech representation 224 for each of the multiple domains. Alternatively, multiple sub-models 215 (each trained for one or more of the multiple domains) may bias the speech conversion model 200 to generate a biased speech representation 224 for each of the multiple domains. For example, if the target speaker 104 has a particular speech type (e.g., stutters) and is located in a particular location (e.g., Boston) associated with a quirk, one or more sub-models 215 may bias the output of the speech conversion model 200 for each of these domains.

[0026] The output 190 may receive the unbiased and biased speech representations 222, 224 generated by the speech conversion model 200. In some examples, the output is a standard and fluent speech representation 222, 224 of the utterance 108 when the speech conversion model 200 is configured to process atypical speech. In other examples, the output 190 includes a natural language understanding (NLU) that performs query interpretation on the speech conversion results. The NLU may further instruct downstream applications / services to perform actions based on the results. The output 190 may also include a user interface generator configured to display the speech conversion results as a transcription on a screen of the user device 110 and / or another device. Similarly, the output 190 may be fed to a vocoder (not shown) to generate a time-domain audio waveform of the synthesized standard and fluent speech representation of the utterance 108.

[0027] 1 is presented for illustrative purposes only and is not intended to be limiting. For example, while only a single instance of each component is shown, system 100 may include any number of components 110, 112, 140, 150, 200, and 215. Additionally, while some components are described as being within cloud computing environment 150, in some implementations these components may be hosted locally on user device 110. Additionally, in various implementations some or all of components 112, 200, and 215 are hosted locally on user device 110, hosted remotely (such as within cloud computing environment 150), or some combination thereof.

[0028] 2A, an exemplary speech conversion model 200 implements a sub-model 215 to generate biased speech representations 224 (e.g., synthesized standard fluent speech representations 224). Here, the speech conversion model 200 receives a speech conversion request 105 including audio data 102 and a speaker identifier 103. In this example, the sub-model 215 includes a single model trained with different inputs and contexts corresponding to different domains. The speaker identifier 103 may be a one-hot vector indicating one or more specific domains (e.g., a specific user and / or a specific voice type) corresponding to the context of the audio data 102. The one-hot vectors may be concatenated and projected to phrase set embeddings before being sent to the sub-model 215. In some implementations, the one-hot vectors are looked up in an embedding matrix before being projected to the phrase set embeddings. Alternatively, the sub-model 215 may project the one-hot vectors to the phrase set embeddings. The sub-model may then activate one or more parameters corresponding to one or more particular domains indicated by the speaker identifier 103 based on the phrase set embeddings to process the audio data 102.

[0029] 2B, another exemplary speech conversion model 200 implements a selected sub-model 215, 215b to generate a biased speech representation 224 (e.g., a synthesized standard fluent speech representation 224). Here, the speech conversion model 200 receives a speech conversion request 105 including audio data 102 and a speaker identifier 103. In this case, the speaker identifier 103 is received by a sub-model selector 218. The sub-model selector 218 selects a particular sub-model 215 from a plurality of sub-models 215, 215a-n, each sub-model 215 corresponding to one or more different domains. The sub-models 215a-n may be trained with data reflecting any domain that provides a context for processing the audio input 102. For example, the selected sub-model 215b is based on a domain such as location, activity, user profile, voice type, or any other domain that can affect how the audio data 102 is processed. The speaker identifier 103 may be a one-hot vector indicating a particular domain corresponding to the context of the voice conversion request 105. Upon receiving the speaker identifier 103, the sub-model selector 218 may perform a lookup to determine the appropriate sub-model(s) 215 to use in processing the voice conversion request 105. When processing the streaming audio input 102 from the user device 110, the various sub-models 215a-n are easily interchangeable such that the voice conversion model 200 can swap in and out various sub-models 215 from the multiple sub-models 215a-n (i.e., disable or enable biases provided by the sub-models 215) while processing the streaming input when a new speaker identifier 103 is received. If the voice conversion request 105 does not include a speaker identifier 103 or if the speaker identifier 103 is not applicable to any of the available sub-models 215, the voice conversion model 200 processes the audio input 102 without activating or enabling any sub-models 215 and generates an unbiased voice representation 222 (FIG. 1).

[0030] The sub-model 215 may be implemented to bias the output of the speech conversion model 200 in various ways. FIG. 3A shows a schematic diagram 300a of the speech conversion model 200 for generating the biased speech representation 224 using the sub-model 215 implemented as a residual adapter layer 361 (FIG. 3B). The speech conversion model 200 may be a recurrent neural network (RNN) including an encoder 310 configured to encode the input audio data 102 into an encoded output 312 (e.g., a hidden feature representation including a series of vectors) and a decoder 320 configured to decode the encoded output 312 into the biased speech representation 224. Typically, the encoded output 312 is sent directly to the decoder 320 to generate the unbiased speech representation 222. However, in this example, the sub-model 215 operates in parallel with the processing of the speech conversion request 105. As a result, the sub-model 215 may generate the sub-model output 325 based on the received audio input 102 of the speech conversion request 105. The voice conversion model 200 may combine the sub-model output 325 and the encoded output 312 to generate a biased encoded output 314 and send it to the decoder 320.

[0031] In another example, the sub-model 215 is implemented between layers of the encoder 310. FIG. 3B shows an exemplary schematic diagram 300b of such an implementation. The encoder 310 may include several components 360, and the sub-model 215 may be placed between layers of the components 360 such that the encoder 310 generates a biased encoded output 319. The encoder components 360 may include a stack of multi-head attention blocks (i.e., conformer blocks), which may include a conformer or a transformer. In some implementations, each multi-head attention block includes a multi-head attention mechanism. Instead of multi-head attention blocks, the encoder 310 may include a stack of long short-term memories (LSTMs). The decoder 320 may receive the biased encoded output 319 and generate the biased representation 224.

[0032] Here, the submodel 215 may be implemented as one or more residual adapter layers 361 placed between the components 360 of the encoder 310 (e.g., a stack of self-attention blocks, each with a multi-headed self-attention mechanism). Each residual adapter layer 361 starts with a layer normalization applied to the input, b A feedforward layer with downward projection to (the bottleneck dimension), nonlinear activation (RELU), and the original input dimension d i This may be followed by another feedforward layer with upward projection to . In some implementations, all weights of the residual adapters 361 are randomly initialized. In a specific example, each submodel 215 includes 17 residual adapter layers 361, each layer added between layers of the encoder. Furthermore, the bottleneck d b may be set at 64, while all weights of the adapters are randomly initialized.

[0033] The residual adapter layer 361 offers several advantages to the implementation of sub-models 215. For example, the residual adapter layer 361 is easily added to the encoder and various sub-models 215 can be easily swapped out as needed. Furthermore, a sub-model 215 can be easily muted / disabled by setting the residual coefficients to zero (i.e., removing the sub-model 215 and allowing the speech conversion model 200 to operate in an unbiased manner). The size of the sub-model 215, when implemented as a residual adapter layer 361, can be scaled to a bottleneck dimension (e.g., d) depending on the task / use case (i.e., the domain associated with the speaker identifier 103). b ) Furthermore, control of the bottleneck dimension is performed inside the submodel 215, allowing for fast inference of precompiled and optimized execution graphs while allowing for dynamic replacement of tensor shapes.

[0034] FIG. 3C shows a schematic diagram of an exemplary speech conversion model 200 used by the speech conversion system 100 of FIG. 1. The speech conversion model 200 includes an encoder 310 and one or more decoders 320, 320a-c. The encoder 310 is configured to encode input audio data 102 of a speech conversion request 105 into an encoded spectrogram 312, where the input audio data 102 includes a sequence of input spectrograms corresponding to an utterance 108 spoken by a target speaker 104. In some implementations, the encoder 310 includes a stack of conformer blocks 360 and / or one or more sub-models 215 disposed between the conformer blocks 360. In these implementations, the encoder 310 subsamples the input audio data 102 using a convolutional layer and then processes the input audio data 102 with the stack of conformer blocks 360. Each conformer block 360 may include a feedforward layer, a self-attention layer, a convolutional layer, and a second feedforward layer. In some examples, the stack of conformer blocks 360 includes 17 layers of conformer blocks 360, each with 512 states, 8 attention heads, and a convolution kernel size of 32 × 1. Instead of conformer blocks, the encoder 310 may use a stack of transformer blocks, lightweight convolutional blocks, or other types of multi-head attention blocks.

[0035] The spectrogram, phoneme, and word piece decoders 320, 320a-c may each include a recurrent neural network based architecture, each receiving the shared encoded spectrogram 312 output by the encoder 310. The spectrogram decoder 320a may include a cross-attention mechanism 331, 331a configured to receive the shared encoded spectrogram 312 from the encoder 310. The spectrogram decoder 320a may further process the shared encoded spectrogram 312 by using several long short-term memory (LSTM) layers 333, 333a and several convolutional layers 335. For example, the spectrogram decoder 320a may include five LSTM layers 333a and five transform layers 335. The spectrogram decoder 320a may generate the output spectrogram 224, 224a (i.e., the speech representation 224). In some implementations, the spectrogram decoder 320a may generate the output spectrogram 224a directly from the encoded spectrogram 312, without performing any intermediate text-to-speech conversion on a text representation corresponding to a transcription of the speech.

[0036] In the illustrated example, the word fragment decoder 320b includes a corresponding cross-attention mechanism 331, 331b configured to receive the shared encoded spectrogram 312 from the encoder 310, followed by two long short-term memory (LSTM) layers 333, 333b and a softmax layer 345, 345a that outputs a text representation 224, 224b corresponding to the transcription of the utterance 108. The text representation 224 may be a transcription of the utterance 108 spoken by the target speaker 104 as a synthesized standard fluent speech representation 224.

[0037] Similar to the word fragment decoder 320b, the phoneme decoder 320c also includes a cross-attention mechanism 331, 331c configured to receive the shared encoded spectrogram 312 from the encoder 310, followed by two long short-term memory (LSTM) layers 333, 333c and a softmax layer 345, 345b that outputs a phoneme representation 224, 224c of the utterance 108, where the phoneme representation 224c includes phonemes that represent a synthesized standard fluent phonetic representation 224 of the utterance 108.

[0038] The above examples of Figures 3A, 3B, and 3C are for illustrative purposes only and are not intended to be limiting. The speech conversion model 200 and the sub-model 215 may include any suitable structure / architecture to perform speech conversion and generate the biased speech representation 224 in response to the speaker identifier 103. Furthermore, the sub-model 215 and the speech conversion model 200 may function in any suitable combination to generate the biased speech representation 224. For example, the sub-model 215 may be located in the speech conversion model 200 at any suitable location in the architecture of the speech conversion model 200. For example, the sub-model 215 may be deployed in a layer of the speech conversion model 200 as a residual adapter layer, as a tensor, as an encoder / decoder layer, as a predictive network, as a combination network, etc. Alternatively, the sub-model 215 and the speech conversion model 200 may generate outputs independently of each other, and the results may be combined by the speech conversion model 200, or another suitable component of the system, to determine the biased speech representation 224. In particular, when the sub-model 215 is disabled from the original frozen state of the speech conversion model 200, the speech conversion model 200 remains unchanged. That is, when the sub-model 215 is disabled (e.g., due to the absence of a speaker identifier 103), the speech conversion model 200 generates an unbiased speech representation 222 that is not influenced by the sub-model 215.

[0039] FIG. 4 illustrates a training process 400 of the speech conversion model 200. The speech conversion model 200 may be specifically adapted to normalize / convert dysarthric and atypical speech into a synthesized standard and fluent speech representation 224 (e.g., typical speech). The speech conversion model 200 may be trained in an end-to-end training process 400 to directly map an input spectrogram to another spectrogram without utilizing or relying on an intermediate discrete representation. The generated output spectrogram (e.g., speech representation 222 / 224) may be passed to a vocoder to generate a time-domain waveform. Alternatively, the output spectrogram may generate a transcript in parallel, so that the speech conversion model 200 effectively becomes an ASR system as well. In some implementations, the process 400 utilizes a two-step training technique that includes pre-training and training. Pre-training the model is a technique used to initialize the model, which can then be further fine-tuned based on additional training data 410. With respect to the speech model 200, pre-training may include starting the speech conversion model 200 with pre-training data 405 including multiple spoken utterances by one or more speakers. The pre-training data 405 may further include the spoken utterances paired with corresponding ground truth synthetic speech representations of the spoken utterances 108. The speech samples used for pre-training may be speech synthesized from a reference transcript in a given voice and / or speech synthesized from non-synthesized speech samples spoken by real humans.

[0040] The process 400 may fine-tune the parameters of the pre-trained speech conversion model 200 after the pre-training is completed. The training process 400 may include, for example, training the encoder 310 and / or the decoder 320 (FIG. 3A) separately or jointly in any suitable combination. The process 400 includes providing a training input 410 to the speech conversion model 200. The speech conversion model 200 may be trained with generalized training data 410 such that the speech conversion model 200 is a generic model that can be further individualized / biased using sub-models. The training data 410 may be speaker and / or domain independent. In some implementations, the training input 410 includes a plurality of speech samples spoken by a variety of different speakers. Here, the training input 410 may include speech samples from users with stereotyped voices. Additionally, the training input 410 may be labeled using a label 420 indicative of a target output associated with the training input 410. Upon receiving the training input 410, the speech conversion model 200 may generate an output 415 (e.g., an unbiased speech representation 222). The speech conversion model 200 may process the training input 410 in a manner described with respect to any of Figures 2A-3C, or in any other suitable manner for speech conversion.

[0041] In some implementations, the output 415 is used by a loss function 430 to generate a loss 440. That is, the loss function 430 compares the output 415 to the label 420 to generate a loss 440, where the loss 440 indicates a mismatch between the label 420 (i.e., the target output) and the output 415. The loss function 430 may implement any suitable technique to determine a loss, such as a regression loss, a mean squared error, a mean squared logarithmic error, a mean absolute error, a binary classification, a binary cross-entropy, a hinge loss, a multi-class loss, etc. The loss 440 may then be fed directly to the speech conversion model 200, where the speech conversion model 200 processes the loss 440 and adjusts one or more parameters of the speech conversion model 200 to compensate for the loss 440. In some implementations, when the speech conversion model 200 is properly trained, the model freezes. In other words, the parameters remain unchanged for a period of time until retraining or replacement of the voice conversion model 200 is deemed necessary (eg, when sufficient new training data 410 has been acquired).

[0042] FIG. 5 illustrates a training process 500 for the sub-model 215. In some implementations, the process 500 utilizes a two-step training technique including pre-training and training. Pre-training is a technique used to initialize the model, which can then be further fine-tuned based on additional training data 510. With respect to the sub-model 215, pre-training may include starting the sub-model 215 with pre-training data 505 including a plurality of spoken utterances by a speaker. The pre-training data 505 may further include the spoken utterances paired with corresponding ground truth synthetic speech representations of the spoken utterances. The speech samples used for pre-training may be speech synthesized from a reference transcript in a given voice. With respect to the sub-model 215, pre-training may further include tuning one or more parameters of the sub-model 215 such that the sub-model 215 is biased toward words and / or phrases corresponding to a particular user and / or a particular domain (e.g., a voice type) associated with the sub-model 215.

[0043] In some examples, the process 500 includes fine-tuning parameters of the pre-trained sub-model 215. In these examples, the process 500 includes providing a training input 510 to the sub-model 215. In some implementations, the training input 510 includes a plurality of voice samples spoken by a particular speaker and / or a speaker corresponding to a particular voice type. The training input 510 may also include a speaker identifier 103 and a transcript with each voice sample. The transcript may be a manually generated text representing the corresponding voice sample. In some implementations, the transcript is generated by a machine. The transcript should accurately reflect the corresponding voice sample such that the transcript is the target output of the sub-model 215. In some implementations, the speaker identifier 103 of the training input 510 may indicate a particular domain (e.g., a particular user or a particular voice type). In other implementations, the speaker identifier 103 of the training input 510 may include a one-hot vector. The sub-model 215 corresponding to the speaker identifier 103 may be trained with speech samples and transcripts such that the sub-model 215 is biased towards a particular user and / or a particular type of speech associated with the speaker identifier 103. In an example of a single sub-model 215 adapted to bias towards multiple speech types, the one-hot vectors of the speaker identifier 103 may be concatenated and projected to a phrase set embedding, and the phrase set embedding may then be used to train the sub-model 215. Furthermore, the training inputs 510 may be labeled with a label 420 indicating a target output associated with the training inputs 510. Upon receiving the training inputs 510, the sub-model 215 may generate an output 515 (e.g., a biased speech representation 224). The sub-model 215 may process the training inputs 510 in a manner described with respect to any of Figures 2A-3C or in any other suitable manner for speech conversion.

[0044] In some implementations, the output 515 is used by a loss function 530 to generate a loss 540. That is, the loss function 530 compares the output 515 with the label 520 to generate a loss 540, where the loss 540 indicates a mismatch between the label 520 (i.e., the target output) and the output 515. The loss function 530 may implement any suitable technique to determine a loss, such as a regression loss, a mean squared error, a mean squared logarithmic error, a mean absolute error, a binary classification, a binary cross-entropy, a hinge loss, a multi-class loss, etc. The loss 540 may then be fed directly to the sub-model 215, where the sub-model 215 processes the loss 540 and adjusts one or more parameters of the sub-model 215 to take the loss 540 into account. In some implementations, the sub-model 215 is continually trained (or retrained) as additional voice conversion requests are received. For example, while the speech conversion model 200 is frozen, the sub-models 215 can continue to adjust parameters based on the received speech conversion request 105, which includes the audio data 102, the speaker identifier 103, and the biased speech representation 224. Here, in the case of multiple sub-models 215, each sub-model 215 can be trained when used by the speech conversion model 200 to be biased towards the associated domain.

[0045] While the examples herein are directed to sub-models 215 biasing the speech conversion model 200 for speech conversion (and speech recognition using a word fragment decoder and / or a phoneme decoder), it is understood that the sub-models 215 and speaker identifiers 103 may be used to bias any type of model used for any purpose. For example, the sub-models 215 may bias image recognition models, recommendation models, filtering (e.g., email) models, medical diagnosis models, or any other models that may use contextual information to bias the results to improve accuracy. As noted above, the sub-models 215 may be trained with appropriate speaker identifiers 103 to appropriately bias the underlying base model.

[0046] 6 is a flow chart of an exemplary configuration of operations for performing voice conversion using the voice conversion model 200 and the sub-models 215. The method 600 may be performed, for example, by various elements of the voice conversion system 100 of FIG. 1. In operation 602, the method 600 includes obtaining a voice conversion model 200 configured to directly convert an input utterance 108 of a human voice into a corresponding output utterance of a synthetic voice. In operation 604, the method 600 includes receiving a voice conversion request 105 including input audio data 102 corresponding to the utterance 108 spoken by a target speaker 104 associated with an atypical voice and a speaker identifier 103 that uniquely identifies the target speaker 104. In operation 606, the method 600 includes using the speaker identifier 103 to activate a particular sub-model 215 for biasing the voice conversion model 200 to recognize a type of atypical voice associated with the target speaker 104 identified by the speaker identifier 103. At operation 608, the method 600 includes using the voice conversion model 200 biased by the particular sub-model 215 that has been activated to convert the input audio data 102 corresponding to the utterance 108 spoken by the target speaker 104 associated with the atypical voice into output audio data corresponding to a synthesized standard, fluent speech representation 224 of the utterance 108 spoken by the target speaker 104.

[0047] 7 is a schematic diagram of an exemplary computing device 700 that can be used to implement the systems and methods described herein. Computing device 700 is intended to represent various types of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown, their connections and relationships, and their functions are meant to be exemplary only and are not intended to limit the embodiments described and / or claimed herein.

[0048] Computing device 700 includes a processor 710, a memory 720, a storage device 730, a high-speed interface / controller 740 that connects to memory 720 and a high-speed expansion port 750, and a low-speed interface / controller 760 that connects to a low-speed bus 770 and storage device 730. Each of the components 710, 720, 730, 740, 750, and 760 are interconnected using various buses and may be mounted on a common motherboard or otherwise implemented as desired. Processor 710 may process instructions for execution within computing device 700, including instructions stored in memory 720 or storage device 730, to display graphical information for a graphical user interface (GUI) on an external input / output device such as a display 780 connected to high-speed interface 740. In other implementations, multiple processors and / or multiple buses may be used as desired, along with multiple memories and memory types. Multiple computing devices 700 may also be connected (e.g., as a bank of servers, a group of blade servers, or a multiprocessor system) with each device providing a portion of the required operations.

[0049] The memory 720 stores information non-transiently within the computing device 700. The memory 720 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transient memory 720 may be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by the computing device 700. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0050] The storage device 730 can provide mass storage for the computing device 700. In some implementations, the storage device 730 is a computer-readable medium. In various different implementations, the storage device 730 can be an array of devices, including a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. In additional implementations, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as the memory 720, the storage device 730, or a memory on the processor 710.

[0051] The high-speed controller 740 manages bandwidth-intensive operations of the computing device 700, and the low-speed controller 760 manages low-bandwidth intensive operations. Such an allocation of roles is merely exemplary. In some implementations, the high-speed controller 740 is connected to a memory 720, a display 780 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 750 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 760 is connected to a storage device 730 and a low-speed expansion port 790. The low-speed expansion port 790 may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), and may be connected to one or more input / output devices such as a keyboard, pointing device, scanner, etc., or may be connected to a network device such as a switch or router, for example, via a network adapter.

[0052] Computing device 700 may be implemented in a number of different forms, as shown in the figure: for example, it may be implemented as a standard server 700a, or multiple times within a group of such servers 700a, as a laptop computer 700b, or as part of a rack server system 700c.

[0053] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, at least one input device, and at least one output device, which may be special purpose or general purpose, coupled to receive data and instructions from the storage system and transmit data and instructions to the storage system.

[0054] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," an "app," or a "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0055] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disks, optical disks, memory, programmable logic circuits (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0056] The processes and logic flows described herein can be executed by one or more programmable processors (also referred to as data processing hardware) executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be executed by special purpose logic circuitry, such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits). Processors suitable for executing computer programs include, by way of example, both general purpose and special purpose microprocessors, as well as any one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively connected to receive data from or transmit data to them, or both. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all types of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0057] To interact with a user, one or more aspects of the present disclosure can be implemented in a computer having a display device, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, for displaying information to the user, and optionally a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to interact with the user. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0058] Although several embodiments have been described, it is understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A computer-implemented method (600) that, when executed on data processing hardware (154), causes the data processing hardware (154) to perform operations, the operations comprising: Obtaining a speech conversion model (200) configured to directly convert an input utterance of a human voice into a corresponding output utterance of a synthetic voice; receiving a speech conversion request (105) including input audio data (102) corresponding to an utterance (108) spoken by a target speaker (104) associated with an atypical speech and a speaker identifier (103) uniquely identifying the target speaker (104); using the speaker identifier (103) to activate a particular sub-model (215) for biasing the speech conversion model (200) to recognize the type of atypical speech associated with the target speaker (104) identified by the speaker identifier (104); converting the input audio data (102) corresponding to the utterance (108) spoken by the target speaker (104) associated with atypical speech into output audio data corresponding to a synthesized standard, fluent speech representation (224) of the utterance (108) spoken by the target speaker (104) using the speech conversion model (200) biased by the activated particular sub-model (215); A computer-implemented method (600), comprising:

2. The voice conversion model (200) trained on generalized training data (410); The computer-implemented method (600) of claim 1, which is speaker and domain independent.

3. The voice conversion model (200) an encoder (310) configured to encode the input audio data (102) corresponding to the speech (108); A decoder (320), receiving as input the encoded audio data (312) from the encoder (310); generating, as an output, the output audio data corresponding to the synthesized standard fluent speech representation (224) of the utterance (108); a decoder (320) configured as follows: The computer-implemented method (600) of claim 1 or 2, comprising:

4. The computer-implemented method (600) of any of claims 1 to 3, wherein the encoder (310) comprises a stack of self-attention blocks, each having a multi-head self-attention mechanism.

5. A computer-implemented method (600) according to any of claims 1 to 4, wherein the sub-model (215) comprises a stack of residual adapters (361) placed between each of the self-attention blocks in the stack of self-attention blocks of the encoder (310).

6. 6. The computer-implemented method (600) of any of claims 1 to 5, wherein each residual adapter (361) includes a normalization layer followed by a feedforward layer with a downward projection into the bottleneck dimension, a non-linear activation, and another feedforward layer with an upward projection.

7. the speech conversion model (200) further comprises a word fragment decoder (320, 320b); The word fragment decoder (320, 320b) receiving as input the encoded audio data (312) from the encoder (310); generating, as output, a text representation (224, 224b) corresponding to the transcription of said utterance (108); A computer-implemented method (600) according to any one of claims 1 to 6, configured to:

8. the voice conversion model (200) further comprises a phoneme decoder (320, 320c); The phoneme decoder (320, 320c) receiving as input the encoded audio data (312) from the encoder (310); generating, as an output, a phonemic representation (224, 224c) of said utterance (108); A computer-implemented method (600) according to any one of claims 1 to 7, configured to:

9. the input audio data (102) comprises one of an input spectrogram or an input audio waveform; The computer-implemented method (600) of any of claims 1 to 8, wherein the output audio data comprises one of an output spectrogram or an output audio waveform.

10. Activating the particular sub-model (215) to bias the speech conversion model (200) comprises: selecting the particular sub-model (215) associated with the type of atypical speech associated with the target speaker (104) from among a plurality of sub-models (215) each associated with a different type of atypical speech; loading the particular sub-model (215) into the speech conversion model (200) for biasing the speech conversion model (200) to recognize the type of atypical speech associated with the target speaker (104); The computer-implemented method of claim 1 , comprising:

11. Data processing hardware (154); memory hardware (156) in communication with said data processing hardware (154); Including, The memory hardware (156) stores instructions that, when executed by the data processing hardware (154), cause the data processing hardware (154) to perform operations, such as: Obtaining a speech conversion model (200) configured to directly convert an input utterance of a human voice into a corresponding output utterance of a synthetic voice; receiving a speech conversion request (105) including input audio data (102) corresponding to an utterance (108) spoken by a target speaker (104) associated with an atypical speech and a speaker identifier (103) uniquely identifying the target speaker (104); using the speaker identifier (103) to activate a particular sub-model (215) for biasing the speech conversion model (200) to recognize the type of atypical speech associated with the target speaker (104) identified by the speaker identifier (104); converting the input audio data (102) corresponding to the utterance (108) spoken by the target speaker (104) associated with atypical speech into output audio data corresponding to a synthesized standard, fluent speech representation (224) of the utterance (108) spoken by the target speaker (104) using the speech conversion model (200) biased by the activated particular sub-model (215); A system (100).

12. The voice conversion model (200) trained on generalized training data (410); The system (100) of claim 11, which is speaker and domain independent.

13. The voice conversion model (200) an encoder (310) configured to encode the input audio data (102) corresponding to the speech (108); A decoder (320), receiving as input the encoded audio data (312) from the encoder (310); generating, as an output, the output audio data corresponding to the synthesized standard fluent speech representation (224) of the utterance (108); a decoder (320) configured as follows: The system (100) of claim 11 or 12, comprising:

14. The system (100) according to any of claims 11 to 13, wherein the encoder (310) comprises a stack of self-attention blocks, each having a multi-head self-attention mechanism.

15. A system (100) according to any one of claims 11 to 14, wherein the sub-model (215) comprises a stack of residual adapters (361) placed between each of the self-attention blocks in the stack of self-attention blocks of the encoder (310).

16. The system (100) of any of claims 11 to 15, wherein each residual adapter (361) includes a normalization layer followed by a feedforward layer with a downward projection into the bottleneck dimension, a non-linear activation, and another feedforward layer with an upward projection.

17. the speech conversion model (200) further comprises a word fragment decoder (320, 320b); The word fragment decoder (320, 320b) receiving as input the encoded audio data (312) from the encoder (310); generating, as output, a text representation (224, 224b) corresponding to the transcription of said utterance (108); A system (100) according to any one of claims 11 to 16, configured so as to

18. the voice conversion model (200) further comprises a phoneme decoder (320, 320c); The phoneme decoder (320, 320c) receiving as input the encoded audio data (312) from the encoder (310); generating, as an output, a phonemic representation (224, 224c) of said utterance (108); A system (100) according to any one of claims 11 to 17, configured so as to

19. the input audio data (102) comprises one of an input spectrogram or an input audio waveform; The system (100) of any of claims 11 to 18, wherein the output audio data comprises one of an output spectrogram or an output audio waveform.

20. Activating the particular sub-model (215) to bias the speech conversion model (200) comprises: selecting the particular sub-model (215) associated with the type of atypical speech associated with the target speaker (104) from among a plurality of sub-models (215) each associated with a different type of atypical speech; loading the particular sub-model (215) into the speech conversion model (200) for biasing the speech conversion model (200) to recognize the type of atypical speech associated with the target speaker (104); The system (100) according to any one of claims 11 to 19, comprising:

Images