Scalable model specialization framework for speech model personalization

Abstract

A method (600) for speech conversion includes obtaining a speech conversion model (200) configured to directly convert an input utterance into a corresponding output utterance of synthetic speech. The method further comprises receiving a voice conversion request (105) comprising input audio data (102) corresponding to an utterance (108) associated with the atypical voice and a speaker identifier (103) uniquely identifying the target speaker (104). The method includes activating a particular sub-model (215) using the speaker identifier for biasing the speech conversion model to identify a type of atypical speech associated with a target speaker identified by the speaker identifier. The method includes using a speech conversion model biased by the activated particular sub-model to convert the input audio data into output audio data corresponding to a synthetic normative fluent speech representation (224) of the utterance spoken by the target speaker.

Description

Technical Field

[0001] The present disclosure relates to a scalable model specialization framework for speech model personalization. Background Art

[0002] A speech conversion model can be used to modify the speech of a source speaker into another form without changing the linguistic information of the speech. For example, a speech conversion model can produce a transcription of a user's speech. Alternatively, a speech conversion model can convert the user's speech into an audio waveform of speech in another language. Machine learning methods can be used to accurately train a speech conversion model and efficiently convert the speech into another form. Speech conversion models are typically large models that are trained on large data sets and are therefore generally suitable for large user sets. Summary of the invention

[0003] One aspect of the present disclosure provides a computer-implemented method for biasing a speech conversion model using a sub-model. The computer-implemented method, when executed by data processing hardware, causes the data processing hardware to perform an operation, the operation comprising: obtaining a speech conversion model, the speech conversion model being configured to directly convert an input utterance of human speech into a corresponding output utterance of synthesized speech. The operation comprises: receiving a speech conversion request, the speech conversion request comprising input audio data and a speaker identifier, the input audio data corresponding to an utterance spoken by a target speaker associated with an atypical speech, the speaker identifier uniquely identifying the target speaker. The operation further comprises: using the speaker identifier to activate a specific sub-model for biasing the speech conversion model to identify the type of atypical speech associated with the target speaker identified by the speaker identifier. The operation also comprises: using the speech conversion model biased by the activated specific sub-model to convert the input audio data corresponding to the utterance spoken by the target speaker associated with the atypical speech into output audio data corresponding to a synthetic canonical fluent speech representation of the utterance spoken by the target speaker.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the speech conversion model is trained on generalized training data and is speaker independent and domain independent. In some implementations, the speech conversion model includes an encoder configured to encode input audio data corresponding to an utterance. In these implementations, the speech conversion model also includes a decoder configured to receive encoded audio data from the encoder as input and generate output audio data corresponding to a synthetic canonical fluent speech representation of the utterance as output. In these implementations, the encoder may include a stack of self-attention blocks each having a multi-head self-attention mechanism. In these implementations, the sub-model may include a stack of residual adapters disposed between each of the self-attention blocks in the stack of the encoder's self-attention blocks. Further, in these implementations, each residual adapter may include a normalization layer, followed by a feedforward layer with a downward projection to the bottleneck dimension, and a nonlinear activation, and another feedforward layer with an upward projection. In these implementations, the speech conversion model may further include a word-piece decoder configured to receive the encoded audio data from the encoder as input and generate a text representation corresponding to the transcription of the utterance as output. Alternatively, in these implementations, the speech conversion model may further include a phoneme decoder configured to receive the encoded audio data from the encoder as input and generate a phoneme representation of the utterance as output.

[0005] The input audio data may include one of an input spectrogram or an input audio waveform, and the output audio data may include one of an output spectrogram or an output audio waveform. In some implementations, activating a specific sub-model for biasing the speech conversion model includes selecting a specific sub-model associated with a type of atypical speech associated with a target speaker from among a plurality of sub-models each associated with a different type of atypical speech. The implementation further includes loading the specific sub-model into the speech conversion model for biasing the speech conversion model to identify the type of atypical speech associated with the target speaker.

[0006] Another aspect of the present disclosure provides a system for biasing a speech conversion model using a sub-model. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include: obtaining a speech conversion model that is configured to directly convert an input utterance of human speech into a corresponding output utterance of synthesized speech. The operations include: receiving a speech conversion request that includes input audio data and a speaker identifier, the input audio data corresponding to an utterance spoken by a target speaker associated with an atypical speech, the speaker identifier uniquely identifying the target speaker. The operations further include: using the speaker identifier to activate a specific sub-model for biasing the speech conversion model to identify the type of atypical speech associated with the target speaker identified by the speaker identifier. The operations also include: using the speech conversion model biased by the activated specific sub-model to convert the input audio data corresponding to the utterance spoken by the target speaker associated with the atypical speech into output audio data corresponding to a synthetic canonical fluent speech representation of the utterance spoken by the target speaker.

[0007] This aspect may include one or more of the following optional features. In some implementations, the speech conversion model is trained on generalized training data and is speaker independent and domain independent. In some implementations, the speech conversion model includes an encoder configured to encode input audio data corresponding to an utterance. In these implementations, the speech conversion model also includes a decoder configured to receive encoded audio data from the encoder as input and generate output audio data corresponding to a synthetic canonical fluent speech representation of the utterance as output. In these implementations, the encoder may include a stack of self-attention blocks each having a multi-head self-attention mechanism. In these implementations, the sub-model may include a stack of residual adapters disposed between each of the self-attention blocks in the stack of self-attention blocks of the encoder. Further, in these implementations, each residual adapter may include a normalization layer, followed by a feed-forward layer with a downward projection to the bottleneck dimension, and a nonlinear activation, and another feed-forward layer with an upward projection. In these implementations, the speech conversion model may further include a word-piece decoder configured to receive the encoded audio data from the encoder as input and generate a text representation corresponding to the transcription of the utterance as output. Alternatively, in these implementations, the speech conversion model may further include a phoneme decoder configured to receive the encoded audio data from the encoder as input and generate a phoneme representation of the utterance as output.

[0008] The input audio data may include one of an input spectrogram or an input audio waveform, and the output audio data may include one of an output spectrogram or an output audio waveform. In some implementations, activating a specific sub-model for biasing the speech conversion model includes selecting a specific sub-model associated with a type of atypical speech associated with a target speaker from among a plurality of sub-models each associated with a different type of atypical speech. The implementation further includes loading the specific sub-model into the speech conversion model for biasing the speech conversion model to identify the type of atypical speech associated with the target speaker.

[0009] The details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a schematic diagram of an example system for a speech conversion system including a speech conversion model.

[0011] Figure 2A is a schematic diagram of a speech conversion model for generating speech representations using sub-models.

[0012] Figure 2B is a schematic diagram of a speech conversion model for generating a speech representation using a selected sub-model.

[0013] Figure 3A is a schematic diagram of a speech conversion model for generating speech representations using a sub-model as a residual adapter layer.

[0014] Figure 3B is a schematic diagram of a speech conversion model for generating speech representations using a sub-model as a residual adapter layer set between layers of an encoder.

[0015] Figure 3C is a schematic diagram of the speech conversion model with various decoders.

[0016] Figure 4 is a diagram of an example training scenario for a speech conversion model.

[0017] Figure 5 is a diagram of an example training scenario for a sub-model of a speech conversion model.

[0018] Figure 6 is a flow diagram of an example arrangement of operations for a method of performing speech conversion using a speech conversion model modified with sub-models.

[0019] Figure 7is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0020] Like reference numerals in the various drawings indicate like elements. DETAILED DESCRIPTION

[0021] There is a growing interest in developing more inclusive speech technologies, especially those that can help people with speech impairments. With the introduction of end-to-end (E2E) deep learning-based models, automated speech recognition (ASR) has made great progress in recognizing speech from speakers with dysarthria or atypical speech patterns for conversion into accurate transcriptions. For example, atypical speech patterns can include, but are not limited to, impaired speech due to physical or neurological conditions (e.g., speakers with amyotrophic lateral sclerosis (ALS)), heavily accented speech, and deaf speech. Similar deep learning-based models can be applied by speech conversion systems to convert speech with atypical speech patterns into canonical and fluent output speech.

[0022] Using machine learning techniques, voice conversion models can be trained on large training data sets including audio samples of speech to produce a robust model for speech recognition / conversion. Typically, these voice conversion models are large because the more extensive the model is trained, the better the model performs. However, there are disadvantages in using such large models, such as a single model for various users with different characteristics. For example, even if some users may have atypical speech patterns, a single voice conversion model can still be built for users with typical speech patterns. Then, the voice conversion model may not be able to perform so accurately for certain user groups. Further, because of the computational cost, it is difficult to retrain or update the model due to size. This can cause the voice conversion model to be outdated and cannot perform well for new / emerging words / phrases (e.g., slang, new TV shows).

[0023] Neural model fine-tuning and adaptation have become the standard approach for model specialization or task customization. Fine-tuning the entire model or a subset of the model's parameters has shown significant gains for a wide range of downstream tasks. However, these techniques have scalability limitations in both training and inference when applied to a large number of tasks, domains, conditions, and / or users.

[0024] The implementation of this article involves integrating one or more sub-models with a basic voice conversion model to personalize the basic voice conversion model for converting voices in a specific domain (i.e., a specific user and / or a voice type associated with the user). The sub-model includes a set of parameters that can be added to or replaced from a general basic voice conversion model. The sub-model can be loaded / enabled / disabled when necessary, thereby allowing the basic voice conversion model to be used without personalization, thereby eliminating problems caused by modifying a larger model, such as catastrophic forgetting problems. In some examples, the basic voice conversion model is trained, and then the parameters of the basic voice conversion model are frozen during operation (i.e., reasoning). In this way, the basic voice conversion model remains stable and can continue to work for general business or multiple users / domains. However, when speaker recognition information is available, the basic voice conversion model can activate appropriate sub-models for specific speakers and / or specific voice types to perform voice conversion based on the type of the user's voice. One advantage of these implementations includes the ability to train sub-models separately without having to train or retrain a large basic voice conversion model. Further, one or more sub-models can be dynamically and real-time loaded and replaced upon request (e.g., using memory cache). The basic speech conversion model remains unchanged regardless of whether the sub-model is used, which eliminates the problem that the model will be damaged by use and / or update.

[0025] As used herein, and unless otherwise specified, the terms "speech conversion system" and "speech conversion model" may refer to any combination of an ASR system / model in which an input atypical speech is recognized and converted into corresponding text (e.g., transcription) and / or a set of phonemes representing the atypical speech, or a speech-to-speech conversion system / model in which an input atypical speech is directly converted into canonical fluent synthesized speech without performing speech recognition. In other words, the speech conversion system / model is configured to directly convert an input audio waveform or spectrogram corresponding to an atypical speech into an output audio waveform or spectrogram corresponding to canonical fluent speech without converting the input audio waveform into an intermediate representation (e.g., text or phonemes). It is apparent that the speech conversion model and techniques for training the speech conversion model will enable users with atypical speech to converse with and be understood by other humans and speech interfaces (e.g., digital assistants) by enabling recognition and reproduction of the user's intended speech. Although the examples herein depict a speech conversion model that receives an input audio waveform or spectrogram corresponding to atypical speech for conversion to an output audio waveform or spectrogram corresponding to canonical fluent speech, the speech conversion model can be similarly adapted to perform other types of speech conversion tasks without departing from the scope of this disclosure. For example, a speech conversion model can convert an input audio waveform or spectrogram corresponding to an utterance in a first language into an output audio waveform or spectrogram corresponding to a translation of an utterance in a different second language. The speech conversion model can similarly receive spoken input by a user and output a synthesized speech containing the same linguistic content of the spoken input but with different voice characteristics of the target speaker.

[0026] Figure 1A speech conversion system 100 including an automatic speech conversion model 200 and a sub-model 215 is illustrated. The speech conversion model 200 using the sub-model 215 is configured to process a speech conversion request 105. The speech conversion request 105 includes input audio data 102 corresponding to an utterance 108 spoken by a target speaker 104 and captured by a user device 110. The speech conversion request 105 may also include a speaker identifier 103. Using the audio data 102 and the speaker identifier 103, the speech conversion model 200 and the sub-model 215 generate or predict an unbiased speech representation 222 or a biased speech representation 224. The biased speech representation 224 is more likely to include words or phrases associated with a specific user and / or a specific speech type, toward which the speech conversion model 200 is biased using the sub-model 215 (e.g., based on the speaker identifier 103). In some implementations, when the target speaker 104 is identified as having an atypical voice, the speech representation 224 is a synthesized canonical fluent speech representation 224 of the utterance 108 spoken by the target speaker 104. In some examples, the input audio data 102 includes an input spectrogram corresponding to the utterance 108. The speaker identifier 103 can provide an indication of a specific user and / or a specific domain (i.e., speech type) among a plurality of different domains, each domain representing or indicating a corresponding bias context of the utterance 108.

[0027] Although not shown, the acoustic front end resident on the user device 110 can convert the time-domain audio waveform of the speech 108 captured via the microphone of the user device 110 into an input spectrogram 102 or other type or form of audio data 102. Further, the front-end device can be configured to determine or obtain data representing a speaker identifier 103 affecting the speech 108 and / or other relevant information corresponding to the target speaker 104 and / or the client device 110.

[0028] The user device 110 associated with the target speaker 104 can capture the utterance 108 spoken by the target speaker 104 and provide the corresponding input audio data 102 to the speech conversion system 100 as part of the speech conversion request 105. Additionally, the user device 110 can determine the speaker identifier 103 to include in the speech conversion request 105. The user device 110 can include, but is not limited to, a smart phone, a tablet, a desktop / laptop computer, a smart speaker, a smart display, a smart home appliance, an assistant-enabled wearable device (e.g., a smart watch, a smart headset, smart glasses, etc.), or a vehicle infotainment system. Alternatively, the remote server 112 can process the audio data 102 and any other additional data or metadata from the user device 110 to determine the speaker identifier 103.

[0029] The speech conversion system 100 may be distributed across multiple devices such that the speech conversion model 200 resides on one of the user device 110 or a remote system 150 (also referred to herein as a cloud computing environment) that communicates with the user device 110 via a network 140. The remote system 150 may be a single computer, multiple computers, or a distributed system (e.g., a cloud environment) having scalable / elastic resources 152 including computing resources 154 (e.g., data processing hardware) and / or storage resources 156 (e.g., memory hardware). A data repository 158 (i.e., a remote storage device) may be overlaid on the storage resources 146 to allow scalable use of the storage resources 146 by one or more user devices 110 or computing resources 154. The speech conversion model 200 and the sub-model 215 may be executed on the remote system 150 or the user device 110. The sub-model 215 may be stored locally at the user device 110 or on a remote system (e.g., at the data repository 158), or some combination thereof.

[0030] The speech conversion system 100 implements a dynamic model including two or more components or models, including a common base model (e.g., speech conversion model 200) and a "swappable" sub-model 215 to generate biased speech representations 224 based on received speech conversion requests 105. The speech conversion model 200 can be trained on a large speech data set. Once trained, the speech conversion model 200 can be frozen so that the parameters of the speech conversion model 200 remain constant during operation. The speech conversion model 200 can be updated, retrained, or replaced as needed or when additional training data becomes available. In some implementations, a single sub-model 215 is used to bias the speech conversion model 200 by activating certain parameters based on the speaker identifier 103 of the speech conversion request 105. In other implementations, the system 100 includes multiple sub-models 215, where each sub-model 215 corresponds to a specific domain (i.e., a specific user and / or a specific speech type). In these implementations, a particular sub-model 215 is selected / activated based on the speaker identifier 103. For example, when the speaker identifier 103 indicates that the target speaker 104 has a voice type associated with atypical speech (e.g., stuttering), the speech conversion model 200 can activate the sub-model 215 associated with the voice type. When the speech conversion request 105 does not have a speaker identifier 103 or when the sub-model 215 corresponding to the speaker identifier 103 does not exist or has not been developed, the speech conversion model 200 generates an unbiased speech representation 222 in some examples. That is, the unbiased speech representation 222 is generated only by the speech conversion model 200 and is not affected or acted upon by any sub-model 215.

[0031] The speaker identifier 103 may be based on any signal or data that can be used to improve the accuracy of the biased speech representation 224. The speaker identifier 103 may be based on information related to the target speaker 104. For example, the target speaker 104 has a particular dialect, accent, mannerisms, patterns of speech, speech disfluencies, etc. Therefore, the system 100 may generate and train a sub-model 215 corresponding to the target speaker 104, wherein the sub-model 215 biases / personalizes the speech conversion model 200 to make predictions specifically suited for the target speaker 104. In some implementations, the speaker identifier 103 includes a one-hot vector, and the system uses the speaker identifier 103 to select an appropriate sub-model 215 (i.e., a sub-model 215 corresponding to the target speaker 104) from a plurality of different sub-models 215.

[0032] In some implementations, the speaker identifier 103 is based on information associated with the user device 110. For example, the user device 110 is associated with a particular user (i.e., the target speaker 104), and thus any utterance 108 associated with the user device 110 is transmitted as a speech conversion request 105, which includes a speaker identifier 103 indicating the particular user. Further, the user device 110 may include a smart device equipped with sensors such as GPS, accelerometers, gyroscopes, microphones, proximity sensors, cameras, etc. Alternatively, the speech conversion model 200 may identify a particular speech type based on an analysis of the audio input 102, and thus determine a speaker identifier 103 associated with the speech type.

[0033] The speaker identifier 103 may indicate a domain associated with the user device 110, as inferred from one of the sensors. For example, the speaker identifier 103 may indicate the geographic location of the user device 110, as inferred from GPS data (which may be revoked at any time if the user 104 explicitly consents to share his / her geographic location). Here, the speaker identifier 103 may correspond to a more geographic location (such as a city of Chicago) or to a more specific location (e.g., a gym). In either instance, the sub-model 215 may bias the biased speech representation 224 toward a particular domain (i.e., speech type) based on the location. Specifically, a speaker identifier 103 identifying the location Chicago may activate a sub-model 215 trained on data from users in that city, state, and / or region, which may be biased based on the accent or other speech features of users from that region and named entities unique to that region (e.g., restaurants, sports teams, street names, etc.). Thus, the sub-model 215 may bias the speech conversion model 200 to generate a biased speech representation 224 that is biased toward a prediction bias appropriate for that domain. For example, the biased speech representation 224 can increase the likelihood that the speech conversion model 200 predicts that the utterance 108 includes a reference to a restaurant or street in Chicago. Similarly, a speaker identifier 103 indicating that the source speaker 104 is at a gym can activate a sub-model 215 trained based on speech from users who are exercising or in a similar location. Here, the speech can be affected by labored breathing, or involve specific words or phrases (e.g., voice commands for operating a music player on a smart device). Thus, the sub-model 215 can take these contextual elements into account when processing the audio data 102 to produce the biased speech representation 224.

[0034] In some implementations, the speaker identifier 103 indicates that multiple domains are applicable to the utterance 108. In this scenario, a single sub-model 215 can bias the speech conversion model 200 to generate a biased speech representation 224 toward each of the multiple domains. Alternatively, multiple sub-models 215 (each trained for one or more of the multiple domains) can bias the speech conversion model 200 to generate a biased speech representation 224 toward each of the multiple domains. For example, when the target speaker 104 has a particular voice type (e.g., stuttering) and is located in a particular location associated with an accent (e.g., Boston), one or more sub-models 215 can bias the output of the speech conversion model 200 toward each of these domains.

[0035] The output 190 can receive the unbiased speech representation 222 and the biased speech representation 224 generated by the speech conversion model 200. In some examples, when the speech conversion model 200 is configured to process atypical speech, the output is the canonical fluent speech representation 222, 224 of the speech 108. In other examples, the output 190 includes a natural language understanding (NLU) that performs query interpretation on the speech conversion result. The NLU can further guide downstream applications / services to perform actions based on the results. The output 190 can also include a user interface generator that is configured to display the speech conversion result as a transcription on the screen of the user device 110 and / or another device. Similarly, the output 190 can be fed to a vocoder (not shown) to generate a time domain audio waveform of a synthetic canonical fluent speech representation of the speech 108.

[0036] Figure 1 The system 100 is presented for illustration purposes only and is not intended to be limiting. For example, although only a single example of each component is illustrated, the system 100 may include any number of components 110, 112, 140, 150, 200, and 215. Further, although some components are described as being located in the cloud computing environment 150, in some implementations, those components may be hosted locally on the user device 110. Further, in various implementations, some or all of the components 112, 200, and 215 are hosted locally on the user device 110, remotely hosted (such as in the cloud computing environment 150), or some combination thereof.

[0037] Reference now Figure 2A , the exemplary speech conversion model 200 implements a sub-model 215 to generate a biased speech representation 224 (e.g., a synthetic canonical fluent speech representation 224). Here, the speech conversion model 200 receives a speech conversion request 105 including audio data 102 and a speaker identifier 103. In this example, the sub-model 215 includes a single model trained on various inputs and contexts corresponding to various domains. The speaker identifier 103 can be a one-hot vector that indicates one or more specific domains (e.g., a specific user and / or a specific voice type) corresponding to the context of the audio data 102. The one-hot vector can be concatenated and projected into a phrase set embedding before being transmitted to the sub-model 215. In some implementations, the one-hot vector is looked up in the embedding matrix before being projected into the phrase set embedding. Alternatively, the sub-model 215 can project the one-hot vector into the phrase set embedding. Then, the sub-model can activate one or more parameters corresponding to one or more specific domains indicated by the speaker identifier 103 based on the phrase set embedding to process the audio data 102.

[0038] Reference now Figure 2B, another exemplary speech conversion model 200 implements the selected sub-model 215, 215b to generate a biased speech representation 224 (e.g., a synthetic canonical fluent speech representation 224). Here, the speech conversion model 200 receives a speech conversion request 105 including audio data 102 and a speaker identifier 103. In this example, the speaker identifier 103 is received by a sub-model selector 218. The sub-model selector 218 selects a specific sub-model 215 from a plurality of sub-models 215, 215a to 215n, wherein each sub-model 215 corresponds to one or more different domains. The sub-models 215a to 215n can be trained on data reflecting any domain of the context providing the processing audio input 102. For example, the selected sub-model 215b is based on a domain such as location, activity, user profile, voice type, or any other domain that can affect how to process the audio data 102. The speaker identifier 103 can be a unique hot vector indicating a specific domain corresponding to the context of the speech conversion request 105. Upon receiving the speaker identifier 103, the sub-model selector 218 may perform a lookup to determine the appropriate sub-model 215 to use when processing the speech conversion request 105. The various sub-models 215a to 215n are easily swappable such that when processing the streaming audio input 102 from the user device 110, the speech conversion model 200 may swap in or out various sub-models 215 from the plurality of sub-models 215a to 215n (i.e., disable or enable the bias provided by the sub-model 215) while processing the streaming input when a new speaker identifier 103 is received. When the speech conversion request 105 does not include a speaker identifier 103 or when the speaker identifier 103 is not applicable to any of the available sub-models 215, then the speech conversion model 200 processes the audio input 102 without activating or enabling any sub-model 215 to produce an unbiased speech representation 222 ( Figure 1 ).

[0039] Sub-model 215 may be implemented to bias the output of speech conversion model 200 in any number of ways. Figure 3A Illustrated is a method for using a residual adapter layer 361 ( Figure 3B) to generate a biased speech representation 224. The speech conversion model 200 can be a recurrent neural network (RNN) that includes an encoder 310 and a decoder 320, the encoder being configured to encode the input audio data 102 into an encoded output 312 (e.g., a hidden feature representation including a series of vectors), and the decoder being configured to decode the encoded output 312 into a biased speech representation 224. Typically, the encoded output 312 is sent directly to the decoder 320 to generate an unbiased speech representation 222. However, in this example, the submodel 215 operates in parallel to process the speech conversion request 105. Then, the submodel 215 can generate a submodel output 325 based on the received audio input 102 of the speech conversion request 105. The speech conversion model 200 can merge the submodel output 325 and the encoded output 312 to generate a biased encoded output 314 to send to the decoder 320.

[0040] In another example, the sub-model 215 is implemented between layers of the encoder 310 . Figure 3B An example schematic diagram 300b illustrates such an implementation. The encoder 310 may include multiple components 360, and the sub-model 215 may be set between the layers of the components 360 so that the encoder 310 produces a biased encoded output 319. The components 360 of the encoder may include a stack of multi-headed attention blocks (i.e., conformer blocks), which may include a conformer or a transformer. In some implementations, each multi-headed attention block may include a multi-headed attention mechanism. The encoder 310 may include a stack of long short-term memory (LSTM) instead of a multi-headed attention block. The decoder 320 may receive the biased encoded output 319 and generate a biased representation 224.

[0041] Here, the sub-model 215 can be implemented as one or more residual adapter layers 361 disposed between components 360 (e.g., a stack of self-attention blocks each having a multi-head self-attention mechanism) of the encoder 310. Each residual adapter layer 361 can start with a layer normalization applied to the input, followed by a layer normalization with a dimension d b (bottleneck dimension), and a nonlinear activation (RELU), and a feed-forward layer with down-projection to the original input dimension d i Another feed-forward layer of upward projection. In some implementations, all weights of the residual adapter 361 are randomly initialized. In a specific example, each sub-model 215 includes 17 residual adapter layers 361, each of which is added between the layers of the encoder. Further, while all weights of the adapter are randomly initialized, the bottleneck d bCan be set at 64.

[0042] The residual adapter layer 361 provides several benefits to the implementation of the sub-model 215. For example, the residual adapter layer 361 can be easily added to an encoder, allowing various sub-models 215 to be easily interchanged as needed. Further, the sub-model 215 can be easily weakened / disabled by setting the residual factor to zero (i.e., removing the sub-model 215 and allowing the speech conversion model 200 to operate in an unbiased manner). The size of the sub-model 215 when implemented as a residual adapter layer 361 can be determined by the bottleneck dimension (e.g., d b ) is controlled according to the task / use case (i.e., the domain associated with the speaker identifier 103). Further, the bottleneck dimension is controlled inside the sub-model 215, thereby allowing pre-compiled and optimized execution graphs for fast reasoning while being able to dynamically replace tensor shapes.

[0043] Figure 3C Shown by Figure 1 Schematic diagram of an example speech conversion model 200 used by the speech conversion system 100 of FIG. The speech conversion model 200 includes an encoder 310 and one or more decoders 320, 320a to 320c. The encoder 310 is configured to encode the input audio data 102 of the speech conversion request 105 into an encoded spectrogram 312. Here, the input audio data 102 includes an input spectrogram sequence corresponding to an utterance 108 spoken by a target speaker 104. In some implementations, the encoder 310 includes a stack of one or more sub-models 215 disposed between the conformer blocks 360 and / or disposed between the conformer blocks 360. In these implementations, the encoder 310 uses a convolutional layer to subsample the input audio data 102, and then processes the input audio data 102 using a stack of conformer blocks 360. Each conformer block 360 may include a feedforward layer, a self-attention layer, a convolutional layer, and a second feedforward layer. In some examples, the stack of conformer blocks 360 includes 17 layers of conformer blocks 360, each with 512 states, 8 attention heads, and a 32x1 convolution kernel size. The encoder 310 may alternatively use a stack of conformer blocks, a stack of lightweight convolution blocks, or other types of multi-head attention blocks in place of a conformer block.

[0044] The spectrogram, phoneme, and word piece decoders 320, 320a to 320c may each include a recurrent neural network-based architecture that each receives a shared encoded spectrogram 312 output by the encoder 310. The spectrogram decoder 320a may include a cross-attention mechanism 331, 331a configured to receive the shared encoded spectrogram 312 from the encoder 310. The spectrogram decoder 320a may further process the shared encoded spectrogram 312 using a plurality of long short-term memory (LSTM) layers 333, 333a and a plurality of convolutional layers 335. For example, the spectrogram decoder 320a may include five (5) LSTM layers 333a and five (5) transformation layers 335. The spectrogram decoder 320a may generate an output spectrogram 224, 224a (i.e., a speech representation 224). In some implementations, the spectrogram decoder 320a can generate the output spectrogram 224a directly from the encoded spectrogram 312 without performing any intermediate text-to-speech conversion on a text representation corresponding to the transcription of the utterance.

[0045] In the example shown, the word patch decoder 320b includes a corresponding cross-attention mechanism 331, 331b configured to receive the shared encoded spectrogram 312 from the encoder 310, followed by two long short-term memory (LSTM) layers 333, 333b, and a Softmax layer 345, 345a that outputs a text representation 224, 224b corresponding to the transcription of the utterance 108. The text representation 224 can be a transcription of the utterance 108 spoken by the target speaker 104 as a synthesized canonical fluent speech representation 224.

[0046] As with the word piece decoder 320b, the phoneme decoder 320c may also include a cross-attention mechanism 331, 331c configured to receive the shared encoded spectrogram 312 from the encoder 310, followed by two long short-term memory (LSTM) layers 333, 333c, and a Softmax layer 345, 345b that outputs the phoneme representation 224, 224c of the utterance 108. Here, the phoneme representation 224c includes phonemes that indicate the synthesized canonical fluent speech representation 224 of the utterance 108.

[0047] Figure 3A , Figure 3B and Figure 3CThe above examples are for illustrative purposes only and are not intended to be limiting. The speech conversion model 200 and the sub-model 215 may include any suitable structure / architecture for performing speech conversion and generating a biased speech representation 224 in response to the speaker identifier 103. Further, the sub-model 215 and the speech conversion model 200 may work in any suitable combination to produce the biased speech representation 224. For example, the sub-model 215 is disposed within the speech conversion model 200 at any suitable location in the architecture of the speech conversion model 200. For example, the sub-model 215 is deployed within a layer of the speech conversion model 200 as a residual adapter layer, as a tensor, as an encoder / decoder layer, as a prediction network, as a joint network, etc. Alternatively, the sub-model 215 and the speech conversion model 200 may generate outputs independently of each other, and the results may be combined by the speech conversion model 200 or another suitable component of the system to determine the biased speech representation 224. It is noteworthy that when the sub-model 215 is disabled from the original frozen state of the speech conversion model 200, the speech conversion model 200 remains unchanged. That is, when the sub-model 215 is disabled (e.g., because the speaker identifier 103 is not present), the speech conversion model 200 produces an unbiased speech representation 222 that is not affected by the sub-model 215.

[0048] Figure 4 A training process 400 for a speech conversion model 200 is illustrated. The speech conversion model 200 may be specifically adapted to normalize / convert dysarthria and atypical speech into a synthetic canonical fluent speech representation 224 (e.g., typical speech). The speech conversion model 200 may be trained in an end-to-end training process 400 to map an input spectrogram directly to another spectrogram without utilizing or relying on any intermediate discrete representation. The generated output spectrogram (e.g., speech representation 222 / 224) may be passed to a vocoder to generate a time domain waveform. Alternatively, the output spectrogram may generate a transcription in parallel, thereby effectively making the speech conversion model 200 also an ASR system. In some implementations, the process 400 employs a two-step training technique including pre-training and training. Pre-training the model is a technique for initializing the model, which may then be further fine-tuned based on additional training data 410. For the speech model 200, pre-training may include initiating the speech conversion model 200 using pre-training data 405 including spoken utterances performed by one or more speakers. The pre-training data 405 may further include spoken utterances paired with corresponding ground-truth synthesized speech representations of the spoken utterances 108. The speech samples used for pre-training may be speech synthesized from reference transcriptions in a predetermined voice and / or non-synthesized speech samples spoken by a real human.

[0049] After the pre-training is completed, the process 400 can fine-tune the parameters of the pre-trained speech conversion model 200. The training process 400 includes: for example, the encoder 310 and / or the decoder 320 ( Figure 3A ) for separate or joint training. Process 400 includes: feeding training input 410 to speech conversion model 200. Speech conversion model 200 can be trained on training data 410, which is generalized so that speech conversion model 200 is a general model that can be further personalized / biased using sub-models. Training data 410 can be speaker and / or domain independent. In some implementations, training input 410 includes multiple speech samples spoken by multiple different speakers. Here, training input 410 may include speech samples from users with typical voices. Further, training input 410 can be labeled with a label 420 indicating a target output associated with training input 410. Upon receiving training input 410, speech conversion model 200 can generate output 415 (e.g., unbiased speech representation 222). Speech conversion model 200 can be trained with respect to Figures 2A to 3C The training input 410 may be processed in the manner described in any of the figures in FIG. 1 or in any other suitable manner for speech conversion.

[0050] In some implementations, the output 415 is used by the loss function 430 to generate the loss 440. That is, the loss function 430 compares the output 415 with the label 420 to generate the loss 440, where the loss 440 indicates the difference between the label 420 (i.e., the target output) and the output 415. The loss function 430 can implement any suitable technique to determine the loss, such as regression loss, mean squared error, mean squared logarithmic error, mean absolute error, binary classification, binary cross entropy, hinge loss, multi-class loss, etc. The loss 440 can then be fed directly to the speech conversion model 200. Here, the speech conversion model 200 processes the loss 440 and adjusts one or more parameters of the speech conversion model 200 to account for the loss 440. In some implementations, when the speech conversion model 200 is properly trained, the model is frozen. In other words, the parameters remain unchanged for a certain period of time until it is deemed that the speech conversion model 200 needs to be retrained (e.g., when sufficient new training data 410 is obtained) or replaced.

[0051] Figure 5A training process 500 for a submodel 215 is illustrated. In some implementations, the process 500 employs a two-step training technique including pre-training and training. Pre-training is a technique for initializing a model that can then be further fine-tuned based on additional training data 510. For the submodel 215, pre-training can include initiating the submodel 215 with pre-training data 505 including spoken utterances by a speaker. The pre-training data 505 can further include spoken utterances paired with corresponding true value synthetic speech representations of the spoken utterances. The speech samples used for pre-training can be speech synthesized from reference transcriptions in a predetermined voice. For the submodel 215, pre-training can further include adjusting one or more parameters of the submodel 215 so as to bias the submodel 215 toward words and / or phrases corresponding to a specific user and / or a specific domain (e.g., speech type) associated with the submodel 215.

[0052] In some examples, process 500 includes fine-tuning the parameters of the pre-trained sub-model 215. In these examples, process 500 includes feeding training input 510 to sub-model 215. In some implementations, training input 510 includes multiple speech samples spoken by a specific speaker and / or a speaker corresponding to a specific voice type. Training input 510 may also include speaker identifier 103 and transcription with each speech sample. The transcription may be manually generated text representing the corresponding voice sample. In some implementations, the transcription is machine generated. The transcription should accurately reflect the corresponding voice sample so that the transcription is the target output of sub-model 215. In some implementations, the speaker identifier 103 of training input 510 may indicate a specific domain (e.g., a specific user or a specific voice type). In other implementations, the speaker identifier 103 of training input 510 may include a one-hot vector. The sub-model 215 corresponding to the speaker identifier 103 can be trained on speech samples and transcriptions so that the sub-model 215 is biased toward a particular speech type and / or a particular user associated with the speaker identifier 103. In the example of a single sub-model 215 adapted to be biased for multiple speech types, the one-hot vectors of the speaker identifier 103 can be concatenated and projected into a phrase set embedding, which can then be used to train the sub-model 215. Further, the training input 510 can be labeled with a label 520 indicating a target output associated with the training input 510. Upon receiving the training input 510, the sub-model 215 can generate an output 515 (e.g., a biased speech representation 224). The sub-model 215 can be biased in terms of the target output 520 associated with the training input 510. Figures 2A to 3C Training input 510 may be processed in the manner described in any of the figures in or in any other suitable manner for speech conversion.

[0053] In some implementations, the output 515 is used by the loss function 530 to generate a loss 540. That is, the loss function 530 compares the output 515 to the label 520 to generate the loss 540, where the loss 540 indicates the difference between the label 520 (i.e., the target output) and the output 515. The loss function 530 can implement any suitable technique to determine the loss, such as regression loss, mean squared error, mean squared logarithmic error, mean absolute error, binary classification, binary cross entropy, hinge loss, multi-class loss, etc. The loss 540 can then be fed directly to the sub-model 215. Here, the sub-model 215 processes the loss 540 and adjusts one or more parameters of the sub-model 215 to account for the loss 540. In some implementations, the sub-model 215 is continuously trained (or retrained) as additional speech conversion requests are received. For example, when the speech conversion model 200 is frozen, the sub-model 215 can continue to adjust parameters based on the received speech conversion request 105, which includes the audio data 102, the speaker identifier 103, and the biased speech representation 224. Here, in the case of multiple sub-models 215, each sub-model 215 can be trained for biasing to the associated domain when used by the speech conversion model 200.

[0054] Although the examples herein relate to a sub-model 215 that biases the speech conversion model 200 for speech conversion (and speech recognition using word patches and / or phoneme decoders), it should be understood that the sub-model 215 and speaker identifier 103 can be used to bias any kind of model for any purpose. For example, the sub-model 215 can bias an image recognition model, a recommendation model, a filtering (e.g., email) model, a medical diagnosis model, or any other model in which contextual information can be used to bias the results to increase accuracy. As described above, the sub-model 215 can be trained on appropriate speaker identifiers 103 to appropriately bias the underlying base model.

[0055] Figure 6 is a flow chart of an exemplary arrangement of operations for performing speech conversion using speech conversion model 200 and sub-model 215. Method 600 may be performed by, for example Figure 1The method 600 is performed by various elements of the speech conversion system 100 of the present invention. At operation 602, the method 600 includes: obtaining a speech conversion model 200, the speech conversion model being configured to directly convert an input utterance 108 of human speech into a corresponding output utterance of synthesized speech. At operation 604, the method 600 includes: receiving a speech conversion request 105, the speech conversion request including input audio data 102 and a speaker identifier 103, the input audio data 102 corresponding to an utterance 108 spoken by a target speaker 104 associated with an atypical speech, the speaker identifier 103 uniquely identifying the target speaker 104. At operation 606, the method 600 includes: using the speaker identifier 103 to activate a specific sub-model 215 for biasing the speech conversion model 200 to recognize the type of atypical speech associated with the target speaker 104 identified by the speaker identifier 103. At operation 608, method 600 includes using the speech conversion model 200 biased by the activated particular sub-model 215 to convert input audio data 102 corresponding to an utterance 108 spoken by a target speaker 104 associated with atypical speech into output audio data corresponding to a synthetic canonical fluent speech representation 224 of the utterance 108 spoken by the target speaker 104.

[0056] Figure 7 is a schematic diagram of an example computing device 700 that can be used to implement the systems and methods described in this document. Computing device 700 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions are intended to be exemplary only, and are not intended to limit the implementations described and / or claimed in this document.

[0057] The computing device 700 includes a processor 710, a memory 720, a storage device 730, a high-speed interface / controller 740 connected to the memory 720 and a high-speed expansion port 750, and a low-speed interface / controller 760 connected to a low-speed bus 770 and the storage device 730. Each of the components 710, 720, 730, 740, 750, and 760 is interconnected using various buses and can be installed on a common motherboard or installed in other ways as appropriate. The processor 710 can process instructions for execution within the computing device 700, including instructions stored in the memory 720 or on the storage device 730, to display graphical information of a graphical user interface (GUI) on an external input / output device (such as a display 780 coupled to the high-speed interface 740). In other implementations, multiple processors and / or multiple buses and multiple memories and multiple types of memories can be used as appropriate. In addition, multiple computing devices 700 can be connected (for example, as a server bank, a blade server group, or a multi-processor system), each of which provides part of the necessary operations.

[0058] The memory 720 stores information non-temporarily within the computing device 700. The memory 720 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-temporary memory 720 may be a physical device for storing programs (e.g., instruction sequences) or data (e.g., program state information) on a temporary or permanent basis for use by the computing device 700. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0059] The storage device 730 is capable of providing mass storage for the computing device 700. In some implementations, the storage device 730 is a computer-readable medium. In various implementations, the storage device 730 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices (including devices in a storage area network or other configuration). In other implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or a machine-readable medium, such as the memory 720, the storage device 730, or a memory on the processor 710.

[0060] The high-speed controller 740 manages bandwidth-intensive operations of the computing device 700, while the low-speed controller 760 manages less bandwidth-intensive operations. This division of responsibilities is exemplary only. In some implementations, the high-speed controller 740 is coupled to the memory 720, the display 780 (e.g., through a graphics processor or accelerator), and the high-speed expansion port 750 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 760 is coupled to the storage device 730 and the low-speed expansion port 790. The low-speed expansion port 790, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner; or to a networking device, such as a switch or a router, for example, through a network adapter.

[0061] As shown, computing device 700 can be implemented in a variety of different forms. For example, the computing device can be implemented as a standard server 700a or multiple implementations in a group of such servers 700a, implemented as a laptop computer 700b, or implemented as part of a rack server system 700c.

[0062] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuit systems, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system that includes at least one programmable processor that can be either special purpose or general purpose and can be coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0063] A software application (i.e., software resource) may refer to computer software that enables a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0064] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0065] The process and logic flow described in this specification can be performed by one or more programmable processors (also referred to as data processing hardware), which execute one or more computer programs to perform functions by operating on input data and generating output. The process and logic flow can also be performed by a dedicated logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). For example, processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as a disk, a magneto-optical disk, or an optical disk, or be operably coupled to receive data from one or more mass storage devices or transfer data to one or more mass storage devices or both. However, a computer does not have to have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and storage devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CDROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated in special purpose logic circuitry.

[0066] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor) or a touch screen for displaying information to the user, and optionally a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0067] A variety of implementations have been described. However, it should be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other implementations are within the scope of the appended claims.

Claims

1. A computer-implemented method (600) which, when executed on data processing hardware (154), causes the data processing hardware (154) to perform operations comprising: obtaining a speech conversion model (200) configured to directly convert an input utterance of human speech into a corresponding output utterance of synthesized speech; receiving a speech conversion request (105), the speech conversion request (105) comprising input audio data (102) corresponding to an utterance (108) spoken by a target speaker (104) associated with an atypical speech, and a speaker identifier (103) uniquely identifying the target speaker (104); using the speaker identifier (103) to activate a specific sub-model (215) for biasing the speech conversion model (200) to recognize the type of atypical speech associated with the target speaker (104) identified by the speaker identifier (104); as well as The speech conversion model (200) biased by the activated particular sub-model (215) is used to convert the input audio data (102) corresponding to the utterance (108) spoken by the target speaker (104) associated with atypical speech into output audio data corresponding to a synthetic canonical fluent speech representation (224) of the utterance (108) spoken by the target speaker (104).

2. The computer-implemented method (600) of claim 1, wherein the speech conversion model (200): is trained (410) on generalization training data; and It is speaker-independent and domain-independent.

3. The computer-implemented method (600) of claim 1 or 2, wherein the speech conversion model (200) comprises: an encoder (310) configured to encode the input audio data (102) corresponding to the utterance (108); as well as A decoder (320), the decoder (320) being configured to: receiving as input encoded audio data (312) from the encoder (310); and The output audio data corresponding to the synthesized canonical fluent speech representation (224) of the utterance (108) is generated as output.

4. The computer-implemented method (600) of any one of claims 1 to 3, wherein the encoder (310) comprises a stack of self-attention blocks each having a multi-head self-attention mechanism.

5. A computer-implemented method (600) as described in any one of claims 1 to 4, wherein the sub-model (215) includes a stack of residual adapters (361) arranged between each of the self-attention blocks in the stack of self-attention blocks of the encoder (310).

6. A computer-implemented method (600) as claimed in any one of claims 1 to 5, wherein each residual adapter (361) comprises a normalization layer, followed by a feed-forward layer with a downward projection to the bottleneck dimension, and a non-linear activation, and another feed-forward layer with an upward projection.

7. The computer-implemented method (600) of any one of claims 1 to 6, wherein the speech conversion model (200) further comprises a word slice decoder (320, 320b), the word slice decoder (320, 320b) being configured to: receiving as input the encoded audio data (312) from the encoder (310); and A textual representation (224, 224b) corresponding to the transcription of the utterance (108) is generated as an output.

8. The computer-implemented method (600) of any one of claims 1 to 7, wherein the speech conversion model (200) further comprises a phoneme decoder (320, 320c), the phoneme decoder (320, 320c) being configured to: receiving as input the encoded audio data (312) from the encoder (310); and A phonemic representation (224, 224c) of the utterance (108) is generated as output.

9. The computer-implemented method (600) of any one of claims 1 to 8, wherein: The input audio data (102) comprises one of an input spectrogram or an input audio waveform; and The output audio data comprises one of an output spectrogram or an output audio waveform.

10. The computer-implemented method (600) of claim 1, wherein activating the particular sub-model (215) for use in biasing the speech conversion model (200) comprises: selecting the particular sub-model (215) associated with the type of atypical speech associated with the target speaker (104) from among a plurality of sub-models (215) each associated with a different type of atypical speech; and The specific sub-model (215) is loaded into the speech conversion model (200) for biasing the speech conversion model (200) to recognize the type of the atypical speech associated with the target speaker (104).

11. A system (100), comprising: Data processing hardware (154); as well as Memory hardware (156) in communication with the data processing hardware (154), the memory hardware (156) storing instructions that, when executed on the data processing hardware (154), cause the data processing hardware (154) to perform operations, the operations comprising: obtaining a speech conversion model (200) configured to directly convert an input utterance of human speech into a corresponding output utterance of synthesized speech; receiving a speech conversion request (105), the speech conversion request (105) comprising input audio data (102) corresponding to an utterance (108) spoken by a target speaker (104) associated with an atypical speech, and a speaker identifier (103) uniquely identifying the target speaker (104); using the speaker identifier (103) to activate a specific sub-model (215) for biasing the speech conversion model (200) to recognize the type of atypical speech associated with the target speaker (104) identified by the speaker identifier (104); as well as The speech conversion model (200) biased by the activated particular sub-model (215) is used to convert the input audio data (102) corresponding to the utterance (108) spoken by the target speaker (104) associated with atypical speech into output audio data corresponding to a synthetic canonical fluent speech representation (224) of the utterance (108) spoken by the target speaker (104).

12. The system (100) of claim 11, wherein the speech conversion model (200): is trained (410) on generalization training data; and It is speaker-independent and domain-independent.

13. The system (100) of claim 11 or 12, wherein the speech conversion model (200) comprises: an encoder (310) configured to encode the input audio data (102) corresponding to the utterance (108); as well as A decoder (320), the decoder (320) being configured to: receiving as input encoded audio data (312) from the encoder (310); and The output audio data corresponding to the synthesized canonical fluent speech representation (224) of the utterance (108) is generated as output.

14. The system (100) of any one of claims 11 to 13, wherein the encoder (310) comprises a stack of self-attention blocks, each having a multi-head attention mechanism.

15. The system (100) of any one of claims 11 to 14, wherein the sub-model (215) comprises a stack of residual adapters (361) disposed between each of the self-attention blocks in the stack of self-attention blocks of the encoder (310).

16. The system (100) of any one of claims 11 to 15, wherein each residual adapter (361) comprises a normalization layer, followed by a feed-forward layer with a downward projection to the bottleneck dimension, and a non-linear activation, and another feed-forward layer with an upward projection.

17. The system (100) of any one of claims 11 to 16, wherein the speech conversion model (200) further comprises a word slice decoder (320, 320b), the word slice decoder (320, 320b) being configured to: receiving as input the encoded audio data (312) from the encoder (310); and A textual representation (224, 224b) corresponding to the transcription of the utterance (108) is generated as an output.

18. The system (100) of any one of claims 11 to 17, wherein the speech conversion model (200) further comprises a phoneme decoder (320, 320c), the phoneme decoder (320, 320c) being configured to: receiving as input the encoded audio data (312) from the encoder (310); and A phonemic representation (224, 224c) of the utterance (108) is generated as output.

19. The system (100) of any one of claims 11 to 18, wherein: The input audio data (102) comprises one of an input spectrogram or an input audio waveform; and The output audio data comprises one of an output spectrogram or an output audio waveform.

20. The system (100) of any one of claims 11 to 19, wherein activating the specific sub-model (215) for biasing the speech conversion model (200) comprises: selecting the particular sub-model (215) associated with the type of atypical speech associated with the target speaker (104) from among a plurality of sub-models (215) each associated with a different type of atypical speech; and The specific sub-model (215) is loaded into the speech conversion model (200) for biasing the speech conversion model (200) to recognize the type of the atypical speech associated with the target speaker (104).

Images