A scalable model specialization framework for individualizing speech models.

Abstract

A method (600) for speech conversion includes obtaining a speech conversion model (200) configured to directly convert an input utterance into a corresponding output utterance of a synthetic voice. The method further includes receiving a speech conversion request (105) including input audio data (102) corresponding to an utterance (108) associated with an atypical speech and a speaker identifier (103) uniquely identifying a target speaker (104). The method includes activating a specific sub-model (215) to bias the speech conversion model to recognize a type of atypical speech associated with the target speaker identified by the speaker identifier using the speaker identifier. The method includes converting the input audio data into output audio data corresponding to a synthesized standard and fluent speech representation (224) of the utterance spoken by the target speaker using the speech conversion model biased by the activated specific sub-model.

Description

Technical Field

[0001] The present disclosure relates to a scalable model specialization framework for customizing voice models.

Background Art

[0002] Using a voice conversion model, the voice of a source speaker can be changed into another format without changing the language information of the voice. For example, the voice conversion model can generate a transcript of the user's voice. Alternatively, the voice conversion model can convert the user's voice into an audio waveform of a voice in another language. By using a machine learning method, the voice conversion model can be trained to accurately and efficiently convert the voice into another format. Since the voice conversion model is usually a large-scale model trained with a large-scale dataset, it is generally adapted to a large set of users.

Summary of the Invention

[0003] One aspect of the present disclosure provides a computer implementation method for biasing a speech-to-translate model using a submodel. The computer implementation method, when executed by data processing hardware, causes the data processing hardware to perform an operation that includes acquiring a speech-to-translate model configured to directly translate human speech input utterances into corresponding output utterances of synthesized speech. The operation includes receiving a speech-to-translate request that includes input audio data corresponding to utterances spoken by a target speaker associated with an ad-hoc speech, and a speaker identifier that uniquely identifies the target speaker. The operation further includes using the speaker identifier to activate a particular submodel to bias the speech-to-translate model in order to recognize the type of ad-hoc speech associated with the target speaker identified by the speaker identifier. The operation also includes using the speech-to-translate model biased by the activated particular submodel to translate the input audio data corresponding to utterances spoken by a target speaker associated with an ad-hoc speech into output audio data corresponding to a synthesized standard, fluent speech representation of the utterances spoken by the target speaker.

[0004] Embodiments of the present disclosure may include one or more of the following optional features. In some embodiments, the speech-to-speech model is trained on generalized training data and is speaker and domain independent. In some embodiments, the speech-to-speech model includes an encoder configured to encode input audio data corresponding to an utterance. In these embodiments, the speech-to-speech model also includes a decoder configured to receive encoded audio data from the encoder as input and to produce output audio data corresponding to a synthesized standard, fluent speech representation of the utterance as output. In these embodiments, the encoder may include a stack of self-attention blocks, each having a multi-head self-attention mechanism. In these embodiments, a submodel may include a stack of residual adapters positioned between each self-attention block in the stack of self-attention blocks of the encoder. Furthermore, in these embodiments, each residual adapter may include a normalization layer followed by a feedforward layer with down-projection to a bottleneck dimension, and a nonlinear activation layer followed by another feedforward layer with up-projection. In these embodiments, the speech-to-text model may further include a word-fragment decoder configured to receive encoded audio data from an encoder as input and to produce a text representation corresponding to the transcription of the utterance as output. Alternatively, in these embodiments, the speech-to-text model may further include a phoneme decoder configured to receive encoded audio data from an encoder as input and to produce a phoneme representation of the utterance as output.

[0005] The input audio data may include either an input spectrogram or an input audio waveform, and the output audio data may include either an output spectrogram or an output audio waveform. In some embodiments, activating a specific submodel to bias the speech-to-speech model includes selecting a specific submodel associated with a type of atypical speech associated with a target speaker from among several submodels, each associated with a different type of atypical speech. Embodiments further include loading a specific submodel into the speech-to-speech model to bias the speech-to-speech model in order to recognize the type of atypical speech associated with a target speaker.

[0006] Another aspect of the present disclosure provides a system for biasing a speech-to-transformation model using submodels. The system includes data processing hardware and memory hardware that communicates with the data processing hardware. The memory hardware stores instructions that cause the data processing hardware to perform an action when executed by the data processing hardware. The action includes obtaining a speech-to-transformation model configured to directly translate human speech input utterances into corresponding synthesized speech output utterances. The action includes receiving a speech-to-transformation request that includes input audio data corresponding to utterances spoken by a target speaker associated with an ad-hoc speech, and a speaker identifier that uniquely identifies the target speaker. The action further includes using the speaker identifier to activate a particular submodel to bias the speech-to-transformation model in order to recognize the type of ad-hoc speech associated with the target speaker identified by the speaker identifier. The action also includes using the speech-to-transformation model biased by the activated particular submodel to translate the input audio data corresponding to utterances spoken by a target speaker associated with an ad-hoc speech into output audio data corresponding to a synthesized standard, fluent speech representation of the utterances spoken by the target speaker.

[0007] This embodiment may include one or more of the following optional features. In some embodiments, the speech-to-speech model is trained on generalized training data and is speaker and domain independent. In some embodiments, the speech-to-speech model includes an encoder configured to encode input audio data corresponding to an utterance. In these embodiments, the speech-to-speech model also includes a decoder configured to receive encoded audio data from the encoder as input and produce output audio data corresponding to a synthesized standard, fluent speech representation of the utterance as output. In these embodiments, the encoder may include a stack of self-attention blocks, each having a multi-head self-attention mechanism. In these embodiments, the submodel may include a stack of residual adapters positioned between each self-attention block in the stack of self-attention blocks of the encoder. Furthermore, in these embodiments, each residual adapter may include a normalization layer, followed by a feedforward layer with downward projection to a bottleneck dimension, a nonlinear activation layer, and another feedforward layer with upward projection. In these embodiments, the speech-to-text model may further include a word-fragment decoder configured to receive encoded audio data from an encoder as input and to produce a text representation corresponding to the transcription of the utterance as output. Alternatively, in these embodiments, the speech-to-text model may further include a phoneme decoder configured to receive encoded audio data from an encoder as input and to produce a phoneme representation of the utterance as output.

[0008] The input audio data may include either an input spectrogram or an input audio waveform, and the output audio data may include either an output spectrogram or an output audio waveform. In some embodiments, activating a specific submodel to bias the speech-to-speech model includes selecting a specific submodel associated with a type of atypical speech associated with a target speaker from among several submodels, each associated with a different type of atypical speech. Embodiments further include loading a specific submodel into the speech-to-speech model to bias the speech-to-speech model in order to recognize the type of atypical speech associated with a target speaker.

[0009] One or more embodiments of the present disclosure are described in detail in the accompanying drawings and the following description. Other embodiments, features, and advantages will become apparent from the description and drawings and the claims. [Brief explanation of the drawing]

[0010] [Figure 1] This is a schematic diagram of an exemplary speech conversion system, including a speech conversion model. [Figure 2A] This is a schematic diagram of a speech conversion model that generates speech representations using submodels. [Figure 2B] This is a schematic diagram of a speech conversion model that generates speech representations using selected submodels. [Figure 3A] This is a schematic diagram of a speech conversion model that uses a submodel as a residual adapter layer to generate a speech representation. [Figure 3B] This is a schematic diagram of a speech conversion model that generates speech representations using submodels as residual adapter layers placed between encoder layers. [Figure 3C] This is a schematic diagram of a speech conversion model with various decoders. [Figure 4] This is a schematic diagram of an exemplary training scheme for a speech conversion model. [Figure 5]This is a schematic diagram illustrating an exemplary training scheme for a submodel of a speech conversion model. [Figure 6] This is an illustrative flowchart of the configuration of a method for performing speech conversion using a modified speech conversion model in a submodel. [Figure 7] This is a schematic diagram of an exemplary computing device that may be used to implement the systems and methods described herein. [Modes for carrying out the invention]

[0011] Similar reference symbols in various drawings indicate the same elements.

[0012] There is growing interest in developing more comprehensive speech technologies, particularly those that can assist people with speech impairments. Automatic speech recognition (ASR) has made significant progress with the introduction of end-to-end (E2E) deep learning-based models to recognize speech from speakers with dysarthria or atypical speech patterns in order to translate it into an accurate transcript. Atypical speech patterns, for example, may include, but are not limited to, speech with physical or neurological conditions (e.g., speakers with amyotrophic lateral sclerosis (ALS)), heavily accented speech, and impaired speech due to hearing impairment. Similar deep learning-based models can be applied by speech conversion systems to convert speech with atypical speech patterns into standard, fluent output speech.

[0013] Using machine learning techniques, speech-to-transformation models can be trained on large training datasets containing audio samples of speech to generate robust models for speech recognition / transformation. Generally, these speech-to-transformation models are large because performance improves the more extensively the model is trained. However, there are drawbacks to using such large models, such as a single model used for a wide variety of users with different characteristics. For example, a single speech-to-transformation model may be built for users with fixed speech patterns, even though some users may have atypical speech patterns. As a result, the speech-to-transformation model may not function very accurately for a particular group of users. Furthermore, retraining or updating the model due to size differences is difficult due to the computational cost. This can cause the speech-to-transformation model to become outdated and not function well for new / emerging words / phrases (e.g., slang, new TV programs).

[0014] Fine-tuning and fitting neural models has become a standard approach to model specialization or task customization. Substantial improvements have been demonstrated across a wide range of downstream tasks by fine-tuning the entire model or a subset of its parameters. However, these techniques, when applied to numerous tasks, domains, conditions, and / or users, introduce scalability limitations to both training and inference.

[0015] Embodiments of this specification involve integrating one or more submodels with a base speech-to-speech model to personalize the base speech-to-speech model for speech-to-speech in specific domains (i.e., specific users and / or the type of speech associated with a user). A submodel includes a set of parameters that can be added to or replaced by a general base speech-to-speech model. Submodels can be loaded / enabled / disabled as needed, enabling the use of the base speech-to-speech model without personalization, thereby eliminating problems arising from modifying larger models, such as concerns about catastrophic forgetting. In some examples, after the base speech-to-speech model is trained, the parameters of the base speech-to-speech model are frozen during operation (i.e., estimation). In this way, the base speech-to-speech model remains stable and can continue to function for general traffic or multiple users / domains. However, where speaker identification information is available, the base speech-to-speech model may activate a submodel appropriate for a specific speaker and / or a specific type of speech to perform speech-to-speech based on the user's speech type. One advantage of these embodiments is the ability to train submodels individually without the need to train or retrain the larger base speech-to-speech model. Furthermore, one or more submodels can be loaded and replaced dynamically and in real time on demand (e.g., using a memory cache). The base speech-to-text model remains unchanged regardless of whether submodels are used, eliminating concerns that the model may be compromised by use and / or updates.

[0016] As used herein, unless otherwise specified, the terms “speech-to-speech system” and “speech-to-speech model” may refer to any combination of speech-to-speech systems / models in which input atypical speech is recognized and converted into corresponding text (e.g., transcript) and / or a set of phonemes representing the atypical speech, or in which input atypical speech is directly converted into standard, fluent synthesized speech without performing speech recognition. In other words, a speech-to-speech system / model is configured to directly convert an input audio waveform or spectrogram corresponding to atypical speech to an output audio waveform or spectrogram corresponding to standard, fluent speech, without converting the input audio waveform to an intermediate representation (e.g., text or phonemes). As will become clear, speech-to-speech models, and the techniques for training speech-to-speech models, enable users with atypical speech to speak to and be understood by both other humans and voice interfaces (e.g., digital assistants) by enabling the recognition and / or reproduction of the user’s intended speech. The embodiments of this specification show a speech-to-speech model that receives an input audio waveform or spectrogram corresponding to atypical speech and converts it to an output audio waveform or spectrogram corresponding to standard, fluent speech. However, the speech-to-speech model can be adapted to perform other types of speech-to-speech tasks without departing from the scope of this disclosure. For example, the speech-to-speech model may convert an input audio waveform or spectrogram corresponding to utterances in a first language to an output audio waveform or spectrogram corresponding to translations of utterances in a different second language. The speech-to-speech model can also receive input spoken by a user and output a synthesized speech that contains the same language content as the spoken input but with different voice characteristics from the target speaker.

[0017] Figure 1 shows a voice conversion system 100 including an automatic voice conversion model 200 and a sub-model 215. The voice conversion model 200 is configured to process a voice conversion request 105 using the sub-model 215. The voice conversion request 105 includes input audio data 102 corresponding to an utterance 108 spoken by a target speaker 104 and captured by a user device 110. The voice conversion request 105 may further include a speaker identifier 103. Using the audio data 102 and the speaker identifier 103, the voice conversion model 200 and the sub-model 215 generate or predict an unbiased voice representation 222 or a biased voice representation 224. The biased voice representation 224 is likely to include words or phrases associated with a particular user and / or a particular voice type that bias the voice conversion model 200 using the sub-model 215 (e.g., based on the speaker identifier 103). In some embodiments, when the target speaker 104 is identified as having non-standard speech, the voice representation 224 is a synthesized standard and fluent voice representation 224 of the utterance 108 spoken by the target speaker 104. In some examples, the input audio data 102 includes an input spectrogram corresponding to the utterance 108. The speaker identifier 103 may provide an indication of a particular user and / or a particular domain (i.e., voice type) among a plurality of different domains, each of which means or represents a respective bias context of the utterance 108.

[0018] Although not shown, an acoustic front end present on the user device 110 can convert the time-domain audio waveform of the utterance 108 captured via the microphone of the user device 110 into the input spectrogram 102, or other types or formats of audio data 102. Further, the front-end device may be configured to determine or obtain data representing the speaker identifier 103 that affects the utterance 108, and / or other relevant information corresponding to the target speaker 104 and / or the client device 110.

[0019] A user device 110 associated with a target speaker 104 may capture an utterance 108 spoken by the target speaker 104 and provide the corresponding input audio data 102 to the speech conversion system 100 as part of a speech conversion request 105. Furthermore, the user device 110 may determine a speaker identifier 103 to be included in the speech conversion request 105. The user device 110 may include, but is not limited to, a smartphone, tablet, desktop / laptop computer, smart speaker, smart display, smart home appliance, assistant-enabled wearable device (e.g., smartwatch, smart headphones, smart glasses, etc.), or vehicle infotainment system. Alternatively, the remote server 112 may process the audio data 102 and any other additional data or metadata from the user device 110 to determine the speaker identifier 103.

[0020] The speech conversion system 100 may be distributed across multiple devices, such that the speech conversion model 200 resides on a user device 110 or on one of the remote systems 150 (also referred to herein as a cloud computing environment) that communicates with the user device 110 via a network 140. The remote system 150 may be a single computer, multiple computers, or a distributed system (e.g., a cloud environment) having scalable / elastic resources 152 including computing resources 154 (e.g., data processing hardware) and / or storage resources 156 (e.g., memory hardware). A datastore 158 (i.e., a remote storage device) may be overlaid on the storage resources 146, enabling scalable use of the storage resources 146 by one or more user devices 110 or computing resources 154. The speech conversion model 200 and submodel 215 may run on the remote system 150 or the user device 110. Submodel 215 may be stored locally on the user device 110, or on a remote system (for example, on the data store 158), or in some combination of the above.

[0021] The voice conversion system 100 implements a dynamic model that includes two or more component parts or models, namely a general base model (e.g., the voice conversion model 200) and an "exchangeable" sub-model 215, to generate a biased voice representation 224 based on the received voice conversion request 105. The voice conversion model 200 may be trained on a large-scale voice dataset. Once trained, the voice conversion model 200 may be frozen so that the parameters of the voice conversion model 200 remain constant during operation. If necessary, or when additional training data becomes available, the voice conversion model 200 may be updated, retrained, or replaced. In some embodiments, a single sub-model 215 is used to bias the voice conversion model 200 by activating specific parameters based on the speaker identifier 103 of the voice conversion request 105. In other embodiments, the system 100 includes multiple sub-models 215, each of which corresponds to a specific domain (i.e., a specific user and / or a specific voice type). In these embodiments, a specific sub-model 215 is selected / activated based on the speaker identifier 103. For example, if the speaker identifier 103 indicates that the target speaker 104 has a voice type associated with non-standard speech (e.g., stuttering), the voice conversion model 200 may activate the sub-model 215 associated with that voice type. If the voice conversion request 105 does not have a speaker identifier 103, or if there is no sub-model 215 corresponding to the speaker identifier 103 or it is under development, in some examples, the voice conversion model 200 generates an unbiased voice representation 222. That is, the unbiased voice representation 222 is generated only by the voice conversion model 200 and is neither affected nor acted upon by any sub-model 215.

[0022] The speaker classifier 103 may be based on any signal or data that can be used to improve the accuracy of the biased speech representation 224. The speaker identifier 103 may be based on information related to the target speaker 104. For example, the target speaker 104 may have a particular dialect, native intonation, accent, voice pattern, hesitation, etc. Thus, the system 100 may generate and train a submodel 215 corresponding to the target speaker, the submodel 215 biasing / individualizing the speech conversion model 200 to make predictions particularly suitable for the target speaker 104. In some embodiments, the speaker identifier 103 includes a one-hot vector, and the system uses the speaker identifier 103 to select the appropriate submodel 215 (i.e., the submodel 215 corresponding to the target speaker 104) from a group of different submodels 215.

[0023] In some embodiments, the speaker identifier 103 is based on information related to the user device 110. For example, the user device 110 is associated with a specific user (i.e., a target speaker 104), and therefore any utterance 108 associated with the user device 110 is sent as a speech conversion request 105 that includes a speaker identifier 103 indicating the specific user. Furthermore, the user device 110 may include a smart device equipped with sensors such as a GPS, accelerometer, gyroscope, microphone, proximity sensor, and camera. Alternatively, the speech conversion model 200 may determine the speaker identifier 103 associated with a speech type by identifying a specific speech type based on an analysis of the audio input 102.

[0024] The speaker identifier 103 may indicate a domain associated with the user device 110, as inferred from one of the sensors. For example, the speaker identifier 103 may indicate the geographic location of the user device 110, as inferred from GPS data (provided that user 104 explicitly agrees to share the user's geographic location, which can be disabled at any time). Here, the speaker identifier 103 may correspond to a more geographic location (e.g., a city such as Chicago) or a more specific location (e.g., a gym). In either case, the submodel 215 may bias the biased speech representation 224 for a specific domain (i.e., speech type) based on the location. Specifically, the speaker identifier 103 identifying the location Chicago may activate a submodel 215 trained on data from users from that city, state, and / or region, and the submodel 215 may be biased based on the user's speech habits or other characteristics from named entities specific to that area and region (e.g., restaurants, sports teams, street names, etc.). As a result, submodel 215 can bias the speech conversion model 200 to generate biased speech representations 224 that are biased towards predictions that fit its domain. For example, a biased speech representation 224 may increase the likelihood that the speech conversion model 200 predicts that utterance 108 contains references to restaurants or streets in Chicago. Similarly, a speaker identifier 103 indicating that source speaker 104 is at a gym may activate a trained submodel 215 based on speech from a user who is exercising or in a similar location. Here, the speech may be affected by breathing disturbances or directed towards specific words or phrases (e.g., voice instructions for operating a music player on a smart device). Thus, submodel 215 can incorporate these contextual elements when processing the audio data 102 to generate biased speech representations 224.

[0025] In some embodiments, the speaker identifier 103 indicates that multiple domains are applicable to the utterance 108. In this scenario, a single submodel 215 may bias the speech-to-transformation model 200 to produce a biased speech representation 224 for each of the multiple domains. Alternatively, multiple submodels 215 (each trained for one or more of the multiple domains) may bias the speech-to-transformation model 200 to produce a biased speech representation 224 for each of the multiple domains. For example, if the target speaker 104 has a particular speech type (e.g., stuttering) and is located in a particular location associated with the habit (e.g., Boston), one or more submodels 215 may bias the output of the speech-to-transformation model 200 for each of these domains.

[0026] Output 190 may receive unbiased and biased speech representations 222, 224 generated by the speech conversion model 200. In some examples, the output is a standard, fluent speech representation 222, 224 of the utterance 108, if the speech conversion model 200 is configured to process atypical speech. In other examples, output 190 includes natural language understanding (NLU) which performs query interpretation on the speech conversion result. The NLU may further instruct downstream applications / services to take action based on the result. Output 190 may also include a user interface generator configured to display the speech conversion result as a transcript on the screen of user device 110 and / or another device. Similarly, output 190 may be fed to a vocoder (not shown) to generate a time-domain audio waveform of the synthesized standard, fluent speech representation of the utterance 108.

[0027] The system in Figure 1 is presented for illustrative purposes only and is not intended to limit the scope of the system. For example, while only a single example of each component is shown, system 100 may include any number of components 110, 112, 140, 150, 200, and 215. Furthermore, while some components are described as being located in the cloud computing environment 150, in some embodiments these components may be hosted locally on the user device 110. Moreover, in various embodiments some or all of components 112, 200, and 215 may be hosted locally on the user device 110, hosted remotely (such as within the cloud computing environment 150), or hosted in any combination thereof.

[0028] Referring here to Figure 2A, the exemplary speech-to-voice model 200 implements a submodel 215 to generate a biased speech representation 224 (e.g., a synthesized standard, fluent speech representation 224). Here, the speech-to-voice model 200 receives a speech-to-voice request 105 containing audio data 102 and a speaker identifier 103. In this example, the submodel 215 includes a single model trained on various inputs and contexts corresponding to various domains. The speaker identifier 103 may be a one-hot vector indicating one or more specific domains (e.g., a specific user and / or a specific voice type) corresponding to the context of the audio data 102. The one-hot vector may be concatenated and projected onto a phrase set embedding before being sent to the submodel 215. In some embodiments, the one-hot vector is looked up in an embedding matrix before being projected onto the phrase set embedding. Alternatively, the submodel 215 may project the one-hot vector onto the phrase set embedding. Next, the submodel may activate one or more parameters corresponding to one or more specific domains indicated by the speaker identifier 103, based on the phrase set embedding, in order to process the audio data 102.

[0029] Referring here to Figure 2B, another exemplary speech-to-voice model 200 implements selected submodels 215, 215b to generate a biased speech representation 224 (e.g., a synthesized standard, fluent speech representation 224). Here, the speech-to-voice model 200 receives a speech-to-voice request 105 containing audio data 102 and a speaker identifier 103. In this case, the speaker identifier 103 is received by a submodel selector 218. The submodel selector 218 selects a particular submodel 215 from a plurality of submodels 215, 215a-n, each submodel 215 corresponding to one or more different domains. Submodels 215a-n may be trained with data reflecting any domain that provides context for processing the audio input 102. For example, the selected submodel 215b is based on domains such as location, activity, user profile, voice type, or any other domain that can influence how the audio data 102 is processed. The speaker identifier 103 may be a one-hot vector indicating a specific domain corresponding to the context of the speech conversion request 105. Upon receiving the speaker identifier 103, the submodel selector 218 may perform a lookup to determine the appropriate submodel(s) 215 to use when processing the speech conversion request 105. When processing the streaming audio input 102 from the user device 110, the various submodels 215a-n are readily interchangeable so that the speech conversion model 200 can swap in and out (i.e., disable or enable the bias provided by the submodels 215) from among the multiple submodels 215a-n while processing the streaming input as a new speaker identifier 103 is received. If the speech conversion request 105 does not include the speaker identifier 103, or if the speaker identifier 103 is not applicable to any of the available submodels 215, the speech conversion model 200 processes the audio input 102 without activating or enabling any submodel 215 and generates an unbiased speech representation 222 (Figure 1).

[0030] Submodel 215 may be implemented to bias the output of the speech conversion model 200 in various ways. Figure 3A shows a schematic diagram 300a of the speech conversion model 200 for generating a biased speech representation 224 using submodel 215 implemented as a residual adapter layer 361 (Figure 3B). The speech conversion model 200 may be a recurrent neural network (RNN) including an encoder 310 configured to encode input audio data 102 into an encoded output 312 (e.g., a hidden feature representation containing a series of vectors) and a decoder 320 configured to decode the encoded output 312 into a biased speech representation 224. Typically, the encoded output 312 is sent directly to the decoder 320 to generate an unbiased speech representation 222. However, in this example, submodel 215 operates in parallel with the processing of the speech conversion request 105. As a result, submodel 215 may generate a submodel output 325 based on the received audio input 102 of the speech conversion request 105. The speech conversion model 200 may combine the submodel output 325 and the encoded output 312 to generate a biased encoded output 314, which it may then transmit to the decoder 320.

[0031] In another example, submodel 215 is implemented between the layers of encoder 310. Figure 3B shows an exemplary schematic diagram 300b of such an embodiment. Encoder 310 may include several components 360, and submodel 215 may be placed between the layers of components 360 so that encoder 310 produces a biased encoded output 319. The components 360 of the encoder may include a stack of multi-head attention blocks (i.e., conformer blocks) which may include conformers or transformers. In some embodiments, each multi-head attention block includes a multi-head attention mechanism. Encoder 310 may include a stack of long short-term memory (LSTM) instead of multi-head attention blocks. Decoder 320 may receive the biased encoded output 319 and produce a biased representation 224.

[0032] Here, submodel 215 may be implemented as one or more residual adapter layers 361 placed between the components 360 of the encoder 310 (e.g., a stack of self-attention blocks, each having a multi-head self-attention mechanism). Each residual adapter layer 361 begins with layer normalization applied to the input, and dimension d b A feedforward layer with downward projection onto the bottleneck dimension, nonlinear activation (RELU), and the original input dimension d i Another feedforward layer with upward projection to may follow. In some embodiments, all weights of the residual adapter 361 are initialized randomly. In a specific example, each submodel 215 includes 17 residual adapter layers 361, each layer being added between the layers of the encoder. Furthermore, bottleneck d b It may be set to 64, while all adapter weights are initialized randomly.

[0033] The residual adapter layer 361 offers several advantages to the implementation of the submodel 215. For example, the residual adapter layer 361 can be easily added to the encoder, and various submodels 215 can be easily swapped out as needed. Furthermore, the submodel 215 can be easily muted / disabled by setting the residual coefficient to zero (i.e., removing the submodel 215 and allowing the speech conversion model 200 to operate in an unbiased manner). The size of the submodel 215, when implemented as the residual adapter layer 361, depends on the task / use case (i.e., the domain associated with the speaker identifier 103) and the bottleneck dimension (e.g., d b This can be controlled by [the specified method]. Furthermore, by controlling the bottleneck dimension within submodel 215, it becomes possible to dynamically replace the tensor shape while enabling fast inference of a precompiled and optimized execution graph.

[0034] Figure 3C shows a schematic diagram of an exemplary speech conversion model 200 used by the speech conversion system 100 of Figure 1. The speech conversion model 200 includes an encoder 310 and one or more decoders 320, 320a-c. The encoder 310 is configured to encode the input audio data 102 of a speech conversion request 105 into an encoded spectrogram 312, where the input audio data 102 includes a sequence of input spectrograms corresponding to an utterance 108 spoken by a target speaker 104. In some embodiments, the encoder 310 includes a stack of conformer blocks 360 and / or one or more submodels 215 placed between the conformer blocks 360. In these embodiments, the encoder 310 subsamples the input audio data 102 using a convolutional layer and then processes the input audio data 102 using a stack of conformer blocks 360. Each conformer block 360 may include a feedforward layer, a self-attention layer, a convolutional layer, and a second feedforward layer. In some examples, a stack of conformer blocks 360 includes 17 layers of conformer blocks 360, each having 512 states, 8 attention heads, and a convolutional kernel size of 32 × 1. The encoder 310 may use a stack of transformer blocks, lightweight convolutional blocks, or other types of multi-head attention blocks instead of conformer blocks.

[0035] Spectrogram, phoneme, and word piece decoders 320, 320a-c may each include a recurrent neural network-based architecture in which they each receive a shared encoded spectrogram 312 output by the encoder 310. Spectrogram decoder 320a may include a cross-attention mechanism 331, 331a configured to receive the shared encoded spectrogram 312 from the encoder 310. Spectrogram decoder 320a may further process the shared encoded spectrogram 312 using several long short-term memory (LSTM) layers 333, 333a and several convolutional layers 335. For example, spectrogram decoder 320a may include five LSTM layers 333a and five transformation layers 335. The spectrogram decoder 320a may generate output spectrograms 224, 224a (i.e., speech representation 224). In some embodiments, the spectrogram decoder 320a can directly generate output spectrogram 224a from the encoded spectrogram 312 without performing any intermediate text-to-speech conversion on the text representation corresponding to the speech transcription.

[0036] In the illustrated example, the word fragment decoder 320b includes corresponding cross-attention mechanisms 331, 331b configured to receive a shared encoded spectrogram 312 from the encoder 310, followed by two long short-term memory (LSTM) layers 333, 333b, and softmax layers 345, 345a that output text representations 224, 224b corresponding to the transcription of the utterance 108. The text representation 224 may be a transcription of the utterance 108 spoken by the target speaker 104, as a synthesized standard, fluent speech representation 224.

[0037] Similar to the word fragment decoder 320b, the phoneme decoder 320c also includes a cross-attention mechanism 331, 331c configured to receive a shared encoded spectrogram 312 from the encoder 310, followed by two long short-term memory (LSTM) layers 333, 333c, and a softmax layer 345, 345b that outputs phoneme representations 224, 224c of the utterance 108, where the phoneme representation 224c contains phonemes representing a synthesized standard, fluent speech representation 224 of the utterance 108.

[0038] The examples in Figures 3A, 3B, and 3C are illustrative only and not intended to limit the scope. The speech conversion model 200 and submodel 215 may include any suitable structure / architecture to perform speech conversion and generate a biased speech representation 224 in accordance with the speaker identifier 103. Furthermore, the submodel 215 and the speech conversion model 200 may function in any suitable combination to generate a biased speech representation 224. For example, the submodel 215 may be placed within the speech conversion model 200 at any suitable location in the architecture of the speech conversion model 200. For example, the submodel 215 may be deployed within a layer of the speech conversion model 200 as a residual adapter layer, as a tensor, as an encoder / decoder layer, as a prediction network, as a coupling network, etc. Alternatively, the submodel 215 and the speech conversion model 200 may generate outputs independently of each other, and the results may be combined by the speech conversion model 200 or another suitable component of the system to determine the biased speech representation 224. In particular, when submodel 215 is disabled from the original frozen state of speech conversion model 200, speech conversion model 200 remains unchanged. That is, when submodel 215 is disabled (for example, because speaker identifier 103 is absent), speech conversion model 200 generates an unbiased speech representation 222 that is not affected by submodel 215.

[0039] Figure 4 shows the training process 400 for the speech-to-voice model 200. The speech-to-voice model 200 may be specifically adapted to normalize / convert dysarticulate and atypical speech to a synthesized standard, fluent speech representation 224 (e.g., stereotyped speech). The speech-to-voice model 200 may be trained in an end-to-end training process 400 to directly map an input spectrogram to another spectrogram without using or relying on intermediate discrete representations. The generated output spectrograms (e.g., speech representations 222 / 224) may be passed to a vocoder to generate a time-domain waveform. Alternatively, the output spectrograms may generate a transcript in parallel, so that the speech-to-voice model 200 is effectively also an ASR system. In some embodiments, the process 400 utilizes a two-step training technique including pre-training and training. Pre-training of the model is the technique used to initialize the model, which can then be further refined based on additional training data 410. With respect to the speech model 200, pre-training may include starting the speech conversion model 200 with pre-training data 405, which includes multiple spoken utterances by one or more speakers. The pre-training data 405 may further include spoken utterances paired with corresponding ground truth synthesized speech representations of the spoken utterances 108. The speech samples used for pre-training may be speech synthesized from a reference transcript in a given voice, and / or speech synthesized from non-synthesized speech samples spoken by actual humans.

[0040] Process 400 may fine-tune the parameters of the pre-trained speech-to-transformation model 200 after pre-training is complete. The training process 400 includes, for example, training the encoder 310 and / or decoder 320 (Figure 3A) separately or jointly in any suitable combination. Process 400 includes supplying the training input 410 to the speech-to-transformation model 200. The speech-to-transformation model 200 may be trained with generalized training data 410 such that the speech-to-transformation model 200 is a general-purpose model that can be further individualized / biased using submodels. The training data 410 may be speaker and / or domain independent. In some embodiments, the training input 410 includes multiple speech samples spoken by various different speakers. Here, the training input 410 may include speech samples from a user with a fixed speech pattern. Furthermore, the training input 410 may be labeled with a label 420 indicating the target output associated with the training input 410. Upon receiving the training input 410, the speech conversion model 200 may generate an output 415 (e.g., an unbiased speech representation 222). The speech conversion model 200 may process the training input 410 in the manner described with respect to any of Figures 2A to 3C, or in any other suitable manner for speech conversion.

[0041] In some embodiments, the output 415 is used by a loss function 430 to generate a loss 440. That is, the loss function 430 compares the output 415 with the label 420 to generate a loss 440, which indicates a mismatch between the label 420 (i.e., the target output) and the output 415. The loss function 430 may implement any appropriate technique to determine the loss, such as regression loss, mean squared error, mean squared logarithmic error, mean absolute error, binary classification, binary cross-entropy, hinge loss, or multiclass loss. The loss 440 may then be fed directly to the speech-to-speech model 200, where the speech-to-speech model 200 processes the loss 440 and adjusts one or more parameters of the speech-to-speech model 200 to compensate for the loss 440. In some embodiments, the model freezes when the speech-to-speech model 200 is properly trained. In other words, the parameters remain unchanged for a certain period until it is deemed necessary to retrain or replace the speech conversion model 200 (for example, when enough new training data 410 has been acquired).

[0042] Figure 5 shows the training process 500 for submodel 215. In some embodiments, process 500 utilizes a two-step training technique including pre-training and training. Pre-training is a technique used to initialize the model, which can then be further refined based on additional training data 510. With respect to submodel 215, pre-training may include starting submodel 215 with pre-training data 505, which includes multiple spoken utterances by a speaker. Pre-training data 505 may further include spoken utterances paired with corresponding ground truth synthesized speech representations of the spoken utterances. Speech samples used for pre-training may be synthesized from a reference transcript in a given voice. With respect to submodel 215, pre-training may further include adjusting one or more parameters of submodel 215 so that submodel 215 can be biased towards words and / or phrases corresponding to a particular user and / or a particular domain (e.g., speech type) associated with submodel 215.

[0043] In some examples, process 500 includes fine-tuning the parameters of a pre-trained submodel 215. In these examples, process 500 includes supplying a training input 510 to the submodel 215. In some embodiments, the training input 510 includes multiple speech samples spoken by speakers corresponding to a specific speaker and / or a specific voice type. The training input 510 may also include a speaker identifier 103 and a transcript using each speech sample. The transcript may be manually generated text representing the corresponding speech sample. In some embodiments, the transcript is machine-generated. The transcript must accurately reflect the corresponding speech sample so that the transcript is the target output of the submodel 215. In some embodiments, the speaker identifier 103 of the training input 510 may indicate a specific domain (e.g., a specific user or a specific voice type). In other embodiments, the speaker identifier 103 of the training input 510 may include a one-hot vector. A submodel 215 corresponding to a speaker identifier 103 may be trained with speech samples and transcripts so that the submodel 215 is biased to a specific user and / or a specific voice type associated with the speaker identifier 103. In an example of a single submodel 215 adapted to bias multiple voice types, the one-hot vector of the speaker identifier 103 may be concatenated and projected onto a phrase set embedding, which can then be used to train the submodel 215. Furthermore, the training input 510 may be labeled with a label 420 indicating a target output associated with the training input 510. Upon receiving the training input 510, the submodel 215 may generate an output 515 (e.g., a biased speech representation 224). The submodel 215 may process the training input 510 in the manner described with respect to any of Figures 2A to 3C, or in any other suitable manner for speech conversion.

[0044] In some embodiments, the output 515 is used by a loss function 530 to generate a loss 540. That is, the loss function 530 compares the output 515 with the label 520 to generate a loss 540, which indicates a mismatch between the label 520 (i.e., the target output) and the output 515. The loss function 530 may implement any appropriate technique to determine the loss, such as regression loss, mean squared error, mean squared logarithmic error, mean absolute error, binary classification, binary cross-entropy, hinge loss, or multiclass loss. The loss 540 may then be fed directly to a submodel 215, where the submodel 215 processes the loss 540 and adjusts one or more parameters of the submodel 215 to take the loss 540 into account. In some embodiments, when additional speech conversion requests are received, the submodel 215 is continuously trained (or retrained). For example, while the speech conversion model 200 is frozen, the submodel 215 can continue to adjust its parameters based on the received speech conversion request 105, which includes audio data 102, a speaker identifier 103, and a biased speech representation 224. Here, in the case of multiple submodels 215, each submodel 215 can be trained in use by the speech conversion model 200 to bias the relevant domain.

[0045] The examples herein concern a submodel 215 that biases a speech conversion model 200 for speech conversion (and speech recognition using a word fragment decoder and / or phoneme decoder), and it is understood that the submodel 215 and speaker identifiers 103 may be used to bias any type of model used for any purpose. For example, the submodel 215 can bias an image recognition model, a recommendation model, a filtering (e.g., email) model, a medical diagnostic model, or any other model that can use contextual information to bias the results in order to improve accuracy. As described above, the submodel 215 may be trained with appropriate speaker identifiers 103 to appropriately bias the underlying base model.

[0046] Figure 6 is a flowchart illustrating an exemplary configuration of an operation for performing speech conversion using a speech conversion model 200 and a submodel 215. Method 600 may be performed, for example, by various elements of the speech conversion system 100 in Figure 1. In operation 602, method 600 includes obtaining a speech conversion model 200 configured to directly convert a human speech input utterance 108 into a corresponding output utterance of synthesized speech. In operation 604, method 600 includes receiving a speech conversion request 105 which includes input audio data 102 corresponding to an utterance 108 spoken by a target speaker 104 associated with an atypical speech, and a speaker identifier 103 that uniquely identifies the target speaker 104. In operation 606, method 600 includes using the speaker identifier 103 to activate a specific submodel 215 to bias the speech conversion model 200 in order to recognize the type of atypical speech associated with the target speaker 104 identified by the speaker identifier 103. In operation 608, method 600 includes using a speech conversion model 200 biased by a specific activated submodel 215 to convert input audio data 102 corresponding to an utterance 108 spoken by a target speaker 104 associated with an atypical speech to output audio data corresponding to a synthesized standard, fluent speech representation 224 of the utterance 108 spoken by the target speaker 104.

[0047] Figure 7 is a schematic diagram of an exemplary computing device 700 that can be used to implement the systems and methods described herein. The computing device 700 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown herein, their connections and relationships, and their functions are illustrative only and are not intended to limit the embodiments described and / or claimed herein.

[0048] The computing device 700 includes a processor 710, memory 720, storage device 730, a high-speed interface / controller 740 connected to memory 720 and a high-speed expansion port 750, and a low-speed bus 770 and a low-speed interface / controller 760 connected to storage device 730. Each component 710, 720, 730, 740, 750, and 760 are interconnected using various buses and may be mounted on a common motherboard or implemented in other ways as needed. The processor 710 processes instructions for execution within the computing device 700, including instructions stored in memory 720 or storage device 730, to display graphical information of a graphical user interface (GUI) on an external input / output device such as a display 780 connected to the high-speed interface 740. In other embodiments, multiple processors and / or multiple buses may be used as needed, along with multiple memories and memory types. Also, multiple computing devices 700 may be connected so that each device provides a required portion of the operation (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

[0049] Memory 720 stores information non-temporarily within the computing device 700. Memory 720 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. Non-temporarily stored memory 720 may be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by the computing device 700. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and disk or tape.

[0050] The storage device 730 can provide high-capacity storage to the computing device 700. In some embodiments, the storage device 730 is a computer-readable medium. In various different embodiments, the storage device 730 may be an array of devices including floppy disk devices, hard disk devices, optical disk devices, or tape devices, flash memory or other similar solid-state memory devices, or devices of a storage area network or other configuration. In additional embodiments, a computer program product is visibly embodied in an information carrier. The computer program product includes instructions that perform one or more of the above-described methods at runtime. The information carrier is a computer-readable or machine-readable medium, such as memory 720, the storage device 730, or memory on the processor 710.

[0051] The high-speed controller 740 manages the bandwidth-intensive operation of the computing device 700, and the low-speed controller 760 manages the low-bandwidth-intensive operation. Such role assignments are merely examples. In some embodiments, the high-speed controller 740 is connected to memory 720, a display 780 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 750 that can accept various expansion cards (not shown). In some embodiments, the low-speed controller 760 is connected to a storage device 730 and a low-speed expansion port 790. The low-speed expansion port 790 may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), and the low-speed expansion port 790 may be connected to one or more input / output devices such as a keyboard, pointing device, scanner, etc., or to network devices such as switches and routers, for example, via a network adapter.

[0052] The computing device 700 may be implemented in several different forms, as shown in the figure. For example, it may be implemented as a standard server 700a, or multiple times within a group of such servers 700a, as a laptop computer 700b, or as part of a rack server system 700c.

[0053] Various embodiments of the systems and technologies described herein can be realized as digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs executable and / or interpretable on a programmable system which may be for a specific purpose or general purpose, comprising at least one programmable processor, at least one input device, and at least one output device connected to receive data and instructions from a storage system and transmit data and instructions to a storage system.

[0054] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform a task. In some examples, a software application may be called an “application,” “app,” or “program.” Exemplary applications include, but are not limited to, system diagnostic applications, system administration applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and game applications.

[0055] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transient computer-readable medium, apparatus and / or device (e.g., magnetic disks, optical disks, memory, programmable logic circuits (PLDs)) used to provide machine instructions and / or data to a programmable processor that includes a machine-readable medium that receives machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0056] The processes and logic flows described herein can be executed by one or more programmable processors (also called data processing hardware) executing one or more computer programs to perform functions by acting on input data and producing outputs. Processes and logic flows can also be executed by special-purpose logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). Processors suitable for executing computer programs include, by example, one or more processors from both general-purpose and dedicated microprocessors, as well as any type of digital computer. Generally, processors receive instructions and data from read-only memory, random-access memory, or both. The basic elements of a computer are a processor for executing instructions, and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operablely connected to receive data from or transmit data to them, or both. However, a computer is not required to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices including EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Processors and memory can be complemented by or incorporated into dedicated logic circuits.

[0057] To interact with a user, one or more aspects of the present disclosure may be implemented in a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touchscreen, and optionally a keyboard and pointing device, such as a mouse or trackball, by which the user can input to the computer. Other types of devices may also be used to interact with a user. For example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or haptic feedback, and input from the user may be received in any form, including acoustic, voice, or haptic input. Furthermore, the computer may interact with the user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from a web browser.

[0058] Several embodiments have been described. Needless to say, it is understood that various modifications can be made without departing from the spirit and scope of this disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A method performed by data processing hardware, To obtain a speech conversion model configured to directly convert human speech input utterances into corresponding synthesized speech output utterances without performing speech recognition, Receiving a speech conversion request that includes input audio data corresponding to an utterance spoken by a target speaker associated with a non-standard speech, and a speaker identifier that uniquely identifies the target speaker, From among multiple submodels associated with different types of atypical speech, the speaker identifier is used to select a specific submodel associated with the type of atypical speech associated with the target speaker identified by the speaker identifier, In order to apply a bias to the aforementioned speech conversion model, the specific submodel is loaded into the speech conversion model, Using the speech conversion model biased by the aforementioned specific submodel, the input audio data corresponding to the utterance spoken by the target speaker associated with the atypical speech is converted into output audio data corresponding to a synthesized standard, fluent speech representation of the utterance spoken by the target speaker. Methods that include...

2. The aforementioned voice conversion model is Trained with generalized training data, The method according to claim 1, which is speaker and domain independent.

3. The aforementioned speech conversion model, An encoder configured to encode the input audio data corresponding to the utterance, It is a decoder, The encoded audio data is received from the encoder as input. As output, the system generates output audio data corresponding to the synthesized standard and fluent speech representation of the utterance. A decoder configured in such a way, The method according to claim 1, including the method described in claim 1.

4. The method according to claim 3, wherein the encoder includes a stack of self-attention blocks, each having a multi-head self-attention mechanism.

5. The method according to claim 4, wherein the submodel includes a stack of residual adapters positioned between each of the self-attention blocks in the stack of the self-attention blocks of the encoder.

6. The method according to claim 5, wherein each residual adapter comprises a normalization layer, followed by a feedforward layer with downward projection to the bottleneck dimension, a nonlinear activation layer, and another feedforward layer with upward projection.

7. The aforementioned speech conversion model further includes a word fragment decoder, The aforementioned word fragment decoder The encoded audio data is received from the encoder as input. As output, it generates a text representation corresponding to the transcription of the utterance. The method according to claim 3, configured as described above.

8. The aforementioned speech conversion model further includes a phoneme decoder, The aforementioned phoneme decoder The encoded audio data is received from the encoder as input. The output generates a phoneme representation of the aforementioned utterance. The method according to claim 3, configured as described above.

9. The aforementioned input audio data includes one of the input spectrogram or input audio waveform. The method according to claim 1, wherein the output audio data includes one of an output spectrogram or an output audio waveform.

10. Data processing hardware and Memory hardware that communicates with the data processing hardware, wherein the memory hardware stores instructions, and when an instruction is executed by the data processing hardware, it causes the data processing hardware to perform an operation, and the operation is To obtain a speech conversion model configured to directly convert human speech input utterances into corresponding synthesized speech output utterances without performing speech recognition, Receiving a speech conversion request that includes input audio data corresponding to an utterance spoken by a target speaker associated with a non-standard speech, and a speaker identifier that uniquely identifies the target speaker, From among multiple submodels associated with different types of atypical speech, the speaker identifier is used to select a specific submodel associated with the type of atypical speech associated with the target speaker identified by the speaker identifier, In order to apply a bias to the aforementioned speech conversion model, the specific submodel is loaded into the speech conversion model, Using the speech conversion model biased by the aforementioned specific submodel, the input audio data corresponding to the utterance spoken by the target speaker associated with the atypical speech is converted into output audio data corresponding to a synthesized standard, fluent speech representation of the utterance spoken by the target speaker. The memory hardware includes, A system that includes this.

11. The aforementioned voice conversion model is Trained with generalized training data, The system according to claim 10, which is speaker and domain independent.

12. The aforementioned voice conversion model is An encoder configured to encode the input audio data corresponding to the utterance, It is a decoder, The encoded audio data is received from the encoder as input. As output, the system generates output audio data corresponding to the synthesized standard and fluent speech representation of the utterance. The decoder is configured as described above, The system according to claim 10, including the following:

13. The system according to claim 12, wherein the encoder includes a stack of self-attention blocks, each having a multi-head self-attention mechanism.

14. The system according to claim 13, wherein the submodel includes a stack of residual adapters positioned between each of the self-attention blocks in the stack of the self-attention blocks of the encoder.

15. The system according to claim 14, wherein each residual adapter includes a normalization layer, followed by a feedforward layer with downward projection to the bottleneck dimension, a nonlinear activation layer, and another feedforward layer with upward projection.

16. The aforementioned speech conversion model further includes a word fragment decoder, The aforementioned word fragment decoder The encoded audio data is received from the encoder as input. As output, it generates a text representation corresponding to the transcription of the utterance. The system according to claim 12, configured as described above.

17. The aforementioned speech conversion model further includes a phoneme decoder, The aforementioned phoneme decoder The encoded audio data is received from the encoder as input. The output generates a phoneme representation of the aforementioned utterance. The system according to claim 12, configured as described above.

18. The aforementioned input audio data includes one of the input spectrogram or input audio waveform. The system according to claim 10, wherein the output audio data includes one of an output spectrogram or an output audio waveform.

19. A computer program that causes the data processing hardware to execute the method according to any one of claims 1 to 9.

Images