Generating audio from multiple inputs using neural networks

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an audio signal. In one aspect, a method comprises obtaining a context input for generating the audio signal, wherein the context input comprises a plurality of inputs; processing each of the inputs using a corresponding encoder neural network to generate a respective representation of the input; processing the respective representations of the inputs using a shared encoder to generate an encoded representation; processing the encoded representation using a token decoder neural network to generate a sequence of output tokens representing the audio signal; and processing the sequence of output tokens representing the audio signal using an audio decoder neural network to generate the audio signal.

Description

[0001] GENERATING AUDIO FROM MULTIPLE INPUTS USING NEURAL NETWORKS

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to U.S. Provisional Application No. 63 / 647,582, filed on May 14. 2024. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application.

[0004] BACKGROUND

[0005] This specification relates to generating audio from multiple inputs using neural networks.

[0006] Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters.

[0007] SUMMARY

[0008] This specification describes a system implemented as computer programs on one or more computers in one or more locations that generates an audio signal from multiple inputs using one or more neural networks. The multiple inputs can include, for example, text or audio.

[0009] Generally, the output audio signal is an output audio example that includes a sample of an audio wave at each of a sequence of output time steps that span a specified time window. For example, the output time steps can be arranged at regular intervals within the specified time window.

[0010] The audio sample at a given output time step can be an amplitude value of the audio wave or an amplitude value that has been compressed, companded, or both. For example, the audio sample can be a raw amplitude value or a mu-law companded representation of the amplitude value.

[0011] In general, one innovative aspect of the subject matter described in this specification can be embodied in methods for generating an audio signal that include the actions of obtaining a context input for generating the audio signal, wherein the context input comprises a plurality of inputs; processing each of the inputs using a corresponding encoder neural network to generate a respective representation of the input; processing the respective representations of the inputs using a shared encoder to generate an encoded representation; processing the encoded representation using a token decoder neural network to generate a sequence of output tokens representing the audio signal; and processing the sequence of output tokens representing the audio signal using an audio decoder neural network to generate the audio signal.

[0012] In some implementations, the respective representation of the input comprises a fixed- length sequence of summarization tokens.

[0013] In some implementations, processing the respective representations of the inputs using a shared encoder comprises providing one or more sequences of embeddings and one or more summarization tokens as input to the shared encoder to generate updated summarization tokens.

[0014] In some implementations, the corresponding encoder neural network is configured to process the input to generate the respective representation of the input, comprising a sequence of embeddings for the input.

[0015] In some implementations, the shared encoder is configured to generate a concatenation of the respective representations of the inputs.

[0016] In some implementations, the shared encoder comprises a shared encoder neural network configured to process the concatenation to generate the encoded representation.

[0017] In some implementations, the shared encoder neural network is a self-attention neural network.

[0018] In some implementations, the plurality of inputs are derived from two or more modalities of data.

[0019] In some implementations, the plurality of inputs comprise one or more first inputs derived from a source audio signal, and the generated audio signal is time synchronous with the source audio signal.

[0020] In some implementations, the audio signal shares one or more prosodic properties with the source audio signal.

[0021] In some implementations, the plurality of inputs comprise a text input, and the generated audio signal represents speech for the text input.

[0022] In some implementations, the plurality of inputs comprise one or more second inputs derived from an input video, and the generated audio signal is time synchronous with the input video. In some implementations, the plurality of inputs comprise a text input comprising a multi-turn dialogue, and the generated audio signal represents speech in context with the multi-turn dialogue.

[0023] In some implementations, obtaining a context input for generating an audio signal comprises: receiving the source audio signal; and generating, from the source audio signal, one or more first inputs.

[0024] In some implementations, the plurality of inputs comprise any two or more of: a text input, an input audio signal representing speech, data representing features of a video, data representing energy features of a source audio signal, data representing pitch features of a source audio signal, data representing spectral features of a source audio signal, or speaker embeddings of a source audio signal.

[0025] In some implementations, the plurality of inputs comprise (i) an input audio signal representing speech by a particular speaker and (ii) a text input, and the audio signal represents the text input spoken by the particular speaker.

[0026] In some implementations, the plurality of inputs comprise (i) an input audio signal representing speech in a first language by a particular speaker and (ii) a text input in a second language, and the audio signal represents the text input spoken by the particular speaker.

[0027] In some implementations, the input audio signal comprises at least a portion of a source audio signal that represents speech in the first language.

[0028] In some implementations, obtaining a context input for generating an audio signal comprises generating the text input by processing the source audio signal using a sequence of one or more neural networks configured to perform automatic speech translation.

[0029] In some implementations, the method further comprises generating a training example comprising the source audio signal and the audio signal.

[0030] In some implementations, the method further comprises providing the training example for training a speech processing model.

[0031] In some implementations, the plurality' of inputs further comprise any one or more of: data representing energy features of the source audio signal, data representing pitch features of the source audio signal, data representing spectral features of the source audio signal, or speaker embeddings of the source audio signal. In some implementations, the shared encoder comprises a shared encoder neural network, and the shared encoder neural network and token decoder neural network are trained on a plurality of training examples, each comprising (i) a representation of a training text input, (ii) one or more representations of data representing features of a training source audio signal representing speech in a first language, and (iii) a ground-truth sequence of output tokens representing a target audio signal representing speech in a second language.

[0032] In some implementations, the shared encoder comprises a shared encoder neural network, and the shared encoder neural network and token decoder neural network are trained on a plurality of training examples, each comprising (i) a representation of a training text input, (ii) one or more representations of data representing features of a target audio signal representing speech in a second language, and (iii) a ground-truth sequence of output tokens representing the target audio signal.

[0033] In some implementations, the plurality of inputs comprise (i) a first text input that comprises a sequence of one or more dialogue turns, (ii) a second text input that comprises a response to a last dialogue turn of the sequence, (iii) one or more input audio signals, each representing speech of one of the one or more dialogue turns, and (iv) an input audio signal representing speech by a particular speaker, and the audio signal represents the second text input spoken by the particular speaker.

[0034] In some implementations, the shared encoder comprises a shared encoder neural network, and the shared encoder neural network and the token decoder neural network are trained on a plurality of training examples, each comprising (i) a representation of a training first text input comprising a sequence of one or more dialogue turns, (ii) a representation of a training second text input comprising a response to a last dialogue turn of the sequence, (iii) one or more representations of one or more training source audio signals, each representing speech of one of the one or more dialogue turns, (iv) a representation of a training input audio signal representing speech by a particular speaker, and (v) a ground-truth sequence of output tokens representing a target audio signal representing the training second text input spoken by the particular speaker.

[0035] In some implementations, the plurality of inputs comprise (i) a text input, and (ii) data representing an input video depicting an environment, and the audio signal represents the text input spoken in the environment.

[0036] In some implementations, the input video further depicts one or more characteristics of a speaker, and the audio signal represents the text input spoken with the one or more characteristics. In some implementations, the input video further depicts one or more lip movements of a speaker, and the audio signal represents the text input spoken to align with the one or more lip movements.

[0037] In some implementations, the shared encoder comprises a shared encoder neural network, and one or more of the corresponding encoder neural networks are pre-trained and frozen prior to training the shared encoder and the token decoder neural network.

[0038] In some implementations, the shared encoder comprises a shared encoder neural network, and one or more of the corresponding encoder neural networks are pre-trained, and wherein parameters for one or more of the corresponding encoder neural networks are updated during training of the shared encoder and the token decoder neural network.

[0039] In general, another innovative aspect of the subject matter described in this specification can be embodied in methods for generating an audio signal that include the actions of obtaining an encoded representation, wherein the encoded representation is derived from a plurality of distinct inputs; for each frame in a sequence of frames corresponding to a sequence of output tokens representing the audio signal, wherein each frame corresponds to one or more output tokens of the sequence of output tokens: generating, using a first machine learning model and conditioned on at least the encoded representation, a frame token for the frame; and generating, using one or more second machine learning models and conditioned on the frame token, one or more output tokens corresponding to the frame; and processing the one or more output tokens corresponding to each frame using an audio decoder neural network to generate the audio signal.

[0040] In some implementations, generating, using a first machine learning model and conditioned on at least the encoded representation, a frame token for the frame comprises generating the frame token conditioned on the encoded representation and one or more combined embeddings, wherein each of the one or more combined embeddings represent one or more output tokens corresponding to a respective preceding frame of the sequence of frames.

[0041] In general, another innovative aspect of the subj ect matter described in this specification can be embodied in methods for generating an encoded representation configured for use in generating an audio signal that include the actions of obtaining a context input, wherein the context input comprises a plurality7of inputs; obtaining, for each input of the plurality of inputs, a respective representation of the input; processing the respective representations of the inputs using a shared encoder to generate the encoded representation, comprising: concatenating the respective representations of the inputs; and processing the concatenation using a shared encoder neural network to generate the encoded representation.

[0042] In general, another innovative aspect of the subject matter described in this specification can be embodied in methods for training a shared encoder that include the actions of training the shared encoder on a plurality of training examples, wherein each training example comprises (i) a training input that comprises one or more representations of inputs for a target audio signal and (ii) a target output that comprises a ground-truth sequence of output tokens representing the target audio signal, and wherein the method comprises training the shared encoder, by a machine learning technique, to optimize an objective function that, for each training example, measures an error between: (i) the target output specified by the training example, and (ii) a predicted target output generated by processing an encoded representation generated by the shared encoder for the training input using a token decoder neural network.

[0043] In general, another innovative aspect of the subj ect matter described in this specification can be embodied in methods for training one or more neural networks for generating an audio signal that include the actions of obtaining a plurality of training examples, wherein each training example comprises (i) a training input that comprises one or more inputs for a target audio signal and (ii) a target output that comprises a ground-truth sequence of output tokens representing the target audio signal; generating, for each training example, and for each of the one or more inputs, a representation of the input; processing, for each training example, the representations of the one or more inputs using a shared encoder to generate an encoded representation for the training input; processing the encoded representation using a token decoder neural network to generate a predicted sequence of output tokens representing the target audio signal; and training the shared encoder and the token decoder neural network, by a machine learning technique, to optimize an objective function that, for each training example, measures an error between: (i) the target output specified by the training example, and (ii) the predicted sequence of output tokens.

[0044] In some implementations, generating, for each training example, and for each of the one or more inputs, a representation of the input comprises generating the representation of the input using a corresponding encoder neural network.

[0045] In some implementations, the corresponding encoder neural network is pre-trained and frozen prior to training the shared encoder and the token decoder neural network. In some implementations, the corresponding encoder neural network is pre-trained, and wherein parameters for the corresponding encoder neural network are updated during training of the shared encoder and the token decoder neural network.

[0046] Other implementations of these aspects include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices.

[0047] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0048] The system described in this specification can generate audio signals conditioned on a variety of inputs. Some conventional systems that conditionally generate audio signals receive only text or a voice sample as conditioning. The system described in this specification can generate audio signals conditioned on a set of multiple inputs, both with and without timing information. For example, the inputs can include text, audio, video, etc. In some examples, the inputs can include features derived from text, audio, or video, e.g., energy, pitch, etc.

[0049] In some examples, the system described in this specification can preserve or reflect the characteristics, e.g., speaker identity, context, of a source audio signal in the audio generated by the system. For example, the inputs can include one or more of a source audio signal representing speech, data representing energy features of a source audio signal, data representing pitch features of a source audio signal, data representing spectral features of a source audio signal, or speaker embeddings of a source audio signal.

[0050] The generated audio signal exhibits the characteristics of the source audio signal and / or the energy' features, the pitch features, the spectral features and / or speaker characteristics represented by the speaker embeddings. This permits a generated audio signal which sounds more natural, and which is also objectively more comprehensible to a listener. This is because the source audio signal can embody non-textual information which is preserved in the generated audio signal. For example, in the case of a source audio signal associated with an input video, the source audio signal may label the content of an associated input video (e.g. identify a person depicted at a certain portion of the video); thus, preserving a timing relationship between the source audio signal and the video preserves correct labelling. In another example, speech characteristics may emphasize the importance of certain spoken words in the source audio signal which are critical for understanding by a listener; for example, words which are unfamiliar (e.g. a name which occurs for the first time in the source audio signal, or an uncommon technical term) may be said more slowly. In another example, the content of the source audio signal may be associated with the speaker (e.g. it may refer to "mv generation”), so that confusion may arise if the generated audio signal uses a voice sounding very different.

[0051] For example, the system can generate audio signals for dubbing. Dubbed speech can be used to provide videos, films, or audio recordings of speech in multiple languages. Dubbing speech in a first language into speech of the same meaning in a second language conventionally requires recording a bilingual speaker, which is difficult or infeasible for many combinations of first and second languages, or recording a speaker of the first language and a different speaker of the second language. In many examples, although the speeches by two speakers have the same meaning, the sound and characteristics of the speeches are different. For example, the two speeches can have different recording conditions, sound quality, prosody, duration, pitch, timbre, etc.

[0052] Some conventional systems that generate audio signals for dubbing perform text to speech to generate audio signals from text in the second language and a voice sample in the first language. In many examples, the speech generated by these conventional systems does not preserve characteristics of the voice sample, such as timing and prosody. In some implementations, the system described in this specification generates speech from text in a second language and a voice sample in a first language that preserves characteristics such as timing, duration, and prosody of the voice sample. The system generates speech that better captures the full effect of the voice sample in the first language, for example, that retains pauses within the voice sample, or that retains the timbre and pitch of the speaker of the voice sample. For example, the system can receive inputs such as data representing energy features of the voice sample, data representing pitch features of the voice sample, data representing spectral features of the voice sample, or speaker embeddings of the voice sample. The system can process the inputs using corresponding encoders to generate respective representations of the inputs, and process the representations of the inputs using a shared encoder to generate an encoded representation. The system can then process the encoded representation using decoder neural networks to generate the audio signal representing speech in the second language that sounds as if it was spoken by the speaker of the voice sample, while preserving characteristics of the voice sample. Thus the system generates high-quality audio signals that represent speech that can be used for dubbing.

[0053] As another example, the system can generate audio signals for dialogue. Dialogue includes multiple turns by two or more speakers responding to each other. Some conventional systems that perform text to speech do not take into account previous turns of dialogue, resulting in generated speech that sounds out of context from the previous turns of dialogue. For example, an utterance of ‘“yes’' may be spoken differently depending on the conversational context. The speech generated by conventional systems that receive only the text to be spoken and a voice sample can thus sound out of context with the dialogue.

[0054] In some implementations, the system described in this specification can generate audio signals that represent speech that sounds more fluent and natural in the context of previous dialogue. For example, the system can receive a text input that includes a multi-turn dialogue, a text input that represents text to be spoken, and a voice sample. The system can process the inputs using corresponding encoders to generate respective representations of the inputs, and process the representations of the inputs using a shared encoder to generate an encoded representation. The system can then process the encoded representation using decoder neural networks to generate the audio signal representing speech that sounds as if it was spoken by the speaker of the voice sample, while sounding natural in the context of the multi-tum dialogue. Thus the system generates high-quality audio signals that represent speech that sounds natural in the context of previous dialogue.

[0055] As another example, the system can generate an audio signal for a video that reflects the environmental conditions of the content of the video and is synchronized with the content of the video. The video can depict content that is set within a particular environment and in some examples can depict a speaker. Some conventional systems that perform text to speech do not take into account timing or environmental conditions depicted in the video, resulting in generated speech that sounds out of context from the video and does not match the timing of the video.

[0056] In some implementations, the system described in this specification can generate audio signals that reflect the timing, e.g., lip movements of a speaker, characteristics of a speaker, or environmental conditions, e.g., reverberation, depicted in the video by receiving data representing an input video. For example, the data representing the input video can include data representing content of the input video, or data representing motion of the input video. The system can process the inputs using corresponding encoders to generate respective representations of the inputs, and process the representations of the inputs using a shared encoder to generate an encoded representation. The system can then process the encoded representation using decoder neural networks to generate the audio signal representing speech that sounds as if it was spoken by the speaker depicted in the video, while sounding natural in the context of the environmental conditions. Thus the system generates high-quality audio signals that represent speech that reflects the environmental conditions of the content of the video and are synchronized with the content of the video.

[0057] In some examples, the system described in this specification can generate audio signals with different combinations of inputs, e.g., a set of inputs that includes one or more additional inputs of another set of inputs, a set of inputs that does not include one or more inputs of another set of inputs, or both. For example, the system can generate audio signals representing speech from text, a voice sample, and other types of inputs that can be derived from source inputs such as audio signals or video. The system can be trained using dropout to generate audio signals from different combinations of inputs. Thus the system is flexible at inference to different combinations of inputs. The system can generate audio signals with different combinations of inputs, without requiring the training and deploying of a different system for each combination of inputs, reducing the consumption of computing resources such as computing time and power that would otherwise be required for training and deployment of multiple systems.

[0058] In some implementations, the system described in this specification can generate synthetic parallel training data for training other machine learning models such as speech processing models. Parallel data that includes input speech in a first language, and output speech in a second language with the same meaning, speaker, and other characteristics such as prosody, as the input speech, is extremely rare and difficult to obtain. The system described in this specification can generate an audio signal in a second language with the same meaning and other characteristics as an input audio signal in a first language. The system can include the generated audio signal and the input audio signal in a training example. The system can thus enable the generation of synthetic parallel data for training speech processing models, resulting in better performance for the speech processing model compared to a speech processing model trained on a limited amount of parallel data.

[0059] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below'. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

[0060] BRIEF DESCRIPTION OF THE DRAWINGS

[0061] FIG. 1 is a block diagram of an example audio generation system.

[0062] FIG. 2 is a block diagram of another example audio generation system.

[0063] FIG. 3 is a block diagram of another example audio generation system. FIG. 4 is a flow diagram of an example process for generating an audio signal from multiple inputs.

[0064] FIG. 5 shows an example process for training an example system for generating audio.

[0065] FIG. 6 shows an example token decoder neural network.

[0066] Like reference numbers and designations in the various drawings indicate like elements.

[0067] DETAILED DESCRIPTION

[0068] FIG. 1 shows an example audio generation system 100. The system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0069] The audio generation system 100 generates a prediction of an audio signal 160 from a context input 101 that includes multiple inputs 102a-n. The audio signal 160 includes a respective audio sample at each of multiple output time steps spanning a time window.

[0070] To generate audio, the system 100 receives the context input 101. The context input 101 includes multiple inputs such as a text input 102a, an audio signal 102b, and, in some examples, one or more additional inputs 102c-n. For example, the audio signal 160 represents speech for the text input 102a. that preserves characteristics, e.g., speaker characteristics, timing, style, etc., of the audio signal 102b.

[0071] In some examples, the system 100 receives one or more inputs 102 of the context input 101 from a user. For example, the system 100 can receive the text input 102a from a user through a user interface of a user device.

[0072] In some examples, the system 100 generates one or more inputs 102 of the context input 101. For example, the system 100 can generate the text input 102a, the audio signal 102b, or both, from a source audio signal as described with reference to FIG. 3.

[0073] The text input 102a includes a natural language sequence of text to be represented as speech in the audio signal 160. In the example of FIG. 1, the text input 102a includes the English text ‘'Hello world."

[0074] The audio signal 102b includes speech spoken by a particular speaker. In some examples, the speech represents text that does not include the text input 102a. In the example of FIG. 1, the audio signal 102b includes the French speech “Bonjour le monde.” The one or more additional inputs 102c-n can include any of a variety of types of inputs. For example, the additional inputs 102c-n can include data representing features of a video, data representing energy features of a source audio signal, data representing pitch features of a source audio signal, data representing spectral features of a source audio signal, speaker embeddings of a source audio signal, etc. In some examples, the additional inputs 102c-n can include a text input representing a sequence of one or more dialogue turns and corresponding input audio signals representing speech of the one or more dialogue turns.

[0075] Generally, the inputs can be continuous or discrete inputs. For example, inputs such as data representing features of a video can be continuous, while text is discrete.

[0076] In some examples, the inputs are derived from two or more modalities of data. For example, the inputs can be derived from two or more of text, video, or audio data.

[0077] The system processes each input 102a-n of the context input 101 using a corresponding encoder neural network of the encoder neural networks 1 lOa-n to generate respective representations 112a-n. Each encoder neural network 1 lOa-n is configured to generate a respective representation 112a-n of the corresponding input 102a-n. For example, each encoder neural network 1 lOa-n can be configured to process the corresponding input 102a-n to generate a respective representation 112a-n of the input that includes a sequence of embeddings for the input. An embedding refers to an ordered collection of numerical values, e.g., a vector, matrix, or other tensor of numerical values. The embeddings for different representations 112 can have the same dimensionality. In some examples, the length of the sequence of embeddings for different representations can vary.

[0078] As an example, each encoder neural network 1 lOa-n can include multiple encoder layers. Typically, each of some or all of the encoder layers included in an encoder neural network can operate on a respective input sequence that includes a respective network input (e.g., in the form of a vector) at each of one or more positions in an input order.

[0079] Each encoder neural network, e.g., encoder neural network 110a, uses the encoder layers to process an input, e.g., input 102a, data derived from the input, or both, to generate a respective representation, e.g., respective representation 112a, of the input. The respective representation has a sequence of embeddings. In some examples, the respective representation is the output of the last encoder layer or a combination of the outputs of multiple encoder layers. For example, for an encoder neural network that generates embeddings for input audio (e.g. embeddings for respective temporal portions of the audio input; the temporal portions corresponding to different embeddings may be overlapping and may be of different lengths), the representation can include the output of the last encoder layer. In some examples, the respective representation can include a sequence of summarization tokens representing the outputs of one or more encoder layers, e.g., as aggregated embeddings that each represent two or more embeddings of the outputs of one or more encoder layers.

[0080] In some examples, each encoder neural network 1 lOa-n includes one or more attention layers. In some such implementations, each encoder neural network 1 lOa-n also includes other layers, e.g., fully-connected layers, embedding layers, and activation layers, either in place of or in addition to the attention layers.

[0081] A self-attention neural network layer has an attention layer input for each element of the input and is configured to apply an attention mechanism over attention layer inputs to generate an attention layer output for each element of the input. There are many different attention mechanisms that may be used.

[0082] Some example encoder neural networks 110 are described below with reference to FIGS. 2-3.

[0083] In some implementations, for one or more of the encoder neural networks, the system 100 can be configured to generate a fixed-length sequence of summarization tokens, e.g., embeddings, from a sequence of embeddings of a corresponding representation 112. In some examples, the number of summarization tokens in the fixed-length sequence is predetermined.

[0084] As an example, one or more of the encoder neural networks can each generate a sequence of embeddings. The system 100 can process each sequence of embeddings, e.g., using a respective aggregation layer, to generate a sequence of one or more summarization tokens that each summarize at least two embeddings from the sequence of embeddings. For example, each summarization token can be an aggregated embedding. For example, for an encoder neural network that generates embeddings for input text, the system 100 can generate summarization tokens using the embeddings for the input text. The system 100 includes the summarization tokens in the representation 112.

[0085] In some other implementations, the system 100 can be configured to generate the fixed-length sequence of summarization tokens to be included in the representation 112 using a neural network, e.g., a Transformer-based neural network. For example, the system 100 can provide a predetermined number, e.g., three, summarization tokens, and tokens representing the input to a neural network, e.g., a Transformer-based neural network. In some examples, the tokens representing the input can have been generated by an encoder neural network of the encoder neural networks 110, e.g., using atokenizer of the appropriate encoder neural network. The neural network processes the summarization tokens and the tokens representing the input to update all of the tokens. The system 100 includes the updated summarization tokens in the representation 112.

[0086] The system processes the respective representations 112 using a shared encoder 120 to generate an encoded representation 122. The shared encoder 120 is configured to generate a combination, e.g., a concatenation, of the respective representations 112. In some examples, the system 100 can use the concatenation as the encoded representation 122. In some examples, the system 100 includes positional encodings for the different representations.

[0087] In some examples, the respective representations 112 can have different lengths. The system 100 can pad representations that are shorter than a maximum length along the temporal dimension to the maximum length. For example, the system 100 can zero-pad the representation (i.e., adds one or more zeros to the end of the representation). The system 100 can concatenate the padded representations and corresponding padding indicators along the temporal dimension. For example, the system 100 can concatenate the padded representations so that embeddings for the same time step or frame are combined. The system can also use corresponding padding indicators to indicate which representations are padded or which part of the representations are padded.

[0088] In some examples, the system 100 can concatenate the representations 112 along the sequence length or channel dimension. For example, the representations 112 can include dimensions for different channels of a source audio signal. The system 100 can concatenate the representations 112 along the same channel dimension. In examples where the system 100 concatenates the representations 112 along the sequence length or length of the representations 112, the system can pad the representations in order to concatenate the representations 112.

[0089] In some examples, the shared encoder 120 includes a shared encoder neural network. In these examples, the shared encoder neural network processes the concatenation to generate the encoded representation 122. An example shared encoder neural network is described below with reference to FIG. 2.

[0090] In some implementations where the system 100 generates a fixed-length sequence of summarization tokens for each respective representation 112a-n, the system 100 can provide the one or more sequences of embeddings generated by the encoder neural networks 110 and the one or more summarization tokens as input to the shared encoder 120 to update the summarization tokens. The system can include the updated summarization tokens in the encoded representation 122. In some implementations where the system 100 generates a fixed-length sequence of summarization tokens for each respective representation 112a-n, the system 100 can provide the sequences of summarization tokens from the representations 112 as input to the shared encoder 120.

[0091] In some implementations, each respective representation 112a-n includes a fixed- length sequence of summarization tokens. The system 100 can provide the representations 112 as input to the shared encoder 120. The shared encoder 120 can concatenate the sequences of summarization tokens. In some examples, the system 100 can provide the concatenated sequences of summarization tokens as input to a shared encoder neural network of the shared encoder 120.

[0092] The system processes the encoded representation 122 using a token decoder neural network 130 to generate a sequence of output tokens 132. Each of the output tokens can be selected from a vocabulary of output tokens. As an example, the token decoder neural network 130 can have a Transformer-based architecture.

[0093] As a particular example, the token decoder neural network 130 can be a Transformerbased neural network that includes a plurality of layers that each apply an attention mechanism. For example, each layer can apply an attention mechanism such as self-attention or cross-attention.

[0094] In particular, the token decoder neural network 130 can be an auto-regressive neural network that auto-regressively generates the sequence of output tokens 132 by generating each particular output token in the sequence conditioned on the encoded representation 122 and a current input sequence that includes any tokens that precede the particular output token in the output sequence. The token decoder neural network 130 can apply a cross-attention mechanism over the encoded representation 122 and the current input sequence.

[0095] In some examples, the output tokens in the vocabulary include semantic tokens. Each semantic token is selected from the vocabulary and represents semantic content of the audio signal 160. Examples of semantic content represented by the semantic tokens can include linguistic content, phonetics, language syntax, and prosodic features for speech.

[0096] In some examples, the output tokens in the vocabulary include acoustic tokens. Each acoustic token is selected from the vocabulary and represents acoustic properties of the audio signal 160. Examples of acoustic properties represented by the acoustic tokens can include reverberation, distortion, speaker identity, and background noise. Any appropriate set of acoustic tokens may be used. For example, an acoustic token can represent one of a plurality of code vectors in a codebook for a quantizer, e.g., a codebook for a vector quantizer included in a residual (i.e. , multi-stage) vector quantizer (RVQ). For example, the set of acoustic tokens may be provided using the codebook of an audio codec such as a Soundstream neural audio codec.

[0097] Throughout this specification, a ‘'residual vector quantizer” (RVQ) can refer to a multi-stage vector quantization technique that is based on a sequence of (residual) vector quantizers. A vector quantizer can quantize an input vector, e.g., by identifying a code vector from a codebook of code vectors associated with the vector quantizer, e.g., that has a smallest distance from the input vector, e.g., according to a distance metric (e.g., based on an LI norm). The residual vector quantizer can quantize an input vector (or '‘signal”) by iteratively quantizing the residual errors from previous quantization stages. Thus each stage in a residual vector quantizer encodes the difference (or residual) between the original signal and the reconstructed signal from the previous stage, thereby progressively refining the approximation of the original signal with each step.

[0098] In this example, the neural audio codec can include a hierarchy of multiple vector quantizers that each generate a respective acoustic token from a corresponding codebook of token vectors for the vector quantizer. The hierarchy includes one or more coarse vector quantizers at one or more first levels in the hierarchy and one or more fine vector quantizers at one or more last levels in the hierarchy. The output tokens can include, for each vector quantizer, a respective acoustic token selected from the codebook for the vector quantizer.

[0099] For example, the hierarchy can include Q vector quantizers arranged in the order of 1...Q', Qr+ 1) ... Q. and the vector quantizers 1... Q’ can be coarse vector quantizers, and the vector quantizers (Q' + 1) ... Q can be fine vector quantizers. The coarse vector quantizers generate coarse acoustic tokens, or acoustic tokens for coarse vector quantizers, that can represent acoustic properties such as speaker identity and recording conditions. The fine vector quantizers generate fine acoustic tokens, or acoustic tokens for fine vector quantizers, that can represent fine acoustic details. For example, fine acoustic tokens can be used to remove lossy compression artifacts in the coarse acoustic tokens.

[0100] In some examples, the output tokens in the vocabulary include acoustic and semantic tokens. The sequence of output tokens 132 can thus include semantic tokens, acoustic tokens, or both. For example, the sequence of output tokens 132 can include interleaved semantic tokens and acoustic tokens. In some examples, the token decoder neural network 130 can use the semantic tokens to guide the decoding process. For example, the token decoder neural network 130 can generate one or more acoustic tokens conditioned on one or more semantic tokens. An example token decoder neural network 130 is described below with reference to FIG. 6.

[0101] The system 100 processes the sequence of output tokens 132 using an audio decoder neural network 140 to generate the audio signal 160. For example, the respective audio sample at each of multiple output time steps spanning the time w indow of the audio signal 160 can be based on one or more output tokens of the sequence of output tokens 132.

[0102] The audio signal 160 represents speech for the text input 102a spoken by. that is, sounds like it is spoken by, the particular speaker of the input audio signal 102b. In the example of FIG. 1, the audio signal 160 includes “Hello world,'’ spoken by the particular speaker of the input audio signal 102b.

[0103] In examples where the output tokens include acoustic tokens, the audio decoder neural network 140 is configured to reconstruct an audio signal by processing acoustic tokens representing the audio signal. For example, the audio decoder neural network 140 can include the decoder of the Soundstream neural audio codec, as described in Zeghidour, Neil, et al. "Soundstream: An end-to-end neural audio codec." IEEE / ACM Transactions on Audio, Speech, and Language Processing 30 (2021): 495-507.

[0104] In some examples where the output tokens include acoustic tokens and semantic tokens, the system 100 is configured to extract the acoustic tokens and provide the acoustic tokens as input to the audio decoder neural network 140. The audio decoder neural network 140 is configured to reconstruct an audio signal by processing the acoustic tokens as described above.

[0105] In examples where the output tokens include semantic tokens, the system 100 is configured to generate acoustic tokens from the semantic tokens and provide the acoustic tokens as input to the audio decoder neural network 140. For example, the system 100 can use one or more generative neural networks to convert semantic tokens to acoustic tokens. Example generative neural netw orks for converting semantic tokens to acoustic tokens are described in Z. Borsos et al., “AudioLM: a Language Modeling Approach to Audio Generation,’" arXiv:2209.03143, which is hereby incorporated by reference in its entirety. The audio decoder neural network 140 is configured to reconstruct an audio signal by processing the acoustic tokens as described above.

[0106] FIG. 2 is a block diagram of the example audio generation system 100 described with reference to FIG. 1. In particular, the audio generation system 100 generates an audio signal 260 from a context input 201. The context input 201 is an example of the context input 101 described above with reference to FIG. 1. The context input 201 includes text input 202a, denoted as "input transcript,” and an input audio signal 202b. In the example of FIG. 2. the input audio signal 202b represents speech spoken by a particular speaker. The text input 202a includes a natural language sequence of text.

[0107] The system processes the text input 202a and the input audio signal 202b using the encoder neural network 110a and the encoder neural network 110b, respectively.

[0108] The encoder neural network 110a is configured to generate a representation 112a for the text input 202a. For example, the encoder neural network 1 10a can tokenize the text input 202a to generate a sequence of text tokens that each represent text of the text input 202a. Each of the text tokens can be selected from a vocabulary of text tokens that represent one or more characters, word pieces, or other text symbols. The encoder neural network 110a can map each text token to corresponding numerical values in accordance with a predefined mapping. Thus the representation 112a includes a sequence of continuous embeddings, each corresponding to a text token. As an example, the predefined mapping can be represented as an embedding matrix that has learned values. In other examples, the predefined mapping can be represented as a different learned module such as a neural network.

[0109] The encoder neural network 110b is configured to generate a representation 112b for the input audio signal 202b. For example, the encoder neural network 110b can generate a sequence of embeddings representing the input audio signal 202b. For example, the encoder neural network 110b can use an audio encoder that downsamples the input audio signal 202b to generate the embeddings. For example, the encoder neural network 110b can use multiple convolutional layers with increasing strides to generate the embeddings. In some examples, the encoder can include an encoder neural netw ork of a neural audio codec. As a particular example, the encoder neural network 110b can use the encoder neural network of a Soundstream neural audio codec to generate the sequence of embeddings.

[0110] In some examples, the encoder neural network 110b can process the sequence of embeddings to generate the representation 112b by further compressing the sequence of embeddings. For example, the encoder neural network 110b can use a prompt encoder that generates a sequence of compressed embeddings representing the sequence of embeddings, where the compressed embeddings are of lower dimensionality than the embeddings. As an example, the prompt encoder can apply a predetermined linear transformation to the embeddings in order to project the embeddings into a lower-dimensional space. Thus the representation 112b includes a sequence of continuous compressed embeddings representing the input audio signal 202b. The system 100 processes the representations 112a and 112b using the shared encoder 120 to generate the encoded representation 122. The encoded representation 122 includes, for example, a vector representing the inputs 202a and 202b.

[0111] In the example of FIG. 2, the shared encoder 120 concatenates the representation 112b and the representation 112a. For example, the shared encoder 120 can concatenate the representation 112b and the representation 112a along the temporal dimension.

[0112] In the example of FIG. 2, the shared encoder 120 includes a shared encoder neural network 220. The shared encoder neural network 220 is configured to process the concatenation to generate the encoded representation 122.

[0113] In some implementations, the shared encoder neural network 220 can be a selfattention neural network. For example, the shared encoder neural network 220 can include one or more attention layers. Each attention layer is configured to receive an input sequence for the layer comprising a respective layer input at each of one or more positions, and thereafter generate an attended input sequence at least in part by applying an attention mechanism to the input sequence for the layer. The attended input sequence includes a respective attended layer input at each of the one or more positions. The shared encoder neural network 220 can also include other layers, e g., fully -connected layers, embedding layers, and activation layers, either in place of or in addition to the attention layers. Examples of configurations of self-attention neural networks and the specifics of the other components of self-attention neural networks, e.g., embedding layers that embed inputs to the encoder, are described in more detail in Vaswani, et al.. Attention Is All You Need, arXiv: 1706.03762, Raffel, et al., and Devlin et al, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, arXiv:1810.04805, the entire contents of which are hereby- incorporated by reference herein in their entirety-.

[0114] The system processes the encoded representation 122 using the token decoder neural network 130 to generate the sequence of output tokens 132. As described with reference to FIG. 1 and FIG. 6, the token decoder neural network 130 can apply cross-attention to the encoded representation 122 and generate the sequence of output tokens 132 autoregressively.

[0115] The system 100 processes the sequence of output tokens 132 using the audio decoder neural network 140 to generate the audio signal 260. The audio signal 260 represents speech for the text input 202a, spoken by the particular speaker of the input audio signal 202b.

[0116] FIG. 3 is a block diagram of the example audio generation system 100 described with reference to FIG. 1. In particular, the audio generation system 100 generates an audio signal 360 from a context input that includes a text input 302a, an input audio signal 302b, and other inputs 302c-d. In some examples, the context input can include additional different combinations of inputs 302a-n. The context input is an example of the context input 101 described above with reference to FIG. 1.

[0117] The context input includes text input 302a (shown in FIG. 3 as ‘'translated transcript”), which includes the Spanish text “El riesgo de incendio en un auto rojo en comparacion con no estar en un auto rojo es casi una diferencia de diez veces. Es bastante cercano. Uh, para usar una de las frases favoritas de mi hermano, es casi una diferencia de orden de magnitud. Asi de significative. ” In other examples, the text input 302a can include text of other languages.

[0118] The context input also includes the input audio signal 302b. In the example of FIG. 3, the input audio signal 302b represents speech in English by a particular speaker. In other examples, the input audio signal 302b can represent speech in other languages. In some examples, the input audio signal 302b includes at least a portion of a source audio signal 305 described below.

[0119] In some examples, the context input also includes data representing energy features 302c of an audio signal, e.g., the source audio signal 305. Data representing energy features can include, for example, a time series of energy features that represent local energy information for the audio signal at different scales. The data representing energy features 302c can have a shape of [Ti, Di] where Ti is the number of frames (number of slices or portions of the audio signal), and Di is the dimensionality of the energy features.

[0120] In some examples, the context input also includes data representing pitch features 302d of an audio signal, e.g., the source audio signal 305. Data representing pitch features can include, for example, a time series of pitch features that represent local pitch histograms over time. The data representing pitch features 302d can have a shape of [Ti, Di] where Ti is the number of frames, and Di is the dimensionality of the pitch features.

[0121] In some examples, the context input can include other inputs that convey audio characteristics, e.g., prosody. As an example, other inputs can include data representing spectral features of an audio signal, e.g., the source audio signal 305. Data representing spectral features can include, for example, a time series of spectral features, e.g., mel- frequency ceptral coefficients (MFCCs), that represent local spectral envelope over time. The data representing spectral features can have a shape of [Ti, Di] where Ti is the number of frames, and Di is the dimensionality of the spectral features.

[0122] In some examples, other inputs can include speaker embeddings of an audio signal. e.g., the source audio signal 305. Speaker embeddings can include, for example, a time series of speaker embeddings that represent local timbre characteristics. In some examples, the speaker embeddings represent local timbre characteristics for multiple different speakers. The speaker embeddings can have a shape of [Ti, Di] where Ti is the number of frames, and Di is the dimensionality of the speaker embeddings.

[0123] In some examples, the system 100 receives the one or more of the inputs of the context input, e.g., from a user. In some examples, the system 100 generates one or more of the inputs of the context input from a source input, e.g.. from the source audio signal 305. The system 100 can receive the source audio signal 305 from a user.

[0124] The source audio signal 305 represents speech by a particular speaker for the English text “fire risk in a red car versus not in a red car is almost a tenfold difference. It's pretty dam close. Uh to use one of my brother's favorite phrases, it's almost an order of magnitude difference. That is how significant.” Note that the translated transcript 302a is a second language (i.e. Spanish) translation of first language (i.e. English) text (here called the automatic speech recognition (ASR) transcript) corresponding to the source audio signal 305.

[0125] The system 100 can generate one or more inputs of the context input using feature extraction engines 150. Each feature extraction engine 150a-n is configured to generate a corresponding type of input from a source input. In some examples, the feature extraction engines 150 can perform operations such as smoothing.

[0126] For example, the system can use the feature extraction engine 150a to generate the text input 302a from the source audio signal 305. As an example, the feature extraction engine 150a includes one or more neural networks configured to perform automatic speech translation. For example, the system 100 can provide the source audio signal 305 as input to the feature extraction engine 150a to obtain the text input 302a.

[0127] In the example of FIG. 3, the feature extraction engine 150a includes a first neural network configured to perform automatic speech recognition (ASR) to generate the ASR transcript, followed by a second neural network configured to perform machine translation (MT) to generate the text input 302a. For example, the feature extraction engine 150a can perform automatic speech recognition using a Universal Speech Model (USM), described in further detail in Zhang, Yu, et al., “Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages”, arXiv prePrint: arXiv:2303.01037 (2023). As another example, the system can perform machine translation using a neural network such as Gemini, described in Gemini Team, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023). As another example, the feature extraction engine 150a can include a neural network configured to perform automatic speech to text translation. An example neural network is described in Rubenstein, Paul, et al., AudioPaLM: A Large Language Model That Can Speak and Listen, arXiv prePrint: arXiv:2306. 12925 (2023).

[0128] In some examples, the system can use the feature extraction engine 150b to generate the input audio signal 302b. For example, the feature extraction engine 150b can be configured to sample at least a portion of the source audio signal 305 to generate the input audio signal 302b. As an example, the feature extraction engine 150b can extract the segment representing speech for “is almost a tenfold difference” from the source audio signal 305.

[0129] In some examples, the system 100 can use the source audio signal 305 as the input audio signal 302b.

[0130] The system can use the feature extraction engine 150c to generate the data representing energy features 302c from the source audio signal 305. As an example, the feature extraction engine 150c can be configured to generate a vector that includes energy features for each frame of the source audio signal 305.

[0131] The system can use the feature extraction engine 150d to generate the data representing pitch features 302d from the source audio signal 305. As an example, the feature extraction engine 150d can be configured to generate a vector that includes pitch features for each frame of the source audio signal 305.

[0132] In examples where the context input includes other inputs such as data representing spectral features or speaker embeddings, the system 100 can use appropriate feature extraction engines to generate the inputs from the source audio signal 305. For example, a feature extraction engine for data representing spectral features can be configured to generate a vector that includes MFCCs for each frame of the source audio signal 305. A feature extraction engine for speaker embeddings can be configured to generate a vector that includes speaker embeddings for each frame of the source audio signal 305.

[0133] In the example of FIG. 3, the generated audio signal 360 represents the text input 302a spoken by the particular speaker of the input audio signal 302b. That is, the generated audio signal 360 represents the Spanish text “El riesgo de incendio en un auto rojo en comparacion con no estar en un auto rojo es casi una diferencia de diez veces. Es bastante cercano. Uh, para usar una de las frases favoritas de mi hermano, es casi una diferencia de orden de magnitud. Asi de significative ”, spoken by the speaker of the English speech of the input audio signal 302b. Furthermore, in some examples, the generated audio signal 360 is time synchronous with the source audio signal 305. The generated audio signal 360 has the same overall duration as the source audio signal 305. The speech of the generated audio signal 360 retains characteristics of the source audio signal 305, such as speaker, style. acoustic environment, or prosodic properties of the source audio signal 305, e.g., overall duration, pauses, pitch profile, etc.

[0134] To generate the audio signal 360, the system 100 processes each of the inputs 302a-d, and other inputs of the context input, using a corresponding encoder neural network to generate representations 112. For example, the system 100 processes the text input 302a using the encoder 110a described with reference to FIG. 2. The system 100 processes the input audio signal 302b using the encoder 110b described with reference to FIG. 2.

[0135] The system 1 0 processes the data representing energy features 302c using an encoder 110c. The encoder 110c is configured to generate a representation of the data representing energy features 302c.

[0136] The system 100 processes the data representing pitch features 302d using an encoder 1 lOd. The encoder 1 lOd is configured to generate a representation of the data representing pitch features 302d.

[0137] A further encoder may be used to generate a representation of a video (e.g. without a soundtrack) which can be one of the additional inputs (e.g., one of inputs 302a-n This encoder operates on low-level features (e.g. edges or pixel attnbutes for images) encoded in the pixel intensity data of frames of the video.

[0138] In some implementations, the system 100 can combine two or more of the respective representations 112 to generate a combined representation. For example, the system can combine the representation of the data representing energy features 302c and the representation of the data representing pitch features 302d to generate a combined representation for pitch and energy', replacing the representation of the data representing energy features 302c and the representation of the data representing pitch features 302d.

[0139] As described above with reference to FIGS. 1-2. the system 100 processes the representations 112 using the shared encoder 120 to generate the encoded representation 122. The system 100 processes the encoded representation 122 using the token decoder neural network 130 to generate the sequence of output tokens 132. The system 100 processes the sequence of output tokens 132 using the audio decoder neural network 140 to generate the audio signal 360.

[0140] In some examples, the system 100 can generate a training example for training a speech processing model using the generated audio signal 360 and the source audio signal 305. For example, the system 100 can include the source audio signal 305, speech representing “fire risk in a red car versus not in a red car is almost a tenfold difference. It’s pretty' dam close. Uh to use one of my brother’s favorite phrases, it’s almost an order of magnitude difference. That is how significant”, spoken by the particular speaker, as the training input. The system 100 can include the generated audio signal 360, speech representing "El nesgo de incendio en un auto rojo en comparacion con no estar en un auto rojo es casi una diferencia de diez veces. Es bastante cercano. Uh, para usar una de las frases favoritas de mi hermano, es casi una diferencia de orden de magnitud. Asi de significative”, spoken by the particular speaker, as the training output.

[0141] Alternatively or in addition, the system 100 can generate a training example that includes the source audio signal 305 as the training output and the generated audio signal 360 as the training input.

[0142] The system 100 can provide the training example(s) for training the speech processing model. The system 100 can thus be used to generate parallel training data which is typically infeasible or difficult to obtain in large quantities.

[0143] FIG. 4 is a flow diagram of an example process 400 for generating an audio signal given multiple inputs. For convenience, the process 400 will be described as being performed by a system. For example, a system, e.g., the audio generation system 100 of FIG. 1, appropriately configured in accordance with this specification, can perform the process 400.

[0144] The system obtains a context input for generating an audio signal (step 410). The context input includes multiple inputs. The inputs of the context input can include any two or more of: a text input, an input audio signal representing speech, data representing features of a video, data representing energy features of a source audio signal, data representing pitch features of a source audio signal, data representing spectral features of a source audio signal, or speaker embeddings of a source audio signal. In some examples, the inputs can be derived from two or more modalities of data, such as text, audio, video, etc. Example combinations of inputs are described below and above with reference to FIGS. 1-3.

[0145] In some examples, the system receives one or more of the inputs from a user. In some examples, the system generates one or more of the inputs from a source input such as a source audio signal or input video (e.g. an input video including a sound track).

[0146] The system processes each of the inputs using a corresponding encoder neural network to generate a respective representation of the input (step 420). Each corresponding encoder neural network can be configured to process the input to generate a respective representation of the input that includes a sequence of embeddings for the input. In some examples, the system can generate, for each respective representation, a fixed-length sequence of summarization tokens. The system processes the respective representations of the inputs using a shared encoder to generate an encoded representation (step 430). For example, the system can combine, e.g., concatenate, the respective representations of the inputs. In some examples, the system can use the concatenation as the encoded representation.

[0147] In some examples, the system can process the concatenation using a shared encoder neural network to generate the encoded representation. In some implementations, the shared encoder neural network is a self-attention neural network.

[0148] In some examples where the system generates a fixed-length sequence of summarization tokens for each respective representation, the system can provide one or more sequences of embeddings and one or more summarization tokens as input to the shared encoder to generate updated summarization tokens. The encoded representation can include the updated summarization tokens.

[0149] The system processes the encoded representation using a token decoder neural network to generate a sequence of output tokens representing the audio signal (step 440). Example token decoder neural networks are described above with reference to FIGS. 1 and 6.

[0150] The system processes the sequence of output tokens representing the audio signal using an audio decoder neural network to generate the audio signal (step 450). Example audio decoder neural networks are described above with reference to FIG. 1.

[0151] The system can provide data representing the audio signal for presentation to a user. For example, the system can provide data representing the audio signal to a user device of the user and cause playback of the audio signal.

[0152] In some examples, the context input includes one or more inputs derived from a source audio signal. For example, the system can receive the source audio signal and generate the one or more inputs from the source audio signal. In these examples, the generated audio signal is time synchronous with the source audio signal (e.g., has substantially the same duration as the source audio signal). In some examples, the generated audio signal shares one or more prosodic properties with the source audio signal.

[0153] In some examples, the context input includes a text input with text to be spoken. In these examples, the generated audio signal represents speech for the text input.

[0154] In some examples, the context input includes a text input that includes multi-turn dialogue, that is, includes a sequence of one or more dialogue turns. In these examples, the generated audio signal represents speech in the context of the multi-turn dialogue.

[0155] For example, the context input can include a first text input that includes the multiturn dialogue, a second text input that includes a response to a last dialogue turn of the sequence, one or more input audio signals that each represent speech of one of the dialogue turns, and an input audio signal that represents speech by a particular speaker. The system can generate the audio signal from the multiple inputs as described above. The generated audio signal represents speech for the second text input, spoken by the particular speaker.

[0156] In examples where the multi-turn dialogue includes multiple speakers, the system learns to perfonn dubbing while preserving speaker turns. For example, for a first speaker, the second text input can represent speech to be spoken by the first speaker, and the input audio signal can represent speech by the first speaker. The system can generate the audio signal for the first speaker given the context input. For a second speaker, the second text input can represent speech to be spoken by the second speaker, and the input audio signal can represent speech by the second speaker. The system can generate the audio signal for the second speaker given the context input. In some examples, the one or more input audio signals that each represent speech of one of the dialogue turns for generating the audio signal for the second speaker can include the output audio signal for the first speaker. The first text input for generating the audio signal for the second speaker can include the second text input for generating the audio signal for the first speaker.

[0157] In some implementations, the system generates the first text input and the one or more input audio signals that each represent speech of one of the dialogue turns. For example, the system can generate the inputs from a source dialogue signal using a machine learning model such as a speaker change detection model or a voice activity detection model.

[0158] In some examples, the context input includes one or more inputs that each include data representing features of a video. A video includes multiple frames, and in some examples, audio. Each frame includes multiple pixels that each have one or more intensity values, e.g., that includes RGB color values or other color values in another colorization scheme for each pixel of the frame.

[0159] In some examples, the inputs can be derived from an input video. In these examples, the generated audio signal is time synchronous with the input video (e.g., has substantially the same duration as the input video).

[0160] In some implementations, the system generates the one or more inputs derived from the input video. For example, the system can use one or more image processing models or video processing models to process the input video. The inputs can represent features of the input video such as visual properties, e.g., content or motion.

[0161] For example, the context input can include a text input and data representing an input video depicting an environment. The data representing the video can represent properties of the environment. The system can generate the audio signal from the multiple inputs as described above. The generated audio signal represents the text input spoken in the environment.

[0162] As another example, the input video can depict one or more characteristics of a speaker. The context input can include data representing the input video that represents characteristics of the speaker, such as gender, emotion, regional accent, level of reverberation, etc. The system can generate the audio signal from the multiple inputs as described above. The generated audio signal represents the text input spoken with the characteristics of the speaker of the input video.

[0163] As another example, the input video can depict movement such as lip movement of the speaker. The context input can include data representing the input video that represents lip movement of the speaker. The system can generate the audio signal from the multiple inputs as described above. The generated audio signal represents the text input spoken to align with the lip movements of the speaker of the input video.

[0164] The components of the system, e.g., the encoder neural networks, shared encoder, token decoder neural network, or audio decoder neural network, can be trained by a training system of the system or another training system on training data. Training components of the system is described below in further detail with reference to FIG. 5.

[0165] FIG. 5 shows an example process 500 for training an example system for generating audio. For convenience, the process 500 will be described as being performed by a system. For example, a system, e.g., a training system of the audio generation system 100 of FIG. 1, appropriately configured in accordance with this specification, can perform the process 500 to train the audio generation system 100.

[0166] The audio generation system, as described above, can include components such as encoder neural networks 1 lOa-n, a shared encoder 120, a token decoder neural network 130, and an audio decoder neural network 140.

[0167] The training system can train one or more of the components on training data. The training data can include, for example, inputs, representations of inputs, or both, and groundtruth output audio signals, also referred to as target audio signals.

[0168] In some examples, the training system can generate inputs from source inputs. In some examples, the training system can generate representations of inputs using pre-trained encoder neural networks.

[0169] In some examples, the training system can generate ground-truth sequences of output tokens from ground-truth output audio signals. In some examples, the training system can generate the ground-truth sequence of output tokens to include semantic tokens. For example, the training system can provide the ground-truth output audio signal as input to a semantic tokenizer to generate the semantic tokens. The semantic tokenizer can include an audio representation neural network that has been trained to generate representations of input audio. For example, the audio representation neural network can be a self-attention based model, e.g., a Transfomier-based model or a Conformer-based model, e.g., a W2v-BERT neural network (described in Chung. Yu-An, et al. "W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training.” 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)). As an example, the audio representation neural network can be trained on a masked language modeling loss or a combination of a masked language modeling loss and a contrastive loss.

[0170] The semantic tokenizer can generate the semantic tokens based on outputs of one or more layers, e.g., of one of the intermediate layers, of an audio representation neural network. For example, the semantic tokenizer can generate the semantic tokens by processing the ground-truth output audio signal using the audio representation neural network. The outputs of the one or more layers of the audio representation neural network can include an embedding of the ground-truth output audio signal for each of multiple time steps of the ground-truth output audio signal. The semantic tokenizer can generate the semantic tokens by assigning each embedding for the ground-truth output audio signal to the closest semantic token, e.g.. an embedding distance in the embedding space, of a set of semantic tokens. The set of semantic tokens can include the centroids of K clusters of embeddings for an intermediate layer of the audio representation neural network for a set of training audio samples.

[0171] In some examples, the training system can generate the ground-truth sequence of output tokens to include acoustic tokens. For example, the system can generate a representation of the ground-truth output audio signal using an encoder, and quantize the representation to generate acoustic tokens. As an example, the encoder can include an encoder neural network of a neural audio codec, and the quantization can be a residual vector quantization as described in Zeghidour, Neil, et al. "Soundstream: An end-to-end neural audio codec." IEEE / ACM Transactions on Audio, Speech, and Language Processing 30 (2021): 495-507.

[0172] In some examples, the training system can generate the ground-truth sequence of output tokens to include semantic tokens and acoustic tokens. In some examples, the groundtruth sequence of output tokens can include interleaved semantic and acoustic tokens. In some implementations, one or more of the corresponding encoder neural networks

[0173] I lOa-n or the audio decoder neural network 140 are trained from scratch.

[0174] In some implementations, one or more of the corresponding encoder neural networks or the audio decoder neural network 140 are pre-trained and frozen prior to training the shared encoder 120 and the token decoder neural network 130 (i.e., not updated during the training of the shared encoder 120 and the token decoder neural network).

[0175] In some implementations, one or more of the corresponding encoder neural networks

[0176] I I Oa-n or the audio decoder neural network 140 are pre-trained, and parameters for one or more of the corresponding encoder neural networks are updated during training of the shared encoder 120 and the token decoder neural network 130. For example, one or more of the corresponding encoder neural networks 1 lOa-n can be optimized along with the shared encoder neural network and the token decoder neural network 130 to reduce the total error on training data through back propagation or the chain rule.

[0177] In some examples, one or more of the corresponding encoder neural networks can include pre-trained components such as text tokenizers, text encoders, audio tokenizers, audio encoders, etc. In some examples, an encoder neural network can be pre-trained on unlabeled training data based on optimizing a self-supervised or unsupervised loss function to generate embeddings. In some examples, the encoder neural network can be the encoder of an autoencoder neural network corresponding to the type of input.

[0178] The training system can train the shared encoder neural network and the token decoder neural network 130 on multiple training examples that each include a training input and a corresponding target output. In some examples, each training input can include a set of inputs, or a set of representations of inputs. In some examples, the corresponding target output can include a corresponding ground-truth sequence of output tokens representing a target audio signal, or a target audio signal.

[0179] For example, the training system can train the shared encoder neural network, the token decoder neural network 130, or both, using a machine learning training technique, e.g., a gradient descent with backpropagation training technique that uses a suitable optimizer, e.g., stochastic gradient descent, RMSprop, Adam optimizer, or Adafactor optimizer, to optimize an objective function for a next token prediction task. For example, the system can train the shared encoder neural network, the token decoder neural network 130, or both, to optimize an objective function that, for each training example, measures an error (e.g., crossentropy error) between (i) the ground-truth sequence of output tokens specified by the training example and (ii) the output tokens generated by the token decoder neural network 130 for the training input specified by the training example.

[0180] For example, the training system can train the token decoder neural network 130 to maximize the probability of selecting the ground-truth next token. The training system can thus encourage the token decoder neural network to maximize the predicted probability for the ground-truth next token, and to minimize the predicted probability for other tokens.

[0181] The training system can thus determine updated values for the neural network parameters of the shared encoder neural network, the token decoder neural network 130, or both.

[0182] In some examples, the training system trains the system to generate audio signals with a particular combination of types of inputs. In some other examples, the training system can train the system to generate audio signals with different combinations of types of inputs.

[0183] In some examples, the training system can apply dropout so that different training inputs can include different sets of inputs, or different sets of representations of inputs. For example, the training system can apply dropout independently to all types of inputs so that a ground-truth sequence of output tokens can correspond to multiple training inputs, that include sets of different combinations of representations. As an example, a first training input can include a set that includes a representation of input text and a representation of an input audio signal. A second training input can include a set that includes a representation of input text, a representation of an input audio signal, and a representation for energy features. In some examples, the system can include an empty vector in each training input for representations of inputs that are dropped out. Thus the training system can train the audio generation system to perform inference on different combinations of inputs.

[0184] As an example, the system can train the shared encoder on multiple training examples. Each training example can include one or more representations of inputs for a target audio signal and (ii) a target output that comprises a ground-truth sequence of output tokens representing the target audio signal. The system can train the shared encoder, e.g., the shared encoder neural network of the shared encoder, by a machine learning technique, to optimize an objective function that, for each training example, measures an error between: (i) the target output specified by the training example, and (ii) a predicted target output generated by processing an encoded representation generated by the shared encoder for the training input using a token decoder neural network. As another example, the system can train the shared encoder and the token decoder neural network by a machine learning technique, to optimize an objective function that, for each training example, measures an error between: (i) the target output specified by the training example, and (ii) a predicted sequence of output tokens. Each training example can include (i) a training input that comprises one or more inputs for a target audio signal and (ii) a target output that comprises a ground-truth sequence of output tokens representing the target audio signal.

[0185] The system can generate the predicted sequence of output tokens for each training example by generating, for each training example, and for each of the one or more inputs, a representation of the input. For example, the system can generate the representation of the input using a corresponding encoder neural network that can be pre-trained and frozen prior to training the shared encoder and the token decoder neural network, or updated during training of the shared encoder and the token decoder neural network. The system can process, for each training example, the representations of the one or more inputs using the shared encoder to generate an encoded representation for the training input. The system can process the encoded representation using the token decoder neural network to generate the predicted sequence of output tokens representing the target audio signal.

[0186] In the example of FIG. 5, the target output for a training example can include a ground-truth output segment of audio, e.g., from a training source audio signal. The training input for the training example can include a segment of audio 502 immediately preceding the ground-truth output segment, a text transcription 504 of the ground-truth output segment, and, in some examples, other inputs 506 derived from the ground-truth output segment.

[0187] The audio generation system processes the segment of audio 502 using the encoder neural network 110a. In the example of FIG. 5, the encoder neural network 110a includes an audio tokenizer and a prompt encoder. For example, the audio tokenizer can be configured to generate semantic tokens, acoustic tokens, or both. Example processes for generating semantic tokens and acoustic tokens are described above. In the example of FIG. 5, the prompt encoder can be configured to generate an embedding for each token. For example, the prompt encoder can map each token to corresponding numerical values in accordance with a predefined mapping. In some examples, the predefined mapping can be represented as an embedding matrix that has learned values. In some examples, the predefined mapping can be represented as a different learned module such as a neural network.

[0188] The audio generation system processes the text transcription 504 using the encoder neural network 1 10b. In the example of FIG. 5, the encoder neural network 110b includes a text tokenizer and a text encoder. Examples of tokenizing and encoding text are described above with reference to FIG. 2.

[0189] In some examples, the audio generation system processes the other inputs 506 using other feature encoders 1 lOc-n. In the example of FIG. 5, the other inputs include energy features and pitch features determined using a feature extraction engine such as the feature extraction engines described above with reference to FIG. 3.

[0190] The audio generation system processes the representations generated by the encoder neural networks 11 Oa-n using the shared encoder 120 to generate the encoded representation 122 as described above with reference to FIGS. 1-4. The audio generation system processes the encoded representation 122 using the token decoder neural network 130 to generate the sequence of output tokens 132 as described above with reference to FIGS. 1-4.

[0191] The system can determine the error for the objective function using the output tokens 132 and tokens, e.g., semantic tokens, acoustic tokens, or both, representing the ground-truth output segment of audio.

[0192] In cases where parallel data is available, each training example can include a training input that includes a training text input that includes text in a second language, and data representing features (e g., an input audio signal, data representing energy features, etc.) of a training source audio signal representing speech in a first language, and a target output that includes a target audio signal representing speech in the second language. In some examples, the system can generate the training text input from the training source audio signal. For example, the system can perform automatic speech recognition on the training source audio signal to generate a transcript that includes the text of the speech in the first language. The system can perform machine translation on the transcript to generate text of the speech in the second language.

[0193] As an example, the training system can train the shared encoder neural network and token decoder neural network on training examples that include (i) a representation of a training text input, (ii) one or more representations of data representing features of a training source audio signal representing speech in a first language, and (iii) a ground-truth sequence of output tokens representing a target audio signal representing speech in a second language.

[0194] In cases where parallel data is not available, each training example can include a training input that includes a training text input that includes text in a first language, and data representing features (e.g., an input audio signal, data representing energy features, etc.) of the target audio signal representing speech in a second language, and a target output that includes the target audio signal. In these cases, the training system can limit the temporal resolution of the inputs such as data representing energy features to mitigate the potential mismatch between training and inference. For example, the training system can use a sample rate of 5Hz when generating the inputs, which can be lower than the sample rate at inference. As another example, the training system can apply augmentation to the inputs, such as masking, e.g., time masking or frequency masking, or smoothing, e.g., by applying a filter, so that the system leams to drive inference with slowly varying or noisy features.

[0195] In cases where parallel data is not available, the system can train the shared encoder neural network and token decoder neural network on training examples that include (i) a representation of a training text input, (ii) one or more representations of data representing features of a target audio signal representing speech in a second language, and (iii) a groundtruth sequence of output tokens representing the target audio signal.

[0196] As another example, for multi-turn dialogue generation, the training system can train the shared encoder neural network and token decoder neural network on training examples that include (i) a representation of a training first text input that includes a sequence of one or more dialogue turns, (ii) a representation of a training second text input that includes a response to a last dialogue turn of the sequence, (iii) one or more representations of one or more training source audio signals, each representing speech of one of the one or more dialogue turns, (iv) a representation of a training input audio signal representing speech by a particular speaker, and (v) a ground-truth sequence of output tokens representing a target audio signal representing the training second text input spoken by the particular speaker.

[0197] In some examples, the system can generate one or more of the training examples from a training source audio signal. The training source audio signal can include speech representing a multi-turn dialogue spoken by one or more speakers. In some examples, the system can generate a training example for each turn of the multi-turn dialogue. For example, for each turn of the multi-turn dialogue, the representation of a training first text input can include a representation of the text of any preceding dialogue turns, the representation of a training second text input can include a representation of the text of the turn, one or more representations of one or more training source audio signals can include representations of speech of any preceding dialogue turns, the representation of a training input audio signal representing speech can include a representation of a training input audio signal representing speech by the speaker of the turn, and the ground-truth sequence of output tokens can represent the turn of the training source audio signal.

[0198] In some examples, the training system can generate the training examples from a training source dialogue signal using a speaker change detection model, a voice activity detection model, or both. For example, the training system can process a training source dialogue signal that includes multiple dialogue turns using a speaker change detection model to identify points, e.g., timestamps, in the training source dialogue signal where the speaker of the speech changes. The training system can use the identified points to generate training source audio signals and transcripts of text for each dialogue turn. In some examples, the training system can process the training source dialogue signal using a voice activity detection model to identify portions of the training source dialogue signal that represent speech. The training system can use the identified portions to generate training source audio signals and transcripts of text for each dialogue turn.

[0199] After training, the training system or a different inference system deploys the audio generation system on one or more computing devices to perform inference as described above with reference to FIGS. 1-4. For example, the audio generation system can obtain a voice prompt audio signal representing speech by a particular speaker, a text input, and source audio with particular characteristics such as energy and pitch. In some examples, the voice prompt audio signal can be a portion of the source audio. The audio generation system can obtain the context input from the voice prompt audio signal, the text input, and the source audio. The audio generation system can generate an audio signal that represents speech for the text input, spoken by the particular speaker, with the particular characteristics of the source audio.

[0200] FIG. 6 is a diagram of an example token decoder neural network 130. The token decoder neural network 130 is configured to generate a sequence of output tokens conditioned on an encoded representation 122, as described above with reference to FIG. 1.

[0201] As a particular example, the token decoder neural network 130 can be configured to generate a multimodal sequence of acoustic tokens (depicted in gray) and semantic tokens (depicted in a dotted pattern).

[0202] The token decoder neural network 130 can include a first machine learning model, e.g., a temporal model 610, to generate a frame token 612a-n for each sequence frame of n frames of the output sequence. The example of FIG. 6 shows the frame tokens 612a-c.

[0203] A frame can be a portion of an output sequence and a frame token can represent multiple output tokens associated with the frame. In some examples, each frame can include a predetermined number of semantic tokens and a predetermined number of acoustic tokens, for example, one semantic token followed by multiple acoustic tokens. In some examples, a portion of an output sequence can have any appropriate number of output tokens, e.g., tokens associated with a time period, a percentage of an output sequence, a fixed number of output tokens, etc.

[0204] The first machine learning model can generate frame tokens autoregressively. In some examples, the first machine learning model can have a Transformer-based architecture.

[0205] For example, the first machine learning model 610 can generate each frame token conditioned on at least the encoded representation 122. In some examples, the first machine learning model 610 can generate each frame token further conditioned on a current input sequence of any previously generated combined embeddings 608a-n. Alternatively or in addition, the first machine learning model 610 can generate each frame token conditioned on a current input sequence of any previously generated frame tokens.

[0206] In some examples, one or more of the previously generated combined embeddings can have been generated from previously generated output tokens in the output sequence. For example, the first machine learning model 610 can generate the first frame token conditioned on the encoded representation 122, and the second frame token conditioned on the encoded representation 122 and the combined embedding for the output tokens of the first frame generated using the first frame token.

[0207] The example of FIG. 6 shows combined embeddings 608a-c. In the example of FIG. 6, the first machine learning model 610 can generate the frame token 612b conditioned on the combined embedding 608a and the encoded representation 122.

[0208] Each of the combined embeddings 608a-n can represent multiple output tokens associated with a frame. For example, the system can generate the combined embedding 608a by generating an embedding for each token in frame A, and combining the embeddings, e.g., through mean pooling. In some examples, the system can generate each of the combined embeddings using a machine learning model configured to process an input sequence of one or more output tokens and generate a combined embedding.

[0209] The token decoder neural network 130 can include one or more second machine learning models, e.g., a depth model 620, to generate output tokens within each frame. For example, each second machine learning model is configured to receive a frame token as input and generate one or more output tokens based on the frame token. In some examples, the token decoder neural network 130 can use multiple second machine learning models to generate tokens within a frame. For example, the token decoder neural network 130 can include a second machine learning model for generating semantic tokens, and a second machine learning model for generating acoustic tokens. In some examples, one or more of the second machine learning models can generate output tokens autoregressively. In some examples, one or more of the second machine learning models can generate output tokens non-autoregressively. The second machine learning models can have any of a variety of architectures. In some examples, one or more of the second machine learning models can have a Transformer-based architecture, or a multilayer perceptron (MLP) architecture.

[0210] For example, the second machine learning model 620 can generate each output token conditioned on one or more previously generated output tokens of the sequence. For example, the second machine learning model 620 can generate each output token for a current frame conditioned on a current input sequence of previously generated output tokens for the current frame. In the example of FIG. 6. the second machine learning model 620 can generate each output token of the output tokens 622 for the frame C conditioned on the frame token 612c and any previously generated output tokens for the frame C.

[0211] In some examples, one or more second machine learning models are configured to generate tokens of a particular mode, e.g., acoustic or semantic tokens. In some examples, one or more second machine learning models are configured to generate tokens of multiple modes. As a particular example, one or more second machine learning models can be configured to generate a predetermined number of semantic tokens followed by a predetermined number of acoustic tokens.

[0212] In some examples, the first machine learning model 610 and the second machine learning models 620 can be configured to generate a sequence of output tokens by generating frame tokens autoregressively, and generating output tokens within frames autoregressively. For example, the first machine learning model 610 and the second machine learning models 620 can generate a sequence of output tokens according to: the number of output tokens associated with a frame, T is the number of frame tokens, xt,qis a q-th token associated with a t-th frame token, and a token ztis a t-th frame token.

[0213] As an example, to generate an audio signal, the system can obtain an encoded representation that is derived from multiple distinct inputs as described above. For example, the system can obtain a context input that includes multiple inputs. The system can obtain, for each input, a respective representation of the input. For example, the system can generate the respective representations using corresponding encoder neural networks as described above. The system can process the respective representations of the inputs using a shared encoder to generate the encoded representation.

[0214] For each frame in a sequence of frames corresponding to a sequence of output tokens representing the audio signal, where each frame corresponds to one or more output tokens of the sequence of output tokens, the system can generate, using the first machine learning model 610 and conditioned on at least the encoded representation 112, a frame token for the frame. In some examples, for one or more frames, e.g.. each frame after the first frame, the system can generate the frame token conditioned on the encoded representation and one or more combined embeddings. Each combined embedding can represent one or more output tokens corresponding to a respective preceding frame of the sequence of frames.

[0215] The system can generate, using the one or more second machine learning models 620, one or more output tokens corresponding to the frame. The system can process the one or more output tokens corresponding to each frame using an audio decoder neural network to generate the audio signal as described above.

[0216] In this specification, the term "configured" is used in relation to computing systems and environments, as well as computer program components. A computing system or environment is considered "configured" to perform specific operations or actions when it possesses the necessary' software, firmware, hardware, or a combination thereof, enabling it to carry out those operations or actions during operation. For instance, configuring a system might involve installing a software library with specific algorithms, updating firmware with new instructions for handling data, or adding a hardware component for enhanced processing capabilities. Similarly, one or more computer programs are "configured" to perform particular operations or actions when they contain instructions that, upon execution by a computing device or hardware, cause the device to perform those intended operations or actions.

[0217] The embodiments and functional operations described in this specification can be implemented in various forms, including digital electronic circuitry', software, firmware, computer hardware (encompassing the disclosed structures and their structural equivalents), or any combination thereof. The subject matter can be realized as one or more computer programs, essentially modules of computer program instructions encoded on a tangible non- transitory storage medium for execution by or to control the operation of a computing device or hardware. The storage medium can be a storage device such as a hard drive or solid-state drive (SSD), a storage medium, a random or serial access memory device, or a combination of these. Additionally or alternatively, the program instructions can be encoded on a transmitted signal, such as a machine-generated electrical, optical, or electromagnetic signal, designed to carry infonnation for transmission to a receiving device or system for execution by a computing device or hardware. Furthermore, implementations may leverage emerging technologies like quantum computing or neuromorphic computing for specific applications, and may be deployed in distributed or cloud-based environments where components reside on different machines or within a cloud infrastructure.

[0218] The term "computing device or hardware" refers to the physical components involved in data processing and encompasses all types of devices and machines used for this purpose. Examples include processors or processing units, computers, multiple processors or computers working together, graphics processing units (GPUs), tensor processing units (TPUs), and specialized processing hardware such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). In addition to hardware, a computing device or hardware may also include code that creates an execution environment for computer programs. This code can take the form of processor firmware, a protocol stack, a database management system, an operating system, or a combination of these elements. Embodiments may particularly benefit from utilizing the parallel processing capabilities of GPUs, in a General-Purpose computing on Graphics Processing Units (GPGPU) context, where code specifically designed for GPU execution, often called kernels or shaders, is employed. Similarly, TPUs excel at running optimized tensor operations crucial for many machine learning algorithms. By leveraging these accelerators and their specialized programming models, the system can achieve significant speedups and efficiency gains for tasks involving artificial intelligence and machine learning, particularly in areas such as computer vision, natural language processing, and robotics.

[0219] A computer program, also referred to as software, an application, a module, a script, code, or simply a program, can be written in any programming language, including compiled or interpreted languages, and declarative or procedural languages. It can be deployed in various forms, such as a standalone program, a module, a component, a subroutine, or any other unit suitable for use within a computing environment. A program may or may not correspond to a single file in a file system and can be stored in various ways. This includes being embedded within a file containing other programs or data (e g., scripts within a markup language document), residing in a dedicated file, or distributed across multiple coordinated files (e.g., files storing modules, subprograms, or code segments). A computer program can be executed on a single computer or across multiple computers, whether located at a single site or distributed across multiple sites and interconnected through a data communication network. The specific implementation of the computer programs may involve a combination of traditional programming languages and specialized languages or libraries designed for GPGPU programming or TPU utilization, depending on the chosen hardware platform and desired performance characteristics.

[0220] In this specification, the term "engine" broadly refers to a software-based system, subsystem, or process designed to perform one or more specific functions. An engine is typically implemented as one or more software modules or components installed on one or more computers, which can be located at a single site or distributed across multiple locations. In some instances, one or more dedicated computers may be used for a particular engine, while in other cases, multiple engines may operate concurrently on the same one or more computers. Examples of engine functions within the context of Al and machine learning could include data pre-processing and cleaning, feature engineering and extraction, model training and optimization, inference and prediction generation, and post-processing of results. The specific design and implementation of engines will depend on the overall architecture and the distribution of computational tasks across various hardware components, including CPUs, GPUs, TPUs, and other specialized processors.

[0221] The processes and logic flows described in this specification can be executed by one or more programmable computers running one or more computer programs to perform functions by operating on input data and generating output. Additionally, graphics processing units (GPUs) and tensor processing units (TPUs) can be utilized to enable concurrent execution of aspects of these processes and logic flows, significantly accelerating performance. This approach offers significant advantages for computationally intensive tasks often found in Al and machine learning applications, such as matrix multiplications, convolutions, and other operations that exhibit a high degree of parallelism. By leveraging the parallel processing capabilities of GPUs and TPUs, significant speedups and efficiency gains compared to relying solely on CPUs can be achieved. Alternatively or in combination with programmable computers and specialized processors, these processes and logic flows can also be implemented using specialized processing hardware, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs), for even greater performance or energy efficiency in specific use cases.

[0222] Computers capable of executing a computer program can be based on general-purpose microprocessors, special-purpose microprocessors, or a combination of both. They can also utilize any other type of central processing unit (CPU). Additionally, graphics processing units (GPUs), tensor processing units (TPUs), and other machine learning accelerators can be employed to enhance performance, particularly for tasks involving artificial intelligence and machine learning. These accelerators often work in conjunction with CPUs, handling specialized computations while the CPU manages overall system operations and other tasks. Typically, a CPU receives instructions and data from read-only memory (ROM), random access memory' (RAM), or both. The elements of a computer include a CPU for executing instructions and one or more memory devices for storing instructions and data. The specific configuration of processing units and memory will depend on factors like the complexity of the Al model, the volume of data being processed, and the desired performance and latency requirements. Embodiments can be implemented on a wide range of computing platforms, from small embedded devices with limited resources to large-scale data center systems with high-performance computing capabilities. The system may include storage devices like hard drives, SSDs, or flash memory for persistent data storage.

[0223] Computer-readable media suitable for storing computer program instructions and data encompass all forms of non-volatile memory', media, and memory' devices. Examples include semiconductor memory devices such as read-only memory (ROM), solid-state drives (SSDs), and flash memory devices; hard disk drives (HDDs); optical media; and optical discs such as CDs, DVDs, and Blu-ray discs. The specific type of computer-readable media used will depend on factors such as the size of the data, access speed requirements, cost considerations, and the desired level of portability or permanence.

[0224] To facilitate user interaction, embodiments of the subject matter described in this specification can be implemented on a computing device equipped with a display device, such as a liquid cry stal display (LCD) or an organic light-emitting diode (OLED) display, for presenting information to the user. Input can be provided by the user through various means, including a keyboard), touchscreens, voice commands, gesture recognition, or other input modalities depending on the specific device and application. Additional input methods can include acoustic, speech, or tactile input, while feedback to the user can take the form of visual, auditory', or tactile feedback. Furthermore, computers can interact with users by exchanging documents with a user's device or application. This can involve sending web content or data in response to requests or sending and receiving text messages or other forms of messages through mobile devices or messaging platforms. The selection of input and output modalities will depend on the specific application and the desired form of user interaction.

[0225] Machine learning models can be implemented and deployed using machine learning frameworks, such as TensorFlow or JAX. These frameworks offer comprehensive tools and libraries that facilitate the development, training, and deployment of machine learning models.

[0226] Embodiments of the subject matter described in this specification can be implemented within a computing system comprising one or more components, depending on the specific application and requirements. These may include a back-end component, such as a back-end server or cloud-based infrastructure; an optional middleware component, such as a middleware server or application programming interface (API), to facilitate communication and data exchange; and a front-end component, such as a client device with a user interface, a web browser, or an app, through which a user can interact with the implemented subject matter. For instance, the described functionality could be implemented solely on a client device (e.g., for on-device machine learning) or deployed as a combination of front-end and back-end components for more complex applications. These components, when present, can be interconnected using any form or medium of digital data communication, such as a communication network like a local area network (LAN) or a wide area network (WAN) including the Internet. The specific system architecture and choice of components will depend on factors such as the scale of the application, the need for real-time processing, data security requirements, and the desired user experience.

[0227] The computing system can include clients and servers that may be geographically separated and interact through a communication network. The specific type of network, such as a local area network (LAN), a wide area network (WAN), or the Internet, will depend on the reach and scale of the application. The client-server relationship is established through computer programs running on the respective computers and designed to communicate with each other using appropriate protocols. These protocols may include HTTP, TCP / IP, or other specialized protocols depending on the nature of the data being exchanged and the security requirements of the system. In certain embodiments, a server transmits data or instructions to a user's device, such as a computer, smartphone, or tablet, acting as a client. The client device can then process the received information, display results to the user, and potentially send data or feedback back to the server for further processing or storage. This allows for dynamic interactions between the user and the system, enabling a wide range of applications and functionalities.

[0228] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0229] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0230] In addition to the embodiments described above, the following embodiments are also innovative:

[0231] Embodiment 1 is a computer-implemented method for generating an audio signal, comprising: obtaining a context input for generating the audio signal, wherein the context input comprises a plurality of inputs; processing each of the inputs using a corresponding encoder neural network to generate a respective representation of the input; processing the respective representations of the inputs using a shared encoder to generate an encoded representation; processing the encoded representation using a token decoder neural network to generate a sequence of output tokens representing the audio signal; and processing the sequence of output tokens representing the audio signal using an audio decoder neural network to generate the audio signal.

[0232] Embodiment 2 is the method of embodiment 1, wherein the respective representation of the input comprises a fixed-length sequence of summarization tokens.

[0233] Embodiment 3 is the method of embodiment 2, wherein processing the respective representations of the inputs using a shared encoder comprises providing one or more sequences of embeddings and one or more summarization tokens as input to the shared encoder to generate updated summarization tokens. Embodiment 4 is the method of embodiment 1, wherein the corresponding encoder neural network is configured to process the input to generate the respective representation of the input, comprising a sequence of embeddings for the input.

[0234] Embodiment 5 is the method of any one of embodiments 1-4, wherein the shared encoder is configured to generate a concatenation of the respective representations of the inputs.

[0235] Embodiment 6 is the method of embodiment 5, wherein the shared encoder comprises a shared encoder neural network configured to process the concatenation to generate the encoded representation.

[0236] Embodiment 7 is the method of embodiment 6, wherein the shared encoder neural network is a self-attention neural network.

[0237] Embodiment 8 is the method of any one of embodiments 1-7, wherein the plurality of inputs are derived from two or more modalities of data.

[0238] Embodiment 9 is the method of any one of embodiments 1-8, wherein the plurality of inputs comprise one or more first inputs derived from a source audio signal, and the generated audio signal is time synchronous with the source audio signal.

[0239] Embodiment 10 is the method of embodiment 9, wherein the audio signal shares one or more prosodic properties with the source audio signal.

[0240] Embodiment 11 is the method of any one of embodiments 1-10, wherein the plurality of inputs comprise a text input, and the generated audio signal represents speech for the text input.

[0241] Embodiment 12 is the method of any one of embodiments 1-11, wherein the plurality of inputs comprise one or more second inputs derived from an input video, and the generated audio signal is time synchronous with the input video.

[0242] Embodiment 13 is the method of any one of embodiments 1-12, wherein the plurality of inputs comprise a text input comprising a multi-turn dialogue, and the generated audio signal represents speech in context with the multi-turn dialogue.

[0243] Embodiment 14 is the method of any one of embodiments 9-11, wherein obtaining a context input for generating an audio signal comprises: receiving the source audio signal; and generating, from the source audio signal, one or more first inputs.

[0244] Embodiment 15 is the method of embodiment 1, wherein the plurality of inputs comprise any two or more of: a text input, an input audio signal representing speech, data representing features of a video, data representing energy features of a source audio signal. data representing pitch features of a source audio signal, data representing spectral features of a source audio signal, or speaker embeddings of a source audio signal.

[0245] Embodiment 16 is the method of embodiment 1, wherein the plurality of inputs comprise (i) an input audio signal representing speech by a particular speaker and (ii) a text input, and the audio signal represents the text input spoken by the particular speaker.

[0246] Embodiment 17 is the method of embodiment 1. wherein the plurality of inputs comprise (i) an input audio signal representing speech in a first language by a particular speaker and (ii) a text input in a second language, and the audio signal represents the text input spoken by the particular speaker.

[0247] Embodiment 18 is the method of any one of embodiments 16-17, wherein the input audio signal comprises at least a portion of a source audio signal that represents speech in the first language.

[0248] Embodiment 19 is the method of embodiment 18, wherein obtaining a context input for generating an audio signal comprises generating the text input by processing the source audio signal using a sequence of one or more neural networks configured to perform automatic speech translation.

[0249] Embodiment 20 is the method of any one of embodiments 17-19, wherein the method further comprises generating a training example comprising the source audio signal and the audio signal.

[0250] Embodiment 21 is the method of embodiment 20, further comprising providing the training example for training a speech processing model.

[0251] Embodiment 22 is the method of any one of embodiments 16-21, wherein the plurality of inputs further comprise any one or more of: data representing energy features of the source audio signal, data representing pitch features of the source audio signal, data representing spectral features of the source audio signal, or speaker embeddings of the source audio signal.

[0252] Embodiment 23 is the method of any one of embodiments 17-22, wherein the shared encoder comprises a shared encoder neural network, and the shared encoder neural network and token decoder neural network are trained on a plurality of training examples, each comprising (i) a representation of a training text input, (ii) one or more representations of data representing features of a training source audio signal representing speech in a first language, and (iii) a ground-truth sequence of output tokens representing a target audio signal representing speech in a second language. Embodiment 24 is the method of any one of embodiments 17-22, wherein the shared encoder comprises a shared encoder neural network, and the shared encoder neural network and token decoder neural network are trained on a plurality of training examples, each comprising (i) a representation of a training text input, (ii) one or more representations of data representing features of a target audio signal representing speech in a second language, and (iii) a ground-truth sequence of output tokens representing the target audio signal.

[0253] Embodiment 25 is the method of embodiment 1. wherein the plurality of inputs comprise (i) a first text input that comprises a sequence of one or more dialogue turns, (ii) a second text input that comprises a response to a last dialogue turn of the sequence, (iii) one or more input audio signals, each representing speech of one of the one or more dialogue turns, and (iv) an input audio signal representing speech by a particular speaker, and the audio signal represents the second text input spoken by the particular speaker.

[0254] Embodiment 26 is the method of embodiment 25, wherein the shared encoder comprises a shared encoder neural network, and the shared encoder neural network and the token decoder neural network are trained on a plurality of training examples, each comprising (i) a representation of a training first text input comprising a sequence of one or more dialogue turns, (ii) a representation of a training second text input comprising a response to a last dialogue turn of the sequence, (iii) one or more representations of one or more training source audio signals, each representing speech of one of the one or more dialogue turns, (iv) a representation of a training input audio signal representing speech by a particular speaker, and (v) a ground-truth sequence of output tokens representing a target audio signal representing the training second text input spoken by the particular speaker.

[0255] Embodiment 27 is the method of embodiment 1 , wherein the plurality of inputs comprise (i) a text input, and (ii) data representing an input video depicting an environment, and the audio signal represents the text input spoken in the environment.

[0256] Embodiment 28 is the method of embodiment 27, wherein the input video further depicts one or more characteristics of a speaker, and the audio signal represents the text input spoken with the one or more characteristics.

[0257] Embodiment 29 is the method of any one of embodiments 27-28, wherein the input video further depicts one or more lip movements of a speaker, and the audio signal represents the text input spoken to align with the one or more lip movements.

[0258] Embodiment 30 is the method of any one of embodiments 1-29, wherein the shared encoder comprises a shared encoder neural network, and one or more of the corresponding encoder neural networks are pre-trained and frozen prior to training the shared encoder and the token decoder neural network.

[0259] Embodiment 31 is the method of any one of embodiments 1 -29, wherein the shared encoder comprises a shared encoder neural network, and one or more of the corresponding encoder neural networks are pre-trained, and wherein parameters for one or more of the corresponding encoder neural netw orks are updated during training of the shared encoder and the token decoder neural network.

[0260] Embodiment 32 is a computer-implemented method for generating an audio signal, comprising: obtaining an encoded representation, wherein the encoded representation is derived from a plurality of distinct inputs; for each frame in a sequence of frames corresponding to a sequence of output tokens representing the audio signal, wherein each frame corresponds to one or more output tokens of the sequence of output tokens: generating, using a first machine learning model and conditioned on at least the encoded representation, a frame token for the frame; and generating, using one or more second machine learning models and conditioned on the frame token, one or more output tokens corresponding to the frame; and processing the one or more output tokens corresponding to each frame using an audio decoder neural network to generate the audio signal.

[0261] Embodiment 33 is the method of embodiment 32, wherein generating, using a first machine learning model and conditioned on at least the encoded representation, a frame token for the frame comprises generating the frame token conditioned on the encoded representation and one or more combined embeddings, wherein each of the one or more combined embeddings represent one or more output tokens corresponding to a respective preceding frame of the sequence of frames.

[0262] Embodiment 34 is a computer-implemented method for generating an encoded representation configured for use in generating an audio signal, comprising: obtaining a context input, wherein the context input comprises a plurality of inputs; obtaining, for each input of the plurality7of inputs, a respective representation of the input; processing the respective representations of the inputs using a shared encoder to generate the encoded representation, comprising: concatenating the respective representations of the inputs; and processing the concatenation using a shared encoder neural network to generate the encoded representation.

[0263] Embodiment 35 is a computer-implemented method for training a shared encoder, comprising: training the shared encoder on a plurality of training examples, wherein each training example comprises (i) a training input that comprises one or more representations of inputs for a target audio signal and (ii) a target output that comprises a ground-truth sequence of output tokens representing the target audio signal, and wherein the method comprises training the shared encoder, by a machine learning technique, to optimize an objective function that, for each training example, measures an error between: (i) the target output specified by the training example, and (ii) a predicted target output generated by processing an encoded representation generated by the shared encoder for the training input using a token decoder neural network.

[0264] Embodiment 36 is a computer-implemented method for training one or more neural networks for generating an audio signal, comprising: obtaining a plurality of training examples, wherein each training example comprises (i) a training input that comprises one or more inputs for a target audio signal and (ii) a target output that comprises a ground-truth sequence of output tokens representing the target audio signal: generating, for each training example, and for each of the one or more inputs, a representation of the input; processing, for each training example, the representations of the one or more inputs using a shared encoder to generate an encoded representation for the training input; processing the encoded representation using a token decoder neural network to generate a predicted sequence of output tokens representing the target audio signal; and training the shared encoder and the token decoder neural network, by a machine learning technique, to optimize an objective function that, for each training example, measures an error between: (i) the target output specified by the training example, and (ii) the predicted sequence of output tokens.

[0265] Embodiment 37 is the method of embodiment 36, wherein generating, for each training example, and for each of the one or more inputs, a representation of the input comprises generating the representation of the input using a corresponding encoder neural network.

[0266] Embodiment 38 is the method of embodiment 37, wherein the corresponding encoder neural network is pre-trained and frozen prior to training the shared encoder and the token decoder neural network.

[0267] Embodiment 39 is the method of embodiment 37, wherein the corresponding encoder neural network is pre-trained, and wherein parameters for the corresponding encoder neural network are updated during training of the shared encoder and the token decoder neural network.

[0268] Embodiment 40 is a system comprising one or more computers and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers. cause the one or more computers to perform operations of the respective method of any one of embodiments 1-39.

[0269] Embodiment 41 is one or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations of the respective method of any of embodiments 1-39.

[0270] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0271] What is claimed is:

Claims

CLAIMS1. A computer-implemented method for generating an audio signal, comprising: obtaining a context input for generating the audio signal, wherein the context input comprises a plurality of inputs; processing each of the inputs using a corresponding encoder neural network to generate a respective representation of the input; processing the respective representations of the inputs using a shared encoder to generate an encoded representation; processing the encoded representation using a token decoder neural network to generate a sequence of output tokens representing the audio signal; and processing the sequence of output tokens representing the audio signal using an audio decoder neural network to generate the audio signal.

2. The method of claim 1, wherein the respective representation of the input comprises a fixed-length sequence of summarization tokens.

3. The method of claim 2, wherein processing the respective representations of the inputs using a shared encoder comprises providing one or more sequences of embeddings and one or more summarization tokens as input to the shared encoder to generate updated summarization tokens.

4. The method of claim 1, wherein the corresponding encoder neural network is configured to process the input to generate the respective representation of the input, comprising a sequence of embeddings for the input.

5. The method of any preceding claim, wherein the shared encoder is configured to generate a concatenation of the respective representations of the inputs.

6. The method of claim 5, wherein the shared encoder comprises a shared encoder neural network configured to process the concatenation to generate the encoded representation.

7. The method of claim 6, wherein the shared encoder neural network is a self-attention neural network.

8. The method of any preceding claim, wherein the plurality of inputs are derived from two or more modalities of data.

9. The method of any preceding claim, wherein the plurality of inputs comprise one or more first inputs derived from a source audio signal, and the generated audio signal is time synchronous with the source audio signal.

10. The method of claim 9, wherein the audio signal shares one or more prosodic properties with the source audio signal.

11. The method of any preceding claim, wherein the plurality of inputs comprise a text input, and the generated audio signal represents speech for the text input.

12. The method of any preceding claim, wherein the plurality of inputs comprise one or more second inputs derived from an input video, and the generated audio signal is time synchronous with the input video.

13. The method of any preceding claim, wherein the plurality of inputs comprise a text input comprising a multi-turn dialogue, and the generated audio signal represents speech in context with the multi-turn dialogue.

14. The method of any of claims 9-1 1, wherein obtaining a context input for generating an audio signal comprises: receiving the source audio signal; and generating, from the source audio signal, one or more first inputs.

15. The method of claim 1 , wherein the plurality’ of inputs comprise any two or more of: a text input, an input audio signal representing speech, data representing features of a video, data representing energy features of a source audio signal, data representing pitch features of a source audio signal, data representing spectral features of a source audio signal, or speaker embeddings of a source audio signal.

16. The method of claim 1, wherein the plurality of inputs comprise (i) an input audio signal representing speech by a particular speaker and (ii) a text input, and the audio signal represents the text input spoken by the particular speaker.

17. The method of claim 1, wherein the plurality of inputs comprise (i) an input audio signal representing speech in a first language by a particular speaker and (ii) a text input in a second language, and the audio signal represents the text input spoken by the particular speaker.

18. The method of any of claims 16-17, wherein the input audio signal comprises at least a portion of a source audio signal that represents speech in the first language.

19. The method of claim 18, wherein obtaining a context input for generating an audio signal comprises generating the text input by processing the source audio signal using a sequence of one or more neural networks configured to perform automatic speech translation.

20. The method of any of claims 17-19, wherein the method further comprises generating a training example comprising the source audio signal and the audio signal.

21. The method of claim 20, further comprising providing the training example for training a speech processing model.

22. The method of any of claims 16-21, wherein the plurality of inputs further comprise any one or more of: data representing energy features of the source audio signal, data representing pitch features of the source audio signal, data representing spectral features of the source audio signal, or speaker embeddings of the source audio signal.

23. The method of any of claims 17-22, wherein the shared encoder comprises a shared encoder neural network, and the shared encoder neural network and token decoder neural network are trained on a plurality7of training examples, each comprising (i) a representation of a training text input, (ii) one or more representations of data representing features of a training source audio signal representing speech in a first language, and (hi) a ground-truth sequence of output tokens representing a target audio signal representing speech in a second language.

24. The method of any of claims 17-22, wherein the shared encoder comprises a shared encoder neural network, and the shared encoder neural network and token decoder neural network are trained on a plurality of training examples, each comprising (i) a representation of a training text input, (ii) one or more representations of data representing features of a target audio signal representing speech in a second language, and (iii) a ground-truth sequence of output tokens representing the target audio signal.

25. The method of claim 1, wherein the plurality of inputs comprise (i) a first text input that comprises a sequence of one or more dialogue turns, (ii) a second text input that comprises a response to a last dialogue turn of the sequence, (iii) one or more input audio signals, each representing speech of one of the one or more dialogue turns, and (iv) an input audio signal representing speech by a particular speaker, and the audio signal represents the second text input spoken by the particular speaker.

26. The method of claim 25, wherein the shared encoder comprises a shared encoder neural network, and the shared encoder neural network and the token decoder neural network are trained on a plurality of training examples, each comprising (i) a representation of a training first text input comprising a sequence of one or more dialogue turns, (ii) a representation of a training second text input comprising a response to a last dialogue turn of the sequence, (iii) one or more representations of one or more training source audio signals, each representing speech of one of the one or more dialogue turns, (iv) a representation of a training input audio signal representing speech by a particular speaker, and (v) a ground-truth sequence of output tokens representing a target audio signal representing the training second text input spoken by the particular speaker.

27. The method of claim 1, wherein the plurality of inputs comprise (i) a text input, and (ii) data representing an input video depicting an environment, and the audio signal represents the text input spoken in the environment.

28. The method of claim 27, wherein the input video further depicts one or more characteristics of a speaker, and the audio signal represents the text input spoken with the one or more characteristics.

29. The method of any of claims 27-28, wherein the input video further depicts one or more lip movements of a speaker, and the audio signal represents the text input spoken to align with the one or more lip movements.

30. The method of any preceding claim, wherein the shared encoder comprises a shared encoder neural network, and one or more of the corresponding encoder neural networks are pre-trained and frozen prior to training the shared encoder and the token decoder neural network.

31. The method of any of claims 1 -29, wherein the shared encoder comprises a shared encoder neural network, and one or more of the cones ponding encoder neural networks are pre-trained, and wherein parameters for one or more of the corresponding encoder neural networks are updated during training of the shared encoder and the token decoder neural network.

32. A computer-implemented method for generating an audio signal, comprising: obtaining an encoded representation, wherein the encoded representation is derived from a plurality of distinct inputs; for each frame in a sequence of frames corresponding to a sequence of output tokens representing the audio signal, wherein each frame corresponds to one or more output tokens of the sequence of output tokens: generating, using a first machine learning model and conditioned on at least the encoded representation, a frame token for the frame; and generating, using one or more second machine learning models and conditioned on the frame token, one or more output tokens corresponding to the frame; and processing the one or more output tokens corresponding to each frame using an audio decoder neural network to generate the audio signal.

33. The method of claim 32, wherein generating, using a first machine learning model and conditioned on at least the encoded representation, a frame token for the frame comprises generating the frame token conditioned on the encoded representation and one or more combined embeddings, wherein each of the one or more combined embeddings represent one or more output tokens corresponding to a respective preceding frame of the sequence of frames.

34. A computer-implemented method for generating an encoded representation configured for use in generating an audio signal, comprising: obtaining a context input, wherein the context input comprises a plurality of inputs; obtaining, for each input of the plurality of inputs, a respective representation of the input; processing the respective representations of the inputs using a shared encoder to generate the encoded representation, comprising: concatenating the respective representations of the inputs; and processing the concatenation using a shared encoder neural network to generate the encoded representation.

35. A computer-implemented method for training a shared encoder, comprising: training the shared encoder on a plurality of training examples, wherein each training example comprises (i) a training input that comprises one or more representations of inputs for a target audio signal and (ii) a target output that comprises a ground-truth sequence of output tokens representing the target audio signal, and wherein the method comprises training the shared encoder, by a machine learning technique, to optimize an objective function that, for each training example, measures an error between: (i) the target output specified by the training example, and (ii) a predicted target output generated by processing an encoded representation generated by the shared encoder for the training input using a token decoder neural network.

36. A computer-implemented method for training one or more neural networks for generating an audio signal, comprising: obtaining a plurality of training examples, wherein each training example comprises (i) a training input that comprises one or more inputs for a target audio signal and (ii) a target output that comprises a ground-truth sequence of output tokens representing the target audio signal; generating, for each training example, and for each of the one or more inputs, a representation of the input; processing, for each training example, the representations of the one or more inputs using a shared encoder to generate an encoded representation for the training input; processing the encoded representation using a token decoder neural network to generate a predicted sequence of output tokens representing the target audio signal; and training the shared encoder and the token decoder neural network, by a machine learning technique, to optimize an objective function that, for each training example, measures an error between: (i) the target output specified by the training example, and (ii) the predicted sequence of output tokens.

37. The method of claim 36, wherein generating, for each training example, and for each of the one or more inputs, a representation of the input comprises generating the representation of the input using a corresponding encoder neural network.

38. The method of claim 37, wherein the corresponding encoder neural network is pretrained and frozen prior to training the shared encoder and the token decoder neural network.

39. The method of claim 37, wherein the corresponding encoder neural network is pretrained, and wherein parameters for the corresponding encoder neural network are updated during training of the shared encoder and the token decoder neural network.

40. A system comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of the respective method of any preceding claim.

41. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations of the respective method of any of claims 1-39.

Images