Audio generation using autoregressive generation neural network

Abstract

To provide a method, a system and a device, which include a computer program encoded on a computer storage medium for generating prediction of an audio signal.SOLUTION: A method according to one embodiment includes the steps of: receiving a request for generating an audio signal; acquiring semantic representation of the audio signal; generating acoustic representation of the audio signal with at least semantic representation as a condition by using one or more generation neural networks; and processing at least acoustic representation by using a decoder neural network and generating prediction of the audio signal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 63 / 404,528, filed September 7, 2022, and U.S. Provisional Application No. 63 / 441,412, filed January 26, 2023. The disclosures of the prior applications are considered part of the disclosure of this application and are incorporated by reference into the disclosure of this application. [Background technology]

[0002] This specification relates to generating audio using neural networks.

[0003] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a received input. Some neural networks contain one or more hidden layers in addition to an output layer. The output of each hidden layer is used as the input to the next layer in the network, either the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the input of the current value of its respective parameter set. Summary of the Invention

[0004] This specification describes a system, implemented as a computer program on one or more computers at one or more locations, that generates an audio signal using one or more generative neural networks.

[0005] In general, the output audio signal is an output audio example that includes samples of a sound wave at each of a sequence of output time steps that span a specified time window, for example, the output time steps can be regularly spaced within the specified time window.

[0006] The audio samples at a given output time step may be amplitude values ​​of sound waves, or may be amplitude values ​​that have been compressed, companded, or both. For example, the audio samples may be raw amplitude values ​​or mu-law companded representations of amplitude values.

[0007] According to a first aspect, there is provided a method for generating a prediction of an audio signal, the method comprising: receiving a request to generate an audio signal having a respective audio sample at each of a plurality of output time steps spanning a time window; obtaining a semantic representation of the audio signal specifying a respective semantic token at each of a plurality of first time steps spanning the time window, each semantic token being selected from a vocabulary of semantic tokens and representing semantic content of the audio signal at a corresponding first time step; generating, using one or more generative neural networks, an acoustic representation of the audio signal conditioned on at least the semantic representation, the acoustic representation specifying a set of one or more respective acoustic tokens at each of a plurality of second time steps spanning the time window, the one or more respective acoustic tokens at each second time step representing acoustic characteristics of the audio signal at the corresponding second time step; and processing at least the acoustic representation using a decoder neural network to generate a prediction of the audio signal.

[0008] In some implementations, the decoder neural network is a decoder neural network of a neural audio codec that is jointly trained with the encoder neural network for the purpose of measuring the reconstruction quality of a predicted audio signal generated by the decoder neural network from an acoustic representation generated using the output generated by the encoder neural network.

[0009] In some implementations, the acoustic representation is a prediction of a ground truth acoustic representation that would be generated from the output of the encoder neural network by processing the audio signal.

[0010] In some implementations, the encoder neural network outputs a respective embedding at each of a plurality of second time steps, and the ground truth acoustic representation is generated by applying quantization to each of the respective embeddings.

[0011] In some implementations, the quantization is residual vector quantization that encodes each embedding using a hierarchy of multiple vector quantizers, each generating a respective acoustic token from a corresponding vocabulary of acoustic tokens for the vector quantizer, the hierarchy including one or more coarse vector quantizers at one or more initial positions in the hierarchy and one or more fine quantizers at one or more final positions in the hierarchy, and the set of one or more respective acoustic tokens at each second time step includes, for each vector quantizer, a respective acoustic token selected from the vector quantizer's vocabulary that is a prediction of a ground truth acoustic token that would be generated by the vector quantizer from the ground truth embedding generated by the encoder neural network at the second time step.

[0012] In some embodiments, the set of one or more respective acoustic tokens at each of the plurality of second time steps includes a plurality of acoustic tokens collectively representing a prediction of the output of residual vector quantization applied to an embedding representing an acoustic characteristic of the audio signal at the second time step, the residual vector quantization encoding the embedding using a hierarchy of a plurality of vector quantizers, each generating a respective acoustic token from a corresponding vocabulary of acoustic tokens for the vector quantizer, the hierarchy including one or more coarse vector quantizers at one or more initial positions in the hierarchy and one or more fine vector quantizers at one or more final positions in the hierarchy, and the set of acoustic tokens at each of the second time steps includes, for each vector quantizer, a respective acoustic token selected from the vocabulary of the vector quantizer.

[0013] In some implementations, generating an acoustic representation of the audio signal using one or more generative neural networks and conditioned at least on the semantic representation includes using a first generative neural network to generate, for each of one or more coarse vector quantizers in the hierarchy, an acoustic token for a second time step of the vector quantizer conditioned at least on the semantic representation.

[0014] In some implementations, the first generation neural network is an autoregressive neural network configured to autoregressively generate acoustic tokens according to a first generation order, wherein each particular acoustic token at each particular second time step for each particular coarse vector quantizer is conditioned on at least the semantic representation and any acoustic tokens that precede the particular acoustic token in the first generation order.

[0015] In some implementations, each particular acoustic token at each particular second time step for each particular coarse vector quantizer is preceded in first generation order by (i) any acoustic token for any of the coarse vector quantizers at any second time step preceding the particular second time step, and (ii) any acoustic token at the particular second time step of any coarse vector quantizer preceding the particular vector quantizer in the hierarchy.

[0016] In some implementations, the first generating neural network has a decoder-only transformer architecture or an encoder-decoder transformer architecture.

[0017] In some implementations, generating an acoustic representation of the audio signal using one or more generative neural networks and conditioned on at least the semantic representation includes using a second generative neural network to generate, for each of one or more fine vector quantizers in the hierarchy, an acoustic token for each second time step of the vector quantizer conditioned on the acoustic token for each second time step of one or more coarse vector quantizers in the hierarchy.

[0018] In some embodiments, the second generation neural network is not conditioned on the semantic representation.

[0019] In some implementations, the second generation neural network is an autoregressive neural network configured to autoregressively generate acoustic tokens according to a second generation order, wherein each particular acoustic token at each particular second time step for each particular fine vector quantizer is conditioned on (i) the respective acoustic tokens for at least a subset of the second time steps of one or more coarse vector quantizers, and (ii) at least a subset of the acoustic tokens that precede the particular acoustic token in the second generation order.

[0020] In some implementations, each particular acoustic token at each particular second time step for each particular fine vector quantizer is preceded in the second generation order by (i) any acoustic token for any of the fine vector quantizers at any second time step preceding the particular second time step, and (ii) any acoustic token at the particular second time step for any fine vector quantizer preceding the particular vector quantizer in the hierarchy.

[0021] In some embodiments, each particular acoustic token at each particular second time step for each particular fine vector quantizer is conditioned on (i) the respective acoustic tokens for one or more coarse vector quantizers that precede the second time step by up to a threshold number of second time steps, and (ii) any acoustic tokens at second time steps that precede the particular second time step in the second generation order and that precede the second time step by up to a threshold number of second time steps.

[0022] In some implementations, the second generating neural network has a decoder-only transformer architecture or an encoder-decoder transformer architecture.

[0023] In some implementations, obtaining a semantic representation of the audio signal includes autoregressively generating the semantic representation using a third generative neural network.

[0024] In some implementations, the request specifies a context for the audio signal, and the audio signal is conditioned on the context.

[0025] In some implementations, the context specifies semantic characteristics of the audio signal, and obtaining a semantic representation of the audio signal includes generating the semantic representation conditioned on the context.

[0026] In some implementations, the context specifies acoustic characteristics of the audio signal, and generating an acoustic representation of the audio signal using one or more generative neural networks and conditioned on at least the semantic representation includes generating an acoustic representation of the audio signal using one or more generative neural networks and conditioned on the semantic representation and the context.

[0027] In some implementations, using a first generative neural network to generate, for each of one or more coarse vector quantizers in the hierarchy, a respective acoustic token for a second time step of the vector quantizer, conditioned on at least the semantic representation, comprises using a first generative neural network to generate, for each of one or more coarse vector quantizers in the hierarchy, a respective acoustic token for a second time step of the vector quantizer, conditioned on the semantic representation and the context.

[0028] In some implementations, processing at least the acoustic representation using a decoder neural network to generate a prediction of the audio signal comprises processing the acoustic representation and the acoustic representation of the context using a decoder neural network to generate a prediction of the audio signal.

[0029] In some implementations, the context includes an audio input.

[0030] In some implementations, the context includes visual data.

[0031] In some implementations, the context includes text data.

[0032] In some implementations, the number of first time steps and the number of second time steps spanning the time window are less than the number of output time steps spanning the time window.

[0033] In some implementations, the number of first time steps spanning the time window is less than the number of second time steps spanning the time window.

[0034] According to a second aspect, there is provided a method for generating a prediction of an audio signal, the method comprising: receiving a request to generate an audio signal having a respective audio sample at each of a plurality of output time steps spanning a time window, the method comprising: processing the input using an embedded neural network to map the input to one or more embedded tokens; generating a semantic representation of the audio signal specifying a respective semantic token at each of a plurality of first time steps spanning the time window, the semantic token being selected from a vocabulary of semantic tokens conditioned on the embedded tokens and representing semantic content of the audio signal at a corresponding first time step; generating an acoustic representation of the audio signal using one or more generative neural networks conditioned on at least the semantic representation and the embedded tokens, the acoustic representation specifying a set of one or more respective acoustic tokens at each of a plurality of second time steps spanning the time window, the one or more respective acoustic tokens at each of the second time steps representing acoustic characteristics of the audio signal at the corresponding second time step; and processing at least the acoustic representation using a decoder neural network to generate a prediction of the audio signal.

[0035] In some implementations, processing the input using an embedded neural network to map the input to one or more embedding tokens includes generating an embedding vector for the input in a joint embedding space using the embedded neural network, and quantizing the embedding vector to generate the embedding tokens.

[0036] In some implementations, the input includes a sequence of text, and an embedding neural network is trained to map the text and audio into a joint embedding space.

[0037] In some implementations, the input includes a sequence of text and the prediction of the audio signal is a prediction of the music described by the sequence of text.

[0038] In some implementations, the input further includes an audio signal representing a melody, and the prediction of the audio signal is described by the sequence of text and is a prediction of music along the melody.

[0039] In some implementations, the method further includes processing the audio signal using a melody embedding neural network to map the audio signal to one or more melody embedding tokens, and concatenating the melody embedding tokens with the embedding tokens.

[0040] In some implementations, the input includes an audio signal representing a melody, and the prediction of the audio signal is a prediction of music along the melody.

[0041] In some implementations, the method further includes processing the audio signal using a melody embedding neural network to map the audio signal into one or more melody embedding tokens, each semantic token selected from a vocabulary of semantic tokens conditioned on the melody embedding tokens.

[0042] In some implementations, processing the audio signal using a melody embedding neural network to map the audio signal to one or more melody embedding tokens includes generating one or more melody embedding vectors for the audio signal in a joint embedding space using the melody embedding neural network, and quantizing the one or more melody embedding vectors to generate the melody embedding tokens.

[0043] In some implementations, the sequence of text includes multiple subsequences of text and the prediction of the audio signal is a musical prediction having a section of music corresponding to and reflecting each of the subsequences.

[0044] In some implementations, the embedded neural network is trained with training data that includes an audio signal.

[0045] In some implementations, the embedding neural network is trained on the objective that the text describing the audio signal and the corresponding audio signal have embeddings that are close to each other in the joint embedding space.

[0046] In some implementations, the decoder neural network is a decoder neural network of a neural audio codec that is jointly trained with the encoder neural network for the purpose of measuring the reconstruction quality of a predicted audio signal generated by the decoder neural network from an acoustic representation generated using the output generated by the encoder neural network.

[0047] In some implementations, the acoustic representation is a prediction of a ground truth acoustic representation that would be generated from the output of the encoder neural network by processing the audio signal.

[0048] In some implementations, the encoder neural network outputs a respective embedding at each of a plurality of second time steps, and the ground truth acoustic representation is generated by applying quantization to each of the respective embeddings.

[0049] In some implementations, the quantization is residual vector quantization that encodes each embedding using a hierarchy of multiple vector quantizers, each generating a respective acoustic token from a corresponding vocabulary of acoustic tokens for the vector quantizer, the hierarchy including one or more coarse vector quantizers at one or more initial positions in the hierarchy and one or more fine quantizers at one or more final positions in the hierarchy, and the set of one or more respective acoustic tokens at each second time step includes, for each vector quantizer, a respective acoustic token selected from the vector quantizer's vocabulary that is a prediction of a ground truth acoustic token that would be generated by the vector quantizer from the ground truth embedding generated by the encoder neural network at the second time step.

[0050] In some embodiments, the set of one or more respective acoustic tokens at each of the plurality of second time steps includes a plurality of acoustic tokens collectively representing a prediction of the output of residual vector quantization applied to an embedding representing an acoustic characteristic of the audio signal at the second time step, the residual vector quantization encoding the embedding using a hierarchy of a plurality of vector quantizers, each generating a respective acoustic token from a corresponding vocabulary of acoustic tokens for the vector quantizer, the hierarchy including one or more coarse vector quantizers at one or more initial positions in the hierarchy and one or more fine vector quantizers at one or more final positions in the hierarchy, and the set of acoustic tokens at each of the second time steps includes, for each vector quantizer, a respective acoustic token selected from the vocabulary of the vector quantizer.

[0051] In some implementations, generating an acoustic representation of the audio signal using one or more generative neural networks and conditioned on at least the semantic representation and the embedded tokens includes using a first generative neural network to generate, for each of one or more coarse vector quantizers in the hierarchy, an acoustic token for a respective second time step of the vector quantizer conditioned on at least the semantic representation and the embedded tokens.

[0052] In some implementations, the first generation neural network is an autoregressive neural network configured to autoregressively generate acoustic tokens according to a first generation order, wherein each particular acoustic token at each particular second time step for each particular coarse vector quantizer is conditioned on at least the semantic representation, and the embedding token, and any acoustic tokens that precede the particular acoustic token in the first generation order.

[0053] In some implementations, each particular acoustic token at each particular second time step for each particular coarse vector quantizer is preceded in generation order by (i) any acoustic token for any of the coarse vector quantizers at any second time step preceding the particular second time step, and (ii) any acoustic token at the particular second time step of any coarse vector quantizer preceding the particular vector quantizer in the hierarchy.

[0054] In some implementations, the first generating neural network has a decoder-only transformer architecture or an encoder-decoder transformer architecture.

[0055] In some implementations, generating an acoustic representation of the audio signal using one or more generative neural networks and conditioned on at least the semantic representation and the embedding tokens includes using a second generative neural network to generate, for each of one or more fine vector quantizers in the hierarchy, an acoustic token for each second time step of the vector quantizer conditioned on the acoustic token for each second time step of one or more coarse vector quantizers in the hierarchy.

[0056] In some implementations, the second generative neural network is not conditioned on the semantic representation and the embedding tokens.

[0057] In some implementations, the second generation neural network is an autoregressive neural network configured to autoregressively generate acoustic tokens according to a second generation order, wherein each particular acoustic token at each particular second time step for each particular fine vector quantizer is conditioned on (i) the respective acoustic tokens for at least a subset of the second time steps of one or more coarse vector quantizers, and (ii) at least a subset of the acoustic tokens that precede the particular acoustic token in the second generation order.

[0058] In some implementations, each particular acoustic token at each particular second time step for each particular fine vector quantizer is preceded in generation order by (i) any acoustic token for any of the fine vector quantizers at any second time step preceding the particular second time step, and (ii) any acoustic token at the particular second time step for any fine vector quantizer preceding the particular vector quantizer in the hierarchy.

[0059] In some embodiments, each particular acoustic token at each particular second time step for each particular fine vector quantizer is conditioned on (i) the respective acoustic tokens for one or more coarse vector quantizers that precede the second time step by up to a threshold number of second time steps, and (ii) any acoustic tokens at second time steps that precede the particular second time step in the second generation order and that precede the second time step by up to a threshold number of second time steps.

[0060] In some implementations, the second generating neural network has a decoder-only transformer architecture or an encoder-decoder transformer architecture.

[0061] In some implementations, generating a semantic representation of the audio signal includes autoregressively generating the semantic representation using a third generative neural network having the embedding tokens as conditioning signals.

[0062] In some implementations, the number of first time steps and the number of second time steps spanning the time window are less than the number of output time steps spanning the time window.

[0063] In some implementations, the number of first time steps spanning the time window is less than the number of second time steps spanning the time window.

[0064] In some implementations, the input comprises a sequence of text, the embedding is an embedding in a joint embedding space, and the one or more generative neural networks are trained at least in part on audio-specific training data, and during training, the semantic and acoustic representations are conditioned on the embedding of the audio input in the joint embedding space.

[0065] According to a further aspect, a system is also described that includes one or more computers and one or more storage devices that store instructions that, when executed by the one or more computers, cause the one or more computers to perform methods described herein.

[0066] According to further aspects, one or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the methods described herein are also described.

[0067] Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages.

[0068] The systems described herein provide high-quality audio generation with long-term consistent structure. For example, to generate a prediction of an audio signal, the system can obtain a semantic representation of the audio signal. The semantic representation can ensure long-term consistency by representing features such as the linguistic content of speech, i.e., melody and rhythm of music. The system can generate an acoustic representation of the audio signal based on the semantic representation using one or more generative neural networks. The acoustic representation can ensure high-quality audio synthesis by representing features such as acoustic details. The system can then process the acoustic representation using a decoder neural network to generate a prediction of the audio signal. In this way, the system can generate high-quality, consistently structured audio using information from both the semantic and acoustic representations.

[0069] Conventional systems for generating audio cannot produce audio with a coherent sound without conditioning or text annotation. The system described herein can generate syntactically and semantically coherent speech even without text annotation. For example, the system can use semantic representations to capture local dependencies, such as phonetics, as well as long-term semantic information, such as linguistic content and temporal structure.

[0070] Furthermore, conventional systems for generating speech may produce speech with limited acoustic diversity or quality. For example, conventional systems may be trained only on clean speech and generate speech in a single speaker's voice. Other conventional systems may produce low-quality audio. The system described herein can generate speech sequences that preserve the voice, intonation, and prosody of any unseen speaker. For example, the system can use acoustic representations to capture the speaker identity and recording conditions of a given contextual audio input.

[0071] In some implementations, the audio generated by the system is music. For example, the system can generate a sequence of audio inputs that includes music that is consistent with the input in terms of melody, harmony, tone, playing style, timbre, and rhythm. The system can use semantic representations to capture information such as harmony, rhythm, and melody. For example, the semantic representations can provide a melody and temporal structure that is consistent with the prompt. The system can then use the semantic representations to guide the generation of an acoustic representation.

[0072] The system can conserve computational resources during training. For example, a decoder neural network that generates predictions of an audio signal from an acoustic representation can be pre-trained and fixed earlier than training one or more generative neural networks. An audio representation neural network that can be used to obtain a target semantic representation and a neural audio codec that can be used to obtain a target acoustic representation can also be pre-trained and fixed earlier than training one or more generative neural networks. Furthermore, while conventional systems are trained with clean audio speech samples, the system described herein performs strongly when trained on more diverse and noisy samples. Increased robustness to the quality of the trained data reduces the computational resources and time required to create and clean training data.

[0073] The system can also conserve computational resources during inference. For example, the system obtains a semantic representation and then uses the semantic representation as conditioning to generate an acoustic representation. Thus, the number of tokens the system processes is reduced compared to alternatives such as processing an interleaved sequence of semantic and acoustic tokens, allowing for more efficient inference and training. Furthermore, by autoregressively generating semantic and acoustic tokens and then mapping the acoustic tokens to audio samples, computational resources are conserved compared to alternatives such as directly processing the audio samples.

[0074] In some implementations, the system can generate music given input that includes text. The system can generate long, high-quality, and consistent music that conforms to a given high-quality text description of higher complexity than conventional music generation systems. For example, because the system uses an embedding neural network with a joint embedding space of audio and text, and the generative neural network described herein, the system can generate music conditional on the text more effectively than conventional systems that generate music from text.

[0075] Furthermore, not all desirable characteristics of music are easy to describe using text. For example, melody can be an important characteristic of a musical composition, but is difficult to describe using text. Given an input audio clip representing a melody, in some implementations, the system can generate music that aligns with the melody. For example, the input audio clip can include a hummed melody. The system can generate music that aligns with the melody by using a melody embedding neural network to provide conditioning to one or more generative neural networks. In some implementations, the system can generate music that aligns with the input melody and reflects an input sequence of text. For example, the system can use an embedding neural network and a melody embedding neural network to generate embedding tokens, also referred to as audio embedding tokens.

[0076] The system can generate music conditioned on text without requiring a large training dataset of text-music paired training data. The system can be trained on a music-only dataset. Training the system on music alone also helps make the system robust to noisy labels, which can be a limitation when implementing traditional music generation systems.

[0077] In some implementations where the system is capable of generating music given input including text, each component of the system can be trained separately, allowing for flexibility and efficiency during training. For example, the embedding neural network and the decoder neural network can be trained separately or simultaneously.

[0078] The details of one or more embodiments of the subject matter herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0079] [Figure 1] FIG. 1 is a block diagram of an exemplary audio production system. [Figure 2] FIG. 2 is a diagram of an exemplary process for generating acoustic tokens. [Figure 3] FIG. 2 is a flow diagram of an example process for generating a prediction of an audio signal. [Figure 4] FIG. 1 is a diagram of an example process for training an example audio production system. [Figure 5] FIG. 2 is a diagram of an example process for generating a prediction of an audio signal. [Figure 6] FIG. 2 is a diagram of an exemplary process for generating acoustic tokens. [Figure 7] FIG. 2 is a flow diagram of an example process for generating a prediction of an audio signal. [Figure 8] FIG. 1 is a diagram of an example process for training an example audio production system. DETAILED DESCRIPTION OF THE INVENTION

[0080] Like reference numbers and designations in the various drawings refer to like elements.

[0081] 1 is a block diagram of an exemplary audio generation system 100. Audio generation system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations in which the systems, components, and techniques described below may be implemented.

[0082] The audio production system 100 generates a prediction of an audio signal 104 given a request 102 to generate an audio signal. The audio signal 104 includes a respective audio sample at each of a plurality of output time steps spanning a time window.

[0083] To generate audio, the system 100 receives a request 102 .

[0084] In some implementations, the request 102 can specify a context for the audio signal 104. In these implementations, the audio signal 104 is subject to the context.

[0085] For example, the context may include an audio input as an input audio signal. In some examples, the audio input may include words spoken by a particular speaker. In these examples, the audio signal 104 may be a sequence of words spoken by the particular speaker. In some examples, the speech input may include music. In these examples, the audio signal 104 may be a sequence of music within the audio input.

[0086] In some implementations, the input audio signal can include a melody and the audio signal 104 can be melodic music. The system 100 can generate an output audio signal 104 that is melodic music, as described in more detail below with reference to Figures 5-8.

[0087] In some examples, the context can also include text data. The audio signal 104 can include speech that reflects the text data. In some implementations, the audio signal 104 can include music that reflects the text data. In these implementations, the system 100 generates an output audio signal 104 that reflects the text data, as described in more detail below with reference to Figures 5-8.

[0088] In some examples, the context may also include visual data, in which case the audio signal 104 may include audio that describes the visual data or music that reflects the visual data.

[0089] In some implementations, the system 100 may also include an embedded neural network 120. In some examples where the context includes an input, the system 100 may process the input and map the input to one or more embedded tokens 122, also referred to as audio embedded tokens.

[0090] For example, if the input includes a text sequence, the embedding neural network 120 can be trained to map the text and audio into a joint text and audio embedding space, also referred to as a joint audio embedding space. For example, the embedding neural network can include a neural network that maps the text input to an embedding and a neural network that maps the audio input to an embedding. In the joint text and audio embedding space, both the text and the audio are mapped to embeddings in the same embedding space. That is, the text and audio embedding vectors have the same dimensionality. Furthermore, embeddings that are close to each other in the joint embedding space mean that the embeddings share semantics within and across modalities. For example, two embeddings that are close to each other can represent two semantically similar text sequences, two semantically similar audio samples, or an audio sample and a text sequence with semantically similar features. The embedding neural network 120 is described in more detail below with reference to FIG. 8.

[0091] The system 100 can obtain a semantic representation 106 of the audio signal 104. The semantic representation 106 specifies a respective semantic token at each of a plurality of first time steps spanning a time window.

[0092] Each semantic token is selected from a semantic token vocabulary and represents the semantic content of the audio signal 104 at the corresponding first time step. Examples of semantic content that a semantic token can represent include linguistic content, phonetic, linguistic syntax, and prosodic features of speech. Examples of semantic content also include musical genre, melody, harmony, and rhythmic characteristics.

[0093] The system can generate the semantic representation 106 of the audio signal 104 in any of a variety of ways. Generating the semantic representation 106 is described in further detail below with reference to Figures 3 and 7.

[0094] The number of first time steps and the number of semantic tokens may depend on the sampling rate of the audio representation neural network, i.e., the number of embeddings per unit time, as described in more detail below with reference to Figures 3 and 7.

[0095] The system 100 then uses one or more generative neural networks 108 to generate an acoustic representation 110 of the audio signal 104 conditioned at least on the semantic representation 106 .

[0096] The acoustic representation 110 specifies a set of one or more respective acoustic tokens at each of a plurality of second time steps spanning a time window. The one or more respective acoustic tokens at each of the second time steps represent acoustic characteristics of the audio signal 104 at the corresponding second time step. The acoustic characteristics capture details of the audio waveform and enable high-quality synthesis. The acoustic characteristics may include, for example, speaker identification. The acoustic characteristics may also include recording conditions such as reverberation level, distortion, background noise, etc. Generating the acoustic representation 110 is described in further detail below with reference to FIGS. 2 and 6.

[0097] The number of second time steps may depend on the sampling rate of the encoder neural network, which will be described in more detail below with reference to Figures 3 and 4. In some implementations, the number of second time steps may be greater than the number of first time steps. For example, the number of second time steps may be twice the number of first time steps. Thus, for each semantic token in a first time step, the acoustic representation may include one or more acoustic tokens in each of two second time steps corresponding to the first time step.

[0098] The system 100 then processes at least the acoustic representation 110 using a decoder neural network 112 to generate a prediction of the audio signal 104. For example, each audio sample in each of a number of output time steps spanning a time window may be based on one or more acoustic tokens of the acoustic representation 110.

[0099] In some implementations, the decoder neural network 112 may be a decoder neural network of a neural audio codec, such as the SoundStream neural audio codec. The decoder neural network 112 is described in more detail below with reference to Figures 3 and 4.

[0100] Thus, the system 100 can generate a satisfying audio signal 104 from acoustic tokens, which requires fewer computational resources and power than generating audio samples directly without tokens. For example, autoregressively generating tokens that represent features of audio samples is more computationally efficient than autoregressively generating audio samples. The system 100 can generate tokens from embeddings at a lower sampling rate than the sampling rate of the audio signal. For example, the audio signal may have a sampling rate of 16 kHz, such that the audio signal contains an audio sample every 0.0625 ms. The embeddings for generating semantic tokens can be computed at a sampling rate of 25 Hz, such that each embedding represents features of a 40 ms window of the audio signal. Furthermore, the embeddings for generating acoustic tokens can be computed at a sampling rate of 50 Hz, such that each embedding represents features of a 20 ms window of the audio signal. After quantizing the embeddings into acoustic tokens, the system 100 can utilize a decoder neural network 112 to decode the acoustic tokens into an audio signal with a higher sampling rate.

[0101] The system 100 can be configured to perform any of a variety of tasks that require producing an audio signal 104 as an output.

[0102] For example, the audio signal 104 may be a speech signal, and the system 100 may generate the speech signal unconditionally, e.g., resulting in the generation of a speech signal that is drawn from a distribution represented by the training data set(s) on which the generative neural network(s) were trained.

[0103] As another example, the audio signal 104 may be a different type of audio signal, e.g., music, animal sounds, etc., and the system may generate the audio signal 104 unconditionally, e.g., resulting in an audio signal 104 that is drawn from a distribution represented by the training dataset(s) on which the generative neural network(s) were trained.

[0104] As another example, as described above, the system 100 can receive a context along with a request to generate the audio signal 104 and generate the audio signal 104 conditioned on the received context.

[0105] For example, the audio signal 104 may be a speech signal or other audio signal, that is, the system 100 generates the output audio signal 104 conditioned on a context that is the input audio signal.

[0106] As one example, the generated output audio signal 104 may be a prediction of the audio signal that follows the input audio signal. For example, the context may be an input speech signal that is a question asked by one speaker, and the output audio signal may be an output speech signal that is an answer to the question spoken by the same speaker or another speaker. As another example, the context may be an input speech signal that is a first part of an utterance spoken by one speaker, and the output audio signal 104 may be an output speech signal that is a completion of an utterance spoken by the speaker or another speaker, or a response to the input utterance.

[0107] As an example, in some implementations, the input audio signal may represent music, and the system 100 may generate an output audio signal 104 that is music in accordance with the input audio signal.

[0108] As another example, system 100 can perform audio separation on an input audio signal to generate output audio signal 104. For example, the input audio signal can include both speech and music or other background noise, and output audio signal 104 can represent only speech. As another example, the input audio signal can include speech from multiple speakers (and optionally background noise), and output audio signal 104 can include only the speech of one of the speakers. In some examples, system 100 can perform audio-conditional separation. That is, when the input audio signal can include additional audio input that is acoustically similar to one of the speakers, output audio signal 104 can include only the speech of the speaker in the input audio signal that is acoustically similar to the additional audio input.

[0109] As another example, the system 100 can perform speech-to-speech conversion in which the input and output speech represent the same semantic content but are spoken differently. For example, the input audio signal can include speech in one natural language, and the output audio signal can represent speech in a target, different natural language that is a conversion from the input speech to the target language. As another example, the input audio signal can include speech spoken by a first speaker, and the output audio signal 104 can represent the same semantic content as the input speech but represented by a different speaker. As an example of this, the input audio signal can include speech spoken by a first speaker with a first accent of a natural language, and the output audio signal can represent the same semantic content as the input speech but represented by a different speaker with a different accent of the natural language. As another example of this, the input audio signal can include speech spoken by a first speaker with a speech disorder, and the output audio signal can represent the same semantic content as the input speech but without the speech disorder. As another example, the input audio signal may include a first speech segment, and the output audio signal may include a second, shorter speech segment that summarizes the semantic content of the first speech.

[0110] As another example, in some implementations, the system 100 can perform melody-conditioned music generation where the input audio signal represents a melody, and the system 100 can generate an output audio signal 104 that is music that conforms to the melody.

[0111] As another example, the context may include both audio and text data.

[0112] For example, system 100 can perform transcript-conditional speech enhancement where the context is a text transcript and noisy audio corresponding to the text transcript, and output audio signal 104 is clean audio corresponding to the text transcript. For example, the clean audio may contain less background noise than the noisy audio.

[0113] As another example, system 100 can perform transcript-based audio infill, where the context is a text transcript and audio corresponding to a portion of the text transcript, and the output audio signal 104 corresponds to another portion of the text transcript. System 100 can be trained to autoregressively predict semantic and acoustic tokens, for example, using supervised constraints.

[0114] As another example, the system 100 can perform speaker-conditioned speech conversion where the context is a text transcript and a speaker's audio, and the output audio signal 104 is a verbalization of the text transcript spoken by the speaker.

[0115] As another example, in some embodiments, system 100 can perform text- and melody-conditioned music generation. The context can include a sequence of text and an input audio signal representing a melody. System 100 can generate an output audio signal 104 that is music described by the sequence of text and that aligns with the melody.

[0116] As another example, the context may include both audio data and visual, eg, image or video, data.

[0117] For example, the system 100 may perform an audio-video sequence, where the system receives a partial audio track along with the corresponding video, and the output audio signal 104 is a sequence of the partial audio track.

[0118] As another example, the system 100 can perform cross-modal infill, where the system receives a video and an audio track corresponding to a portion of the video, and the output audio signal 104 is an audio track corresponding to another portion of the video.

[0119] As another example, the context input may include only visual data.

[0120] For example, the system 100 can perform image-conditioned audio generation, in which the system receives an input image and generates an output audio signal 104 that describes the image.

[0121] As another example, the context input may include only text data.

[0122] For example, the system 100 can perform sound synthesis based on text descriptions, where the input is text describing an audio signal, and the output is an audio signal 104 characterized by the text.

[0123] As an example, system 100 can perform text-conditional music generation, where an input can include a sequence of text and system 100 can generate an output audio signal 104 that is music that can be described by the sequence of text. Generating music given text that describes the music is described in more detail below with reference to Figures 5-8.

[0124] As another example, in some implementations, system 100 can perform story-mode generation. The input can include multiple subsequences of text. The system can generate an output audio signal 104 that includes a section of music corresponding to each of the subsequences. Each section of music can be described by a corresponding subsequence of text. Furthermore, system 100 can generate smooth transitions between each section of music that are consistent in tempo and semantically relevant.

[0125] As another example, the system 100 can perform text-conditioned speech sampling, for example, where the input is a text transcript and the output audio signal 104 is a verbalization of the text transcript spoken by a random speaker with random prosody.

[0126] In some implementations where the context input includes non-audio data, the system 100 can be trained with training data that includes only audio, as described in more detail below with reference to FIG.

[0127] 2 is a diagram of an exemplary process for generating acoustic tokens. For convenience, process 200 is described as being performed by one or more computer systems located at one or more locations. For example, an audio generation system, such as audio generation system 100 of FIG. 1, suitably programmed in accordance with this specification can perform process 200.

[0128] The system can generate the acoustic tokens of the acoustic representation using one or more generative neural networks, such as a coarse generative neural network 210 and a fine generative neural network 220. The coarse generative neural network 210 is also referred to as a first generative neural network. The fine generative neural network 220 is also referred to as a second generative neural network. The system can condition the generation of the acoustic representation based on at least the semantic representation 106.

[0129] The encoder neural network may be part of a neural audio codec that includes a decoder neural network and an encoder neural network, which may be trained jointly, as described in more detail below with reference to Figure 4. The coarse generative neural network 210 and the fine generative neural network 220 may be trained to predict acoustic representations that are generated based on the output of a vector quantizer of the neural audio codec.

[0130] In some implementations, the set of one or more respective acoustic tokens at each of the plurality of second time steps includes a plurality of acoustic tokens collectively representing a prediction of the output of residual vector quantization applied to an embedding representing acoustic characteristics of the audio signal at the second time step. The residual vector quantization encodes the embedding using a hierarchy of multiple vector quantizers, each generating a respective acoustic token from a corresponding vocabulary of acoustic tokens for the vector quantizer. The hierarchy includes one or more coarse vector quantizers at one or more initial positions in the hierarchy and one or more fine vector quantizers at one or more final positions in the hierarchy. The set of acoustic tokens at each of the second time steps includes, for each vector quantizer, a respective acoustic token selected from the vocabulary of the vector quantizer.

[0131] For example, a hierarchy may include Q vector quantizers, where vector quantizer 1...Q' may be coarse vector quantizers and vector quantizer (Q'+1)...Q may be fine vector quantizers. The coarse vector quantizers generate coarse acoustic tokens or acoustic tokens for coarse vector quantizer 212 that represent acoustic characteristics such as speaker identity and recording conditions. The fine vector quantizers generate fine acoustic tokens or acoustic tokens for fine vector quantizer 222 that represent fine acoustic details. For example, the fine acoustic tokens can be used to remove lossy compression artifacts in the coarse acoustic tokens.

[0132] In some implementations, the number of first time steps and the number of second time steps spanning the time window are less than the number of output time steps spanning the time window. That is, the sampling rate of the encoder neural network that outputs the respective embeddings in each of the second time steps may be lower than the sampling rate of the audio signal. Furthermore, the sampling rate of the audio representation neural network described below with reference to FIG. 3 may be lower than the sampling rate of the audio signal. Thus, the number of embeddings that can generate semantic tokens and the number of embeddings that can generate acoustic tokens may be less than the number of output time steps.

[0133] In some implementations, the number of first time steps spanned in the time window is less than the number of second time steps spanned in the time window. For example, for every semantic token, the system may generate two acoustic tokens per coarse vector quantizer and two acoustic tokens per fine vector quantizer. That is, the sampling rate of the encoder neural network that outputs the respective embeddings in each of the second time steps may be higher than the sampling rate of the audio representation neural network described below with reference to FIG. 3.

[0134] To generate the acoustic representation, the coarse generative neural network 210 can generate acoustic tokens for the coarse vector quantizer 212 conditioned on at least the semantic representation 106. For example, for each of one or more coarse vector quantizers in a hierarchy, the coarse generative neural network 210 can generate respective acoustic tokens for the second time step of the vector quantizer conditioned on at least the semantic representation. The acoustic tokens for the coarse vector quantizer 212 represent acoustic characteristics such as speaker identity and recording conditions.

[0135] In some implementations where the request specifies a context, such as when the context includes an audio input, and the context specifies acoustic characteristics of the audio signal, the system can generate an acoustic representation of the audio signal conditioned on the semantic representation and the context. For example, the coarse generative neural network 210 can generate, for each of one or more coarse vector quantizers in the hierarchy, respective acoustic tokens for a second time step of the vector quantizer conditioned on the semantic representation 106 and the acoustic tokens representing the context.

[0136] The coarse generation neural network 210 may be an autoregressive neural network configured to autoregressively generate acoustic tokens for the coarse vector quantizer 212 according to a first generation order.

[0137] Each particular acoustic token at each particular second time step for each particular coarse vector quantizer is conditioned on at least the semantic representation and any acoustic tokens that precede it in the first generation order. Furthermore, each particular acoustic token at each particular second time step for each particular coarse vector quantizer is preceded in generation order by (i) any acoustic token for any of the coarse vector quantizers at any second time step that precedes the particular second time step, and (ii) any acoustic token at the particular second time step of any coarse vector quantizer that precedes the particular vector quantizer in the hierarchy.

[0138] For example, a hierarchy may include Q vector quantizers, where vector quantizers 1...Q' may be coarse vector quantizers, while vector quantizers (Q'+1)...Q may be fine vector quantizers. An acoustic token at a particular second time step t and a particular coarse vector quantizer q≦Q' is conditioned on all of the semantic tokens of the semantic representation, any acoustic tokens at the second time step preceding the second time step t, and any acoustic tokens at the second time step t in the coarse vector quantizer preceding Q', and in the coarse vector quantizer preceding the particular coarse vector quantizer q.

[0139] In some implementations, the coarse-grained generative neural network 210 has a decoder-only transformer architecture. In some implementations, the coarse-grained generative neural network 210 has an encoder-decoder transformer architecture.

[0140] To generate the acoustic representation, the fine generative neural network 220 can generate acoustic tokens for the fine vector quantizer 222 conditioned on the acoustic tokens of at least the coarse vector quantizer 212. For example, the fine generative neural network 220 can generate, for each of one or more fine vector quantizers in a hierarchy, respective acoustic tokens for the second time step of the vector quantizer conditioned on the respective acoustic tokens for the second time step of one or more coarse vector quantizers in the hierarchy. Thus, the fine generative neural network 220 cannot be conditioned on the semantic representation 106. The acoustic tokens of the fine vector quantizer 222 can be used to further improve audio quality, for example, by removing lossy compression artifacts.

[0141] The fine generation neural network 220 may be an autoregressive neural network configured to autoregressively generate acoustic tokens according to a second generation order. Each particular acoustic token at each particular second time step for each particular fine vector quantizer is conditioned on (i) the respective acoustic tokens for at least a subset of the second time steps of one or more coarse vector quantizers and (ii) at least a subset of the acoustic tokens that precede the particular acoustic token in the second generation order. Each particular acoustic token at each particular second time step for each particular fine vector quantizer is preceded in the second generation order by (i) any acoustic token for any of the fine vector quantizers at any second time step that precedes the particular second time step, and (ii) any acoustic token at the particular second time step for any fine vector quantizer that precedes the particular vector quantizer in the hierarchy.

[0142] For example, a hierarchy may include Q vector quantizers, where vector quantizers (Q'+1)...Q may be fine vector quantizers. An acoustic token at a particular second time step t and a particular fine vector quantizer q>Q' is conditioned on all of the acoustic tokens of the coarse vector quantizer 312, any acoustic tokens in the second time steps preceding the second time step t and in the fine vector quantizers following Q', and any acoustic tokens at the second time step t in the coarse vector quantizers and fine vector quantizers preceding the particular vector quantizer q.

[0143] In some implementations, each particular acoustic token at each particular second time step for each particular fine vector quantizer is conditioned on (i) the respective acoustic tokens for one or more coarse vector quantizers that precede the second time step by up to a threshold number of second time steps, and (ii) any acoustic tokens at second time steps that precede the particular second time step in second generation order and that precede the second time step by up to a threshold number of second time steps, i.e., the second time steps can be divided into non-overlapping batches of consecutive second time steps.

[0144] For example, an acoustic token at a particular second time step t and a particular fine vector quantizer q>Q' is conditioned on the acoustic tokens of the coarse vector quantizer 312 preceding the particular second time step t in the corresponding batch of second time steps that is the same as the particular second time step t, any acoustic tokens in the second time steps preceding the second time step t in the corresponding batch and in the fine vector quantizer following Q', and any acoustic tokens at the second time step t in the coarse vector quantizer and fine vector quantizer preceding the particular fine vector quantizer q.

[0145] In some implementations, the fine generative neural network 220 has a decoder-only transformer architecture. In some implementations, the fine generative neural network 220 has an encoder-decoder transformer architecture.

[0146] In some implementations, the system can simultaneously generate acoustic tokens for the coarse vector quantizer 212 and the fine vector quantizer 222. For example, in the case of a hierarchy of Q vector quantizers, the acoustic tokens at a particular second time step t of a particular coarse or fine vector quantizer q are conditioned on all of the acoustic tokens at the second time step preceding second time step t and in the vector quantizer preceding q, as well as any acoustic tokens at the second time step t of the vector quantizer preceding the particular vector quantizer q.

[0147] The system can include in the acoustic representation the acoustic tokens of the fine vector quantizer 222. The system can use the acoustic representation to generate a prediction of the audio signal, as described below with reference to FIG.

[0148] 3 is a flow diagram of an exemplary process for generating a prediction of an audio signal. For convenience, process 300 is described as being performed by one or more computer systems located at one or more locations. For example, an audio generation system, such as audio generation system 100 of FIG. 1, suitably programmed in accordance with this specification can perform process 300.

[0149] The system receives a request to generate an audio signal (step 310). The audio signal has a respective audio sample at each of a plurality of output time steps spanning a time window. In some examples, the request specifies a context for the audio signal. The audio signal can be conditioned on the context. For example, the context can include audio input, visual data, and / or text data, as described above.

[0150] The system may obtain a semantic representation of the audio signal (step 320). The semantic representation specifies a respective semantic token at each of a plurality of first time steps spanning a time window. Each semantic token may be selected from a vocabulary of semantic tokens and may represent the semantic content of the audio signal at the corresponding first time step.

[0151] In some embodiments, the system can generate the semantic representation. For example, the system can autoregressively generate the semantic representation using a semantic representation generation neural network. The semantic representation generation neural network can also be referred to as a third generative neural network.

[0152] For example, the semantic representation generation neural network can be trained to autoregressively generate semantic tokens one after the other. The semantic representation generation neural network can be, for example, a decoder-only or encoder-decoder transformer-based neural network. For example, the semantic representation generation neural network may be trained to predict a generated semantic representation based on the output of one or more layers, such as, for example, one of the hidden layers of an audio representation neural network.

[0153] An audio representation neural network can generate outputs or embeddings at regular time intervals. For example, an audio representation neural network can generate one embedding every 40ms seconds of the input audio signal.

[0154] The training of the semantic representation generation neural network and the audio representation neural network is described in more detail below with reference to FIG.

[0155] In some implementations, the request specifies a context that specifies semantic characteristics of the audio signal. For example, the context can include the audio input. The system can generate the semantic representation conditioned on the context. For example, when the generation is conditioned on a context that includes the audio input, the system can use an audio representation neural network to generate a semantic representation of the audio input as described above.

[0156] In some examples, the audio signal spans the same time window as the audio input. The system can use a semantic representation of the audio input as a semantic representation of the audio signal. For example, the semantic representation of the audio signal can include semantic tokens of the semantic representation of the audio input.

[0157] In some examples, the audio signal spans a longer time window than the audio input. For example, the audio signal may be a continuation of the audio input. For example, the audio signal may be a speech continuation or a music continuation. The system may condition the semantic representation generation neural network based on at least the semantic representation of the audio input while autoregressively generating the semantic representation of the audio signal. For example, the semantic representation generation neural network may autoregressively generate semantic tokens for the audio signal conditioned on semantic tokens of the audio input. The semantic representation of the audio signal may include semantic tokens of the semantic representation of the audio input followed by the autoregressively generated semantic tokens.

[0158] In some implementations, the request specifies a context that includes text or image data. When generation is conditioned at least in part on non-audio context input, the system can generate the semantic tokens for the non-audio portion of the audio input by using a suitable encoder neural network to map the context input from a vocabulary to semantic tokens and use the semantic tokens as (at least part of) the semantic representation of the audio signal, or can autoregressively generate the semantic representation of the audio signal while conditioning a semantic representation generation neural network on at least the semantic tokens.

[0159] For example, if the context includes image data, the system can map the image data from a vocabulary to semantic tokens using an encoder neural network configured to map the image data to audio. In some examples, the system can use the semantic tokens as a semantic representation of the audio signal. In some examples, the system can autoregressively generate the semantic representation of the audio signal while conditioning the semantic representation generation neural network based on at least the semantic tokens.

[0160] For example, if the context includes text data, the system can map the text data from a vocabulary to semantic tokens using an encoder neural network configured to map the text data to audio. In some implementations, the encoder neural network can be an embedded neural network, as described in more detail with reference to FIGS. 1 and 8. For example, the text data can include sequences of text describing music, and the embedded neural network can map the musical descriptions to audio. The system can use the semantic tokens corresponding to the audio as at least part of a semantic representation of the audio signal. In some examples, the system can autoregressively generate a semantic representation of the audio signal while conditioning the semantic representation generation neural network based at least on the semantic tokens.

[0161] The system generates an acoustic representation of the audio signal (step 330). The system can use one or more generative neural networks to generate the acoustic representation conditioned on at least the semantic representation. The acoustic representation specifies a set of one or more respective acoustic tokens at each of a plurality of second time steps spanning the time window. The one or more respective acoustic tokens at each of the second time steps can represent acoustic characteristics of the audio signal at the corresponding second time step. Generating the acoustic representation of the audio signal is described in further detail above with reference to FIG. 2.

[0162] In some implementations, the context specifies acoustic characteristics of the audio signal to be generated, such as when the context specifies a target speaker of the output audio signal, when the output audio signal includes a following voice, or when the output audio signal includes audio that should otherwise resemble the output audio signal. In these implementations, the system can map the context to an acoustic representation, for example, using a neural audio codec as described above, and can use at least a portion of the acoustic representation of the context when generating the acoustic representation of the audio signal. Generating an acoustic representation conditioned on a context is described in further detail above with reference to FIG. 2.

[0163] The system processes at least the acoustic representation to generate a prediction of the audio signal (step 340). The system can process at least the acoustic representation using a decoder neural network to generate a prediction of the audio signal. For example, each audio sample at each of multiple output time steps spanning a time window can be based on one or more acoustic tokens of all vector quantizers in the hierarchy.

[0164] In some implementations where the context specifies acoustic characteristics of the audio signal, the system can use a decoder neural network to process the acoustic representation and the acoustic representation of the context to generate a prediction of the audio signal. For example, the decoder neural network can process acoustic tokens of the acoustic representation and acoustic tokens representing the context.

[0165] In some embodiments, the number of first time steps and the number of second time steps spanning the time window are less than the number of output time steps spanning the time window. In some embodiments, the number of first time steps spanning the time window is less than the number of second time steps spanning the time window, as described in more detail above with reference to FIG.

[0166] 4 is a diagram of an example process 400 for training an example audio production system. For convenience, the process 400 is described as being performed by one or more computer training systems located at one or more locations.

[0167] The training system can train the training data to an audio production system, such as the audio production system 100 of Figure 1. In some implementations, components of the audio production system, such as the semantic representation generative neural network 130, the one or more generative neural networks 108, the neural audio codec 420, and the audio representation neural network 410, can be trained separately.

[0168] The training system can train the neural audio codec 420 and the audio representation neural network 410 with training data that includes audio clips. An exemplary audio clip of the training data is shown as target audio 450 in FIG.

[0169] The neural audio codec 420 can include a decoder neural network and an encoder neural network. For example, the encoder neural network can convert the target audio 450 into a quantized coded signal into an acoustic representation. The decoder neural network can convert the acoustic representation into a predicted audio signal. In some implementations, the neural audio codec 420 can be pre-trained and fixed before training the audio generation system.

[0170] The neural audio codec 420 can be trained to minimize the adversarial loss and the reconstruction loss. For example, the decoder neural network and the encoder neural network can be jointly trained with the goal of measuring the reconstruction quality of a predicted audio signal generated by the decoder neural network from an acoustic representation generated using the output generated by the encoder neural network.

[0171] The audio representation neural network 410 may be trained to generate a representation of the input audio signal. In some implementations, the audio representation neural network 410 can be pre-trained and fixed before training the audio generation system. The audio representation neural network 410 can be trained to minimize masked language model (MLM) loss and contrast loss. The audio representation neural network 410 can be, for example, a w2v-BERT model that maps the input audio signal to a set of linguistic features.

[0172] The training system can train the semantic representation generative neural network 130 and one or more generative neural networks 108 with training data that includes audio clips. In some implementations, the generative neural networks can be trained using teacher constraints.

[0173] The training system can train the semantic representation generation neural network 130 to predict generated semantic representations based on the output of one or more layers, such as, for example, one of the hidden layers of the audio representation neural network 410.

[0174] For example, the audio representation neural network 410 can include a model having multiple layers, such as a self-attention based model, e.g., a transformer-based model or a conformer-based model having multiple layers.

[0175] The self-attention based model may have been trained on a speech representation task, for example, through self-supervised learning. The self-attention based model may have been trained on other tasks, such as automatic speech recognition.

[0176] The output of one or more layers of the audio representation neural network 410 can include embeddings of the input audio. For example, a self-attention-based model can generate dense embeddings for some or all of the audio samples in the training data. The embeddings of the intermediate layers of the self-attention-based model for some or all of the audio samples can be clustered into K clusters using k-means. The centroids of the clusters can be used as semantic tokens. In some implementations, the output can be normalized so that each dimension has zero mean and unit variance before clustering.

[0177] An exemplary target semantic representation for training the semantic representation generation neural network 130 may have been generated by providing the target audio 450 to a self-attention-based model. The training system can generate semantic tokens for the target semantic representation by assigning each output of the target audio 450 to the centroid of the nearest cluster.

[0178] In some implementations, consecutive repetitions of semantic tokens in the target semantic representation can be removed. For example, the target semantic representation of target audio 450 includes a sequence of semantic tokens. The training system can remove consecutively repeated semantic tokens in the sequence and then use the target semantic representation without the consecutively repeated semantic tokens for training.

[0179] The training system can train one or more generative neural networks 108 to generate acoustic representations conditioned on at least the semantic representations. In some implementations, the acoustic representations can be predictions of ground truth acoustic representations that would be generated from the output of the encoder neural network by processing the audio signal. For example, the encoder neural network can be a convolutional encoder that maps the audio signal to a sequence of embeddings. The encoder neural network can output a respective embedding at each of a plurality of second time steps. Each respective embedding at each of the plurality of second time steps can correspond to features of the audio signal at the second time step. The ground truth acoustic representations can be generated by applying quantization to each of the respective embeddings. As described above, the encoder neural network can be part of a neural audio codec, such as neural audio codec 420.

[0180] For example, the quantization may be residual vector quantization, which encodes each embedding using a hierarchy of multiple vector quantizers, each generating a respective acoustic token from a corresponding vocabulary of acoustic tokens for the vector quantizer. The hierarchy may include Q vector quantizers, each using a corresponding vocabulary. The hierarchy may include one or more coarse vector quantizers at one or more initial positions in the hierarchy and one or more fine vector quantizers at one or more final positions in the hierarchy. The set of one or more respective acoustic tokens at each of the second time steps may include, for each vector quantizer, a respective acoustic token selected from the vector quantizer's vocabulary that is a prediction of the ground truth acoustic token that would be generated by the vector quantizer from the ground truth embeddings generated by the encoder neural network at the second time step.

[0181] The training system can train one or more generative neural networks 108 to predict generated acoustic representations based on the output of a residual vector quantizer of the neural audio codec 420 .

[0182] The target acoustic representations for training the one or more generative neural networks 108 may have been generated by applying quantization to each of the respective embeddings output by the encoder neural network of the neural audio codec 420 at each of the plurality of second time steps. The quantization may be residual vector quantization (RVQ), as described above with reference to FIG. 2.

[0183] 5 is a diagram of an exemplary process for generating a prediction of an audio signal. For convenience, process 500 is described as being performed by one or more computer systems located at one or more locations. For example, an audio generation system, such as audio generation system 100 of FIG. 1, suitably programmed in accordance with this specification can perform process 500.

[0184] The system receives a request to generate an audio signal conditioned on input 502. For example, input 502 may be included in the audio signal context described above with reference to FIGS. 1-3. In the exemplary process 500, the input includes text data. The system can perform text-conditioned music generation such that the generated audio 504 is music described by the text data: "hip hop song with violin solo." The text data can include descriptions of considerable complexity, such as "a captivating jazz song with an impressive saxophone solo and solo singer" or "90s Berlin techno with bass and a powerful kick."

[0185] The system processes text data using an embedded neural network 120, as described above with reference to FIG. 1. The embedded neural network 120 maps text to one or more embedded tokens 122. The embedded neural network 120 can be trained to map text and audio into a joint embedding space of text and audio, also referred to as a joint audio embedding space. The system can use the embedded neural network 120 to generate embedding vectors for input text data in the joint embedding space. The system can then quantize the embedding vectors to generate the embedded tokens 122.

[0186] In some implementations, the embedded neural network 120 can include a text network and a music network. For example, the embedded neural network 120 can include a neural network trained to map text to an embedding in a joint embedding space and a neural network trained to map audio to an embedding in the joint embedding space. For example, the embedded neural network 120 can be a joint embedding model with two embedding towers, one for text and one for music. The towers use contrastive learning to map text and music to a shared embedding space. Training of the embedded neural network 120 is described below with reference to FIG. 8.

[0187] The system generates a semantic representation 106 of the audio signal 504. Each semantic token in the semantic representation 106 is selected from a vocabulary of semantic tokens conditioned on the embedding tokens 122.

[0188] For example, the system can autoregressively generate the semantic representation 106 using a semantic representation generation neural network having the embedding tokens 122 as a conditioning signal. For example, the semantic representation generation neural network can condition on the embedding tokens 122 while autoregressively generating the semantic representation 106. That is, the semantic tokens in each first time step can be conditioned on the semantic tokens that preceded the first time step and the embedding tokens 122. The semantic representation generation neural network can be S t |S <t , M T where S t represents the semantic token at the first time step t, and M T represents the embedded token 122.

[0189] The system then generates an acoustic representation 110 of the audio signal 504 conditioned on at least the semantic representation 106 and the embedded tokens 122. The system can use one or more generative neural networks to generate the acoustic representation 110 conditioned on at least the semantic representation 106 and the embedded tokens 122. For example, the one or more generative neural networks can autoregressively generate the acoustic representation 110 conditioned on the semantic representation 106 and the embedded tokens 122. That is, the set of one or more acoustic tokens in each second time step can be conditioned on the acoustic tokens, semantic representation 106, and embedded tokens 122 in the preceding second time step. The one or more generative neural networks can t |A <t , S, M T wherein A t denotes the acoustic token at the first time step t, S denotes the semantic representation, and M T represents the embedded token 122 generated from the text.

[0190] The system then processes at least the acoustic representation 110 using a decoder neural network 112 to generate a prediction of the audio signal 504 .

[0191] Thus, the generated audio 504 contains music that can be described by the text data of input 502: "hip hop song with violin solo." Although input 502 in this example contains text data, the system can be trained on audio signals, as described below with reference to FIG.

[0192] 6 is a diagram of an exemplary process 600 for generating acoustic tokens. For convenience, process 600 is described as being performed by one or more computer systems located at one or more locations. For example, an audio generation system, such as audio generation system 100 of FIG. 1, suitably programmed in accordance with this specification can perform process 600.

[0193] Process 600 is similar to process 200 described above with reference to Figure 2 in that the system can generate acoustic tokens for the acoustic representation using one or more generative neural networks, such as a coarse generative neural network 610 and a fine generative neural network 620. The coarse generative neural network 610, also referred to as the first generative neural network, is similar to the coarse generative neural network 210 described with reference to Figure 2. The fine generative neural network 620, also referred to as the second generative neural network, is similar to the fine generative neural network 620 described with reference to Figure 2. The system can condition the generation of the acoustic representation based on at least the semantic representation 106 and the embedding tokens 122, also referred to as audio embedding tokens.

[0194] The encoder neural network may be part of a neural audio codec that includes a decoder neural network and an encoder neural network, which may be trained jointly, as described in more detail below with reference to Figure 8. The coarse generative neural network 610 and the fine generative neural network 620 may be trained to predict acoustic representations that are generated based on the output of a vector quantizer of the neural audio codec.

[0195] In some implementations, the set of one or more respective acoustic tokens at each of the plurality of second time steps includes a plurality of acoustic tokens collectively representing a prediction of the output of residual vector quantization applied to an embedding representing acoustic characteristics of the audio signal at the second time step. The residual vector quantization encodes the embedding using a hierarchy of multiple vector quantizers, each generating a respective acoustic token from a corresponding vocabulary of acoustic tokens for the vector quantizer. The hierarchy includes one or more coarse vector quantizers at one or more initial positions in the hierarchy and one or more fine vector quantizers at one or more final positions in the hierarchy. The set of acoustic tokens at each of the second time steps includes, for each vector quantizer, a respective acoustic token selected from the vocabulary of the vector quantizer.

[0196] For example, a hierarchy may include Q vector quantizers, where Vector Quantizer 1...Q' may be coarse vector quantizers and Vector Quantizer (Q'+1)...Q may be fine vector quantizers. The coarse vector quantizers generate coarse acoustic tokens or acoustic tokens for Coarse Vector Quantizer 612 that represent acoustic characteristics, such as speaker identity and recording conditions. The fine vector quantizers generate fine acoustic tokens or acoustic tokens for Fine Vector Quantizer 622 that represent fine acoustic details. For example, the fine acoustic tokens can be used to remove lossy compression artifacts in the coarse acoustic tokens.

[0197] In some implementations, the number of first time steps and the number of second time steps spanning the time window are less than the number of output time steps spanning the time window. That is, the sampling rate of the encoder neural network that outputs the respective embeddings in each of the second time steps may be lower than the sampling rate of the audio signal. Furthermore, the sampling rate of the audio representation neural network described with reference to FIG. 7 may be lower than the sampling rate of the audio signal. Thus, the number of embeddings that can generate semantic tokens and the number of embeddings that can generate acoustic tokens may be less than the number of output time steps.

[0198] In some implementations, the number of first time steps spanned in the time window is less than the number of second time steps spanned in the time window. For example, for every semantic token, the system may generate two acoustic tokens per coarse vector quantizer and two acoustic tokens per fine vector quantizer. That is, the sampling rate of the encoder neural network that outputs the respective embeddings in each of the second time steps may be higher than the sampling rate of the audio representation neural network described with reference to FIG. 7.

[0199] To generate the acoustic representation, the coarse generative neural network 610 can generate acoustic tokens for the coarse vector quantizer 612 conditioned on at least the semantic representation 106 and the embedding tokens 122. In some examples, the embedding tokens 122 can include melody embedding tokens, also referred to as melodic audio embedding tokens. In some examples, the embedding tokens 122 can be concatenated with the melody embedding tokens.

[0200] For example, for each of one or more coarse vector quantizers in the hierarchy, the coarse generative neural network 610 can generate respective acoustic tokens for the second time step of the vector quantizer, conditioned on at least the semantic representation 106 and the embedding tokens 122. The acoustic tokens of the coarse vector quantizers 612 represent acoustic characteristics such as recording conditions.

[0201] The coarse generative neural network 610 may be an autoregressive neural network configured to autoregressively generate acoustic tokens for the coarse vector quantizer 612 according to a first generation order.

[0202] Each particular acoustic token at each particular second time step for each particular coarse vector quantizer is conditioned on at least the semantic representation 106, and the embedded token 122, and any acoustic tokens that precede it in the first generation order. Furthermore, each particular acoustic token at each particular second time step for each particular coarse vector quantizer is preceded in generation order by (i) any acoustic token for any of the coarse vector quantizers at any second time step that precedes the particular second time step, and (ii) any acoustic token at the particular second time step of any coarse vector quantizer that precedes the particular vector quantizer in the hierarchy.

[0203] For example, a hierarchy may include Q vector quantizers, where vector quantizers 1...Q' may be coarse vector quantizers, while vector quantizers (Q'+1)...Q may be fine vector quantizers. The acoustic tokens at a particular second time step t and a particular coarse vector quantizer q≦Q' are conditioned on all of the semantic tokens of the semantic representation 106, all of the embedding tokens 122, any acoustic tokens at the second time step preceding the second time step t, and any acoustic tokens at the second time step t in the coarse vector quantizer preceding Q', and in the coarse vector quantizer preceding the particular coarse vector quantizer q.

[0204] In some implementations, the coarse-grained generative neural network 610 has a decoder-only transformer architecture. In some implementations, the coarse-grained generative neural network 610 has an encoder-decoder transformer architecture.

[0205] To generate the acoustic representation, the fine generative neural network 620 can generate acoustic tokens for the fine vector quantizer 622 conditioned on the acoustic tokens of at least the coarse vector quantizer 612. For example, the fine generative neural network 620 can generate, for each of one or more fine vector quantizers in a hierarchy, acoustic tokens for each second time step of the vector quantizer conditioned on the acoustic tokens for each second time step of one or more coarse vector quantizers in the hierarchy. Thus, the fine generative neural network 620 cannot be conditioned on the semantic representation 106 and the embedding tokens 122. The acoustic tokens of the fine vector quantizer 622 can be used to further improve audio quality, for example, by removing lossy compression artifacts.

[0206] The fine generation neural network 620 may be an autoregressive neural network configured to autoregressively generate acoustic tokens according to a second generation order. Each particular acoustic token at each particular second time step for each particular fine vector quantizer is conditioned on (i) the respective acoustic tokens for at least a subset of the second time steps of one or more coarse vector quantizers and (ii) at least a subset of the acoustic tokens that precede the particular acoustic token in the second generation order. Each particular acoustic token at each particular second time step for each particular fine vector quantizer is preceded in the second generation order by (i) any acoustic token for any of the fine vector quantizers at any second time step that precedes the particular second time step, and (ii) any acoustic token at the particular second time step for any fine vector quantizer that precedes the particular vector quantizer in the hierarchy.

[0207] For example, a hierarchy may include Q vector quantizers, where vector quantizers (Q'+1)...Q may be fine vector quantizers. An acoustic token at a particular second time step t and a particular fine vector quantizer q>Q' is conditioned on all of the acoustic tokens of the coarse vector quantizer 612, any acoustic tokens in the second time steps preceding the second time step t and in the fine vector quantizers following Q', and any acoustic tokens at the second time step t in the coarse vector quantizers and fine vector quantizers preceding the particular fine vector quantizer q.

[0208] In some implementations, each particular acoustic token at each particular second time step for each particular fine vector quantizer is conditioned on (i) the respective acoustic tokens for one or more coarse vector quantizers that precede the second time step by up to a threshold number of second time steps, and (ii) any acoustic tokens at second time steps that precede the particular second time step in second generation order and that precede the second time step by up to a threshold number of second time steps, i.e., the second time steps can be divided into non-overlapping batches of consecutive second time steps.

[0209] For example, an acoustic token at a particular second time step t and a particular fine vector quantizer q>Q' is conditioned on the acoustic tokens of the coarse vector quantizer 612 preceding the particular second time step t in the corresponding batch of second time steps that is the same as the particular second time step t, any acoustic tokens in the second time steps preceding the second time step t in the corresponding batch and in the fine vector quantizer following Q', and any acoustic tokens at the second time step t in the coarse vector quantizer and fine vector quantizer preceding the particular fine vector quantizer q.

[0210] In some implementations, the fine generative neural network 620 has a decoder-only transformer architecture. In some implementations, the fine generative neural network 620 has an encoder-decoder transformer architecture.

[0211] In some implementations, the system can simultaneously generate acoustic tokens for the coarse vector quantizer 612 and the fine vector quantizer 622. For example, for a hierarchy of Q vector quantizers, the acoustic tokens for a particular coarse or fine vector quantizer q at a particular second time step t are conditioned on all of the acoustic tokens at the second time steps preceding second time step t and in the vector quantizers preceding q, as well as any acoustic tokens at the second time step t in the vector quantizers preceding the particular vector quantizer q.

[0212] The system can include in the acoustic representation the acoustic tokens of the fine vector quantizer 622. The system can use the acoustic representation to generate a prediction of the audio signal, as described below with reference to FIG.

[0213] 7 is a flow diagram of an example process 700 for generating a prediction of an audio signal. For convenience, process 700 is described as being performed by one or more computer systems located at one or more locations. For example, an audio generation system, such as audio generation system 100 of FIG. 1, suitably programmed in accordance with this specification can perform process 700.

[0214] The system receives a request to generate an audio signal conditioned on an input (step 710). The audio signal has a respective audio sample at each of a plurality of output time steps spanning a time window. The audio signal can be conditioned on the input. The input can be included in a context specified in the request, for example. The input can include an input audio signal, visual data, and / or text data.

[0215] In an example where the input includes textual data, such as a sequence of text, the prediction of the audio signal may be a prediction of the music described by the sequence of text.

[0216] In some examples, the sequence of text may include multiple subsequences of text, and the prediction of the audio signal may be a musical prediction having sections of music corresponding to and reflecting each of the subsequences.

[0217] In an example where the input includes text data and an input audio signal representing a melody, the prediction of the audio signal may be a prediction of the music described by the text sequence and along the melody.

[0218] In an example where the input includes an input audio signal representing a melody, the prediction of the audio signal may be a prediction of music along the melody, for example, the input audio signal may represent whistling or humming.

[0219] In an example where the input includes an input audio signal representing music, the prediction of the audio signal may be a continuation of the input audio signal.

[0220] In some examples where the input includes visual data, the system can obtain a textual description of the visual data. In some implementations, the system can generate a textual description of the visual data. The prediction of the audio signal can be a prediction of the music described by the textual description.

[0221] The system processes the input and maps the input to one or more embedding tokens (step 715). The embedding tokens are also referred to as audio embedding tokens. The system can process the input using an embedding neural network. For example, the system can use the embedding neural network to generate an embedding vector for the input and quantize the embedding vector to generate the embedding tokens.

[0222] In some examples, the input includes a sequence of text. The embedding neural network may be trained to map the text and audio into a joint embedding space, also referred to as a joint audio embedding space. In examples where the input includes a sequence of text, embedding tokens are generated from the text.

[0223] The embedded neural network may have been trained with training data that includes an audio signal. Training of the embedded neural network is described in further detail below with reference to FIG. 8.

[0224] In some instances, the input may include multiple subsequences of text, in which case the system may generate an embedding token for each subsequence, thereby varying the conditioning signal for each subsequence when generating the semantic and acoustic representations.

[0225] In some examples, the input includes a sequence of text and an input audio signal representing a melody. The system can process the input audio signal using a melody embedding neural network to map the input audio signal to one or more melody embedding tokens, also referred to as melody audio embedding tokens. The system can then concatenate the melody embedding tokens with the embedding tokens.

[0226] In some examples, the input includes an input audio signal representing a melody. The system can process the input audio signal using a melody embedding neural network to map the audio signal to one or more melody embedding tokens. The system can use the melody embedding tokens as embedding tokens used as conditioning to generate the semantic and acoustic representations.

[0227] In these examples where the input audio signal represents a melody, to process the input audio signal using a melody embedding neural network, the system can use the melody embedding neural network to generate one or more melody embedding vectors for the input audio signal in a joint embedding space, also referred to as the joint audio embedding space. In some implementations, the melody embedding neural network can be a vision transformer (ViT) that receives a time frame of a mel spectrogram of the input audio signal and generates a melody embedding vector for the input audio signal.

[0228] The system can quantize the melody embedding vector to generate melody embedding tokens. For example, the system can use residual vector quantization to quantize the melody embedding vector into melody embedding tokens. Training the melody embedding neural network is described in further detail below with reference to FIG. 8.

[0229] The system generates a semantic representation of the audio signal (step 720). The semantic representation specifies a respective semantic token at each of a plurality of first time steps spanning a time window. Each semantic token is selected from a vocabulary of semantic tokens conditioned on the embedding token and represents the semantic content of the audio signal at the corresponding first time step.

[0230] In some implementations, the system can autoregressively generate the semantic representations using a semantic representation generation neural network having the embedding tokens as conditioning signals, which may also be referred to as a third generative neural network and is similar to the semantic representation generation neural network described above with reference to FIG.

[0231] For example, the semantic representation generation neural network can be trained to autoregressively generate semantic tokens one after the other. The semantic representation generation neural network can be, for example, a decoder-only or encoder-decoder transformer-based neural network. For example, the semantic representation generation neural network may be trained to predict a generated semantic representation based on the output of one or more layers, such as, for example, one of the hidden layers of an audio representation neural network.

[0232] An audio representation neural network can generate outputs or embeddings at regular time intervals. For example, an audio representation neural network can generate one embedding every 40 ms of the input audio signal.

[0233] The training of the semantic representation generation neural network and the audio representation neural network is described in more detail below with reference to FIG.

[0234] In some examples, the input includes a sequence of text and the embedding tokens are generated from the text. In these examples, the system can autoregressively generate a semantic representation of the audio signal while conditioning the semantic representation generation neural network based on at least the embedding tokens.

[0235] In some examples, the input can include multiple subsequences of text, and the embedding tokens include an embedding token for each subsequence. For example, for a first subsequence, the system can condition the semantic representation generation neural network based on the embedding tokens of the first subsequence while autoregressively generating a semantic representation of the audio signal. For a second subsequence, the system can condition the semantic representation generation neural network based on the embedding tokens of the second subsequence and the semantic tokens generated for the first subsequence.

[0236] In some examples, the input includes a sequence of text and an input audio signal representing a melody, and embedding tokens are generated from the text and melody embedding tokens are generated from the input audio signal. The system can concatenate the melody embedding tokens with the embedding tokens. The system can select each semantic token from a vocabulary of semantic tokens conditioned on the melody embedding tokens and the embedding tokens. The system can condition a semantic representation generation neural network on the embedding tokens and melody embedding tokens while autoregressively generating a semantic representation of the audio signal.

[0237] In some examples, the input includes an input audio signal representing a melody, and melody embedding tokens are generated from the input audio signal. In these examples, the system can use the melody embedding tokens as the embedding tokens. The system can select each semantic token from a vocabulary of semantic tokens conditional on the embedding tokens. The system can autoregressively generate a semantic representation of the audio signal while conditioning a semantic representation generation neural network on the embedding tokens.

[0238] In some examples, the system can generate predictions for audio signals longer than the audio signals the system was trained on. For example, the system can be trained on a 30-second audio signal. To generate the longer audio signal, the semantic representation generation neural network can advance in 15-second strides to generate an additional 15 seconds, using 15 seconds as a prefix, while conditioning on the same embedding tokens from the input text.

[0239] The system generates an acoustic representation of the audio signal (step 730). The system can use one or more generative neural networks to generate the acoustic representation conditioned on at least the semantic representation and the embedding tokens. The acoustic representation specifies a set of one or more respective acoustic tokens at each of a plurality of second time steps spanning the time window. The one or more respective acoustic tokens at each of the second time steps represent acoustic characteristics of the audio signal at the corresponding second time step. Generating the acoustic representation is described in further detail above with reference to FIG. 6.

[0240] In some implementations, the input includes a sequence of text. The one or more generative neural networks may have been trained, at least in part, on audio-only training data. During training, the semantic and acoustic representations are conditioned on an embedding of the audio input within a joint text and audio embedding space. Training is described in further detail below with reference to FIG. 8.

[0241] The system processes at least the acoustic representation to generate a prediction of the audio signal (step 740). The system can process the acoustic representation using a decoder neural network. For example, each audio sample at each of multiple output time steps spanning a time window can be based on one or more acoustic representations of all vector quantizers in the hierarchy.

[0242] In some implementations, the number of first time steps and the number of second time steps spanning the time window are less than the number of output time steps spanning the time window. In some implementations, the number of first time steps spanning the time window is less than the number of second time steps spanning the time window, as described in more detail above with reference to FIG.

[0243] 8 is a diagram of an example process 800 for training an example audio production system. For convenience, the process 800 is described as being performed by one or more computer training systems located at one or more locations.

[0244] Process 800 is similar to process 400 described with reference to Figure 4 in that an audio generation system, such as audio generation system 100 of Figure 1, may be trained with training data. In some implementations, components of the audio generation system, such as embedding neural network 120, semantic representation generative neural network 130 (shown in Figure 4 as "generative neural network 130"), one or more generative neural networks 108, neural audio codec 420, and audio representation neural network 410, may be trained separately.

[0245] The embedding neural network 120 can be a joint embedding model with two embedding towers, one for text and one for music. The embedding neural network 120 can be, for example, a MuLan model. The towers use contrastive learning to map text and music into a shared embedding space. The text network can be BERT, pre-trained on a large corpus of text-only data. The music network can be, for example, a residual convolutional network.

[0246] The embedded neural network 120 can be trained with training data that includes an audio signal. For example, the training data can include pairs of music clips and corresponding text annotations. The embedded neural network 120 can be trained to link music to unconstrained natural language descriptions. For example, the embedded neural network 120 can be trained with the objective that text describing an audio signal and the corresponding audio signal have close embeddings to each other in a joint embedding space of audio and text. In some implementations, the embedded neural network 120 can be pre-trained and fixed before training the audio generation system.

[0247] The audio generation system can be trained on audio-only training data, which is easily scalable and not limited to audio data with text captions. Furthermore, the embedded neural network 120 can be trained with contrastive loss to increase robustness to noisy text descriptions.

[0248] To generate embedding tokens for training an audio production system, a training system can provide the target audio signal 850 to the embedding neural network 120. The embedding neural network 120 can generate a representation of the target audio signal 850 in a joint embedding space. The training system can quantize the representation into individual embedding tokens. The embedding tokens are based on an embedding of the target audio signal 850 in the joint embedding space.

[0249] In some implementations, the training data may include audio signals that are longer than the audio signals on which the embedded neural network 120 was pre-trained to operate. For example, the target audio signal 850 may have a length of 30 seconds, and the embedded neural network 120 may have been pre-trained to operate on 10-second sequences. The training system may use the embedded neural network 120 to compute representations of the target audio signal 850 over 10-second windows with 1-second strides and average the resulting representations. The training system may then quantize the averaged representations into distinct embedded tokens of the target audio signal 850.

[0250] The training system can train the neural audio codec 420 and the audio representation neural network 410 with training data that includes audio clips, such as music clips. An exemplary audio clip of the training data is shown in FIG. 8 as target audio 850.

[0251] As described above with reference to Figure 4, the neural audio codec 420 can include a decoder neural network and an encoder neural network. For example, the encoder neural network can convert the target audio 850 into a quantized coded signal into an acoustic representation. The decoder neural network can convert the acoustic representation into a predicted audio signal. In some implementations, the neural audio codec 420 can be pre-trained and fixed before training the audio generation system.

[0252] The neural audio codec 420 can be trained to minimize the adversarial loss and the reconstruction loss. For example, the decoder neural network and the encoder neural network can be jointly trained with the goal of measuring the reconstruction quality of a predicted audio signal generated by the decoder neural network from an acoustic representation generated using the output generated by the encoder neural network.

[0253] The audio representation neural network 410 may be trained to generate a representation of the input audio signal. In some implementations, the audio representation neural network 410 can be pre-trained and fixed before training the audio generation system. The audio representation neural network 410 can be trained to minimize masked language model (MLM) loss and contrast loss. The audio representation neural network 410 can be, for example, a w2v-BERT model.

[0254] The training system can train the semantic representation generative neural network 130 and one or more generative neural networks 108 with training data that includes audio clips. In some implementations, the generative neural networks can be trained using teacher constraints.

[0255] The training system can train the semantic representation generation neural network 130 to predict the generated semantic representation based on the output of one or more layers, such as one of the hidden layers of the audio representation neural network 410. The training system can train the semantic representation generation neural network 130 conditioned on the embedding tokens. The semantic representation generation neural network 130 can be trained based on the distribution p(S t |S <t , M A ) can be modeled, where S t represents the semantic token at the first time step t, and M Arepresents the embedded token generated from the audio.

[0256] For example, the audio representation neural network 410 can include a network with multiple layers, such as a self-attention based model, such as a transformer-based model or a conformer-based model with multiple layers.

[0257] Self-attention based models may have been trained on music representation tasks, for example through self-supervised learning.

[0258] The output of one or more layers of the audio representation neural network 410 can include embeddings of the input audio. For example, a self-attention-based model can generate dense embeddings for some or all of the audio samples in the training data. The embeddings of the intermediate layers of the self-attention-based model for some or all of the audio samples can be clustered into K clusters using k-means. The centroids of the clusters can be used as semantic tokens. In some implementations, the output can be normalized so that each dimension has zero mean and unit variance before clustering.

[0259] An exemplary target semantic representation for training the semantic representation generation neural network 130 may have been generated by providing the target audio 450 to a self-attention-based model. The training system can generate semantic tokens for the target semantic representation by assigning each output of the target audio 450 to the centroid of the nearest cluster.

[0260] In some implementations, consecutive repetitions of semantic tokens in the target semantic representation can be removed. For example, the target semantic representation of target audio 850 includes a sequence of semantic tokens. The training system can remove consecutively repeated semantic tokens in the sequence and then use the target semantic representation without the consecutively repeated semantic tokens for training.

[0261] The training system can train one or more generative neural networks 108 to generate acoustic representations conditioned on at least the semantic representation and the embedding tokens. In some implementations, the acoustic representations are predictions of ground truth acoustic representations that would be generated from the output of the encoder neural network by processing the audio signal. For example, the encoder neural network can be a convolutional encoder that maps the audio signal to a sequence of embeddings. Each respective embedding at each of the plurality of second time steps can correspond to features of the audio signal at the second time step. The ground truth acoustic representations can be generated by applying quantization to each of the respective embeddings. As mentioned above, the encoder neural network can be part of a neural audio codec, such as a neural audio codec.

[0262] For example, the quantization may be residual vector quantization, which encodes each embedding using a hierarchy of multiple vector quantizers, each generating a respective acoustic token from the vector quantizer's corresponding vocabulary of acoustic tokens. The hierarchy may include one or more coarse vector quantizers at one or more initial positions in the hierarchy and one or more fine vector quantizers at one or more final positions in the hierarchy. The set of one or more respective acoustic tokens at each second time step may include, for each vector quantizer, a respective acoustic token selected from the vector quantizer's vocabulary that is a prediction of a ground truth acoustic token that would be generated by the vector quantizer from the ground truth embeddings generated by the encoder neural network at the second time step.

[0263] The training system can train one or more generative neural networks 108 to predict the generated acoustic representation based on the output of a residual vector quantizer of the neural audio codec 420. The training system can train one or more generative neural networks 408 conditioned on the embedding tokens. The one or more generative neural networks 408 can be trained to predict the generated acoustic representation based on the output of a residual vector quantizer of the neural audio codec 420. t |A <t , S, M A ) can be modeled, where A t denotes the acoustic token at the first time step t, S denotes the semantic representation, and M A represents the embedded token generated from the audio.

[0264] The target acoustic representations for training the one or more generative neural networks 108 may have been generated by applying quantization to each of the respective embeddings output by the encoder neural network of the neural audio codec 420 at each of the plurality of second time steps. The quantization may be residual vector quantization (RVQ), as described above with reference to FIG. 6.

[0265] In some implementations, the audio generation system also includes a melody embedding neural network. The melody embedding neural network may be trained on audio pairs that match melodically but differ acoustically. For example, the training data may include different versions of the same music clip, such as covers, instrumental, vocal, humming, and singing versions.

[0266] A melody embedding neural network can be trained so that when two audio clips contain the same melody, their corresponding embeddings in a joint embedding space, also referred to as the joint audio embedding space, are close to each other. For example, a melody embedding neural network can be trained using a semi-hard triplet loss to generate audio embeddings that represent the melody in the input audio signal while being invariant to the acoustic properties associated with the instruments being played.

[0267] This specification uses the term "configured" in reference to systems and computer program components. In the case of a system of one or more computers configured to perform particular operations or actions, it means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when in operation, causes the system to perform the operations or actions. In the case of one or more computer programs configured to perform particular operations or actions, it means that the one or more programs contain instructions that, when executed by a data processing device, cause the device to perform the operations or actions.

[0268] Embodiments of the subject matter and functional operations described herein can be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware, or one or more combinations thereof, including the structures disclosed herein and structural equivalents thereof. Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by or to control the operation of a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random-access or serial-access memory device, or one or more combinations thereof. Alternatively, or additionally, the program instructions can be encoded in an artificially generated, propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, generated to encode information for transmission to a receiver apparatus suitable for execution by the data processing apparatus.

[0269] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus may also be or further include special-purpose logic circuitry, such as, for example, an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). An apparatus may also optionally include, in addition to hardware, code that creates an execution environment for a computer program, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0270] A computer program, which may also be referred to as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted, declarative or procedural, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple cooperating files, e.g., files that store one or more modules, subprograms, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communications network.

[0271] The term "engine" is used broadly herein to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers are dedicated to a particular engine, and in other cases, multiple engines can be installed and run on the same one or more computers.

[0272] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by special purpose logic circuitry such as an FPGA or an ASIC, or a combination of special purpose logic circuitry and one or more programmed computers.

[0273] A computer suitable for running a computer program can be based on a general-purpose or special-purpose microprocessor, or both, or on another type of central processing unit. Typically, the central processing unit receives instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by, or incorporated in, special-purpose logic circuitry. Typically, a computer also includes one or more mass storage devices for storing data, such as magnetic, magneto-optical, or optical disks, or is operatively coupled to receive data from, transfer data to, or both of, the one or more mass storage devices. However, a computer need not have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name just a few.

[0274] Computer-readable media suitable for storing computer program instructions and data include, by way of example, all types of non-volatile memory, media devices and memory devices, including, for example, semiconductor memory devices such as EPROMs, EEPROMs, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0275] To facilitate user interaction, embodiments of the subject matter described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices can also be used to facilitate user interaction. For example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user can be received in any form, including acoustic, spoken, or tactile input. Furthermore, a computer can interact with a user by sending documents to and receiving documents from a device used by the user, such as, for example, by sending a web page to a web browser on the user's device in response to a request received from the web browser. A computer can also interact with a user by, for example, sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving reply messages from the user in response.

[0276] A data processing apparatus for implementing machine learning models may also include dedicated hardware accelerator units, for example, for handling the common computationally intensive parts of machine learning training or generation, i.e., inference, workloads.

[0277] The machine learning model can be implemented and deployed using a machine learning framework, such as the TensorFlow framework or the Jax framework.

[0278] Embodiments of the subject matter described herein can be implemented in a computing system that includes back-end components such as, for example, a data server, or that includes middleware components such as, for example, an application server, or that includes front-end components such as a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0279] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server may send data, e.g., HTML pages, to a user device for the purpose of displaying the data to a user interacting with the device acting as a client and receiving user input from the user. Data generated at the user device, e.g., the results of user interactions, may be received from the device by the server.

[0280] While the specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what can be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable subcombination. Furthermore, even if features are described above as functioning in a particular combination and originally claimed as such, one or more features from a claimed combination may, in some cases, be deleted from that combination, and the claimed combination may be directed to a subcombination or variations of the subcombination.

[0281] Similarly, while operations are depicted in the figures and described in the claims in a particular order, it should not be understood that desirable results require that such operations be performed in the particular order or sequential order shown, or that all of the operations shown be performed. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated into a single software product or packaged into multiple software products.

[0282] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. By way of example, the processes depicted in the accompanying figures do not necessarily require following the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. 1. A computer-implemented method for generating a prediction of an audio signal, comprising: receiving a request to generate an audio signal having a respective audio sample at each of a plurality of output time steps spanning a time window; obtaining a semantic representation of the audio signal specifying a respective semantic token at each of a plurality of first time steps spanning the time window, each semantic token being selected from a vocabulary of semantic tokens and representing semantic content of the audio signal at a corresponding first time step; generating, using one or more generative neural networks and conditioned on at least the semantic representation, an acoustic representation of the audio signal, the acoustic representation specifying a set of one or more respective acoustic tokens at each of a plurality of second time steps spanning the time window, the one or more respective acoustic tokens at each second time step representing an acoustic characteristic of the audio signal at a corresponding second time step; processing at least the acoustic representation using a decoder neural network to generate the prediction of the audio signal; A method comprising:

2. 2. The method of claim 1, wherein the decoder neural network is a decoder neural network of a neural audio codec that is jointly trained with the encoder neural network for the purpose of measuring the reconstruction quality of a predicted audio signal generated by the decoder neural network from an acoustic representation generated using outputs generated by an encoder neural network.

3. The method of claim 1 or 2, wherein the acoustic representation is a prediction of a ground truth acoustic representation that would be generated from the output of an encoder neural network by processing the audio signal.

4. 4. The method of claim 3 , wherein the encoder neural network outputs a respective embedding at each of the plurality of second time steps, and the ground truth acoustic representation is generated by applying quantization to each of the respective embeddings.

5. the quantization is residual vector quantization that encodes each embedding using a hierarchy of multiple vector quantizers, each generating a respective acoustic token from a corresponding vocabulary of acoustic tokens for the vector quantizer, the hierarchy comprising one or more coarse vector quantizers in one or more initial positions within the hierarchy and one or more fine quantizers in one or more final positions within the hierarchy; the set of one or more respective acoustic tokens at each second time step includes, for each vector quantizer, a respective acoustic token selected from the vocabulary of the vector quantizer and which is a prediction of a ground truth acoustic token that would be generated by the vector quantizer from a ground truth embedding generated by the encoder neural network at the second time step. The method of claim 4.

6. the set of one or more respective acoustic tokens at each of the plurality of second time steps comprises a plurality of acoustic tokens collectively representing a prediction of an output of residual vector quantization applied to an embedding representing an acoustic characteristic of the audio signal at the second time step; the residual vector quantization encodes the embedding using a hierarchy of a plurality of vector quantizers, each generating a respective acoustic token from a corresponding vocabulary of acoustic tokens for the vector quantizer, the hierarchy comprising one or more coarse vector quantizers at one or more initial positions in the hierarchy and one or more fine vector quantizers at one or more final positions in the hierarchy; the set of acoustic tokens at each second time step includes, for each vector quantizer, a respective acoustic token selected from the vocabulary of the vector quantizer.

6. The method according to any one of claims 1 to 5.

7. generating an acoustic representation of the audio signal using one or more generative neural networks and conditioned on at least the semantic representation; using a first generative neural network to generate, for each of the one or more coarse vector quantizers in the layer, the respective acoustic tokens for the second time step of the vector quantizer conditioned on at least the semantic representation; The method according to claim 5 or claim 6.

8. 8. The method of claim 7, wherein the first generative neural network is an autoregressive neural network configured to autoregressively generate the acoustic tokens according to a first generation order, and wherein each particular acoustic token at each particular second time step for each particular coarse vector quantizer is conditioned on at least the semantic representation and any acoustic tokens that precede the particular acoustic token in the first generation order.

9. 9. The method of claim 8, wherein each particular acoustic token at each particular second time step for each particular coarse vector quantizer is preceded in the first generation order by (i) any acoustic token for any of the coarse vector quantizers at any second time step preceding the particular second time step, and (ii) any acoustic token at the particular second time step of any coarse vector quantizer preceding the particular vector quantizer in the hierarchy.

10. 10. The method of claim 6, wherein the first generator neural network has a decoder-only transformer architecture or an encoder-decoder transformer architecture.

11. generating an acoustic representation of the audio signal using one or more generative neural networks and conditioned on at least the semantic representation; using a second generator neural network to generate, for each of the one or more fine vector quantizers in the layer, the respective acoustic tokens for the second time step of the vector quantizer conditioned on the respective acoustic tokens for the second time step of the one or more coarse vector quantizers in the layer; 11. The method according to any one of claims 7 to 10.

12. The method of claim 11 , wherein the second generative neural network is not conditioned on the semantic representation.

13. 13. The method of claim 11 or claim 12, wherein the second generation neural network is an autoregressive neural network configured to autoregressively generate the acoustic tokens according to a second generation order, and wherein each particular acoustic token at each particular second time step for each particular fine vector quantizer is conditioned on (i) the respective acoustic tokens for at least a subset of the second time steps of the one or more coarse vector quantizers, and (ii) at least a subset of the acoustic tokens that precede the particular acoustic token in the second generation order.

14. 14. The method of claim 13, wherein each particular acoustic token at each particular second time step for each particular fine vector quantizer is preceded in the second generation order by (i) any acoustic token for any of the fine vector quantizers at any second time step preceding the particular second time step, and (ii) any acoustic token at the particular second time step for any fine vector quantizer preceding the particular vector quantizer in the hierarchy.

15. 15. The method of claim 13 or 14, wherein each particular acoustic token at each particular second time step for each particular fine vector quantizer is conditioned on (i) the respective acoustic token for the one or more coarse vector quantizers that precedes the second time step by up to a threshold number of second time steps, and (ii) any acoustic token at a second time step that precedes the particular second time step in the second generation order and that is up to a threshold number of second time steps before the second time step.

16. 16. The method of any one of claims 11 to 15, wherein the second generator neural network has a decoder-only transformer architecture or an encoder-decoder transformer architecture.

17. obtaining a semantic representation of the audio signal, autoregressively generating the semantic representation using a third generative neural network.

17. The method of any one of claims 1 to 16.

18. 18. The method of claim 1, wherein the request specifies a context for the audio signal, and the audio signal is conditioned on the context.

19. The context specifies semantic characteristics of the audio signal, and a semantic representation of the audio signal is obtained. generating the semantic representation conditional on the context; 20. The method of claim 18.

20. the context specifies acoustic characteristics of the audio signal; generating an acoustic representation of the audio signal using one or more generative neural networks and conditioned on at least the semantic representation; generating an acoustic representation of the audio signal conditioned on the semantic representation and the context using one or more generative neural networks.

20. The method of claim 18 or claim 19.

21. using a first generative neural network to generate, for each of the one or more coarse vector quantizers in the layer, the respective acoustic tokens for the second time step of the vector quantizer conditioned on at least the semantic representation; using a first generative neural network to generate, for each of the one or more coarse vector quantizers in the layer, the respective acoustic tokens for the second time step of the vector quantizer conditioned on the semantic representation and the context; The method of claim 20 when dependent on claim 7.

22. processing at least the acoustic representation using a decoder neural network to generate the prediction of the audio signal; processing the acoustic representation and the acoustic representation of the context using the decoder neural network to generate the prediction of the audio signal.

22. The method of claim 20 or claim 21.

23. 23. The method of any one of claims 18 to 22, wherein the context comprises an audio input.

24. 24. The method of any one of claims 18 to 23, wherein the context comprises visual data.

25. 25. The method of any one of claims 18 to 24, wherein the context comprises textual data.

26. 26. The method of claim 1, wherein the number of first time steps and the number of second time steps spanning the time window are less than the number of output time steps spanning the time window.

27. 27. The method of claim 26, wherein the number of first time steps spanned by the time window is less than the number of second time steps spanned by the time window.

28. 1. A computer-implemented method for generating a prediction of an audio signal, comprising: receiving a request to generate an audio signal, conditional on an input, having a respective audio sample at each of a plurality of output time steps spanning a time window; processing the input using an embedded neural network to map the input to one or more embedded tokens; generating a semantic representation of the audio signal specifying a respective semantic token at each of a plurality of first time steps spanning the time window, each semantic token selected from a vocabulary of semantic tokens conditional on the embedding token and representing semantic content of the audio signal at a corresponding first time step; generating, using one or more generative neural networks and conditioned on at least the semantic representation and the embedding tokens, an acoustic representation of the audio signal, the acoustic representation specifying a set of one or more respective acoustic tokens at each of a plurality of second time steps spanning the time window, the one or more respective acoustic tokens at each second time step representing acoustic characteristics of the audio signal at a corresponding second time step; processing at least the acoustic representation using a decoder neural network to generate the prediction of the audio signal; A method comprising:

29. processing the input using an embedded neural network to map the input to one or more embedded tokens; generating an embedding vector for the input in a joint embedding space using the embedding neural network; quantizing the embedding vector to generate an embedding token.

29. The method of claim 28.

30. 29. The method of claim 28, wherein the input comprises a sequence of text, and the embedding neural network is trained to map text and audio into a joint embedding space.

31. 31. A method according to any one of claims 28 to 30, wherein the input comprises a sequence of text and the prediction of the audio signal is a prediction of music described by the sequence of text.

32. 32. The method of claim 31, wherein the input further comprises an audio signal representing a melody, and wherein the prediction of the audio signal is a prediction of music described by the sequence of text and consistent with the melody.

33. The method comprises: processing the audio signal using a melody embedding neural network to map the audio signal to one or more melody embedding tokens; concatenating the melody embedding token with the embedding token; 33. The method of claim 32, further comprising:

34. 30. The method of claim 28, wherein the input comprises an audio signal representing a melody, and the prediction of the audio signal is a prediction of music along the melody.

35. The method comprises: further comprising processing the audio signal using a melody embedding neural network to map the audio signal to one or more melody embedding tokens; each semantic token is selected from a vocabulary of semantic tokens conditioned on said melody embedding tokens; 35. The method of claim 34.

36. processing the audio signal using a melody embedding neural network to map the audio signal to one or more melody embedding tokens; generating one or more melody embedding vectors for the audio signal in a joint embedding space using the melody embedding neural network; and quantizing the one or more melody embedding vectors to generate melody embedding tokens.

36. The method of claim 33 or claim 35.

37. 31. The method of claim 30, wherein the sequence of text includes a plurality of subsequences of text, and the prediction of the audio signal is a musical prediction having a section of music corresponding to and reflecting each of the subsequences.

38. 29. The method of claim 28, wherein the embedded neural network is trained with training data that includes an audio signal.

39. 30. The method of claim 28, wherein the embedding neural network is trained on an objective such that text describing an audio signal and the corresponding audio signal have embeddings that are close to each other in the joint embedding space.

40. 29. The method of claim 28, wherein the decoder neural network is a decoder neural network of a neural audio codec that is jointly trained with the encoder neural network for the purpose of measuring the reconstruction quality of a predicted audio signal generated by the decoder neural network from an acoustic representation generated using an output generated by the encoder neural network.

41. 41. The method of any one of claims 28 to 40, wherein the acoustic representation is a prediction of a ground truth acoustic representation that would be produced from the output of an encoder neural network by processing the audio signal.

42. 42. The method of claim 41 , wherein the encoder neural network outputs a respective embedding at each of the plurality of second time steps, and the ground truth acoustic representation is generated by applying quantization to each of the respective embeddings.

43. the quantization is residual vector quantization that encodes each embedding using a hierarchy of multiple vector quantizers, each generating a respective acoustic token from a corresponding vocabulary of acoustic tokens for the vector quantizer, the hierarchy comprising one or more coarse vector quantizers in one or more initial positions within the hierarchy and one or more fine quantizers in one or more final positions within the hierarchy; the set of one or more respective acoustic tokens at each second time step includes, for each vector quantizer, a respective acoustic token selected from the vocabulary of the vector quantizer and which is a prediction of a ground truth acoustic token that would be generated by the vector quantizer from a ground truth embedding generated by the encoder neural network at the second time step.

43. The method of claim 42.

44. the set of one or more respective acoustic tokens at each of the plurality of second time steps comprises a plurality of acoustic tokens collectively representing a prediction of an output of residual vector quantization applied to an embedding representing an acoustic characteristic of the audio signal at the second time step; the residual vector quantization encodes the embedding using a hierarchy of a plurality of vector quantizers, each generating a respective acoustic token from a corresponding vocabulary of acoustic tokens for the vector quantizer, the hierarchy comprising one or more coarse vector quantizers at one or more initial positions in the hierarchy and one or more fine vector quantizers at one or more final positions in the hierarchy; the set of acoustic tokens at each second time step includes, for each vector quantizer, a respective acoustic token selected from the vocabulary of the vector quantizer.

44. The method of any one of claims 28 to 43.

45. generating an acoustic representation of the audio signal using one or more generative neural networks and conditioned on at least the semantic representation and the embedding tokens; using a first generative neural network to generate, for each of the one or more coarse vector quantizers in the layer, the respective acoustic tokens for the second time step of the vector quantizer conditioned on at least the semantic representation and the embedding tokens; 45. The method of claim 43 or claim 44.

46. 46. ​​The method of claim 45, wherein the first generative neural network is an autoregressive neural network configured to autoregressively generate the acoustic tokens according to a first generation order, and wherein each particular acoustic token at each particular second time step for each particular coarse vector quantizer is conditioned on at least the semantic representation, and the embedding token, and any acoustic tokens that precede the particular acoustic token in the first generation order.

47. 47. The method of claim 46, wherein each particular acoustic token at each particular second time step for each particular coarse vector quantizer is preceded in the generation order by (i) any acoustic token for any of the coarse vector quantizers at any second time step preceding the particular second time step, and (ii) any acoustic token at the particular second time step of any coarse vector quantizer preceding the particular vector quantizer in the hierarchy.

48. 48. The method of any one of claims 44 to 47, wherein the first generator neural network has a decoder-only transformer architecture or an encoder-decoder transformer architecture.

49. generating an acoustic representation of the audio signal using one or more generative neural networks and conditioned on at least the semantic representation and the embedding tokens; using a second generator neural network to generate, for each of the one or more fine vector quantizers in the layer, the respective acoustic tokens for the second time step of the vector quantizer conditioned on the respective acoustic tokens for the second time step of the one or more coarse vector quantizers in the layer; 49. The method of any one of claims 45 to 48.

50. 50. The method of claim 49, wherein the second generative neural network is not conditioned on the semantic representation and the embedding tokens.

51. 51. The method of claim 49 or claim 50, wherein the second generation neural network is an autoregressive neural network configured to autoregressively generate the acoustic tokens according to a second generation order, and wherein each particular acoustic token at each particular second time step for each particular fine vector quantizer is conditioned on (i) the respective acoustic token for at least a subset of the second time steps of the one or more coarse vector quantizers, and (ii) at least a subset of the acoustic tokens that precede the particular acoustic token in the second generation order.

52. 52. The method of claim 51 , wherein each particular acoustic token at each particular second time step for each particular fine vector quantizer is preceded in the generation order by (i) any acoustic token for any of the fine vector quantizers at any second time step preceding the particular second time step, and (ii) any acoustic token at the particular second time step for any fine vector quantizer preceding the particular vector quantizer in the hierarchy.

53. 53. The method of claim 51 or claim 52, wherein each particular acoustic token at each particular second time step for each particular fine vector quantizer is conditioned on (i) the respective acoustic token for the one or more coarse vector quantizers that precedes the second time step by up to a threshold number of second time steps, and (ii) any acoustic token at a second time step that precedes the particular second time step in the second generation order and that is up to a threshold number of second time steps before the second time step.

54. 54. A method according to any one of claims 49 to 53, wherein the second generator neural network has a decoder-only transformer architecture or an encoder-decoder transformer architecture.

55. generating a semantic representation of the audio signal, autoregressively generating the semantic representation using a third generative neural network having the embedding tokens as a conditioned signal.

55. The method of any one of claims 28 to 54.

56. 56. The method of any one of claims 28 to 55, wherein the number of first time steps and the number of second time steps spanned by the time window are less than the number of output time steps spanned by the time window.

57. 57. The method of claim 56, wherein the number of first time steps spanned by the time window is less than the number of second time steps spanned by the time window.

58. 58. The method of any one of claims 28 to 57, wherein the input comprises a sequence of text, the embedding is an embedding in a joint embedding space, the one or more generative neural networks are trained at least in part with audio-specific training data, and during the training, the semantic representation and the acoustic representation are conditioned on an embedding of audio input in the joint embedding space.

59. 1. A system comprising: one or more computers; one or more storage devices storing instructions that, when executed by said one or more computers, cause said one or more computers to perform the respective operations of any one of claims 28 to 58; A system comprising:

60. 59. One or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform each of the operations of the method of any one of claims 28 to 58.

Images