Generating audio using an autoregressive generative neural network

Abstract

Methods, systems, and apparatus for generating predictions of audio signals, including computer programs encoded on a computer storage medium. One method includes: receiving a request to generate an audio signal; obtaining a semantic representation of the audio signal; generating an acoustic representation of the audio signal conditioned on at least the semantic representation using one or more generative neural networks; and processing the at least acoustic representation using a decoder neural network to generate a prediction of the audio signal.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 404,528, filed September 7, 2022, and U.S. Provisional Application No. 63 / 441,412, filed January 26, 2023. The disclosures of the prior applications are considered part of the disclosures of this application and are incorporated herein by reference. Background Technology

[0003] This manual relates to the use of neural networks to generate audio.

[0004] A neural network is a machine learning model that uses one or more non-linear units to predict the output of a received input. In addition to the output layer, some neural networks also include one or more hidden layers. The output of each hidden layer serves as the input to the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates an output from the received input based on the current values ​​of its corresponding set of parameters. Summary of the Invention

[0005] This specification describes a system implemented as a computer program on one or more computers in one or more locations, the system using one or more generative neural networks to generate audio signals.

[0006] Generally, the output audio signal is an example of output audio that includes samples of the audio wave at each output time step across a series of output time steps spanning a specified time window. For example, the output time steps may be arranged at regular intervals within the specified time window.

[0007] An audio sample at a given output time step can be the amplitude value of the audio wave or an amplitude value that has been compressed, companded, or both. For example, the audio sample can be the original amplitude value or a mu-law companded representation of the amplitude value.

[0008] According to a first aspect, a computer-implemented method for generating a prediction of an audio signal is provided, the method comprising: receiving a request to generate an audio signal having corresponding audio samples at each of a plurality of output time steps spanning a time window; obtaining a semantic representation of the audio signal, the semantic representation specifying corresponding semantic words at each of a plurality of first time steps spanning the time window, each semantic word being selected from a semantic word lexicon and representing the semantic content of the audio signal at the corresponding first time step; generating an acoustic representation of the audio signal conditioned on at least the semantic representation using one or more generative neural networks, the acoustic representation specifying a set of one or more corresponding acoustic words at each of a plurality of second time steps spanning the time window, the one or more corresponding acoustic words at each second time step representing the acoustic characteristics of the audio signal at the corresponding second time step; and processing at least the acoustic representation using a decoder neural network to generate the prediction of the audio signal.

[0009] In some implementations, the decoder neural network is a decoder neural network of a neural audio codec, which has been jointly trained based on a target and an encoder neural network, the target being the reconstructed quality of a predicted audio signal generated by the decoder neural network from an acoustic representation, the acoustic representation being generated using the output of the encoder neural network.

[0010] In some implementations, the acoustic representation is a prediction of the true acoustic representation generated from the output of the encoder neural network by processing the audio signal.

[0011] In some implementations, the encoder neural network outputs a corresponding embedding at each of the plurality of second time steps, and wherein the true acoustic representation is generated by applying quantization to each of the corresponding embeddings.

[0012] In some implementations, the quantization is residual vector quantization, which uses a hierarchical structure of multiple vector quantizers to encode each embedding. Each of the multiple vector quantizers generates a corresponding acoustic lexicon from its corresponding acoustic lexicon vocabulary. The hierarchical structure includes one or more coarse vector quantizers located at one or more preceding positions and one or more fine vector quantizers located at one or more subsequent positions. The set of one or more corresponding acoustic lexicons at each second time step includes corresponding acoustic lexicons for each vector quantizer. These corresponding acoustic lexicons are selected from the vocabulary of the vector quantizer and are predictions of the real value acoustic lexicons generated by the vector quantizer from the real value embeddings, which are generated by the encoder neural network at the second time step.

[0013] In some implementations, the set of one or more corresponding acoustic lexical units at each of the plurality of second time steps includes a plurality of acoustic lexical units that collectively represent a prediction of the output of a residual vector quantization applied to an embedding, the embedding representing the acoustic characteristics of the audio signal at the second time step. The residual vector quantization encodes the embedding using a hierarchy of multiple vector quantizers, each of which generates a corresponding acoustic lexical unit from its corresponding acoustic lexicon. The hierarchy includes one or more coarse vector quantizers located at one or more preceding positions and one or more fine vector quantizers located at one or more subsequent positions. The set of acoustic lexical units at each second time step includes, for each vector quantizer, a corresponding acoustic lexical unit selected from the lexicon of the vector quantizer.

[0014] In some implementations, generating an acoustic representation of the audio signal using one or more generative neural networks conditioned on at least the semantic representation includes: for each of the one or more coarse vector quantizers in the hierarchy, generating a corresponding acoustic lexical for the second time step of the vector quantizer conditioned on at least the semantic representation using a first generative neural network.

[0015] In some implementations, the first generative neural network is an autoregressive neural network configured to autoregressively generate the acoustic lexical units according to a first generation order, wherein each particular acoustic lexical unit of each particular coarse vector quantizer at each particular second time step is conditioned on at least the semantic representation and any acoustic lexical units preceding the particular acoustic lexical unit in the first generation order.

[0016] In some implementations, each specific acoustic lexical at each specific second time step of each specific coarse vector quantizer is, in the first generation order, following: (i) any acoustic lexical at any second time step preceding any coarse vector quantizer of the coarse vector quantizer, and (ii) any acoustic lexical at the specific second time step preceding any coarse vector quantizer of the coarse vector quantizer in the hierarchy.

[0017] In some implementations, the first generative neural network has a decoder-only Transformer architecture or an encoder-decoder Transformer architecture.

[0018] In some implementations, generating an acoustic representation of the audio signal using one or more generative neural networks conditioned on at least the semantic representation includes: for each of the one or more fine vector quantizers in the hierarchy, using a second generative neural network conditioned on the corresponding acoustic lexical at the second time step of the one or more coarse vector quantizers in the hierarchy to generate a corresponding acoustic lexical at the second time step of the vector quantizer.

[0019] In some implementations, the second generative neural network is not conditioned on the semantic representation.

[0020] In some implementations, the second generative neural network is an autoregressive neural network configured to autoregressively generate the acoustic lexical units according to a second generation order, wherein each specific acoustic lexical unit of each particular fine vector quantizer at each particular second time step is conditional on: (i) the corresponding acoustic lexical units of at least a subset of the second time step of the one or more coarse vector quantizers, and (ii) at least a subset of the acoustic lexical units preceding the specific acoustic lexical unit in the second generation order.

[0021] In some implementations, each specific acoustic lexical at each specific second time step of each specific fine vector quantizer is, in the second generation order, following: (i) any acoustic lexical at any second time step preceding any fine vector quantizer of the fine vector quantizer, and (ii) any acoustic lexical at the specific second time step preceding any fine vector quantizer of the fine vector quantizer in the hierarchy.

[0022] In some implementations, each specific acoustic lexical at each specific second time step of each specific fine vector quantizer is conditioned on: (i) the corresponding acoustic lexical of the one or more coarse vector quantizers at most a threshold number of second time steps prior to the second time step, and (ii) any acoustic lexical at the second time step prior to the specific second time step and at most a threshold number of second time steps prior to the second time step in the second generation order.

[0023] In some implementations, the second generative neural network has a decoder-only Transformer architecture or an encoder-decoder Transformer architecture.

[0024] In some implementations, obtaining a semantic representation of the audio signal includes generating the semantic representation regressively using a third generative neural network.

[0025] In some implementations, the request specifies the context of the audio signal and the audio signal is conditional on the context.

[0026] In some implementations, the context specifies the semantic characteristics of the audio signal, and obtaining the semantic representation of the audio signal includes generating the semantic representation based on the context.

[0027] In some implementations, the context specifies the acoustic characteristics of the audio signal, and generating an acoustic representation of the audio signal using one or more generative neural networks conditioned on at least the semantic representation includes: generating an acoustic representation of the audio signal using one or more generative neural networks conditioned on the semantic representation and the context.

[0028] In some implementations, generating corresponding acoustic lexical units for the second time step of the vector quantizer using a first generative neural network conditioned on at least the semantic representation for each of the one or more coarse vector quantizers in the hierarchy includes: generating corresponding acoustic lexical units for the second time step of the vector quantizer using a first generative neural network conditioned on the semantic representation and the context for each of the one or more coarse vector quantizers in the hierarchy.

[0029] In some implementations, using a decoder neural network to process at least the acoustic representation to generate the prediction of the audio signal includes: using the decoder neural network to process the acoustic representation and the acoustic representation of the context to generate the prediction of the audio signal.

[0030] In some implementations, the context includes audio input.

[0031] In some implementations, the context includes visual data.

[0032] In some implementations, the text includes text data.

[0033] In some implementations, the number of first time steps and second time steps spanning the time window is less than the number of output time steps spanning the time window.

[0034] In some implementations, the number of first time steps spanning the time window is less than the number of second time steps spanning the time window.

[0035] According to a second aspect, a computer-implemented method for generating a prediction of an audio signal is provided, the method comprising: receiving a request to generate an audio signal conditioned on an input, the audio signal having corresponding audio samples at each of a plurality of output time steps spanning a time window; processing the input using an embedded neural network to map the input to one or more embedded lexical units; generating a semantic representation of the audio signal, the semantic representation specifying corresponding semantic lexical units at each of a plurality of first time steps spanning the time window, each semantic lexical unit being conditioned on the embedded lexical unit from a semantic word... The process involves selecting from a metavocabulary and representing the semantic content of the audio signal at a corresponding first time step; generating an acoustic representation of the audio signal using one or more generative neural networks conditioned on at least the semantic representation and the embedded lexical units, the acoustic representation specifying a set of one or more corresponding acoustic lexical units at each of a plurality of second time steps spanning the time window, the one or more corresponding acoustic lexical units at each second time step representing the acoustic characteristics of the audio signal at the corresponding second time step; and processing at least the acoustic representation using a decoder neural network to generate the prediction of the audio signal.

[0036] In some implementations, using an embedding neural network to process the input to map the input to one or more embedding lexical units includes: using the embedding neural network to generate an embedding vector of the input in a joint embedding space; and quantizing the embedding vector to generate embedding lexical units.

[0037] In some implementations, the input includes a sequence of text, and the embedding neural network is trained to map text and audio into a joint embedding space.

[0038] In some implementations, the input includes a text sequence, and the prediction of the audio signal is a prediction of music described by the text sequence.

[0039] In some implementations, the input further includes an audio signal representing a melody, and the prediction of the audio signal is a prediction of music described by the text sequence and following the melody.

[0040] In some implementations, the method further includes: using a melody embedding neural network to process the audio signal to map the audio signal to one or more melody embedding morphemes; and concatenating the melody embedding morphemes with the embedding morphemes.

[0041] In some implementations, the input includes an audio signal representing a melody, and the prediction of the audio signal is a prediction of music following the melody.

[0042] In some implementations, the method further includes: using a melody embedding neural network to process the audio signal to map the audio signal to one or more melody embedding lexical units; and wherein each semantic lexical unit is selected from a semantic lexical vocabulary conditioned on the melody embedding lexical unit.

[0043] In some implementations, using a melody embedding neural network to process the audio signal to map the audio signal to one or more melody embedding lemmas includes: using the melody embedding neural network to generate one or more melody embedding vectors of the audio signal in a joint embedding space; and quantizing the one or more melody embedding vectors to generate melody embedding lemmas.

[0044] In some implementations, the text sequence comprises multiple text subsequences, and the prediction of the audio signal is a prediction of music with musical segments corresponding to and reflecting each of the subsequences.

[0045] In some implementations, the embedded neural network is trained based on training data including audio signals.

[0046] In some implementations, the embedding neural network has been trained based on a target such that the text describing the audio signal and the corresponding audio signal have embeddings that are close to each other in the joint embedding space.

[0047] In some implementations, the decoder neural network is a decoder neural network of a neural audio codec, which has been jointly trained based on a target and an encoder neural network, the target being the reconstructed quality of a predicted audio signal generated by the decoder neural network from an acoustic representation, the acoustic representation being generated using the output of the encoder neural network.

[0048] In some implementations, the acoustic representation is a prediction of the true acoustic representation generated from the output of the encoder neural network by processing the audio signal.

[0049] In some implementations, the encoder neural network outputs a corresponding embedding at each of the plurality of second time steps, and wherein the true acoustic representation is generated by applying quantization to each of the corresponding embeddings.

[0050] In some implementations, the quantization is residual vector quantization, which uses a hierarchical structure of multiple vector quantizers to encode each embedding. Each of the multiple vector quantizers generates a corresponding acoustic lexicon from its corresponding acoustic lexicon vocabulary. The hierarchical structure includes one or more coarse vector quantizers located at one or more preceding positions and one or more fine vector quantizers located at one or more subsequent positions. The set of one or more corresponding acoustic lexicons at each second time step includes corresponding acoustic lexicons for each vector quantizer. These corresponding acoustic lexicons are selected from the vocabulary of the vector quantizer and are predictions of the real value acoustic lexicons generated by the vector quantizer from the real value embeddings, which are generated by the encoder neural network at the second time step.

[0051] In some implementations, the set of one or more corresponding acoustic lexical units at each of the plurality of second time steps includes a plurality of acoustic lexical units that collectively represent a prediction of the output of a residual vector quantization applied to an embedding, the embedding representing the acoustic characteristics of the audio signal at the second time step. The residual vector quantization encodes the embedding using a hierarchy of multiple vector quantizers, each of which generates a corresponding acoustic lexical unit from its corresponding acoustic lexicon. The hierarchy includes one or more coarse vector quantizers located at one or more preceding positions and one or more fine vector quantizers located at one or more subsequent positions. The set of acoustic lexical units at each second time step includes, for each vector quantizer, a corresponding acoustic lexical unit selected from the lexicon of the vector quantizer.

[0052] In some implementations, generating an acoustic representation of the audio signal using one or more generative neural networks conditioned on at least the semantic representation and the embedded lexical units includes: for each of the one or more coarse vector quantizers in the hierarchy, using a first generative neural network conditioned on at least the semantic representation and the embedded lexical units to generate a corresponding acoustic lexical unit for the second time step of the vector quantizer.

[0053] In some implementations, the first generative neural network is an autoregressive neural network configured to autoregressively generate the acoustic lexical units according to a first generation order, wherein each specific acoustic lexical unit of each particular coarse vector quantizer at each particular second time step is conditioned on at least the semantic representation and the embedded lexical units, as well as any acoustic lexical units preceding the specific acoustic lexical unit in the first generation order.

[0054] In some implementations, each specific acoustic lexical at each specific second time step of each specific coarse vector quantizer is, in the generation order, following: (i) any acoustic lexical at any second time step preceding any coarse vector quantizer in the specific coarse vector quantizer, and (ii) any acoustic lexical at the specific second time step preceding any coarse vector quantizer in the hierarchy preceding the specific vector quantizer.

[0055] In some implementations, the first generative neural network has a decoder-only Transformer architecture or an encoder-decoder Transformer architecture.

[0056] In some implementations, generating an acoustic representation of the audio signal using one or more generative neural networks conditioned on at least the semantic representation and the embedded lexical units includes: for each of the one or more fine vector quantizers in the hierarchy, using a second generative neural network conditioned on the corresponding acoustic lexical units of the second time step of the one or more coarse vector quantizers in the hierarchy to generate corresponding acoustic lexical units of the second time step of the vector quantizer.

[0057] In some implementations, the second generative neural network is not conditioned on the semantic representation and the embedded lexical units.

[0058] In some implementations, the second generative neural network is an autoregressive neural network configured to autoregressively generate the acoustic lexical units according to a second generation order, wherein each specific acoustic lexical unit of each particular fine vector quantizer at each particular second time step is conditioned on: (i) the corresponding acoustic lexical units of at least a subset of the second time step of the one or more coarse vector quantizers, and (ii) at least a subset of the acoustic lexical units preceding the specific acoustic lexical unit in the second generation order.

[0059] In some implementations, each specific acoustic lexical at each specific second time step of each specific fine vector quantizer is, in the generation order, following: (i) any acoustic lexical at any second time step preceding any fine vector quantizer of the specific fine vector quantizer, and (ii) any acoustic lexical at the specific second time step preceding any fine vector quantizer of the specific fine vector quantizer in the hierarchy.

[0060] In some implementations, each specific acoustic lexical at each specific second time step of each specific fine vector quantizer is conditioned on: (i) the corresponding acoustic lexical of the one or more coarse vector quantizers at most a threshold number of second time steps prior to the second time step, and (ii) any acoustic lexical at the second time step prior to the specific second time step and at most a threshold number of second time steps prior to the second time step in the second generation order.

[0061] In some implementations, the second generative neural network has a decoder-only Transformer architecture or an encoder-decoder Transformer architecture.

[0062] In some implementations, generating the semantic representation of the audio signal includes using a third generative neural network to regressively generate the semantic representation using embedded lexical units as conditional signals.

[0063] In some implementations, the number of first time steps and second time steps spanning the time window is less than the number of output time steps spanning the time window.

[0064] In some implementations, the number of first time steps spanning the time window is less than the number of second time steps spanning the time window.

[0065] In some implementations, the input includes a text sequence, and the embedding is an embedding in a joint embedding space, wherein the one or more generative neural networks have been trained at least in part on audio-only training data, wherein during the training, the semantic representation and the acoustic representation are conditioned on the embedding of the audio input in the joint embedding space.

[0066] According to another aspect, a system is also described, the system comprising: one or more computers; and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the methods described herein.

[0067] According to another aspect, one or more computer-readable storage media are also described, which store instructions that, when executed by one or more computers, cause the one or more computers to perform the methods described herein.

[0068] Specific embodiments of the subject matter described in this specification may be implemented to achieve one or more of the following advantages.

[0069] The system described in this specification provides high-quality audio generation with long-term coherent structure. For example, to generate predictions of audio signals, the system can obtain a semantic representation of the audio signal. The semantic representation can ensure long-term consistency by representing features such as linguistic content in speech or melody and rhythm in music. The system can use one or more generative neural networks to generate an acoustic representation of the audio signal based on the semantic representation. The acoustic representation can ensure high-quality audio synthesis by representing features such as acoustic details. The system can then use a decoder neural network to process the acoustic representation to generate predictions of the audio signal. Therefore, the system can use information from both the semantic and acoustic representations to generate high-quality and structurally coherent audio.

[0070] Conventional audio generation systems may fail to generate audio with coherent speech without conditionalization or text annotation. The system described in this specification can generate syntactically and semantically coherent speech without any text annotation. For example, the system can use semantic representations to capture local dependencies (such as phonetic) and long-term semantic information (such as linguistic content and temporal structure).

[0071] Furthermore, conventional speech generation systems may produce speech with limited acoustic diversity or quality. For example, a conventional system may be trained solely on clean speech and generate speech using only the voice of a single speaker. Other conventional systems may generate low-quality audio. The system described in this specification can generate speech continuations that preserve the voice, intonation, and prosody of any unseen speaker. For example, the system may use acoustic representations to capture the speaker identity and recording conditions of audio input given a context.

[0072] In some implementations, the audio generated by the system is music. For example, the system may generate a continuation of the audio input, which includes music consistent with the input in terms of melody, harmony, pitch, performance style, timbre, and rhythm. The system may use semantic representations to capture information such as harmony, rhythm, and melody. For example, semantic representations may provide a melody and temporal structure consistent with the cues. The system may then use the semantic representations to guide the generation of acoustic representations.

[0073] The system can save computational resources during training. For example, the decoder neural network that generates predictions of audio signals from acoustic representations can be pre-trained and frozen before training one or more generative neural networks. Similarly, the audio representation neural network that can be used to obtain a target semantic representation and the neural audio codec that can be used to obtain a target acoustic representation can also be pre-trained and frozen before training one or more generative neural networks. Furthermore, while conventional systems are trained on clean audio speech samples, the system described in this specification demonstrates strong performance when trained on more diverse and noisy samples. This robustness to increased training data quality reduces the computational resources and time required to prepare and clean the training data.

[0074] The system can also save computational resources during inference. For example, the system obtains a semantic representation and then uses it as a conditionalization for generating an acoustic representation. Therefore, compared to alternatives such as processing intertwined sequences of semantic and acoustic lexical units, the system processes fewer lexical units, allowing for more efficient inference and training. Furthermore, compared to alternatives such as directly processing audio samples, autoregressively generating semantic and acoustic lexical units and then mapping the acoustic units to audio samples saves computational resources.

[0075] In some implementations, the system can generate music based on input including text. Compared to conventional music generation systems, the system can generate longer, higher-quality, and more coherent music that follows a more complex, given high-level text description. For example, because the system uses an embedding neural network with a joint embedding space of audio and text, as well as a generative neural network as described in this specification, the system can generate music more efficiently based on text compared to conventional systems that generate music from text.

[0076] Furthermore, not all desired musical characteristics are easily described using text. For example, melody may be an important characteristic of a piece of music, but it is difficult to describe using text. Given an input audio clip representing a melody, in some implementations, the system can generate music that follows the melody. For example, the input audio clip may include a hummed melody. The system can generate melody-following music by using a melody embedding neural network to conditionalize one or more generative neural networks. In some implementations, the system can generate music that follows the input melody and reflects the input text sequence. For example, the system can use an embedding neural network and a melody embedding neural network to generate embedded lexical units (also known as audio embedding lexical units).

[0077] The system can generate music conditioned on text without requiring a large training dataset of paired text-music training data. The system can be trained on a music-only dataset. Training the system solely on music also helps make it robust to noisy labels, which could be a limitation in implementing conventional music generation systems.

[0078] In some implementations of the system that can generate music from input including text, each component of the system can be trained independently, thereby achieving flexibility and efficiency during training. For example, the embedded neural network and the decoder neural network can be trained individually or simultaneously.

[0079] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0080] Figure 1 This is a block diagram of an example audio generation system.

[0081] Figure 2 This is a diagram of an example process for generating acoustic lexical units.

[0082] Figure 3 This is a flowchart of an example process for generating predictions for audio signals.

[0083] Figure 4 This is a diagram of an example process used to train the example audio generation system.

[0084] Figure 5 This is a diagram illustrating an example process for generating predictions for audio signals.

[0085] Figure 6 This is a diagram of an example process for generating acoustic lexical units.

[0086] Figure 7 This is a flowchart of an example process for generating predictions for audio signals.

[0087] Figure 8 This is a diagram of an example process used to train the example audio generation system.

[0088] The same reference numerals and names in the various figures indicate the same elements. Detailed Implementation

[0089] Figure 1 This is a block diagram of an example audio generation system 100. The audio generation system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, wherein the systems, components, and technologies described below are implemented.

[0090] In response to request 102 to generate an audio signal, audio generation system 100 generates a prediction of audio signal 104. Audio signal 104 includes corresponding audio samples at each of multiple output time steps spanning a time window.

[0091] In order to generate audio, system 100 receives request 102.

[0092] In some implementations, request 102 may specify the context of audio signal 104. In these implementations, audio signal 104 is context-dependent.

[0093] For example, the context may include audio input as an input audio signal. In some examples, the audio input may include words spoken by a specific speaker. In these examples, audio signal 104 may be a continuation of the words spoken by the specific speaker. In some examples, the audio input may include music. In these examples, audio signal 104 may be a continuation of the music in the audio input.

[0094] In some implementations, the input audio signal may include a melody, and the audio signal 104 may be music following a melody. System 100 may generate an output audio signal 104, which is music following a melody, as referenced below. Figures 5 to 8 Further detailed description.

[0095] In some examples, the context may also include text data. Audio signal 104 may include speech reflecting the text data. In some implementations, audio signal 104 may include music reflecting the text data. In these implementations, system 100 generates output audio signal 104 reflecting the text data, as referenced below. Figures 5 to 8 Further detailed description.

[0096] In some examples, the context may also include visual data. In these examples, the audio signal 104 may include speech describing the visual data or music reflecting the visual data.

[0097] In some implementations, system 100 may also include an embedded neural network 120. In some examples where the context includes the input, system 100 may process the input to map it to one or more embedded lexical units 122, also known as audio embedded lexical units.

[0098] For example, if the input includes a text sequence, an embedding neural network 120 can be trained to map text and audio to a joint embedding space for text and audio, also known as a joint audio embedding space. For example, the embedding neural network may include a neural network that maps text input to embeddings, and a neural network that maps audio input to embeddings. In the joint embedding space for text and audio, both text and audio are mapped to embeddings in the same embedding space. That is, the embedding vectors for text and audio have the same dimension. Furthermore, embeddings that are close together in the joint embedding space mean that the embeddings share semantics both intramodally and cross-modally. For example, two embeddings that are close to each other may represent two semantically similar text sequences, two semantically similar audio samples, or audio samples and text sequences with semantically similar features. The embedding neural network 120 is referenced below. Figure 8 Further detailed description.

[0099] System 100 can obtain a semantic representation 106 of audio signal 104. Semantic representation 106 specifies the corresponding semantic lexical at each of a plurality of first time steps spanning a time window.

[0100] Each semantic unit is selected from a semantic unit vocabulary and represents the semantic content of the audio signal 104 at the corresponding first time step. Examples of semantic content that can be represented by semantic units include the linguistic content, pronunciation, linguistic syntax, and prosodic features of speech. Examples of semantic content may also include musical genres, melodies, harmonics, and rhythmic characteristics.

[0101] The system can generate the semantic representation 106 of the audio signal 104 in any of a variety of ways. The generation of the semantic representation 106 is described below. Figure 3 and Figure 7 Further detailed description.

[0102] The number of time steps and the number of semantic units can be determined by the sampling rate of the audio representation neural network, i.e., the number of embeddings per unit time. The audio representation neural network is referenced below. Figure 3 and Figure 7 Further detailed description.

[0103] Then, system 100 uses one or more generative neural networks 108 and conditioned on at least semantic representation 106 to generate an acoustic representation 110 of audio signal 104.

[0104] Acoustic representation 110 specifies a set of one or more corresponding acoustic terms at each of a plurality of second time steps spanning a time window. The one or more corresponding acoustic terms at each second time step represent the acoustic characteristics of the audio signal 104 at the corresponding second time step. Acoustic characteristics capture details of the audio waveform and allow for high-quality synthesis. Acoustic characteristics may include, for example, speaker identity. Acoustic characteristics may also include recording conditions such as the levels of reverberation, distortion, and background noise. The generation of acoustic representation 110 is referenced below. Figure 2 and Figure 6 Further detailed description.

[0105] The number of second time steps can depend on the sampling rate of the encoder neural network, which will be referenced below. Figure 3 and Figure 4 Further details are provided. In some implementations, the number of second time steps may be greater than the number of first time steps. For example, the number of second time steps may be twice the number of first time steps. Therefore, for each semantic lexical at the first time step, the acoustic representation may include one or more acoustic lexical units at each of the two second time steps corresponding to the first time step.

[0106] Then, system 100 uses decoder neural network 112 to process at least the acoustic representation 110 to generate a prediction of audio signal 104. For example, the corresponding audio sample of each of the multiple output time steps spanning a time window may be based on one or more acoustic lexical units of the acoustic representation 110.

[0107] In some implementations, the decoder neural network 112 can be a decoder neural network of a neural audio codec. The neural audio codec can be, for example, the SoundStream neural audio codec. The decoder neural network 112 is referenced below. Figure 3 and Figure 4 Further detailed description.

[0108] Therefore, system 100 can generate a requested audio signal 104 from acoustic lexical units, requiring less computational resources and power compared to directly generating audio samples without lexical units. For example, generating lexical units representing features of audio samples autoregressively is more computationally efficient than generating audio samples autoregressively. System 100 can generate lexical units from embeddings at a sampling rate lower than the sampling rate of the audio signal. For example, the audio signal may have a sampling rate of 16 kHz, such that the audio signal includes one audio sample every 0.0625 ms. Embeddings for generating semantic lexical units can be computed at a sampling rate of 25 Hz, such that each embedding represents features of a 40 ms window of the audio signal. Furthermore, embeddings for generating acoustic lexical units can be computed at a sampling rate of 50 Hz, such that each embedding represents features of a 20 ms window of the audio signal. After quantizing the embeddings into acoustic lexical units, system 100 can utilize decoder neural network 112 to decode the acoustic lexical units into an audio signal at a higher sampling rate.

[0109] System 100 can be configured to perform any of a variety of tasks that require generating an audio signal 104 as output.

[0110] For example, audio signal 104 may be a speech signal, and system 100 may generate speech signals unconditionally, for example, resulting in the generation of speech signals extracted from a distribution represented by a training dataset on which a trained generative neural network is based.

[0111] As another example, the audio signal 104 can be a different type of audio signal, such as music, animal sounds, etc., and the system can generate the audio signal 104 unconditionally, for example, resulting in the generation of the audio signal 104 extracted from a distribution represented by a training dataset on which the trained generative neural network is based.

[0112] As another example, as described above, system 100 may receive a context along with a request to generate audio signal 104, and generate audio signal 104 conditionally based on the received context.

[0113] For example, the audio signal 104 can be speech or other audio signals, that is, the system 100 generates the output audio signal 104 in a context that serves as the input audio signal.

[0114] As an example, the generated output audio signal 104 can be a prediction of an audio signal following an input audio signal. For instance, the context can be the input speech signal, which is a question posed by a speaker, and the output audio signal can be an output speech signal, which is an answer to that question spoken by the same speaker or another speaker. As another example, the context can be the input speech signal, which is the first part of a utterance spoken by a speaker, and the output audio signal 104 can be an output speech signal, which is the completion of that utterance spoken by the speaker or another speaker, or a response to the input utterance.

[0115] As an example, in some implementations, the input audio signal may represent music, and the system 100 may generate an output audio signal 104, which is music following the input audio signal.

[0116] As another example, system 100 may perform sound separation on the input audio signal to generate an output audio signal 104. For example, the input audio signal may include both speech and music or other background noise, and the output audio signal 104 may represent only speech. As another example, the input audio signal may include speech from multiple speakers (and optionally background noise), and the output audio signal 104 may include only the speech of one of the speakers. In some examples, system 100 may perform audio conditional separation, i.e., when the input audio signal may include additional audio input that is acoustically similar to one of the speakers, and the output audio signal 104 may include only the speech of the speaker in the input audio signal that is acoustically similar to the additional audio input.

[0117] As another example, system 100 can perform speech-to-speech translation, where the input speech and output speech represent the same semantic content but are spoken differently. For example, the input audio signal may include speech in a natural language, while the output audio signal may represent speech in a different target natural language; it is a translation of the input speech into the target language. As another example, the input audio signal may include speech spoken by a first speaker, and the output audio signal 104 may represent the same semantic content as the input speech but spoken by a different speaker. As an example of this, the input audio signal may include speech spoken by a first speaker with a first accent in a natural language, and the output audio signal may represent the same semantic content as the input speech but spoken by a different speaker with a different accent in a natural language. As another example, the input audio signal may include speech spoken by a first speaker with a speech impairment, and the output audio signal may represent the same semantic content as the input speech but without a speech impairment. As another example, the input audio signal may include a first speech segment, and the output audio signal may include a shorter second speech segment that summarizes the semantic content of the first speech.

[0118] As another example, in some implementations, system 100 may perform melody conditional music generation, wherein the input audio signal represents a melody, and system 100 may generate an output audio signal 104, which is music following the melody.

[0119] As another example, the context can include both audio data and text data.

[0120] For example, system 100 can perform transcriptional conditional speech enhancement, where the context is text transcription and noisy audio corresponding to the text transcription, and the output audio signal 104 is clean audio corresponding to the text transcription. For example, the clean audio may include less background noise than the noisy audio.

[0121] As another example, system 100 can perform transcription-based audio padding, where the context is a text transcription and audio corresponding to a portion of the text transcription, and the output audio signal 104 corresponds to different portions of the text transcription. System 100 can be trained, for example, by teacher-forcing to autoregressively predict semantic and acoustic lexical units.

[0122] As another example, system 100 can perform speaker-conditional text-to-speech, where the context is the text transcription and the speaker's audio, and the output audio signal 104 is the speech expression that the speaker is uttering the text transcription.

[0123] As another example, in some implementations, system 100 may perform conditional music generation based on text and melody. The context may include a text sequence and an input audio signal representing the melody. System 100 may generate an output audio signal 104, which is music described by the text sequence and follows the melody.

[0124] As another example, the context may include both audio data and visual data (e.g., images or videos).

[0125] For example, system 100 can perform audio-video continuation, wherein the system receives a portion of an audio track and a corresponding video, and outputs an audio signal 104 that is a continuation of the portion of the audio track.

[0126] As another example, system 100 can perform cross-modal filling, wherein the system receives video and an audio track corresponding to a portion of the video, and outputs an audio signal 104 that is an audio track corresponding to a different portion of the video.

[0127] As another example, contextual input can include only visual data.

[0128] For example, system 100 can perform image-conditional audio generation, wherein the system receives an input image and generates an output audio signal 104 describing the image.

[0129] As another example, contextual input can include only text data.

[0130] For example, system 100 may perform sound synthesis based on text description, where the input is text describing an audio signal and the output is an audio signal 104 represented by the text.

[0131] As an example, system 100 can perform text-based conditional music generation, where the input may include a text sequence, and system 100 can generate an output audio signal 104, which is music that can be described by the text sequence. Text-based music generation describing music is referenced below. Figures 5 to 8 To describe in more detail.

[0132] As another example, in some implementations, system 100 may perform story pattern generation. The input may include multiple text subsequences. The system may generate an output audio signal 104 comprising a musical section corresponding to each subsequence. Each musical section may be described by a corresponding text subsequence. Furthermore, system 100 may generate rhythmically consistent and semantically reasonable smooth transitions between each musical section.

[0133] As another example, system 100 can perform text-conditional speech sampling, for example, where the input is a text transcription and the output audio signal 104 is a speech expression of the text transcription spoken by a random speaker at a random prosody.

[0134] In some implementations where the context input includes non-audio data, system 100 can be trained based on training data that includes only audio, as referenced below. Figure 8 To describe in more detail.

[0135] Figure 2 This is a diagram illustrating an example process for generating acoustic lexical units. For convenience, process 200 will be described as being executed by a system of one or more computers located in one or more locations. For example, an audio generation system appropriately programmed according to this specification, such as... Figure 1 The audio generation system 100, executable process 200.

[0136] The system may use one or more generative neural networks (such as coarse generative neural network 210 and fine generative neural network 220) to generate acoustic lexical units of acoustic representations. Coarse generative neural network 210 is also referred to as a first generative neural network. Fine generative neural network 220 is also referred to as a second generative neural network. The system may condition the generation of acoustic representations with at least semantic representation 106.

[0137] An encoder neural network can be part of a neural audio codec that includes both a decoder neural network and an encoder neural network. See the reference below. Figure 4 In further detail, the decoder neural network and the encoder neural network can be jointly trained. The coarse generative neural network 210 and the fine generative neural network 220 can be trained to predict the acoustic representation generated by the output of the vector quantizer based on the neural audio codec.

[0138] In some implementations, the set of one or more corresponding acoustic lexics at each of a plurality of second time steps comprises a plurality of acoustic lexics that collectively represent a prediction of the output of residual vector quantization applied to the embedding, the embedding representing the acoustic characteristics of the audio signal at the second time step. Residual vector quantization encodes the embedding using a hierarchy of multiple vector quantizers, each generating a corresponding acoustic lexicogram from its corresponding acoustic lexicon. The hierarchy includes one or more coarse vector quantizers located at one or more preceding positions and one or more fine vector quantizers located at one or more subsequent positions. The set of acoustic lexics at each second time step includes, for each vector quantizer, a corresponding acoustic lexicogram selected from the lexicon of that vector quantizer.

[0139] For example, the hierarchy may include Q vector quantizers, where vector quantizer 1…Q′ can be coarse vector quantizers, and vector quantizers (Q′+1)…Q can be fine vector quantizers. The coarse vector quantizer generates coarse acoustic lexics representing acoustic characteristics such as speaker identity and recording conditions, or acoustic lexics of coarse vector quantizer 212. The fine vector quantizer generates fine acoustic lexics representing fine acoustic details, or acoustic lexics of fine vector quantizer 222. For example, fine acoustic lexics can be used to remove lossy compression artifacts from coarse acoustic lexics.

[0140] In some implementations, the number of first and second time steps spanning the time window is less than the number of output time steps spanning the time window. That is, the sampling rate of the corresponding embedded encoder neural network output at each second time step within the second time step can be lower than the sampling rate of the audio signal. Furthermore, see the following reference... Figure 3 The sampling rate of the audio representation neural network can be lower than the sampling rate of the audio signal. Therefore, the number of semantic word embeddings and the number of acoustic word embeddings that can be generated from it can be less than the number of output time steps.

[0141] In some implementations, the number of first time steps spanning the time window is less than the number of second time steps spanning the time window. For example, for each semantic lexical unit, the system may generate two acoustic lexical units for each coarse vector quantizer and two acoustic lexical units for each fine vector quantizer. That is, the sampling rate of the corresponding embedded encoder neural network output at each second time step in the second time step may be higher than the following reference. Figure 3 The audio represents the sampling rate of the neural network.

[0142] To generate acoustic representations, the coarse generative neural network 210 can conditionally generate acoustic lexical units of the coarse vector quantizer 212, at least based on the semantic representation 106. For example, for each of one or more coarse vector quantizers in a hierarchical structure, the coarse generative neural network 210 can conditionally generate corresponding acoustic lexical units for a second time step of the vector quantizer, at least based on the semantic representation. The acoustic lexical units of the coarse vector quantizer 212 represent acoustic characteristics such as speaker identity and recording conditions.

[0143] In some implementations where a specified context is requested and the context specifies the acoustic characteristics of the audio signal (such as the context including the audio input), the system can generate an acoustic representation of the audio signal conditioned on a semantic representation and the context. For example, for each coarse vector quantizer in one or more coarse vector quantizers in a hierarchy, the coarse generative neural network 210 can generate corresponding acoustic terms for a second time step of the vector quantizer conditioned on a semantic representation 106 and acoustic terms representing the context.

[0144] The coarse generative neural network 210 can be an autoregressive neural network configured to autoregressively generate acoustic lexics of the coarse vector quantizer 212 according to a first generation order.

[0145] Each specific acoustic lexical of each particular coarse vector quantizer at each particular second time step is conditioned on at least the semantic representation and any acoustic lexical preceding the specific acoustic lexical in the first generation order. Furthermore, each specific acoustic lexical of each particular coarse vector quantizer at each particular second time step follows, in the generation order, any acoustic lexical of any coarse vector quantizer at any second time step preceding the specific second time step, and (ii) any acoustic lexical of any coarse vector quantizer at the specific second time step preceding the specific vector quantizer in the hierarchy.

[0146] For example, the hierarchy may include Q vector quantizers, where vector quantizer 1…Q′ may be coarse vector quantizers, and vector quantizers (Q′+1)…Q may be fine vector quantizers. Acoustic lexical units at a specific second time step t and at a specific coarse vector quantizer q≤Q′ are conditional upon: all semantic lexical units of the semantic representation; any acoustic lexical unit at the second time step prior to the second time step t and at the coarse vector quantizer prior to Q′; and any acoustic lexical unit at the second time step t at the coarse vector quantizer prior to the specific coarse vector quantizer q.

[0147] In some implementations, the coarse-coarse generative neural network 210 has a decoder-only Transformer architecture. In other implementations, the coarse-coarse generative neural network 210 has an encoder-decoder Transformer architecture.

[0148] To generate an acoustic representation, the fine generative neural network 220 can conditionally generate acoustic terms for the fine vector quantizer 222 based on at least the acoustic terms of the coarse vector quantizer 212. For example, for each of the one or more fine vector quantizers in the hierarchy, the fine generative neural network 220 can conditionally generate the corresponding acoustic terms for the second time step of that vector quantizer based on the corresponding acoustic terms for the second time step of the one or more coarse vector quantizers in the hierarchy. Therefore, the fine generative neural network 220 may not be conditional on the semantic representation 106. The acoustic terms of the fine vector quantizer 222 can be used to further improve audio quality, for example, by removing lossy compression artifacts.

[0149] The fine generative neural network 220 may be an autoregressive neural network configured to autoregressively generate acoustic lexical units according to a second generation order. Each specific acoustic lexical unit of each particular fine vector quantizer at each particular second time step is conditional upon: (i) the corresponding acoustic lexical unit of at least a subset of the second time steps of one or more coarse vector quantizers, and (ii) at least a subset of the acoustic lexical units preceding the specific acoustic lexical unit in the second generation order. Each specific acoustic lexical unit of each particular fine vector quantizer at each particular second time step follows the following acoustic lexical units in the second generation order: (i) any acoustic lexical unit of any fine vector quantizer at any second time step preceding the specific second time step, and (ii) any acoustic lexical unit of any fine vector quantizer preceding the specific vector quantizer in the hierarchy at the specific second time step.

[0150] For example, the hierarchy may include Q vector quantizers, and vector quantizers (Q′+1)…Q may be fine vector quantizers. Acoustic lexics at a specific second time step t and at a specific fine vector quantizer q>Q′ are conditional upon: all acoustic lexics of the coarse vector quantizer 312, any acoustic lexics at the second time step before the second time step t and after Q′ at the fine vector quantizer, and any acoustic lexics at the second time step t at both the coarse and fine vector quantizers preceding the specific fine vector quantizer q.

[0151] In some implementations, each specific acoustic word at each specific second time step of each particular fine vector quantizer is conditioned on: (i) the corresponding acoustic words of one or more coarse vector quantizers at most a threshold number of second time steps prior to that second time step, and (ii) any acoustic word in the second generation order that is prior to that specific second time step and is at most a threshold number of second time steps prior to that second time step. That is, the second time step can be divided into multiple non-overlapping batches of consecutive second time steps.

[0152] For example, the acoustic lexical at a specific second time step t and a specific fine vector quantizer q>Q′ is conditional on the following: any acoustic lexical before the coarse vector quantizer 312 and within the same corresponding second time step batch as the specific second time step t, and any acoustic lexical at the fine vector quantizer at the second time step before the second time step t and after Q′ within the corresponding batch, and any acoustic lexical at the second time step t of the coarse vector quantizer before the specific fine vector quantizer q and any acoustic lexical at the second time step t of the fine vector quantizer.

[0153] In some implementations, the refined generative neural network 220 has a decoder-only Transformer architecture. In other implementations, the refined generative neural network 220 has an encoder-decoder Transformer architecture.

[0154] In some implementations, the system can simultaneously generate acoustic tokens for both the coarse vector quantizer 212 and the fine vector quantizer 222. For example, for a hierarchy of Q vector quantizers, the acoustic token of a particular coarse or fine vector quantizer q at a particular second time step t is conditional upon the following: all acoustic tokens at the vector quantizers preceding q at the second time step t, and any acoustic tokens at the second time step t of the vector quantizer preceding q.

[0155] The system may include acoustic terms of a fine vector quantizer 222 in the acoustic representation. The system can use the acoustic representation to generate predictions of audio signals, as described below. Figure 3 As described.

[0156] Figure 3 This is a flowchart of an example process for generating a prediction of an audio signal. For convenience, process 300 will be described as being executed by a system of one or more computers located in one or more locations. For example, an audio generation system appropriately programmed according to this specification, such as... Figure 1 The audio generation system 100, executable process 300.

[0157] The system receives a request to generate an audio signal (step 310). The audio signal has corresponding audio samples at each of a plurality of output time steps spanning a time window. In some examples, the request may specify the context of the audio signal. The audio signal may be context-conditional. For example, the context may include audio input, visual data, and / or text data as described above.

[0158] The system obtains a semantic representation of the audio signal (step 320). The semantic representation specifies a corresponding semantic unit at each of a plurality of first time steps spanning a time window. Each semantic unit can be selected from a semantic unit vocabulary and can represent the semantic content of the audio signal at the corresponding first time step.

[0159] In some implementations, the system can generate semantic representations. For example, the system can use a semantic representation generating neural network to regressively generate semantic representations. A semantic representation generating neural network can also be called a third generative neural network.

[0160] For example, a semantic representation generative neural network can be trained to autoregressively generate semantic lexical units one after another. The semantic representation generative neural network can be, for example, a neural network based on a decoder-only or encoder-decoder Transformer. For example, a semantic representation generative neural network can be trained to predict semantic representations generated based on the output of one or more layers (e.g., an intermediate layer in an intermediate layer) of an audio representation neural network.

[0161] Audio representation neural networks can generate outputs or embeddings at regular time intervals. For example, an audio representation neural network can generate an embedding every 40ms for an input audio signal.

[0162] The training of semantic representation generative neural networks and audio representation neural networks is described in the following reference. Figure 4 Further detailed description.

[0163] In some implementations, a context is specified, which defines the semantic characteristics of the audio signal. For example, the context may include the audio input. The system can generate a semantic representation conditionally based on the context. For example, when generating a semantic representation conditionally based on a context including the audio input, the system can use an audio representation neural network to generate the semantic representation of the audio input as described above.

[0164] In some examples, the audio signal spans the same time window as the audio input. The system can use a semantic representation of the audio input as a semantic representation of the audio signal. For example, the semantic representation of the audio signal may include semantic units of the semantic representation of the audio input.

[0165] In some examples, the audio signal spans a time window longer than the audio input. For example, the audio signal can be a continuation of the audio input. For example, the audio signal can be a continuation of speech or music. The system can cause the semantic representation generating neural network to conditionally generate a semantic representation of the audio signal with at least the semantic representation of the audio input. For example, the semantic representation generating neural network can conditionally generate semantic words for the audio signal from the semantic words of the audio input. The semantic representation of the audio signal may include semantic words from the semantic representation of the audio input, followed by the autoregressively generated semantic words.

[0166] In some implementations, a request is made to specify a context including text or image data. When generating a context input that is at least partially not audio, the system can generate semantic lexical units by: mapping the context input to semantic lexical units from a vocabulary using an appropriate encoder neural network for the non-audio portion of the audio input, and using the semantic lexical units as a semantic representation (at least a portion) of the audio signal; or the system can cause the semantic representation generation neural network to be conditioned on at least the semantic lexical units when autoregressively generating a semantic representation of the audio signal.

[0167] For example, if the context includes image data, the system can use an encoder neural network configured to map image data to audio to map the image data to semantic units from a vocabulary. In some examples, the system can use semantic units as a semantic representation of the audio signal. In some examples, the system can cause the semantic representation generating neural network to be conditioned on at least semantic units when autoregressively generating a semantic representation of the audio signal.

[0168] For example, if the context includes text data, the system can use an encoder neural network configured to map text data to audio to map the text data to semantic lexical units from a vocabulary. In some implementations, the encoder neural network may be a reference... Figure 1 and Figure 8 An embedded neural network is described in further detail. For example, text data may include a text sequence describing music, and the embedded neural network may map the description of the music to audio. The system may use semantic terms corresponding to the audio as at least a portion of the semantic representation of the audio signal. In some examples, the system may enable the semantic representation generating neural network to be conditioned on at least semantic terms when autoregressively generating a semantic representation of the audio signal.

[0169] The system generates an acoustic representation of the audio signal (step 330). The system may use one or more generative neural networks and generate the acoustic representation conditioned on at least a semantic representation. The acoustic representation specifies a set of one or more corresponding acoustic terms at each of a plurality of second time steps spanning a time window. The one or more corresponding acoustic terms at each second time step may represent the acoustic characteristics of the audio signal at the corresponding second time step. The acoustic representation of the generated audio signal is referenced above. Figure 2 A further detailed description was provided.

[0170] In some implementations, the context specifies the acoustic characteristics of the audio signal to be generated, such as when the context specifies the target speaker of the output audio signal, includes the speech to be followed by the output audio signal, or includes audio that should be similar to the output audio signal in another way. In these implementations, the system may, for example, use a neural audio codec as described above to map the context to an acoustic representation, and use at least a portion of the acoustic representation of the context when generating the acoustic representation of the audio signal. The generation of acoustic representations conditional on the context is referenced above. Figure 2 A further detailed description was provided.

[0171] The system processes at least the acoustic representation to generate a prediction of the audio signal (step 340). The system may use a decoder neural network to process at least the acoustic representation to generate the prediction of the audio signal. For example, the corresponding audio sample at each of multiple output time steps across a time window may be based on one or more acoustic lexes of each vector quantizer in the hierarchy.

[0172] In some implementations where the acoustic properties of an audio signal are specified by context, the system may use a decoder neural network to process the acoustic representation and the acoustic representation of the context to generate a prediction of the audio signal. For example, the decoder neural network may process acoustic lexics of the acoustic representation and acoustic lexics representing the context.

[0173] In some implementations, the number of first and second time steps spanning the time window is less than the number of output time steps spanning the time window. In some implementations, the number of first time steps spanning the time window is less than the number of second time steps spanning the time window, as shown in the reference above. Figure 2 Further detailed description.

[0174] Figure 4 This is a diagram of an example process 400 used to train an example audio generation system. For convenience, process 400 will be described as being performed by a training system located on one or more computers in one or more locations.

[0175] The training system can train the audio generation system based on training data, such as... Figure 1 An audio generation system 100. In some implementations, components of the audio generation system, such as a semantic representation generative neural network 130, one or more generative neural networks 108, a neural audio codec 420, and an audio representation neural network 410, can be trained independently.

[0176] The training system can train the neural audio codec 420 and the audio representation neural network 410 based on training data including audio clips. Example audio clips of the training data are shown in... Figure 4 The target audio is shown as 450.

[0177] The neural audio codec 420 may include a decoder neural network and an encoder neural network. For example, the encoder neural network may convert the target audio 450 into a coded signal, which is quantized into an acoustic representation. The decoder neural network may convert the acoustic representation into a predicted audio signal. In some implementations, the neural audio codec 420 may be pre-trained and frozen before training the audio generation system.

[0178] The neural audio codec 420 can be trained to minimize adversarial and reconstruction losses. For example, the decoder neural network and the encoder neural network can be jointly trained based on a target metric: the reconstruction quality of the predicted audio signal generated by the decoder neural network from an acoustic representation, which is generated using the output of the encoder neural network.

[0179] The audio representation neural network 410 can be trained to generate representations of the input audio signal. In some implementations, the audio representation neural network 410 can be pre-trained and frozen before training the audio generation system. The audio representation neural network 410 can be trained to minimize the masked language model (MLM) loss and the contrastive loss. The audio representation neural network 410 can be, for example, a w2v-BERT model that maps the input audio signal to a set of language features.

[0180] The training system can train a semantic representation generative neural network 130 and one or more generative neural networks 108 based on training data including audio clips. In some implementations, the generative neural networks can be trained through forced instruction.

[0181] The training system can train the semantic representation generation neural network 130 to predict semantic representations generated based on the output of one or more layers (e.g., an intermediate layer in the intermediate layers) of the audio representation neural network 410.

[0182] For example, the audio representation neural network 410 may include a model with multiple layers. For example, the audio representation neural network 410 may include a self-attention-based model, such as a Transformer-based model or a Conformer-based model with multiple layers.

[0183] Self-attention-based models can be trained on speech representation tasks, such as through self-supervised learning. Self-attention-based models can also be trained on other tasks, such as automatic speech recognition.

[0184] The output of one or more layers of the audio representation neural network 410 may include embeddings of the input audio. For example, a self-attention-based model may generate dense embeddings for some or all audio samples in the training data. The embeddings of some or all audio samples used in the intermediate layers of the self-attention-based model can be clustered into K clusters using k-means. The centroids of the clusters can be used as semantic units. In some implementations, the output may be normalized before clustering so that each dimension has zero mean and unit variance.

[0185] The example target semantic representation used to train the semantic representation generation neural network 130 can be generated by feeding the target audio 450 to the self-attention-based model. The training system can generate semantic units of the target semantic representation by assigning each output of the target audio 450 to the centroid of the nearest cluster.

[0186] In some implementations, consecutive repetitions of semantic terms within the target semantic representation can be removed. For example, the target semantic representation of target audio 450 includes a sequence of semantic terms. The training system can remove consecutively repeated semantic terms from the sequence and then train using the target semantic representation without consecutively repeated semantic terms.

[0187] The training system can train one or more generative neural networks 108 to generate acoustic representations conditioned on at least a semantic representation. In some implementations, the acoustic representation can be a prediction of a ground truth acoustic representation generated from the output of an encoder neural network by processing an audio signal. For example, the encoder neural network can be a convolutional encoder that maps an audio signal to an embedding sequence. The encoder neural network can output a corresponding embedding at each of a plurality of second time steps. Each corresponding embedding at each of the plurality of second time steps can correspond to a feature of the audio signal at that second time step. By applying quantization to each of the corresponding embeddings, a ground truth acoustic representation can be generated. As described above, the encoder neural network can be part of a neural audio codec such as a neural audio codec 420.

[0188] For example, quantization can be residual vector quantization, which uses a hierarchical structure of multiple vector quantizers to encode each embedding, each vector quantizer generating a corresponding acoustic lexical from its corresponding acoustic lexical vocabulary. The hierarchy may include Q vector quantizers, each using a corresponding vocabulary. The hierarchy may include one or more coarse vector quantizers located at earlier or later positions in the hierarchy, and one or more fine vector quantizers located at later or later positions in the hierarchy. At each second time step, a set of one or more corresponding acoustic lexicals may be included for each vector quantizer, comprising corresponding acoustic lexicals selected from the quantizer's vocabulary and predictions of the ground truth acoustic lexicals generated by the vector quantizer from the ground truth embeddings, which are generated by the encoder neural network at the second time step.

[0189] The training system can train one or more generative neural networks 108 to predict acoustic representations generated by the output of the residual vector quantizer based on the neural audio codec 420.

[0190] The target acoustic representation used to train one or more generative neural networks 108 can be generated by applying quantization to each embedding in the corresponding embedding output by the encoder neural network of the neural audio codec 420 at each of a plurality of second time steps. Quantization can be as described above. Figure 2 The aforementioned residual vector quantization (RVQ).

[0191] Figure 5 This is a diagram illustrating an example process for generating a predicted audio signal. For convenience, process 500 will be described as being executed by a system of one or more computers located in one or more locations. For example, an audio generation system appropriately programmed according to this specification, such as... Figure 1 The audio generation system 100 has an executable process 500.

[0192] The system receives a request to generate an audio signal based on input 502. For example, input 502 may include the information referenced above. Figures 1 to 3 In the context of the aforementioned audio signal. In example process 500, the input includes text data. The system can perform text-conditional music generation such that the generated audio 504 is music described by the text data “hip hop song with violin solo”. The text data may include descriptions of significant complexity, such as “enchanting jazz song with a memorable saxophone solo and a solosinger” or “Berlin 90stechno with a low bass and strong kick”.

[0193] The system uses the above reference. Figure 1 The embedding neural network 120 is used to process text data. The embedding neural network 120 maps text to one or more embedding terms 122. The embedding neural network 120 can be trained to map text and audio to a joint embedding space of text and audio, also known as a joint audio embedding space. The system can use the embedding neural network 120 to generate embedding vectors of the text data input in the joint embedding space. The system can then quantize the embedding vectors to generate the embedding terms 122.

[0194] In some implementations, the embedding neural network 120 may include a text network and a music network. For example, the embedding neural network 120 may include a neural network trained to map text to embeddings in a joint embedding space, and a neural network trained to map audio to embeddings in the joint embedding space. For example, the embedding neural network 120 may be a joint embedding model with two embedding towers, one for text and one for music. The towers use contrastive learning to map text and music to a shared embedding space. See below for reference. Figure 8 Describe the training of the embedded neural network 120.

[0195] The system generates a semantic representation 106 of the audio signal 504. Each semantic unit in the semantic representation 106 is selected from the semantic unit lexicon based on the embedded unit 122.

[0196] For example, the system can use the embedded word 122 as a conditional signal to regressively generate semantic representation 106 using a semantic representation generation neural network. For example, the semantic representation generation neural network can conditionally generate semantic representation 106 with embedded word 122. That is, the semantic word at each first time step can be conditional on the semantic word at the previous first time step and embedded word 122. The semantic representation generation neural network can generate S t |S <t M T S t Let M represent the semantic unit at the first time step t, and M T This indicates the embedded word element 122.

[0197] The system then generates an acoustic representation 110 of the audio signal 504 conditioned on at least semantic representation 106 and embedded lexical units 122. The system may use one or more generative neural networks and conditioned on at least semantic representation 106 and embedded lexical units 122 to generate the acoustic representation. For example, one or more generative neural networks may regressively generate the acoustic representation 110 conditioned on semantic representation 106 and embedded lexical units 122. That is, the set of one or more acoustic lexical units at each second time step may be conditioned on the acoustic lexical units, semantic representation 106, and embedded lexical units 122 at previous second time steps. One or more generative neural networks may generate A t |A <t ,S,M T A t Let S represent the acoustic lexical unit at the first time step t, and let M represent the semantic representation. T This indicates the embedded word 122 generated from the text.

[0198] The system then uses a decoder neural network 112 to process at least the acoustic representation 110 to generate a prediction of the audio signal 504.

[0199] Therefore, the generated audio 504 includes music that can be described by the text data "hip hop song with violin solo" from input 502. Although input 502 in this example includes text data, the system can be trained based on audio signals, as referenced below. Figure 8 As stated above.

[0200] Figure 6 This is a diagram of an example process 600 for generating acoustic lexical units. For convenience, process 600 will be described as being executed by a system of one or more computers located in one or more locations. For example, an audio generation system appropriately programmed according to this specification, such as... Figure 1 The audio generation system 100 has an executable process 600.

[0201] Process 600 and the above reference Figure 2 The similarity of process 200 lies in that the system can use one or more generative neural networks (such as coarse generative neural network 610 and fine generative neural network 620) to generate acoustic lexical representations of acoustic terms. The coarse generative neural network 610 is also referred to as the first generative neural network and is similar to the reference... Figure 2 The coarse generative neural network 210 is described above. The fine generative neural network 620, also known as a second generative neural network, is similar to the reference network. Figure 2 The aforementioned fine generative neural network 620. The system allows the generation of acoustic representations to be conditioned on at least semantic representation 106 and embedded lexical units 122 (also called audio embedded lexical units).

[0202] An encoder neural network can be part of a neural audio codec that includes both a decoder neural network and an encoder neural network. See the reference below. Figure 8 In further detail, the decoder neural network and the encoder neural network can be jointly trained. The coarse generative neural network 610 and the fine generative neural network 620 can be trained to predict the acoustic representation generated by the output of the vector quantizer based on the neural audio codec.

[0203] In some implementations, the set of one or more corresponding acoustic lexics at each of a plurality of second time steps comprises a plurality of acoustic lexics that collectively represent a prediction of the output of residual vector quantization applied to the embedding, the embedding representing the acoustic characteristics of the audio signal at the second time step. Residual vector quantization encodes the embedding using a hierarchy of multiple vector quantizers, each generating a corresponding acoustic lexicogram from its corresponding acoustic lexicon. The hierarchy includes one or more coarse vector quantizers located at one or more preceding positions and one or more fine vector quantizers located at one or more subsequent positions. The set of acoustic lexics at each second time step includes, for each vector quantizer, a corresponding acoustic lexicogram selected from the quantizer's lexicon.

[0204] For example, the hierarchy may include Q vector quantizers, where vector quantizer 1…Q′ can be coarse vector quantizers, and vector quantizers (Q′+1)…Q can be fine vector quantizers. The coarse vector quantizer generates coarse acoustic lexics representing acoustic characteristics such as speaker identity and recording conditions, or acoustic lexics of coarse vector quantizer 612. The fine vector quantizer generates fine acoustic lexics representing fine acoustic details, or acoustic lexics of fine vector quantizer 622. For example, fine acoustic lexics can be used to remove lossy compression artifacts from coarse acoustic lexics.

[0205] In some implementations, the number of first and second time steps spanning the time window is less than the number of output time steps spanning the time window. That is, the sampling rate of the corresponding embedded encoder neural network output at each second time step within the second time step can be lower than the sampling rate of the audio signal. Furthermore, refer to... Figure 7 The sampling rate of the audio representation neural network can be lower than that of the audio signal. Therefore, the number of semantic word embeddings and acoustic word embeddings that can be generated from it can be less than the number of output time steps.

[0206] In some implementations, the number of first time steps spanning the time window is less than the number of second time steps spanning the time window. For example, for each semantic lexical unit, the system may generate two acoustic lexical units for each coarse vector quantizer and two acoustic lexical units for each fine vector quantizer. That is, the sampling rate of the corresponding embedded encoder neural network output at each second time step in the second time step can be higher than the reference sampling rate. Figure 7 The audio represents the sampling rate of the neural network.

[0207] To generate acoustic representations, the coarse generative neural network 610 can generate acoustic lexies of the coarse vector quantizer 612 conditioned on at least the semantic representation 106 and the embedded lexies 122. In some examples, the embedded lexies 122 may include melody embedded lexies, also known as melody audio embedded lexies. In some examples, the embedded lexies 122 may be concatenated with melody embedded lexies.

[0208] For example, for each of one or more coarse vector quantizers in the hierarchy, the coarse generative neural network 610 can generate corresponding acoustic lexical units for a second time step of the vector quantizer, conditioned at least on the semantic representation 106 and the embedded lexical units 122. The acoustic lexical units of the coarse vector quantizer 612 represent acoustic characteristics such as recording conditions.

[0209] The coarse generative neural network 610 can be an autoregressive neural network configured to autoregressively generate acoustic lexics of the coarse vector quantizer 612 according to a first generation order.

[0210] Each specific acoustic lexical of each particular coarse vector quantizer at each particular second time step is conditioned on at least semantic representation 106 and embedded lexical 122, as well as any acoustic lexical preceding the specific acoustic lexical in the first generation order. Furthermore, each specific acoustic lexical of each particular coarse vector quantizer at each particular second time step follows, in the generation order, any acoustic lexical of any coarse vector quantizer at any second time step preceding the specific second time step, and (ii) any acoustic lexical of any coarse vector quantizer at the specific second time step preceding the specific vector quantizer in the hierarchy.

[0211] For example, the hierarchy may include Q vector quantizers, and vector quantizers 1…Q′ may be coarse vector quantizers, while vector quantizers (Q′+1)…Q may be fine vector quantizers. Acoustic lexical units at a specific second time step t and at a specific coarse vector quantizer q≤Q′ are conditional upon: all semantic lexical units of semantic representation 106, all embedded lexical units 122, any acoustic lexical unit at the second time step prior to the second time step t and at the coarse vector quantizer prior to Q′, and any acoustic lexical unit at the second time step t at the coarse vector quantizer prior to the specific coarse vector quantizer q.

[0212] In some implementations, the coarse-coarse generative neural network 610 has a decoder-only Transformer architecture. In other implementations, the coarse-coarse generative neural network 610 has an encoder-decoder Transformer architecture.

[0213] To generate acoustic representations, the fine generative neural network 620 can conditionally generate acoustic terms for the fine vector quantizer 622 based on at least the acoustic terms of the coarse vector quantizer 612. For example, for each of one or more fine vector quantizers in a hierarchy, the fine generative neural network 620 can conditionally generate the corresponding acoustic terms for the second time step of that vector quantizer based on the corresponding acoustic terms for the second time step of one or more coarse vector quantizers in the hierarchy. Therefore, the fine generative neural network 620 may not be conditional on the semantic representation 106 and the embedded terms 122. The acoustic terms of the fine vector quantizer 622 can be used to further improve audio quality, for example, by removing lossy compression artifacts.

[0214] The fine generative neural network 620 may be an autoregressive neural network configured to autoregressively generate acoustic lexical units according to a second generation order. Each specific acoustic lexical unit of each particular fine vector quantizer at each particular second time step is conditional upon: (i) the corresponding acoustic lexical unit of at least a subset of the second time steps of one or more coarse vector quantizers, and (ii) at least a subset of the acoustic lexical units preceding the specific acoustic lexical unit in the second generation order. Each specific acoustic lexical unit of each particular fine vector quantizer at each particular second time step follows the following acoustic lexical units in the second generation order: (i) any acoustic lexical unit of any fine vector quantizer at any second time step preceding the specific second time step, and (ii) any acoustic lexical unit of any fine vector quantizer preceding the specific vector quantizer in the hierarchy at the specific second time step.

[0215] For example, the hierarchy may include Q vector quantizers, and vector quantizers (Q′+1)...Q may be fine vector quantizers. Acoustic lexics at a specific second time step t and at a specific fine vector quantizer q>Q′ are conditional upon: all acoustic lexics of the coarse vector quantizer 612, any acoustic lexics at the second time step prior to the second time step t and after Q′ at the fine vector quantizer, and any acoustic lexics at the second time step t at both the coarse and fine vector quantizers preceding the specific fine vector quantizer q.

[0216] In some implementations, each specific acoustic word at each specific second time step of each particular fine vector quantizer is conditioned on: (i) the corresponding acoustic words of one or more coarse vector quantizers at most a threshold number of second time steps prior to that second time step, and (ii) any acoustic word in the second generation order that is prior to that specific second time step and is at most a threshold number of second time steps prior to that second time step. That is, the second time step can be divided into multiple non-overlapping batches of consecutive second time steps.

[0217] For example, the acoustic lexical at a specific second time step t and a specific fine vector quantizer q>Q′ is conditional on the following: any acoustic lexical before the coarse vector quantizer 612 and within the same corresponding second time step batch as the specific second time step t, and any acoustic lexical at the fine vector quantizer at the second time step before the second time step t and after Q′ within the corresponding batch, and any acoustic lexical at the second time step t of the coarse vector quantizer before the specific fine vector quantizer q and any acoustic lexical at the second time step t of the fine vector quantizer.

[0218] In some implementations, the Generated Neural Network 620 has a decoder-only Transformer architecture. In other implementations, the Generated Neural Network 620 has an encoder-decoder Transformer architecture.

[0219] In some implementations, the system can simultaneously generate acoustic tokens for both the coarse vector quantizer 612 and the fine vector quantizer 622. For example, for a hierarchy of Q vector quantizers, the acoustic token of a particular coarse or fine vector quantizer t at a particular second time step q is conditional upon the following: all acoustic tokens at the vector quantizers before the second time step t and before q, and any acoustic token at the vector quantizer before the particular vector quantizer q at the second time step t.

[0220] The system may include acoustic terms from a fine vector quantizer 622 in the acoustic representation. The system can use the acoustic representation to generate predictions of audio signals, as referenced below. Figure 7 As stated above.

[0221] Figure 7 This is a flowchart of an example process 700 for generating a predicted audio signal. For convenience, process 700 will be described as being executed by a system of one or more computers located in one or more locations. For example, an audio generation system appropriately programmed according to this specification, such as... Figure 1 The audio generation system 100, executable process 700.

[0222] The system receives a request to generate an audio signal conditioned on an input (step 710). The audio signal has corresponding audio samples at each of a plurality of output time steps spanning a time window. The audio signal can be conditioned on an input. For example, the input can be included in a context specified in the request. The input can include an input audio signal, visual data, and / or text data.

[0223] In examples where the input includes text data (such as a text sequence), the prediction of the audio signal could be a prediction of music described by the text sequence.

[0224] In some examples, a text sequence may include multiple text subsequences. The prediction of an audio signal may be a prediction of music that corresponds to and reflects musical segments within each subsequence.

[0225] In an example where the input includes text data and an input audio signal representing a melody, the prediction of the audio signal could be a prediction of music described by a text sequence and following the melody.

[0226] In an example where the input includes an input audio signal representing a melody, the prediction of the audio signal could be a prediction of music following the melody. For example, the input audio signal could represent whistling or humming.

[0227] In an example where the input includes an input audio signal representing music, the prediction of the audio signal can be a continuation of the input audio signal.

[0228] In some examples where the input includes visual data, the system can obtain a textual description of the visual data. In some implementations, the system can generate a textual description of the visual data. The prediction of the audio signal can be a prediction of the music described in the textual description.

[0229] The system processes the input to map it to one or more embedded lexical units (step 715). These embedded lexical units are also called audio embedded lexical units. The system may use an embedding neural network to process the input. For example, the system may use an embedding neural network to generate an embedding vector of the input and quantize the embedding vector to generate embedded lexical units.

[0230] In some examples, the input includes a sequence of text. The embedding neural network can be trained to map text and audio to a joint embedding space, also known as a joint audio embedding space. In examples where the input contains a sequence of text, embedding terms are generated from the text.

[0231] Embedded neural networks can be trained using training data that includes audio signals. The training of embedded neural networks is explained in the reference below. Figure 8 Further detailed description.

[0232] In some examples, the input may include multiple subsequences of text. In these examples, the system may generate embedded lexical units for each subsequence, and thus modify the conditional signal of each subsequence when generating semantic and acoustic representations.

[0233] In some examples, the input includes a text sequence and an input audio signal representing a melody. The system can use a melody embedding neural network to process the input audio signal to map it to one or more melody embedding lexicons, also known as melody audio embedding lexicons. The system can then concatenate the melody embedding lexicons with the embedding lexicons.

[0234] In some examples, the input includes an input audio signal representing a melody. The system can use a melody embedding neural network to process the input audio signal to map the audio signal to one or more melody embedding lexical units. The system can use the melody embedding lexical units as embedding units for conditionalization of the generated semantic and acoustic representations.

[0235] In these examples where the input audio signal represents a melody, in order to process the input audio signal using a melody embedding neural network, the system can use the melody embedding neural network to generate one or more melody embedding vectors of the input audio signal in a joint embedding space (also known as a joint audio embedding space). In some implementations, the melody embedding neural network can be a visual transformer (ViT) that receives time frames of the Melody spectrogram of the input audio signal and generates melody embedding vectors of the input audio signal.

[0236] The system can quantize melody embedding vectors to generate melody embedding lexics. For example, the system can use residual vector quantization to quantize melody embedding vectors into melody embedding lexics. Training of the melody embedding neural network is described in the following reference. Figure 8 Further detailed description.

[0237] The system generates a semantic representation of the audio signal (step 720). The semantic representation specifies a corresponding semantic unit at each of a plurality of first time steps spanning a time window. Each semantic unit is selected from a semantic unit vocabulary conditioned on embedded units and represents the semantic content of the audio signal at the corresponding first time step.

[0238] In some implementations, the system can embed lexical units as conditional signals and use a semantic representation generative neural network to regressively generate semantic representations. This semantic representation generative neural network can also be called a third generative neural network, and is similar to the one referenced above. Figure 3 The semantic representation generative neural network is described above.

[0239] For example, a semantic representation generative neural network can be trained to autoregressively generate semantic lexical units one after another. The semantic representation generative neural network can be, for example, a neural network based on a decoder-only or encoder-decoder Transformer. For example, a semantic representation generative neural network can be trained to predict semantic representations generated based on the output of one or more layers (e.g., an intermediate layer in an intermediate layer) of an audio representation neural network.

[0240] Audio representation neural networks can generate outputs or embeddings at regular time intervals. For example, an audio representation neural network can generate an embedding every 40ms for an input audio signal.

[0241] The training of semantic representation generative neural networks and audio representation neural networks is described in the following reference. Figure 8 Further detailed description.

[0242] In some examples, the input includes a sequence of text, and embedded words are generated from the text. In these examples, the system enables the semantic representation generative neural network to autoregressively generate semantic representations of audio signals with at least embedded words as a condition.

[0243] In some examples, the input may include multiple text subsequences, and the embedded terms include the embedded terms for each subsequence. For example, for a first subsequence, the system may conditioned the semantic representation generating neural network on the embedded terms of the first subsequence when autoregressively generating a semantic representation of the audio signal. For a second subsequence, the system may conditioned the semantic representation generating neural network on the embedded terms of the second subsequence and the semantic terms generated for the first subsequence.

[0244] In some examples, the input includes a text sequence and an input audio signal representing a melody, and embedding terms are generated from the text and melody embedding terms are generated from the input audio signal. The system can concatenate melody embedding terms with the embedding terms. The system can select each semantic term from a semantic term vocabulary conditioned on both the melody embedding terms and the embedding terms. The system can enable a semantic representation generative neural network to autoregressively generate semantic representations of the audio signal conditioned on both the embedding terms and the melody embedding terms.

[0245] In some examples, the input includes an input audio signal representing a melody, and melody embedding lexical units are generated from the input audio signal. In these examples, the system can use melody embedding lexical units as embedding lexical units. The system can select each semantic lexical unit from a semantic lexical vocabulary conditioned on the embedding lexical units. The system can enable a semantic representation generative neural network to conditionally generate semantic representations of audio signals in an autoregressive manner.

[0246] In some examples, the system can generate predictions of audio signals that are longer than the audio signal on which the system was trained. For example, the system can be trained on a 30-second audio signal. To generate longer audio signals, the semantic representation generative neural network can move in 15-second strides, using 15 seconds as a prefix to generate another 15 seconds, while conditioned on the same embedded terms from the input text.

[0247] The system generates an acoustic representation of the audio signal (step 730). The system may use one or more generative neural networks and generate the acoustic representation conditioned on at least semantic representations and embedded lexical units. The acoustic representation specifies a set of one or more corresponding acoustic lexical units at each of a plurality of second time steps spanning a time window. The one or more corresponding acoustic lexical units at each second time step represent the acoustic characteristics of the audio signal at the corresponding second time step. The generated acoustic representation is referenced above. Figure 6 Further detailed description.

[0248] In some implementations, the input includes a sequence of text. One or more generative neural networks may be trained, at least in part, on audio-only training data. During training, the semantic and acoustic representations are conditioned on the embeddings of the audio input in the joint embedding space of text and audio. Training is described in the reference below. Figure 8 Further detailed description.

[0249] The system processes at least the acoustic representation to generate a prediction of the audio signal (step 740). The system may use a decoder neural network to process the acoustic representation. For example, the corresponding audio sample at each of a plurality of output time steps spanning a time window may be based on one or more acoustic lexes of each vector quantizer in a hierarchical structure.

[0250] In some implementations, the number of first and second time steps spanning the time window is less than the number of output time steps spanning the time window. (See reference above.) Figure 6 Further detailed description.

[0251] Figure 8 This is a diagram of an example process 800 used to train an example audio generation system. For convenience, process 800 will be described as being executed by a training system located on one or more computers in one or more locations.

[0252] Process 800 and Reference Figure 4 The similarity of process 400 lies in the fact that the training system can train the audio generation system based on training data, such as... Figure 1An audio generation system 100. In some implementations, components of the audio generation system, such as an embedded neural network 120 and a semantic representation generative neural network 130, can be trained independently. Figure 4 The above are shown as "Generative Neural Network 130" and one or more generative neural networks 108, a neural audio codec 420 and an audio representation neural network 410.

[0253] The embedding neural network 120 can be a joint embedding model with two embedding towers, one for text and one for music. For example, the embedding neural network 120 can be a MuLan model. The towers use contrastive learning to map text and music to a shared embedding space. The text network can be a BERT pre-trained on a large corpus of text-only data. For example, the music network can be a residual convolutional network.

[0254] The embedding neural network 120 can be trained based on training data including audio signals. For example, the training data may include pairs of music clips and corresponding text annotations. The embedding neural network 120 can be trained to associate music with unconstrained natural language descriptions. For example, the embedding neural network 120 can be trained based on a goal such that the text describing the audio signal and the corresponding audio signal have embeddings that are close to each other in the joint embedding space of audio and text. In some implementations, the embedding neural network 120 can be pre-trained and frozen before training the audio generation system.

[0255] Because the audio generation system can be trained on audio-only training data, the training data can be easily scaled. The training data is not limited to audio data with text narration. Furthermore, robustness to noisy text descriptions can be improved by training the embedding neural network 120 based on contrastive loss.

[0256] To generate embedded lexical units for training an audio generation system, the training system can feed a target audio signal 850 to an embedding neural network 120. The embedding neural network 120 can generate a representation of the target audio signal 850 in a joint embedding space. The training system can quantize the representation into discrete embedded lexical units. The embedded lexical units are embeddings of the target audio signal 850 in the joint embedding space.

[0257] In some implementations, the training data may include an audio signal longer than the audio signal on which the embedding neural network 120 is pre-trained to operate. For example, the target audio signal 850 may be 30 seconds long, and the embedding neural network 120 may have been pre-trained to operate on a 10-second sequence. The training system can use the embedding neural network 120 to compute a representation based on a 10-second window of the target audio signal 850 in 1-second strides, and average the resulting representation. The training system can then quantize the averaged representation into discrete embedding units of the target audio signal 850.

[0258] The training system can train the neural audio codec 420 and the audio representation neural network 410 based on training data including audio clips such as music clips. Example audio clips of the training data are shown in... Figure 8 The target audio is shown as 850.

[0259] As per the above reference Figure 4 The neural audio codec 420 may include a decoder neural network and an encoder neural network. For example, the encoder neural network can convert the target audio 850 into a coded signal, which is quantized into an acoustic representation. The decoder neural network can convert the acoustic representation into a predicted audio signal. In some implementations, the neural audio codec 420 may be pre-trained and frozen before training the audio generation system.

[0260] The neural audio codec 420 can be trained to minimize adversarial and reconstruction losses. For example, the decoder neural network and the encoder neural network can be jointly trained based on a target metric: the reconstruction quality of the predicted audio signal generated by the decoder neural network from an acoustic representation, which is generated using the output of the encoder neural network.

[0261] The audio representation neural network 410 can be trained to generate representations of the input audio signal. In some implementations, the audio representation neural network 410 can be pre-trained and frozen before training the audio generation system. The audio representation neural network 410 can be trained to minimize the masked language model (MLM) loss and the contrastive loss. For example, the audio representation neural network 410 can be a w2v-BERT model.

[0262] The training system can train a semantic representation generative neural network 130 and one or more generative neural networks 108 based on training data including audio clips. In some implementations, the generative neural networks can be trained through forced instruction.

[0263] The training system can train a semantic representation generation neural network 130 to predict semantic representations generated from the output of one or more layers (e.g., an intermediate layer) of an audio representation neural network 410. The training system can conditionally train the semantic representation generation neural network 130 using embedded lexical units. The semantic representation generation neural network 130 can predict semantic representations generated by the distribution p(S) t |S <t M A Modeling is performed, where S t Let M represent the semantic unit at the first time step t, and M A This represents embedded words generated from audio.

[0264] For example, the audio representation neural network 410 may include a network with multiple layers. For example, the audio representation neural network 410 may include a self-attention-based model, such as a Transformer-based model or a Conformer-based model with multiple layers.

[0265] Self-attention-based models can be trained on music representation tasks, for example, through self-supervised learning.

[0266] The output of one or more layers of the audio representation neural network 410 may include embeddings of the input audio. For example, a self-attention-based model may generate dense embeddings for some or all audio samples in the training data. The embeddings of some or all audio samples from the intermediate layers of the self-attention-based model can be clustered into K clusters using k-means. The centroids of the clusters can be used as semantic units. In some implementations, the output may be normalized before clustering so that each dimension has zero mean and unit variance.

[0267] The example target semantic representation used to train the semantic representation generation neural network 130 can be generated by feeding the target audio 450 to the self-attention-based model. The training system can generate semantic units of the target semantic representation by assigning each output of the target audio 450 to the centroid of the nearest cluster.

[0268] In some implementations, consecutive repetitions of semantic terms within the target semantic representation can be removed. For example, the target semantic representation of target audio 850 includes a sequence of semantic terms. The training system can remove consecutively repeated semantic terms from the sequence and then train using the target semantic representation without consecutively repeated semantic terms.

[0269] The training system can train one or more generative neural networks 108 to generate acoustic representations conditioned on at least semantic representations and embedded lexical units. In some implementations, the acoustic representation is a prediction of the ground truth acoustic representation generated from the output of an encoder neural network by processing an audio signal. For example, the encoder neural network can be a convolutional encoder that maps an audio signal to a sequence of embeddings. Each corresponding embedding at each of a plurality of second time steps can correspond to a feature of the audio signal at the second time step. By applying quantization to each corresponding embedding, a ground truth acoustic representation can be generated. As described above, the encoder neural network can be part of a neural audio codec, such as a neural audio codec.

[0270] For example, quantization can be residual vector quantization, which uses a hierarchical structure of multiple vector quantizers to encode each embedding, each of which generates a corresponding acoustic lexicon from its corresponding acoustic lexicon vocabulary. The hierarchy may include one or more coarse vector quantizers located at earlier or later positions in the hierarchy, and one or more fine vector quantizers located at later or later positions. The set of one or more corresponding acoustic lexicons at each second time step may include corresponding acoustic lexicons for each vector quantizer, selected from the vector quantizer's vocabulary and predicted by the vector quantizer from the ground truth acoustic lexicons generated by the encoder neural network at the second time step.

[0271] The training system can train one or more generative neural networks 108 to predict acoustic representations generated by the output of the residual vector quantizer based on the neural audio codec 420. The training system can conditionally train one or more generative neural networks 408 using embedded lexical units. One or more generative neural networks 408 can train on a distribution p(A) t |A <t ,S,M A Modeling is performed, where A t Let S represent the acoustic lexical unit at the first time step t, and let M represent the semantic representation. A This represents embedded words generated from audio.

[0272] The target acoustic representation used to train one or more generative neural networks 108 can be generated by applying quantization to each embedding in the corresponding embedding output by the encoder neural network of the neural audio codec 420 at each of a plurality of second time steps. Quantization can be as described above. Figure 6 The aforementioned residual vector quantization (RVQ).

[0273] In some implementations, the audio generation system also includes a melody embedding neural network. This neural network can be trained on audio pairs that match the melody but have different acoustic effects. For example, training data could include different versions of the same music clip, such as covers, instrumental versions, vocal versions, humming versions, and singing versions.

[0274] Melody embedding neural networks can be trained such that when two audio clips contain the same melody, corresponding embeddings in the joint embedding space (also known as the joint audio embedding space) are close to each other. For example, a semi-hard triplet loss can be used to train a melody embedding neural network to generate embeddings of audio that represent the melody in the input audio signal while remaining invariant to the acoustic properties associated with the instrument being played.

[0275] This specification uses the term "configuration" in conjunction with system and computer program components. For a system of one or more computers to be configured to perform specific operations or actions, this means that the system has software, firmware, hardware, or a combination thereof installed thereon that causes the system to perform those operations or actions in operation. For one or more computer programs configured to perform specific operations or actions, this means that the one or more programs include instructions that, when executed by a data processing device, cause that device to perform those operations or actions.

[0276] Embodiments of the subject matter and functional operation described in this specification can be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, or in computer hardware, including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium, for execution by a data processing device or to control the operation of the data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof. Alternatively or additionally, program instructions can be encoded on artificially generated propagation signals (e.g., machine-generated electrical, optical, or electromagnetic signals) to generate information for transmission to a suitable receiver device for execution by the data processing device.

[0277] The term "data processing device" refers to data processing hardware and includes all types of devices, apparatuses, and machines for processing data, such as programmable processors, computers, or multiple processors or computers. The device may also be or further include special-purpose logic circuit systems, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the device may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0278] A computer program, which may also be referred to or described as a program, software, software application, application, module, software module, script, or code, may be written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages); and it may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not necessarily, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinating files (e.g., a file storing one or more modules, subroutines, or code portions). A computer program may be deployed to execute on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected via a data communication network.

[0279] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines may be installed and run on the same one or more computers.

[0280] The processes and logic flows described in this specification can be executed by one or more programmable computers, which execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by a dedicated logic circuit system (e.g., an FPGA or ASIC) or by a combination of a dedicated logic circuit system and one or more programmable computers.

[0281] A computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated into a special-purpose logic circuit system. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or operatively coupled to receive data from or transfer data to or from them, or both. However, a computer does not necessarily have to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0282] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM discs.

[0283] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's device in response to a request received from a web browser. Additionally, the computer can interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and receiving response messages from the user in response.

[0284] Data processing devices used to implement machine learning models may also include, for example, dedicated hardware accelerator units for handling the general and computationally intensive portions of machine learning training or production (i.e., inference) workloads.

[0285] Machine learning frameworks (such as TensorFlow or Jax) can be used to implement and deploy machine learning models.

[0286] Embodiments of the subject matter described in this specification can be implemented in computing systems that include back-end components (e.g., as data servers), or middleware components (e.g., application servers), or front-end components (e.g., client computers having a graphical user interface, web browser, or application through which users can interact with implementations of the subject matter described in this specification), or any combination of one or more such back-end components, middleware components, or front-end components. Components of the system can be interconnected via digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0287] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact via a communication network. The client-server relationship is established by computer programs running on respective computers that establish a client-server relationship between them. In some embodiments, the server transmits data (e.g., HTML pages) to a user device, for example, for the purpose of displaying data to a user interacting with the device acting as a client and receiving user input from that user. Data generated at the user device, such as the result of user interaction, may be received at the server from the device.

[0288] While this specification contains numerous details of specific implementations, these details should not be construed as limiting the scope of any invention or the scope of claims, but rather as descriptions of features specific to particular embodiments of the invention. Certain features described in the context of individual embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases one or more features from the claimed combination may be removed from the combination, and the claimed combination may involve sub-combinations or variations thereof.

[0289] Similarly, although operations are depicted in the accompanying drawings and described in a specific order in the claims, this should not be construed as requiring such operations to be performed in the specific order shown or in a sequential order, or requiring all shown operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0290] Specific embodiments of this subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the figures do not necessarily require the specific or sequential order shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented method for generating predictions for audio signals, the method comprising: Receive a request to generate an audio signal having corresponding audio samples at each of a plurality of output time steps spanning a time window; Obtain a semantic representation of the audio signal, wherein the semantic representation specifies a corresponding semantic unit at each of a plurality of first time steps spanning the time window, and each semantic unit is selected from a semantic unit vocabulary and represents the semantic content of the audio signal at the corresponding first time step; An acoustic representation of the audio signal is generated using one or more generative neural networks conditioned on at least the semantic representation, the acoustic representation specifying a set of one or more corresponding acoustic terms at each of a plurality of second time steps spanning the time window, the one or more corresponding acoustic terms at each second time step representing the acoustic characteristics of the audio signal at the corresponding second time step. as well as A decoder neural network is used to process at least the acoustic representation to generate the prediction of the audio signal.

2. The method as described in claim 1, wherein, The decoder neural network is a decoder neural network of a neural audio codec. The decoder neural network has been jointly trained based on a target and an encoder neural network. The target metric is the reconstruction quality of the predicted audio signal generated by the decoder neural network from an acoustic representation, which is generated using the output generated by the encoder neural network.

3. The method as described in claim 1, wherein, The acoustic representation is a prediction of the true acoustic representation generated from the output of the encoder neural network by processing the audio signal.

4. The method of claim 3, wherein, The encoder neural network outputs a corresponding embedding at each of the plurality of second time steps, wherein the true acoustic representation is generated by applying quantization to each of the corresponding embeddings.

5. The method of claim 4, wherein: The quantization is residual vector quantization, which uses a hierarchical structure of multiple vector quantizers to encode each embedding. Each vector quantizer generates a corresponding acoustic lexicon from its corresponding acoustic lexicon vocabulary. The hierarchical structure includes one or more coarse vector quantizers located at one or more preceding positions and one or more fine vector quantizers located at one or more subsequent positions. The set of one or more corresponding acoustic lexical units at each second time step includes corresponding acoustic lexical units for each vector quantizer, the corresponding acoustic lexical units being selected from the vocabulary of the vector quantizer and being predictions of real value acoustic lexical units generated by the vector quantizer from real value embeddings generated by the encoder neural network at the second time step.

6. The method of claim 1, wherein: The set of one or more corresponding acoustic lexical units at each of the plurality of second time steps comprises a plurality of acoustic lexical units that collectively represent a prediction of the output of the residual vector quantization applied to the embedding, the embedding representing the acoustic characteristics of the audio signal at the second time step. The residual vector quantization uses a hierarchical structure of multiple vector quantizers to encode the embedding. Each vector quantizer generates a corresponding acoustic lexicon from its corresponding acoustic lexicon vocabulary. The hierarchical structure includes one or more coarse vector quantizers located at earlier positions and one or more fine vector quantizers located at later positions. The set of acoustic lexical units at each second time step includes, for each vector quantizer, the corresponding acoustic lexical units selected from the vocabulary of the vector quantizer.

7. The method of claim 6, wherein, Generating an acoustic representation of the audio signal using one or more generative neural networks, conditioned on at least the semantic representation, includes: For each of the one or more coarse vector quantizers in the hierarchy, a first generative neural network is used to generate corresponding acoustic lexical units for the second time step of the coarse vector quantizer, conditioned on at least the semantic representation.

8. The method of claim 7, wherein, The first generative neural network is an autoregressive neural network configured to autoregressively generate the acoustic lexical units according to a first generation order, wherein each specific acoustic lexical unit of each specific coarse vector quantizer at each specific second time step is conditioned on at least the semantic representation and any acoustic lexical units preceding the specific acoustic lexical unit in the first generation order.

9. The method of claim 8, wherein, Each specific acoustic lexical at each specific second time step of each specific coarse vector quantizer is, in the first generation order, following: (i) any acoustic lexical at any second time step preceding any coarse vector quantizer in the first generation order, and (ii) any acoustic lexical at the specific second time step preceding any coarse vector quantizer in the first generation order.

10. The method of claim 7, wherein, The first generative neural network has a decoder-only Transformer architecture or an encoder-decoder Transformer architecture.

11. The method of claim 7, wherein, Generating an acoustic representation of the audio signal using one or more generative neural networks, conditioned on at least the semantic representation, includes: For each of the one or more fine vector quantizers in the hierarchy, a second generative neural network is used to generate the corresponding acoustic lexical at the second time step of the fine vector quantizer, conditioned on the corresponding acoustic lexical at the second time step of the one or more coarse vector quantizers in the hierarchy.

12. The method of claim 11, wherein, The second generative neural network is not conditioned on the semantic representation.

13. The method of claim 11, wherein, The second generative neural network is an autoregressive neural network configured to autoregressively generate the acoustic lexical units according to a second generation order, wherein each specific acoustic lexical unit of each specific fine vector quantizer at each specific second time step is conditioned on: (i) the corresponding acoustic lexical units of at least a subset of the second time step of the one or more coarse vector quantizers, and (ii) at least a subset of the acoustic lexical units preceding the specific acoustic lexical unit in the second generation order.

14. The method of claim 13, wherein, Each specific acoustic lexical at each specific second time step of each specific fine vector quantizer is, in the second generation order, following: (i) any acoustic lexical at any second time step preceding any fine vector quantizer of the fine vector quantizer, and (ii) any acoustic lexical at the specific second time step preceding any fine vector quantizer of the fine vector quantizer in the hierarchy.

15. The method of claim 13, wherein, Each specific acoustic lexical at each specific second time step of each specific fine vector quantizer is conditioned on: (i) the corresponding acoustic lexical of the one or more coarse vector quantizers at a maximum of a threshold number of second time steps prior to the second time step, and (ii) any acoustic lexical in the second generation order at a second time step prior to the specific second time step and at a maximum of a threshold number of second time steps prior to the second time step.

16. The method of claim 11, wherein, The second generative neural network has a decoder-only Transformer architecture or an encoder-decoder Transformer architecture.

17. The method of claim 1, wherein, Obtaining the semantic representation of the audio signal includes: The semantic representation is generated regressively using a third generative neural network.

18. The method of claim 1, wherein, The request specifies the context of the audio signal, and the audio signal is conditional on the context.

19. The method of claim 18, wherein, The context specifies the semantic characteristics of the audio signal, and wherein obtaining the semantic representation of the audio signal includes: The semantic representation is generated based on the context.

20. The method of claim 18, wherein, The context specifies the acoustic properties of the audio signal, and wherein generating the acoustic representation of the audio signal using one or more generative neural networks conditioned on at least the semantic representation includes: The acoustic representation of the audio signal is generated using one or more generative neural networks, conditioned on the semantic representation and the context.

21. The method of claim 20, wherein, The set of one or more corresponding acoustic lexical units at each of the plurality of second time steps comprises a plurality of acoustic lexical units that collectively represent a prediction of the output of the residual vector quantization applied to the embedding, the embedding representing the acoustic characteristics of the audio signal at the second time step. The residual vector quantization uses a hierarchical structure of multiple vector quantizers to encode the embedding. Each vector quantizer generates a corresponding acoustic lexicon from its corresponding acoustic lexicon vocabulary. The hierarchical structure includes one or more coarse vector quantizers located at earlier positions and one or more fine vector quantizers located at later positions. The set of acoustic lexical units at each second time step includes, for each vector quantizer, the corresponding acoustic lexical units selected from the vocabulary of the vector quantizer, and wherein the acoustic representation of the audio signal is generated using one or more generative neural networks conditioned on the semantic representation and the context. For each of the one or more coarse vector quantizers in the hierarchy, a first generative neural network is used to generate the corresponding acoustic lexical for the second time step of the coarse vector quantizer, conditioned on the semantic representation and the context.

22. The method of claim 20, wherein, The predictions made using a decoder neural network to process at least the acoustic representation to generate the audio signal include: The decoder neural network is used to process the acoustic representation and the acoustic representation of the context to generate the prediction of the audio signal.

23. The method of claim 18, wherein, The context includes audio input.

24. The method of claim 18, wherein, The context includes visual data.

25. The method of claim 18, wherein, The context includes text data.

26. The method of claim 1, wherein, The number of first time steps and second time steps spanning the time window is less than the number of output time steps spanning the time window.

27. The method of claim 26, wherein, The number of first time steps spanning the time window is less than the number of second time steps spanning the time window.

28. A system for generating predictions of audio signals, comprising one or more computers and one or more storage devices storing instructions, the instructions, when executed by the one or more computers, causing the one or more computers to perform operations including a method for generating predictions of audio signals, the method comprising: Receive a request to generate an audio signal having corresponding audio samples at each of a plurality of output time steps spanning a time window; Obtain a semantic representation of the audio signal, wherein the semantic representation specifies a corresponding semantic unit at each of a plurality of first time steps spanning the time window, and each semantic unit is selected from a semantic unit vocabulary and represents the semantic content of the audio signal at the corresponding first time step; An acoustic representation of the audio signal is generated using one or more generative neural networks conditioned on at least the semantic representation, the acoustic representation specifying a set of one or more corresponding acoustic terms at each of a plurality of second time steps spanning the time window, the one or more corresponding acoustic terms at each second time step representing the acoustic characteristics of the audio signal at the corresponding second time step. as well as A decoder neural network is used to process at least the acoustic representation to generate the prediction of the audio signal.

29. The system of claim 28, wherein: The set of one or more corresponding acoustic lexical units at each of the plurality of second time steps comprises a plurality of acoustic lexical units that collectively represent a prediction of the output of the residual vector quantization applied to the embedding, the embedding representing the acoustic characteristics of the audio signal at the second time step. The residual vector quantization uses a hierarchical structure of multiple vector quantizers to encode the embedding. Each vector quantizer generates a corresponding acoustic lexicon from its corresponding acoustic lexicon vocabulary. The hierarchical structure includes one or more coarse vector quantizers located at earlier positions and one or more fine vector quantizers located at later positions. The set of acoustic lexical units at each second time step includes, for each vector quantizer, the corresponding acoustic lexical units selected from the vocabulary of the vector quantizer.

30. The system of claim 29, wherein, Generating an acoustic representation of the audio signal using one or more generative neural networks, conditioned on at least the semantic representation, includes: For each of the one or more coarse vector quantizers in the hierarchy, a first generative neural network is used to generate corresponding acoustic lexical units for the second time step of the coarse vector quantizer, conditioned on at least the semantic representation.

31. The system of claim 30, wherein, The first generative neural network is an autoregressive neural network configured to autoregressively generate the acoustic lexical units according to a first generation order, wherein each specific acoustic lexical unit of each specific coarse vector quantizer at each specific second time step is conditioned on at least the semantic representation and any acoustic lexical units preceding the specific acoustic lexical unit in the first generation order.

32. The system of claim 30, wherein, Generating an acoustic representation of the audio signal using one or more generative neural networks, conditioned on at least the semantic representation, includes: For each of the one or more fine vector quantizers in the hierarchy, a second generative neural network is used to generate the corresponding acoustic lexical at the second time step of the fine vector quantizer, conditioned on the corresponding acoustic lexical at the second time step of the one or more coarse vector quantizers in the hierarchy.

33. The system of claim 32, wherein, The second generative neural network is an autoregressive neural network configured to autoregressively generate the acoustic lexical units according to a second generation order, wherein each specific acoustic lexical unit of each specific fine vector quantizer at each specific second time step is conditioned on: (i) the corresponding acoustic lexical units of at least a subset of the second time step of the one or more coarse vector quantizers, and (ii) at least a subset of the acoustic lexical units preceding the specific acoustic lexical unit in the second generation order.

34. The system of claim 28, wherein, Obtaining the semantic representation of the audio signal includes: The semantic representation is generated regressively using a third generative neural network.

35. The system of claim 28, wherein, The request specifies the context of the audio signal, and the audio signal is conditional on the context.

36. The system of claim 35, wherein, The context specifies the semantic characteristics of the audio signal, and wherein obtaining the semantic representation of the audio signal includes: The semantic representation is generated based on the context.

37. The system of claim 35, wherein, The context specifies the acoustic properties of the audio signal, and wherein generating the acoustic representation of the audio signal using one or more generative neural networks conditioned on at least the semantic representation includes: The acoustic representation of the audio signal is generated using one or more generative neural networks, conditioned on the semantic representation and the context.

38. The system of claim 35, wherein, The context includes audio input.

39. One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations, said operations including a method for generating a prediction of an audio signal, said method comprising: Receive a request to generate an audio signal having corresponding audio samples at each of a plurality of output time steps spanning a time window; Obtain a semantic representation of the audio signal, wherein the semantic representation specifies a corresponding semantic unit at each of a plurality of first time steps spanning the time window, and each semantic unit is selected from a semantic unit vocabulary and represents the semantic content of the audio signal at the corresponding first time step; An acoustic representation of the audio signal is generated using one or more generative neural networks conditioned on at least the semantic representation, the acoustic representation specifying a set of one or more corresponding acoustic terms at each of a plurality of second time steps spanning the time window, the one or more corresponding acoustic terms at each second time step representing the acoustic characteristics of the audio signal at the corresponding second time step. as well as A decoder neural network is used to process at least the acoustic representation to generate the prediction of the audio signal.

Images