Audio generation using non-autoregressive decoding

Abstract

A method, system, and apparatus are provided that include a computer program encoded on a computer storage medium for generating an output audio signal. In some embodiments, the output audio signal may be generated by using a generative neural network with a number of iterations independent of the length of the output audio signal.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims priority to U.S. Provisional Application No. 63 / 462,498, filed Apr. 27, 2023, and U.S. Provisional Application No. 63 / 465,332, filed May 10, 2023. The disclosure of the prior applications is considered a part of the disclosure of this application and is incorporated herein by reference.

Background Art

[0002] This specification relates to generating audio using a neural network.

[0003] A neural network is a machine - learning model that uses one or more layers of non - linear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as an input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values of its respective set of parameters.

Summary of the Invention

[0004] This specification describes an audio generation system implemented as a computer program on one or more computers at one or more locations that generates an output audio signal. In some embodiments, the audio generation system can generate the output audio signal with a number of iterations that does not depend on the length of the output audio signal.

[0005] In one embodiment, a method is provided which is performed by one or more computers, the method comprising: receiving a masked representation of an audio signal as a sequence of tokens, the sequence of tokens comprising one or more masked tokens; processing the masked representation of the audio signal using a neural network, the method comprising processing the masked representation of the audio signal using an embedding layer of the neural network to generate a sequence of embeddings comprising an embedding for each token in the sequence of tokens; processing the sequence of embeddings using a pooling layer of the neural network to generate a sequence of pooled embeddings, each pooled embedding in the sequence of pooled embeddings being a result of pooling each of the multiple embeddings from the sequence of embeddings; processing the sequence of pooled embeddings using an attention block of the neural network to update the sequence of pooled embeddings, the method comprising applying one or more self-attention operations to the sequence of pooled embeddings; and processing the updated sequence of pooled embeddings to generate, for each of one or more positions in the sequence of tokens, a prediction characterizing the token that should occupy the position in the sequence of tokens.

[0006] In some embodiments, the attention block is a convolutional extension attention block comprising one or more convolutional layers and one or more self-attention layers.

[0007] In some embodiments, applying one or more self-attention actions to a pooled sequence of embeddings includes applying one or more local self-attention actions to a pooled sequence of embeddings.

[0008] In some embodiments, applying one or more local self-attention actions to a pooled sequence of embeddings includes sequentially applying multiple local self-attention actions to a pooled sequence of embeddings.

[0009] In some embodiments, receiving a representation of an audio signal as a sequence of tokens includes receiving a sequence of acoustic tokens generated by a neural audio codec that includes multiple residual vector quantizers.

[0010] In some embodiments, receiving a representation of an audio signal as a sequence of tokens includes receiving a sequence of semantic tokens generated by an intermediate layer of an auxiliary audio processing neural network.

[0011] In some embodiments, receiving a sequence of semantic tokens includes resampling the sequence of semantic tokens based on the sampling rate of the sequence of acoustic tokens.

[0012] In some embodiments, processing a sequence of embeddings using a pooling layer to generate a pooled sequence of embeddings involves the pooling layer processing the sequence of embeddings according to a stride that depends on the total number of residual vector quantizers.

[0013] In some embodiments, the stride value is equal to the total number of residual vector quantizers.

[0014] In some embodiments, the output block of the neural network includes multiple output heads, and processing an updated pooled embedding sequence to generate a respective prediction for each of one or more locations in the sequence of tokens that should occupy a location in the sequence of tokens, includes processing the updated pooled embedding sequence independently and in parallel with other output heads, for each output head to generate a respective prediction for multiple tokens that are located at different locations in the sequence of tokens.

[0015] In some embodiments, each output head is configured to receive each pooled embedding from an updated sequence of pooled embeddings and to process the pooled embeddings to generate each prediction for each position in the same frame of the sequence of tokens, wherein the output head is configured to generate and process predictions for the same frame positions in parallel.

[0016] In some embodiments, the method further includes selecting a plurality of positions in a sequence of tokens to be unmasked, where each position selected to be unmasked is occupied by a masked token, and for each of the plurality of positions selected to be unmasked, determining which unmasked token occupies the position based on predictions generated by a neural network for the token that should occupy the position.

[0017] In some embodiments, selecting multiple positions in a sequence of tokens to be demasked includes, for each position in the sequence of tokens occupied by a masked token, deciding whether to select a position to be demasked based on the confidence of a prediction generated by a neural network characterizing the token that should occupy that position in the sequence of tokens.

[0018] In some embodiments, some or all of the positions in a sequence of tokens are associated with each residual vector quantizer in a sequence of residual vector quantizers, and selecting multiple positions in a sequence of tokens to be unmasked involves determining, for each position in a sequence of tokens occupied by a masked token, whether to select a position to be unmasked based on the residual vector quantizer associated with the position.

[0019] In some embodiments, selecting multiple positions in a sequence of tokens to be demasked includes identifying a set of masked tokens that are eligible to be demasked, which includes selecting each masked token associated with a particular residual vector quantizer to be included in the set of masked tokens that are eligible to be demasked, and simultaneously selecting multiple tokens from the set of masked tokens that are eligible to be demasked as tokens to be demasked.

[0020] In some embodiments, all tokens associated with any residual vector quantizer preceding a particular residual vector quantizer in a sequence of residual vector quantizers are already unmasked, while all tokens associated with any residual vector quantizer following a particular residual vector quantizer in a sequence of residual vector quantizers are not yet unmasked.

[0021] In some embodiments, for each of the multiple locations selected to be unmasked, the prediction of which token should occupy the location, generated by the neural network, includes a score distribution across the set of tokens.

[0022] In some embodiments, for each of a set of locations selected to be unmasked, determining the unmasked tokens that occupy the location involves sampling tokens from a score distribution of the set of tokens.

[0023] In some embodiments, the method comprises, for each of one or more positions of a sequence of tokens occupied by masked tokens, determining a first difference between (i) the token to occupy the position and (ii) each prediction characterizing the token to occupy the position, and determining an update to a value of a set of parameters of a neural network based at least on the first difference.

[0024] In some embodiments, the method further comprises, for each of one or more positions of a sequence of tokens occupied by unmasked tokens, determining a second difference between (i) the token to occupy the position and (ii) each prediction characterizing the token to occupy the position, and determining an update to a value of a set of parameters of a neural network based on both the first difference and the second difference.

[0025] In some embodiments, determining the second difference comprises downweighting the second difference relative to the first difference.

[0026] In some embodiments, the method further comprises generating a masked representation of an audio signal as a sequence of tokens, the generating comprising determining a start time step within a plurality of time steps of the audio signal and replacing tokens of a predetermined number of consecutive time steps following the start time step with masked tokens.

[0027] In some embodiments, the method further includes generating a masked representation of an audio signal as a sequence of tokens, obtaining a representation of the audio signal as a sequence of tokens generated by a sequence of residual vector quantization layers of a neural audio codec, selecting a target vector quantization layer, and masking only the tokens associated with the target vector quantization layer or a vector quantization layer following the target vector quantization layer in a sequence of vector quantization layers.

[0028] In some embodiments, the method further includes generating an output audio signal based at least in part on each unmasked token.

[0029] According to other aspects, one or more non-transitory computer storage media storing instructions are provided, which, when executed by one or more computers, cause the one or more computers to perform the methods described herein.

[0030] According to other aspects, a system is provided that includes one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform respective operations of the methods described herein.

[0031] Particular embodiments of the subject matter described herein can be implemented so as to realize one or more of the following advantages.

[0032] This specification describes a system that uses generative neural networks to generate high-quality, semantically consistent audio. Compared to existing audio generation systems, the system described can generate audio that is longer in duration and has a wider variety of semantic representations, such as not only speech but also music, ambient sounds, acoustic events, or both. By relying on local attention, the complexity and inference cost of the model depend linearly, rather than quadratically (as in the case of self-attention), on the length of the model input. Therefore, by using the system described, processor cycles, memory, and power consumption are reduced while generating audio of a similar duration to that produced by existing systems.

[0033] Furthermore, the system is configured to generate audio data by a non-autoregressive decoding procedure, particularly a parallel decoding procedure, where multiple tokens (implicitly representing audio data) are predicted simultaneously with higher quality in a single forward pass, resulting in significant speed improvements, for example, in terms of wall clock time, and not only that, the computing resources consumed are reduced compared to what would otherwise be required by autoregressively predicting audio tokens one after another. In some embodiments, the audio generation system can generate the output audio signal in exactly a predetermined number of iterative time steps, regardless of the length of the output audio signal.

[0034] Advantageously, in some embodiments, the generative neural network generates tokens representing code vectors that can be generated by a hierarchy of multiple vector quantizers, which are further processed by a decoder neural network to generate output audio, and the parallel decoding configuration allows the system to reduce the consumption of computing resources when generating output audio by leveraging the hierarchical ordering of the multiple vector quantizers. For example, multiple tokens associated with the same vector quantizer can be generated in parallel.

[0035] More specifically, some conventional systems autoregressively generate audio data by, for example, starting with a first audio sample and generating a sequence of audio samples one at a time, with each audio sample being generated in a manner conditioned on previously generated audio samples. However, generating high-quality audio data may require very long sequences of audio data samples, for example, generating more than 40,000 audio data samples for one second of audio. Autoregressively generating long sequences of audio data samples can be computationally inefficient and time-consuming, for example, because it requires running a new forward pass through the audio generation system for each new sample.

[0036] The system described herein addresses these problems by enabling the efficient generation of long sequences of audio data through a parallel decoding procedure. Specifically, the system can generate audio data by iteratively demasking a sequence of tokens representing the code vectors of a residual vector quantizer codebook. The system can then process the sequence of tokens to generate a compressed representation of the audio signal, and then process this compressed representation with a decoder neural network to generate the complete audio signal.

[0037] The sequence of tokens (i.e., iteratively unmasked by the system) is divided into “frames,” each containing the respective tokens for each vector quantizer in the sequence of vector quantizers, for example, a residual vector quantizer (RVQ). The system can unmask the sequence of tokens level by level, for example, by first unmasking all tokens associated with the first vector quantizer, then all tokens associated with the second vector quantizer, then all tokens associated with the third vector quantizer, and so on. (The “level” of the tokens can refer to the position of the vector quantizer associated with the tokens in the sequence of vector quantizers). The system can unmask tokens at a specific level by a parallel decoding procedure. For example, multiple tokens may be unmasked simultaneously at a given level. Level-by-level unmasking of the sequence of tokens takes into account the conditional dependencies between levels in the RVQ hierarchy and utilizes the conditional independence of tokens from “fine” (i.e., later) levels given all tokens from “coarse” (i.e., earlier) levels.

[0038] Details of one or more embodiments of the subject matter of this specification are described in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from this specification, the drawings, and the claims. [Brief explanation of the drawing]

[0039] [Figure 1A] An exemplary audio generation system is shown. [Figure 1B] An exemplary audio generation system is shown. [Figure 2] This is an illustrative diagram of the operations performed by a generative neural network. [Figure 3] This is an illustrative process flowchart for processing a masked representation of an audio signal using a neural network. [Figure 4A]This specification provides quantitative examples of the performance gains that can be achieved by the audio generation systems described herein. [Figure 4B] This specification provides quantitative examples of the performance gains that can be achieved by the audio generation systems described herein. [Modes for carrying out the invention]

[0040] Similar reference numbers and names in various drawings refer to the same elements.

[0041] Figure 1A is a block diagram of an exemplary audio generation system 100. The audio generation system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations where the systems, components, and technologies described below are implemented.

[0042] The audio generation system 100 generates an output audio signal 104 in response to a request 102 for generating an audio signal. In some embodiments, the request 102 can specify the context for the audio signal 104. In these embodiments, the output audio signal 104 is context-dependent.

[0043] Generally, the output audio signal 104 is an example of output audio containing samples of sound waves in each of a sequence of output time steps spanning a specified time window. For example, the output time steps can be placed at regular intervals within the specified time window.

[0044] An audio sample at a given time step can be the amplitude value of a sound wave, or an amplitude value that has been compressed, decompressed, or both. For example, an audio sample can be the raw amplitude value, or a mu-law decompressed representation of the amplitude value.

[0045] The audio generation system 100 can be configured to perform any of a variety of tasks that require generating an audio signal as an output.

[0046] For example, the output audio signal 104 could be a speech signal, and the system could unconditionally generate a speech signal, for example, resulting in a speech signal extracted from a distribution represented by the training dataset(s) on which the generative neural network(s) were trained. In other examples, the output audio signal 104 could be a different type of audio signal, such as music, ambient sounds, or sound events, and the system could unconditionally generate an audio signal, for example, resulting in an audio signal extracted from a distribution represented by the training dataset(s) on which the generative neural network(s) were trained.

[0047] As another example, the audio generation system 100 can receive a context along with a request 102 to generate an audio signal, and generate an output audio signal 104 conditional on the received context.

[0048] For example, the output audio signal 104 could be a voice signal or another audio signal; that is, the system generates the output audio 104 given the context of an input audio signal.

[0049] For example, the context can include an audio input as the input audio signal. In some examples, the audio input may contain words spoken by a specific speaker, and the output audio signal 104 may be a continuation of the words spoken by that specific speaker. In some examples, the audio signal may contain music, and the output audio signal 104 may be a continuation of the music from the audio input.

[0050] In some embodiments, the generated output audio signal 104 may be a prediction of an audio signal following an input audio signal. For example, the context may be an input audio signal which is a question asked by one speaker, and the output audio signal may be an output audio signal which is an answer to a question spoken by the same speaker or another speaker. In another example, the context may be an input audio signal which is the first part of an utterance spoken by one speaker, and the input audio signal may be the completion of an utterance spoken by the speaker or another speaker, or an output audio signal which is a response to the input utterance.

[0051] In some embodiments, the input audio signal may include a melody, and the output audio signal 104 may be music that follows the melody. The system can generate the output audio signal 104, which is music that follows the melody.

[0052] As another example, the audio generation system 100 can perform speech separation on an input audio signal to generate an output audio signal 104. For example, the input audio signal may contain both speech and music, or other background noise, and the output audio signal may represent only speech. As yet another example, the input audio signal may contain speech from multiple speakers (and optionally background noise), and the output audio signal 104 may contain only the speech of one of the speakers. In some examples, the system may perform conditional audio separation, i.e., when the input audio signal may contain additional audio inputs that are acoustically similar to one of the sources.

[0053] As another example, the audio generation system 100 can perform speech-to-speech conversion in which the input and output speech represent the same semantic content but are spoken differently. For example, the input audio signal may contain speech in one natural language, and the output audio signal 104 may represent speech in another natural language of a target, which is a conversion from the input speech to the target language. As yet another example, the input audio signal may contain speech spoken by a first speaker, and the output audio signal 104 may represent speech spoken by a different speaker, but with the same semantic content as the input speech. As yet another example, the input audio signal may contain a first speech segment, and the output audio signal 104 may contain a second, shorter speech segment that summarizes the semantic content of the first speech.

[0054] As another example, the context can include both audio and text data. In this example, the output audio signal 104 can include speech that reflects the text data. In some embodiments, the output audio signal 104 can include music that reflects the text data. In these embodiments, the system generates the output audio signal 104 that reflects the text data.

[0055] For example, the output generation system 100 can perform transcript-conditional speech enhancement, where the context is a text transcript and noisy audio corresponding to the text transcript, and the output audio signal 104 is clean audio corresponding to the text transcript.

[0056] As another example, the output generation system 100 can perform transcript-based audio infill where the context is audio corresponding to a text transcript and a portion of the text transcript, and the output audio signal 104 corresponds to another portion of the text transcript.

[0057] As another example, the output generation system 100 can perform text-to-speech, where the context is a sequence of texts and the output audio signal 104 is the verbalization of the sequence of texts. In that particular example, the output generation system 100 can perform speaker-conditional speech conversion, where the context is a text transcript and speaker audio, and the output audio signal 104 is the verbalization of the text transcript spoken by the speaker.

[0058] As another example, the context can include both audio data and visual data, such as images or video data. In this example, the output audio signal 104 could include audio describing the visual data or music reflecting the visual data.

[0059] For example, the output generation system 100 can perform an audio-video sequence, the system receives a partial audio track along with the corresponding video, and the output audio signal 104 is a sequence of partial audio tracks.

[0060] As another example, the output generation system 100 can perform cross-modal infill, where the system receives video and audio tracks corresponding to portions of the video, and the output audio signal 104 is an audio track corresponding to another portion of the video.

[0061] As another example, context input may contain only visual data.

[0062] For example, the output generation system 100 can perform image-conditional audio generation, where the system receives an input image and generates an output audio signal 104 that describes the image.

[0063] As another example, context input can contain only text data.

[0064] For example, the output generation system 100 can perform sound synthesis based on a text description, where the input is text describing an audio signal and the output is an output audio signal 104 characterized by the text.

[0065] As shown in Figure 1B, in order to generate the output audio signal 104, the audio generation system 100 implements a pre-processing engine 108, a generative neural network 112, and a post-processing engine 116.

[0066] At a high level, the audio generation system 100 uses a generative neural network 112 to generate a sequence of output tokens 114 from a sequence of input tokens 110 over multiple iterations. As will be described in more detail below, in some embodiments, the generative neural network 112 can be used to generate a sequence of output tokens 114 in a strictly fixed number of iterations, regardless of the length of the sequence of output tokens 114. This is faster and more computationally efficient than autoregressively generating output tokens one after another.

[0067] Prior to the first iteration of the operation performed by the generative neural network 112, the preprocessing engine 108 generates a sequence of input tokens 110 containing each of the input tokens at each of the multiple positions in the sequence of input tokens 110.

[0068] The positions generally correspond to time steps spanning a specified time window of the output audio signal 104. In some embodiments, the correspondence is one-to-one, and thus each position may correspond to a certain time step. In other embodiments, the correspondence is many-to-one, and thus two or more positions may correspond to a certain time step. In any embodiment, the positions can be divided into multiple frames (or segments), each of which may contain a fixed number of positions.

[0069] Throughout this specification, “embedding” refers to an ordered set of numbers, such as a vector of numbers, a matrix, or a tensor of other numbers (e.g., integers or floating-point numbers). “Token” can represent an individual data element from a separate (e.g., finite) set of data elements. For example, a token can represent one of several code vectors in a quantizer codebook, such as a vector quantizer codebook contained in a residual (i.e., multi-stage) vector quantizer (RVQ). For this purpose, tokens may also be in the form of an ordered set of numbers. For example, each token may contain one or more numbers.

[0070] Throughout this specification, “residual vector quantizer” (RVQ) may refer to a multi-stage vector quantization technique based on a sequence of (residual) vector quantizers. A vector quantizer can quantize an input vector, for example, by identifying the code vector that is least far from the input vector, according to a distance metric (e.g., based on the L1 norm), from a codebook of code vectors associated with the vector quantizer. A residual vector quantizer can quantize an input vector (or “signal”) by iteratively quantizing the residuals from previous quantization stages. Thus, each stage of a residual vector quantizer encodes the difference (or residual) between the original signal and the reconstructed signal from the previous stage, thereby progressively refining the approximation of the original signal with each step.

[0071] The sequence of input tokens 110 includes a masked token. A "masked token" is a token that contains a predetermined number and means that the corresponding token in the sequence of input tokens 110 has not yet been generated, for example, has not yet been selected from a given set of tokens.

[0072] The preprocessing engine 108 can generate a sequence of input tokens 110 that is the same length as the sequence of output tokens 114. That is, the number of positions in the sequence of input tokens 110 is the same as the number of positions in the sequence of output tokens 114. The length of the sequence of output tokens 114 can be predetermined or specified by the user of the audio generation system 100.

[0073] In some embodiments, the sequence of input tokens 110 consists entirely of masked tokens, that is, each of the multiple positions in the sequence of input tokens 110 contains a masked token.

[0074] In some embodiments, the sequence of input tokens 110 includes both masked tokens and conditional tokens, for example, each of a subset of positions in the sequence of input tokens 110 includes a masked token, and each of the other subsets of positions in the sequence of input tokens 110 includes a conditional token. In other words, each position in the sequence of input tokens 110 is occupied by either a masked token or a conditional token.

[0075] In some of these embodiments, the sequence of input tokens 110 is arranged in a specific order. For example, the sequence of input tokens 110 includes a conditional token followed by a masked token. That is, the input tokens are masked after a certain time step (i.e., corresponding to the end of the context).

[0076] The sequence of input tokens 110 may include conditional tokens when the audio generation system 100 receives a context in addition to the request 102, such as an input audio signal, text data, visual data, or any combination thereof. In other words, the conditional tokens may be generated from the context.

[0077] In some of these embodiments, conditional tokens include semantic tokens. If included, semantic tokens define the semantic representation of the output audio signal. For example, a semantic token at a given position may represent the semantic content of the output audio signal at the time step corresponding to that position. Each semantic token can be selected from a vocabulary of semantic tokens. Examples of semantic content that can be represented by tokens include the linguistic content of an utterance, as well as the melody and rhythm of music.

[0078] Semantic tokens can be generated in one of several ways. For example, they may be generated by an auxiliary audio processing neural network included in or accessible by the preprocessing engine 108. The auxiliary audio processing neural network may be a self-attention-based model, such as a transformer-based or conformer-based model, such as the W2v-BERT neural network (described in Chung, Yu-An, et al. “W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training.” 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)). The preprocessing engine 108 can generate semantic tokens based on the output of one or more layers of the auxiliary audio processing neural network, such as one of the hidden layers.

[0079] In some of these embodiments, the conditional tokens include acoustic tokens. If included, the acoustic tokens define the acoustic representation of the output audio signal. For example, an acoustic token at a given position may represent the acoustic characteristics of the output audio signal at a time step corresponding to a given position. The acoustic characteristics capture the details of the audio waveform, enabling high-quality synthesis. The acoustic characteristics may include, for example, speaker identification. The acoustic characteristics may also include recording conditions such as reverberation level, distortion, and background noise.

[0080] Acoustic tokens can be generated in one of several ways. For example, acoustic tokens may be generated by a neural audio codec included in or accessible by the preprocessing engine 108. Examples of neural audio codecs and the operations they perform are described in more detail in Borsos, Zalan, et al. “Audiolm: a language modeling approach to audio generation,” IEEE / ACM Transactions on Audio, Speech, and Language Processing (2023), Kharitonov, Eugene, et al. “Speak, read and prompt: High-fidelity text-to-speech with minimal supervision,” Transactions of the Association for Computational Linguistics 11 (2023): 1703-1718, and Agostinelli, Andrea, et al. “Musiclm: Generating music from text.” arXiv preprint arXiv:2301.11325 (2023).

[0081] Such a neural audio codec can generate acoustic tokens using one or more vector quantizers, for example, a residual vector quantizer including a cascade of multiple vector quantizers, and is conditioned on other conditional tokens, for example, embedded tokens generated from context, such as input audio signals or text data, semantic tokens, or both.

[0082] In this example, the neural audio codec encodes conditional tokens using a hierarchy of vector quantizers, each generating an acoustic token from a corresponding codebook of the vector quantizer's token vector. The hierarchy includes one or more coarse vector quantizers at one or more initial positions in the hierarchy, and one or more fine vector quantizers at one or more final positions in the hierarchy. The set of acoustic tokens at a particular position within a frame includes, for each vector quantizer, the respective acoustic tokens selected from the codebook for that vector quantizer.

[0083] For example, the hierarchy may include Q vector quantizers arranged in the order 1...Q', (Q'+1)...Q, where vector quantizer 1...Q' may be a coarse vector quantizer and vector quantizer (Q'+1)...Q may be a fine vector quantizer. A coarse vector quantizer can generate coarse acoustic tokens or acoustic tokens of a coarse vector quantizer that represent acoustic characteristics such as speaker identification and recording conditions. A fine vector quantizer can generate fine acoustic tokens or acoustic tokens of a fine vector quantizer that can represent fine acoustic details. For example, fine acoustic tokens can be used to remove lossy compression artifacts within coarse acoustic tokens.

[0084] In some of these embodiments, the conditional tokens include both acoustic and semantic tokens. When the position of the sequence of input tokens 110 is divided into multiple frames, the preprocessing engine 108 can generate a sequence of input tokens 110 in a temporally aligned manner, for example, to include both acoustic and semantic tokens, such that each frame includes an equal number of acoustic and semantic tokens.

[0085] In practice, acoustic tokens and semantic tokens may be generated at different sampling rates, so the preprocessing engine 108 can convert the sampling rate of one token to that of another token, for example, through resampling. For example, assuming that acoustic tokens have a higher sampling rate than semantic tokens, the preprocessing engine 108 can upsample the semantic tokens to match the higher sampling rate of the acoustic tokens when generating conditional tokens. In some embodiments, semantic tokens can be upsampled by appropriately duplicating the tokens.

[0086] At that time, the audio generation system 100 uses a generative neural network 112 to generate a sequence of output tokens 114 from a sequence of input tokens 110 over multiple iterations. Similar to the sequence of input tokens 110, the sequence of output tokens 114 contains each output token at each of several positions in the sequence of output tokens 114, but the tokens present at these positions do not contain any masked tokens. In other words, the audio generation system 100 generates the sequence of output tokens 114 by gradually unmasking all of the masked tokens that were initially included in the sequence of input tokens 110.

[0087] During each iteration, the audio generation system 100 performs a forward pass through the generative neural network 112, that is, it uses the generative neural network 112 to process the network input according to its parameters and generate an updated sequence of input tokens. In the first iteration, the network input contains a sequence of input tokens 110. For each subsequent iteration, the network input contains an updated sequence of input tokens generated in the previous iteration.

[0088] Next, in each iteration, the audio generation system 100 uses a generative neural network 112 to process the network input and generate one or more new tokens that replace each masked token in the sequence of input tokens 110. That is, in each iteration, the generative neural network 112 is used to generate an updated sequence of input tokens having fewer masked tokens.

[0089] After the last iteration, the audio generation system 100 uses the updated sequence of input tokens generated in the last iteration as the sequence of output tokens 114, and uses the post-processing engine 116 to generate an output audio signal 104 that satisfies the request 102 from the sequence of output tokens 114.

[0090] Next, the audio generation system 100 outputs an output audio signal 104. For example, the audio generation system 100 can provide the output audio signal 104 to present or play to the user via a speaker or other audio output device, or it can store the output audio signal 104 for later use.

[0091] To generate the output audio signal 104, the post-processing engine 116 implements or accesses software that can convert the tokens into audio waveforms representing the audio signal, for example, time-domain audio waveforms that define the amplitude of the audio signal over time. For example, the software may include a detokenizer, as mentioned in Borsos, Z., Marinier, et al. Audiolm: a language modeling approach to audio generation. arXiv:2209.03143, 2022.

[0092] Therefore, the sequence of output tokens 114 is decodeable into an audio signal. In some embodiments, the output tokens correspond to the compressed encoding of the audio signal. The output tokens can be decoded / restored as needed to generate the output audio signal 104. In embodiments where the tokens represent code vectors from the residual vector quantizer (RVQ) codebook, for each frame in the sequence of output tokens 114, the post-processing engine 116 can combine, e.g., concatenate or sum the code vectors represented by the tokens in the frame, and then process at least the combined code vectors using a decoder neural network to generate a prediction of the output audio signal 104. For example, the decoder neural network could be the SoundStream decoder neural network mentioned in Zeghidour, Neil, et al. “Soundstream: An end-to-end neural audio codec.” IEEE / ACM Transactions on Audio, Speech, and Language Processing 30 (2021): 495-507.

[0093] Figure 2 illustrates an exemplary operation performed by the generative neural network 112 in a given iteration. The generative neural network 112 includes an embedding layer 111, followed by a pooling layer 121, followed by one or more attention blocks 140, and followed by an output block 160.

[0094] The generative neural network 112 receives network inputs during a given iteration. When the given iteration is the first iteration, the network inputs consist of a sequence of input tokens 110. When the given iteration is a subsequent iteration, the network inputs consist of an updated sequence of input tokens generated in the previous iteration.

[0095] The embedding layer 111 processes the network input to generate a sequence of embeddings 120. The sequence of embeddings includes the respective embedding for each token in the sequence of input tokens 110 (or the updated sequence of input tokens).

[0096] The pooling layer 121 processes the sequence of embeddings 120 to generate a sequence of pooled embeddings 130. The pooling layer 121 generates each pooled embedding in the sequence of pooled embeddings 130 by applying a pooling operation to each of the multiple embeddings from the sequence of embeddings. For example, to generate pooled embedding 130-A in Figure 2, the pooling layer 121 can apply a pooling operation to the embedding of tokens that occupy a position in a first frame in the sequence of input tokens 110. The first frame may contain a first number of positions (4 in the example in Figure 2) in the sequence of input tokens 110.

[0097] The pooling operation can be, for example, average pooling, minimum pooling, or maximum pooling. The pooling layer 121 can apply the pooling operation according to a predetermined stride value. For example, the stride value may depend on the total number of vector quantizers in the SoundStream neural audio codec used to generate the acoustic tokens contained in the sequence of input tokens 110, for example, the stride value may be the same as the total number of vector quantizers.

[0098] Each attention block 140 includes one or more convolutional layers and one or more attention layers. Therefore, each attention block 140 may also be referred to as a “convolution-augmented attention block.” The convolution-augmented attention block is described in more detail in Anmol Gulati, et al. Conformer: Convolution-augmented transformer for speech recognition. In Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Shanghai, China, 25-29 October 2020, pages 5036-5040. ISCA, 2020, the entirety of which is incorporated herein by reference.

[0099] Each attention block 140 updates the pooled embedding sequence by processing the pooled embedding sequence, or data derived from the pooled embedding sequence, and applying both convolution and attention operations to generate an updated sequence of the pooled embedding 150.

[0100] In the case of multiple attention blocks, they can be stacked, that is, arranged in a sequence where the output of any of the attention blocks except the last one is an input to another attention block, and the multiple attention blocks apply multiple convolution and attention operations sequentially to a pooled embedding sequence, with the updated pooled embedding sequence becoming the output of the last attention block.

[0101] In some embodiments, one or more attention layers within the attention block 140 are self-attention attention layers, each configured to apply a self-attention action. In some of those embodiments, the self-attention attention layers are local self-attention layers, each configured to apply a local self-attention action. Local self-attention is a self-attention mechanism that considers only a subset of all positions contained in the sequence of input tokens 110.

[0102] Output block 160 then includes a plurality of output heads 160-A to 160-T. The plurality of output heads 160-A to 160-T generally correspond to frames in the sequence of input tokens 110. In some embodiments, the number of output heads is the same as the number of frames in the sequence of input tokens 110. Thus, in the example in Figure 2, the generative neural network 112 includes a total of T output heads, each corresponding to T frames in the sequence of input tokens 110. T can be any natural number.

[0103] Each output head may have the same architecture. In some embodiments, each output head includes one or more layers, for example, one or more high-density layers. Each output head processes at least a portion of the updated sequence of pooled embeddings to generate, for each of one or more positions in the sequence of input tokens 110, a respective prediction characterizing a new unmasked token that should occupy the position in the sequence of input tokens 110.

[0104] More specifically, each output head is configured to operate independently and in parallel with other output heads included in the output block 160. That is, each output head processes at least a portion of the updated pooled embedding sequence to generate, in parallel with each other output head, predictions characterizing unmasked tokens that should occupy some of the positions in the sequence of input tokens 110, and each other output head processes at least a portion of the updated pooled embedding sequence to generate, predictions characterizing unmasked tokens that should occupy some other positions in the sequence of input tokens 110.

[0105] For example, in Figure 2, the output head 160-A receives the updated pooled embeddings 150-A corresponding to the first frame of the sequence of input tokens 110, and processes the updated pooled embeddings 150-A within the first frame of the sequence of input tokens 110 to generate predictions characterizing each of the tokens that occupy at least a portion of the location, such as the location occupied by the masked token.

[0106] Predictions characterizing the unmasked tokens can be generated probabilistically, for example, the output of an output head parameterizes a score distribution over a given set of tokens that a token can determine for any given position, and the determined token is then used as the unmasked token that should occupy the given position. For example, the given set of tokens may include tokens that can represent multiple code vectors in a quantizer's codebook, for example, a codebook for a residual vector quantizer. For example, tokens can be determined by greedily selecting the token with the highest score, or by sampling from the score distribution using, for example, nuclear sampling or other sampling techniques.

[0107] Next, an updated sequence of input tokens 113 for a given iteration can be generated by replacing some of the masked tokens in the sequence of input tokens 110 with unmasked tokens, that is, by including unmasked tokens in place of the masked tokens in the sequence of input tokens 110. Thus, at the end of a given iteration, the generative neural network 112 can generate an updated sequence of input tokens, or in other words, a partially masked representation of the audio signal, with fewer masked tokens.

[0108] Furthermore, this architecture of the generative neural network 112 gives it the flexibility to be configured either as an encoder for representation learning or as a decoder for audio generation. That is, the generative neural network 112 can be trained for either self-supervised audio representation learning via masked audio modeling or non-autoregressive audio generation via parallel confidence-based decoding. Training the generative neural network 112 is described further below.

[0109] Figure 3 is a flowchart of an exemplary process 300 for processing a masked representation of an audio signal using a neural network. For convenience, the process 300 is described as being performed by a system of one or more computers located in one or more locations. For example, an audio generation system appropriately programmed according to this specification, such as the audio generation system 100 depicted in Figure 1, can perform the process 300.

[0110] In some embodiments, the audio generation system is implemented on a local user device, such as a smartphone, tablet, or personal computer, and the steps of process 300 are performed partially or entirely on the local user device. In some embodiments, some components of the audio generation system are implemented on the local device, and other components of the audio generation system are implemented in a data center, and some steps of process 300 are performed on the local device, while other steps of process 300 are performed in the data center. In some embodiments, the audio generation system is implemented in a data center that is communicatively coupled to a client device. After the steps of process 300 are performed partially or entirely in the data center, the data center then transmits the output audio signal to the client device.

[0111] The audio generation system includes a generative neural network. The generative neural network includes an embedding layer, followed by a pooling layer, followed by one or more attention blocks, and followed by an output block.

[0112] Before the first iteration of process 300, the system obtains a masked representation of the audio signal as a sequence of input tokens. The sequence of input tokens contains each input token at each of several positions in the sequence of input tokens. The sequence of input tokens contains masked tokens; that is, at least some of the positions in the sequence of input tokens are occupied by masked tokens.

[0113] The system can repeatedly run multiple iterations of process 300 to iteratively update the sequence of input tokens, and can use the updated sequence of input tokens generated in the last iteration of process 300 as the sequence of output tokens. By repeatedly running iterations of process 300, the system can generate a sequence of output tokens that is an unmasked representation of the audio signal, containing the respective output tokens at each of multiple positions in the sequence of input tokens.

[0114] At the start of a given iteration of process 300, the generative neural network receives a network input for the given iteration. When the given iteration is the first iteration, the network input contains a sequence of input tokens. When the given iteration is a subsequent iteration, the network input contains an updated sequence of input tokens generated in the previous iteration.

[0115] The system processes the network input using the embedding layer of a generative neural network to generate an embedding sequence (step 302). The embedding sequence contains the respective embedding for each token in the sequence of input tokens (or the updated sequence of input tokens).

[0116] The system processes the sequence of embeddings using a pooling layer of a generative neural network to generate a pooled sequence of embeddings (step 304). Each pooled embedding in the pooled sequence of embeddings is the result of pooling each subset of embeddings from the sequence of embeddings.

[0117] The system processes the pooled embedding sequence using one or more attention blocks of a generative neural network in order to update the pooled embedding sequence (step 306). Each attention block includes one or more convolutional layers and one or more attention layers. Each attention block processes the pooled embedding sequence, or data derived from the pooled embedding sequence, in order to update the pooled embedding sequence by applying both convolutional and attention operations.

[0118] The system processes at least a portion of the updated sequence of the pooled embeddings using output blocks of a generative neural network to generate a prediction for each of one or more positions in the sequence of input tokens that characterizes the token that should occupy that position in the sequence of input tokens (step 308).

[0119] The output block includes multiple output heads that operate independently and in parallel with other output heads. Each output head can correspond to a frame of the sequence of input tokens. Each frame can contain a subset of the positions in the sequence of input tokens.

[0120] In other words, each output head processes at least a portion of the updated pooled embedding sequence to generate, in parallel with each other output head, a prediction characterizing an unmasked token that should occupy some of the positions in the sequence of input tokens, and each other output head processes at least a portion of the updated pooled embedding sequence to generate, each prediction characterizing an unmasked token that should occupy some other positions in the sequence of input tokens.

[0121] The system selects multiple positions in the sequence of input tokens to be demasked (step 310). Each position selected to be demasked is occupied by a masked token.

[0122] In particular, the system can begin by identifying a subset of the sequence of input tokens that are eligible to be unmasked in the current iteration. For example, as described above, each token in the sequence of input tokens may be associated with each vector quantizer at a specific level / position in the sequence of vector quantizers. The system can be configured to unmask the tokens of the input sequence level by level, starting from a first level in the sequence of vector quantizers. Thus, the system can identify a subset of the sequence of input tokens that are eligible to be unmasked in the current iteration as any masked tokens in the input sequence associated with the level to be unmasked in the current iteration.

[0123] After identifying a subset of the sequence of input tokens eligible for masking in the current iteration, the system can identify some or all of the token sets of tokens eligible for demasking in the current iteration.

[0124] In some cases, the system may select all tokens that are eligible to be unmasked in the current iteration as the token to be unmasked in the current iteration.

[0125] In other cases, the system may select a number of tokens that are more than one but less than all of the tokens eligible to be unmasked in the current iteration as the token to be unmasked in the current iteration. For example, for each token eligible to be unmasked in the current iteration, the system may use a generative neural network to generate a score distribution over a set of possible tokens that can be selected to occupy the position currently occupied by the masked token. The system may identify multiple tokens from among the tokens eligible to be unmasked in the current iteration that are associated with the highest confidence score as the token to be unmasked in the current iteration. The "confidence score" of a masked token eligible to be unmasked may be based on the score distribution generated by the generative neural network for the masked token. For example, the confidence score may be the highest score assigned to the token by the score distribution, or the confidence score may be based on, for example, a measure of the sparseness of the confidence score distribution, for example, the reciprocal of a measure of the sparseness of the confidence score distribution.

[0126] A generative neural network can generate tokens that can be decoded into an audio signal. These tokens may correspond to an encoded / compressed representation of the audio signal. More specifically, the generative neural network generates tokens that represent code vectors that can be generated by a hierarchy of multiple vector quantizers, which are then further processed by a decoder neural network to produce output audio, and as mentioned above, this selection can be based on a predetermined order in which the multiple vector quantizers are arranged.

[0127] Specifically, when different positions in a sequence of input tokens are associated with different vector quantizers in a sequence of vector quantizers 1...Q', (Q'+1)...Q contained in a neural audio codec arranged in a hierarchical order (for example, a neural audio codec configured to generate acoustic tokens as described above), for each position occupied by a masked token in the sequence of input tokens, the system can decide whether to select a position to be unmasked based on the residual vector quantizer associated with that position.

[0128] For example, a hierarchical order can range from coarse to fine. That is, a hierarchy may include one or more coarse vector quantizers at one or more initial levels of the hierarchy, and one or more fine vector quantizers at one or more final levels of the hierarchy.

[0129] In this example, the system can proceed to select an additional position associated with the fine vector quantizer Q'+1, which is to be unmasked only after all positions within the sequence of input tokens associated with the coarse vector quantizer Q' have been unmasked.

[0130] Furthermore, several embodiments of the system can employ a predetermined schedule that specifies the number of iterations for each vector quantizer in the hierarchy. For example, assuming a total of 12 vector quantizers in the hierarchy, the predetermined schedule could specify (16;1;1;::;1) iterations for each vector quantizer. That is, the system performs 16 demasks of the tokens present at the positions associated with the first (coarse) level, and then performs one iteration of demasking the tokens present at the positions associated with each subsequent level, in a coarse-to-fine order. Thus, this predetermined schedule results in a total of 27 forward passes using a generative neural network.

[0131] For each of the multiple tokens selected to be unmasked, the system determines which unmasked token will occupy the position currently occupied by the masked token, based on the prediction of the token that should occupy that position, generated by the generative neural network (step 312). Positions not selected in an iteration remain occupied by the masked token and may be predicted again by the generative neural network in the next iteration.

[0132] In some embodiments, the prediction of which token should occupy a given position, generated by a generative neural network, includes a score distribution of a given set of tokens, i.e., the score of each token in the given set of tokens. Then, for each of the multiple positions selected to be unmasked, the unmasked token occupying that position can be determined by greedily selecting the token with the highest score, or by sampling, for example, using nuclear sampling or other sampling techniques from the score distribution.

[0133] Next, an updated sequence of input tokens for iteration can be generated by replacing some of the masked tokens in the sequence of input tokens with sampled tokens, that is, by including unmasked tokens in place of the masked tokens in the sequence of input tokens.

[0134] Process 300 may be performed as part of predicting a sequence of output tokens from a sequence of input tokens, where the desired sequence of output tokens, i.e., the output tokens that the system should generate for the sequence of input tokens, is unknown.

[0135] Process 300 may also be performed as part of the processing of input tokens obtained from a training dataset in order to train a generative neural network to determine trained values ​​for the parameters of the generative neural network, i.e., the output tokens to be generated by the system are inputs obtained from a known set of inputs.

[0136] The system (or other training system) can train generative neural networks on such training datasets via self-supervised audio representation learning. Specifically, the system can train generative neural networks on an audio representation learning task, such as gradient descent using conventional optimizers, e.g., stochastic gradient descent, RMSprop, or a backpropagation training technique using the Adam optimizer, by repeatedly running process 300 on selected training examples from the training dataset to optimize an appropriate objective function for the task.

[0137] For example, an audio representation learning task could be a masked audio modeling task. For each training example selected from the training dataset, the masked audio modeling task is a task in which, given a sequence of input tokens containing masked tokens, the task must predict a sequence of output tokens that represent an audio signal, containing unmasked tokens in place of the masked tokens.

[0138] During training, the system can generate a sequence of input tokens, which are masked representations of audio signals, by generating a sequence of tokens from selected training examples from the training dataset. Each training example represents an audio signal, and then some of the masked tokens are replaced. The sequence of input tokens may contain at least a predetermined number of masked tokens, for example, about 20%, 30%, or 40% of the input tokens in the sequence being masked.

[0139] For example, the system can do this by determining the start time step within multiple time steps of an audio signal, and then replacing a predetermined number of consecutive time steps following the start time step, such as tokens of 100ms, 200ms, 300ms, etc., with masked tokens.

[0140] As another example, the system can obtain a sequence of tokens representing an audio signal generated by a sequence of residual vector quantizers in a neural audio codec. The system can then select a target vector quantizer from the sequence of vector quantizers and mask only the tokens associated with the target vector quantizer, or the vector quantizers that follow the target vector quantizer in the sequence of vector quantizers.

[0141] For example, the system can employ the following masking scheme to generate a sequence of input tokens from selected training examples from the training dataset. Sample the context, delimiter, and time step t ~ U{0, T-1}. • Sample the current residual vector quantizer (RVQ) level q~U{1,Q}. • Mask M ∈ {0,1} according to the cosine schedule of level q TWe sample the masking ratio p = cos(u), where u ~ U[0, π / 2], and sample Mi ~ Bernoulli(p) at iid. • Mask the unconditional token selected at the current RVQ level q (M t’ Y if =1 and t'>t t’,q (Mask) and all unconditional tokens to a finer RVQ level (Y >t;>q ) is masked, where t∈{1,...,T}, and T represents the length of the maximum sequence, and no tokens prior to time step t are masked. Conditional tokens are also not masked. Y∈{1,...,C} T×Q represents an acoustic token, and C is the size of the codebook used for each RVQ level at the Q level.

[0142] In general, the objective function can be any function that evaluates the loss of the prediction output generated by the generative neural network with respect to the masked positions. That is, for each of one or more positions in the sequence of tokens occupied by the masked tokens, the objective function may include a first term that evaluates a first difference between (i) the token that should occupy that position and (ii) each prediction characterizing the token that should occupy the position generated by the generative neural network. For example, the first difference can be evaluated as the cross-entropy loss.

[0143] In some cases, the loss is evaluated with respect to a subset of all positions in the token sequence. For example, if different positions in the token sequence are associated with different vector quantizers in the sequence of vector quantizers 1...Q', (Q'+1)...Q included in the neural audio codec, the loss can be evaluated only with respect to the positions in the token sequence associated with one of the Q vector quantizers.

[0144] In some cases, the objective function may also evaluate the loss of the prediction outputs generated by the generative neural network with respect to the unmasked positions. That is, the objective function may also include a second term that evaluates a second difference between (i) the token that should occupy that position and (ii) each prediction characterizing the token that should occupy that position, for each of one or more positions in the sequence of tokens occupied by the unmasked tokens. Similarly, the second difference can be evaluated as a cross-entropy loss. In embodiments, such a loss with respect to the unmasked positions can result in faster convergence and improved performance of the neural network on various downstream tasks.

[0145] In some of these cases, the objective function can assign different weights to the first and second terms. For example, the objective function can weight the second term lower than the first term, for instance, using loss weights of 10 and 1, 20 and 1, or 50 and 1 for masked and unmasked positions, respectively.

[0146] In this example, the system can then determine one or more updates to the values ​​of the generative neural network parameters by calculating the gradient of the objective function, which includes a first difference and possibly a second difference, with respect to the generative neural network parameters.

[0147] Figures 4A and 4B show quantitative examples of performance gains that can be achieved by the audio generation systems described herein.

[0148] Figure 4A shows a comparison between the quality of audio signals generated by the SoundStreamBERT system (corresponding to the audio generation system described herein) and the AudioLM system (described in Zalan Borsos, et al. AudioLM: a language modeling approach to audio generation. arXiv preprint arXiv:2209.03143, 2022).

[0149] As recognized, the SoundStreamBERT system achieves lower character error rates (CER) and lower word error rates (WER) than the AudioLM system. Furthermore, with respect to the MOS proxy metric, the quality of the audio signal produced by the SoundStreamBERT system more closely matches the quality of the ground truth audio signal. The MOS proxy metric is an estimate of the perceived quality of the produced audio signal and can be determined using a MOS estimator similar to the DNSMOS estimator (described in Reddy, CKA, et al. DNSMOS: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. In IEEE International Conference on Acoustics, Speech and Signal Processing (DNSMOS), 2021). Figure 4B shows a comparison of runtimes between different systems for producing audio signals of 10 seconds, 20 seconds, and up to 30 seconds when run on the same hardware device (TPUv4 chip). Figure 4B shows that the SoundStorm system can generate audio signals two orders of magnitude faster than the audio LM system ("Audio LM Stages 2 & 3"), with a real-time coefficient of 0.017, including decoding to waveforms by the SoundStorm decoder ("SoundStorm Dec").

[0150] This specification uses the term “configured” in relation to systems and computer program components. When one or more computer systems are configured to perform a particular operation or action, it means that, while in operation, software, firmware, hardware, or a combination thereof is installed on the system that causes the system to perform that operation or action. When one or more computer programs are configured to perform a particular operation or action, it means that one or more programs, when executed by a data processing device, contain instructions that cause the device to perform that operation or action.

[0151] The subject matter and functional embodiments described herein can be implemented in digital electronic circuits, tangibly embodied computer software or firmware, computer hardware, and include structures disclosed herein and their structural equivalents, or one or more combinations thereof. Embodiments of the subject matter described herein can be implemented as one or more modules of computer program instructions, that is, as one or more computer programs encoded in a tangible non-temporary storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage board, a random-access memory device or a serial-access memory device, or one or more combinations thereof. Alternatively or additionally, the program instructions may be encoded into artificially generated propagated signals, such as mechanically generated electrical signals, optical signals or electromagnetic signals, which are generated to encode information for transmission to a receiving device suitable for execution by a data processing device.

[0152] The term "data processing device" refers to data processing hardware and encompasses all types of devices, machines, and equipment for processing data, including, for example, programmable processors, computers, or multiple processors or computers. A device may also be, or further include, specialized logic circuits such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). Optionally, in addition to hardware, a device may include code that constitutes the execution environment for computer programs, such as processor firmware, protocol stacks, database management systems, operating systems, or one or more combinations thereof.

[0153] Computer programs, which may be called or described as programs, software, software applications, apps, modules, software modules, scripts, or code, can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A program may, but is not required to, correspond to a file in a file system. A program may be stored in a single file dedicated to a program of interest, in part with other programs or data, for example, in a file holding one or more scripts stored in a markup language document, or in multiple collaborative files, for example, in a file storing one or more modules, subprograms, or parts of code. A computer program can be deployed to run on one computer, or it can be located in one place or distributed across multiple locations and interconnected by a data communication network to run on multiple computers.

[0154] In this specification, the term “database” is used broadly to refer to any collection of data. The data does not need to be structured in any particular way, or not structured at all, and can be stored on one or more storage devices. Therefore, for example, an index database may contain multiple collections of data, each of which may be organized and accessed in a different way.

[0155] Similarly, in this specification, the term “engine” is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine is implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers are dedicated to a particular engine, while in other cases, multiple engines may be installed and run on the same one or more computers.

[0156] The processes and logic flows described herein can be performed by one or more programmable computers executing one or more computer programs to act on input data and produce outputs, thereby performing their functions. Alternatively, the processes and logic flows can be performed by special-purpose logic circuits, such as FPGAs or ASICs, or by a combination of special-purpose logic circuits and one or more programmed computers.

[0157] A computer suitable for running computer programs may be based on a general-purpose or dedicated microprocessor, or both, or other types of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory, random-access memory, or both. The basic elements of a computer are the central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be complemented by or incorporated into special-purpose logic circuits. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operablely connected to them to receive data from them, transmit data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer may be incorporated into other devices, such as mobile phones, personal digital assistants (PDAs), mobile audio or video players, game consoles, Global Positioning System (GPS) receivers, or portable storage devices (such as Universal Serial Bus (USB) flash drives) (these are just a few examples).

[0158] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0159] Embodiments of the subject matter described herein may be implemented in a computer having a display device for displaying information to a user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a key vector board and pointing device that allows the user to input into the computer, such as a mouse or trackball, in order to provide user interaction. Other types of devices can also be used to interact with the user. For example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, voice, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from a web browser. The computer can also interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and then receiving response messages from the user.

[0160] Data processing equipment for implementing machine learning models may include, for example, dedicated hardware accelerator units for processing the general and numerical computation portions (i.e., inference, workloads) of machine learning training or production.

[0161] Machine learning models can be implemented and deployed using machine learning frameworks, such as the TensorFlow framework or the JAX framework.

[0162] Embodiments of the subject matter described herein may be implemented in a computing system that includes, for example, a data server as a backend component, or in a computing system that includes a middleware component, for example, an application server, or in a computing system that includes a client computer having a frontend component, for example, a graphical user interface, a web browser, or an application that enables a user to interact with embodiments of the subject matter described herein, or in a computing system that includes one or more such backend, middleware, or frontend components in any combination. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and, for example, the Internet.

[0163] A computing system can include clients and servers. Clients and servers are generally geographically separated from each other and typically interact through a communication network. The client-server relationship arises from computer programs that run on each computer and have a client-server relationship with each other. In some embodiments, the server sends data (e.g., an HTML page) to a user device for the purpose of displaying data to a user interacting with a device acting as a client and receiving user input from that user. Data generated on the user device, such as the results of user interactions, can be received from the device by the server.

[0164] While this specification includes details of many specific embodiments, these should not be interpreted as limiting the scope of any invention or claimable content, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described herein in the context of individual embodiments may also be realized in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be realized individually or in any suitable subcombination in multiple embodiments. Furthermore, features may be described above as functioning in a particular combination, and even if initially claimed as such, one or more features from the claimed combination may be removed from the combination, and the claimed combination may cover a subcombination or a variation of a subcombination.

[0165] Similarly, while operations are shown in the drawings and described in a specific order in the claims, this should not be understood as requiring that such operations be performed in a specific or sequential order shown, or that all shown operations be performed, in order to obtain the desired results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described can generally be integrated into a single software product or packaged into multiple software products.

[0166] This specification also provides the subject matter of the following numbered clauses:

[0167] Clause 1. A method performed by one or more computers, Receiving a masked representation of an audio signal as a sequence of tokens, wherein the sequence of tokens includes one or more masked tokens. Processing the masked representation of the audio signal using a neural network, Processing the masked representation of the audio signal using the embedding layer of the neural network to generate a sequence of embeddings that includes the embedding of each token in the sequence of tokens, Processing the sequence of embeddings using a pooling layer of the neural network to generate a sequence of pooled embeddings, wherein each pooled embedding in the sequence of pooled embeddings is a result of pooling each of the multiple embeddings from the sequence of embeddings. Processing the pooled embedding sequence using the attention block of the neural network to update the pooled embedding sequence, including applying one or more self-attention actions to the pooled embedding sequence, and Processing the updated sequence of the pooled embedding to generate, for each of one or more positions in the sequence of tokens, a prediction characterizing the token that should occupy the position in the sequence of tokens; Methods that include...

[0168] Clause 2. The method according to Clause 1, wherein the attention block is a convolutional extension attention block comprising one or more convolutional layers and one or more self-attention layers.

[0169] Clause 3. Applying one or more self-attention actions to the pooled embedding sequence is permitted. The method according to clause 1 or 2, comprising applying one or more localized self-attention actions to the pooled sequence of embeddings.

[0170] Clause 4. Applying one or more local self-attention actions to the pooled embedding sequence is permitted. The method according to Clause 3, comprising sequentially applying a plurality of local self-attention actions to the pooled sequence of embeddings.

[0171] Clause 5. Receiving the aforementioned representation of the audio signal as a sequence of tokens means that The method according to any one of the clauses 1 to 4, comprising receiving a sequence of acoustic tokens generated by a neural audio codec that includes multiple residual vector quantization layers.

[0172] Clause 6. Receiving the aforementioned representation of the audio signal as a sequence of tokens means that The method according to Clause 5, comprising receiving a sequence of semantic tokens generated by an intermediate layer of an auxiliary audio processing neural network.

[0173] Clause 7. The method of Clause 6, wherein receiving the sequence of semantic tokens comprises re-sampling the sequence of semantic tokens based on the sampling rate of the sequence of acoustic tokens.

[0174] Clause 8. Processing the sequence of the pooled embeddings using the pooling layer in order to generate the sequence of the pooled embeddings is: The method according to any one of claims 5 to 7, comprising processing the sequence of embeddings with respect to a stride dependent on the total number of residual vector quantization layers using the pooling layer.

[0175] Clause 9. The method according to Clause 8, wherein the value of the stride is the same as the total number of residual vector quantization layers.

[0176] Clause 10. The output block of the neural network comprises a plurality of output heads, and for each of one or more positions in the sequence of tokens, processing the updated sequence of the pooled embeddings to generate a respective prediction characterizing the token that should occupy the position in the sequence of tokens, The method according to any one of the claims 1 to 9, comprising processing the updated sequence of pooled embeddings by each output head independently and in parallel with other output heads in order to generate respective predictions characterizing multiple tokens located at different positions in the sequence of tokens.

[0177] Clause 11. Each output head, Receiving each pooled embedding from the updated sequence of the pooled embeddings, and Processing the pooled embeddings to generate a corresponding prediction for each position in the same frame of the sequence of tokens, wherein the output head processes the predictions for the same frame positions in parallel. The method described in clause 10, configured to perform the following:

[0178] Clause 12. Selecting multiple positions in the sequence of tokens to be demasked, where each position selected to be demasked is occupied by a masked token, and For each of the plurality of locations selected to be unmasked, the unmasked tokens occupying the location are determined based on predictions generated by the neural network for the tokens that should occupy the location. The method described in any one of the clauses 1 to 11, further including the method described in any one of the clauses 1 to 11.

[0179] Clause 13. Selecting multiple positions in the sequence of tokens to be demasked means that for each position in the sequence of tokens occupied by a masked token, The method according to clause 12, comprising determining whether to select the unmasked position based on the confidence of a prediction generated by the neural network that characterizes the token that should occupy the position in the sequence of tokens.

[0180] Clause 14. Some or all of the positions in the sequence of tokens are associated with each residual vector quantization layer of the sequence of residual vector quantization layers. Selecting multiple positions in the sequence of tokens to be demasked means, for each position in the sequence of tokens occupied by the masked token, The method according to any one of the claims 12 to 13, comprising determining whether to select the position to be unmasked based on the residual vector quantization layer associated with the position.

[0181] Clause 15. The method according to any one of Clauses 12 to 14, wherein for each of the plurality of locations selected to be unmasked, the prediction of the token that should occupy the location, generated by the neural network, includes a score distribution across the set of tokens.

[0182] Clause 16. For each of the multiple locations selected to be demasked, determining the demasked token occupying the location is: The method according to clause 15, comprising sampling tokens from the score distribution of the set of tokens.

[0183] Clause 17. For each of the one or more positions in the sequence of tokens occupied by the masked token, determine a first difference between (i) the token that should occupy the position and (ii) the respective predictions that characterize the token that should occupy the position, and Based on at least one difference, determine the update of the set of parameters of the neural network to their values. The method described in any one of the clauses 1 to 16, further including the method described in any one of the clauses 1 to 16.

[0184] Clause 18. For each of the one or more positions in the sequence of tokens occupied by the unmasked token, determine a second difference between (i) the token that should occupy the position and (ii) the respective predictions characterizing the token that should occupy the position, and Based on both the first difference and the second difference, determine the update to the values ​​of the set of parameters of the neural network. The method described in Article 17, further including the method described in Article 17.

[0185] Clause 19. The method of Clause 18, wherein determining the second difference includes weighting the second difference lower than the first difference.

[0186] Clause 20. To generate the masked representation of the audio signal as a sequence of tokens, Determining the start time step within a plurality of time steps of the audio signal, and Replacing the tokens of a predetermined number of consecutive time steps following the aforementioned start time step with masked tokens, The method described in any one of the clauses 1 to 19, including, further including, generating.

[0187] Clause 21. To generate the masked representation of the audio signal as a sequence of tokens, The representation of the audio signal is obtained as a sequence of tokens generated by the sequence of residual vector quantization layers of the neural audio codec. Selecting a target vector quantization layer, This includes masking only the tokens associated with the target vector quantization layer, or the vector quantization layer that follows the target vector quantization layer in the sequence of vector quantization layers, A method of description in any one of clauses 1 through 19, further including generating.

[0188] Clause 22. The method according to any one of Clauses 1 to 22, further comprising generating an output audio signal based on at least each of the aforementioned unmasked tokens.

[0189] Clause 23. One or more computer storage media for storing instructions, wherein when the instructions are executed by one or more computers, the one or more computers cause the one or more computers to implement the neural network described in any one of Clauses 1 to 22.

[0190] Clause 24. A system, One or more computers, One or more storage devices for storing instructions, wherein, when executed by the one or more computers, the instructions cause the one or more computers to perform the operations described in any one of the clauses 1 to 22; A system that includes this.

[0191] Specific embodiments of the present invention have been described. Other embodiments are within the scope of the following claims. For example, the actions described in the claims may be performed in a different order, and this may still yield desirable results. As an example, the process shown in the accompanying figures does not necessarily require to be performed in a specific or sequential order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method performed by one or more computers, Receiving a masked representation of an audio signal as a sequence of tokens, wherein the sequence of tokens includes one or more masked tokens; Processing the masked representation of the audio signal using a neural network, Processing the masked representation of the audio signal using the embedding layer of the neural network in order to generate a sequence of embeddings that includes the embedding of each token in the sequence of tokens, Processing the sequence of embeddings using a pooling layer of the neural network to generate a sequence of pooled embeddings, wherein each pooled embedding in the sequence of pooled embeddings is a result of pooling each of the multiple embeddings from the sequence of embeddings. Processing the pooled embedding sequence using the attention block of the neural network to update the pooled embedding sequence, including applying one or more self-attention actions to the pooled embedding sequence, and Processing the updated sequence of the pooled embedding to generate, for each of one or more positions in the sequence of tokens, a prediction characterizing the token that should occupy the position in the sequence of tokens; Including processing, Methods that include...

2. The method according to claim 1, wherein the attention block is a convolutional extension attention block comprising one or more convolutional layers and one or more self-attention layers.

3. Applying one or more self-attention actions to the pooled embedded sequence means The method according to claim 1 or 2, comprising applying one or more localized self-attention actions to the pooled sequence of embeddings.

4. Applying one or more local self-attention actions to the pooled sequence of embeddings is: The method according to claim 3, comprising sequentially applying a plurality of local self-attention actions to the pooled embedding sequence.

5. Receiving the aforementioned representation of the audio signal as a sequence of tokens means The method according to any one of claims 1 to 4, comprising receiving a sequence of acoustic tokens generated by a neural audio codec including a plurality of residual vector quantizers.

6. Receiving the aforementioned representation of the audio signal as a sequence of tokens means The method according to claim 5, comprising receiving a sequence of semantic tokens generated by an intermediate layer of an auxiliary audio processing neural network.

7. The method according to claim 6, wherein receiving the sequence of semantic tokens includes re-sampling the sequence of semantic tokens based on the sampling rate of the sequence of acoustic tokens.

8. Processing the sequence of the pooled embeddings using the pooling layer in order to generate the sequence of the pooled embeddings is The method according to any one of claims 5 to 7, comprising processing the sequence of embeddings with respect to a stride dependent on the total number of residual vector quantizers using the pooling layer.

9. The method according to claim 8, wherein the value of the stride is the same as the total number of residual vector quantizers.

10. The output block of the neural network includes a plurality of output heads, and for each of one or more positions in the sequence of tokens, processing the updated sequence of the pooled embeddings to generate a respective prediction characterizing the token that should occupy the position in the sequence of tokens, The method according to any one of claims 1 to 9, comprising processing the updated sequence of pooled embeddings independently and in parallel with other output heads by each output head to generate respective predictions characterizing multiple tokens located at different positions in the sequence of tokens.

11. Each output head, Receiving each pooled embedding from the updated sequence of the pooled embeddings, and Processing the pooled embeddings to generate a corresponding prediction for each position in the same frame of the sequence of tokens, wherein the output head processes the predictions for the same frame positions in parallel. The method according to claim 10, configured to perform the following:

12. The selection of multiple positions in the sequence of tokens to be demasked, wherein each position selected to be demasked is occupied by a masked token. For each of the plurality of locations selected to be unmasked, the unmasked tokens occupying the location are determined based on predictions generated by the neural network for the tokens that should occupy the location. The method according to any one of claims 1 to 11, further comprising:

13. Selecting multiple positions in the sequence of tokens to be demasked means that for each position in the sequence of tokens occupied by the masked token, The method according to claim 12, comprising determining whether to select the position to be demasked based on the confidence of a prediction generated by the neural network that characterizes the token that should occupy the position in the sequence of tokens.

14. Some or all of the positions in the sequence of tokens are associated with each residual vector quantizer in the sequence of residual vector quantizers, Selecting multiple positions in the sequence of tokens to be demasked means, for each position in the sequence of tokens occupied by the masked token, The method according to any one of claims 12 to 13, comprising determining whether to select a position to be demasked based on the residual vector quantizer associated with the position.

15. Selecting the plurality of positions in the sequence of tokens to be demasked is, Identifying a set of masked tokens that are eligible to be demasked, and including selecting each masked token associated with a particular residual vector quantizer to be included in the set of masked tokens that are eligible to be demasked; Simultaneously, the tokens to be demasked include selecting multiple tokens from the set of masked tokens that are eligible to be demasked, The method according to claim 14, including the method described in claim 14.

16. All tokens associated with any residual vector quantizer preceding the particular residual vector quantizer in the sequence of residual vector quantizers have already been demasked. The method according to claim 15, wherein all tokens associated with residual vector quantizers that follow a particular residual vector quantizer in the sequence of residual vector quantizers have not yet been demasked.

17. The method according to any one of claims 12 to 16, wherein for each of the plurality of locations selected to be unmasked, the prediction generated by the neural network regarding the token that should occupy the location includes a score distribution over a set of tokens.

18. For each of the plurality of locations selected to be demasked, determining the demasked token occupying the location is: The method according to claim 17, comprising sampling tokens from the score distribution over the set of tokens.

19. For each of the one or more positions in the sequence of tokens occupied by the masked token, a first difference is determined between (i) the token that should occupy the position and (ii) the respective predictions that characterize the token that should occupy the position. Based on at least the first difference, determine the update of the values ​​for the set of parameters of the neural network, The method according to any one of claims 1 to 18, further comprising:

20. For each of the one or more positions in the sequence of tokens occupied by the unmasked token, determine a second difference between (i) the token that should occupy the position and (ii) the respective predictions characterizing the token that should occupy the position. Based on both the first difference and the second difference, the update to the set of parameters of the neural network is determined, The method according to claim 19, further comprising:

21. The method according to claim 20, wherein determining the second difference includes weighting the second difference lower than the first difference.

22. The process involves generating the masked representation of the audio signal as a sequence of tokens, Determining the start time step within the multiple time steps of the aforementioned audio signal, Replacing tokens at a predetermined number of consecutive time steps following the aforementioned start time step with masked tokens, The method according to any one of claims 1 to 21, further comprising generating, including

23. The process involves generating the masked representation of the audio signal as a sequence of tokens, The representation of the audio signal is obtained as a sequence of tokens generated by the sequence of residual vector quantization layers of the neural audio codec, Selecting a target vector quantization layer, This includes masking only the tokens associated with the target vector quantization layer, or the vector quantization layer that follows the target vector quantization layer in the sequence of vector quantization layers, The method according to any one of claims 1 to 19, further comprising generating.

24. The method according to any one of claims 1 to 23, further comprising generating an output audio signal based on at least each of the aforementioned unmasked tokens.

25. One or more computer storage media for storing instructions, wherein when an instruction is executed by one or more computers, one or more computer storage media cause the one or more computers to perform an operation according to any one of claims 1 to 24.

26. It is a system, One or more computers, A system comprising one or more storage devices for storing instructions, wherein, when executed by the one or more computers, the instructions cause the one or more computers to perform an operation according to any one of claims 1 to 24.

Images