Compression of audio waveforms using neural networks and vector quantizers

Abstract

This provides a method for compressing audio waveforms using machine learning models. [Solution] The method includes the steps of: receiving an audio waveform containing an audio sample for each of a plurality of time steps; processing the audio waveform using an encoder neural network to generate a plurality of feature vectors representing the audio waveform; and generating a coded representation of each of the plurality of feature vectors using a plurality of vector quantizers, each associated with a codebook of the code vectors. Each coded representation of each feature vector identifies a plurality of code vectors containing the respective code vectors from the codebooks of each vector quantizer. The method also includes the step of compressing each coded representation of the plurality of feature vectors to generate a compressed representation of the audio waveform.

Description

[Technical Field]

[0001] Cross-reference of related applications This application claims priority to U.S. Patent Application No. 17 / 856,856, filed on July 1, 2022, which in turn claims priority to U.S. Provisional Application No. 63 / 218,139, filed on July 2, 2021. The disclosures of the prior applications are deemed to be part of the disclosures of this application and are incorporated into the disclosures of this application by reference.

[0002] This specification relates to processing data using machine learning models. [Background technology]

[0003] A machine learning model receives input and, based on the received input, generates an output, such as a predicted output. Some machine learning models are parametric models, which generate an output based on the received input and the values ​​of the model's parameters.

[0004] Some machine learning models are deep models that employ multiple layers of the model to produce an output for a given input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that apply nonlinear transformations to the received inputs, respectively, in order to produce an output. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] "SoundStream: An End-to-End Neural Audio Codec" in N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, IEEE / ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 495-507, 2022.

Non-Patent Document 2

Summary of the Invention

Means for Solving the Problems

[0006] This specification generally describes a compression system implemented as a computer program on one or more computers in one or more locations that can compress an audio waveform. This specification further describes a restoration system implemented as a computer program on one or more computers in one or more locations that can restore an audio waveform.

[0007] Generally, the compression system and the restoration system can be located at any suitable location. In particular, the compression system can, in some cases, be located remotely from the restoration system. For example, the compression system can be implemented by one or more first computers at a first location, while the restoration system can be implemented by one or more second (different) computers at a second (different) location.

[0008] In some implementations, the compression system can generate a compressed representation of the input audio waveform and store the compressed representation in a datastore, such as a logical data storage area or a physical data storage device. The recovery system can later access the compressed representation from the datastore, process it, and generate the corresponding output audio waveform. The output audio waveform may be, for example, a reconstruction of the input audio waveform, or an enhanced (e.g., denoised) version of the input audio waveform.

[0009] In some implementations, a compression system can generate a compressed representation of an input audio waveform and transmit the compressed representation to a destination over a data communication network, such as a local area network, wide area network, or the internet. A decompression system can access the compressed representation at the destination, process it, and generate a corresponding output waveform.

[0010] According to a first aspect, a method is provided that can be performed by one or more computers, the method comprising: receiving an audio waveform having respective audio samples for each of a plurality of time steps; processing the audio waveform using an encoder neural network to generate a plurality of feature vectors representing the audio waveform; generating a respective coded representation of each of the plurality of feature vectors using a plurality of vector quantizers, each associated with a respective codebook of the code vectors, wherein each coded representation of each feature vector identifies a plurality of code vectors, each including a code vector from a codebook of each vector quantizer that defines the quantized representation of the feature vectors; and generating a compressed representation of the audio waveform by compressing the respective coded representations of the plurality of feature vectors.

[0011] In some implementations, multiple vector quantizers are ordered in a sequence, and for each of the multiple feature vectors, the step of generating a coded representation of the feature vector includes, for the first vector quantizer in the sequence of vector quantizers, the step of receiving the feature vector, the step of identifying, based on the feature vector, each code vector from the vector quantizer's codebook to represent the feature vector, and the step of determining the current residual vector based on the error between (i) the feature vector and (ii) the code vector representing the feature vector, wherein the coded representation of the feature vector identifies the code vector representing the feature vector.

[0012] In some implementations, for each of a plurality of feature vectors, the step of generating a coded representation of the feature vector further includes, for each vector quantizer after the first vector quantizer in the sequence of vector quantizers, the step of receiving the current residual vector generated by the previous vector quantizer in the sequence of vector quantizers; the step of identifying, based on the current residual vector, each code vector from the vector quantizer codebook to represent the current residual vector; and, if the vector quantizer is not the last vector quantizer in the sequence of vector quantizers, the step of updating the current residual vector based on the error between (i) the current residual vector and (ii) the code vector representing the current residual vector, so that the coded representation of the feature vector identifies the code vector representing the current residual vector.

[0013] In some implementations, the step of generating a compressed representation of the audio waveform includes the step of entropy encoding each of the coded representations of multiple feature vectors.

[0014] In some implementations, the quantized representation of each feature vector is defined by the sum of multiple code vectors identified by the coded representation of the feature vector.

[0015] In some implementations, the codebooks for multiple vector quantizers all contain the same number of code vectors.

[0016] In some implementations, the encoder neural network and the codebooks for multiple vector quantizers are trained together with the decoder neural network, and the decoder neural network is configured to receive the quantized representation of each of the multiple feature vectors representing the input audio waveform, which are generated using the encoder neural network and the multiple vector quantizers, and to process the quantized representations of the feature vectors representing the input audio waveform to generate the output audio waveform.

[0017] In some implementations, training includes the steps of: acquiring multiple training examples, each containing (i) an input audio waveform and (ii) a corresponding target audio waveform; processing each input audio waveform from each training example using an encoder neural network, multiple vector quantizers from a sequence of vector quantizers, and a decoder neural network to generate an output audio waveform which is an estimate of the corresponding target audio waveform; determining the gradient of an objective function which depends on the respective output and target waveforms for each training example; and updating one or more of the encoder neural network parameters, the decoder neural network parameters, or the codebooks of the multiple vector quantizers using the gradient of the objective function.

[0018] In some implementations, for one or more training examples, the target audio waveform is an extended version of the input audio waveform.

[0019] In some implementations, for one or more of the training examples, the target audio waveform is a denoised version of the input audio waveform.

[0020] In some implementations, for one or more training examples, the target audio waveform is the same as the input audio waveform.

[0021] In some implementations, the step of processing each input audio waveform to generate a corresponding output audio waveform includes the step of conditioning an encoder neural network, a decoder neural network, or both, on data that defines whether the corresponding target audio waveform is (i) an input audio waveform or (ii) an extended version of the input audio waveform.

[0022] In some implementations, the method further includes, for each training example, the steps of selecting the number of vector quantizers that will be used to quantize the feature vector representing the input audio waveform, and generating the corresponding output audio waveform using only the selected number of vector quantizers from the sequence of vector quantizers.

[0023] In some implementations, the number of vector quantizers selected to quantize the feature vector representing the input audio waveform varies among the training examples.

[0024] In some implementations, for each training example, the step of selecting the number of vector quantizers that will be used to quantize the feature vector representing the input audio waveform includes the step of randomly sampling the number of vector quantizers that will be used to quantize the feature vector representing the input audio waveform.

[0025] In some implementations, the objective function includes a reconstruction loss that measures the error between (i) the output audio waveform and (ii) the corresponding target audio waveform for each training example.

[0026] In some implementations, for each training example, the reconstruction loss measures the multiscale spectral error between (i) the output audio waveform and (ii) the corresponding target audio waveform.

[0027] In some implementations, training comprises the step of, for each training example, using a discriminator neural network to process data derived from an output audio waveform to generate a set of one or more discriminator scores, each discriminator score characterizing the estimated likelihood that the output audio waveform is an audio waveform generated using an encoder neural network, a plurality of vector quantizers, and a decoder neural network, and the objective function comprises an adversarial loss that depends on the discriminator scores generated by the discriminator neural network.

[0028] In some implementations, the data derived from the output audio waveform includes the output audio waveform, a downsampled version of the output audio waveform, or a Fourier transformed version of the output audio waveform.

[0029] In some implementations, for each training example, the reconstruction loss measures the error between (i) one or more intermediate outputs generated by the discriminator neural network by processing the output audio waveform, and (ii) one or more intermediate outputs generated by the discriminator neural network by processing the corresponding target audio waveform.

[0030] In some implementations, the codebooks for multiple vector quantizers are repeatedly updated during training using an exponential moving average of the feature vectors generated by the encoder neural network.

[0031] In some implementations, the encoder neural network comprises a sequence of encoder blocks, each configured to process each set of input feature vectors according to a set of encoder block parameters, to generate a set of output feature vectors having a lower temporal resolution than the set of input feature vectors.

[0032] In some implementations, the decoder neural network comprises a sequence of decoder blocks, each configured to process each set of input feature vectors according to a set of decoder block parameters, to generate a set of output feature vectors having a higher temporal resolution than the set of input feature vectors.

[0033] In some implementations, the audio waveform is either a speech waveform or a music waveform.

[0034] In some implementations, the method further includes the step of transmitting a compressed representation of the audio waveform over the network.

[0035] In another embodiment, a method is provided that is performed by one or more computers, the method comprising: receiving a compressed representation of an input audio waveform; restoring the compressed representation of the input audio waveform to obtain a coded representation of each of a plurality of feature vectors representing the input audio waveform, wherein the coded representation of each feature vector identifies a plurality of code vectors, each including a code vector from each of a plurality of vector quantizers' respective codebooks that define a quantized representation of the feature vector; generating a quantized representation of each feature vector from the coded representation of the feature vectors; and processing the quantized representations of the feature vectors using a decoder neural network to generate an output audio waveform.

[0036] In another embodiment, a system is provided, comprising one or more computers and one or more storage devices communicably coupled to one or more computers, wherein one or more storage devices store instructions, and when an instruction is executed by one or more computers, the one or more computers cause one or more computers to perform the operation of each of the methods described herein.

[0037] In another embodiment, one or more non-temporary computer storage media are provided for storing instructions, and when an instruction is executed by one or more computers, the one or more computers are made to perform the operation of each of the methods described herein.

[0038] The subject matter described herein may be implemented in certain embodiments to achieve one or more of the following advantages:

[0039] The compression / decompression systems described herein can enable audio data to be compressed more efficiently than some conventional systems. By enabling more efficient audio data compression, the systems enable more efficient audio data transmission (for example, by reducing the communication network bandwidth required to transmit audio data) and more efficient audio data storage (for example, by reducing the amount of memory required to store audio data).

[0040] The compression / decompression system includes an encoder neural network, a set of vector quantizers, and a decoder neural network, all trained together (i.e., "end-to-end"). By training the respective neural network parameters of the encoder and decoder neural networks together with the codebook of the vector quantizers, the parameters of the compression / decompression system can be harmonically adapted to achieve more efficient audio compression than would otherwise be possible. For example, as the neural network parameters of the encoder neural network are iteratively tuned, the codebook of the vector quantizers is simultaneously optimized, enabling more accurate quantization of the feature vectors generated by the encoder neural network. The neural network parameters of the decoder neural network are also simultaneously optimized, enabling more accurate reconstruction of the audio waveform from the quantized feature vectors generated using the updated codebook of the vector quantizers.

[0041] The vector quantization of feature vectors representing an audio waveform is performed using a single vector quantizer, where each feature vector is represented using r bits, and the size is 2 rA codebook may be required. That is, the size of a vector quantizer's codebook can increase exponentially with the number of bits allocated to represent each feature vector. As the number of bits allocated to represent each feature vector increases, learning and memorizing the codebook becomes computationally impossible. To address this problem, a compression / decompression system performs vector quantization using a sequence of multiple vector quantizers, each maintaining its own codebook. The first vector quantizer can directly quantize the feature vectors generated by the encoder neural network, while each subsequent vector quantizer can quantize a residual vector that defines the quantization error generated by the previous vector quantizer.

[0042] A sequence of vector quantizers can iteratively improve the quantization of feature vectors while maintaining each one a dramatically smaller codebook that would otherwise be required by a single vector quantizer. For example, each vector quantizer is size

[0043]

number

[0044] The codebook can be maintained, where r is the number of bits allocated to represent each feature vector, and N q is the number of vector quantizers. Therefore, by performing vector quantization using a sequence of multiple vector quantizers, the compression / decompression system can reduce the memory required to store the quantizer codebook, and vector quantization can be performed in situations where it would otherwise be computationally impossible to do so.

[0045] By performing vector quantization using a set of multiple vector quantizers (i.e., not a single vector quantizer), the compression / decompression system can also control the compression bitrate, for example, the number of bits used to represent each second of audio data. To reduce the bitrate, the compression / decompression system can perform vector quantization using fewer vector quantizers. Conversely, to increase the bitrate, the compression / decompression system can perform vector quantization using more vector quantizers. During training, the number of vector quantizers used for compression / decompression of each audio waveform can be varied (e.g., randomly) across training examples, allowing the compression / decompression system to learn a single set of parameter values ​​that enable effective compression / decompression across a range of possible bitrates. Thus, the compression / decompression system can reduce the consumption of computational resources by eliminating any requirement to train and maintain multiple encoders, decoders, and vector quantizers, each optimized for their respective bitrates.

[0046] Compression / decompression systems can be trained to perform both audio data compression and audio data augmentation, such as denoising, together. That is, compression and decompression systems can be trained to simultaneously augment (e.g., denoise) audio waveforms as part of compressing and decompressing the waveforms, without increasing overall latency. In contrast, some conventional systems apply separate audio augmentation algorithms to the audio waveform either at the transmitter side (i.e., before compression) or at the receiver side (i.e., after decompression), which can result in increased latency.

[0047] Details of one or more embodiments of the subject matter of this specification are described in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawing]

[0048] [Figure 1] This figure shows an exemplary audio compression system that can compress audio waveforms using an encoder neural network and a residual vector quantizer. [Figure 2] This figure illustrates an exemplary audio restoration system that can restore a compressed audio waveform using a decoder neural network and a residual vector quantizer. [Figure 3] This is a schematic diagram of an exemplary training system in which an encoder neural network, a decoder neural network, and a residual vector quantizer can be trained together. [Figure 4] This is a flowchart illustrating an exemplary process for compressing audio waveforms. [Figure 5] This is a flowchart illustrating an exemplary process for restoring a compressed audio waveform. [Figure 6] This is a flowchart illustrating an exemplary process for generating a quantized representation of a feature vector using a residual vector quantizer. [Figure 7] This is an illustrative process flowchart for training an encoder neural network, a decoder neural network, and a residual vector quantizer together. [Figure 8A] This figure shows an example of a fully convolutional neural network architecture. [Figure 8B] This figure shows an example of a fully convolutional neural network architecture. [Modes for carrying out the invention]

[0049] Similar reference numbers and names in various drawings refer to the same elements.

[0050] Figure 1 shows an exemplary audio compression system 100 that can compress an audio waveform using an encoder neural network 102 and a residual vector quantizer 106. Similarly, Figure 2 shows an exemplary audio restoration system 200 that can restore a compressed audio waveform using a decoder neural network 104 and a residual vector quantizer 106. For clarity, both Figures 1 and 2 will be referred to when describing the various components involved in compression and restoration. The audio compression / restoration systems 100 / 200 are examples of systems implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below are implemented.

[0051] The compression / decompression system 100 / 200 utilizes a neural network architecture (neural codec) which can outperform older codecs, such as waveform and parametric codecs, in terms of operating bitrate and general-purpose audio compression. For comparison, waveform codecs typically use time / frequency domain transformations for compression / decompression but make few or no assumptions about the source audio. As a result, waveform codecs produce high-quality audio at medium to high bitrates but tend to introduce coding artifacts at low bitrates. Parametric codecs can overcome this problem by making specific assumptions about the source audio (e.g., speech), but this is ineffective for general-purpose audio compression. Conversely, the audio compression and decompression system 100 / 200 can compress and decompress speech, music, and general audio at the target bitrate, which is typically achieved by speech-adapted codecs (e.g., parametric codecs). Therefore, the audio compression / decompression system 100 / 200 can operate in modalities where conventional codecs are not possible. In some implementations, the compression and decompression system 100 / 200 can be configured for specific types of audio content (e.g., speech) due to the flexibility provided by the neural network architecture.

[0052] Referring to Figure 1, the compression system 100 receives an audio waveform 112 to be compressed. The waveform 112 may contain audio samples at each time step, where the time step typically corresponds to a specific sampling rate. Higher sampling rates capture higher frequency components of the audio waveform 112. For example, a standard audio sampling rate used by professional digital equipment is 48 kHz, because professional digital equipment can reconstruct sounds at frequencies up to 24 kHz (e.g., the upper limit of human hearing). While such sampling rates are ideal for comfortable listening, the compression system 100 can generally be configured to process the waveform 112 at any sampling rate, even with waveforms involving non-uniform sampling.

[0053] The audio waveform 112 can be generated from any suitable audio source. For example, the waveform 112 could be a recording from an external audio device (e.g., voice from a microphone), a purely digital work (e.g., electronic music), or general audio such as sound effects and background noise (e.g., white noise, room tone). In some implementations, the audio compression system 100 can perform audio enhancement while compressing the waveform 112, for example, by suppressing unwanted background noise.

[0054] In conventional audio processing pipelines, compression and expansion are typically performed by separate modules. For example, an audio expansion algorithm can be applied at the input of encoder 102 before the waveform is compressed, or at the output of decoder 104 after the waveform has been restored. In this setup, each processing step introduces end-to-end latency, for example, due to buffering the waveform to the expected frame length of the algorithm. Conversely, by utilizing thoughtful training of various neural network components (see Figure 3), the audio compression system 100 can achieve joint compression and expansion without relying on separate modules and without additional latency. In some implementations, the audio restoration system 200 implements joint restoration and expansion. Generally, the compression system 100, the restoration system 200, or both, can be designed to perform audio expansion by thoroughly training neural network components.

[0055] The audio waveform 112 is processed (e.g., encoded) by the encoder 102 to generate a sequence of feature vectors 208 representing the waveform 112. The feature vectors 208 (e.g., embeddings, latent representations) are a compressed representation of the waveform, extracting the most relevant information about the audio content of the waveform. The encoder 102 can downsample the input waveform 202 to generate a compressed feature vector 208 such that the feature vector 208 has a lower sampling rate than the original audio waveform 112. For example, the encoder neural network 102 can use multiple convolutional layers with increased stride to generate feature vector 208 at a lower sampling rate (e.g., lower temporal resolution).

[0056] Next, the feature vector 208 is processed by a residual (e.g., multi-stage) vector quantizer RVQ106 to produce a coded representation (CFV) 210 of the feature vector and a corresponding quantized representation (QFV) 212 of the feature vector. RVQ106 can generate QFV212 at a given bitrate by leveraging one or more vector quantizers 108. RVQ106 achieves (lossy) compression by mapping the higher-dimensional space of the feature vector 208 to separate subspaces of the code vectors. As detailed below, the CFV210 specifies a codeword (e.g., an index) from each codebook 110 of each vector quantizer 108, where each codeword identifies a code vector stored in the associated codebook 110. Thus, QFV212 is an approximation of the feature vector 208 defined by the combination of code vectors specified by the corresponding CFV210. Generally, QFV212 is the sum (e.g., a linear combination) of the code vectors specified by CFV210.

[0057] In some cases, RVQ106 uses a single vector quantizer 108 with a single codebook 110. The quantizer 108 can compress the feature vector 208 to QFV212 by selecting a code vector in its codebook 110 to represent the feature vector 208. The quantizer 106 uses any appropriate distance metric (e.g., error) between the two vectors, e.g., L n The code vector can be selected based on the norm, cosine distance, etc. For example, RVQ106 has the smallest Euclidean norm (e.g., L) for the feature vector 208. 2 A code vector with a norm can be selected. The quantizer 106 can then store the corresponding codeword in the CFV210. Since a codeword generally requires fewer bits than a code vector, the CFV210 can consume less space in memory and achieve greater compression than QFV212 without additional loss.

[0058] Nevertheless, the single vector quantizer 108 method can be extremely expensive because the size of the codebook 110 increases exponentially as the bit rate increases. To overcome this problem, RVQ106 can utilize a sequence of vector quantizers 108. In this case, each vector quantizer 108 in the sequence contains the corresponding codebook 110 of the code vector. RVQ106 can then use an iterative method to generate the CFV210 and the corresponding QFV212, such that each vector quantizer 108 in the sequence further improves the quantization.

[0059] For example, in the first vector quantizer 108, quantizer 106 receives the feature vector 208 and, based on the smallest distance metric, can select a code vector from its codebook 110 to represent the feature vector 208. The residual vector can be calculated as the difference between the feature vector 208 and the code vector representing the feature vector 208. The residual vector can be received by the next quantizer 108 in the sequence to select a code vector from its codebook 110 to represent the residual vector, based on the smallest distance metric. The difference between these two vectors can be used as the residual vector for the next iteration. This iteration can be continued for each vector quantizer 108 in the sequence. Any code vectors identified in this way can be summed in QFV212, and the codeword of each code vector can be stored in its respective CFV210.

[0060] In general, RVQ106 can utilize any suitable number of vector quantizers 108. Number of quantizers N q and the size N of each codebook icontrols the trade-off between computational complexity and coding efficiency. Thus, the sequence of quantizers 108 provides a flexible means of balancing these two opposing factors. In some cases, the size N of each codebook is such that the total bit budget is uniformly allocated across each vector quantizer 108 i = N is equal. The uniform allocation provides practical modularity to the RVQ106 since each codebook 110 consumes the same amount of space in memory.

[0061] Moreover, for a fixed-size codebook N i the number of vector quantizers N q in the sequence determines the resulting bitrate of the QFV212, where a higher bitrate corresponds to a larger number of quantizers 108. Thus, the RVQ106 provides a convenient framework for variable (e.g., scalable) bitrates by employing structured dropout of quantizers 108. That is, the audio compression and restoration system 100 / 200 can vary the number of quantizers 108 in the sequence to target any desired bitrate, facilitating adjustable performance and reducing the overall memory footprint compared to multiple fixed-bitrate coders. These capabilities enable the compression / restoration system 100 / 200 to be adaptable to low-latency implementations suitable especially for devices with limited computing resources (e.g., smartphones, tablets, wristwatches, etc.).

[0062] Next, the CFV210 can further compress the compressed representation 114 of the audio waveform, for example, using an entropy coder 302. The entropy coder 302 can implement any suitable reversible entropy coding, such as arithmetic coding, Huffman coding, etc.

[0063] Referring to Figure 2, the restoration system 200 receives a compressed representation 114 of the audio waveform. Generally, the compressed audio waveform 114 can represent any type of audio content, such as speech, music, or general audio. Indeed, as mentioned above, the audio compression and restoration systems 100 / 200 may be implemented for specific tasks (e.g., speech-adapted compression / restoration) to optimize around a particular type of audio content.

[0064] The compressed audio waveform 114 can be restored to a CFV210, for example, using an entropy codec 302. The CFV210 is then processed to a QFV212 by RVQ106. As described above, each CFV210 contains a codeword (e.g., an index) that identifies a code vector in the respective codebook 110 of each vector quantizer 108. The combination of code vectors specified by each CFV210 identifies the corresponding QFV212. In general, the code vectors identified by each CFV210 are summed to the corresponding QFV212.

[0065] Next, the QFV212 may be processed (e.g., decoded) by the decoder 104 to generate an audio waveform 112. The decoder 104 generally replicates the process of the encoder 102 by starting with (quantized) feature vectors and outputting a waveform. The decoder 104 can upsample the QFV212 to generate an output waveform 206 at a higher sampling rate than the input QFV212. For example, the decoder 104 may use multiple convolutional layers with reduced stride to generate the output waveform 206 at a higher sampling rate (e.g., higher time resolution).

[0066] It should be noted that the compression / decompression systems 100 / 200 can be implemented in various different forms, such as being integrated as a single system or as separate systems. Furthermore, each component of the compression / decompression systems 100 / 200 is not confined to a single client device. For example, in some implementations, the compression system 100 stores the compressed audio waveform 114 in local storage, and then the compressed audio waveform 114 is retrieved from local storage by the decompression system 200. In other implementations, the compression system 100 on the transmitter client transmits the compressed audio waveform 114 over a network (e.g., the internet, a 5G cellular network, Bluetooth, Wi-Fi, etc.), and the compressed audio waveform 114 can be received by the decompression system 200 on the receiver client.

[0067] As will be described in more detail below, the neural network architecture can be trained using the training system 300. The training system 300 can enable efficient general-purpose or adapted compression (e.g., speech adapted) by utilizing a suitable set of training examples 116 and various training procedures. In detail, the training system 300 can train an encoder neural network 102 and a decoder neural network 104 together to efficiently encode and decode the feature vectors 208 of various waveforms contained in the training example 116. Furthermore, the training system 300 can train an RVQ 106 to efficiently quantize the feature vectors 208. In particular, each codebook 110 of each cascading vector quantizer 108 can be trained to minimize the quantization error. To facilitate the trainable codebook 110, each vector quantizer 108 can be implemented, for example, as a vector quantized variational autoencoder (VQ-VAE).

[0068] The audio compression / reconstruction system 100 / 200 can be a fully "end-to-end" machine learning method when implementing this data-driven training solution. In the end-to-end implementation, the compression / reconstruction system 100 / 200 leverages a neural network for all tasks involved in training, as well as for post-training inference. Processing such as feature extraction is not performed by an external system. Generally, the training system 300 can utilize unsupervised learning algorithms, semi-supervised learning algorithms, supervised learning algorithms, or more complex combinations thereof. For example, the training system 300 can balance reconstruction loss and adversarial loss to enable audio compression that is both faithful to the original audio in playback and perceptually similar to the original audio.

[0069] In general, the neural networks included in the audio compression / reconstruction systems 100 / 200 may have any suitable neural network architecture that enables the audio compression / reconstruction systems 100 / 200 to perform the functions they describe. In detail, the neural network may each include any suitable neural network layers (e.g., fully connected layers, convolutional layers, attention layers, etc.) in any suitable number (e.g., 5, 10, or 100 layers) and arranged in any suitable configuration (e.g., as a linear sequence of layers).

[0070] In some implementations, the compression / decompression system 100 / 200 leverages a fully convolutional neural network architecture. Figures 8A and 8B show exemplary implementations of such an architecture for the neural networks of the encoder 102 and decoder 104. The fully convolutional architecture can be particularly advantageous for low-latency compression because, compared to fully connected networks (e.g., multilayer perceptrons), it has lower-scale connectivity and filters (e.g., kernels) that can be optimized to limit coding artifacts. Furthermore, the convolutional neural network provides an effective means of resampling the waveform 112, i.e., changing the time resolution of the waveform 112, by using different strides for different convolutional layers.

[0071] In further implementations, the compression / reconstruction system 100 / 200 uses strict causal convolution when implementing a fully convolutional architecture, so that padding is applied only to the past in both training and offline inference, and not to the future. Padding is not required for streaming inference. In this case, the overall latency of the compression and reconstruction system 100 / 200 is entirely determined by the temporal resampling ratio between the waveforms and their corresponding feature vectors.

[0072] Figure 3 shows the operation performed by an exemplary training system 300 to train the encoder neural network 102, the decoder neural network 104, and the residual vector quantizer 106 together. The neural networks are trained end-to-end on an objective function 214 which may include a number of reconstruction losses. In some implementations, the discriminator neural network 216 is also trained to accommodate adversarial losses 218 and, optionally, additional reconstruction losses.

[0073] The training system 300 receives a set of training examples 116. Each training example 116 includes a respective input audio waveform 202 and a corresponding target audio waveform 204 that the neural network is trained to reconstruct. That is, the target waveform 204 may be compared to the resulting output audio waveform 206 to evaluate the performance of the neural network using an objective function 214. In detail, the objective function 214 may include a reconstruction loss that measures the error between the target waveform 204 and the output waveform 206. In some cases, a point-wise reconstruction loss in the raw waveforms is implemented, for example, using the mean squared error between the waveforms.

[0074] However, this type of reconstruction loss can have limitations in some cases, for example, because two different waveforms may sound perceptually equivalent, while similar point-wise waveforms may sound very different. To mitigate this problem, the objective function 214 can utilize multiscale spectral reconstruction loss, measuring the error between the Mel spectrograms of the target waveform 204 and the output waveform 206. A spectrogram characterizes the frequency spectrum of an audio waveform over time, for example, using the Short-Time Fourier Transform (STFT). A Mel spectrogram is a spectrogram transformed to the Mel scale. Since humans generally do not perceive sound frequencies on a linear scale, the Mel scale allows for a more accurate evaluation of frequency components to improve fidelity. For example, the target waveform

[0075]

number

[0076] and output waveform

[0077]

number

[0078] Reconstruction loss L between rec This can include terms that measure the absolute and logarithmic errors (log error) of the Mel spectrogram.

[0079]

number

[0080] However, ||...|| n L n This shows the norm. Other reconstruction losses are possible, but this form of L rec This satisfies strictly appropriate scoring rules that may be desirable for training purposes. Here,

[0081]

number

[0082] This shows the t-th frame (e.g., time slice) of the 64-Bimmer spectrogram, calculated using a window length equal to s and a hop length equal to s / 4. α s The coefficient is,

[0083]

number

[0084] It can be set to [this value].

[0085] As mentioned above, the set of training examples 116 can be selected to enable various modalities of the compression / decompression system 100 / 200, such as general-purpose audio compression and speech-adapted compression. For example, to train for general-purpose audio compression, training examples 116 may include speech, music, and general audio waveforms. In other implementations, training examples 116 may include only music waveforms to facilitate optimal music compression and playback.

[0086] In some cases, the target waveform 204 is equal to the input waveform 202, thereby allowing the neural network to be trained toward a faithful and perceptually similar reconstruction. However, the target waveform 204 may also be modified relative to the input waveform 202 to facilitate more advanced functionalities such as joint compression and expansion. The nature of the expansion can be determined by designing training examples 116 with several qualities. For example, the target waveform 204 could be an audio-expanded version of the input waveform 202 so that the neural network improves upon audio dialogue when reconstructing the waveform. Alternatively or additionally, the target waveform 204 could be a denoising version of the input waveform 202, training the network to suppress background noise. In general, any desired audio expansion can be made possible using this technique.

[0087] In further implementations, the encoder 102 and / or decoder 104 may be conditioned on data typically included in training example 116, which defines whether the target waveform 204 is equal to the input waveform 202 or an extended version of the waveform 202. For example, training example 116 may include a conditioning signal representing two modes (extension enabled or disabled), so that the neural network is trained to enable extension only when a signal is present. To implement this, the encoder 102 and / or decoder 104 may have a dedicated layer, such as a Feature-wise Linear Modulation (FilM) layer, to process the conditioning signal. After training, this technique can enable the audio compression / decompression system 100 / 200 to flexibly control the extension in real time by supplying the conditioning signal through the network. Thus, the compression system 100 can implement this controllable extension to enable compression of acoustic scenes and natural sounds that would otherwise be removed by extension (e.g., denoising).

[0088] Next, we return to the encoder neural network 102. The input waveform 202 is processed by the encoder 102 and encoded into a sequence of feature vectors 208. This process, which may involve numerous encoder network layers, is represented by an encoder function ε that maps the input waveform x to feature vectors y. θ It can be expressed collectively as follows: y(x)=ε θ (x) is obtained. Encoder function ε θ This is parameterized by encoder network parameters θ, which can be updated using the objective function 214 to minimize loss during encoding.

[0089] Next, the feature vector 208 is compressed by RVQ106 to produce the coded CFV210 and the corresponding QFV212. The quantization process of RVQ106, which may involve multiple vector quantizers 108, is performed on the feature vector y to QFV

[0090]

number

[0091] The RVQ function Q that maps to ψ It can be expressed collectively by,

[0092]

number

[0093] Please note that this is how it will be. RVQ function Q ψ It is parameterized by a codebook parameter ψ, which can be updated using the objective function 214 to minimize losses during quantization.

[0094] The training system 300 can minimize the quantization loss associated with RVQ106 by appropriately aligning the code vector with the vector space of the feature vector 208. That is, the codebook parameter ψ can be updated by the training system 300 by backpropagating the gradient of the objective function 214. For example, the codebook 110 can be iteratively updated during training using the exponential moving average of the feature vector 208. The training system 300 can also improve the use of the codebook 110 by performing a k-means algorithm on the first set of training examples 116 and using the learned centroid as the initialization for subsequent training examples 116. Alternatively or additionally, if the code vector is not assigned to the feature vector 208 for a number of training examples 116, the training system 300 can replace it with a random feature vector 208 sampled during the current training example 116. For example, the training system 300 can track the exponential moving average of its assignments to each code vector (with a decay function of 0.99) and replace any code vectors whose statistic is less than 2.

[0095] To adequately train the neural network for variable (e.g., scalable) bitrates, the training system 300 determines a specific number of vector quantizers 108 that will be used for each training example 116, such that the number of quantizers 108 differs among the training examples 116. q You can select [1;N] for each training example 116. For example, the training system 300 selects [1;N] for each training example 116. q In ], n are uniformly randomly selected q Sample the first i=1...n in the sequence. q Only 10⁸ quantizers can be used. Therefore, the network has a range of n. q =1...N qThe system is trained to encode and decode audio waveforms for all corresponding target bitrates, and no architectural changes are required for encoder 102 or decoder 104. After training, the audio compression and decompression system 100 / 200 adapts to the desired bitrate, with a specific number of n quantizers during compression and decompression. q You can choose this option.

[0096] Next, we refer to the decoder neural network 104. QFV212 is processed by decoder 104 and decoded into output audio waveform 206. Similar to encoder 102, decoding is performed by decoder function D φ It can be represented collectively by and may involve numerous decoder network layers. Decoder function D φ The input QFV

[0097]

number

[0098] Output waveform

[0099]

number

[0100] Mapping to,

[0101]

number

[0102] This is how it works. In some implementations, the input waveform x and the output waveform

[0103]

number

[0104] The decoder function D has the same sampling rate, but it does not have to. φ This is parameterized by decoder network parameters φ, which can be updated using the objective function 214 to minimize losses during decoding.

[0105] Output waveform

[0106]

number

[0107] Since the results generally depend on the encoder network parameter θ, the codebook parameter ψ, and the decoder network parameter φ, an objective function 214 containing the reconstruction loss between the output waveform 206 and the target waveform 204 of each training example 116 may be used to update these network parameters. More specifically, the gradient of the objective function 214 may be calculated, for example, using gradient descent to iteratively update the network parameters using backpropagation. Generally, the network parameters are updated with the aim of optimizing the objective function 214.

[0108] In some implementations, the training system 300 leverages a discriminator neural network 216 to incorporate an adversarial loss 218 into the objective function 214, and potentially an additional reconstruction loss. The adversarial loss 218 can enhance the perceived quality of the waveform reconstructed by the neural network. In this case, the discriminator 216 is trained together with the training system 300 to compete with the encoder 102, decoder 104, and RVQ 106. That is, the discriminator 216 is trained to distinguish the target waveform 204 from the output waveform 206, while the encoder 102, decoder 104, and RVQ 106 are trained to deceive the discriminator 216.

[0109] The discriminator 216 has a discriminator score L of k = {1, 2, ..., K}. kBy using a set of these, this can be implemented in the adversarial loss 218, so that each score characterizes the estimated likelihood that the output waveform 206 is not produced as the output from the decoder 104. For example, the discriminator 216 receives the output waveform from the decoder 104.

[0110]

number

[0111] The system receives the signal and processes the waveform using one or more neural network layers to obtain a logit.

[0112]

number

[0113] It is possible to generate g k,t is a discriminator function that maps the input waveform to output logits. k indexes a particular discriminator output, and t indexes a particular logit of the discriminator output. In some implementations, the discriminator 216 utilizes a fully convolutional neural network such that the number of logits is proportional to the length of the input waveform.

[0114] The discriminator 216 uses logit to determine the respective discriminator score L for each discriminator output k. k This can be determined. For example, each score L k This can be determined from the average over the logit, as follows:

[0115]

number

[0116] Here, T k E is the total number of logits for output k. x L is the expected value over x. In some implementations, the adversarial loss L is used. advThis is the discriminator score L k This is an average over a certain period.

[0117]

number

[0118] Adversarial loss L adv This can be included in the objective function 214 to improve the perceived quality of the reconstructed waveform. Furthermore, the discriminator 216 has a discriminator loss function L dis By minimizing the target waveform,

[0119]

number

[0120] Output waveform

[0121]

number

[0122] To distinguish it from the others, it can be trained by the training system 300. In some implementations, L dis It has the following format:

[0123]

number

[0124] Target waveform

[0125]

number

[0126] Generally, it is equal to the input waveform.

[0127]

number

[0128] Note that this depends on the input waveform x, which may be either or an extended version. dis Regarding this, the encoder 102, decoder 104, and RVQ 106 are trained to efficiently classify the target waveform 204 from the output waveform 206 by training the discriminator 216. adv By minimizing this, the discriminator learns to be deceived.

[0129] In some implementations, the discriminator 216 utilizes different versions of the waveform to achieve a discriminator score L k This determines the discriminator 216. For example, in addition to the original waveform, the discriminator 216 may also use downsampled versions of the waveform (e.g., downsampled twice, downsampled four times, etc.) or Fourier transformed versions of the waveform (e.g., STFT, Hartley transform, etc.), thereby adding diversity to the adversarial loss 218. As a specific implementation form using four discriminator scores, L k=1 The score can correspond to the STFT waveform, but the discriminator score L k=2,3,4 This can correspond to the original waveform, a waveform downsampled twice, and a waveform downsampled four times.

[0130] In further implementations, the discriminator 216 introduces a reconstruction loss in the form of a "feature loss". Specifically, the feature loss L feat This measures the error between the discriminator's internal layer output for the target audio waveform 204 and the discriminator's internal layer output for the output audio waveform 206. For example, feature loss L feat For each layer l∈{1,2,...,L}, the target waveform

[0131]

number

[0132] and output waveform

[0133]

number

[0134] Discriminator output for

[0135]

number

[0136] It can be expressed as the absolute difference between them, and is as follows:

[0137]

number

[0138] Feature loss can be a useful tool for facilitating increased fidelity between the output waveform 206 and the target waveform 204. Taking all the above loss terms into account, the objective function L can control the trade-offs between reconstruction loss, adversarial loss, and feature loss. L=λ rec L rec +λ adv L adv +λ feat L feat

[0139] Weight coefficient λ rec , λ adv , and λ feat By using and weighting the appropriate loss term, the objective function 214 can emphasize several properties, such as faithful reconstruction, fidelity, and perceived quality. In some implementations, the weighting coefficient is λ rec =λ adv =1 and λ feat Set to =100.

[0140] Figure 4 is a flowchart of an exemplary process 400 for compressing an audio waveform. For convenience, process 400 is described as being performed by a system of one or more computers located in one or more locations. For example, an audio compression system appropriately programmed according to this specification, e.g., the audio compression system 100 in Figure 1, can perform process 400.

[0141] The system receives an audio waveform (402). The audio waveform contains each audio sample at each of several time steps. In some cases, the time steps may correspond to a specific sampling rate.

[0142] The system uses an encoder neural network to process the audio waveform and generate a feature vector representing the audio waveform (404).

[0143] The system processes each feature vector using multiple vector quantizers to generate a coded representation of the feature vector (406), where each vector quantizer is associated with a codebook of the code vector. Each coded representation of the feature vector identifies multiple code vectors, including the code vectors from the codebooks of each vector quantizer that define each quantized representation of the feature vector. In some implementations, each quantized representation of a feature vector is defined by the sum of multiple code vectors.

[0144] The system compresses the coded representation of the feature vectors to generate a compressed representation of the audio waveform (408). In some implementations, the system uses entropy coding to compress the coded representation of the feature vectors.

[0145] Figure 5 is a flowchart of an exemplary process 500 for restoring a compressed audio waveform. For convenience, process 500 is described as being performed by a system of one or more computers located in one or more locations. For example, an audio restoration system appropriately programmed according to this specification, such as the audio restoration system 200 in Figure 2, can perform process 500.

[0146] The system receives a compressed representation of the input audio waveform (502).

[0147] The system reconstructs a compressed representation of the audio waveform to obtain a coded representation of the feature vectors representing the input audio waveform (504). In some implementations, the system uses entropy decoding to reconstruct the compressed representation of the input audio waveform.

[0148] The system identifies multiple code vectors for each coded representation of a feature vector, including code vectors from the codebook of each vector quantizer that define each quantized representation of the feature vector (506). In some implementations, each quantized representation of a feature vector is defined by the sum of multiple code vectors.

[0149] The system uses a decoder neural network to process a quantized representation of the feature vector to generate an output audio waveform (510). The output audio waveform may contain each audio sample at each of multiple time steps. In some cases, the time steps may correspond to a specific sampling rate.

[0150] Figure 6 is a flowchart of an exemplary process 600 for generating a quantized representation of a feature vector using a residual vector quantizer. For convenience, process 600 is described as being performed by one or more computer systems located in one or more locations.

[0151] The system receives a feature vector in the first vector quantizer in the sequence of vector quantizers (602).

[0152] The system identifies, based on the feature vector, the code vector from the codebook of the first vector quantizer in the sequence to represent the feature vector (604). For example, a distance metric (e.g., error) may be calculated between the feature vector and each code vector in the codebook. The code vector with the smallest distance metric may be selected to represent the feature vector.

[0153] The system determines the current residual vector based on the error between the feature vector and the code vector representing the feature vector (606). For example, the residual vector may be the difference between the feature vector and the code vector representing the feature vector. The codeword corresponding to the code vector representing the feature vector may be stored in the coded representation of the feature vector.

[0154] The system receives the current residual vector generated by the previous vector quantizer in the sequence at the next vector quantizer in the sequence (608).

[0155] The system identifies a code vector from the codebook of the next vector quantizer in the sequence to represent the current residual vector, based on the current residual vector (610). For example, a distance metric (e.g., error) may be calculated between the current residual vector and each code vector in the codebook. The code vector with the smallest distance metric may be selected to represent the current residual vector. The codeword corresponding to the code vector representing the current residual vector may be stored in the coded representation of the feature vector.

[0156] The system updates the current residual vector based on the error between the current residual vector and the code vector representing the current residual vector (612). For example, the current residual vector may be updated by subtracting the code vector representing the current residual vector from the current residual vector.

[0157] Steps 608–612 may be repeated for each remaining vector quantizer in the sequence. The last coded representation of a feature vector contains the codewords of each code vector selected from its respective codebook during process 600. The quantized representation of a feature vector corresponds to the sum of all code vectors specified by the codewords of the coded representation of the feature vector. In some implementations, the codebooks of vector quantizers in the sequence contain an equal number of code vectors so that each codebook is allocated the same space in memory.

[0158] Figure 7 is a flowchart of an exemplary process 700 for training an encoder neural network, a decoder neural network, and a residual vector quantizer together. For convenience, process 700 is described as being performed by a system of one or more computers located in one or more locations. For example, a training system appropriately programmed according to this specification, e.g., training system 300 in Figure 3, can perform process 700.

[0159] The system obtains training examples containing each input audio waveform and its corresponding target audio waveform (702). In some implementations, one or more target audio waveforms in the training examples may be extended versions of the input audio waveforms, such as denoised versions of the input audio waveforms. One or more target audio waveforms in the training examples may also be the same as the input audio waveforms. Alternatively or additionally, the input audio waveforms may be speech or musical waveforms.

[0160] The system uses an encoder neural network, multiple vector quantizers, and a decoder neural network to process the input audio waveform for each training example to generate the corresponding output audio waveform (704), where each vector quantizer is associated with its respective codebook. In some implementations, the encoder and / or decoder neural networks are conditioned on data that defines whether the corresponding target audio waveform is the same as the input audio waveform or an extended version of the input audio waveform.

[0161] The system uses backpropagation, for example, to determine the gradient of the objective function, which depends on the respective output and target audio waveforms for each training example (706).

[0162] The system uses the gradient of the objective function to update one or more of the encoder network parameters, the decoder network parameters, or the codebooks of multiple vector quantizers (708). For example, the parameters can be updated using any suitable gradient descent optimization technique, such as update rules like RMSprop or Adam.

[0163] Figures 8A and 8B show an example of a fully convolutional neural network architecture for the encoder 102 and decoder 104. C represents the number of channels, and D is the number of dimensions of the feature vector 208. The architecture in Figures 8A and 8B is based on the SoundStream model developed by N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi in "SoundStream: An End-to-End Neural Audio Codec" in IEEE / ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 495-507, 2022. This model is an adaptation of the SEANet encoder-decoder network without skip coupling, designed by Y. Li, M. Tagliasacchi, O. Rybakov, V. Ungureanu, and D. Roblek, "Real-Time Speech Frequency Bandwidth Extension," ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 691-695.

[0164] Encoder 102 includes a Conv1D layer 802 and four subsequent EncoderBlocks 804. Each block includes three ResidualUnits 812 containing dilated convolutions with dilation rates of 1, 3, and 9, respectively, and a downsampling layer in the form of a subsequent strided convolution. The internal convolutional layers of EncoderBlock 804 and ResidualUnit 812 are shown in Figure 8B. The number of channels is doubled whenever downsampling occurs. The final Conv1D layer 802, with a kernel of length 3 and a stride of 1, is used to set the dimensionality of the feature vector 208 to D. A FiLM conditioning layer 806 may also be implemented to process a conditioning signal for use in joint compression and expansion. The FiLM layer 806 performs a feature-wise affine transformation on the neural network's feature vector 208, using the conditioning signal as a condition.

[0165] In this case, the decoder 104 effectively replicates the encoder 102. The DecoderBlock 810 includes a transposed Conv1D layer 814 for upsampling and three subsequent ResidualUnits 812. The internal convolutional layers of the DecoderBlock 810 and ResidualUnits 812 are shown in Figure 8B. The decoder 104 reconstructs the waveform with the same resolution as the input waveform, using the same stride as the encoder 102, but in reverse. The number of channels is halved whenever upsampling occurs. A final Conv1D layer 802 with one filter, a kernel of size 7, and a stride of 1 projects the feature vector 208 back onto the waveform 112. A FiLM conditioning layer 806 may also be implemented to process the conditioning signal for joint reconstruction and expansion. In some implementations, both the encoder 102 and decoder 104 perform audio expansion, while in others, only one of the encoder 102 or decoder 104 is responsible.

[0166] This specification uses the term “configured” in relation to system and computer program components. One or more computer systems being configured to perform a particular operation or action means that the system has installed software, firmware, hardware, or a combination thereof that causes the system to perform the operation or action while in operation. One or more computer programs being configured to perform a particular operation or action means that one or more programs, when executed by a data processing device, contain instructions that cause the device to perform the operation or action.

[0167] The subject matter and functional operating embodiments described herein may be implemented in digital electronic circuits, in tangibly implemented computer software or firmware, in computer hardware, or in one or more combinations thereof, including the structures disclosed herein and their structural equivalents. Embodiments of the subject matter described herein may be implemented as one or more modules of computer program instructions encoded on a tangible non-temporary storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage board, a random or sequential access memory device, or one or more combinations thereof. Alternatively or additionally, the program instructions may be encoded on artificially generated propagating signals, such as machine-generated electrical signals, optical signals, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0168] The term “data processing device” refers to data processing hardware and encompasses all types of devices, machines, and equipment for processing data, including, for example, programmable processors, computers, or multiple processors or computers. A device may also be, or further include, dedicated logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In some cases, in addition to hardware, a device may include code that creates an execution environment for computer programs, such as processor firmware, protocol stacks, database management systems, operating systems, or code that constitutes one or more of these.

[0169] Computer programs, sometimes called, or sometimes described as, programs, software, software applications, apps, modules, software modules, scripts, or code, may be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and may be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A program may, but does not have to, correspond to a file in a file system. A program may be stored in a single file dedicated to the program in question, in part of a file holding one or more scripts stored in other programs or data, such as a markup language document, or in a set of coordinated files, such as a file storing one or more modules, subprograms, or parts of code. A computer program may be deployed to run on one computer, or on multiple computers located in one site, or distributed across multiple sites and interconnected by a data communication network.

[0170] In this specification, the term “engine” is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine, while in other cases, multiple engines may be installed and running on the same one or more computers.

[0171] The processes and logic flows described herein may be executed by one or more programmable computers running one or more computer programs to perform their functions by operating on input data and generating outputs. The processes and logic flows may also be executed by dedicated logic circuits, such as FPGAs or ASICs, or by a combination of dedicated logic circuits and one or more programmed computers.

[0172] A computer suitable for running computer programs may be based on a general-purpose or dedicated microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit will receive instructions and data from read-only memory or random-access memory, or both. Essential elements of a computer are the central processing unit for executing or running instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated into dedicated logic circuits. Generally, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or be operablely coupled to them for receiving data from them, transferring data to them, or both. However, a computer is not required to have such devices. Moreover, a computer may be embedded in another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, such as a Universal Serial Bus (USB) flash drive, to name just a few.

[0173] Computer-readable media suitable for storing computer program instructions and data include, for example, all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0174] To provide user interaction, embodiments of the subject matter described herein may be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices may be used similarly to provide user interaction, for example, the feedback given to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic input, voice input, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser in response to a request received from a web browser on the user's device. The computer may also interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving a response message from the user in return.

[0175] Data processing devices for implementing machine learning models may also include, for example, dedicated hardware accelerator units for handling machine learning training or fabrication, i.e., inference, common and computationally intensive parts of the workload.

[0176] Machine learning models can be implemented and deployed using machine learning frameworks, such as the TensorFlow framework.

[0177] Embodiments of the subject matter described herein may be implemented in a computing system that includes, for example, a data server as a backend component, or a middleware component, for example, an application server, or a frontend component, for example, a client computer having a graphical user interface, a web browser, or an application through which a user can interact with the implementation of the subject matter described herein, or any combination of one or more such backend components, middleware components, or frontend components. The components of the system may be interconnected by digital data communications in any form or medium, for example, a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), for example, the Internet.

[0178] A computing system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises from computer programs running on each computer and having a client-server relationship with each other. In some embodiments, the server sends data, such as an HTML page, to a user device to display data to a user interacting with a device acting as a client and to receive user input from the user. Data generated on the user device, such as the results of user interaction, may be received from the device to the server.

[0179] This specification includes numerous specific implementation details, which should not be construed as limitations on the scope of any invention or claim, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Some features described herein in relation to separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in relation to a single embodiment may also be implemented separately in multiple embodiments or in any preferred partial combination. Furthermore, features may be described above as acting in several combinations, and may even be initially claimed as such, but one or more features from a claimed combination may, in some cases, be removed from that combination, and the claimed combination may be a partial combination or a variation of a partial combination.

[0180] Similarly, while operations are illustrated in the drawings and described in the claims in a specific order, this should not be understood as requiring such operations to be performed in a specific illustrated order or sequence, or that all illustrated operations must be performed to achieve a desired result. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged in multiple software products.

[0181] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions described in the claims may be performed in a different order and still achieve the desired results. As an example, the process illustrated in the accompanying drawings does not necessarily require the specific order or sequence shown to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of Symbols]

[0182] 100 Audio Compression Systems, Compression Systems 102 Encoder Neural Network, Encoder 104 Decoder Neural Network, Decoder 106 Residual vector quantizer, residual (e.g., multi-stage) vector quantizer RVQ, RVQ, quantizer 108 Vector quantizer, quantizer 110 Codebooks, Training Codebooks 112 Audio waveforms, waveforms 114 Compressed representation of audio waveforms, compressed audio waveforms 116 Training Examples 200 Audio Restoration System, Restoration System 202 Input waveforms, each input audio waveform, waveform 204 Corresponding target audio waveform, target waveform, target audio waveform 206 Output waveform, obtained output audio waveform, output audio waveform 208 feature vectors, compressed feature vectors 210 Codified representation of feature vectors (CFV), CFV 212 Quantized representation of feature vectors (QFV), QFV, Input QFV 214 Objective Function 216 Discriminant neural network, discriminant 218 Hostile Losses 300 Training Systems 302 Entropy Codec 802 Conv1D layer 804 EncoderBlock 806 FiLM conditioning layer, FiLM layer 810 DecoderBlock 812 ResidualUnit 814 Transposed Conv1D layer

Claims

1. A method performed by one or more computers, A step of receiving each coded representation of each of a plurality of feature vectors representing an audio waveform, The coded representation of each feature vector comprises multiple codebook symbols that identify each covector from each codebook of each vector quantizer in the sequence of vector quantizers, and A step of generating a quantized representation of each feature vector from each of the code vectors identified by the coded representation of the feature vector, To generate the estimation of the audio waveform, the decoder neural network is used to process the quantized representation of the feature vector. A method that includes [a certain feature].

2. The method according to claim 1, wherein the quantized representation of each of the feature vectors is defined by the sum of the coded representations identified by the coded representation of the feature vector.

3. The method according to claim 1, wherein all of the codebooks of the vector quantizer contain an equal number of code vectors.

4. The sequence of vector quantizers is configured to generate a coded representation of the input feature vector that defines a quantized representation of the input feature vector, and for the first vector quantizer in the sequence of vector quantizers, The steps include receiving the aforementioned input feature vector, A step of identifying each code vector from the codebook of the vector quantizer to represent the feature vector based on the input feature vector, (i) a step of determining the current residual vector based on the error between the input feature vector and (ii) the code vector representing the input feature vector, The coded representation of the input feature vector identifies the code vector that represents the input feature vector, and The method according to claim 1, comprising:

5. To generate the coded representation of the input feature vector that defines the quantized representation of the input feature vector, for each vector quantizer after the first vector quantizer in the sequence, The steps include receiving the current residual vector generated by the previous vector quantizer in the sequence of the vector quantizers, A step of identifying, based on the current residual vector, each code vector from the codebook of the vector quantizer to represent the current residual vector, If the vector quantizer is not the last vector quantizer in the sequence of vector quantizers, (i) updating the current residual vector based on the error between the current residual vector and (ii) the code vector representing the current residual vector, wherein the coded representation of the input feature vector identifies the code vector representing the current residual vector. The method according to claim 4, further comprising:

6. The decoder neural network and the codebook of the vector quantizer are trained together with the encoder neural network, and the encoder neural network Receiving the input audio waveform, To process the input audio waveform in order to generate multiple feature vectors representing the input audio waveform, The method according to claim 4, configured to perform the following:

7. The training is (i) obtain multiple training examples, each containing an input audio waveform and (ii) a corresponding target audio waveform. Using the encoder neural network, the vector quantizer sequence, and the decoder neural network, the respective input audio waveforms from each training example are processed to generate the respective output audio waveforms for the training example, which are estimates of the corresponding target audio waveforms. The gradient of the objective function is determined, depending on the output audio waveform and the target audio waveform of each training example. (i) a set of encoder neural network parameters, (ii) a set of decoder neural network parameters, or (iii) using the gradient of the objective function to update one or more of the codebooks of the vector quantizer The method according to claim 6, including the method described in claim 6.

8. The method according to claim 7, wherein for one or more of the training examples, the target audio waveform is an extended version of the input audio waveform.

9. The method according to claim 8, wherein for one or more of the training examples, the target audio waveform is a denoised version of the input audio waveform.

10. The method according to claim 8, wherein for one or more of the training examples, the target audio waveform is the same as the input audio waveform.

11. Processing the respective input audio waveforms of each training example in order to generate the corresponding output audio waveform, Conditioning the encoder neural network, the decoder neural network, or both on data that defines whether the corresponding target audio waveform is (i) the input audio waveform, or (ii) an extended version of the input audio waveform. The method according to claim 10, comprising:

12. For each training example, The steps include selecting the number of vector quantizers to be used when quantizing the feature vector representing the input audio waveform, A step of generating the corresponding output audio waveform using only the selected number of vector quantizers from the sequence of vector quantizers. The method according to claim 7, further comprising:

13. The method according to claim 12, wherein the selected number of vector quantizers used to quantize the feature vector representing the input audio waveform varies among training examples.

14. For each training example, the step of selecting the respective number of vector quantizers to be used when quantizing the feature vector representing the input audio waveform is: The step of randomly sampling the number of vector quantizers that will be used when quantizing the feature vector representing the input audio waveform. The method according to claim 13, including the method described in claim 13.

15. The method according to claim 7, wherein the objective function comprises, for each training example, a reconstruction loss for measuring the error between (i) each output audio waveform and (ii) the corresponding target audio waveform.

16. The method according to claim 15, wherein for each training example, the reconstruction loss measures the multiscale spectral error between (i) each output audio waveform and (ii) the corresponding target audio waveform.

17. The training further includes, for each training example, The process comprises the step of processing data derived from the output audio waveform using a discriminant neural network to generate a set of one or more discriminant scores, Each discriminator score characterizes the estimated likelihood that the output audio waveform is an audio waveform generated using the encoder neural network, the vector quantizer sequence, and the decoder neural network. The objective function comprises an adversarial loss that depends on the discriminator score generated by the discriminator neural network. The method according to claim 7.

18. The method according to claim 17, wherein the data derived from the output audio waveform comprises the output audio waveform, a downsampled version of the output audio waveform, or a Fourier transformed version of the output audio waveform.

19. A system, One or more computers, One or more storage devices that are communicably coupled to one or more computers, and when executed by one or more computers, Receiving the respective coded representations of each of the multiple feature vectors representing the audio waveform, Each feature vector's coded representation includes multiple codebook symbols that identify each covector from each codebook of the vector quantizer in the sequence of vector quantizers, From each of the code vectors identified by the coded representation of the feature vector, a quantized representation of each feature vector is generated. To generate the estimation of the audio waveform, a decoder neural network is used to process the quantized representation of the feature vector. One or more storage devices that store the commands to perform the action A system equipped with these features.

20. One or more non-temporary computer-readable storage media storing instructions, wherein when an instruction is executed by one or more computers, the one or more computers shall Receiving the respective coded representation of each of the multiple feature vectors representing the audio waveform, Each feature vector's coded representation includes multiple codebook symbols that identify each covector from each codebook of the vector quantizer in the sequence of vector quantizers, From each of the code vectors identified by the coded representation of the feature vector, a quantized representation of each feature vector is generated. To generate the estimation of the audio waveform, a decoder neural network is used to process the quantized representation of the feature vector. One or more non-temporary computer-readable storage media that perform operations including the following.

Images