Signal coding using potential feature prediction

Abstract

Techniques and solutions for encoding and decoding signals, such as audio data, are described. The disclosed innovations can be particularly used for voice coding and decoding applications, such as for real-time communication. Using a neural network, context coding may be used to encode potential features for a current frame using predictions of the potential features from reconstruction of past frames as a context. The extractor learns residual-like features based on this prediction and potential features of the current frame obtained using the encoder. And quantizing similar residual features. At a decoder portion of the codec framework, the quantized features are de-quantized and then combined with predictions from potential features of previous reconstruction to provide reconstructed features of the current frame, which may then be processed by a decoder to provide a reconstructed signal.

Description

Technical Field

[0001] The present disclosure relates generally to signal coding. Certain implementations provide neural coding of audio data using latent feature prediction. Background Art

[0002] Digital technology has been used to record, store and transmit audio information at least since the early 1970s.With the advent of the Internet, digital audio transmission is used explosively, including real-time streaming use for applications such as IP voice and services, including Microsoft Teams (Microsoft Corporation, Redmond, Washington). Although the computing power of personal computing devices continues to improve, with the networking infrastructure, it is still of interest to provide improved audio quality while reducing the amount of data required to convey audio information.Especially, real-time audio may be more sensitive to transmission and processing delays, because only limited buffering can be used for audio signals.For example, the delay in audio processing may hinder the participants in the call from communicating effectively with each other.Therefore, there is room for improvement. Summary of the invention

[0003] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0004] Techniques and solutions for encoding and decoding signals such as audio data are described. The disclosed innovations may be particularly useful in speech codec applications, such as for real-time communications. Using a neural network, context coding may be used to encode latent features of a current frame using predictions of reconstructed latent features from past frames as context. An extractor learns class residual features based on the latent features of the current frame obtained using an encoder and such predictions. The class residual features are then quantized. At the decoder portion of the codec framework, the quantized class residual features are dequantized and then combined with predictions from previously reconstructed latent features to provide reconstructed features for the current frame, which may then be processed by a decoder to provide a reconstructed signal.

[0005] In one aspect, a method for encoding a signal such as digital audio data is provided. One or more latent features are extracted from a frame of an input signal using an encoder. A prediction of the one or more latent features is determined using reconstructed latent features for a plurality of previous frames. A class residual feature is extracted from the one or more latent features and the prediction. The class residual feature or data sufficient to reconstruct the class residual feature is sent to a client.

[0006] The present disclosure also includes a computing system and a tangible, non-transitory computer-readable storage medium configured to implement or include instructions for implementing the above-described methods.As described herein, various other features and advantages can be incorporated into the technology as desired. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] FIG. 1 is a diagram of a prior art vector quantized variational autoencoder.

[0008] Figure 2 is a diagram of a vector quantized variational autoencoder according to an embodiment of the present disclosure.

[0009] Figure 3 It presents the Figure 2 Illustration of the filtering technique used in a vector quantized variational autoencoder.

[0010] Figure 4 Provides a Figure 2 Illustration of a variation of a variational encoder for vector quantization.

[0011] Figure 5 is a graphic representation of the audio quality of different audio encoding techniques at different bit rates.

[0012] Figure 6 Provides a Figure 2 Table of comparison results of various implementations of vector quantized variational autoencoders.

[0013] Figure 7 is an illustration of an overall system for using latent feature predictions as context in a vector quantized variational autoencoder according to the present disclosure.

[0014] Fig. 8A and 8B Provided Figure 7 Details of the encoder and decoder parts of the system.

[0015] Fig. 9 is a diagram showing a technique for group-by-group vector quantization.

[0016] Fig.10 is a flow chart of an example signal encoding technique according to the present disclosure.

[0017] Fig.11 is a diagram of an example computing system in which some described embodiments may be implemented.

[0018] Fig.12 is an example cloud computing environment that can be used in conjunction with the techniques described herein. DETAILED DESCRIPTION Example 1 - Overview

[0019] At least since the early 1970s, digital technology has been used to record, store and transmit audio information. With the advent of the Internet, digital audio transmission has been used explosively, including real-time streaming use for applications such as IP voice and services, including Microsoft Teams (Microsoft Corporation, Redmond, Washington). Although the computing power of personal computing devices continues to improve, with networking infrastructure, it is still of interest to provide improved audio quality while reducing the amount of data required to convey audio information. In particular, real-time audio may be more sensitive to transmission and processing delays, because only limited buffering can be used for audio signals. For example, the delay in audio processing may hinder the participants in a call from communicating effectively with each other. Therefore, there is room for improvement.

[0020] Artificial intelligence / machine learning techniques (such as neural networks) have been applied to audio data, including for real-time communications. Existing neural audio codecs can be divided into two categories. One type of neural audio codec is based on a generative decoder model. At least some generative decoder models extract acoustic features from audio data for encoding after quantization and entropy coding. Strong decoders are used to recover the waveform based on the generative model.

[0021] Another type of audio codec that has been studied is based on end-to-end neural audio codecs. End-to-end neural networks typically use a VQ-VAE (Vector Quantization-Variational Autoencoder) framework (an example of which is shown in FIG1 ) to learn an encoder 110, a vector quantizer 120, and a decoder 130 in an end-to-end manner, as shown in FIG1 . Most of the latent features to be quantized produced by the encoder are blindly learned using a convolutional network (CNN) without any prior knowledge of their semantics. These methods can improve codec efficiency by achieving high quality at low bit rates. However, temporal correlations are not fully utilized in these algorithms. There is still a lot of redundancy between adjacent frames in the encoded features. In contrast, the disclosed innovation incorporates context coding into a VQ-VAE-based neural codec framework to remove such redundancy in the latent domain, thereby further improving codec efficiency.

[0022] Prediction has been used for redundancy removal in image, video and audio codecs such as JPEG, HEVC, H.264 / AVC and DPCM / ADPCM. In image and intra-frame codecs of video, reconstructed neighboring blocks are used to predict the current block in the pixel or frequency domain, and the predicted residual is quantized and encoded into the bitstream. In inter-frame codecs of video codes, reconstructed reference frames are used to predict the current frame with motion compensation. The residual after prediction is too sparse and the entropy is greatly reduced. In neural video codecs, such temporal correlation can be exploited by utilizing motion-aligned reference frames as predictions or contexts for encoding the current frame. In audio codecs, DPCM / ADPCM has been used to encode audio samples or acoustic parameters. However, such techniques have not yet been studied for use in neural audio codecs.

[0023] The present disclosure provides the introduction of contextual coding with temporal prediction into the VQ-VAE framework for neural audio coding. To reduce latency, the prediction is performed in a latent representation. Unlike traditional video / audio codecs that determine the residual by subtracting samples from the prediction, a learnable extractor and synthesizer are used to fuse the prediction with the latent features and quantized output.

[0024] The disclosed innovations are particularly applicable to low-latency speech codecs, but can be incorporated into other codec technologies and can be used with other types of signals other than audio speech data, and including data other than audio data. The present disclosure provides many innovations that can, but are not required to, be used together. These innovations include the use of time-frequency bins as inputs to neural encoders, learnable amplitude compression, delay domain context coding for end-to-end neural audio codecs, rate-controllable improved vector quantization techniques, and scalable coding frameworks in which the availability of higher transmission bit rates can be used to provide scalable quality using the same coding framework. Example 2 - Example Variational Autoencoder with Temporal Filtering

[0025] In one aspect, the present disclosure provides a codec that includes a neural network that uses time-frequency input and can be referred to as a "TFNet". A specific implementation 200 of TFNet is described in Figure 2. The implementation includes a causal 2D encoder 204, the output of which is processed using a temporal filter 208, which includes a temporal convolution module (TCM) and a group-by-group gated recurrent unit (G-GRU) in an interleaved manner. The output of the filter 208 is quantized by a vector quantizer 212 using a codebook 216 to provide a quantized input 220. The quantized input 220 (such as after transmission through a network) is then provided to a temporal filter 224 configured for the filter 208, which includes a temporal convolution module interleaved with a group-by-group gated recurrent unit. The output of the filter 224 is provided to a causal 2D decoder 228. The operation of the TFNet implementation 200 will now be further described.

[0026] TNets-based codecs take a time-frequency spectral input. The time-frequency spectral input can be obtained by dividing the audio samples into overlapping windows and applying a short-time Fourier transform (STFT) to each windowed input to obtain a frame, where the hop size determines the frequency at which the input is processed. Although these parameters can be selected as desired, a 20ms window size with a 5ms hop length can provide good results when used for speech processing.

[0027] Optionally, the input may be further processed before being provided to the encoder neural network. In particular, power law compression of the amplitude may be applied to the input. The dynamic range of speech may be high due to harmonics. Compression is used to normalize the input so that the importance of different frequencies is balanced and training is more stable. Optionally, other compression techniques may be used to compress the amplitude of the input to the encoder 204.

[0028] The encoder 204 utilizes its local two-dimensional (2D) correlation. The time filter 208, 224 utilizes the long-term time dependency with the past frame for feature extraction. This two-stage feature extraction helps to learn to extract features with good representation ability, provides error recovery to packet loss, and if necessary, may remove undesirable information, such as background noise. The learned features are then quantized by the learned vector quantizer and encoded and decoded with a fixed length codec or Huffman codec. For decoding, there are several time filtering blocks, followed by a decoder for reconstruction. If power law compression is applied to amplitude in encoding, inverse power law compression can be applied to the amplitude of the decoded spectrum. Considering the packet loss in real-time communication, decoding should preferably be resilient to these losses with recovery ability and minimum error propagation. Therefore, a heterogeneous structure is provided, which has more time filtering blocks for decoding than encoding.

[0029] The entire network is trained end-to-end to optimize reconstruction quality under rate constraints. Convolutions are causal in the time dimension, allowing the system to maintain low latency, such as 20ms in some examples. Example 3 - TFNet Encoder and Decoder

[0030] refer to Figure 2 , the encoder 204 includes several causal 2D convolutional layers, each of which is followed by batch normalization (BN) and parameterized ReLU (PRe Lu) for nonlinearity. After each convolutional layer, the features are downsampled by 2 or 4 in the frequency dimension, and finally all frequency information is folded into channels.

[0031] Let X I ∈R T×F×2 represents the input feature. After being processed by the encoder 204, the feature is R ∈R T ×F×2 , which is used as input to the temporal filter 208. T, F and C are the number of frames, frequency bins and channels respectively. The convolution is causal along the time dimension, so T is preserved without any downsampling. The decoder 228 is symmetrical to the encoder 204 with a causal 2D deconvolution layer. The output of the decoder is the reconstructed spectrum X R ∈R T×F×2 , which is processed using an inverse short-time Fourier transform to provide an output waveform. Example 4 - Example Temporal Filtering

[0032] As described in Example 2, and as Figure 3 As shown, the filters 208, 224 implemented by TFNet include a dilated temporal convolution module (TCM) 300 and a group-by-group gated recurrent unit (G-GRU) 350. Both filter elements are causal and low complexity. The TCM module can be similar to the implementation described in Pandey et al., "TCNN: Temporal convolutional neural network for real-time speech enhancement in the time domain", IEEE International Conference on Acoustics, Speech and Signal Processing, 6875-6879 (2019). According to the Pandey reference, it provides: The residual block consists of 3 convolutions: input 1x1 convolution, depthwise convolution, and output 1x1 convolution. Depthwise convolution is used to further reduce the number of parameters. In depthwise convolution, the number of channels remains the same and only one filter per input channel is used for output calculation.

[0033] The TCM module includes two convolutions 304, 308 with a kernel size of 1×1 for changing the channel dimension, and a dilated depthwise convolution 312 for exploiting temporal correlation with low complexity. Several TCM blocks with different dilation rates are grouped into large blocks to increase the receptive field and diversity.

[0034] The group-wise GRU portion of filters 208, 224 splits the channels into N groups and exploits the temporal dependencies within each group independently. The operation of the gated recurrent unit is described in Cho et al., "On the Properties of Neural Machine Translation: Encoder-Decoder Approaches", arXiv:1409.1259 (2014). In particular, the Cho reference describes that an activation function can be used to provide gating: …which adds the usual logistic sigmoid activation function with two gating units called reset r and update z. Each gate depends on the previous hidden state h (t-1) , and the current input x t Control the flow of information.

[0035] Further details of gated recurrent units are provided in Cho et al., “Learning Phase Representations using RNN Encoder-Decoder for Statistical Machine Translation”, arXiv1406.1078v3 (2014). The Cho reference describes: When the reset gate is close to 0, the hidden state is forced to ignore the previous hidden state and reset only with the current input. This effectively allows the hidden state to discard any information that is found to be irrelevant in the future, allowing for a more compact representation. On the other hand, the update gate controls how much information from the previous hidden state will be carried to the current hidden state. This works similarly to how the memory cells in LSTM networks work and helps the RNN remember long-term information.

[0036] This per-group GRU variant not only reduces complexity but also improves flexibility and representational power for providing frequency-aware temporal filtering, as channels are able to learn from frequency. TCM can help explore short-term and mid-term temporal evolutions, while GRU can help capture long-term dependencies. Thus, interleaving these two techniques helps capture short-term and long-term temporal correlations at different depths. The experimental results provided in Example 8 verify that the interleaved structure is more effective than the single structure. Example 5 - Example Vector Quantization

[0037] The vector quantizer discretizes the features learned in encoding using a set of learnable codebooks according to the target bitrate. Before quantization, the encoded feature X Q ∈R T×1×C is reduced to X Q ∈R T×1×C′ . Group quantization is obtained by splitting the channels C’ into N groups and encoding and decoding each group with an independent codebook. Let S denote the number of codewords in each codebook and K = C’ / N the dimension of each codeword. In a specific example of implementation 200, a window length of 20 ms and a hop length of 5 ms are adopted for STFT, and if fixed-length encoding and decoding are used, the bitrate is given by N × log 2 S / 5 kbps. Although other parameter values can be used depending on the situation, for 6 kbps, C’, N, S, and K can be set to 120, 3, 1024, and 40 respectively. Following the technique described in “Neural discrete representation learning” by van den Oord et al., arXiv:1711.00937 (2017), the codebook is learned using exponential moving average. According to this technique, the encoder network outputs discrete codes instead of continuous codes and uses a learned prior instead of a static one. A shared embedding space can be used to determine the discrete codes using a nearest neighbor search process. Learning is provided by passing the gradient from the decoder input to the encoder, as the encoder and decoder share the same dimensional space. The shared embedding space, i.e., the codebook, is updated as a function of the moving average of the encoder output z e (x).

[0038] Specifically, the input x can be passed through the encoder to generate the output z e (x), where a shared embedding space e (with embedding vector e j ) can be used for nearest neighbor search to determine the discrete latent variable z. Then, the encoder output can pass through a discretization bottleneck and then be mapped to the nearest embedding e. The following equation can be used, where q(z = k|x) is the posterior classification distribution probability and z q (x) is the nearest embedding: z q (x) = e k

[0039] Quantified features It may be upscaled to a shape of T×1×C before being provided to the temporal filter 224 in the decoding portion of the implementation 200 . Example 6 - Example Loss Function

[0040] An example loss function that can be used in system 200 is a combination of two terms: is the reconstruction loss, and Put constraints on the vector quantization. For the reconstruction loss, the mean square error can be used on the power-law compressed spectrum between the original and decoded signals. To help provide STFT consistency, the decoded spectrum can be first transformed into the waveform domain by inverse STFT and then transformed again into the time-frequency domain by STFT to calculate the loss. The second term is the commitment loss used in VQ-VAE, which forces the encoder 204 to generate representations close to its codewords, and α is a weighting factor used to balance the two terms. Example 7 - Sample TFNet implementation

[0041] In real-time communication, there are several types of degradations, such as background noise and packet loss, in addition to the quality loss caused by audio codecs. Due to the disclosed end-to-end learnable codec, when used for audio applications, joint optimization of audio codecs using speech enhancement (SE) and packet loss concealment (PLC) is feasible. Two joint optimization approaches are provided: (1) a cascaded network with an enhancer before the codec and a PLC network after the codec ( Figure 4 , network 400), and (2) an integrated network ( Figure 4 , network 450).

[0042] Figure 4 The cascade network 400 of FIG. 4 includes three modules: an enhancer 410 for preprocessing, an audio codec (encoder 420 and decoder 424), and a PLC network 440 for postprocessing. Since speech is more effective in compression than noisy audio, the enhancer 410 is placed before the codec 420. The enhancer 410, the encoder 420 and the decoder 424, and the PLC network 440 can all be based on a TFNet-like structure (such as in Figure 2In other words, for example, encoder 420 may include the functions of encoder 208 and filter 208, and decoder 424 may include the functions of filter 224 and decoder 228.

[0043] The preprocessing enhancer 410 takes noisy audio as input and outputs enhanced audio for feeding into the codec. Unlike the TFNet-based codec implementation 200, there are skip connections between the encoder and decoder in the enhancer 410 to eliminate TFNet information loss. A causal gating block can be used in the decoder to output amplitude gain and phase for reconstruction, which can be implemented in a similar manner to that described in Zheng et al., “Interactive speech and noise modeling for speech enhancement,” AAAI 2021. In Zheng, the gating block “learns a multiplicative mask over the corresponding features from the encoder, aiming to suppress its undesirable parts.”

[0044] In case of packet loss, the neural codec is adjusted so that when decoding it takes as input both the quantized features with zeros for the lost packets and a mask showing where the loss occurred. The mask is also injected into each temporal filtering block in the decoding. The post-processing PLC module 440 operates in the waveform domain, employing a TNet-based structure with both the decoded audio and the mask as input. There are also skip connections in the PLC network 440 as in the enhancer 410. As a recovery task, the PLC network 440 outputs a complex residual in the time-frequency domain, which is added to the spectrum of the decoded audio for reconstruction.

[0045] For training, the three networks can be cascaded and trained jointly end-to-end. For better quality, a two-stage training can be used. First, the enhancer 410 and the codec 420 can be trained separately with noisy and clean data, respectively. Then, by using With the same reconstruction loss, the cascaded network 440 can be fine-tuned with two additional supervisions at the output of the enhancer and the encoder-decoder respectively.

[0046] The integrated network 450 is resilient to both background noise and packet loss using only a single codec network having the same overall structure as the TFNet implementation 200, including an encoder 460 (which includes the functionality of both the encoder 204 and the filter 208) and a decoder 470 (which includes the functionality of both the filter 224 and the decoder 228). To accommodate packet loss, the decoding portion in the codec is adjusted to be similar to the decoding portion in the cascaded network 400. It is trained from scratch with auxiliary supervision added to the encoding portion to remove noise for efficient encoding and decoding. This is achieved by adding a decoder after the temporal filtering block of the encoder, which is forced to output clean audio during training. During inference, the decoder is not needed. Example 8 - Example comparison results

[0047] 890 hours of 16kHz noise audio with clean speech, noise, and room impulses were synthesized from the Deep Noise Suppression Challenge at ICASSP2021. Clean audio includes multilingual speech, emotions, and singing clips. The signal-to-noise ratio is randomly selected between -5dB and 20dB, and the speech level is within -40dB to -10dB. Each audio is cut into 3-second segments for training. Speech enhancement performs both denoising and de-reverberation. Packet loss is simulated according to the three-state model described in Milner et al., "An analysis of packet loss model for distributed speech recognition", Proceedings INTERSPEECH, the 8th International Conference on Spoken Language Processing (2004). In the three-state model, one state corresponds to a "good" state in which no packet loss occurs, another state corresponds to a "bad" state with a probability of packet loss, and the final state can represent a transition from a "good" state to a new state that is not associated with packet loss either. For testing, 1400 audios are used, each 10 seconds long and without any overlap with the training data.

[0048] During training, the Adam optimizer (see Kingma et al., "Adam: A Method for Stochastic Optimization", arXiv:1412:6980 (2014)) was used with a learning rate of 0.0004. The network was trained for 100 epochs using batches of size 200. The "Adam" algorithm is "first-order gradient-based optimization of stochastic objective functions with adaptive estimation of low-order moments".

[0049] In the evaluation, in addition to the subjective listening tests, three metrics were used in the ablation study to evaluate the joint optimization and for evaluating the temporal filter types - PESQ (Perceptual Evaluation of Speech Quality), STOI (Short-term Objective Intelligibility), and DNSMOS (Deep Noise Suppression Mean Opinion Score). Although these metrics were not designed and optimized for exactly the same task, they were found to match the perceptual quality well for the same kinds of distortions in all compared scenarios.

[0050] The codec network was trained and measured on clean data from the Deep Noise Suppression Challenge. Subjective listening tests were performed using a crowdsourcing approach inspired by MUSHRA (Multiple Stimuli with Hidden References and Anchors). There were 10 participants. Each participant evaluated 12 samples. The TNet-based neural codec was compared with Lyra (Neural Speech Codec, Google LLC) and Opus (Xiph.Org Foundation), two codecs for real-time communication. Figure 5 As shown, the disclosed TFNet technology is significantly better than Lyra at around 3kbps, and TFNet at 6kbps is much better than Opus at 6kbps, which proves the superiority of the disclosed TFNet technology.

[0051] Joint optimization of codec, speech enhancement, and PLC (packet loss concealment) is evaluated using noisy / clean paired data with simulated packet loss traces. Three methods are compared: a baseline with separately trained enhancement, codec, and PLC models; a cascaded network; and an all-in-one network. In the baseline, only the original clean data is used to train the codec and PLC networks. The enhancer and PLC networks have 470K parameters and 1.2M MACs per 20ms, much smaller than the codec network with 5M parameters.

[0052] exist Figure 6 Tables 610 and 620 of present the comparison results for the joint optimization of two and three tasks, respectively. It is observed that the two joint optimization methods significantly outperform the baseline in all metrics. Although no pre-processing or post-processing network is used, the performance of the integrated network is competitive with the cascaded network, showing the strong discrimination and representation capabilities of TFNet. Another observation is that the PLC network trained on the original clean data in the baseline method is sensitive to mismatches in the input.

[0053] The interleaved structure in the TFNet neural codec is compared with the separate use of two modules, TCM and GRU, in a regression task commonly used for speech enhancement. All schemes are compared at the same computational complexity, with 1.4M parameters and 3.3M MACs for each 20ms window for encoding and decoding. All temporal filtering modules are only used for decoding to evaluate their recovery capabilities.

[0054] Figure 6 The comparison results are shown in Table 630. It can be seen that the staggered structure performs best for capturing both short-term and long-term temporal correlations. Example 9 - Overview of Vector Quantized Variational Autoencoder with Latent Feature Prediction

[0055] Examples 9-13 describe a low bitrate and scalable contextual neural audio codec for real-time communication based on the VQ-VAE framework. The codec incorporates the features of the codecs described in Examples 1 to 8. The codec of Examples 9-13 learns encoding, vector quantization codebooks, and decoding in an end-to-end manner. Unlike existing neural audio codecs that use acoustic features or blind features learned using convolutional neural networks for encoding, there is still temporal redundancy within the quantized features through convolutional neural networks, and context coding using latent feature prediction is introduced into the VQ-VAE framework to further remove such redundancy. Channel-by-channel group vector quantization with random dropout is used to help provide bitrate scalability in a single model and a single bitstream. Subjective evaluations show that the disclosed technology can achieve acceptable speech quality at 1kbps and near-transparent quality at 6kbps.

[0056] The disclosed technology provides many features and advantages that can be used in real-time communication applications as well as other applications, including for compressing other types of audio information. One feature is that time-frequency bins are used as network inputs for an end-to-end neural audio codec. Another feature is the use of learnable amplitude compression for low bitrate codecs. Latent domain context codecs are used for end-to-end neural audio codecs. The disclosed technology also provides vector quantization features that support rate control. Another feature is channel-by-channel bitrate scalability, where the audio quality can be scaled to a higher level as the bitrate increases. Example 10 - Example Vector Quantized Variational Autoencoder with Latent Feature Prediction

[0057] Figure 7 An example neural codec 700 is shown that performs context encoding and decoding in a latent representation to reduce latency in accordance with the present disclosure and in particular Examples 9-13. The codec 700 is split into an encoding portion 800 and a decoding portion 850, as shown in FIG. Fig. 8A and 8BThe technique is described with particular application to low-delay speech codecs, but can be used in other applications, including as a codec for other types of audio. The basic encoder and decoder networks 800, 850 are similar to those described with respect to Examples 1 to 8.

[0058] Encoder 704 is applied to convert the input audio x( Fig. 8A ) extracts the latent representation r. For each frame r in r t , encoder 704 utilizes the latent code reconstructed from the past by predictor 708 The learned prediction, predictor 708 has a receptive field of N past frames. Then, extractor 712 extracts t and p t The information of the learning class residual in is used for quantization. By using this autoregressive operation, the temporal redundancy can be effectively reduced without introducing any error propagation between frames. Then, the vector quantizer 716 ( Fig. 8A ) using the learned codebook 720 ( Figure 7 ) quantizes the extracted residual-like features and entropy encodes and decodes them using Huffman encoding (although other types of encodings may be used). In particular, the output of the vector quantizer 716 may include quantization indices for the learned codebook 720, which are then entropy encoded, such as into a bitstream. In turn, the bitstream may be sent to a client for decoding. The quantization indices (including encoding into the bitstream) may be referred to as "data sufficient to reconstruct the residual image features."

[0059] In the decoding section 850 ( Figure 8B ), the dequantized class residual features are combined with the prediction p of the latent features from the past reconstruction by the synthesizer 730 t Merge to get the current reconstruction potential code The decoder 734 is then used to reconstruct the waveform In the following examples, these modules are described in detail. Example 11 - Example Amplitude Compression of Input Data

[0060] Typical neural networks obtain time domain samples in end-to-end neural codecs, or obtain Mel-scale features in generative neural codecs. The disclosed technology uses the short-time Fourier transform (STFT) domain for feature extraction. The time-frequency spectrum X of the STFT t,f is used as the encoder input. Due to the harmonics of speech, t,f There is a large dynamic range in X that may make training unstable. In order to balance the importance of different frequencies and bit rates, we also t,f The amplitude of Introduced learnable power compression, where A t,f Yes X t,f The amplitude of γ is the power parameter to be learned during training. Through this learnable amplitude compression, more attention is paid to the main component at low bitrates, while more details will also receive attention at high bitrates. Example 12 - Example Latent Domain Context Encoding and Decoding

[0061] Since the context codec is autoregressive, it is studied in the latent domain in order to reduce the delay (at 750). Figure 7 shown, and in Fig. 8A and 8B The encoding / decoding split is shown in FIG. 7 , and there is a predictor 708 to reconstruct the latent features based on the past To predict the latent feature r t Each frame in. Since the prediction p t may contain some undesirable information for encoding frame t, so in a particular implementation, instead of passing r t -p t The residual codec of r t and p t An extractor is employed to extract new information n for frame t that may not be estimated from the past t . Then use vector quantization to quantize n t . Thus, as used herein, “extractor” / “extract” / “extracting” refers to obtaining “new” information from the latent features and the predictions, rather than simply computing the difference between them (such as via subtraction). The term “residual-like features” is intended to distinguish this “new” information from the residual computed as the difference between the latent features and the predictions. However, in other aspects, residuals computed via such differences (as opposed to residual-like features), or simply encoder outputs in the absence of context encoding and decoding, may be used with various disclosed innovations. Symmetrically, to reconstruct the latent code for the current frame, a synthesizer 730 is employed to merge p t and the dequantized output To get

[0062] Predictor 708 provides a nonlinear prediction of the current frame based on the past, which is obtained by a window of N frames. Given. Two convolutional layers may be used, such as with kernel sizes of 5 and 3, followed by a parameterized ReLU (PRe LU), where ReLU is a rectified linear unit activation function) to obtain a receptive field of N = 7 frames.

[0063] In order to guide the predictor 708 to have good prediction accuracy, a prediction loss L is introduced in the training.p =E(D(p t ,s g (r t ))), where D(·) is the distance metric given by L1. sg(·) is the stopping gradient operator for more stable training.

[0064] Both the extractor 712 and the synthesizer 730 include one convolutional layer with a kernel size of 1, followed by a parameterized ReLU as a non-linear activation function.

[0065] Since quantization is non-differentiable, a technique is used to learn a codebook and perform backpropagation through a vector quantization process. Suitable methods include VQ-VAE with commitment loss, exponential moving average (EMA), Gumbel-Softmax (see Jang et al., "Categorical Reparameterization with Gumbel-Softmax", ICLR 2017), and soft-to-hard (see Agustsson et al., "Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations") arXiv1704.00648v2 (2017). According to Jang, Gumbel-Softmax involves "a continuous distribution over a simplex that can approximate categorical samples and whose parameter gradients can be easily computed using the reparameterization trick". In addition, "the Gumbel-Softmax distribution interpolates between a discrete one-hot encoded categorical distribution and a continuous categorical density". According to Agustsson, "soft-to-hard" uses A soft assignment of a given scalar or vector will be quantized to a quantization level. A parameter controls the "hardness" of the assignment and allows for a gradual transition from soft to hard assignment during training. In contrast to rounding-based or stochastic quantization schemes, our encoding / decoding scheme is directly differentiable and thus trainable end-to-end. Among these methods, Gumbel-Softmax and Soft-to-Hard allow the probability of selecting codewords and thus make rate control feasible.

[0066] However, Gumbel-Softmax uses linear projection to select codewords without explicitly relating them to quantization error. Soft-to-hard techniques give soft assignments based on the distances to different codewords, but a weighted average of codewords rather than a single codeword is used for quantization in training, which leads to a gap between training and inference.

[0067] In view of this, in a specific implementation, a modified mechanism combines the distance-to-soft mapping with Gumbel-Softmax to provide a nonlinear projection as opposed to the linear projection of Gumbel-Softmax. Let K denote the number of codewords in the codebook C. The kth codeword is selected to quantize n t The probability of is given by: d t,k =D(n t ,c k )

[0068] where τ is the temperature of the Gumbel Softmax and vk∈Gumbel(0,1) is a sample drawn from a Gumbel distribution in the given case (0,1). where D(·) is a distance metric and L2 is used for a particular implementation. α is a control for the distance from D(n t ,c k ) to logical values ​​(logit). During the forward pass, a hard assignment can be used, while during the backward pass, the gradient with respect to the logit is used.

[0069] Using q t,k , the rate of each small batch is controlled in the following way:

[0070] Among them, b,t (·) is the expected value over all T frames of all B audio in each training mini-batch. target is based on the average number of target bits per frame. This loss L r Not only does it limit the average rate, but it also Perform rate distortion optimization. When the current entropy is higher than R target When it is lower than R, similar features quantized to the same codeword will be pushed through the trade-off between rate and distortion; and when it is lower than R target When , similar features can be quantized into different codewords to maintain higher quality but higher rate.

[0071] In order to reduce the codebook size for training, group vector quantization is used. t Along the channel dimension ( Fig. 9 ) is split into G groups, and each group n t,c Quantization is performed using a separate codebook with K codewords. In certain implementations, a large codebook size can be used to help capture the actual distribution of the underlying features through rate-distortion optimization. For example, if it is desired to implement 16khz audio for 6kbps, then it is expected that every 20ms of new data will consume 120 bits. The codebook is then calculated by G·log2 (K)>120 is set. Example 13 - Example techniques for bitrate scalability

[0072] Bitrate scalability is a desirable feature for streaming and real - time communication to support different receivers with different network conditions without any transcoding. Bitrate scalability can support multiple bitrates in a single bitstream. Specifically, the bitstream can be split into S layers {Bi|i = 0, 1, …, S - 1}, where B 0 is the base layer, B 1 , B 2 , …, B S-1 are enhancement layers. A receiver with only B 0 will obtain the lowest quality, while a receiver with B 0 , B 1 , B 2 , …, B S-1 , i < S will get higher quality. The best quality is achieved when i = S.

[0073] Existing scalable neural audio codecs typically utilize residual vector quantization to achieve bitrate scalability, where a single codebook is trained for all channels for the lowest bitrate, and more codebooks are used for encoding the residuals between the encoder features and their previous reconstructions for higher bitrates. Instead, per - channel bitrate scalability can be used by leveraging per - channel group VQ that is inactivated (dropped out) during training. As Fig. 9 shown, the i - th group of features n t,i is quantized using a separate codebook. n t,0 can be regarded as the base layer, n t,1 , n t,2 , …, n t,G-1 can be regarded as enhancement layers, and thus G bitrates can be supported. During training, for each mini - batch, a bitrate b s is randomly selected, which only uses {n t,i |i = 0, 1, …, s, s < S}, and the features are set to zero to achieve bitrates higher than b s , i.e., {n t,i |i = s + 1, s + 2, …, S - 1} are set to zero. In this way, the decoder is guided to learn recovery from multiple bitrates, so it can achieve scalability in a single model. Example 14 - Example encoder with latent feature prediction

[0074] Fig.101 is a flow chart of an example signal encoding method 1000 according to the present disclosure. At 1010, one or more latent features are extracted from a frame of an input signal using an encoder. At 1020, a prediction of the one or more latent features is determined using reconstructed latent features for a plurality of previous frames. At 1030, a residual-like feature is extracted from the extracted one or more latent features and the prediction. At 1040, the residual-like feature or data sufficient to reconstruct the residual-like feature is sent to a client for decoding. Example 15 - Additional Examples

[0075] Example 1 is a computing system comprising at least one memory and at least one hardware processor coupled to the at least one memory. The computing system also includes one or more computer-readable storage media storing computer-executable instructions, which when executed cause the computing system to perform various operations. The operations include extracting one or more latent features from a frame of an input signal using an encoder to provide extracted one or more latent features. Determining a prediction of one or more latent features using reconstructed latent features for multiple previous frames. Extracting a class residual feature from the extracted one or more latent features and the prediction. Sending the class residual feature or data sufficient to reconstruct the class residual feature to a client.

[0076] Example 2 includes the subject matter of Example 1, and further specifies that the input signal includes audio data, such as speech data.

[0077] Example 3 includes the subject matter of Example 1 or Example 2, and further specifies that the extracting includes use of at least one convolutional layer.

[0078] Example 4 includes the subject matter of any of Examples 1 to 3, and further specifies that the input signal includes time-frequency spectral data.

[0079] Example 5 includes the subject matter of Example 4, and further specifies that the time-frequency spectral data is obtained using a short-time Fourier transform of a time window of the input signal.

[0080] Example 6 includes the subject matter of Example 4 or Example 5, and further specifies applying amplitude compression to the time-frequency spectral data.

[0081] Example 7 includes the subject matter of Example 6, and further specifies that amplitude compression is applied using values ​​determined during training of the encoder.

[0082] Example 8 includes the subject matter of Example 7, and further specifies that the value is different for different encoding bit rates.

[0083] Example 9 includes the subject matter of any one of Examples 1 to 8, and further specifies that the encoder comprises a plurality of convolutional layers.

[0084] Example 10 includes the subject matter of any of Examples 1 to 9, and further specifies that determining the prediction comprises processing reconstructed latent features for multiple previous frames using multiple convolutional layers.

[0085] Example 11 includes the subject matter of any one of Examples 1 to 10, and further specifies that quantizing the residual-like features comprises splitting the residual-like features into a plurality of groups along a channel dimension, and quantizing the groups in the plurality of groups separately.

[0086] Example 12 includes the subject matter of Example 11, and further specifies that a given group in the plurality of groups includes a plurality of frequencies.

[0087] Example 13 includes the subject matter of Example 12, and further specifies that channels are quantized using different codebooks. For an input training data set used during training of an encoder, a group of the plurality of groups is randomly selected, wherein the groups are associated with sets of progressively higher bit rates. During training of the encoder using the input training data set, only the selected group of the plurality of groups and groups of the plurality of groups associated with a lower bit rate than the selected group are used.

[0088] Example 14 includes the subject matter of any one of Examples 1 to 13, and further specifies that quantizing the class residual feature includes: determining, for a frame, a distance between the class residual and a codeword used for a codebook, the codebook being used for vectorization of the class residual feature, and using at least in part the distance to determine a probability of selecting the codeword.

[0089] Example 15 includes the subject matter of Example 14, and further specifies that the probability is determined as a non-linear projection.

[0090] Example 16 includes the subject matter of Example 14 or Example 15, and further specifies that determining the probability includes selecting an element of a Gumbel distribution.

[0091] Example 17 includes the subject matter of any of Examples 1 to 16, and further specifies that the residual-like features, or data sufficient to reconstruct the residual-like features, are sent as part of a bitstream having a rate. During training of the encoder, a bit rate for training input data is determined, wherein determining the bit rate comprises determining a difference between a target bit rate and an entropy of a probability that a particular codeword of a codebook is selected for a frame of the training input data.

[0092] Example 18 includes the subject matter of Example 17, and further specifies an optimized rate-distortion factor determined as a tradeoff of the determined distortion and a bit rate for the training input data.

[0093] Example 19 is one or more computer-readable media storing computer-executable instructions that, when executed, cause a computing system to perform various operations. The operations include extracting one or more latent features from a frame of an input signal using an encoder to provide extracted one or more latent features. Determine a prediction of one or more latent features using reconstructed latent features for a plurality of previous frames. Extract a class residual feature from the extracted one or more latent features and the prediction. Send the class residual feature or data sufficient to reconstruct the class residual feature to a client. Additional examples include the subject matter of Example 19 and the subject matter of any of Examples 2-18 and 27-31, in the form of computer-executable instructions.

[0094] Example 20 is a method that can be implemented in hardware, software, or a combination thereof. One or more latent features are extracted from a frame of an input signal using an encoder to provide one or more extracted latent features. Predictions of the one or more latent features are determined using reconstructed latent features for a plurality of previous frames. Class residual features are extracted from the extracted one or more latent features and the predictions. The class residual features or information sufficient to reconstruct the class residual features are sent to a client. Additional examples include the subject matter of Example 20 and the subject matter of any of Examples 2-18 and 27-31, in the form of additional elements of the method.

[0095] Example 21 is a computing system comprising at least one memory and at least one hardware processor coupled to the at least one memory. The computing system also includes one or more computer-readable storage media storing computer-executable instructions, which when executed cause the computing system to perform various operations. The operations include receiving a class residual feature or data sufficient to reconstruct a class residual feature. A prediction of one or more potential values ​​is determined using reconstructed potential features for multiple previous frames. The prediction and the class residual feature are combined to provide one or more reconstructed potential features for a frame of an input signal. The one or more reconstructed potential features are provided to a decoder to provide a decoded output signal.

[0096] Example 22 includes the subject matter of Example 21, and further specifies that the output signal includes audio data, such as speech data.

[0097] Example 23 includes the subject matter of Example 21 or Example 22, and further specifies that the decoder comprises a plurality of convolutional layers.

[0098] Example 24 includes the subject matter of any of Examples 21 to 23, and further specifies that determining the prediction comprises processing reconstructed latent features for multiple previous frames using multiple convolutional layers.

[0099] Example 25 includes the subject matter of any one of Examples 21 to 24, and further specifies that data sufficient to reconstruct the residual-like feature comprises: a quantization index for a codebook for dequantization.

[0100] Example 26 includes the subject matter of Example 25, and further specifies that the quantization index is received in the bitstream.

[0101] Example 27 includes the subject matter of any of Examples 1 to 18, and further includes quantizing the class residual features.

[0102] Example 28 includes the subject matter of Example 27, wherein the quantized class residual features provide quantization indices into a codebook used in quantization.

[0103] Example 29 includes the subject matter of Example 28, and further includes encoding and decoding the quantization index into the bitstream.

[0104] Example 30 includes the subject matter of Example 29, and further specifies that the codec is an entropy codec.

[0105] Example 31 includes the subject matter of Example 30, and further specifies that the entropy codec is Huffman codec. Example 16 - Computing System

[0106] Fig.11 A generalized example of a suitable computing system 1100 is depicted in which the described innovations may be implemented. The computing system 1100 is not intended to suggest any limitation as to the scope of use or functionality of the present disclosure, as the innovations may be implemented in various general-purpose or special-purpose computing systems.

[0107] refer to Fig.11 , the computing system 1100 includes one or more processing units 1110, 1115 and memories 1120, 1125. Fig.11 , the basic configuration 1130 is included within the dashed line. Processing units 1110, 1115 execute computer executable instructions, such as for implementing the features described in Examples 1-15. The processing units may be general-purpose central processing units (CPUs), processors in application-specific integrated circuits (ASICs), or any other type of processor. In a multi-processing system, multiple processing units execute computer executable instructions to increase processing power. For example, Fig.11 A central processing unit 1110 and a graphics processing unit or synergistic processing unit 1115 are shown. Tangible memory 1120, 1125 may be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of both accessible by the processing unit(s) 1110, 1115. The memory 1120, 1125 stores software 1180 in the form of computer executable instructions suitable for execution by the processing unit(s) 1110, 1115 that implements one or more innovations described herein.

[0108] The computing system 1100 may have additional features. For example, the computing system 1100 includes storage 1140, one or more input devices 1150, one or more output devices 1160, and one or more communication connections 1170, including input devices, output devices, and communication connections for interacting with a user. An interconnection mechanism (not shown) such as a bus, controller, or network interconnects the components of the computing system 1100. Typically, operating system software (not shown) provides an operating environment for other software executed in the computing system 1100 and coordinates the activities of the components of the computing system 1100.

[0109] Tangible storage 1140 may be removable or non-removable and include disks, tapes or cassettes, CD-ROMs, DVDs, or any other medium that can be used to store information in a non-transitory manner and that can be accessed within computing system 1100. Storage 1140 stores instructions for software 1180 implementing one or more innovations described herein.

[0110] The input device(s) 1150 may be a touch input device such as a keyboard, a mouse, a pen or trackball, a voice input device, a scanning device, or another device that provides input to the computing system 1100. The output device(s) 1160 may be a display, a printer, a speaker, a CD burner, or another device that provides output from the computing system 1100.

[0111] Communication connection(s) 1170 enable communication with another computing entity over a communication medium. The communication medium transmits information such as computer executable instructions, audio or video input or output, or other data in a modulated data signal. A modulated data signal is a signal that has one or more characteristics set or changed in such a way as to encode the information in the signal. By way of example and not limitation, the communication medium may use an electrical, optical, RF or other carrier wave.

[0112] Innovations may be described in the general context of computer executable instructions such as those included in program modules and executed in a computing system on a target real or virtual processor. Typically, program modules or components include routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The functionality of program modules may be combined or split between program modules as desired in various embodiments. Computer executable instructions for program modules may be executed within a local or distributed computing system.

[0113] The terms "system" and "device" are used interchangeably herein. Unless the context clearly indicates otherwise, neither term implies any limitation on the type of computing system or computing device. In general, a computing system or computing device may be local or distributed, and may include any combination of special-purpose hardware and / or general-purpose hardware and software that implements the functionality described herein.

[0114] In various examples described herein, a module (e.g., component or engine) may be "encoded" to perform certain operations or provide certain functionality, indicating that computer executable instructions for the module may be executed to perform such operations, so that such operations are performed, or otherwise provide such functionality. Although functionality described with respect to a software component, module, or engine may be performed as a discrete software unit (e.g., a procedure, function, class method), it need not be implemented as a discrete unit. That is, functionality may be incorporated into a larger or more general program, such as one or more lines of code in a larger or general program.

[0115] For purposes of presentation, the detailed description uses terms such as "determine" and "use" to describe computer operations in a computing system. These terms are high-level abstractions for operations performed by a computer and should not be confused with actions performed by a human. The actual computer operations corresponding to these terms vary depending on the implementation. Example 17 - Cloud computing environment

[0116] Fig.12 An example cloud computing environment 1200 is depicted in which the described techniques may be implemented. Cloud computing environment 1200 includes cloud computing services 1210. Cloud computing services 1210 may include various types of cloud computing resources, such as computer servers, data repositories, networking resources, etc. Cloud computing services 1210 may be centrally located (e.g., provided by a data center of an enterprise or organization) or distributed (e.g., provided by various computing resources located in different locations, such as different data centers and / or located in different cities or countries).

[0117] The cloud computing service 1210 is utilized by various types of computing devices (e.g., client computing devices), such as computing devices 1220, 1222, and 1224. For example, the computing devices (e.g., 1220, 1222, and 1224) may be computers (e.g., desktop or laptop computers), mobile devices (e.g., tablet computers or smart phones), or other types of computing devices. For example, the computing devices (e.g., 1220, 1222, and 1224) may utilize the cloud computing service 1210 to perform computing operations (e.g., data processing, data storage, etc.). Example 18 - Implementation

[0118] Although the operations of some of the disclosed methods are described in a specific, sequential order for ease of presentation, it should be understood that this description includes rearrangement unless a specific ordering is required by the specific language set forth herein. For example, in some cases, the operations described in sequence may be rearranged or performed simultaneously. In addition, for simplicity, the accompanying drawings may not illustrate the various ways in which the disclosed methods can be used in conjunction with other methods.

[0119] Any of the disclosed methods may be implemented as computer-executable instructions or computer program products stored on one or more computer-readable storage media and executed on a computing device (e.g., any available computing device, including a smartphone or other mobile device that includes computing hardware). Tangible computer-readable storage media are any available tangible media that can be accessed within a computing environment (e.g., one or more optical media disks such as DVDs or CDs, volatile memory components such as DRAM or SRAM, or non-volatile memory components such as flash memory or a hard drive). As an example and with reference to Fig.11 , computer-readable storage media include memories 1120 and 1125 and storage device 1140. The term computer-readable storage medium does not include signals and carrier waves. In addition, the term computer-readable storage medium does not include communication connections (eg, 1170).

[0120] Any computer executable instructions for implementing the disclosed techniques and any data created and used during the implementation of the disclosed embodiments may be stored on one or more computer readable storage media. Computer executable instructions may be, for example, a dedicated software application or a portion of a software application that is accessed or downloaded via a web browser or other software application (such as a remote computing application). Such software may be executed, for example, on a single local computer (e.g., any suitable commercially available computer) or in a network environment (e.g., via the Internet, a wide area network, a local area network, a client-server network (such as a cloud computing network or other such network) using one or more network computers.

[0121] For clarity, only certain selected aspects of software-based implementations are described. It should be understood that the disclosed techniques are not limited to any particular computer language or program. For example, the disclosed techniques can be implemented by software written in C++, Java, Perl, JavaScript, Python, Ruby, ABAP, SQL, Adobe Flash, or any other suitable programming language, or in some examples, can be implemented in a markup language such as html or XML, or a combination of suitable programming languages ​​and markup languages. Likewise, the disclosed techniques are not limited to any particular computer or type of hardware.

[0122] In addition, any software-based embodiment (including, for example, computer-executable instructions for causing a computer to perform any disclosed method) can be uploaded, downloaded, or remotely accessed via suitable communication means. Such suitable communication means include, for example, the Internet, the World Wide Web, an intranet, a software application, cables (including fiber optic cables), magnetic communications, electromagnetic communications (including RF, microwave, and infrared communications), electronic communications, or other such communication means.

[0123] The disclosed methods, apparatuses, and systems should not be construed as limiting in any way. Rather, the present disclosure relates to all novel and non-obvious features and aspects of the various disclosed embodiments, both individually and in various combinations and sub-combinations with each other. The disclosed methods, apparatuses, and systems are not limited to any particular aspect or feature or combination thereof, nor do the disclosed embodiments require the presence of any one or more specific advantages or problems solved.

[0124] The technology from any example can be combined with the technology described in any one or more of the other examples. In view of the many possible embodiments to which the principles of the disclosed technology can be applied, it should be recognized that the illustrated embodiments are examples of the disclosed technology and should not be considered as limiting the scope of the disclosed technology. On the contrary, the scope of the disclosed technology includes what is covered by the scope and spirit of the appended claims.

Claims

1. A computing system comprising: at least one hardware processor; at least one memory coupled to the at least one hardware processor; as well as One or more computer-readable storage media including computer-executable instructions that, when executed, cause the computing system to perform operations comprising: extracting one or more latent features from frames of an input signal using an encoder to provide extracted one or more latent features; determining a prediction of the one or more latent features using the reconstructed latent features for a plurality of previous frames; extracting class residual features from the extracted one or more latent features and the prediction; and The class residual feature or data sufficient to reconstruct the class residual feature is sent to the client.

2. The computing system of claim 1, wherein the input signal comprises audio data.

3. The computing system of claim 1, wherein the extracting comprises use of at least one convolutional layer. The computing system of claim 1 , wherein the input signal comprises time-frequency spectral data.

5. The computing system of claim 4, wherein the time-frequency spectral data is obtained using a short-time Fourier transform of a time window of the input signal.

6. The computing system of claim 4, further comprising applying amplitude compression to the time-frequency spectral data.

7. The computing system of claim 6, wherein the amplitude compression is applied using values ​​determined during training of the encoder.

8. The computing system of claim 7, wherein the value is different for different encoding bit rates.

9. The computing system of claim 1, wherein the encoder comprises a plurality of convolutional layers.

10. The computing system of claim 1, wherein said determining a prediction comprises: The reconstructed latent features for the plurality of previous frames are processed using a plurality of convolutional layers.

11. The computing system of claim 1 , the operations further comprising: The class residual features are split into multiple groups along the channel dimension, and the groups in the multiple groups are quantized respectively.

12. The computing system of claim 11, wherein a given group of the plurality of groups comprises a plurality of frequencies.

13. The computing system of claim 12, wherein the channels are quantized using different codebooks, the operations further comprising during training of the encoder: For an input training data set used during training of the encoder, randomly selecting a group from the plurality of groups, wherein groups are associated with sets of progressively higher bit rates; and During training of the encoder using the input training data set, only the selected group of the plurality of groups and groups of the plurality of groups associated with a lower bit rate than the selected group are used.

14. The computing system of claim 1, the operations further comprising: Quantifying the residual-like feature, the quantization comprising: For the frame, determining a distance between the residual-like feature and a codeword of a codebook, the codebook being used for vector quantization of the residual-like feature; and The distance is used, at least in part, to determine a probability of selecting the codeword.

15. The computing system of claim 14, wherein the determined probability is determined as a non-linear projection.

16. The computing system of claim 14, wherein said determining a probability comprises selecting an element of a Gumbel distribution.

17. The computing system of claim 1, wherein the residual-like features or data sufficient to reconstruct the residual-like features are sent as part of a bitstream having a rate, the operations further comprising: During training of the encoder, a bit rate for training input data is determined, the determining the bit rate comprising determining a difference between a target bit rate and an entropy of a probability of selecting a particular codeword in a codebook for a frame of the training input data.

18. The computing system of claim 17, the operations further comprising: A rate-distortion factor is optimized, the rate-distortion factor being determined as a tradeoff between the determined distortion and the bit rate for the training input data.

19. A method implemented in a computing system, the computing system comprising at least one hardware processor and at least one memory coupled to the at least one hardware processor, the method comprising: extracting one or more latent features from frames of an input signal using an encoder to provide extracted one or more latent features; determining a prediction of the one or more latent features using the reconstructed latent features for a plurality of previous frames; extracting class residual features from the extracted one or more latent features and the prediction; and The class residual feature or data sufficient to reconstruct the class residual feature is sent to the client.

20. One or more computer-readable storage media comprising: computer executable instructions that, when executed by a computing system comprising at least one hardware processor and at least one memory coupled to the at least one hardware processor, cause the computing system to extract one or more latent features from frames of an input signal using an encoder to provide extracted one or more latent features; computer executable instructions that, when executed by the computing system, cause the computing system to determine a prediction of the one or more latent features using reconstructed latent features for a plurality of previous frames; computer executable instructions that, when executed by the computing system, cause the computing system to extract a class residual feature from the extracted one or more latent features and the prediction; as well as The computer executable instructions, when executed by the computing system, cause the computing system to send the class residual feature or data sufficient to reconstruct the class residual feature to a client.

Images