High Frequency Reconstruction Using Neural Network Systems

Abstract

A method for reconstructing an audio signal, comprising: receiving a bitstream including an encoded lowband audio signal representation and a set of high-frequency reconstruction (HFR) parameters; decoding the lowband audio signal representation to provide a lowband audio signal in a filterbank domain; reconstructing a highband audio signal in the filterbank domain using a neural network system trained to predict samples of the highband audio signal in the filterbank domain given samples of the lowband signal in the filterbank domain and the HFR parameters; and synthesizing a time-domain output audio signal from the lowband signal in the filterbank domain and the reconstructed highband signal in the filterbank domain. By reconstructing the high-frequency range using a generative model in the form of a neural network system, a perceptually improved audio output can be achieved.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 63 / 331,056, filed April 14, 2022 (Docket No. D21075USP1), and European Application No. EP22168469.9, filed April 14, 2022 (Docket No. D21075EP), each of which is incorporated by reference in its entirety herein.

[0002] TECHNICAL FIELD OF THEINVENTION The present invention relates to high-frequency reconstruction using generative deep neural networks operating in the filterbank domain. [Background technology]

[0003] For very low bitrate audio coding systems, existing encoders for audio transmission are not able to encode the full bandwidth signal and are therefore forced to encode only the lower frequency range. For example, for 32 kbps stereo encoded by mp3 (ISO / MPEG-II Layer 3), the codec bandwidth can be as low as 4-6 kHz. While this may be sufficient for some use cases, it is generally desirable to also convey higher frequencies in the audio output.

[0004] According to one approach, called "blind bandwidth extension," higher frequency bands are generated based only on information in lower frequency bands. Such processing schemes can successfully provide bandwidth extension for certain isolated signal classes where the signal statistics are well determined, such as speech and piano music, but fail for more complex signal types (e.g., general audio containing mixed music and speech or other signal classes).

[0005] In more sophisticated techniques, called "high frequency reconstruction" (HFR), side information describing the characteristics of the higher frequency (HFR) band, such as the spectral envelope, tone-to-noise ratio, or other characteristics of the high band, is used to reconstruct the HFR band. Such high frequency reconstruction guided by side information is known to work well for most signal classes. Examples include A-SPX in AC-4 (developed by Dolby Laboratories) and SBR in HE-AAC (ISO / MPEG standard).

[0006] In such HFR systems, the coded bitstream contains a waveform-coded low-frequency band and HFR side information that parameterizes the HFR band. Only a fraction of the available bitrate is allocated to the HFR side information. The frequency at which the HFR range begins is called the "crossover" frequency. On the decoder side, the low band is decoded by the decoder and the HFR side information is used by the HFR module to correctly reconstruct the HFR band.

[0007] In the HFR module, a unit called a "transposer" first generates an initial highband approximation. This approximation is then modified in various ways to resemble the original highband, in a process guided by side information in the bitstream. One method of transposition is the "copy up" method (used, for example, in AC-4 and HE-AAC), where frequency chunks (contiguous collections of subband samples) from the decoded lowband are copied to the HFR frequency range. This is a robust method with very low computational complexity, but often suffers from single sideband (SSB) distortion when the crossover frequency is low. This is usually the case when encoding for low bitrates, because the available bits only allow a limited frequency range to be encoded by the waveform core encoder. Another transposition method is the harmonic transposer, used, for example, in the MPEG USAC standard, where a phase vocoder is used to generate second, third, and even fourth harmonics from the lowband. This type of transition avoids SSB distortion, but the high frequencies produced are sometimes perceived as metallic and synthetic. Summary of the Invention [Problem to be solved by the invention]

[0008] One objective of the present invention is to provide an improved approach to high-frequency reconstruction by using a filterbank-based neural network to generate the high-frequency band given the decoded low-band and side information. [Means for solving the problem]

[0009] According to a first aspect of the present invention, this object is achieved by a method for reconstructing an audio signal, the method comprising the steps of: receiving a bitstream comprising an encoded lowband audio signal representation and a set of high-frequency reconstruction (HFR) parameters; decoding the lowband audio signal representation to provide a lowband audio signal in a filterbank domain; reconstructing the highband audio signal in the filterbank domain using a neural network system trained to predict samples of the highband audio signal in the filterbank domain given samples of the lowband signal in the filterbank domain and the HFR parameters; and synthesizing a time-domain output audio signal from the lowband signal in the filterbank domain and the reconstructed highband signal in the filterbank domain.

[0010] By reconstructing the high frequency range using a generative model in the form of a neural network system, a perceptually improved audio output can be achieved.

[0011] By a representation in the "filterbank domain" we intend a time-frequency representation that includes (implicit or explicit) phase information (or, in other words, facilitates synthesis of signals with correct phase). It can contain real or complex coefficients. Examples of well-known filterbank representations are the MDCT (Modified Discrete Cosine Transform), QMF (Quadrature Mirror Filter), and STFT (Short-Term Fourier Transform). Note that spectrograms or Mel-spectra are not considered filterbank representations in the context of this application (since these representations are based only on the magnitude spectrum and thus discard information about the phase).

[0012] In some embodiments, the neural network system is trained to predict filter bank domain samples with reduced signal dynamics. In this case, the method further comprises restoring the signal dynamics of the reconstructed filter bank domain high band signal. Experience has shown that training the neural network system with signals with reduced signal dynamics can result in faster convergence and better performance of the trained model.

[0013] In this disclosure, the processes of reducing and restoring signal dynamics are referred to as "flattening" and "inverse flattening", although terms such as "compression / expansion" or "whitening / de-whitening" may also be used.

[0014] In some embodiments, the method further includes reconstructing the improved filter bank domain low-band audio signal using a neural network system trained to predict samples of the filter bank domain low-band signal given the decoded samples of the filter bank domain low-band signal, wherein said synthesizing is based on the reconstructed filter bank domain low-band signal and the reconstructed filter bank domain high-band signal.

[0015] In this implementation, the neural network system (or potentially two different neural network systems) provides two advantages. 1) Use the decoded samples as conditioning to "clean up" (improve) the low-band range. 2) Given the decoded lower band and the side information, reconstruct the higher band as conditioned (as above).

[0016] For example, the decoded low-band samples may be coarsely quantized (due to limited bitrate for the low-band samples), in which case the neural network system will work to reconstruct (predict) the improved low-band samples in addition to reconstructing the missing high-band samples.

[0017] In yet another application, the lowband audio signal representation includes quantized filterbank domain coefficients and associated control data, and the method further includes decoding the lowband audio signal representation using a neural network system trained to predict filterbank domain samples given the quantized filterbank domain coefficients.

[0018] According to a second aspect of the present invention, the above object is achieved by a decoder system comprising a demultiplexer for separating a bitstream into an encoded lowband audio signal representation and a set of high frequency reconstruction (HFR) parameters, a decoder for decoding the lowband audio signal representation to provide a lowband audio signal in a filterbank domain, a generative model for reconstructing the highband signal in the filterbank domain using a neural network system trained to predict samples of the highband audio signal in the filterbank domain given samples of the lowband signal in the filterbank domain and said HFR parameters, and a synthesis filterbank for synthesizing a time domain audio signal from the lowband signal in the filterbank domain and the reconstructed highband signal in the filterbank domain.

[0019] Yet another aspect of the invention relates to a neural network system for autoregressively generating a current sample for a current time slot of a filterbank representation of an audio signal, the current sample including a plurality of values ​​each corresponding to a channel of the filterbank, the system comprising first and second sub-models, each sub-model including: 1) a processing layer trained to generate conditioning information for the current sample; and 2) an output layer subdivided into a plurality of sequentially executed sub-layers, each sub-layer trained to generate a subset of values ​​of the current sample given the conditioning information from the processing layer and samples generated by any previously executed sub-layers. The first sub-model is trained to generate values ​​of the current sample corresponding to a low-band frequency range given previously generated samples of the filterbank representation and conditioned by quantized samples of the filterbank representation, and the second sub-model is trained to generate values ​​of the current sample corresponding to a high-band frequency range given previously generated samples of the filterbank representation and conditioned by quantized samples of the filterbank representation and by a set of high-frequency reconstruction parameters.

[0020] It should be noted that this aspect of the invention may form patentable subject matter independently of the first and second aspects of the invention.

[0021] In such a decoder system with a neural network system, the first sub-model can, with appropriate training, partially replace the decoder. Such a decoder system is compatible with existing legacy encoders, e.g. as defined by the AC-4 (or HE-AAC) syntax, but provides superior reconstruction compared to the legacy decoders.

[0022] The sub-models may operate in different filterbank domains, for example a first sub-model may operate in the MDCT domain, which is particularly advantageous when the bitstream contains encoded MDCT samples. [Brief description of the drawings]

[0023] The present invention will now be described in more detail with reference to the accompanying drawings showing presently preferred embodiments of the invention.

[0024] [Figure 1] FIG. 2 is a process diagram of a decoder system according to a first embodiment of the present invention.

[0025] [Diagram 2] 1 is a schematic topology of the generative model of FIG. 1 according to one implementation.

[0026] [Diagram 3] The training of the model is shown in Figure 2.

[0027] [Figure 4] FIG. 11 is a process diagram of a decoder system according to a second embodiment of the present invention.

[0028] [Diagram 5] 5 is a schematic topology of the generative model of FIG. 4 according to one implementation.

[0029] [Figure 6] FIG. 11 is a process diagram of a decoder system according to a third embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030] The systems and methods disclosed in this application may be implemented as software, firmware, hardware, or a combination thereof. In a hardware implementation, the division of tasks does not necessarily correspond to a division into physical units; conversely, one physical component may have multiple functions and one task may be performed by several cooperating physical components.

[0031] The computer hardware may be, for example, a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a smart phone, a web appliance, a network router, a switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by the computer hardware. Furthermore, the present disclosure relates to any collection of computer hardware that individually or collectively executes instructions to perform any one or more of the concepts described herein.

[0032] Some or all of the components may be implemented by one or more processors that accept computer-readable (also called machine-readable) code that includes a set of instructions that, when executed by the one or more processors, perform at least one of the methods described herein. Any processor capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken is included. Thus, an example is a typical processing system (i.e., computer hardware) that includes one or more processors. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system may further include a memory subsystem that includes a hard drive, SSD, RAM, and / or ROM. A bus subsystem may be included for communication between the components. Software may reside in the memory subsystem and / or in the processor during its execution by the computer system.

[0033] The processor or processors may operate as stand-alone devices or may be connected to other processors, e.g., networked. Such networks may be built on a variety of different network protocols and may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0034] Software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to those skilled in the art, the term computer storage media includes both volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, various forms of physical (non-transitory) storage media, such as EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by a computer. Additionally, communication media (transitory) typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media, as is well known to those skilled in the art.

[0035] In the following description, the neural network system is said to generate the current sample of the audio signal representation. In practice, the neural network system typically generates a probability distribution. The reconstruction of the samples is performed by sampling according to the probability distribution.

[0036] FIG. 1 illustrates a block diagram of a method for receiving a bitstream, e.g., an AC-4 bitstream, and converting it into a low-band audio signal representation and a high-frequency reconstruction parameter a k The decoder system 10 includes a bitstream demultiplexer 11 configured to separate the lowband audio signal representation into a sequence of lowband filterbank domain samples, each sample x k LFcontains a number of values ​​corresponding to filterbank channels in the lowband frequency range. The decoder system k LF and the high-frequency reconstruction parameter a k Given a highband sample x k HF The synthesis filterbank 14 is configured to generate a combined audio signal in the time domain based on the lowband filterbank samples and the predicted highband samples.

[0037] In one example, the synthesis filter bank 14 is a quadrature mirror filter (QMF) bank, and the neural network system 13 is trained to output high-band QMF samples (or, strictly speaking, a probability distribution of such samples). The decoder 12 is configured to decode the bitstream into low-band QMF samples. In many applications, the bitstream contains encoded MDCT samples, in which case the decoder 12 first decodes the MDCT samples, inverse transforms these samples to the time domain, and finally applies a QMF analysis filter bank to obtain the QMF samples.

[0038] In use, the decoder system 10 receives a bitstream which is demultiplexed by a demultiplexer 11 into a low-band audio signal representation and high-frequency reconstruction parameters a k The low-band audio signal representation is decoded by the decoder 12. k LF , where the QMF samples are provided to a generative model 13, which calculates the high-frequency reconstruction parameters a k is used as additional conditioning to obtain the high-band sample x k HFThe resulting filterbank samples (decoded lowband and generated highband) are then synthesized by the synthesis filterbank 14 into a time-domain (e.g. PCM) signal.

[0039] An example of a neural network-based generative model operating in the filter bank (QMF) domain is disclosed in PCT / EP2021 / 078652, which is incorporated herein by reference. The model processes QMF samples end-to-end, i.e., it receives QMF subband samples as ground truth input during training and generates QMF samples during inference (generation). The neural network system in PCT / EP2021 / 078652 is a modification of a previously disclosed time domain generative model and serves to facilitate the prediction of filter bank domain samples. The model uses upper layers with a coarser time resolution that condition a lower layer with a finer time resolution. This lower layer (layer 0) is composed of several layers, each predicting a subset of the filter bank channels. Subsequent layers are conditioned by the previous layers by a recurrent neural network (RNN).

[0040] An example of a neural network system 13 following this approach is shown in more detail in Figure 2. Here, the model 13 includes a (single) upper layer 21 and a lower layer 22. In the upper layer 21, the previously generated filterbank samples are received together with the decoded lowband samples and the high frequency reconstruction parameters.

[0041] The upper layer 21 is a set of previously generated samples {x <m The upper layer 21 includes a convolutional network 23 that takes as input the decoded low-band samples x for the current time slot m (and possibly future time slots > m). The convolutional network 23 may, for example, include 32 channels and use a kernel size of 15. The upper layer 21 receives the decoded low-band samples x for the current time slot m (and possibly future time slots > m).m LF and the high-frequency reconstruction parameter a m The convolutional network 24 may, for example, include 22 channels. The exact kernel size and stride of the convolutional networks 23, 24 may, for example, be determined by the previous sample {x <m} Provided frame size, high frequency reconstruction parameters a m Note that the time resolution (singular or plural) of the input signal may be adjusted based on several factors, including the time resolution (singular or plural).

[0042] The outputs of the convolutional networks 23, 24 are summed at a summing junction 25 and input to a recurrent neural network (RNN) 26. The RNN 26 may be implemented, for example, using one or more stateful network units, such as gated recurrent units (GRUs), long short-term memory units (LSTMs), pseudo-recurrent neural networks, Elman networks, etc. The main property of such an RNN is that it is stateful, i.e., it updates its hidden latent state through time steps.

[0043] The output from the RNN 26 is provided as input to an upsampling stage 27, which may be implemented using, for example, a transposed convolutional network. It is envisaged that the network itself can learn how specifically to perform such upsampling, i.e. the upsampling provided by stage 27 may be "learned upsampling".

[0044] The lower layer 22 is divided into a number of sequentially executed sublayers 28-j, where j=1,...,L, where L is the total number of sublayers, which in this case is 4. Each sublayer is configured to generate a set of reconstructed channels in the high frequency band.

[0045] Each sublayer 28-j is a set of previously generated samples {x <m} as input. The convolutional networks 29 may have different numbers of channels. All convolutional networks may have the same kernel size, for example 4. Each sublayer 28 further includes another convolutional network 30 that takes as its input the decoded low-band samples x m LF and the high-frequency reconstruction parameter a m The convolutional network 30 may correspond to the convolutional network 24 in the upper layer 21.

[0046] The outputs of the convolutional networks 29, 30 and the output from the upper layer 21 are all summed at a summing point 31 and then split to form two different tensors of equal size. One of the tensors is input to a recurrent neural network (RNN) 32, e.g., another GRU. This RNN 32 may be of a similar structure to the RNN 26. The output of the RNN 32 and the second tensor from the split are summed at a second summing point 33 and fed to a further RNN 34, e.g., another GRU, common to all layers. While the RNN 32 operates in the "time direction", the RNN 34 operates in the "layer direction" or "frequency band direction". Thus, the RNN 34 can predict samples for higher filter bank channels from lower filter bank channels. The RNN 34 is provided with conditioning in the form of a parameter h0, which becomes the initial hidden state of the RNN 34. The parameter h0 may be a learned parameter, i.e., trained by a neural network, or may be set to a constant value, e.g., zero.

[0047] The output from RNN 34 is provided as conditioning to a neural network 35, e.g., a multi-layer perceptron MLP, which ultimately outputs the set of highband channels for the current QMF sample. As mentioned above, in a practical implementation, the output from 35 is the parameters, e.g., mean and variance, for a selected distribution, e.g., Gaussian, Logistic, or Laplacian. During generation (inference), filter bank samples are sampled from the distribution given these parameters.

[0048] Processing occurs first in sublayer 28-1, followed by the next sublayer, and so on until the last sublayer, in this case 28-4. To allow each sublayer to compute one or a small number of the total number of channels, the kernels of each convolutional network 29 are "masked" accordingly. As an example, with L=4 sublayers, 10 frequency channels in the decoded low band, and 22 frequency channels in the reconstructed high band, the first two layers 28-1, 28-2 may generate four high band channels each, the third layer 28-3 may generate six high band channels, and the fourth layer 28-4 may generate eight high band channels. In a practical implementation, the number of layers is usually greater than this.

[0049] The training of the neural network system (generative model) 13 is illustrated in Fig. 3. Ground truth data in the form of filter bank domain audio data 41, e.g., QMF samples including high band samples 42 and low band samples 43, are used to generate the high frequency reconstruction parameters a kThe low-band samples 43 are also fed to an encoder / decoder 45, 46 to provide a decoded version 47 of the low-band samples. The neural network system 13 uses the high-band samples 42 as a ground truth and the high-frequency reconstruction parameters a as conditioning. k and trained using the decoded low band 47.

[0050] In Figure 3, the ground truth highband samples 42 are flattened 48 using the transmitted spectral envelope in the high frequency reconstruction parameters. Flattening applies an inverse gain calculated from the envelope energy to reduce or whiten the dynamics of the highband QMF samples. In one embodiment, the highband samples 42 are not "perfectly" flattened, but instead use a compression factor of about 0.9 instead of 1.0 (where 1.0 corresponds to perfect flattening towards unit variance for each band).

[0051] Returning to Figure 1, the decoder system 10 in this case further includes an inverse flattening unit 15 for inverse flattening the predicted highband samples. As explained below, the model 13 is trained to predict samples in the flattened QMF domain. The inverse flattening unit 15 is configured to apply inverse flattening using the spectral envelope information in the side information.

[0052] Instead of a "straightforward" inverse flattening before QMF synthesis, i.e. applying exactly the inverse of the gain applied during flattening of the ground truth input, a further envelope adjustment can also be applied to the output QMF samples flattened by the inverse flattening unit 15, where the adjustment aims to ensure that the envelope of the reconstructed signal exactly follows the envelope information transmitted in the bitstream. The advantage of this approach is that since the envelope data is based on absolute values, the envelope adjuster acts as a safeguard to ensure that the spectral envelope is always correct (within the accuracy of the spectral envelope values ​​estimated by the high-frequency reconstruction parameter generator 44).

[0053] Finally, the decoder system 10 extracts the low-band samples x k LF It may include a decompression unit 16 for removing any compression applied during encoding of the image. *

[0054] 4 includes, similar to the decoder system 10, a demultiplexer 111, a decoder 112, a generative model 113 in the form of a neural network system and a synthesis filter bank 114, e.g. a QMF filter bank. Also in this case, the decoder system 100 receives a bitstream which is divided by the demultiplexer 111 into a low-band audio signal representation and high-frequency reconstruction parameters a k The lowband audio signal representation is decoded by the decoder 112.

[0055] However, in this implementation, the neural network system 113 is given the decoded low-band samples and computes the high-band samples x k HF Not just low-band samples^x k LFThe synthesis filterbank 114 then generates the combined audio signal in the time domain based only on the generated (predicted) filterbank samples (lowband and highband).

[0056] The neural network system 113 is further described with reference to Fig. 5. In this case, the system 113 is separated into two sub-models 113A and 113B. The first sub-model is trained to predict low-band samples (among multiple channels) given previously generated samples and decoded low-band samples. The second sub-model is trained to predict high-band samples (among multiple channels) given previously generated samples, decoded low-band samples, and high-frequency reconstruction parameters.

[0057] Each sub-model 113A, 113B includes an upper layer 121A, 121B and a lower layer 122A, 122B. The lower layer 122A of the first sub-model 113A includes a first group of layers predicting the low band range, and the lower layer 122B of the second sub-model 113B includes a second group of layers predicting the high band range as described above. It is important to note that the RNN 123A in the lower layer 122A and the RNN 123B in the lower layer 122B may not be the same in the two sub-models. They may each have a separate set of coefficients (weights) determined during training for their specific purpose.

[0058] During training of this model (neural network system 113), the ground truth low-band samples may also be flattened, in which case the spectral envelope of the low-band range is calculated, giving the time resolution of the spectral envelope for the high-band. Both spectral envelopes are concatenated and used in conditioning.

[0059] In this case, the decoded filterbank samples need to be flattened before being provided to the neural network system 113, using the same type of spectral envelope used to flatten the ground truth samples during training. To this end, the decoder system 100 may include a flattening unit 115 arranged before the neural network 113. Furthermore, both the low-band and high-band ranges predicted by the model 113 are flattened. Thus, the decoder 100 may also include an inverse flattening unit 116 arranged between the neural network system 113 and the synthesis filterbank 114. The inverse flattening unit 116 is configured to inverse flatten, and optionally envelope adjust, both the low-band and high-band samples generated by the neural network system 113.

[0060] Furthermore, similar to the decoder system 10, any compression applied during the encoding of the low-band samples needs to be removed. For this purpose, the flattening unit 115 may be preceded by an expansion unit 117 similar to unit 16 in FIG.

[0061] In the implementation of Fig. 4, the model (neural network system) 113 is also able to reconstruct missing samples. In other words, given proper training, the decoder system 100 can improve the quality of the reconstruction of the coarsely quantized filterbank samples given the appropriate side information (decoded low-band samples from the decoder and high-frequency reconstruction parameters from the bitstream).

[0062] FIG. 6 shows a decoder system 200 that has similar capabilities to the decoder system 100, but operates in two different filterbank domains, in the illustrated case the MDCT domain and the QMF domain.

[0063] Again, the bitstream contains information about the quantized MDCT coefficients for the low band and side information containing high frequency reconstruction parameters. The bitstream is received by a demultiplexer 211 and the low band information is decoded by a decoder 212 to recreate the actual quantized MDCT coefficients. Compared to the example of Fig. 1, where the decoder 12 transformed the samples into the QMF domain, the decoder 212 only needs to perform decoding of the MDCT waveform. Therefore, the decoder 212 may be referred to as a waveform decoder.

[0064] The MDCT coefficients are fed to a generative model formed by a first neural network system 213, which is trained to reconstruct a set of MDCT coefficients for the low band having higher resolution. The generative model 213 may be an MDCT predictor as described in PCT / US2021 / 054617, which is incorporated herein by reference. The generated (predicted) MDCT coefficients are then fed to an inverse MDCT transform 214, followed by a QMF analysis filter bank 215, which reconstructs the low band QMF samples x k LF Generate.

[0065] The remaining blocks in Figure 6 are a second neural network system 216 and a QMF synthesis filter bank 217. The second neural network system 216 may be equivalent to the generative model 13 of Figure 1 and may be trained in a similar manner.

[0066] Similar to the decoder system 10 of FIG. 1, an inverse flattening unit 218 may be provided before the synthesis filter bank 217, where the second neural network system 216 is trained to predict flattened samples.

[0067] Unless otherwise indicated, and as will be apparent from the following description, throughout this disclosure, discussions utilizing terms such as "processing," "calculating," "computing," "determining," "analyzing," and the like, will be understood to refer to operations and / or processes of computer hardware or computing systems, or similar electronic computing devices, that manipulate and / or transform data represented as physical quantities, such as electronic quantities, into other data also represented as physical quantities.

[0068] In the above description of exemplary embodiments of the present invention, it should be understood that various features of the invention may be grouped together in a single embodiment, figure, or description thereof for the purpose of facilitating the flow of the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, this method of disclosure should not be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in fewer than all features of a single above-disclosed embodiment. Thus, the claims following the detailed description are expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of the present invention. Furthermore, although some embodiments described herein include some features but not other features included in other embodiments, combinations of features of different embodiments are intended to be within the scope of the present invention and form different embodiments. This will be understood by those skilled in the art. For example, in the following claims, any of the claimed embodiments may be used in any combination.

[0069] Furthermore, some of the embodiments are described herein as a method or a combination of elements of a method that can be implemented by a processor of a computer system or by other means for performing the function. Thus, a processor having the necessary instructions for performing such a method or elements of a method forms a means for performing the method or elements of a method. It should be noted that when a method includes several elements, e.g., several steps, no order of such elements is implied unless specifically stated. Furthermore, the elements described herein of an apparatus embodiment are examples of means for performing the functions performed by the elements for implementing the invention. In the description provided herein, numerous specific details are described. However, it is understood that embodiments of the invention can be implemented without these specific details. On the other hand, well-known methods, structures, and techniques have not been shown in detail in order not to obscure an understanding of this description.

[0070] Those skilled in the art will appreciate that the present invention is in no way limited to the specific embodiments described above. On the contrary, many modifications and variations are possible within the scope of the appended claims. For example, as mentioned above, other filter bank domains may be used.

[0071] Various aspects of the present invention can be understood from the following enumerated example embodiments (EEE). [EEE1] 1. A method for reconstructing an audio signal, comprising: receiving a bitstream comprising an encoded lowband audio signal representation and a set of high frequency reconstruction (HFR) parameters; decoding the encoded lowband audio signal representation to provide a filterbank domain lowband audio signal; reconstructing the highband audio signal in the filterbank domain using a neural network system trained to predict samples of the highband audio signal in the filterbank domain given samples of the lowband signal in the filterbank domain and the HFR parameters; synthesizing a time domain output audio signal from the filter bank domain low band signal and the reconstructed filter bank domain high band signal. method. [EEE2] The method according to EEE1, wherein the neural network system is trained to predict highband samples of a filter bank region having reduced signal dynamics, and the method further comprises increasing the dynamics of the highband signal in the reconstructed filter bank region. [EEE3] The method of claim 8, further comprising envelope adjusting the high band signal in the reconstructed filter bank domain using envelope data in the HFR parameters. [EEE4] The method according to any one of EEE1 to EEE3, wherein the low band signal in the filter bank domain is compressed in the encoding process, the method further comprising expanding the low band signal in the filter bank domain prior to synthesis. [EEE5] reconstructing the improved filterbank domain lowband audio signal using a neural network system trained to predict samples of the lowband signal in the filterbank domain given the decoded samples of the lowband signal in the filterbank domain; the combining is based on a low-band signal of the reconstructed filter bank domain and a high-band signal of the reconstructed filter bank domain; The method according to any one of EEE1 to 4. [EEE6] reconstructing the low-band audio signal representation using a neural network system trained to predict low-band filterbank domain samples given the quantized filterbank domain coefficients. The method according to any one of EEE1 to 5. [EEE7] The method described in EEE6, wherein the neural network system used to reconstruct the lowband audio signal representation operates in a first filter bank domain and the neural network system used to reconstruct the highband audio signal in the filter bank domain operates in a second filter bank domain. [EEE8] The method according to EEE7, wherein the first filter bank domain is an MDCT domain and the second filter bank domain is a QMF domain. [EEE9] a demultiplexer for separating the bitstream into an encoded lowband audio signal representation and a set of high frequency reconstruction (HFR) parameters; a decoder for decoding the encoded lowband audio signal representation to provide a lowband audio signal in a filterbank domain; a generative model for reconstructing the highband signal in the filterbank domain using a neural network system trained to predict samples of the highband audio signal in the filterbank domain given samples of the lowband signal in the filterbank domain and the HFR parameters; a synthesis filterbank for synthesizing a time domain audio signal from the low-band signal in the filterbank domain and the high-band signal in the reconstructed filterbank domain; Decoder system. [EEE10] The decoder system as described in EEE9, wherein the neural network system is also trained to predict samples of the low band signal in the filter bank domain given decoded samples of the low band signal in the filter bank domain. [EEE11] The neural network system: a first sub-model trained to predict samples of the low-band signal in the filter bank domain given decoded samples of the low-band signal in the filter bank domain; a second sub-model trained to predict samples of a high-band signal in the filter bank domain given the predicted samples of the low-band signal in the filter bank domain and the HFR parameters; The method described in EEE10 includes two sub-models: [EEE12] The decoder system of claim 11, wherein the first sub-model operates in a first filter bank domain and the second sub-model operates in a second filter bank domain. [EEE13] 13. The decoder system of claim 12, wherein the first filter bank domain is an MDCT domain and the second filter bank domain is a QMF domain. [EEE14] m is the current sample (x m ), where a current sample includes a number of values, each value corresponding to a channel of the filter bank, and the system comprises: It has a first and a second submodel, each submodel: a processing layer trained to generate conditioning information for a current sample; an output layer subdivided into a plurality of sequentially executed sublayers, each sublayer being trained to generate a subset of values ​​for a current sample given the conditioning information from the processing layer and samples, if any, generated by previously executed sublayers; the first sub-model is trained to generate values ​​for a current sample corresponding to a low-band frequency range given previously generated samples of the filter bank representation and conditioned by quantized samples of the filter bank representation; the second sub-model is trained to generate values ​​for a current sample corresponding to a highband frequency range given previously generated samples of the filterbank representation and conditioned by the quantized samples of the filterbank representation and by a set of high-frequency reconstruction parameters. Neural network systems. [EEE15] A computer program product comprising computer program code portions configured to perform the method according to any one of claims 8 to 10, when the computer program product is executed on a computer processor.

Claims

1. 1. A method for reconstructing an audio signal, comprising: receiving a bitstream comprising an encoded lowband audio signal representation and a set of high frequency reconstruction (HFR) parameters; decoding the encoded lowband audio signal representation to provide a filterbank domain lowband audio signal; reconstructing the highband audio signal in the filterbank domain using a neural network system trained to predict samples of the highband audio signal in the filterbank domain given samples of the lowband signal in the filterbank domain and the HFR parameters, wherein the neural network system predicts a current sample (x) for a current time slot (m) of the highband signal in the filterbank domain. m ), wherein a current sample includes a plurality of values ​​each corresponding to a channel of the filter bank, and the system is configured to: a processing layer trained to generate conditioning information for a current sample based on the quantized samples of the low-band signal in the filter bank domain and the HFR parameters; and including an output layer subdivided into a plurality of sequentially executed sublayers, each sublayer being trained to generate a subset of values ​​for a current sample given the conditioning information from the processing layer and samples generated by previously executed sublayers, if any; synthesizing a time domain output audio signal from the filter bank domain low band signal and the reconstructed filter bank domain high band signal. method.

2. 2. The method of claim 1, wherein the neural network system is trained to predict highband samples of a filterbank region with reduced signal dynamics, the method further comprising increasing dynamics of the highband signal of the reconstructed filterbank region.

3. The method of claim 2 , further comprising envelope adjusting a highband signal in the reconstructed filterbank domain using envelope data in the HFR parameters.

4. 2. The method of claim 1, wherein the low-band signal in the filter bank domain is compressed in the encoding process, the method further comprising: expanding the low-band signal in the filter bank domain prior to synthesis.

5. reconstructing the improved filterbank domain low-band audio signal using a neural network system trained to predict samples of the low-band signal in the filterbank domain given the decoded samples of the low-band signal in the filterbank domain; the combining is based on a low-band signal of the reconstructed filter bank domain and a high-band signal of the reconstructed filter bank domain; The method of claim 1.

6. reconstructing the low-band audio signal representation using a neural network system trained to predict low-band filterbank domain samples given the quantized filterbank domain coefficients. The method of claim 1.

7. 7. The method of claim 6, wherein the neural network system used to reconstruct the lowband audio signal representation operates in a first filter bank domain and the neural network system used to reconstruct the highband audio signal in the filter bank domain operates in a second filter bank domain.

8. The method of claim 7 , wherein the first filter bank domain is an MDCT domain and the second filter bank domain is a QMF domain.

9. a demultiplexer for separating the bitstream into an encoded lowband audio signal representation and a set of high frequency reconstruction (HFR) parameters; a decoder for decoding the encoded lowband audio signal representation to provide a lowband audio signal in a filterbank domain; a generative model for reconstructing a highband signal in a filterbank domain using a neural network system trained to predict samples of a highband audio signal in the filterbank domain given samples of a lowband signal in the filterbank domain and the HFR parameters, the neural network system predicting a current sample (x m ), wherein a current sample includes a plurality of values ​​each corresponding to a channel of the filter bank, and the system is configured to: a processing layer trained to generate conditioning information for a current sample based on the quantized samples of the low-band signal in the filter bank domain and the HFR parameters; and a generative model including an output layer subdivided into a number of sequentially executed sublayers, each sublayer trained to generate a subset of values ​​for a current sample given the conditioning information from the processing layer and samples generated by previously executed sublayers, if any; a synthesis filterbank for synthesizing a time domain audio signal from the low-band signal in the filterbank domain and the high-band signal in the reconstructed filterbank domain; Decoder system.

10. 10. The decoder system of claim 9, wherein the neural network system is also trained to predict samples of a low-band signal in a filterbank domain given decoded samples of a low-band signal in the filterbank domain.

11. The neural network system: a first sub-model trained to predict samples of the low-band signal in the filter bank domain given decoded samples of the low-band signal in the filter bank domain; a second sub-model trained to predict samples of a high-band signal in the filter bank domain given the predicted samples of the low-band signal in the filter bank domain and the HFR parameters; 11. The decoder system of claim 10, comprising two sub-models:

12. 12. The decoder system of claim 11, wherein the first sub-model operates in a first filter bank domain and the second sub-model operates in a second filter bank domain.

13. 13. The decoder system of claim 12, wherein the first filter bank domain is an MDCT domain and the second filter bank domain is a QMF domain.

14. The current sample (x) for the current time slot (m) of the filterbank representation of the audio signal m ), where a current sample includes a number of values, each value corresponding to a channel of the filter bank, the system comprising: A first and a second sub-model, each sub-model comprising: a processing layer trained to generate conditioning information for a current sample; an output layer subdivided into a plurality of sequentially executed sublayers, each sublayer being trained to generate a subset of values ​​for a current sample given the conditioning information from the processing layer and samples generated by previously executed sublayers, if any; the first sub-model is trained to generate values ​​for a current sample corresponding to a low-band frequency range given previously generated samples of the filter bank representation and conditioned by quantized samples of the filter bank representation; the second sub-model is trained to generate values ​​of a current sample corresponding to a highband frequency range given previously generated samples of the filterbank representation and conditioned by the quantized samples of the filterbank representation and by a set of high-frequency reconstruction parameters. Neural network system.

15. A computer program product comprising computer program code portions configured to perform the method according to any one of claims 1 to 8 when said computer program product is executed on a computer processor.

Images