High-frequency reconstruction using neural network systems

Abstract

A method for reconstructing an audio signal, comprising: receiving a bitstream including an encoded lowband audio signal representation and a set of high-frequency reconstruction (HFR) parameters; decoding the lowband audio signal representation to provide a lowband audio signal in a filterbank domain; reconstructing a highband audio signal in the filterbank domain using a neural network system trained to predict samples of the highband audio signal in the filterbank domain given samples of the lowband signal in the filterbank domain and the HFR parameters; and synthesizing a time-domain output audio signal from the lowband signal in the filterbank domain and the reconstructed highband signal in the filterbank domain. By reconstructing the high-frequency range using a generative model in the form of a neural network system, a perceptually improved audio output can be achieved.

Description

[Technical Field]

[0001] Cross-references to related applications This application claims priority to U.S. Provisional Application No. 63 / 331,056 (reference number: D21075USP1), filed on 14 April 2022, and European Application No. EP22168469.9 (reference number: D21075EP), filed on 14 April 2022, each of which is incorporated herein by reference in whole.

[0002] Technical field of inventions This invention relates to high-frequency reconstruction using a generative deep neural network operating in the filter bank region. [Background technology]

[0003] For very low bitrate audio encoding systems, existing encoders for audio transmission cannot encode the entire bandwidth signal and are therefore forced to encode only the lower frequency range. For example, for 32kbps stereo encoded with mp3 (ISO / MPEG-II Layer 3), the codec bandwidth can be as low as 4-6kHz. While this may be sufficient for some use cases, it is generally desirable to transmit higher frequencies in the audio output as well.

[0004] One technique, called "blind bandwidth expansion," generates higher frequency bands based solely on information in lower frequency bands. While such processing methods can successfully provide bandwidth expansion for certain isolated signal classes where signal statistics are well determined, such as speech and piano music, they fail for more complex signal types (e.g., general audio involving mixed music and speech or other signal classes).

[0005] A more sophisticated technique called "high-frequency reconstruction" (HFR) uses side information that describes characteristics of higher frequency (HFR) bands, such as spectral envelopes, tone-to-noise ratios, or other high-band characteristics, to reconstruct the HFR band. Such side-information-guided high-frequency reconstructions are known to work well for most signal classes. Examples include A-SPX in AC-4 (developed by Dolby Laboratories) and SBR in HE-AAC (ISO / MPEG standard).

[0006] In such an HFR system, the encoded bitstream includes a waveform-encoded low-frequency band and HFR side information that parameterizes the HFR band. Only a portion of the available bitrate is allocated to the HFR side information. The frequency at which the HFR range begins is called the "crossover" frequency. On the decoder side, the low-frequency band is decoded by the decoder, and the HFR side information is used by the HFR module to correctly reconstruct the HFR band.

[0007] In HFR modules, a unit called a "transposer" first generates an initial high-bandwidth approximation. This approximation is then modified in various ways to resemble the original high-bandwidth in a process guided by side information in the bitstream. One method of transposition is the "copy-on" method (used, for example, in AC-4 and HE-AAC), where frequency chunks (a continuous set of subband samples) from the decoded low-bandwidth are copied to the HFR frequency range. This is a robust method with very low computational complexity, but it often suffers from single-sideband (SSB) distortion at low crossover frequencies. Typically, this is the case when encoding for low bitrates, because the available bits only allow a limited frequency range to be encoded by the waveform core encoder. Another transposition method is the harmonic converter, used, for example, in the MPEG USAC standard, where a phase vocoder is used to generate second, third, and even fourth harmonics from the low-bandwidth. This type of transition avoids SSB distortion, but the resulting high-bandwidth can sometimes be perceived as metallic and synthetic. [Overview of the project] [Problems that the invention aims to solve]

[0008] One objective of the present invention is to provide an improved method for high-frequency reconstruction by using a filter bank-based neural network to generate a high-frequency band given decoded low-band and side information. [Means for solving the problem]

[0009] According to a first aspect of the present invention, this objective is achieved by a method for reconstructing an audio signal, comprising the steps of: receiving a bitstream comprising an encoded low-bandwidth audio signal representation and a set of high-frequency reconstruction (HFR) parameters; decoding the low-bandwidth audio signal representation to provide a low-bandwidth audio signal in a filter bank region; reconstructing a high-bandwidth audio signal in a filter bank region using a neural network system trained to predict a sample of a high-bandwidth audio signal in a filter bank region, given a sample of the low-bandwidth signal in the filter bank region and HFR parameters; and synthesizing a time-domain output audio signal from the low-bandwidth signal in the filter bank region and the reconstructed high-bandwidth signal in the filter bank region.

[0010] Perceptually improved audio output can be achieved by reconstructing the high-frequency range using a generative model in the form of a neural network system.

[0011] The representation in the "filter bank region" is intended to be a time-frequency representation that includes (implicit or explicit) phase information (or, in other words, facilitates the synthesis of signals with correct phase). It may include real or complex coefficients. Well-known examples of filter bank representations are the MDCT (Modified Discrete Cosine Transform), QMF (Quaternary Mirror Filter), and STFT (Short-Term Fourier Transform). Note that spectrograms or Mel spectra are not considered filter bank representations in the context of this application (because these representations are based solely on amplitude spectra and therefore discard phase information).

[0012] In some embodiments, the neural network system is trained to predict filter bank region samples with reduced signal dynamics. In this case, the method further includes restoring the signal dynamics of the high-band signals of the reconstructed filter bank region. Experience has shown that training the neural network system with signals having reduced signal dynamics can result in faster convergence and better performance of the trained model.

[0013] In the present disclosure, the process of reducing and restoring signal dynamics is referred to as "flattening" and "inverse flattening", although expressions such as "compression / expansion" or "whitening / de-whitening" may also be used.

[0014] In some embodiments, the method further includes using a neural network system trained to predict samples of the low-band signal of the filter bank region, given the decoded samples of the low-band signal of the filter bank region, to reconstruct an improved low-band audio signal of the filter bank region, and said synthesizing is based on the reconstructed low-band signal of the filter bank region and the reconstructed high-band signal of the filter bank region.

[0015] In this implementation, the neural network system (or possibly two different neural network systems) provides two advantages. 1) Use the decoded samples as a condition to "clean up" (improve) the low-band range. 2) Given the decoded low-band and side information, reconstruct the high-band as a condition (as above).

[0016] For example, decoded low-bandwidth samples may be coarsely quantized (due to the limited bitrate for low-bandwidth samples), in which case the neural network system will work to reconstruct (predict) improved low-bandwidth samples in addition to reconstructing the missing high-bandwidth samples.

[0017] In further applications, the lowband audio signal representation includes quantized filter bank region coefficients and associated control data, and the method further includes decoding the lowband audio signal representation using a neural network system trained to predict filter bank region samples given the quantized filter bank region coefficients.

[0018] According to a second aspect of the present invention, the above objective is achieved by a decoder system having: a demultiplexer that separates a bitstream into an encoded low-bandwidth audio signal representation and a set of high-frequency reconstruction (HFR) parameters; a decoder that decodes the low-bandwidth audio signal representation to provide a low-bandwidth audio signal in a filter bank region; a generative model for reconstructing a high-bandwidth signal in a filter bank region using a neural network system trained to predict a sample of a high-bandwidth audio signal in a filter bank region, given a sample of the low-bandwidth signal in the filter bank region and the HFR parameters; and a synthetic filter bank for synthesizing a time-domain audio signal from the low-bandwidth signal in the filter bank region and the reconstructed high-bandwidth signal in the filter bank region.

[0019] A further aspect of the present invention relates to a neural network system for autoregressively generating a current sample for a current time slot of a filter bank representation of an audio signal, wherein the current sample comprises a plurality of values, each corresponding to a channel in the filter bank, and the system comprises first and second submodels, each submodel comprising 1) a processing layer trained to generate conditioned information about the current sample, and 2) an output layer subdivided into a plurality of sequentially executed sublayers, each sublayer being trained to generate a subset of the current sample values, given the conditioned information from the processing layer and samples generated by any previously executed sublayers. The first submodel is trained to generate current sample values ​​corresponding to a low-bandwidth frequency range, given previously generated samples of the filter bank representation and conditioned by quantized samples of the filter bank representation, and the second submodel is trained to generate current sample values ​​corresponding to a high-bandwidth frequency range, given previously generated samples of the filter bank representation and conditioned by quantized samples of the filter bank representation and a set of high-frequency reconstruction parameters.

[0020] It should be noted that this aspect of the present invention may form patentable subject matter independently of the first and second aspects of the present invention.

[0021] In a decoder system equipped with such a neural network system, the first submodel can partially replace the decoder with appropriate training. Such a decoder system is compatible with existing legacy encoders, such as those defined by AC-4 (or HE-AAC) syntax, but offers superior reconfiguration compared to legacy decoders.

[0022] Submodels can operate in different filter bank regions. For example, the first submodel may operate in the MDCT region, which is particularly advantageous when the bitstream contains encoded MDCT samples. [Brief explanation of the drawing]

[0023] The present invention will be described in more detail with reference to the accompanying drawings illustrating currently preferred embodiments of the invention.

[0024] [Figure 1] This is a process diagram of a decoder system according to a first embodiment of the present invention.

[0025] [Figure 2] This is a schematic topology of the generative model shown in Figure 1, based on a particular implementation.

[0026] [Figure 3] The training of the model shown in Figure 2 is illustrated.

[0027] [Figure 4] This is a process diagram of a decoder system according to a second embodiment of the present invention.

[0028] [Figure 5] This is a schematic topology of the generative model shown in Figure 4, based on a particular implementation.

[0029] [Figure 6] This is a process diagram of a decoder system according to a third embodiment of the present invention. [Modes for carrying out the invention]

[0030] The systems and methods disclosed in this application may be implemented as software, firmware, hardware, or a combination thereof. In hardware implementations, task division does not necessarily correspond to division into physical units; conversely, a single physical component may have multiple functions, and a single task may be performed by several collaborating physical components.

[0031] Computer hardware may include, for example, a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a smartphone, a web appliance, a network router, a switch or bridge, or any machine capable of executing instructions (sequentially or otherwise) that specify actions to be performed by such computer hardware. Furthermore, this disclosure relates to any set of computer hardware that individually or collectively execute instructions to perform any one or more of the concepts described herein.

[0032] Certain or all components may be implemented by one or more processors that accept computer-readable (also called machine-readable) code containing a set of instructions that, when executed by one or more processors, perform at least one of the methods described herein. This includes any processor capable of executing a set of instructions (sequential or otherwise) specifying an action to be taken. Thus, an example is a typical processing system (i.e., computer hardware) comprising one or more processors. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system may further include a memory subsystem, including a hard drive, SSD, RAM, and / or ROM. A bus subsystem may be included for communication between components. Software may reside within the memory subsystem and / or within the processors while the computer system is running it.

[0033] One or more processors may operate as standalone devices, or they may be connected to other processors, or they may be networked, for example. Such a network may be built on a variety of different network protocols and may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0034] Software may be distributed on computer-readable media, which may include computer storage media (or non-temporary media) and communication media (or temporary media). As is well known to those skilled in the art, the term computer storage media includes both volatile and non-volatile, removable and non-removable media implemented in any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include, but are not limited to, various forms of physical (non-temporary) storage media, such as EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or any other media that can be used to store desired information and can be accessed by a computer. Furthermore, as is well known to those skilled in the art, communication media (temporary) typically include any information delivery medium, which embodies computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transport mechanisms.

[0035] In the following explanation, a neural network system is said to generate current samples of an audio signal representation. In reality, a neural network system typically generates a probability distribution. Sample reconstruction is performed by sampling according to the probability distribution.

[0036] Figure 1 shows the process of receiving a bitstream, for example, an AC-4 bitstream, and converting it into a low-bandwidth audio signal representation and high-frequency reconstruction parameter a. k The decoder system 10 includes a bitstream demultiplexer 11 configured to separate side information including the decoder 12, which is configured to decode a lowband audio signal representation into a series of lowband filter bank region samples, with each sample x k LFincludes a plurality of values corresponding to filter bank channels in a low-frequency range. The decoder system further includes an autoregressive generative model in the form of a neural network system 13 configured to perform high-frequency reconstruction by predicting high-frequency samples x k LF and high-frequency reconstruction parameters a k to give high-frequency samples x k HF . The synthesis filter bank 14 is configured to generate a combined audio signal in the time domain based on the low-band filter bank samples and the predicted high-band samples.

[0037] In one example, the synthesis filter bank 14 is a quadrature mirror filter (QMF) bank, and the neural network system 13 is trained to output high-band QMF samples (or, strictly speaking, the probability distribution of such samples). The decoder 12 is configured to decode the bitstream into low-band QMF samples. In many applications, the bitstream includes encoded MDCT samples, in which case the decoder 12 first decodes the MDCT samples, inverse-transforms these samples into the time domain, and finally applies a QMF analysis filter bank to obtain QMF samples.

[0038] In use, the decoder system 10 receives a bitstream, which is split by a demultiplexer 11 into a low-band audio signal representation and high-frequency reconstruction parameters a k . The low-band audio signal representation is decoded by the decoder 12. The low-band samples x k LF , here QMF samples, are provided to the generative model 13, which uses the high-frequency reconstruction parameters a k as an additional conditioning to predict high-frequency samples x k HFThis predicts the result. The resulting filter bank samples (decoded lowband and generated highband) are then synthesized into a time-domain (e.g., PCM) signal by the synthesized filter bank 14.

[0039] An example of a neural network-based generative model operating in the filter bank (QMF) region is disclosed in PCT / EP2021 / 078652, which is incorporated herein by reference. This model processes QMF samples end-to-end; that is, it receives QMF subband samples as ground truth input during training and generates QMF samples during inference (generation). The neural network system in PCT / EP2021 / 078652 is a modification of a previously disclosed time-domain generative model that works to facilitate the prediction of filter bank region samples. This model uses upper layers with coarser temporal resolution, which in turn condition lower layers with finer temporal resolution. These lower layers (layer 0) consist of several layers, each predicting a subset of filter bank channels. Subsequent layers are conditioned by the preceding layers via a recurrent neural network (RNN).

[0040] An example of a neural network system 13 that follows this approach is shown in more detail in Figure 2. Here, model 13 includes a (single) upper layer 21 and a lower layer 22. In the upper layer 21, previously generated filter bank samples are received along with decoded low-band samples and high-frequency reconstruction parameters.

[0041] The upper layer 21 is a set of previously generated samples {x <m The layer includes a convolutional network 23 that takes} (i.e., generated before the current time slot m) as input. The convolutional network 23 may have, for example, 32 channels and use a kernel size of 15. The upper layer 21 receives decoded low-bandwidth samples x for the current time slot m (and possibly future time slots > m).m LF and DOM reconstruction parameter a m It further includes another convolutional network 24 that takes as input. Convolutional network 24 can contain, for example, 22 channels. The exact kernel size and stride of convolutional networks 23 and 24 are, for example, the previous sample {x <m Provided frame size, high-frequency reconstruction parameters a m Please note that this can be adjusted based on multiple factors, including the time resolution (one or more).

[0042] The outputs of the convolutional networks 23 and 24 are added together at the addition point 25 and input to the recurrent neural network (RNN) 26. The RNN 26 may be implemented using, for example, one or more stateful network units, such as gated recurrent units (GRUs), long short-term memory units (LSTMs), pseudo-recurrent neural networks, Elman networks, etc. The main characteristic of such an RNN is that it is stateful, that is, it updates its hidden latent state throughout the time steps.

[0043] The output from RNN 26 is provided as input to an upsampling stage 27, which can be implemented, for example, using a transposed convolutional network. It is assumed that the network itself can learn how to perform such upsampling in detail, i.e., the upsampling provided by stage 27 can be "learned upsampling".

[0044] The lower layer 22 is divided into multiple sequentially executed sublayers 28-j (where j=1, ..., L, where L is the total number of sublayers, which is 4 in this case). Each sublayer is configured to generate a reconstructed set of high-frequency band channels.

[0045] Each sublayer 28-j is a set of previously generated samples {x <m The sublayer 28 includes a convolutional network 29 that takes} as input. The convolutional network 29 may have a different number of channels. All convolutional networks may have the same kernel size, for example, 4. Each sublayer 28 further includes another convolutional network 30, which takes decoded low-bandwidth samples x for the current time slot m (and possibly future time slots as well) as its input. m LF and high-frequency reconstruction parameter a m The convolutional network 30 can correspond to the convolutional network 24 in the upper layer 21.

[0046] The outputs of the convolutional networks 29 and 30 and the output from the upper layer 21 are all added together at addition point 31 and then split to form two different tensors of equal size. One of these tensors is input to a recurrent neural network (RNN) 32, for example, another GRU. This RNN 32 may have a similar structure to RNN 26. The output of RNN 32 and the second tensor from the split are added together at a second addition point 33 and fed to a further RNN 34 common to all layers, for example, another GRU. While RNN 32 operates in the "time direction," RNN 34 operates in the "layer direction" or "frequency band direction." Thus, RNN 34 can predict samples for higher filter bank channels from lower filter bank channels. RNN 34 is provided with a condition in the form of a parameter h0 that becomes the initial hidden state of RNN 34. The parameter h0 may be a learned parameter, i.e., one that has been trained by a neural network, or it may be a constant value, such as zero.

[0047] The output from the RNN 34 is ultimately provided as a condition to a neural network 35, such as a multi-layer perceptron MLP, which outputs a set of high-band channels of the current QMF sample. As described above, in an actual implementation, the output from 35 is the parameters, such as the mean and variance, for a selected distribution, such as Gaussian, logistic, or Laplacian. During generation (inference), the filter bank samples are given these parameters and sampled from the distribution.

[0048] The processing starts with sublayer 28-1 and continues with the next sublayers, etc. until the processing in the last sublayer, in this case 28-4. To allow each sublayer to compute one or a few of the total number of channels, the kernels of each convolutional network 29 are accordingly "masked". As an example, using L = 4 sublayers, 10 frequency channels in the decoded low band, and 22 frequency channels in the reconstructed high band, the first two layers 28-1, 28-2 may each generate 4 high-band channels, the third layer 28-3 may generate 6 high-band channels, and the fourth layer 28-4 may generate 8 high-band channels. In a realistic implementation, the number of layers is usually more than this.

[0049] The training of the neural network system (generative model) 13 is shown in FIG. 3. The correct data in the form of filter bank region audio data 41, such as QMF samples including high-band samples 42 and low-band samples 43, are high-frequency reconstruction parameters a kThe low-bandwidth sample 43 is supplied to a parameter generator 44 configured to provide the high-frequency reconstruction parameter a. This parameter generator 44 typically forms part of the encoder in the codec system. The low-bandwidth sample 43 is also supplied to encoders / decoders 45, 46 to provide a decoded version 47 of the low-bandwidth sample. The neural network system 13 uses the high-bandwidth sample 42 as the ground truth and the high-frequency reconstruction parameter a as conditioning parameter. k It is then trained using the decoded low-bandwidth 47.

[0050] In Figure 3, the ground truth highband sample 42 is flattened using the transmitted spectral envelope in the high-frequency reconstruction parameters (48). Flattening reduces, or whitens, the dynamics of the highband QMF sample by applying an inverse gain calculated from the envelope energy. In one embodiment, the highband sample 42 is not "perfectly" flattened, but instead uses a compression factor of about 0.9 instead of 1.0 (where 1.0 corresponds to perfect flattening toward unit variance for each band).

[0051] Returning to Figure 1, the decoder system 10 in this case further includes an inverse flattening unit 15 for inverse flattening the predicted high-bandwidth samples. As described below, Model 13 is trained to predict samples in the flattened QMF region. The inverse flattening unit 15 is configured to apply inverse flattening using spectral envelope information in side information.

[0052] Instead of a "direct" inverse flattening before QMF synthesis, i.e., precisely applying the inverse of the gain applied during the flattening of the ground truth input, further envelope adjustment may also be applied to the output QMF samples flattened by the inverse flattening unit 15, where the adjustment aims to ensure that the envelope of the reconstructed signal follows exactly the envelope information transmitted in the bitstream. The advantage of this approach is that, since the envelope data is based on absolute values, the envelope adjuster acts as a safeguard to ensure that the spectral envelope is always correct (within the precision of the spectral envelope values ​​estimated by the high-frequency reconstruction parameter generator 44).

[0053] Finally, the decoder system 10 performs low-bandwidth sample x k LF It may include a decompression unit 16 to remove any compression applied during encoding. *

[0054] The decoder system 100 in Figure 4, like the decoder system 10, includes a demultiplexer 111, a decoder 112, a generative model 113 in the form of a neural network system, and a composite filter bank 114, such as a QMF filter bank. In this case, the decoder system 100 receives a bitstream, which is then demultiplexed by the demultiplexer 111 into a low-bandwidth audio signal representation and a high-frequency reconstruction parameter a k It is divided into two parts. The low-bandwidth audio signal representation is decoded by decoder 112.

[0055] However, in this implementation, the neural network system 113 is given a decoded low-bandwidth sample and a high-bandwidth sample x k HF Not only that, but also low-bandwidth samples^x k LFIt is also trained to predict. The synthesized filter bank 114 then generates a combined audio signal in the time domain based only on the generated (predicted) filter bank samples (lowband and highband) in this case.

[0056] The neural network system 113 is further described with reference to Figure 5. In this case, system 113 is separated into two submodels 113A and 113B. The first submodel is trained to predict low-bandwidth samples (in multiple channels) given previously generated samples and decoded low-bandwidth samples. The second submodel is trained to predict high-bandwidth samples (in multiple channels) given previously generated samples, decoded low-bandwidth samples, and high-frequency reconstruction parameters.

[0057] Each submodel 113A and 113B includes upper layers 121A and 121B, and lower layers 122A and 122B. The lower layer 122A of the first submodel 113A includes a first group of layers that predict the low-bandwidth range, and the lower layer 122B of the second submodel 113B includes a second group of layers that predict the high-bandwidth range as described above. It is important to note that the RNN 123A in the lower layer 122A and the RNN 123B in the lower layer 122B do not have to be the same within the two submodels. They may each have a separate set of coefficients (weights) determined during training for their particular purposes.

[0058] During training of this model (neural network system 113), the ground truth low-bandwidth samples may also be flattened, in which case the spectral envelope for the low-bandwidth range is calculated, and the temporal resolution of the spectral envelope for the high-bandwidth range is given. Both spectral envelopes are concatenated and used in conditioning.

[0059] In this case, the decoded filter bank samples need to be flattened before being provided to the neural network system 113 using the same type of spectral envelope used to flatten the ground truth samples during training. For this purpose, the decoder system 100 may include a flattening unit 115 placed before the neural network 113. Furthermore, both the low-bandwidth and high-bandwidth ranges predicted by the model 113 are flattened. Thus, the decoder 100 may also include an inverse flattening unit 116 placed between the neural network system 113 and the synthetic filter bank 114. The inverse flattening unit 116 is configured to inversely flatten both the low-bandwidth and high-bandwidth samples generated by the neural network system 113 and optionally also perform envelope adjustment.

[0060] Furthermore, similar to the decoder system 10, any compression applied during the encoding of low-bandwidth samples must also be removed. For this purpose, an extension unit 117 similar to unit 16 in Figure 1 may be placed before the flattening unit 115.

[0061] In the implementation shown in Figure 4, the model (neural network system) 113 is also capable of reconstructing missing samples. In other words, given appropriate training, the decoder system 100 can improve the quality of reconstruction of coarsely quantized filter bank samples by being given appropriate side information (low-band samples decoded from the decoder and high-frequency reconstruction parameters from the bitstream).

[0062] Figure 6 shows a decoder system 200 that has similar capabilities to decoder system 100 but operates in two different filter bank regions, the MDCT region and the QMF region in the illustrated case.

[0063] In this case as well, the bitstream contains information about the quantized MDCT coefficients for the low bandwidth and side information including high-frequency reconstruction parameters. The bitstream is received by the demultiplexer 211, and the low-bandwidth information is decoded by the decoder 212 to recreate the actual quantized MDCT coefficients. Compared to the example in Figure 1 where decoder 12 converts the sample to the QMF region, decoder 212 only needs to perform decoding of the MDCT waveform. Therefore, decoder 212 is sometimes called a waveform decoder.

[0064] The MDCT coefficients are fed to a generative model formed by a first neural network system 213, which is trained to reconstruct a set of MDCT coefficients for low bandwidth with higher resolution. The generative model 213 may be an MDCT predictor as described in PCT / US2021 / 054617, which is incorporated herein by reference. The generated (predicted) MDCT coefficients are then fed to an inverse MDCT transform 214, followed by a QMF analysis filter bank 215, and low bandwidth QMF samples x k LF Generates.

[0065] The remaining blocks in Figure 6 are the second neural network system 216 and the QMF synthesis filter bank 217. The second neural network system 216 may be equivalent to the generative model 13 in Figure 1 and may be trained in a similar manner.

[0066] Similar to the decoder system 10 in Figure 1, if a second neural network system 216 is trained to predict flattened samples, an inverse flattening unit 218 may be provided before the synthetic filter bank 217.

[0067] Unless otherwise specified, as will be apparent from the following description, discussions throughout this disclosure using terms such as “processing,” “computation,” “calculation,” “determination,” and “analysis” are understood to refer to the operation and / or process of computer hardware or computing systems or similar electronic computing devices that manipulate and / or convert data, which is expressed as physical quantities such as electronic quantities, into other data, which is similarly expressed as physical quantities.

[0068] In the above description of exemplary embodiments of the present invention, it should be understood that, for the purpose of improving the flow of disclosure and aiding in the understanding of one or more of the various aspects of the invention, various features of the invention may be grouped together in a single embodiment, figure, or description thereof. However, this method of disclosure should not be interpreted as reflecting an intention that the claimed invention requires more features than are explicitly described in each claim. Rather, as reflected in the following claims, the aspects of the invention consist of fewer features than all the features of a single disclosed embodiment above. Thus, the claims following the detailed description are explicitly incorporated into this detailed description, and each claim stands alone as a distinct embodiment of the invention. Furthermore, while some embodiments described herein include some features included in other embodiments, and others do not, combinations of features of different embodiments are intended to be within the scope of the invention and form different embodiments. This will be understood by those skilled in the art. For example, in the following claims, any of the claimed embodiments may be used in any combination.

[0069] Furthermore, some embodiments are described herein as methods or combinations of elements of methods that can be implemented by a processor of a computer system or by other means of performing the function. Thus, a processor having the instructions necessary to perform such a method or element of a method forms means for performing the method or element of a method. When a method includes several elements, for example, several steps, it should be noted that, unless otherwise stated, the order of such elements is not implied. Furthermore, the elements of apparatus embodiments described herein are examples of means for performing the function performed by the elements for the purpose of carrying out the present invention. Numerous specific details are described in the description provided herein. However, it should be understood that embodiments of the present invention can be carried out without these specific details. On the other hand, well-known methods, structures, and arts are not described in detail so as not to obscure the understanding of this description.

[0070] Those skilled in the art will understand that the present invention is by no means limited to the specific embodiments described above. Rather, many modifications and variations are possible within the scope of the appended claims. For example, other filter bank regions may be used, as described above. Several aspects are described below. [Aspect 1] A method for reconstructing an audio signal; A step of receiving a bitstream containing an encoded low-bandwidth audio signal representation and a set of high-frequency reconstruction (HFR) parameters; The steps include: decoding the encoded low-bandwidth audio signal representation to provide a low-bandwidth audio signal in the filter bank region; A step of reconstructing a high-bandwidth audio signal in a filter bank region using a neural network system trained to predict a sample of a high-bandwidth audio signal in a filter bank region, given a sample of a low-bandwidth signal in the filter bank region and the HFR parameters, wherein the neural network system reconstructs the high-bandwidth audio signal in the filter bank region using the current sample (x) for the current time slot (m) of the high-bandwidth signal in the filter bank region. m The system is configured to autoregressively generate the current sample, each containing multiple values ​​corresponding to the channels of the filter bank, and the system: A processing layer trained to generate conditioning information about the current sample based on the quantized sample of the lowband signal in the filter bank region and the HFR parameters, and A stage comprising an output layer subdivided into multiple sequentially executed sublayers, each sublayer being trained to generate a subset of the current sample values, given the conditioning information from the processing layer and samples generated by previously executed sublayers, if any; The process includes the step of synthesizing a time-domain output audio signal from the low-bandwidth signal of the filter bank region and the high-bandwidth signal of the reconstructed filter bank region. method. [Aspect 2] The method according to embodiment 1, wherein the neural network system is trained to predict high-bandwidth samples in a filter bank region having reduced signal dynamics, and the method further comprises increasing the high-bandwidth signal dynamics in the reconstructed filter bank region. [Aspect 3] The method according to embodiment 2, further comprising using envelope data in the HFR parameters to envelope-tune the high-bandwidth signals in the reconstructed filter bank region. [Aspect 4] The method according to embodiment 1, wherein the low-bandwidth signals in the filter bank region are compressed in the encoding process, and the method further comprises decompressing the low-bandwidth signals in the filter bank region before synthesis. [Aspect 5] The process further includes the step of reconstructing an improved low-band audio signal in a filter bank region using a neural network system trained to predict a sample of the low-band signal in the filter bank region, given a decoded sample of the low-band signal in the filter bank region. The above synthesis is based on the low-bandwidth signal of the reconstructed filter bank region and the high-bandwidth signal of the reconstructed filter bank region. The method described in Embodiment 1. [Aspect 6] The process further includes the step of reconstructing the low-bandwidth audio signal representation using a neural network system trained to predict low-bandwidth filter bank region samples, given quantized filter bank region coefficients. The method described in Embodiment 1. [Aspect 7] The method according to embodiment 6, wherein the neural network system used to reconstruct the low-bandwidth audio signal representation operates in a first filter bank region, and the neural network system used to reconstruct the high-bandwidth audio signal in the filter bank region operates in a second filter bank region. [Aspect 8] The method according to embodiment 7, wherein the first filter bank region is an MDCT region and the second filter bank region is a QMF region. [Aspect 9] A demultiplexer that separates the bitstream into an encoded low-bandwidth audio signal representation and a set of high-frequency reconstruction (HFR) parameters; A decoder that decodes the encoded low-bandwidth audio signal representation to provide a low-bandwidth audio signal in the filter bank region; A generative model for reconstructing a high-bandwidth signal in a filter bank region, wherein the neural network system is trained to predict a sample of a high-bandwidth audio signal in the filter bank region, given a sample of the low-bandwidth signal in the filter bank region and the HFR parameters, wherein the neural network system is trained to predict a sample of the high-bandwidth audio signal in the filter bank region for the current time slot (m) of the high-bandwidth signal in the filter bank region (x m The system is configured to autoregressively generate the current sample, each containing multiple values ​​corresponding to the channels of the filter bank, and the system: A processing layer trained to generate conditioning information about the current sample based on the quantized sample of the lowband signal in the filter bank region and the HFR parameters, and A generative model comprising an output layer subdivided into multiple sequentially executed sublayers, each sublayer being trained to generate a subset of the current sample values, given the conditioning information from the processing layer and samples generated by previously executed sublayers, if any; It has a combining filter bank for synthesizing a time-domain audio signal from low-bandwidth signals in the filter bank region and high-bandwidth signals in the reconstructed filter bank region. Decoder system. [Aspect 10] The decoder system according to embodiment 9, wherein the neural network system is also trained to predict samples of low-bandwidth signals in the filter bank region, given decoded samples of low-bandwidth signals in the filter bank region. [Aspect 11] The aforementioned neural network system is: Given a decoded sample of the low-bandwidth signal in the filter bank region, a first submodel trained to predict the sample of the low-bandwidth signal in the filter bank region; A second submodel, trained to predict high-bandwidth signal samples in the filter bank region, given predicted low-bandwidth signal samples in the filter bank region and the HFR parameters, The decoder system according to embodiment 10, including two submodels. [Aspect 12] The decoder system according to embodiment 11, wherein the first submodel operates in a first filter bank region, and the second submodel operates in a second filter bank region. [Aspect 13] The decoder system according to embodiment 12, wherein the first filter bank region is an MDCT region and the second filter bank region is a QMF region. [Aspect 14] Current sample(x) for the current time slot(m) of the filter bank representation of the audio signal m A neural network system that autoregressively generates ), wherein the current sample contains multiple values, each corresponding to a channel in the filter bank, and the system: It comprises a first and second submodel, each submodel being: A processing layer trained to generate conditioning information about the current sample, It includes an output layer subdivided into multiple sequentially executed sublayers, each sublayer being trained to generate a subset of the current sample values, given the conditioning information from the processing layer and samples generated by previously executed sublayers, if any. The first submodel is given previously generated samples of the filter bank representation and is trained to generate the value of the current sample corresponding to the low-band frequency range, conditioned by the quantized samples of the filter bank representation. The second submodel is given previously generated samples of the filter bank representation and is trained to generate the value of the current sample corresponding to a high-bandwidth frequency range, conditioned by the quantized samples of the filter bank representation and by a set of high-frequency reconstruction parameters. Neural network system. [Aspect 15] A computer program product comprising a portion of computer program code configured to perform the method described in any one of embodiments 1 to 8 when executed on a computer processor.

[0071] Various aspects of the present invention can be understood from the following enumerated example embodiments (EEEs). [EEE1] A method for reconstructing an audio signal; A step of receiving a bitstream containing an encoded low-bandwidth audio signal representation and a set of high-frequency reconstruction (HFR) parameters; The steps include: decoding the encoded low-bandwidth audio signal representation to provide a low-bandwidth audio signal in the filter bank region; The steps include: reconstructing the high-bandwidth audio signal in the filter bank region using a neural network system trained to predict the sample of the high-bandwidth audio signal in the filter bank region, given the low-bandwidth signal sample and the HFR parameters of the filter bank region; The process includes the step of synthesizing a time-domain output audio signal from the low-bandwidth signal of the filter bank region and the high-bandwidth signal of the reconstructed filter bank region. method. [EEE2] The method according to EEE1, wherein the neural network system is trained to predict high-bandwidth samples in a filter bank region having reduced signal dynamics, and the method further comprises increasing the dynamics of the high-bandwidth signal in the reconstructed filter bank region. [EEE3] The method of EEE2, further comprising using envelope data in the HFR parameters to envelope-tune the high-bandwidth signals of the reconstructed filter bank region. [EEE4] The method according to any one of EEE1 to 3, wherein the low-bandwidth signals in the filter bank region are compressed in the encoding process, and the method further comprises decompressing the low-bandwidth signals in the filter bank region before synthesis. [EEE5] The process further includes the step of reconstructing an improved low-band audio signal in a filter bank region using a neural network system trained to predict a sample of the low-band signal in the filter bank region, given a decoded sample of the low-band signal in the filter bank region. The above synthesis is based on the low-bandwidth signal of the reconstructed filter bank region and the high-bandwidth signal of the reconstructed filter bank region. The method described in any one of EEE1 to EEE4. [EEE6] The process further includes the step of reconstructing the low-bandwidth audio signal representation using a neural network system trained to predict low-bandwidth filter bank region samples, given quantized filter bank region coefficients. The method described in any one of EEE1 to EEE5. [EEE7] The method according to EEE6, wherein the neural network system used to reconstruct the low-bandwidth audio signal representation operates in a first filter bank region, and the neural network system used to reconstruct the high-bandwidth audio signal in the filter bank region operates in a second filter bank region. [EEE8] The method according to EEE7, wherein the first filter bank region is an MDCT region and the second filter bank region is a QMF region. [EEE9] A demultiplexer that separates the bitstream into an encoded low-bandwidth audio signal representation and a set of high-frequency reconstruction (HFR) parameters; A decoder that decodes the encoded low-bandwidth audio signal representation to provide a low-bandwidth audio signal in the filter bank region; A generative model for reconstructing the high-bandwidth signal in the filter bank region, using a neural network system trained to predict samples of high-bandwidth audio signals in the filter bank region, given samples of low-bandwidth signals in the filter bank region and the HFR parameters; It has a combining filter bank for synthesizing a time-domain audio signal from low-bandwidth signals in the filter bank region and high-bandwidth signals in the reconstructed filter bank region. Decoder system. [EEE10] The decoder system according to EEE9, wherein the neural network system is also trained to predict samples of low-bandwidth signals in the filter bank region, given decoded samples of low-bandwidth signals in the filter bank region. [EEE11] The aforementioned neural network system is: Given a decoded sample of the low-bandwidth signal in the filter bank region, a first submodel trained to predict the sample of the low-bandwidth signal in the filter bank region; A second submodel, trained to predict high-bandwidth signal samples in the filter bank region, given predicted low-bandwidth signal samples in the filter bank region and the HFR parameters, The method described in EEE10, including two sub-models. [EEE12] The decoder system according to EEE11, wherein the first submodel operates in a first filter bank region and the second submodel operates in a second filter bank region. [EEE13] The decoder system according to EEE12, wherein the first filter bank region is an MDCT region and the second filter bank region is a QMF region. [EEE14] Current sample(x) for the current time slot(m) of the filter bank representation of the audio signal m A neural network system that autoregressively generates ), wherein the current sample contains multiple values, each corresponding to a channel in the filter bank, and the system: It comprises a first and second submodel, each submodel being: A processing layer trained to generate conditioning information about the current sample, It includes an output layer subdivided into multiple sequentially executed sublayers, each sublayer being trained to generate a subset of the current sample values, given the conditioning information from the processing layer and samples generated by previously executed sublayers, if any. The first submodel is given previously generated samples of the filter bank representation and is trained to generate the value of the current sample corresponding to the low-band frequency range, conditioned by the quantized samples of the filter bank representation. The second submodel is given previously generated samples of the filter bank representation and is trained to generate the value of the current sample corresponding to a high-bandwidth frequency range, conditioned by the quantized samples of the filter bank representation and by a set of high-frequency reconstruction parameters. Neural network system. [EEE15] A computer program product comprising a portion of computer program code configured to perform the method described in any one of the EEE1 through EEE8 when executed on a computer processor.

Claims

1. A method for reconstructing an audio signal; The steps include receiving a bitstream containing an encoded low-bandwidth audio signal representation and a set of high-frequency reconstruction (HFR) parameters; The steps include: decoding the encoded low-bandwidth audio signal representation to provide a low-bandwidth audio signal in the filter bank region; A step of reconstructing the high-bandwidth audio signal in the filter bank region using a neural network system trained to predict a sample of the high-bandwidth audio signal in the filter bank region, given a sample of the low-bandwidth signal in the filter bank region and the HFR parameters, wherein the neural network system reconstructs the high-bandwidth audio signal in the filter bank region using the current sample (x) for the current time slot (m) of the high-bandwidth signal in the filter bank region. m The system is configured to autoregressively generate the current sample, each containing multiple values ​​corresponding to the channels of the filter bank, and the system: A processing layer trained to generate conditioning information about the current sample based on the quantized sample of the lowband signal in the filter bank region and the HFR parameters, and A stage comprising an output layer subdivided into multiple sequentially executed sublayers, each sublayer being trained to generate a subset of the current sample values, given the conditioning information from the processing layer and samples generated by previously executed sublayers, if any; The process includes the step of synthesizing a time-domain output audio signal from the low-bandwidth signal of the filter bank region and the high-bandwidth signal of the reconstructed filter bank region. method.

2. The method according to claim 1, wherein the neural network system is trained to predict high-bandwidth samples in a filter bank region having reduced signal dynamics, and the method further comprises increasing the high-bandwidth signal dynamics in a reconstructed filter bank region.

3. The method according to claim 2, further comprising using envelope data in the HFR parameters to envelope-tune the high-bandwidth signals of the reconstructed filter bank region.

4. The method according to claim 1, wherein the low-bandwidth signals in the filter bank region are compressed in the encoding process, and the method further comprises decompressing the low-bandwidth signals in the filter bank region before synthesis.

5. The process further includes the step of reconstructing an improved low-band audio signal in the filter bank region using a neural network system trained to predict the low-band signal sample in the filter bank region, given a decoded sample of the low-band signal in the filter bank region. The above synthesis is based on the low-bandwidth signal of the reconstructed filter bank region and the high-bandwidth signal of the reconstructed filter bank region. The method according to claim 1.

6. The process further includes the step of reconstructing the low-bandwidth audio signal representation using a neural network system trained to predict low-bandwidth filter bank region samples, given quantized filter bank region coefficients. The method according to claim 1.

7. The method according to claim 6, wherein the neural network system used to reconstruct the low-bandwidth audio signal representation operates in a first filter bank region, and the neural network system used to reconstruct the high-bandwidth audio signal in the filter bank region operates in a second filter bank region.

8. The method according to claim 7, wherein the first filter bank region is an MDCT region and the second filter bank region is a QMF region.

9. A demultiplexer that separates the bitstream into an encoded low-bandwidth audio signal representation and a set of high-frequency reconstruction (HFR) parameters; A decoder that decodes the encoded low-bandwidth audio signal representation to provide a low-bandwidth audio signal in the filter bank region; A generative model for reconstructing a high-bandwidth signal in a filter bank region, wherein the neural network system is trained to predict a sample of a high-bandwidth audio signal in the filter bank region, given a sample of the low-bandwidth signal in the filter bank region and the HFR parameters, wherein the neural network system is trained to predict a sample of the high-bandwidth audio signal in the filter bank region for the current time slot (m) of the high-bandwidth signal in the filter bank region (x m The system is configured to autoregressively generate the current sample, each containing multiple values ​​corresponding to the channels of the filter bank, and the system: A processing layer trained to generate conditioning information about the current sample based on the quantized sample of the lowband signal in the filter bank region and the HFR parameters, and A generative model comprising an output layer subdivided into multiple sequentially executed sublayers, each sublayer being trained to generate a subset of the current sample values, given the conditioning information from the processing layer and samples generated by previously executed sublayers, if any; It has a combining filter bank for synthesizing a time-domain audio signal from low-bandwidth signals in the filter bank region and high-bandwidth signals in the reconstructed filter bank region. Decoder system.

10. The decoder system according to claim 9, wherein the neural network system is also trained to predict samples of low-bandwidth signals in the filter bank region, given decoded samples of low-bandwidth signals in the filter bank region.

11. The aforementioned neural network system is: Given a decoded sample of the low-bandwidth signal in the filter bank region, a first submodel trained to predict the sample of the low-bandwidth signal in the filter bank region; A second submodel, trained to predict high-bandwidth signal samples in the filter bank region, given predicted low-bandwidth signal samples in the filter bank region and the HFR parameters, The decoder system according to claim 10, comprising two submodels.

12. The decoder system according to claim 11, wherein the first submodel operates in a first filter bank region, and the second submodel operates in a second filter bank region.

13. The decoder system according to claim 12, wherein the first filter bank region is an MDCT region and the second filter bank region is a QMF region.

14. Current sample (x) for the current time slot (m) of the filter bank representation of the audio signal m A neural network system that autoregressively generates ) wherein the current sample contains multiple values, each corresponding to a channel in the filter bank, and the system: It comprises a first and second submodel, each submodel being: A processing layer trained to generate conditioning information about the current sample, It includes an output layer subdivided into multiple sequentially executed sublayers, each sublayer being trained to generate a subset of the current sample values, given the conditioning information from the processing layer and samples generated by previously executed sublayers, if any. The first submodel is given previously generated samples of the filter bank representation and is trained to generate the value of the current sample corresponding to the low-band frequency range, conditioned by the quantized samples of the filter bank representation. The second submodel is given previously generated samples of the filter bank representation and is trained to generate the value of the current sample corresponding to a high-bandwidth frequency range, conditioned by the quantized samples of the filter bank representation and by a set of high-frequency reconstruction parameters. Neural network system.

15. A computer program for causing a computer processor to perform the method described in any one of claims 1 to 8.

Images