IVAS SPAR filter banks in the QMF domain
By emulating SPAR filter banks within the QMF domain using time-domain filters, the method addresses the challenge of decoder filter bank utilization in immersive audio, achieving efficient and low-latency signal processing with reduced computational complexity.
Patent Information
- Application Number
- JP2024537477
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-12-20
- Filing Date
- 2022-12-20
- Publication Date
- 2024-12-03
AI Technical Summary
Existing techniques face challenges in efficiently utilizing decoder filter banks within the QMF domain for SPAR decoded content, particularly in immersive audio processing, due to the need for accurate parameter application and mismatch between encoder and decoder filter banks.
A method and apparatus for processing multi-channel audio signals by emulating the filters of a first filter bank, such as a SPAR filter bank, within the domain of a second filter bank, like an oversampled QMF filter bank, using time-domain filters and prototype filters to reduce computational complexity and maintain audio quality.
This approach allows for efficient signal processing in the QMF domain without additional computational burden, maintaining the advantages of the first filter bank's frequency resolution while enabling further signal manipulation, thus reducing overall complexity and latency.
Smart Images

Figure 2024544730000001_ABST
Abstract
Description
[Technical field]
[0001] (Reference to Related Application) This application claims priority to U.S. Provisional Application No. 63 / 291,817, filed December 20, 2021, the contents of which are incorporated herein by reference.
[0002] The present disclosure relates to techniques for processing representations of multi-channel audio signals. In particular, the present disclosure describes SPAR decoding, which involves the implementation of a SPAR filter bank in a domain of a QMF bank (e.g., an oversampled QMF bank) suitable for signal manipulation. [Background technology]
[0003] IVAS SPAR is a low-latency codec for First Order Ambisonics (FOA) and Higher Order Ambisonics (HOA) spatial audio based on the Low Latency Core codec.
[0004] Immersive Audio and Video Services (IVAS) Spatial Reconstruction (SPAR) uses the Modified Discrete Fourier Transform (MDFT) for signal analysis and as a fast convolution kernel for the SPAR Finite Impulse Response (FIR) filter bank. The SPAR filter bank consists of carefully designed low-delay FIR band filters (typically 12) with time and frequency resolution adapted to the human auditory system. The SPAR filter bank operates in the encoder and decoder. In the encoder, the active downmix signal and the residual signal are calculated and sent to the decoder along with parameters (e.g., SPAR parameters). In the decoder, the encoder-side process is reversed and the original signal is reconstructed using the transmitted parameters. To faithfully reconstruct the signal, the filter banks in the encoder and decoder need to match exactly.
[0005] On the other hand, using an oversampled QMF bank in the decoder may be more suitable for signal manipulation (eg parametric audio processing and decoding), possibly on a finer time grid, than the SPAR MDFT domain.
[0006] Therefore, there is a need for techniques to enable efficient use of the decoder filter bank in the QMF domain for SPAR decoded content, and more generally, to enable use of filters of a first filter bank in the domain of a second filter bank. Summary of the Invention
[0007] In view of this need, the present disclosure provides a method and an apparatus for processing a representation of a multi-channel audio signal, as well as a corresponding program and a computer-readable storage medium, having the features of the respective independent claims.
[0008] An aspect of the present disclosure relates to a method for processing a representation of a multi-channel audio signal. The method may be implemented on a computer, for example. The processing may involve decoding, such as SPAR decoding. The multi-channel audio signal may be a spatial audio signal, such as a FOA audio signal or a HOA audio signal. The representation may include a first channel and metadata related to a second channel. Furthermore, the representation of the multi-channel audio signal may include more than one second channel. The first channel may be a carrier channel (or a channel encoded into the carrier channel). The second channel may be a channel other than the carrier channel (or a channel encoded into the carrier channel), in particular a parametrically encoded channel. The metadata may include, for each of a plurality of first bands of a first filter bank, respective prediction parameters (e.g., gain parameters) for making a prediction for the second channel based on the first channel in the first band. The method may include applying a second filter bank having a plurality of second bands to the first channel to obtain, for each of the second bands, a banded version of the first channel in the second band. The second filter bank may be different from the first filter bank. The method may further include generating, for each of the second bands, a respective time-domain filter based on the prediction parameters and a first filter of the first filter bank, where the first filter may correspond to the first band. The method may further include generating a prediction for the second channel based on the banded version of the first channel in the second band and the time-domain filter. This may include, for example, generating, for each of the second bands, a prediction for the second channel within the second band based on the filtered version of the first channel in the second band.Here, the filtered version of the first channel may be obtained by applying a respective time domain filter in the second band to a banded version of the first channel in the second band.
[0009] Thus, the reconstruction of the original multi-channel audio signal and the subsequent audio processing do not require a transformation into the domain of the first filter bank and then into the domain of the second filter bank. Instead, the filters of the first filter bank may be "emulated" in the domain of the second filter bank, thereby avoiding further transformation steps. This allows enjoying the specific advantages of the first filter bank for coding (such as bands specifically adapted to human hearing) while also enjoying the specific advantages of the second filter bank for further signal processing of the reconstructed multi-channel audio signal (such as better time resolution) without any additional computational burden.
[0010] In some embodiments, the multi-channel audio signal may be a First Order Ambisonics (FOA) or a Higher Order Ambisonics (HOA) audio signal.
[0011] In some embodiments, the prediction parameter may be a SPAR parameter (eg, a gain parameter).
[0012] In some embodiments, the first filter bank may be a SPAR filter bank with FIR band filters and may use MDFT. For SPAR, there may be, for example, 12 first banks.
[0013] In some embodiments, the second filter bank may be a QMF filter bank. Furthermore, the second filter bank may be, for example, an oversampled filter bank, in particular an oversampled QMF filter bank.
[0014] In some embodiments, the time domain filter may be a multi-tap FIR filter.
[0015] In some embodiments, generating a time domain filter for a given second band may include generating a plurality of adapted first filters based on a prototype filter for each first filter and the filter transformation.
[0016] In some embodiments, for a given second band l, a first filter h for a given first band b is b The first filter adapted to
number
[0017]
number
[0018] In some embodiments, the method may further include generating a prototype filter for the filter transformation based on the prototype filter of the second filter bank.
[0019] In some embodiments, a prototype filter for the filter transformation may be generated based on a prototype filter of the second filter bank by solving a least squares problem.
[0020] In some embodiments, generating a prototype filter for the filter transformation comprises generating a non-causal prototype filter p based on a prototype filter p of the second filter bank. AThe generating step may further include generating an acausal prototype filter p A and a prototype filter p of the second filter bank, the generating step further comprising: generating a cross-correlation p2 of the second filter bank with a dimension S×R, where n,m is an integer multiple of S and only for indices n,m where n is a non-zero element v n,m A set of matrices V for some integer K, (k) , k=-K, ..., K, where R is a length of a prototype filter for the filter transformation. The generating may further include generating V (k) This may include solving a set of least squares problems for q, where q is a vector of dimension R×1 containing filter coefficients of a prototype filter q for the filter transformation.
[0021] In some embodiments, generating the time domain filter for a given second band may further include taking a weighted sum of the adapted first filters, where the adapted first filters may be weighted with a prediction coefficient (e.g., a gain) for each first band.
[0022] In some embodiments, the prototype filter for the filter transformation may be an asymmetric prototype filter.
[0023] In some embodiments, the processing stride for each tap may be less than or equal to the number of second bands.
[0024] In some embodiments, generating a time domain filter for a given second band may include approximating a given first filter by the first and second fundamental signals, where the first fundamental signal may be obtainable as a result of applying the second filter bank, the fundamental real single tap filter, and a synthesis filter bank of the second filter bank to a fundamental signal having a single non-zero sample at each sample position. The fundamental real single tap filter may be a filter for each single one of the second bands having a single non-zero filter coefficient at each tap position. Furthermore, the second fundamental signal may be obtainable as a result of applying the second filter bank, the fundamental imaginary single tap filter, and a synthesis filter bank of the second filter bank to a fundamental signal, where the fundamental imaginary single tap filter may be a filter for each single one of the second bands having a single non-zero filter coefficient at each tap position. The generating may further include generating a time domain filter adapted for the first filter in the second band based on the coefficients of the first and second fundamental signals in the approximation.
[0025] In some embodiments, generating the time domain filter for a given second band includes generating a second filter bank, a real single tap filter
number
number
[0026]
number
number
number
[0027] In some embodiments, the method may further include truncating a filter length of the time domain filter.
[0028] Thus, computational complexity can be reduced, potentially without perceptual effects.
[0029] In some embodiments, the filter length of a given time domain filter after truncation may depend on the second band of each of the time domain filters.
[0030] In some embodiments, generating a time domain filter for the given second band may include generating a respective base (or adapted) time domain filter (e.g., adapted filter) in the given second band for each of the first filters, and generating a time domain filter in the given second band based on the base time domain filter in the given second band and the prediction parameters. Then, the truncation of the time domain filter for the given second band may be based on thresholds for filter coefficients of the base time domain filters, where each threshold corresponds to a respective one of the first filters. The threshold for the base time domain filter for the given first filter may be derived from a maximum magnitude of the base time domain filter in the plurality of second bands.
[0031] In some embodiments, the method may further include determining, for each first band, a maximum magnitude of a corresponding basic time domain filter in the plurality of second bands. The method may further include determining, for each first band, a minimum truncated filter length for a corresponding basic time domain filter in the plurality of second bands based on a threshold derived from the maximum magnitude. The method may further include determining, for each second band, a filter length of a time domain filter in the second band based on the minimum truncated filter length for a basic time domain filter in the second band.
[0032] In some embodiments, the time domain filter may be a single tap FIR filter.
[0033] By implementing single tap FIR filters, the filters of the first filter bank can be emulated within the domain of the second filter bank with minimal computational burden.
[0034] In some embodiments, generating a time domain filter for a given second band may include determining a first band of the plurality of first bands having a highest energy within the second band, and may further include generating the time domain filter based on a linear phase approximation of the first filter corresponding to the determined first band and corresponding prediction coefficients for the determined first band.
[0035] In some embodiments, generating a time domain filter for a given second band may include determining a set of first bands of the plurality of first bands that have the highest energy in the second band. The generating may further include generating a time domain filter based on a weighted sum of linear phase approximations of the first filters corresponding to the determined set of first bands, where the weights in the weighted sum may depend on the corresponding prediction coefficients for the determined set of first bands and the normalized magnitudes or energies of each of the first bands of the determined set of first bands in the second band, where it is understood that the normalized magnitudes or energies sum to one.
[0036] According to another aspect, a method for generating a representation of a multi-channel audio signal is provided. The representation may include metadata related to a first channel and a second channel. The metadata may include, for each of a plurality of first bands of a first filter bank, respective prediction parameters for making a prediction for the second channel based on the first channel in the first band. The method may include generating a prediction for the second channel based on a first filter of the first filter bank and the prediction parameters, where the prediction for the second channel may be represented by a time-domain signal (e.g., a prediction signal). The method may further include generating a residual for the second channel by subtracting, in the time domain, the prediction for the second channel from the second channel.
[0037] In some embodiments, the representation of the multi-channel audio signal may further include a residual for a second channel.
[0038] According to another aspect, there is provided an apparatus for processing a representation of a multi-channel audio signal. The apparatus may comprise a processor and a memory coupled to the processor for storing instructions for the processor. The processor may be configured to perform all steps of the method according to the above aspect and embodiments thereof.
[0039] According to another aspect, a computer program is described, which may include executable instructions for performing the methods or method steps outlined throughout this disclosure when executed by a computing device.
[0040] According to yet another aspect, a computer readable storage medium is described, which may store a computer program adapted for execution on a processor and, when executed by the processor, to perform the methods or method steps outlined throughout this disclosure.
[0041] It should be noted that the methods and systems, including preferred embodiments thereof, outlined in this disclosure may be used alone or in combination with other methods and systems disclosed herein. Furthermore, all aspects of the methods and systems outlined in this disclosure may be combined in any manner. In particular, the features of the claims may be combined with each other in any manner.
[0042] It will be understood that apparatus features and method steps are interchangeable in many ways. In particular, details of the disclosed methods can be implemented by corresponding apparatus, and vice versa, as will be appreciated by those skilled in the art. Furthermore, any statements made above with respect to methods (and, e.g., steps thereof) will be understood to apply equally to corresponding apparatus (and, e.g., blocks, stages, units thereof), and vice versa. [Brief description of the drawings]
[0043] The invention will now be described, by way of example only, with reference to the accompanying drawings, in which:
[0044] [Figure 1] FIG. 1 is a block diagram that diagrammatically illustrates an example of SPAR encoding and decoding, followed by processing in the QMF filter band domain. [Diagram 2] FIG. 2 is a block diagram that diagrammatically illustrates an example of SPAR encoding and decoding in the QMF filter bank domain, according to an embodiment of the present disclosure. [Diagram 3] FIG. 3 is a flow chart that diagrammatically illustrates an example of a method for processing a representation of a multi-channel audio signal, according to an embodiment of the present disclosure. [Figure 4] FIG. 4 illustrates a schematic example of a conversion of a SPAR filter bank FIR band filter to a QMF domain FIR filter, according to an embodiment of the present disclosure. [Diagram 5] FIG. 5 is a diagram illustrating an example of a low-delay SPAR FIR band filter used within a SPAR encoder. [Figure 6] FIG. 6 shows an example of a low-delay asymmetric QMF prototype filter. [Figure 7] FIG. 7 is a diagram illustrating an example prototype filter for transforming a SPAR FIR filter to a QMF domain SPAR FIR filter using the asymmetric prototype filter of FIG. [Figure 8] FIG. 8 is a diagram illustrating an example of FIR filter length after truncation of a transformed FIR filter according to an embodiment of the present disclosure. [Figure 9] 9A, 9B, 9C, and 9D include diagrams illustrating example magnitudes of filter coefficients of a transformed FIR filter according to an embodiment of the present disclosure. [Figure 10]10A, 10B, 10C, and 10D include diagrams illustrating an example of the first 400 samples of the original SPAR filter impulse response (solid line) and an approximation using a QMF filter (dashed line), according to an embodiment of the present disclosure. [Figure 11] FIG. 11 includes diagrams illustrating examples of cumulative SPAR filters in the QMF domain and modified cumulative SPAR filters in the QMF domain with processing in band 8, according to an embodiment of the present disclosure. [Figure 12] FIG. 12 includes diagrams illustrating example SPAR filter frequency responses (1 ms latency, 12 bands) for possible designs having a bandwidth below 400 Hz at the low center frequency and for possible designs with a minimum bandwidth of 400 Hz and band boundaries aligned to the QMF band boundaries in accordance with an embodiment of the present disclosure. [Figure 13] FIG. 13 is a diagram illustrating an example of the overlay of (QMF-compatible) SPAR encoder filter bands (dashed lines, 12 bands) and QMF decoder filter bands (solid lines, 60 bands) according to an embodiment of the present disclosure. [Figure 14] FIG. 14 illustrates an example of a single tap SPAR filter in the QMF domain (magnitude frequency response in QMF bands) as a column per each SPAR band filter, according to an embodiment of the present disclosure. [Figure 15] FIG. 15 is a flow chart that conceptually illustrates an example method for low-complexity SPAR filtering in the QMF filter bank domain, according to an embodiment of the present disclosure. [Figure 16] FIG. 16 is a flow chart that conceptually illustrates another example of a method for low-complexity SPAR filtering in the QMF filter bank domain, according to an embodiment of the present disclosure. [Figure 17] FIG. 17 includes diagrams illustrating example signal-to-noise ratios (SNRs) for decoded binaural signals for IVAS SPAR with and without QMF domain reconstruction, according to an embodiment of the present disclosure. [Figure 18]FIG. 18 includes diagrams illustrating example signal-to-noise ratios (SNRs) for decoded binaural signals for IVAS SPAR with and without QMF domain reconstruction, according to an embodiment of the present disclosure. [Figure 19] FIG. 19 illustrates a schematic example of an apparatus for implementing a method according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0045] Generally, the present invention relates to parametric filter bank processing for audio coding, where parameters are applied with one filter bank (e.g., a SPAR filter bank) at the encoder and the parameter application is reversed with another filter bank (e.g., a complex QMF filter bank) at the decoder. The present disclosure solves the problem of mismatch of encoder and decoder filter banks for accurate parameter application.
[0046] One advantage of using two different filter banks is the different performance tradeoffs: the filter bank at the encoder may have very low delay but a relatively large processing stride due to the requirement for an efficient FFT-based implementation, while the filter bank at the decoder may have a higher delay but the ability to apply parameters with a smaller stride as needed for efficient subsequent processing.
[0047] In view of the above, embodiments of the present disclosure relate to SPAR decoding and merging of a SPAR decoder filter bank (as a non-limiting example of a first filter bank domain) into a QMF domain (as a non-limiting example of a second, different filter bank domain), e.g., by FIR filtering along time in the QMF bands.
[0048] System Overview The FIR filters may be time-varying depending on the transmitted SPAR parameters. Similar to the SPAR filter bank operation in the MDFT domain, a weighted sum of all band filters may be performed rather than each band filter individually. To reduce complexity, the QMF domain FIR filters may be truncated in a manner that depends on the QMF band frequencies. Possibly, some processing can take advantage of the good frequency resolution SPAR filter bank and be implemented efficiently by merging the processing into the SPAR filters (and still take advantage of the relatively high time resolution of the QMF domain). Other processing steps may only be performed in the QMF domain after the SPAR filtering.
[0049] It may be noted that the QMF filter bank should have nearly perfect reconstruction properties and sufficiently large aliasing rejection to allow high quality signal modification, but these requirements must always be met if the QMF domain is used for signal modification.
[0050] FIG. 1 illustrates diagrammatically an example of a default IVAS SPAR system 100 with subsequent QMF domain processing.
[0051] In the encoder, a multi-channel audio signal 10 is input to an MDFT analysis block 105 for applying a SPAR MDFT filter bank (as a non-limiting example of a first filter bank). The multi-channel audio signal 10 is also input to a signal analysis block 110 for generating prediction parameters (e.g., SPAR parameters, gain parameters) 115 for predicting an audio channel other than the audio channel associated with the transport channel (second audio channel) from the audio channel associated with the transport channel (first audio channel). The output of the MDFT analysis block 105 is input to a filter / prediction block 120, where the prediction parameters 115 are used to generate a prediction for the second channel and, based on the prediction, a residual for the second channel (e.g., a residual for a reconstructed version of the first channel). The first channel signal and the residual signal are then provided to an MDFT synthesis block 130, which performs the inverse operation of the MDFT analysis block 105. The prediction parameters 115 are also provided to the output of the decoder and output as metadata.
[0052] Thus, the encoder outputs a representation 20 of the multi-channel audio signal including metadata related to the first channel (e.g., a waveform-encoded version of the first channel) and the second channel. Possibly, the representation may relate to multiple second channels, but the following description is limited to a single second channel for reasons of brevity and without intending to be limiting. The metadata includes, for each of multiple first bands of the first filter bank, respective prediction parameters for making a prediction for the second channel based on the first channel in that first band. The representation may further include a residual for the second channel.
[0053] In an alternative implementation, instead of transmitting the residual for the second channel, active downmixing may be performed: in this case, the transmitted first channel may be generated in the encoder by time- and frequency-variant downmixing using a first filter bank (e.g., a SPAR filter bank).
[0054] In the decoder, the MDFT is applied by the MDFT analysis block 135, and the inverse prediction is performed by the filter / inverse prediction block 140 using the prediction parameters 115 and the filters of the MDFT analysis block 105 of the encoder. Specifically, within each MDFT band, a prediction for the second channel is generated based on each filtered version of the first channel and each prediction parameter, and can be used to reconstruct the second channel together with the residual for the second channel. The inverse of the process of the MDFT analysis block 135 is then performed by the MDFT synthesis block 150. Thus, the process of the filter / inverse prediction block 140 can be said to be the inverse of the process of the filter / prediction block 120.
[0055] In implementations using active downmixing, the active downmixing may be at least partially undone by time and frequency change scaling based on the transmitted prediction parameters at the decoder using the same filterbank processing techniques.
[0056] The output of the MDFT synthesis block 150, e.g. the reconstructed multi-channel audio signal, is then input to a QMF analysis block 160 for applying a QMF analysis filter bank (as a non-limiting example of a second filter bank). Within the QMF domain, a desired QMF processing is applied to the output of the QMF analysis block 160 by a QMF processing block 170, using processing parameters 175 as necessary. The result is input to a QMF synthesis block 180 for applying a QMF synthesis filter bank that corresponds (e.g. inverse) to the QMF analysis filter bank. This produces a reconstructed and processed multi-channel audio signal 30.
[0057] The processing chain of the default IVAS SPAR system 100 of Figure 1 may have high computational complexity at the decoder side, since it requires MDFT analysis and synthesis followed by QMF analysis and synthesis. In addition, the processing chain may have a delay corresponding to the combined delay of the SPAR filter bank and the QMF filter bank.
[0058] FIG. 2 illustrates a schematic example of a modified IVAS SPAR system 200 for joint QMF domain SPAR decoding and processing according to an embodiment of the present disclosure.
[0059] The blocks 105, 110, 120, and 130 (i.e., the encoder) may be identical to the corresponding blocks in the default IVAS SPAR system 100 of FIG. 1. At the decoder side, a representation 20 of the multi-channel audio signal is input to a QMF analysis block 210, which may have the same functionality as the QMF analysis block 160. Then, unlike the default IVAS SPAR system 100, an inverse prediction is performed in the QMF domain by a filter / inverse prediction block 220, which takes as input the prediction parameters (e.g., SPAR parameters) 115 and the filters of the MDFT analysis block 105 of the encoder. Then, in a QMF processing block 230, the desired QMF processing is applied. To the processed result, in a QMF synthesis block 240, a QMF synthesis filter bank corresponding to the QMF analysis filter bank of the QMF analysis block 210 is applied, which finally outputs a reconstructed and processed multi-channel audio signal 40.
[0060] In some implementations, the encoder does not transmit the (prediction) residual to the decoder. In this case, the QMF domain processing in the decoder may include filling in the missing energy with a decorrelated first channel (e.g., W) signal. The decorrelated signal may be derived using the transmitted parameters. In the case of active downmixing, the QMF domain processing may include active mixing that at least partially reverses the active downmixing.
[0061] Figures 1 and 2 also show the delay and time stride. In the default and modified IVAS SPAR systems of Figures 1 and 2, the following applies with respect to delay, time stride, and computational complexity:
[0062] Delay ○ SPAR filter delay "delay 1" can be 1-4 ms (e.g., typically 1 ms) 〇QMF analysis-synthesis delay "Delay 2" can typically be 2.5-5.0 ms o The overall delay of system 100 and system 200 can be the same (delay1 + delay1 + delay2) ● Time stride 〇Stride 2<Stride 1 ■ SPAR prediction and processing time strides in the MDFT domain “Stride 1” can be relatively large (e.g., typically 10-20 ms) to allow for the most efficient fast convolution with the SPAR filter. ■ QMF domain strides are typically 1.25 ms, 1.33 ms, or 1 ms, which may allow fine time-grid signal modification, e.g., dedicated handling of transients Calculation complexity The complexity of the system 100 without QMF analysis-synthesis can be roughly comparable to the complexity of the system 200 with QMF analysis-synthesis.
[0063] In general, the encoding and decoding process can be described for the example of two encoded audio signals x1 (a first signal related to a first channel) and x2 (a second signal related to a second channel). To simplify the labeling of the signals, both signal and parameter quantization are omitted. Also, for simplicity, the gain parameter (generally as an example of a SPAR parameter or prediction parameter) is assumed to be frequency-dependent but static over time (e.g., over the period of one frame).
[0064] In the encoder, the first signal x1 is filtered through a SPAR filter bank and its FIR filter h b (as an example of the first filter bank) into frequency bands. The second signal x2 is filtered within each band with a gain parameter g bThen, the prediction residual of x2 is calculated, and the prediction residuals of x1 and x2 are transformed back to the wideband time domain by SPAR filter bank synthesis to generate x′1 and x′2. The resulting signals x′1 and x′2 are then transmitted in the bitstream along with gain parameters (generally as an example of SPAR parameters or prediction parameters).
[0065] At the decoder in the IVAS SPAR system 100 of FIG. 1, the encoder process is reversed using a SPAR filter bank and transmitted gain parameters (as examples of SPAR parameters or prediction parameters in general) to generate reconstructed signals x"1 and x"2. For further processing, QMF analysis is applied to these signals, adding delay and computational complexity.
[0066] At the decoder in the modified IVAS SPAR system 200 of Figure 2, the encoder processing is reversed in the QMF domain using QMF domain SPAR filter and gain parameters. Further processing in the QMF domain can either be merged with the SPAR signal reconstruction or occur as a second processing step in the QMF domain.
[0067] Processing Details Next, example implementation details for the above processes in the example systems 100 and 200 are described.
[0068] [Notation] It is understood that all signals and filters are defined for any integer argument by extending with zeros for arguments outside their support (defined by a range explicitly occupied by finite range data).
[0069] [SPAR filter bank] The SPAR filters of the SPAR filter bank can be FIR bandpass filters. Their length can be, for example, 960 or 480 or 240 taps. Furthermore, the center frequency and the bandwidth can be motivated by auditory perception. The FIR filters form a perfect reconstruction filter bank in the sense that they are summed with delayed Dirac pulses (e.g., typically the delay is 1 or 2 or 4 ms). In this way, the filter bank synthesis operation can be a simple sum of banded signals. The FIR filtering can be implemented via fast convolution using the MDFT. Band modification with parameters can occur in the MDFT domain, and a subsequent time-domain cross-fade can be applied to avoid jumps between parameter sets.
[0070] Since a SPAR filter bank may perform perfect or near-perfect reconstruction, the impulse response h of a SPAR filter bank may be given as:
[0071]
number
[0072] The SPAR filter bank response when applying a gain parameter (generally as an example of a SPAR parameter or prediction parameter) within each frequency band may be given by:
[0073]
number
[0074] [QMF filter bank] A time domain signal x can be transformed to the complex QMF domain X, for example via:
[0075]
number
[0076] l=0,1,...,L-1, where N is the length of the prototype filter p, non-zero for n=0,1,...,N-1 and zero otherwise. L is the number of QMF frequency channels (e.g., typically L=60), S is the processing stride in samples, k refers to the time slot index, and D is the analysis-synthesis delay in samples (the delay associated with sample-wise processing). An example of a prototype filter is shown in the diagram of Figure 6.
[0077] In general, this can be expressed in a more compact form using QMF analysis operators as follows:
[0078]
number
[0079] The time domain signal x′ can be reconstructed from the QMF representation X, for example, as follows:
[0080]
number
[0081] In general, this can be expressed in a more compact form using the QMF compositing operator as follows:
[0082]
number
[0083] The QMF analysis-synthesis system is assumed to provide near perfect reconstruction, for example in the systems 100, 200 of FIGS. 1 and 2, with a delay of D2 samples.
[0084]
number
[0085] SPAR band filter h for QMF band l and SPAR filter b b QMF representation of (as an example of a second filter bank representation)
number
[0086]
number
[0087] The SPAR filter bank response in the QMF domain, for example, is the sum over all the SPAR filters.
[0088]
number
[0089] The same is true when applying the SPAR gain parameters (as an example of a prediction parameter) within each SPAR frequency band.
[0090]
number
[0091] An example of such a SPAR filter bank response in the QMF domain is shown in the bottom panel of FIG.
[0092] The SPAR filter bank delay can be modeled in the QMF domain using a converter as follows:
[0093]
number
[0094] Signal Processing The encoder signal may be calculated, for example, as follows:
[0095]
number
number
[0096] Therefore, the prediction for the second channel signal is made by using the filter of the first filter bank (first filter) and the prediction parameters (e.g., filter h g (k)) for the second channel. This prediction may be represented by a time domain signal, as in the example of equation (12). Then, the residual for the second channel
number
[0097] Alternatively, the residual signal
number
[0098]
number
[0099] However, this implementation is more computationally expensive than the implementation of equation (12) and may result in larger reconstruction errors if the SPAR filter bank is not perfect reconstruction.
[0100] In particular, the residual of the second channel signal
number
[0101] In the case of active downmixing, the transmitted signal may be calculated as follows:
[0102]
number
number
[0103] The decoder signal in the system 100 of FIG.
[0104]
number
number
[0105] The decoder signal in the system 200 of FIG. 2 may be computed by first transforming to the QMF domain via:
[0106]
number
number
[0107] Next, we implement the SPAR filter bank, for example as follows:
[0108]
number
number
[0109] If no residual signal is transmitted, the signal can be reconstructed as follows:
[0110]
number
number
number
[0111]
number
number
[0112] Finally, the time-domain decoded signal can be calculated via QMF synthesis, for example as follows:
[0113]
number
number
[0114] Examples of how to handle the representation of multi-channel audio signals An example method 300 for processing (e.g., SPAR decoding) a representation of a multi-channel audio signal (e.g., a First Order Ambisonics (FOA) or Higher Order Ambisonics (HOA) audio signal) using techniques according to the present disclosure is shown in the flow chart of Figure 3. Method 300 includes steps S310 to S330. These steps may be repeated, for example, for each frame of the multi-channel audio signal.
[0115] According to the above, it is understood that the representation includes metadata related to the first channel (e.g., a waveform-coded version of the first channel, corresponding to signal x1) and the second channel (e.g., corresponding to signal x2). Possibly, the representation relates to multiple second channels, and the discussion below can be easily extended to such a case. The metadata includes, for each of multiple first bands of the first filter bank, respective prediction parameters (e.g., SPAR parameters, or gain parameters) for making predictions for the second channel based on the first channel in the first band. The first filter bank can be, for example, a SPAR filter bank including FIR band filters and using MDFT. The representation can further include a residual for the second channel.
[0116] In step S310, a second filter bank having a plurality of second bands is applied to the first channel, and for each of the second bands, a banded version of the first channel in the second band is obtained. It is understood that the second filter bank is different from the first filter bank used in the process generating the representation (e.g., in the encoder). The second filter bank can be, for example, a QMF filter bank.
[0117] In step S320, for each of the second bands, a respective time domain filter is generated based on the prediction parameters and a first filter of the first filter bank. The first filter corresponds to the first band. In one example, the time domain filter may be a multi-tap FIR filter.
[0118] In step S330, within the second band, a prediction for the second channel is generated based on the banded version of the first channel and the time domain filter. For example, this may include, for each of the second bands, generating a prediction for the second channel in the second band based on a filtered version of the first channel in the second band, where the filtered version of the first channel is obtained by applying the respective time domain filter in the second band to the banded version of the first channel in the second band.
[0119] The generation of the time-domain filter for a given second band in step S320 may be based on a prototype filter, which may be an asymmetric prototype filter. In particular, step S320 may include generating a plurality of adapted (or basic) first filters based on the respective first filters and the prototype filter (e.g., the asymmetric prototype filter).
[0120] Said generating of a time domain filter for a given second band may further comprise taking a weighted sum of the adapted first filters. To this end, the adapted first filters may be weighted with prediction coefficients (e.g., prediction parameters, SPAR parameters, gain parameters) for the respective first band. Here, a processing stride for each tap of the adapted first filters may be less than or equal to the number of second bands.
[0121] Step S320 of the method 300 may be said to relate to a filter transformation step, for example from a (MDFT) SPAR FIR filter to a QMF domain SPAR FIR filter, which may correspond to application of the QMF transformation operator of equation (8). Details of the filter transformation are now described.
[0122] Filter Transformation For example, the implementation of joint QMF domain SPAR decoding and processing shown in FIG. 2 or FIG. 3 involves a transformation of the MDFT SPAR filters used for encoding into the QMF domain (e.g., a filter transformation operator H=QMF in Eq. (8)). C {h}), or in general a transformation of the filters of the first filter bank domain into a second, different filter bank domain is required, e.g. by FIR filtering along time in the bands of the second filter bank domain.
[0123] An example of a filter transformation, e.g., from a (MDFT) SPAR FIR filter to a QMF domain SPAR FIR filter, is shown diagrammatically in FIG. 4. In this example, a SPAR FIR filter 410 is transformed from FIR to QMF-FIR in block 430 to generate a QMF domain SPAR FIR filter. Block 430 may receive as further input a set of transformation parameters 420. These transformation parameters 420 may include, for example, an indication of a maximum number of QMF domain taps and / or an indication of a minimum relative coefficient magnitude. Based on the transformation parameters 420, the filter transformation in block 430 may include, for example, a filter truncation, as described in more detail below.
[0124] In general, in the filter transformation for each SPAR filter, a set of complex FIR filters is derived, one for each QMF band. For example, there may be a total of 60 QMF bands. When applied in the QMF domain, this approximates the operation of FIR filtering with one SPAR filter and subsequent QMF analysis. To mimic the parameter modification (e.g., prediction) in all SPAR bands and filter bank synthesis, (e.g., 60) complex FIR filters, one for each QMF band, can be derived by summing (e.g., by filter bank synthesis) over (e.g., 12) parameter-modified complex FIR filters per QMF band.
[0125] For the wideband SPAR FIR to QMF domain FIR conversion, a new prototype filter is first derived based on the QMF prototype, processing stride, QMF analysis synthesis delay, and a minimum square error objective based on the number of QMF bands. This new prototype typically has a length, e.g., three times the processing stride, and is generally asymmetric. Then, a QMF domain complex FIR filter can be computed by taking one SPAR FIR filter as input and performing a QMF analysis using this new prototype filter.
[0126] In general, a new prototype filter for the filter conversion (filter converter prototype) may be derived based on the prototypes of the second filter bank.
[0127] [Prerequisites and Notation] As mentioned above, the prototype filter p of the QMF synthesis filter bank may be assumed to have support on {0,1,...,N-1}. Furthermore, let S be the time stride in samples and L be the number of subbands of the QMF filter bank (e.g., typically 60). For the modeling used here (e.g., relying on a zero-delay filter bank), the non-causal analytical prototype filter is defined, for example, by:
[0128]
number
[0129] [Filter converter prototype calculation] This section generally relates to generating a filter converter prototype q (a prototype filter for filter conversion) based on a second filter bank prototype filter p. As described in more detail below, the filter converter prototype q may be generated based on the second filter bank prototype filter p by solving one or more least squares problems, such as a least squares problem involving a matrix representation derived from the second filter bank prototype filter p.
[0130] For example, to arrive at a filter converter prototype filter q that is supported on {-F,-F+1,...,RF-1}, the following steps may be taken: Thus, R is the length of the filter converter prototype and F is an offset parameter, both in samples.
[0131] First, the cross-correlation can be defined, for example, as follows:
number
[0132] It can be seen that the infinite sum is in fact finite (for l∈{D-N+1,…,D}) and that the support of p2 is finite.
[0133] Second, a finite set of matrices V of size S × R (k) ,k=-K,...,K can be defined by its elements.
[0134]
number
number
number
[0135] Finally, the entries of the filter converter prototype filter q can be found, for example, as the entries of a vector q of size R×1 that solves a least squares problem.
[0136]
number
[0137]
number
[0138] Next, the least squares problem is Vq≒r, and M=V T V is a normal equation Mq=V T r, where V T denotes the matrix transpose of V. For better numerical stability, we can add small positive numbers to all diagonal entries of M before solving this system of equations. The entries of the solution vector q can be used for the entries of filter q on {(-F,-F+1,…,RF-1}.
[0139] A design example for q where L=S=60, R=180, F=120, D=299, and N=600 is shown in the diagram of FIG.
[0140] [Filter conversion using the filter converter prototype] Given a filter converter prototype q, the filter h b Conversion of H b =QMF C {hb} may be defined, for example, by:
[0141]
number
[0142] In general, a number of matched first filters
number
[0143] Note that for k<0
number
[0144] [Conventional filter transformation] An example of a prior art technique for filter transformations that are not applicable to the IVAS SPAR framework with integrated QMF processing is described in U.S. Patent No. 8,315,859 (hereinafter referred to as the reference). In particular, the filter transformations in this document are not applicable to the above-mentioned SPAR FIR to QMF domain SPAR FIR transformations that are particularly relevant to low-latency SPAR processing.
[0145] The filter transformation described in the above document is limited to the following cases in the above document. ● Symmetric QMF prototype filter A QMF filter bank with as many subbands as the time stride in samples, i.e. L=S
[0146] On the other hand, the QMF filter bank design associated with the low delay processing used in the IVAS SPAR can have the following: Asymmetric QMF prototype filter Oversampling allows the number of subbands, L, to be greater than the time stride, S, in samples.
[0147] In contrast to the above cited documents, the filter transformation according to the present disclosure allows in particular for asymmetric QMF prototype filters and / or filter banks that can have oversampling where the number of subbands is larger than the time stride in samples.
[0148] Transformed Filter Truncation The filter transformation (e.g., in step S320 of method 300 or shown in FIG. 4) may further include a step of truncating the filter length of the time domain filter (e.g., QMF domain SPAR filter truncation). In particular, in an efficient implementation of QMF domain SPAR filter bank processing, the filter order (e.g., filter length N along a time slot per one QMF frequency channel l) may be truncated by setting filter taps that have a small effect (e.g., perceptual effect) on the filtering to zero. l It may be advantageous to reduce ) as much as possible. This, if done correctly, may provide computational efficiency for decoding without affecting perception. One way to do this is described below.
[0149] First, a magnitude threshold can be derived for each of all SPAR band filters in the QMF domain as follows:
[0150] For all k and l=0,1,…,L−1 and a reasonable threshold level L of, say, −70 dB. thr In contrast,
number
[0151] Then, for every QMF frequency channel l, the maximum time slot index k is calculated as follows: max may be required. For b = 0,1,…,B-1,
number
[0152] Next, the filter length N in QMF frequency channel l l is N l =k max may be selected as:
[0153] In other words, the truncation may proceed as follows: Define a relative size threshold (for example,
number
number
[0154] Note that in general, one can identify groups of QMF band-adjacent FIR filters that have the same filter length, for example, multiple FIR filters in the highest frequency QMF band have the same truncated filter length, which can simplify implementation.
[0155] In general, in the terminology of method 300, the filter length of a given time domain filter after truncation may depend on each second band of the time domain filter (eg, each QMF band l).
[0156] Further, according to the above, generating a time domain filter for a given second band (e.g., a QMF band) may include generating a respective basic (or adapted) time domain filter (e.g., a transformed FIR filter) in the given second band for each of the first filters (e.g., for each SPAR filter), and generating a time domain filter in the given second band based on the basic time domain filter in the given second band and the prediction parameters (e.g., as a weighted sum as further described above). Then, the truncation of the time domain filter for the given second band may be based on thresholds for the filter coefficients of the basic time domain filter. Each of these thresholds may correspond to a respective one of the first filters. Furthermore, the threshold for the basic time domain filter for a given first filter may be derived from the maximum magnitude of the basic time domain filter in the multiple second bands. For example, the threshold for a given first filter may be derived from the maximum coefficient magnitude for the basic time domain filter for the first filter scaled by a relative threshold (e.g., -20 dB).
[0157] The step of truncating the time domain filter may further include determining, for each first band (e.g., for each SPAR filter), a maximum magnitude (of filter coefficients) of a corresponding basic time domain filter in a plurality of second bands (e.g., in a plurality of QMF bands). Then, for each first band, a minimum truncated filter length (i.e., one minimum truncated filter length for each first filter and second band) may be determined for a corresponding basic time domain filter in a plurality of second bands based on a threshold derived from the maximum magnitude. Finally, for each second band, a filter length of a time domain filter in the second band may be determined based on the minimum truncated filter length of a basic time domain filter in the second band (i.e., one for each first filter). The filter length in the second band may be the maximum of the minimum filter lengths.
[0158] For example, there may be B first filters (e.g., B = 12 SPAR filters) in a first filter bank and L second bands (e.g., L = 60 QMF bands) in a second filter bank. Then, for the first filters b ∈ 0, ..., B-1, a threshold thr b can be derived from all the coefficients of the L basic time-domain filters generated for the first filter b by taking the maximum coefficient value and applying it to a relative threshold thr rel Then, for a given second frequency band l ∈ 0, ..., L-1, we define such thresholds thr b ,b∈0,…,B-1. These thresholds thr b The step of applying to each of the basic time domain filters in the second band l is performed by applying B different minimum filter lengths len l,b, b∈0,…,B−1. Beyond this minimum filter length, the coefficients of the basic time-domain filters in the second band l exceed the respective thresholds thr b Next, for the second band l, the filter length for truncation is set to N l is the minimum filter length len in the second band l l,b can be determined as the maximum of
number
[0159] Figure 8 shows the results for different relative thresholds thr rel 1 shows examples of FIR filter lengths after truncation for an FIR filter transformed across QMF bands for 1. The top graph (diamond symbols) corresponds to a relative threshold of -80 dB, the middle graph (square symbols) corresponds to a relative threshold of -60 dB, and the bottom graph (cross symbols) corresponds to a relative threshold of -40 dB, where the smaller the difference or scaling factor between the maximum coefficient magnitude and the threshold, the shorter the filter length, and vice versa.
[0160] Filter conversion to a single tap filter There may be situations where the computational complexity of multi-tap FIR filtering in the QMF domain is excessively high. To address this issue, two alternative, low-complexity, SPAR parameter processing methods are described below, for example for QMF-tuned SPAR filter banks. It will be understood that these methods are generally applied to the first and second filter banks, but are not limited to SPAR and QMF filter banks.
[0161] In this regard, Fig. 12 shows examples of SPAR filter frequency responses (1 ms latency, 12 bands) for possible designs with bandwidths below 400 Hz at low center frequencies (top panel) and for possible designs with a minimum bandwidth of 400 Hz and band boundaries aligned to the QMF band boundaries (bottom panel). Fig. 13 further shows an example of the superposition of (QMF-adapted) SPAR encoder filter bands (dashed lines, 12 bands) and QMF decoder filter bands (solid lines, 60 bands). The QMF-adjusted SPAR filter bank is shown in the bottom panel of Fig. 12 and is shown in Fig. 13 as a dashed curve (e.g., the SPAR filter bandwidth matches the QMF bandwidth and the SPAR filter bandwidth is equal to or greater than the QMF bandwidth).
[0162] The idea is to approximate the SPAR filterbank band filters by linear phase filters such that the QMF domain multi-tap filters shown in Figures 9A-9D can be expressed as real, non-negative single-tap filters (i.e., only the first column is non-zero). c = const = 0, the sum in equation (17) disappears, leaving only tap n = 0. For reference, Figures 10A, 10B, 10C, and 10D include diagrams showing examples of the first 400 samples of the original SPAR filter impulse response (solid line) and an approximation using a QMF filter (dashed line) according to an embodiment of the present disclosure.
[0163] When approximated by a real single-tap filter, the overall delay of system 200 (see FIG. 2) reduces to Delay1+Delay2 (as compared to Delay1+Delay1+Delay2).
[0164] Although, in some implementations of the present disclosure, the time domain filter may be a single tap FIR filter, it is understood that this may require processing steps to generate the single tap FIR filter.
[0165] If the single-tap filter coefficients are arranged in the columns of a matrix M of size [C × B], then an example of a single-tap SPAR filter in the QMF domain (magnitude frequency response in the QMF bands) can be visualized as a column per SPAR band filter, as shown in Figure 14.
[0166] [Zeroth-order QMF domain SPAR filter calculation] The real coefficients of the single tap filter can be calculated using the (modified) Fourier transform as follows:
[0167]
number
number
[0168] Note that the overall SPAR filter bank response in equation (9) reduces to:
[0169]
number
[0170] To reduce the computational complexity of the filter bank response using gain parameters, e.g., as in equation (10),
number
[0171] For every QMF band l and every SPAR band b,
number
[0172] Further, in some embodiments, generating a time domain filter for a given second band may include steps S1510 and S1520 of the method 1500 shown in Figure 15. In step S1510, a first band of the plurality of first bands having the highest energy within the second band is determined. Then, in step S1520, a time domain filter is generated based on a linear phase approximation of the first filter corresponding to the determined first band and corresponding prediction coefficients for the determined first band.
[0173] Further simplification and complexity reduction can be achieved for QMF frequency bands where only one SPAR filter contributes significantly, e.g., for the lowest seven QMF frequency bands. This case is shown in the example of Figure 13. Let us consider such a matching SPAR band as b for QMF band l. l If we define it as follows:
[0174]
number
[0175] Further, in some embodiments, generating a time domain filter for a given second band may include steps S1610 and S1620 of the method 1600 shown in FIG. 16. In step S1610, a set of first bands of a plurality of first bands having the highest energy in the second band is determined. Then, in step S1620, a time domain filter is generated based on a weighted sum of linear phase approximations of the first filters corresponding to the determined set of first bands, where the weights in the weighted sum depend on the prediction coefficients corresponding to the determined set of first bands in the second band and the normalized magnitude or energy of each of the first bands of the determined set of first bands.
[0176] In one embodiment, the SPAR filter responses for some QMF bands may be calculated using equation (32+x), and for the remaining QMF bands equation (33+x) may be used.
[0177] Finally, Figures 17 and 18 show examples of SNR for decoded binaural signals for IVAS SPAR with and without QMF domain reconstruction, where Figure 17 relates to the use of a modified SPAR filter bank adapted to the QMF bands and a brick-wall application of SPAR parameters in the QMF domain, and Figure 18 relates to the original SPAR filter bank and multi-tap SPAR filtering in the QMF domain according to an embodiment of the present disclosure.
[0178] Direct Filter Conversion An alternative transformation method, which has higher computational complexity, is to transform a given length N l For a given SPAR frequency band b with
number
[0179]
number
[0180] And y=Ψ F Let {x} define the combined effect of QMF analysis, filtering in the QMF domain, and QMF synthesis. JPEG2024544730000066.jpg537 The design goal is F=H b Ψ F But the SPAR filter hb The goal is to approximate the filtering of the input signal x p Consider (k) = δ(kp). x p (k) may be said to represent the fundamental signal with a single non-zero sample (value 1) at each sample location. For each l=0,1,...,L-1 and k=0,1,...,N l For -1, it is a single tap filter.
number
number
number
number
[0181]
number
[0182] The desired result is that for all p=0,1,…,S-1, h b (n-D3-p). If this holds, then Ψ F By the shift invariance of in steps of S samples, it is extended to be true for all p, and therefore
number
[0183]
number
[0184] next,
number
[0185] Therefore, a given first filter h (with appropriate delay) is generated by the first and second fundamental signals. b can be approximated, and then the first filter fitted in the second band l
number
[0186] Apparatus for implementing the method according to the present disclosure Finally, the present disclosure also relates to an apparatus (e.g., a computer-implemented apparatus) for performing the methods and techniques described throughout the present disclosure. FIG. 19 shows an example of such an apparatus 1900. In particular, the apparatus 1900 comprises a processor 1910 and a memory 1920 coupled to the processor 1910. The memory 1920 may store instructions for the processor 1910. The processor 1910 may also receive appropriate input data (e.g., audio input), in particular, depending on the embodiment and / or implementation. The processor 1910 may be adapted to perform the methods / techniques described throughout the present disclosure (e.g., the method 300 of FIG. 3) and generate corresponding output data 1940 (e.g., a reconstructed multi-channel audio signal), depending on the embodiment and / or implementation.
[0187] Disclosure Summary In summary, the present disclosure relates to the following: Filter bank processing of a first filter bank (e.g., a SPAR filter bank) in the domain of another second filter bank (e.g., a QMF filter bank), taking advantage of each of the filter banks in terms of time and frequency resolution and processing stride. Efficient and low-latency SPAR FIR filter transformation to the QMF domain, specifically using asymmetric QMF prototype filters QMF band-dependent QMFFIR length truncation to reduce complexity, if required QMF domain FIR length truncation based on maximum threshold for individual filters, if required SPAR filter bank Combining filtering and signal manipulation
[0188] Furthermore, the technology disclosed herein may have the following features and advantages: No need to adapt SPAR filters to QMF banding Saves computational complexity by avoiding MDFT-based filter bank processing before QMF analysis
[0189] interpretation Aspects of the systems described herein may be implemented in a suitable computer-based sound processing network environment (e.g., a server or cloud environment) for processing digital or digitized audio files. Part of an adaptive audio system may include one or more networks with any desired number of individual machines, including one or more routers (not shown) that function to buffer and route data transmitted between computers. Such networks may be built based on a variety of different network protocols and may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.
[0190] One or more of the components, blocks, processes, or other functional components may be implemented via a computer program that controls the execution of a processor-based computing device of the system. It should also be noted that the various functions disclosed herein may be described in terms of their behavior, register transfers, logical components, and / or other characteristics using any number of combinations of hardware, firmware, and / or as data and / or instructions embodied in various machine-readable or computer-readable media. The computer-readable media on which such formatted data and / or instructions may be embodied include, but are not limited to, various forms of physical (non-transitory) non-volatile storage media, such as optical, magnetic, or semiconductor storage media.
[0191] In particular, it should be understood that the embodiments may include hardware, software, and electronic components or modules, which for purposes of discussion may be illustrated and described as if the majority of the components were implemented solely in hardware. However, based on reading this detailed description, one skilled in the art will recognize that in at least one embodiment, the electronic-based aspects may be implemented in software (e.g., stored in a non-transitory computer-readable medium) executable by one or more electronic processors, such as a microprocessor and / or an application-specific integrated circuit ("ASIC"). Thus, it should be noted that multiple hardware and software-based devices, as well as multiple different constituent components, may be utilized to implement the embodiments. For example, the systems, encoders, decoders, or blocks described with respect to Figures 1 and 2 or Figure 19 above may include one or more electronic processors, one or more computer-readable media modules, one or more input / output interfaces, and various connections (e.g., a system bus) connecting the various constituent components.
[0192] While one or more implementations have been described by way of example and in terms of specific embodiments, it is to be understood that the one or more implementations are not limited to the disclosed embodiments. On the contrary, it is intended to cover various modifications and similar arrangements as would be apparent to those skilled in the art. Therefore, the scope of the appended claims should be accorded the broadest interpretation so as to encompass all such modifications and similar arrangements.
[0193] It should also be understood that the phraseology and terminology used herein is for purposes of description and should not be regarded as limiting. The use of "including," "comprising," "including," or "having" and variations thereof is meant to encompass the items listed thereafter and equivalents thereof as well as additional items. Unless otherwise noted, the terms "mounted," "connected," "supported," and "coupled" and variations thereof are used broadly to encompass both direct and indirect mounting, connecting, supporting, and coupling.
[0194] Example enumerated embodiments Various aspects and implementations of the present disclosure can also be seen from the following enumerated example embodiments (EEE), which are not claims.
[0195] EEE1. 1. A method for processing a representation of a multi-channel audio signal, the representation comprising metadata relating to a first channel and a second channel, the metadata comprising, for each of a plurality of first bands of a first filter bank, respective prediction parameters for performing a prediction for the second channel based on the first channel in the first band; applying a second filter bank having a plurality of second bands to the first channel to obtain, for each of the second bands, a banded version of the first channel in that second band, the second filter bank being different from the first filter bank; generating, for each of the second bands, a respective time-domain filter based on the prediction parameters and a first filter of the first filter bank, the first filter corresponding to the first band; generating a prediction for the second channel based on the banded version of the first channel in the second band and the time domain filter; A method comprising:
[0196] EEE2. generating the prediction for the second channel includes, for each of the second bands, generating a prediction for the second channel in the second band based on a filtered version of the first channel in the second band, the filtered version of the first channel being obtained by applying the respective time domain filter in the second band to the banded version of the first channel in the second band; EEE1 method.
[0197] EEE3. the multi-channel audio signal is a First Order Ambisonics (FOA) or Higher Order Ambisonics (HOA) audio signal; A method relating to EEE1 or EEE2.
[0198] EEE4. The prediction parameters are SPAR parameters. A method relating to any one of EEE1 to EEE3.
[0199] EEE5. The first filter bank is a SPAR filter bank with FIR band filters and uses MDFT. A method relating to any one of EEE1 to EEE4.
[0200] EEE6. the second filter bank is a QMF filter bank; A method relating to any one of EEE1 to EEE5.
[0201] EEE7. The time domain filter is a multi-tap FIR filter. A method relating to any one of EEE1 to EEE6.
[0202] EEE8. The step of generating the time domain filter for a given second band comprises: generating a plurality of adapted first filters based on each of the first filters and the prototype filter; Including, A method relating to any one of EEE1 to EEE7.
[0203] EEE9. For a given second band l, the first filter h for a given first band b b The adapted first filter
number
number
[0204] EEE10. generating a prototype filter for the filter transformation based on a prototype filter of the second filter bank; The method according to EEE8 or EEE9, further comprising:
[0205] EEE11. a prototype filter for the filter transformation is generated based on the prototype filter of the second filter bank by solving a least squares problem. A method relating to EEE10.
[0206] EEE12. The step of generating a prototype filter for the filter transformation comprises: Based on the prototype filter p of the second filter bank, a non-causal prototype filter p A generating a The non-causal prototype filter p A generating a cross-correlation p2 between the prototype filter p of the second filter bank and The dimension is S × R, and there is a nonzero element v if and only if the indices n, m are integer multiples of S. n,m A set of matrices V for some integer K, (k) , k=-K, ..., K, where R is the length of a prototype filter for the filter transformation; V (k) solving a set of least squares problems for q, where q is a vector of dimension R×1 containing filter coefficients of a prototype filter q for said filter transformation; include, A method according to EEE10 or EEE11 when dependent on claim 9.
[0207] EEE13. The step of generating the time domain filter for a given second band comprises: taking a weighted sum of the adapted first filters, the adapted first filters being weighted with a prediction coefficient for each of the first bands; Further comprising: A method relating to any one of EEE8 to EEE12.
[0208] EEE14. the prototype filter for the filter transformation is an asymmetric prototype filter; A method relating to any one of EEE8 to EEE13.
[0209] EEE15. The processing stride for each tap is less than or equal to the number of second bands. A method relating to any one of EEE8 to EEE14.
[0210] EEE16. The step of generating the time domain filter for a given second band comprises: approximating a given first filter by first and second fundamental signals, The first base signal may be obtained as a result of applying the second filter bank, a basic real single-tap filter, and a synthesis filter bank of the second filter bank to a base signal having a single non-zero sample at each sample position, the basic real single-tap filter being a filter for each single one of the second bands having a single non-zero filter coefficient at each tap position; the second fundamental signal may be obtained as a result of applying the second filter bank, a basic imaginary single tap filter, and the synthesis filter bank of the second filter bank to the fundamental signal, the basic imaginary single tap filter being a filter for each single one of the second bands having a single non-zero filter coefficient at each tap position; generating a time domain filter adapted for the first filter in the second band based on coefficients of first and second fundamental signals in the approximation; Including, A method relating to any one of EEE1 to EEE7.
[0211] EEE17. The step of generating the time domain filter for a given second band comprises: The second filter bank, a real single tap filter
number
number
number
number
[0212] EEE18. truncating the filter length of the time domain filter. The method according to any one of EEE1 to EEE17, further comprising:
[0213] EEE19. the filter length of a given time domain filter after truncation depends on the respective second band of the time domain filter. Method according to EEE18.
[0214] EEE20. generating the time domain filter for the given second band includes generating, for each of the first filters, a respective basic time domain filter in the given second band; and generating the time domain filter in the given second band based on the basic time domain filter in the given second band and the prediction parameters; the truncation of the time domain filter for the given second band is based on thresholds for filter coefficients of the basic time domain filters, each threshold corresponding to a respective one of the first filters, and the threshold for the basic time domain filter for a given first filter is derived from a maximum magnitude of that basic time domain filter in the plurality of second bands. A method according to EEE18 or EEE19.
[0215] EEE21. determining, for each first band, a maximum magnitude of the corresponding fundamental time domain filter in the plurality of second bands; determining, for each first band, a minimum truncated filter length for the corresponding fundamental time domain filter in the plurality of second bands based on a threshold derived from the maximum magnitude; for each second band, determining a filter length of the time domain filter in that second band based on a minimum truncated filter length for the base time domain filter in that second band; The method according to EEE20, further comprising:
[0216] EEE22. The time domain filter is a single tap FIR filter. A method relating to any one of EEE1 to EEE6.
[0217] EEE23. The step of generating the time domain filter for a given second band comprises: determining a first band from the plurality of first bands that has a highest energy within the second band; generating the time domain filter based on a linear phase approximation of the first filter corresponding to the determined first band and corresponding prediction coefficients for the determined first band; Including, A method relating to EEE22.
[0218] EEE24. The step of generating the time domain filter for a given second band comprises: determining a set of first bands from the plurality of first bands that have a highest energy within the second band; generating the time domain filter based on a weighted sum of linear phase approximations of the first filters corresponding to the determined set of first bands, where weights in the weighted sum depend on the corresponding prediction coefficients for the determined set of first bands and on a normalized magnitude or energy of each of the first bands of the determined set of first bands in the second band; Including, A method relating to EEE22.
[0219] EEE25. 1. A method for generating a representation of a multi-channel audio signal, the representation comprising metadata relating to a first channel and a second channel, the metadata comprising, for each of a plurality of first bands of a first filter bank, respective prediction parameters for performing a prediction for the second channel based on the first channel in the first band; generating a prediction for the second channel based on a first filter of the first filter bank and the prediction parameters, the prediction for the second channel being represented by a time domain signal; generating a residual for the second channel by subtracting, in a time domain, the prediction of the second channel from the second channel; A method comprising:
[0220] EEE26. the representation of the multi-channel audio signal further comprises the residual of the second channel. A method relating to EEE25.
[0221] EEE27. a processor; and a memory coupled to the processor for storing instructions for the processor, the processor adapted to perform a method according to any one of EEE1 to EEE26. Device.
[0222] EEE28. A program comprising instructions which, when executed by a processor, cause the processor to perform a method according to any one of EEE1 to EEE26.
[0223] EEE29. A computer-readable storage medium storing a program related to EEE28.
Claims
1. 1. A method for processing a representation of a multi-channel audio signal, the representation comprising metadata relating to a first channel and a second channel, the metadata comprising, for each of a plurality of first bands of a first filter bank, respective prediction parameters for making a prediction for the second channel based on the first channel in the first band, applying a second filter bank having a plurality of second bands to the first channel to obtain, for each of the second bands, a banded version of the first channel in that second band, the second filter bank being different from the first filter bank; generating, for each of the second bands, a respective time-domain filter based on the prediction parameters and a first filter of the first filter bank, the first filter corresponding to the first band; generating a prediction for the second channel based on the banded version of the first channel in the second band and the time domain filter; A method comprising:
2. The step of generating the prediction for the second channel may include, for each of the second bands: generating a prediction for the second channel in the second band based on a filtered version of the first channel in the second band, the filtered version of the first channel being obtained by applying the respective time domain filter in the second band to the banded version of the first channel in the second band; Including, The method of claim 1.
3. The multi-channel audio signal is a First Order Ambisonics (FOA) or Higher Order Ambisonics (HOA) audio signal. The method according to claim 1 or 2.
4. The prediction parameters are SPAR parameters.
13. A method according to any one of the preceding claims.
5. the first filter bank is a SPAR filter bank comprising FIR band filters and uses MDFT; 13. A method according to any one of the preceding claims.
6. the second filter bank is a QMF filter bank; 13. A method according to any one of the preceding claims.
7. The time domain filter is a multi-tap FIR filter.
13. A method according to any one of the preceding claims.
8. The step of generating the time domain filter for a given second band comprises: generating a plurality of adapted first filters based on the respective first filters and a prototype filter for the filter transformation; Including, 13. A method according to any one of the preceding claims.
9. For a given second band l, a first filter h for a given first band b b The adapted first filter [0010] is calculated as follows: [0025] q is a prototype filter for the filter transform, S is the stride of the second filter bank, L is the number of second bands, and the summation over n is over the support of prototype filter q for the filter transform. The method according to claim 8.
10. generating a prototype filter for the filter transformation based on a prototype filter of the second filter bank; 10. The method of claim 8 or 9, further comprising:
11. a prototype filter for the filter transformation is generated based on the prototype filters of the second filter bank by solving a least squares problem. The method of claim 10.
12. The step of generating a prototype filter for the filter transformation comprises: Based on the prototype filter p of the second filter bank, a non-causal prototype filter p A generating The non-causal prototype filter p A and the prototype filter p of the second filter bank. 2 generating The dimension is S×R, and there is a nonzero element v only for indices n, m where n−m is an integer multiple of S. n,m A set of matrices V for some integer K, (k) , k=−K, ..., K, where R is the length of a prototype filter for the filter transformation; V (k) solving a set of least squares problems for q, where q is a vector of dimension R×1 containing filter coefficients of a prototype filter q for said filter transformation; include, A method according to claim 10 or 11 when dependent on claim 9.
13. The step of generating the time domain filter for a given second band comprises: taking a weighted sum of the adapted first filters, the adapted first filters being weighted with a prediction coefficient for each of the first bands; Further comprising:
13. The method according to any one of claims 8 to 12.
14. the prototype filter for the filter transformation is an asymmetric prototype filter; 14. The method according to any one of claims 8 to 13.
15. The processing stride for each tap is less than or equal to the number of second bands.
15. The method according to any one of claims 8 to 14.
16. The step of generating the time domain filter for a given second band comprises: approximating a given first filter by first and second fundamental signals, The first fundamental signal may be obtained as a result of applying the second filter bank, a basic real single-tap filter, and a synthesis filter bank of the second filter bank to a fundamental signal having a single non-zero sample at each sample position, the basic real single-tap filter being a filter for each single one of the second bands having a single non-zero filter coefficient at each tap position; the second fundamental signal may be obtained as a result of applying the second filter bank, a basic imaginary single tap filter, and the synthesis filter bank of the second filter bank to the fundamental signal, the basic imaginary single tap filter being a filter for each single one of the second bands having a single non-zero filter coefficient at each tap position; generating a time domain filter adapted for the first filter in the second band based on coefficients of first and second fundamental signals in the approximation; Including, 8. The method according to any one of claims 1 to 7.
17. The step of generating the time domain filter for a given second band comprises: The second filter bank, a real single tap filter [0030] and the synthesis filter bank of the second filter bank is applied to the signal x p (k) = δ(k-p) applied to the result u p,l,k where l denotes a given second band, p denotes a given sample position, and k denotes a filter tap position; The second filter bank, an imaginary single tap filter [0045] and applying the synthesis filter bank of the second filter bank to the signal x p (k) = δ(k-p) applied to the result v p,l,k and Given a delay D 3 For l and b l determining a least squares solution for [0050] Here, h b is the first filter for the first band b, L is the number of second bands, and N l is a predetermined number of filter taps for the second band l; The first filter h in the second band l b The adapted first filter [006] of 【number】 and Including, 8. The method according to any one of claims 1 to 7.
18. truncating the filter length of the time domain filter.
2. The method of any one of the preceding claims, further comprising:
19. the filter length of a given time domain filter after truncation depends on the respective second band of the time domain filter.
20. The method of claim 18.
20. generating the time domain filter for the given second band includes: generating, for each of the first filters, a respective adapted time domain filter in the given second band; and generating the time domain filter in the given second band based on the adapted time domain filter in the given second band and the prediction parameters; the truncation of the time domain filter for the given second band is based on thresholds for filter coefficients of the adapted time domain filter, each threshold corresponding to a respective one of the first filters, and the threshold for the adapted time domain filter for a given first filter is derived from a maximum magnitude of the adapted time domain filter in the plurality of second bands.
20. The method of claim 18 or 19.
21. determining, for each first band, a maximum magnitude of the corresponding adapted time domain filter in the plurality of second bands; determining, for each first band, a minimum truncated filter length for the corresponding adapted time domain filter in the plurality of second bands based on a threshold derived from the maximum magnitude; for each second band, determining a filter length of the time domain filter in that second band based on a minimum truncated filter length for the fitted time domain filter in that second band; 21. The method of claim 20, further comprising:
22. The time domain filter is a single tap FIR filter.
7. The method according to any one of claims 1 to 6.
23. The step of generating the time domain filter for a given second band comprises: determining a first band from the plurality of first bands that has a highest energy within the second band; generating the time domain filter based on a linear phase approximation of the first filter corresponding to the determined first band and corresponding prediction coefficients for the determined first band; Including, 23. The method of claim 22.
24. The step of generating the time domain filter for a given second band comprises: determining a set of first bands from the plurality of first bands that have a highest energy within the second band; generating the time domain filter based on a weighted sum of linear phase approximations of the first filters corresponding to the determined set of first bands, where weights in the weighted sum depend on the corresponding prediction coefficients for the determined set of first bands and on a normalized magnitude or energy of each of the first bands of the determined set of first bands in the second band; Including, 23. The method of claim 22.
25. 1. A method for generating a representation of a multi-channel audio signal, the representation comprising metadata relating to a first channel and a second channel, the metadata comprising, for each of a plurality of first bands of a first filter bank, respective prediction parameters for making a prediction for the second channel based on the first channel in the first band; generating a prediction for the second channel based on a first filter of the first filter bank and the prediction parameters, the prediction for the second channel being represented by a time domain signal; generating a residual for the second channel by subtracting, in the time domain, the prediction of the second channel from the second channel; A method comprising:
26. the representation of the multi-channel audio signal further comprises the residual of the second channel.
26. The method of claim 25.
27. A method for implementing a method according to any one of claims 1 to 26, comprising: a processor; and a memory coupled to said processor for storing instructions for said processor, said processor being adapted to implement a method according to any one of claims 1 to 26, Device.
28. A program comprising instructions which, when executed by a processor, cause the processor to perform a method according to any one of claims 1 to 26.
29. A computer readable storage medium storing the program according to claim 28.
