Comb filter artifact removal for multichannel downmixing using adaptive phase alignment

Abstract

An audio signal processing decoder comprising at least one frequency band (36), and the audio signal processing decoder is used to process an input audio signal (37) having a plurality of input channels (38) within the at least one frequency band (36), wherein the decoder (2) is configured to analyze the input audio signal (37), identify inter-channel dependencies between the input channels (38); is configured to calibrate the phases of the input channels (38) based on the identified inter-channel dependencies (39), wherein the more the phases of the input channels (38) are aligned with each other, the higher their inter-channel dependencies (39) are; and is configured to downmix the calibrated input audio signal to an output audio signal (40), the output audio signal (40) having a smaller number of output channels (41) than the number of the input channels (38).

Description

[0001] This application is a divisional application of the applicant Fraunhofer Gesellschaft, with the application date of July 18, 2014, the application number of 201480041810.X, and the invention name of "Artifact Elimination of Comb Filters for Multi-channel Downmixing Using Adaptive Phase Calibration". Technical Field

[0002] The present invention relates to audio signal processing, and in particular to comb filter artifact removal for multi-channel downmixing using adaptive phase alignment. Background Art

[0003] To date, several multi-channel sound formats have been used, ranging from the typical 5.1 surround sound of movie soundtracks to more extensive 3D surround sound formats. In some cases, the sound content must be delivered through a smaller number of speakers.

[0004] Furthermore, in recent low bit rate audio decoding methods, such as in J. Breebaart, S. van de Par, A. Kohlrausch, and E. Schuijers, “Parametric coding of stereoaudio,” EURASI PJournal on Applied Signal Processing, vol. 2005, pp. 1305–1322, 2005 and in J. Herre, K. J.Breebaart,C.Faller,S.Disch,H.Purnhagen,J.Koppens,J.Hilpert,J. As described in W.Oomen, K.Linzmeier, and K.S.Chong, "MPEG Surround-The ISO / MPEG standardfor efficient and compatible multichannel audio coding," J.Audio Eng.Soc, vol.56, no.11, pp.932–955, 2008, a higher number of channels are transmitted in the form of a set of downmix signals and spatial auxiliary side information, so that the multichannel signal of the original channel configuration is restored. These use cases have promoted the development of downmixing methods, which well preserve the sound quality.

[0005] The simplest downmixing method is the sum of the channels using a static downmix matrix. However, if the input channels contain coherent sounds but are not aligned in time, the downmix signal may achieve perceived spectral deviations, such as the characteristics of a comb filter.

[0006] In J. Breebaart and C. Faller, "Spatial audio processing: MPEG Surround and other applications". Wiley-Interscience, 2008, a phase calibration method for calibrating two input signals is described, which adjusts the phase of the input channels according to the inter-channel phase difference parameter (ICPD) estimated in the frequency band. This scheme provides similar basic functions as the method mentioned in the paper, but cannot be applied to downmixing of more than two internally correlated channels.

[0007] In WO 2012 / 006770 and PCT / CN2010 / 075107 (Huawei, Faller, Lang, Xu), a phase calibration process for two-to-one channels (stereo to mono) is described. This process is not directly applicable to multi-channel audio.

[0008] In Wu et al, "Parametric Stereo Coding Scheme with a new Downmix Method and whole Band Inter Channel Time / Phase Differences", Proceedings of the ICASSP, 2013, a method for downmixing stereo using full-band intra-channel phase differences is proposed. The phase of the mono signal is configured to be the phase difference between the left channel and all phase differences. At the same time, this method is only applicable to stereo to mono downmixing. More than two internal dependent channels cannot be downmixed in this method. Summary of the invention

[0009] The object of the present invention is to provide an improved concept for audio signal processing. This object of the present invention is achieved by an encoder as claimed in claim 1, a decoder as claimed in claim 12, a system as claimed in claim 13, a method as claimed in claim 14 and a computer program as claimed in claim 15.

[0010] An audio signal processing decoder is provided, comprising at least one frequency band, and the audio signal processing decoder is configured to process an input audio signal having a plurality of input channels within the at least one frequency band. The decoder is configured to calibrate the phases of the input channels according to an inter-channel dependency between the input channels, wherein the more the phases of the input channels are calibrated with respect to each other, the higher their inter-channel dependency is. In addition, the decoder is configured to downmix the calibrated input audio signal to an output audio signal, the output audio signal having a smaller number of output channels than the number of the input channels.

[0011] The basic working principle of the decoder is that in the phase of a specific frequency band, the mutually dependent (coherent) input channels of the input audio signal attract each other, while those input channels of the input audio signal that are independent (incoherent) are not affected. The decoder proposed in this paper aims to improve the downmix quality of the post-equalization method with respect to critical signal cancellation conditions, while providing the same performance under non-critical conditions.

[0012] In addition, at least some functions of the decoder can be transmitted to the external device, such as an encoder, which provides the input audio signal. This can provide the possibility of interacting with the signal, which in the prior art the decoder may generate artifacts. In addition, it is possible to update the downmix processing rules without changing the decoder and ensure a high downmix quality. The transmission of the functions of the decoder will be described in detail below.

[0013] In some embodiments, the decoder is used to analyze the input audio signal in frequency bands in order to identify inter-channel dependencies between the input audio channels. In this case, when the analysis of the input audio signal is done by the decoder itself, the encoder providing the input audio signal can be a standard encoder.

[0014] In some embodiments, the decoder may receive the inter-channel dependencies between input channels from an external device providing the input audio signal, such as an encoder. This version allows for flexible rendering settings in the decoder, but requires more additional data transfer between the encoder and decoder, typically in the bitstream containing the decoder's input signal.

[0015] In some embodiments, the decoder is configured to normalize the energy of the output audio signal according to a determined energy of the input audio signal, wherein the decoder is configured to determine the signal energy of the input audio signal.

[0016] In some embodiments, the decoder is configured to normalize the energy of the output audio signal according to a determined energy of the input audio signal, wherein the decoder is configured to receive the determined energy of the input audio signal from an external device providing the input audio signal, such as an encoder.

[0017] By determining the signal energy of the input audio signal and normalizing the energy of the output audio signal, it is ensured that the energy of the output audio signal is at a comparable level compared to other frequency bands. For example, the normalization may be done in such a way that the energy of the audio output signal of each frequency band is the same as the sum of the energy of the input audio signal of the frequency band multiplied by the square of the corresponding downmix gain.

[0018] In various embodiments, the decoder may include a downmixer for downmixing an input audio signal according to a downmix matrix, wherein the decoder is configured to calculate the downmix matrix so that the phases of the input channels are calibrated according to the identified inter-channel dependencies. Matrix operations are a mathematical tool for efficiently solving multidimensional problems. Therefore, the use of a downmix matrix provides a flexible and simple method of downmixing the input audio signal to an output audio signal, wherein the output audio signal has a number of output channels that is less than the number of input channels of the input audio signal.

[0019] In some embodiments, the decoder comprises a downmixer for downmixing an input audio signal according to a downmix matrix, wherein the decoder is configured to receive the downmix matrix, the downmix matrix being calculated such that the phases of the input channels are calibrated according to the identified inter-channel dependencies from an external device, such as an encoder, providing the input audio signal. Hereby, the processing complexity of the output audio signal in the decoder can be significantly reduced.

[0020] In some specific embodiments, the decoder may be configured to calculate the downmix matrix such that the energy of the output audio signal is normalized according to the determined energy of the input audio signal. In this case, the normalization of the energy of the output audio signal is integrated into the downmix process, making signal processing simple.

[0021] In some embodiments, the decoder may be configured to receive the downmix matrix M calculated such that the energy of the output audio signal is normalized according to the determined energy of the input audio signal from an external device providing the input audio signal, such as an encoder.

[0022] The energy balancing step can be included in the encoding process or performed in the decoder, as it is a simple and well-defined processing step.

[0023] In some embodiments, the decoder may be configured to analyze time intervals of the input audio signal using a window function, wherein the inter-channel dependency is determined for each time frame.

[0024] In some embodiments, the decoder may be configured to receive an analysis of a time interval of the input audio signal using a window function, wherein the inter-channel dependency is determined for each time frame from an external device, such as an encoder, providing the input audio signal.

[0025] The processing may be done in both cases in an overlapping frame-by-frame manner, although other options are also possible, for example using a recursive window to evaluate the relevant parameters. In principle, any window function may be chosen.

[0026] In some embodiments, the decoder is configured to calculate a matrix of covariance values, wherein the covariance values ​​represent the inter-channel dependency of a pair of input audio channels. Calculating a matrix of covariance values ​​is a simple method for obtaining short-term random properties of the frequency band, which can be used to determine the coherence of the input channels of the input audio signal.

[0027] In some embodiments, the decoder is used to receive a covariance value matrix from an external device, such as an encoder, that provides the input audio signal, wherein the covariance values ​​represent the inter-channel dependencies of a pair of input audio channels. In this case, the calculation of the covariance matrix can be passed to the encoder. The covariance values ​​of the covariance matrix must then be transmitted in the bitstream between the encoder and the decoder. This version allows for flexible rendering settings at the receiver, but requires additional data in the output audio signal.

[0028] In some preferred embodiments, a normalized covariance value matrix may be established, wherein the normalized covariance value matrix is ​​based on the covariance value matrix. Through this feature, further processing may be simplified.

[0029] In some embodiments, the decoder may be operable to create an attractiveness value matrix by applying a mapping function to the covariance value matrix or to a matrix derived from the covariance value matrix.

[0030] In some embodiments, the gradient of the mapping function may be greater than or equal to 0 for all covariance values ​​or values ​​derived from the covariance values.

[0031] In some preferred embodiments, for an input value between 0 and 1, the mapping function can achieve a value between 0 and 1.

[0032] In some embodiments, the decoder can be used to receive an attractiveness value matrix A, which is established by applying a mapping function to the covariance value matrix or to a matrix obtained from the covariance value matrix. The phase calibration can be adjusted in both cases by applying a nonlinear function to the covariance value matrix or a matrix obtained from the covariance value matrix, such as a normalized covariance matrix.

[0033] The phase attraction value matrix provides control data in the form of phase attraction coefficients, which are used to determine the phase attraction between channel pairs. Based on the measured covariance value matrix, the phase adjustment for each time frequency slice is obtained so that channels with low covariance values ​​do not affect each other and channels with high covariance values ​​perform phase search for each other.

[0034] In some embodiments, the mapping function is a non-linear function.

[0035] In some embodiments, for covariance values ​​or values ​​derived from the covariance values ​​less than a first mapping threshold, the mapping function is equal to 0, and / or for covariance values ​​or values ​​derived from the covariance values ​​greater than a second mapping threshold, the mapping function is equal to 1. With this feature, the mapping function consists of three intervals. For all covariance values ​​or values ​​derived from the covariance values ​​less than the first mapping threshold, the phase attraction coefficient is calculated to be 0, so that no phase adjustment is performed. For all covariance values ​​or values ​​derived from the covariance values ​​greater than the first mapping threshold but less than the second mapping threshold, the phase attraction coefficient is calculated to be a value between 0 and 1, so that a partial phase adjustment is performed. For all covariance values ​​or values ​​derived from the covariance values ​​greater than the second mapping threshold, the phase attraction coefficient is calculated to be 1, so that a complete phase adjustment is performed.

[0036] Let's take the following mapping function as an example:

[0037] f(c′ i,j )=a i,j =max(0,min(1,3c′ i,j -1))

[0038] Another preferred embodiment is as follows:

[0039]

[0040] In some embodiments, the mapping function is represented by a function forming an S-shaped curve.

[0041] In a specific embodiment, the decoder is configured to calculate a phase alignment coefficient matrix, wherein the phase alignment coefficient matrix is ​​based on the covariance value matrix and a prototype downmix matrix.

[0042] In some embodiments, the decoder is configured to receive a phase calibration coefficient matrix from an external device, such as an encoder, providing the input audio signal, wherein the phase calibration coefficient matrix is ​​based on the covariance value matrix from and a prototype downmix matrix.

[0043] The phase correction coefficient matrix describes the number of phase corrections that are required to correct the non-zero channels of interest of the input audio signal.

[0044] The prototype downmix matrix defines which input channels are mixed to which output channels. The coefficients of the downmix matrix may be scaling factors, which are used to downmix the input channels to the output channels.

[0045] It is also possible to transfer the complete calculation of the phase calibration coefficient matrix to the encoder. The phase calibration coefficient matrix then has to be transmitted within the input audio signal, but its elements are often zero and can only be quantized in a positive way. When the phase calibration coefficient matrix is ​​closely dependent on the prototype downmix matrix, this phase calibration coefficient matrix is ​​considered known at the encoder end. This limits the possible output channel configurations.

[0046] In some embodiments, the phase and / or amplitude of the downmix coefficients of the downmix matrix are planned to be smooth over time so that temporal artifacts generated by signal cancellation between adjacent time frames are avoided. Here, "smooth over time" means that no sudden changes occur in the downmix coefficients over time. In particular, the downmix coefficients may vary over time according to a continuous or quasi-continuous function.

[0047] In some embodiments, the phase and / or amplitude of the downmix coefficients of the downmix matrix are planned to be smooth with frequency so that spectral artifacts caused by signal cancellation between adjacent frequency bands are avoided. Here, "smooth with frequency" means that no sudden changes occur in the downmix coefficients as the frequency passes. In particular, the downmix coefficients can vary with frequency according to a continuous or quasi-continuous function.

[0048] In some embodiments, the decoder is configured to calculate or receive a normalized phase calibration coefficient matrix, wherein the normalized phase calibration coefficient matrix is ​​based on the phase calibration coefficient matrix. By this feature, further processing can be simplified.

[0049] In some preferred embodiments, the decoder is configured to establish a regularized phase calibration coefficient matrix according to the phase calibration coefficient matrix.

[0050] In some embodiments, the decoder is configured to receive a regularized phase calibration coefficient matrix based on the phase calibration coefficient matrix from an external device, such as an encoder, that provides the input audio signal.

[0051] The proposed downmix method provides effective regularization in the critical condition of opposite phase signals, where the phase alignment process may suddenly change its polarity.

[0052] The additional regularization step is defined to reduce the cancellation in the transition region between adjacent frames caused by abrupt changes in the phase adjustment coefficient. Regularization of abrupt phase changes between adjacent time-frequency slices and avoidance are advantages of the downmix proposed in this paper. It reduces unwanted artifacts generated when phase jumps between adjacent time-frequency slices or grooves between adjacent frequency bands occur.

[0053] The regularized phase-calibrated downmix matrix can be obtained by applying the phase regularization coefficient θ i,j The normalized phase calibration matrix is ​​obtained.

[0054] This regularization coefficient can be calculated in the processing cycle of each time frequency slice. Said regularization can be applied recursively in time and frequency direction. Considering the phase difference between adjacent time slots and frequency bands, they are weighted by the said attraction value that produces weighting matrix. From this matrix, can obtain the regularization coefficient as discussed in more detail below.

[0055] In some preferred embodiments, the downmix matrix is ​​based on the regularized phase calibration coefficient matrix. In this way, it can be ensured that the downmix coefficients of the downmix matrix are smooth over time and frequency.

[0056] Furthermore, an audio signal processing encoder comprises at least one frequency band, and the audio signal processing decoder is used to process an input audio signal having a plurality of input channels in the at least one frequency band, wherein the encoder is used to

[0057] calibrating the phases of the input channels according to the inter-channel dependency between the input channels, wherein the more the phases of the input channels are aligned with each other, the higher the inter-channel dependency thereof; and

[0058] The calibrated input audio signal is downmixed to an output audio signal having a smaller number of output channels than the number of input channels.

[0059] The audio signal processing encoder may be configured similarly to the audio signal processing decoder discussed in this application.

[0060] Furthermore, an audio signal processing encoder comprises at least one frequency band, the audio signal processing encoder being configured to output a bit stream, wherein the bit stream comprises an encoded audio signal in the frequency band, wherein the encoded audio signal has a plurality of encoded channels in the at least one frequency band, wherein the encoder

[0061] for determining inter-channel dependencies between the encoded channels of the input audio signal, and outputting the inter-channel dependencies in the bitstream; and / or

[0062] for determining the energy of the encoded audio signal and outputting the determined energy of the encoded audio signal in the bitstream; and / or

[0063] for calculating a downmix matrix M of a downmixer for downmixing the input audio signal according to the downmix matrix such that the phases of the encoded channels are calibrated according to the identified inter-channel dependencies, preferably such that the energy of an output audio signal of the downmixer is normalized according to the determined energy of the encoded audio signal, and for transmitting the downmix matrix M within the bitstream, wherein in particular the downmix coefficients of the downmix matrix are configured to be smoothed over time such that temporal artifacts due to signal cancellation between adjacent time frames are avoided, and / or wherein in particular the downmix coefficients of the downmix matrix are configured to be smoothed over frequency such that spectral artifacts due to signal cancellation between adjacent frequency bands are avoided; and / or

[0064] for analyzing a time interval of the encoded audio signal using a window function, wherein the inter-channel dependency is determined for each time frame, and for outputting the inter-channel dependency to the bitstream for each time frame; and / or

[0065] for calculating a matrix of covariance values, wherein the covariance values ​​represent said inter-channel dependency of a pair of coded audio channels, and for outputting the matrix of covariance values ​​in said bitstream; and / or

[0066] for establishing an attractiveness value matrix by applying a mapping function to the covariance value matrix or a matrix derived from the covariance value matrix, and for outputting the attractiveness value matrix in the bitstream, wherein for all covariance values ​​or values ​​derived from the covariance values, the gradient of the mapping function is preferably greater than or equal to 0, and the mapping function preferably can reach a value between 0 and 1 for input values ​​between 0 and 1, in particular a non-linear function, in particular a mapping function, for covariance values ​​less than a first mapping threshold, the mapping function is equal to 0, and / or for covariance values ​​less than a second mapping threshold, the mapping function is equal to 0, and / or the mapping function is represented by a function forming an S-shaped curve; and / or

[0067] for calculating a phase calibration coefficient matrix, wherein the phase calibration coefficient matrix is ​​based on the covariance value matrix and the prototype downmix matrix, and / or

[0068] Used to establish a regularized phase calibration coefficient matrix according to the phase calibration coefficient matrix V and used to output the regularized phase calibration coefficient matrix in the bit stream.

[0069] The bit stream of the encoder can be transmitted to the decoder and decoded. For further details, please refer to the description of the decoder.

[0070] The present invention also provides a system, which includes the audio signal processing decoder and the audio signal processing encoder proposed by the present invention.

[0071] In addition, the present invention also provides a method for processing an input audio signal, wherein the input audio signal has multiple input channels in a frequency band, the method comprising the following steps: analyzing the input audio signal in the frequency band, wherein inter-channel dependencies between the input audio channels have been identified; calibrating the phases of the input channels according to the identified inter-channel dependencies, wherein the more the phases of the input channels are calibrated with each other, the higher their inter-channel dependencies; and downmixing the calibrated input audio signal to an output audio signal, which has a smaller number of output channels than the number of input channels in the frequency band.

[0072] In addition, the present invention also provides a computer program, which implements the above method when executed on a computer or a signal processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Hereinafter, embodiments of the present invention will be described in more detail with reference to the accompanying drawings, in which:

[0074] Figure 1 A block diagram showing the proposed adaptive phase calibration downmix;

[0075] Figure 2 The working principle of the proposed method is shown;

[0076] Figure 3 The processing steps for calculating the downmix matrix M are described;

[0077] Figure 4 Shows a formula that can be used to normalize the covariance matrix C′ to calculate the attractiveness value matrix A;

[0078] Figure 5 Schematic block diagram showing a conceptual overview of a three-dimensional audio encoder.

[0079] Figure 6 Schematic block diagram showing a conceptual overview of a three-dimensional audio decoder.

[0080] Figure 7 Schematic block diagram showing a conceptual overview of a format converter.

[0081] Figure 8 An initial signal processing example with two channels varying over time is shown.

[0082] Fig. 9 An example of initial signal processing with two channels varying with frequency is shown.

[0083] Fig.10A 77-band synthesis filter bank is shown. DETAILED DESCRIPTION

[0084] Before describing embodiments of the present invention, more background on prior art encoder and decoder systems is provided.

[0085] Figure 5 is a schematic block diagram of a conceptual overview of a three-dimensional audio encoder 1, and Figure 6 is a schematic block diagram of a conceptual overview of a 3D audio decoder 2.

[0086] The three-dimensional coding and decoding systems 1 and 2 can be based on the MPEG-D Joint Speech and Audio Coding (USAC) encoder 3 for encoding the channel signal 4 and the object signal 5, and based on the MPEG-D Joint Speech and Audio Coding (USAC) decoder 6 for decoding the output audio signal 7 of the encoder 3.

[0087] The bitstream 7 may contain a coded audio signal 37 with reference to a frequency band of the encoder 1, wherein the coded audio signal 37 has a plurality of coded channels 38. This coded audio signal 37 may be fed to a frequency band 36 of the decoder 2 (see Figure 1 ) as the input audio signal 37.

[0088] To increase the coding efficiency for a large number of objects 5, the Spatial Audio Object Coding (SAOC) technique is improved. Three types of renderers 8, 9 and 10 render objects 11 and 12 to channels 13, render channels 13 to headphones or render channels to different speaker setups.

[0089] When object signals in spatial audio object coding are transmitted explicitly or encoded parametrically, the corresponding object metadata (OAM) 14 information is compressed and multiplexed into the 3D audio bitstream 7 .

[0090] Prior to encoding, a pre-renderer / mixer 15 may be optionally used to convert channel object input scenes 4 and 5 into channel scenes 4 and 16, with the same functionality as the object renderer / mixer 15 described below.

[0091] Pre-rendering of the objects 5 ensures deterministic signal entropy at the input of the encoder 3, which is substantially independent of multiple simultaneously activated object signals 5. By pre-rendering of the object signals 5, no object metadata 14 needs to be transmitted.

[0092] The discrete object signals 5 are rendered to a channel layout for use by the encoder 3. For each channel 16, the weight of the object 5 is taken from the associated object metadata 14.

[0093] The core codec can be applied to the speaker channel signals 4, discrete object signals 5, object downmix signals 14 and pre-rendered signals 16 according to the MPEG-D USAC technology. The core codec handles the encoding of the plurality of signals 4, 5 and 14 by generating channel and object mapping information according to the geometric information and semantic information of the input channel and object allocation. The mapping information describes how the input channels 4 and objects 5 are mapped to USAC channel elements, i.e., mapped to dual channel elements (CPE), single channel elements (SCE), low frequency enhancement (LFE), and the corresponding information is transmitted to the decoder 6.

[0094] All additional payload, such as SAOC data 17 or object metadata 14 can be transmitted via the extension element and can be taken into account in the rate control of the encoder 3 .

[0095] The encoding of the object 5 can use different methods, depending on the rate / distortion requirements and the interactivity requirements applied to the renderer. The following object encoding variants are possible:

[0096] - Pre-rendered objects 16: Before encoding, the object signal 5 is pre-rendered and mixed to the channel signal 4, for example before encoding, to a 22.2 channel signal 4. The subsequent encoding chain sees the 22.2 channel signal 4.

[0097] - Discrete Object Waveform: Object 5 is supplied as a mono waveform to the encoder 3. The encoder 3 uses mono elements (SCE) in addition to the channel signal 4 to transmit the object 5. The decoded object 18 is rendered and mixed at the receiver end. Compressed object metadata information 19 and 20 are transmitted side by side to the receiver / renderer 21.

[0098] - Parameterized object waveform 17: SAOC parameters 22 and 23 are used to describe the object properties and the relationship between the object properties. The downmix of the object signal 17 is encoded using USAC. The parameterization information 22 is transmitted side by side. The number of downmix channels 17 selected depends on the number of objects 5 and the overall data rate. The compressed object metadata information 23 is transmitted to the SAOC renderer 24.

[0099] The SAOC encoder 25 and decoder 24 for the object signal 5 are based on the MPEG SAOC technology. This system is able to recreate, modify and render a plurality of audio objects 5 based on a smaller number of transmitted channels 7 and additional parameterized data 22 and 23, such as object level differences (OLD), inter-object correlations (IOC) and downmix gain values ​​(DMG). The additional parameterized data 22 and 23 enable the data rate to be significantly lower than that required for the individual transmission of all objects 5, which makes the encoding very efficient.

[0100] The SAOC encoder 25 takes the object / channel signal 5 as input to become a mono waveform, and outputs parameterization information 22 (filled into the stereo bitstream 7) and SAOC transmission channels 17 (encoded using mono elements and transmitted). The SAOC decoder 24 reconstructs the object / channel signal 5 from the decoded SAOC transmission channels 26 and parameterization information 23, and generates the output audio scene 27 according to the reproduction layout, the decompressed object metadata information 20, and the optional user's interaction information.

[0101] For each object 5, the associated object metadata 14 specifically defines the geometric position and volume of the object in three-dimensional space, and the object metadata encoder 28 can efficiently encode the object metadata 14 by quantizing the object properties in time and space. The compressed object metadata (cOAM) 19 is transmitted to the receiver as side information 20, which can be decoded using an OAM decoder 29.

[0102] The object renderer 21 generates the object waveform 12 using the compressed object metadata 20 according to the given reproduction format. Each object 5 is rendered to a specific output channel 12 according to its object metadata 19 and 20. The output of block 21 is generated from the sum of the partial results. If the channel-based content 11, 30 and the discrete / parametric objects 12, 27 are decoded, the channel-based content 11 and 30 and the rendered object waveforms 12, 27 will be mixed before the waveform 13 is output by the mixer 8 (or before the generated waveform is fed back to the post-processor modules 9 and 10, such as the binaural renderer 9 or the speaker renderer module 10).

[0103] This binaural renderer module 9 generates a binaural downmix of the multi-channel audio material 13 such that each input channel 13 is represented by a virtual sound source. This process is applied frame by frame in the quadrature mirror filter (QMF) domain. The binauralization is based on the measured binaural room impulse response.

[0104] Figure 7The loudspeaker renderer 10, shown in more detail in Figure 1, converts between a transmitted channel configuration 13 and a desired reproduction format 31. The loudspeaker renderer is referred to as a "format converter" 10 in the following. The format converter 10 performs the conversion to reduce the number of output channels 31, i.e. the format converter generates a downmix via a downmixer 32. The DMX configurator 33 automatically generates an optimal downmix matrix for a given combination of input format 13 and output format 31 and uses the downmix matrix in a downmix process 32, wherein a mixer output layout 34 and a reproduction layout 35 are used. The format converter 10 allows standard loudspeaker configurations as well as random configurations of non-standard loudspeaker positions.

[0105] Figure 1 An audio signal processing device having at least one frequency band 36 is shown and is used for processing an input audio signal 37 having a plurality of input channels 38 in the at least one frequency band 36, wherein the device:

[0106] for analyzing the input audio signal 37, wherein inter-channel dependencies between input channels 38 are identified; and

[0107] for aligning the phases of the input channels 38 according to the identified inter-channel dependency 39, wherein the more the phases of the input channels 38 are aligned with each other, the higher their inter-channel dependency 39;

[0108] The method is used for downmixing the calibrated input audio signal to an output audio signal 40 , wherein the number of output channels 41 of the output audio signal 40 is less than the number of input channels 38 .

[0109] The audio signal processing device may be an encoder 1 or a decoder. For example, the present invention is applicable to the encoder 1 and the decoder.

[0110] The downmixing method proposed in the present invention, for example Figure 1 As shown in the block diagram, the design is based on the following principles:

[0111] 1. This phase adjustment is obtained from each time-frequency slice based on the measured signal covariance matrix C, so that it has low c i,j The channels do not affect each other and have high c i,j The channels are phase locked relative to each other;

[0112] 2. The phase adjustment is regularized over time and frequency to avoid signal cancellation artifacts due to phase adjustment differences in the overlapping regions of adjacent time-frequency slices;

[0113] 3. The downmix matrix gains are adjusted to preserve the downmix energy.

[0114] The basic operating principle of the encoder 1 is that mutually dependent (coherent) input channels 38 of the input audio signal 37 attract each other in terms of phase in a specific frequency band 36, while mutually independent (incoherent) input channels 38 of the input audio signal 37 remain unaffected. The encoder 1 is proposed with the aim of improving the downmix quality corresponding to a post-equalization method under critical signal cancellation conditions, while providing the same performance under non-critical conditions.

[0115] Since the inter-channel dependencies 39 are usually not known in advance, an adaptive method for downmixing is proposed.

[0116] A straightforward approach to reproducing the signal spectrum is to apply an adaptive equalizer 42 to attenuate or amplify the signal within the frequency band 36. However, if the frequency notches are sharper than the resolution of the applied frequency conversion, it is reasonable to expect that such an approach will not be able to robustly reproduce the signal 41. This problem is addressed by pre-processing the phase of the input signal 37 before downmixing to avoid such frequency notches in the first place.

[0117] In the following, a method according to an embodiment of the invention is discussed for adaptively downmixing two or more channels 38 in a frequency band 36, i.e. in a so-called time-frequency slice, into a smaller number of channels 41. This method comprises the following features:

[0118] - analysis of the signal energy and inter-channel dependencies 39 in the frequency bands 36 (contained by the covariance matrix C);

[0119] - before downmixing, adjusting the frequency band phase input channel signals 38 such that signal cancellation effects during downmixing are reduced and / or the coherent signal sum is increased;

[0120] - Adjusting the phase so that pairs or groups of channels with high interdependence (but potentially phase offset) are more aligned relative to each other, while channels with less or no interdependence (and potentially phase offset) are less or not all phase aligned relative to each other;

[0121] -Phase adjustment coefficient being (optionally) configured to smooth over time for avoiding temporal artifacts due to signal cancellation between adjacent time frames;

[0122] -Phase adjustment coefficient being (optionally) configured to smooth over frequency for avoiding spectral artifacts due to signal cancellation between adjacent frequency bands;

[0123] The energies of the frequency band downmix channel signals 41 are normalized, for example such that the energy of each frequency band downmix signal 41 is equal to the sum of the energies of the frequency band input signals 38 multiplied by the corresponding downmix gain.

[0124] Furthermore, the proposed downmixing method provides an effective regularization of the critical condition of opposite phase signals, where the opposite phase signals may abruptly switch their polarity during the phase alignment process.

[0125] Next, a mathematical description of the downmixer is provided, which is a specific implementation of the above content. For a person skilled in the art, another specific implementation having the features according to the above description can be foreseen.

[0126] like Figure 2 The method shown is based on the principle that the mutually correlated signals SC1, SC2 and SC3 attract each other in terms of phase in the frequency band 36, while these signals SI1 are incoherent and remain unaffected. The method aims to simply improve the downmix quality of the post-equalization method in critical signal cancellation conditions, while providing the same performance as in non-critical conditions.

[0127] The method is designed based on the short-term random characteristics of the band signal 37 and the static prototype downmix matrix Q to formulate the band 36 adaptive phase alignment and energy balance downmix matrix M. In particular, the method is only used to mutually implement phase alignment to the interdependent channels SC1, SC2, and SC3.

[0128] Figure 1 The general operation is shown in Figure 1. This processing is performed in an overlapping frame-by-frame manner, although other options are readily available, such as using recursive windows to estimate the relevant parameters.

[0129] For each audio input signal frame 43, the phase alignment downmix matrix M comprises phase alignment matrix coefficients which are defined according to the random data of the input signal frame 43 and a prototype downmix matrix Q, and the prototype downmix matrix Q defines which input channels 38 are downmixed to which output channels 41. The signal frame 43 is generated in a windowing step 44. This random data is contained in the complex-valued covariance matrix C of the input signal 37, which is estimated from the signal frame 43 in an estimation step 45 (or using a recursive window). From this complex-valued covariance matrix C, the phase alignment matrix The configuration of the phase-calibrated downmix coefficients in step 46 is obtained.

[0130] Let the number of input channels be N x And the number of downmix channels N y <N x The prototype downmix matrix Q and the phase calibration downmix matrix M are usually sparse matrices with a dimension of N. y ×N x The phase-aligned downmix matrix M typically varies as a function of time and frequency.

[0131] Phase-aligned downmix solutions reduce signal cancellation between channels, but can introduce cancellation in the transition region between adjacent time-frequency slices if the phase adjustment coefficients are changed abruptly. When adjacent opposite-phase input signals are downmixed, abrupt phase changes over time may occur, but at least there are small changes in amplitude or phase. In this case, the polarity of the phase alignment can switch rapidly, even if the signal itself is a fairly stable signal. This effect can occur, for example, when tonal signal components coincide with inter-channel time differences, and which in turn can be based, for example, from the use of spaced-apart microphone recording techniques or from delay-based audio effects.

[0132] On the frequency axis, abrupt phase shifts between slices may occur, for example when two coherent but differently delayed wideband signals are downmixed. Larger phase differences for higher frequency bands, and wrapping around certain band boundaries may cause notches in the transition region.

[0133] Preferably, The phase adjustment coefficients are regularized in another step to avoid processing artifacts due to abrupt phase shifts. The phase adjustment coefficients vary with time or with frequency, or with both time and frequency. In this way, the regularization matrix If the regularization 47 is omitted, signal cancellation artifacts may occur here due to differences in phase adjustment in the overlapping regions of adjacent time frames and / or adjacent frequency bands.

[0134] Next, energy regularization 48 adaptively ensures the dynamic level of energy in the downmix signal 40. In the overlapping step 49, the processed signal frame 43 is overlapped and superimposed on the output data stream 40. Please note that when designing this time-frequency processing structure, a lot of variations will be obtained. It is possible to obtain similar processing with signal processing blocks with different orders. In addition, some blocks can be combined into a single processing step. In addition, when similar processing characteristics are achieved, the method for windowing 44 or block processing can be reformulated in various ways.

[0135] Figure 3 The different steps of phase-aligned downmixing are described.After the downmix matrix M is obtained in the three overall processing steps, it is used to downmix the original multi-channel input audio signal 37 to a different number of channels.

[0136] The detailed description of each sub-step of calculating the matrix M is as follows.

[0137] According to an embodiment of the present invention, the downmixing method can be implemented in a 64-band QMF domain. A 64-band complex modulated uniform QMF filter bank can be used.

[0138] Calculation From the input audio signal x in the time-frequency domain (equivalent to the input audio signal 38), the complex-valued covariance matrix C is calculated as the matrix C = E{xx H}, where E{·} is the expectation operator and x H is the conjugate transpose of x. In actual implementation, the expected operator is replaced by an average operator that varies over multiple time and / or frequency samples.

[0139] Next, in a covariance regularization step 50, the absolute values ​​of the matrix C are regularized so that the matrix C contains values ​​between 0 and 1 (elements are referred to as c′ i,j and the matrix is ​​referred to as C'). These values ​​represent the portion of the sound energy that is correlated between different channel pairs, but may be phase offset. In other words, when the incoherent signal produces a value of 0, the in-phase, anti-phase and reverse phase signals will each produce a normalized value of 1.

[0140] In the attraction value calculation step 51, they are converted into control data (attraction value matrix A)), which is obtained by mapping function f(c′ i,j ) to represent the phase attraction between the channel pairs, this function f(c′ i,j ) is applied to all inputs of the absolute regularized normalized covariance matrix M′. Here, the formula

[0141] f(c′ i,j )=a i,j =max(0,min(1,3c′ i,j -1))

[0142] Can be used (see Figure 4 )

[0143] In this embodiment, for normalized covariance values ​​c′ less than the first mapping threshold 54, i,j , mapping function f(c′ i,j ) is equal to 0, and / or for a normalized covariance value c′ greater than a second mapping threshold 55 i,j Among them, the mapping function f(c′ i,j ) is equal to 1. With these features, the mapping function consists of three intervals. For all normalized covariance values ​​c′ less than the first mapping threshold 54 i,j , phase attraction coefficient a i,j is calculated as zero, so no phase adjustment is performed. For all normalized covariance values ​​c′ that are greater than the first mapping threshold 54 but less than the second mapping threshold 55 i,j , phase attraction coefficient a i,j is calculated as a value between 0 and 1, so that partial phase adjustment is performed. For all normalized covariance values ​​c′ above the second mapping threshold 55i,j , phase attraction coefficient a i,j is evaluated to 1 and a full phase adjustment is performed.

[0144] From the attraction value, calculate the phase calibration coefficient v i,j . It describes the amount of phase calibration that needs to be used to calibrate the non-zero attraction channels of the signal x.

[0145]

[0146] in For elements on the diagonal The result is the phase calibration coefficient matrix V.

[0147] In the phase calibration coefficient matrix normalization step 52, the coefficients v i,j It is then normalized to the magnitude of the downmix matrix Q to produce a normalized phase-aligned downmix matrix The downmix matrix With elements

[0148]

[0149] The advantage of this downmix is ​​that the channels 38 with low attractiveness do not influence each other, since the phase adjustment is derived from the measured signal covariance matrix C. The channels 38 with high attractiveness are phase locked with respect to each other. The strength of the phase correction depends on the properties of the coherence.

[0150] If the phase adjustment coefficients change abruptly, the phase-aligned downmixing scheme reduces signal cancellation between channels, but may produce cancellation in the transition region between adjacent time-frequency slices. When adjacent opposite-phase input signals are downmixed, sudden phase changes over time may occur, but at least there are small changes in amplitude or phase. In this case, the polarity of the phase alignment can be switched quickly.

[0151] Due to the sudden change of the phase adjustment factor v i,j , an additional regularization step 47 is defined to reduce the cancellation in the transition region between adjacent frames. The regularization and avoidance of abrupt phase changes between audio frames provide the advantage of downmixing for this. It reduces the artifacts generated when phase jumps between adjacent audio frames or notches between adjacent frequency bands occur.

[0152] Regularization can be performed in a variety of ways to avoid large phase shifts between adjacent time-frequency slices. In one embodiment, a simple regularization method is used and is described in detail below. In this method, a processing loop can be used to perform each slice in chronological order from the lowest to the highest frequency slice, and phase regularization can be recursively applied relative to the previous slice in time and frequency.

[0153] Figure 8 and Fig. 9 shows the practical effects of the design steps described below. Figure 8 An initial signal 37 with two channels 38 is shown which varies over time. There is a slowly increasing inter-channel phase difference (IPD) 56 between the two channels 38. An abrupt phase shift from +π to -π produces an abrupt change in the non-regularized phase adjustment 57 of the first channel 38 and an abrupt change in the non-regularized phase adjustment 58 of the second channel 38.

[0154] However, the normalized phase adjustment 59 of the first channel 38 and the normalized phase adjustment 60 of the second channel 38 do not show any abrupt changes.

[0155] Fig. 9 An example of an original signal 37 with two channels 38 is shown. In addition, an original spectrum 61 of one channel 38 of the signal 37 is shown. The calibrated downmix spectrum (passive downmix spectrum) 62 shows the effect of the comb filter. The effect of the comb filter is reduced in the uncalibrated downmix spectrum 63. However, the comb filter effect is not obvious in the regularized downmix spectrum 64.

[0156] Regularized Phase Aligned Downmix Matrix This can be achieved by applying the phase regularization coefficient θ i,j To Matrix And get.

[0157] The regularization coefficients are calculated as each time-frequency frame changes in the processing loop. Regularization 47 is applied recursively in the time and frequency directions. The phase differences between adjacent time slots and frequency bands are taken into account and are weighted by the attraction value to produce a weighted matrix M dA . From the matrix, the regularization coefficient can be obtained:

[0158]

[0159] The continuous phase shift is regularized to avoid The phase shift gradually decreases towards zero, and this phase shift depends on the relative signal energy:

[0160]

[0161] in

[0162]

[0163] Regularized Phase Aligned Downmix Matrix The input is:

[0164]

[0165] Finally, the energy normalized phase aligned downmix vectors are defined in an energy normalization step 53 for each channel j, forming the columns of the final phase aligned downmix matrix:

[0166]

[0167] After computing the matrix M, the output audio material is computed. The QMF domain output channels are weighted sums of the QMF input channels. The complex-valued weights are incorporated into the adaptive phase calibration process as the elements of the matrix M:

[0168] y=M·x

[0169] Some processing steps may be transferred to the encoder 1. The processing steps will significantly reduce the processing complexity of the downmix 7 in the decoder 2. This also provides the possibility to interact with the input audio signal 37, where a standard version of the downmixer would generate artifacts. Without changing the decoder 2, this processing step can update the downmix processing rules and improve the downmix quality.

[0170] There are several possibilities when parts of the phase-aligned downmix can be transferred to the encoder 1. It is possible to transfer the phase alignment coefficients v i,j The complete calculation of the phase calibration factor v i,j Then it needs to be transferred to bitstream 7, but the phase calibration coefficient v i,j is always zero and is quantized in an aggressive manner. i,j When closely relying on the prototype downmix matrix Q, this matrix Q must be known at the encoder side. This will limit the possible output channel configurations. The equalizer or energy normalization step may be included in the encoding process or also performed at the decoder 2, because the normalization step is a simple and clearly defined processing step.

[0171] Another possibility is to transfer the calculation of the covariance matrix C to the encoder 1. Then, the elements of the covariance matrix C have to be transferred to the bitstream 7. This version allows a flexible choice of rendering scheme in the receiver 2, but requires more additional data in the bitstream 7.

[0172] Hereinafter, a preferred embodiment of the present invention is described.

[0173] In the following, the audio signal 37 is fed into the format converter 42 and is referred to as the input signal. The audio signal 40 is the result of the format conversion process and is referred to as the output signal. Please note that the audio input signal 37 of the format converter is the audio output signal of the core decoder 6.

[0174] Vectors and matrices are represented by boldface symbols. Vector elements or matrix elements are represented by italic variables supplemented by indices indicating the column / row of the vector / matrix element within the vector / matrix, e.g. [y1…y A …y N ] = y represents a vector and its elements. Similarly, M a,b represents the element in the a-th column and b-th row of the matrix M.

[0175] The following variables will be used:

[0176] N in The number of channels in the input channel configuration

[0177] N out The number of channels in the output channel configuration

[0178] M DMX Downmix matrix, containing real-valued non-negative downmix coefficients (downmix gains), M DMX The dimension is (N out ×N in )

[0179] G EQ A matrix consisting of gain values ​​for each processed frequency band, which determines the frequency response of the equalization filter

[0180] I EQ A vector signaling which equalization filters to apply to the input channels (if any)

[0181] L is the measured frame length in time-domain audio samples.

[0182] ν Time domain sample index

[0183] n QMF slot index (=subband sample index)

[0184] L n The frame length measured in the QMF slot

[0185] F Frame index (number of frames)

[0186] K is the number of hybrid QMF bands, K = 77

[0187] k QMF band index (1..64) or hybrid QMF band index (1..K)

[0188] A, B channel index (number of channels in channel configuration)

[0189] eps numerical constant, eps=10 -35

[0190] Before processing of the audio samples delivered by the core decoder 6 takes place, initialization of the format converter 42 is performed.

[0191] The initialization takes the following data as input parameters

[0192] The sampling rate of the audio data to be processed

[0193] Parameter format_in: It signals the channel configuration of the audio data to be processed by the format converter

[0194] Parameter format_out: signals the channel configuration of the desired output format

[0195] Optional: Parameters that signal the offset of the speaker positions from the standard speaker solution (random setting function).

[0196] Output

[0197] Enter the number of channels in the speaker configuration, N in ,

[0198] The number of channels configured for the output speakers, N out ,

[0199] Downmix matrix M DMX and the filter parameters of the equalization (I EQ ,G EQ ), which is applied to the audio signal processing of the format converter 42.

[0200] Fine-tune gain and delay values ​​(T g,A and T d,A ): Used to compensate for different speaker distances.

[0201] The audio processing block of the format converter 42 obtains the N in The time domain audio samples 37 of the channel 38 are generated by N out The downmixed time domain audio output signal 40 is composed of the channels 41.

[0202] This process takes the following data as input:

[0203] audio data decoded by the core decoder 6,

[0204] The downmix matrix M returned by the initialization of the format converter 42 DMX ,

[0205] The equalization filter parameters (I EQ ,G EQ ).

[0206] The process returns N out The time domain output signal 40 of the channel is applied to the format_out channel configuration and is signaled during the initialization of the format converter 42 .

[0207] The format converter 42 may operate on consecutive and non-overlapping frames of length L=2048 time domain samples of the input audio signal and output a frame of length L of L samples per processed input frame.

[0208] Furthermore, a T / F conversion (hybrid QMF analysis) can be performed. As a first processing step, the converter converts N in Channel time domain input signal L = 2048 samples to hybrid QMF N in Channel signal performance, and this channel signal performance is L n = 32 QMF time slots (slot index n) and K = 77 frequency bands (band index k ). QMF analysis is performed according to Section 7.14.2.2 of ISO / IEC 23003-2:2010, first performing:

[0209] where 0≤ν<L and 0≤n<L n ,

[0210] Then the mixed analysis

[0211]

[0212] Hybrid filtering shall be performed as described in 8.6.4.3 of ISO / IEC 14496-3:2009. However, the low frequency separation definition (Table 8.36 of ISO / IEC 14496-3:2009) may be replaced by the following table:

[0213] Overview of Low Frequency Separation of 77-Band Hybrid Filter Bank

[0214]

[0215] Furthermore, in the following table, the prototype filter definition must be replaced by the coefficients:

[0216] Prototype filter coefficients of the filter separating the low QMF sub-band of the 77-band hybrid filter bank

[0217] n <![CDATA[g 0 [n],Q 0 =8]]> <![CDATA[g 1,2 [n],Q 1,2 =4]]> 0 0.00746082949812 -0.00305151927305 1 0.02270420949825 -0.00794862316203 2 0.04546865930473 0.0 3 0.07266113929591 0.04318924038756 4 0.09885108575264 0.12542448210445 5 0.11793710567217 0.21227807049160 6 0.125 0.25 7 0.11793710567217 0.21227807049160 8 0.09885108575264 0.12542448210445 9 0.07266113929591 0.04318924038756 10 0.04546865930473 0.0 11 0.02270420949825 -0.00794862316203 12 0.00746082949812 -0.00305151927305

[0218] Furthermore, contrary to 8.6.4.3 of ISO / IEC 14496-3:2009, no subbands are combined, i.e. a 77-band hybrid filter bank is formed by splitting the lowest 3 QMF subbands into (8,4,4) subbands. Fig.10 , the 77 hybrid QMF bands are not reordered but follow the transmission order of the hybrid filter bank.

[0219] Now, a static equalizer gain can be used. Transformer 42 applies a zero phase gain to input channel 38, and the input channel is passed through I EQ and G EQ Variables are signaled.

[0220] I EQ The length is N in vector, then for the N in Each channel A of the input channel sends a signal

[0221] No equalization filters must be applied to specific input channels: I EQ,A =0,

[0222] or with index I EQ,A The G corresponding to the equalization filter of >0 EQ A gain must be applied.

[0223] If for input channel A, I EQ,A > 0, the input signal of channel A is transmitted from G EQ The rows of the matrix are filtered by multiplication to obtain zero phase gain, the G EQ The matrix is ​​I EQ,A Signaling:

[0224]

[0225] Note that all the following processing steps until the conversion back to the time domain signal are performed individually for each hybrid QMF band k and independently of k. The band parameter k is therefore omitted in the equations below, e.g. for each band k,

[0226] Furthermore, an update of the input data and signal adaptive input data windowing is performed. Let F be a monotonically increasing frame index used to represent the current frame of input data, such as for frame F, After initialization of the format converter 42, the first frame of input data starts at F=0. The length is 2*L n The analysis frame from the input mixed QMF spectrum is formulated as

[0227]

[0228] The analysis frame is multiplied by the analysis window w according to the following formula F,n

[0229]

[0230] Among them, w F,n is a signal adaptive window, which is calculated and applied to each frame F, as follows:

[0231]

[0232]

[0233]

[0234]

[0235] Now, a covariance analysis can be performed. The covariance analysis is performed on the windowed input data, and the expected budget sub E(·) is performed as the sum of the auto / cross terms and with the 2L of the windowed input data frame F. n QMF time slot changes. For each processed frame F, the next processing step is performed independently. The index F is therefore omitted until it is explicitly needed, e.g. for frame F,

[0236] Please note that in the case of in In the case of input channels, Represents N in Therefore, the covariance value matrix is ​​formed as follows:

[0237]

[0238] Here (·) T represents transpose and (·) * represents the complex conjugate of the variable, and C y is N, where F is calculated once in each frame. in ×N in The matrix of .

[0239] From the covariance matrix C y Get the inter-channel coherence coefficient between channels A and B

[0240]

[0241] Among them, in the symbol C y,a,b The two indices represent the y The matrix element in the a-th column and b-th row.

[0242] Furthermore, the phase calibration matrix can be formulated. A,B The values ​​are mapped to an attraction measurement matrix T, which has elements

[0243]

[0244] And the intermediate phase calibration mixing matrix M int (equivalent to the normalized phase calibration coefficient matrix in the previous embodiment ) is formulated. With the attraction value matrix:

[0245] P A,B =T A,B ·C y,A,B and

[0246] V=M DMX P,

[0247] The matrix elements are obtained as follows:

[0248] M int,A,B =M DMX,A,B ·exp(jarg(V A,B )),

[0249] where exp(·) represents the exponential function, is the imaginary unit, and arg(·) is the argument of the returned complex variable.

[0250] To avoid abrupt phase shifts, the intermediate phase calibration mixing matrix M int Modified to produce M mod : First, for each frame F, the weighted matrix D F is defined as having elements The phase of the mixing matrix that changes with time (that is, changes with frames) is obtained by comparing the current weighted intermediate mixing matrix and the weighted mixing matrix M generated by the previous frame. mod To measure:

[0251]

[0252]

[0253]

[0254]

[0255]

[0256] The measured phase change of the intermediate mixing matrix is ​​processed to obtain a phase correction parameter, and this phase correction parameter is applied to the intermediate mixing matrix M int , generating M mod (equivalent to the regularized phase calibration coefficient matrix ):

[0257]

[0258]

[0259] The energy conversion is applied to the mixing matrix to obtain the final phase-calibrated mixing matrix M PA .in

[0260]

[0261] in(·) H represents the conjugate transpose operator, and

[0262]

[0263] where the limit is defined as S max =10 0.4 and S min =10 -0.5 , the final phase calibration mixing matrix elements are as follows

[0264] M PA,B,A =S lim,B ·M mod,B,A .

[0265] In a further step, the output data can be calculated. The output signal of the current frame F is obtained by applying the same complex-valued downmix matrix To the windowed input data vector All 2L n Time slot n is used to calculate

[0266]

[0267] The overlap step is applied to the newly calculated output signal frame To achieve the final frequency domain output signal, including L for each channel of frame F n sample,

[0268]

[0269] Now, the F / T conversion (hybrid QMF synthesis) can be performed. Please note that the processing steps described above must be performed independently for each hybrid QMF band k. In the following equation, the band index k is reintroduced, i.e. Hybrid QMF frequency domain output signal is converted into N time domain samples of length L for each output channel B out The time domain signal frame of the channel to obtain the final time domain output signal

[0270] The mixed synthesis

[0271]

[0272] Can be implemented as ISO / IEC 14496-3:2009 Figure 8 .21, i.e., the sum of the subbands of the lowest three QMF subbands is calculated to obtain the three lowest QMF subbands of the 64-band QMF representation. However, the Figure 8 The .21 process must be applied to the (8,4,4) low-band separation, instead of the (6,2,2) low-band separation shown.

[0273] The subsequent QMF synthesis

[0274]

[0275] This may be performed as defined in section 7.14.2.2 of ISO / IEC 23003-2:2010.

[0276] If the radius of the output speaker locations is different (that is, if for all output channels A, trim A The compensation parameters obtained in the initialization are applied to the output signal. The signal of output channel A will be T d,A The time domain samples are delayed and the signal will also be multiplied by the linear gain T g,A .

[0277] Regarding the decoder and the encoder as well as the methods of the described embodiments are mentioned in the following.

[0278] Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent the description of the corresponding method, where the blocks or apparatus correspond to method steps or features of method steps. Similarly, the aspects described in the context of method steps also represent the description of the corresponding blocks or items or features of the corresponding apparatus.

[0279] According to certain implementation requirements, embodiments of the present invention can be implemented in hardware or software. The embodiments can be implemented using a digital storage medium, such as a floppy disk, DVD, CD, ROM, PROM, EPROM, EEPROM or flash memory, with electronically readable control signals stored thereon, which cooperate with (or can cooperate with) a programmable computer system to perform the respective methods.

[0280] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that the system performs one of the methods described herein.

[0281] Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may, for example, be stored on a machine readable carrier.

[0282] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier or a non-transitory storage medium.

[0283] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0284] A further embodiment of the inventive methods is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein.

[0285] A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may be, for example, configured to be transmitted over a data communication connection, for example over the Internet.

[0286] A further embodiment comprises a processing means, for example a computer or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0287] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0288] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may collaborate with a microprocessor to perform one of the methods described herein. Typically, the method is preferably performed by a hardware device.

[0289] Although the present invention has described several embodiments, it is within the scope of the present invention to make changes, substitutions and equivalents thereto. It should also be noted that there are many ways to replace the implementation method and composition of the present invention. Therefore, the claims attached hereto should be understood to include all such changes, substitutions and equivalents, which do not depart from the spirit and scope of the present invention.

Claims

1. An audio signal processing decoder comprising at least one frequency band (36) and arranged to process an input audio signal (37) having a plurality of input channels (38) within the at least one frequency band (36), wherein the decoder (2) is arranged to calibrating the phases of the input channels (38) according to an inter-channel dependency (39) between the input channels (38), wherein the more the phases of the input channels (38) are aligned with each other, the higher the inter-channel dependency (39) thereof; downmixing an input audio signal having calibrated phases of input channels (38) to an output audio signal (40), the output audio signal (40) having a smaller number of output channels (41) than the number of input channels (38); The decoder (2) is configured to analyze the input audio signal (37) within the frequency band (36) in order to identify inter-channel dependencies (39) between input channels (38) or to receive inter-channel dependencies (39) between input channels (38) from an external device providing the input audio signal (37), the external device comprising an encoder (1); In order to normalize the energy of the output audio signal (40) according to the already determined energy of the input audio signal (37), wherein the decoder (2) is configured to determine the signal energy of the input audio signal (37) or to receive the already determined energy of the input audio signal (37) from an external device providing the input audio signal (37), the external device comprising an encoder (1).

2. A decoder as claimed in claim 1, wherein the decoder (2) comprises a downmixer (42) for downmixing the input audio signal (37) according to a downmix matrix, wherein the decoder (2) is configured to calculate the downmix matrix so that the phases of the input channels (38) are calibrated according to the identified inter-channel dependencies (39), or the decoder (2) is configured to receive the calculated downmix matrix so that the phases of the input channels (38) are calibrated according to the identified inter-channel dependencies (39) from an external device providing the input audio signal (37), the external device comprising the encoder (1).

3. A decoder as claimed in claim 2, wherein the decoder (2) is configured to calculate the downmix matrix so that the energy of the output audio signal (40) is normalized according to the already determined energy of the input audio signal (37), or is configured to receive the downmix matrix, which is calculated so that the energy of the output audio signal (40) is normalized according to the already determined energy of the input audio signal (37) from an external device providing the input audio signal (37), the external device comprising an encoder (1).

4. A decoder as claimed in claim 1, wherein the decoder (2) is configured to analyze time frames (43) of the input audio signal (37) using a window function, wherein the inter-channel dependency (39) is determined for each time frame (43), or wherein the decoder (2) is configured to receive an analysis of time frames (43) of the input audio signal (37) using a window function from an external device providing the input audio signal (37), wherein the inter-channel dependency (39) is determined for each time frame (43), the external device comprising an encoder (1).

5. A decoder as claimed in claim 1, wherein the decoder (2) is configured to calculate a matrix of covariance values, wherein the covariance values ​​represent inter-channel dependencies (39) of a pair of input channels (38), or wherein the decoder (2) is configured to receive a matrix of covariance values, wherein the covariance values ​​represent inter-channel dependencies (39) of a pair of input channels (38), from an external device providing the input audio signal (37), the external device comprising an encoder (1).

6. A decoder as claimed in claim 5, wherein the decoder (2) is used to establish an attraction value matrix by applying a mapping function to the covariance value matrix or a matrix obtained from the covariance value matrix, or to receive an attraction value matrix, wherein the attraction value matrix is ​​established by applying a mapping function to the covariance value matrix or a matrix obtained from the covariance value matrix, wherein for all covariance values ​​or values ​​obtained from the covariance values, the gradient of the mapping function is greater than or equal to 0, and wherein for input values ​​between 0 and 1, the mapping function reaches a value between 0 and 1. The decoder of claim 6 , wherein the mapping function is a non-linear function.

8. A decoder as described in claim 6, wherein for the covariance value or the numerical value obtained from the covariance value that is less than a first mapping threshold, the mapping function is equal to 0; and / or wherein for the covariance value or the numerical value obtained from the covariance value that is greater than a second mapping threshold, the mapping function is equal to 1.

9. The decoder of claim 6, wherein the mapping function is represented by a function forming an S-shaped curve.

10. A decoder as claimed in claim 5, wherein the decoder (2) is used to calculate a phase calibration coefficient matrix, wherein the phase calibration coefficient matrix is ​​based on the covariance value matrix and the prototype downmix matrix or to receive a phase calibration coefficient matrix from an external device providing the input audio signal (37), wherein the phase calibration coefficient matrix is ​​based on the covariance value matrix and the prototype downmix matrix, and the external device comprises an encoder (1).

11. The decoder of claim 10, wherein the phase and / or amplitude of the downmix coefficients of the downmix matrix are arranged to be smoothed over time such that temporal artifacts due to signal cancellation between adjacent time frames (43) are avoided.

12. The decoder of claim 10, wherein the phase and / or amplitude of the downmix coefficients of the downmix matrix are configured to be smoothed over frequency such that spectral artifacts due to signal cancellation between adjacent frequency bands (36) are avoided.

13. A decoder as claimed in claim 10, wherein the decoder (2) is used to establish a regularized phase calibration coefficient matrix based on the phase calibration coefficient matrix, or to receive a regularized phase calibration coefficient matrix based on the phase calibration coefficient matrix from an external device providing the input audio signal (37), and the external device includes an encoder (1).

14. A decoder as claimed in claim 11 or 13, wherein the decoder (2) is used to establish a regularized phase calibration coefficient matrix based on the phase calibration coefficient matrix, or to receive a regularized phase calibration coefficient matrix based on the phase calibration coefficient matrix from an external device providing the input audio signal (37), the external device comprising an encoder (1), wherein the downmix matrix is ​​based on the regularized phase calibration coefficient matrix.

15. An audio signal processing encoder comprising at least one frequency band (36) for processing an input audio signal (37) having a plurality of input channels (38) within the at least one frequency band (36), wherein the encoder (1) for calibrating the phases of the input channels (38) according to an inter-channel dependency (39) between the input channels (38), wherein the more the phases of the input channels (38) are aligned with each other, the higher their inter-channel dependency (39); for downmixing an input audio signal having calibrated phases of input channels (38) to an output audio signal (40), the output audio signal (40) having a smaller number of output channels (41) than the number of input channels (38); for analyzing the input audio signal (37) within the frequency band (36) in order to identify inter-channel dependencies (39) between input channels (38) or to receive inter-channel dependencies (39) between input channels (38) from an external device providing the input audio signal (37), the external device comprising an encoder (1); and In order to normalize the energy of the output audio signal (40) according to the already determined energy of the input audio signal (37), a method for determining the signal energy of the input audio signal (37) or for receiving the already determined energy of the input audio signal (37) from an external device providing the input audio signal (37) is provided.

16. An audio signal processing system, comprising: An audio signal processing encoder (1) comprising at least one frequency band (36) and being used to output a bit stream (7), wherein: The bitstream (7) comprises an input audio signal (37) in the frequency bands (36), wherein the input audio signal (37) has a plurality of input channels (38) in the at least one frequency band (36), and An audio signal processing decoder (2) as claimed in claim 1, configured to process an input audio signal (37) having a plurality of input channels (38) in at least one frequency band (36); The encoder (1) is configured for determining inter-channel dependencies (39) between input channels (38) of an input audio signal (37) and for outputting the inter-channel dependencies (39) in a bitstream (7); wherein the decoder (2) is configured For receiving inter-channel dependencies (39) between input channels (38) from an encoder (1).

17. A method for operating a decoder (2) to process an input audio signal (37), the input audio signal having a plurality of input channels (38) within a frequency band (36), the method comprising the following steps: calibrating the phases of the input channels (38) according to the identified inter-channel dependency (39), wherein the more the phases of the input channels (38) are aligned with each other, the higher their inter-channel dependency (39); downmixing an input audio signal having calibrated phases of input channels (38) to an output audio signal (40), the output audio signal (40) having a smaller number of output channels (41) than the number of input channels (38) within the frequency band (36); The decoder (2) is configured to analyze the input audio signal (37) within the frequency band (36) in order to identify inter-channel dependencies (39) between input channels (38) or to receive inter-channel dependencies (39) between input channels (38) from an external device providing the input audio signal (37), the external device comprising an encoder (1); The energy of the output audio signal (40) is normalized based on the already determined energy of the input audio signal (37), wherein the decoder (2) is configured to determine the signal energy of the input audio signal (37) or to receive the already determined energy of the input audio signal (37) from an external device providing the input audio signal (37), the external device comprising an encoder (1).

18. A storage medium having stored thereon a computer program for implementing the method according to claim 17 when executed on a computer or a signal processor.

Images