Audio encoder, audio decoder, method and computer readable medium

Abstract

An audio decoder for providing at least four audio channel signals based on an encoded representation is configured to use multi-channel decoding to provide a first residual signal and a second residual signal based on a joint encoded representation of the first residual signal and the second residual signal. The audio decoder is configured to use residual signal-assisted multi-channel decoding to provide a first audio channel signal and a second audio channel signal based on a first down-mix signal and the first residual signal. The audio decoder is configured to use residual signal-assisted multi-channel decoding to provide a third audio channel signal and a fourth audio channel signal based on a second down-mix signal and the second residual signal. The audio encoder is based on corresponding considerations.

Description

[0001] The present application is a divisional application of the international application PCT / EP2014 / 064915 with an international application date of July 11, 2014 and a priority date of July 22, 2013, which has entered the Chinese national phase and has the corresponding Chinese invention patent application number 201480041694.1 entitled "Audio decoder, audio encoder, method and computer-readable medium using jointly coded residual signal". Technical Field

[0002] An embodiment according to the invention relates to an audio decoder for providing at least four audio channel signals based on an encoded representation.

[0003] Another embodiment according to the invention relates to an audio encoder for providing an encoded representation based on at least four audio channel signals.

[0004] Another embodiment according to the invention relates to a method for providing at least four audio channel signals based on an encoded representation and a method for providing an encoded representation based on at least four audio channel signals.

[0005] A further exemplary embodiment according to the invention relates to a computer program for performing one of the methods described.

[0006] Generally speaking, embodiments according to the present invention relate to joint coding of n channels. Background Art

[0007] In recent years, the demand for storage and transmission of audio content has been steadily increasing. In addition, the quality requirements for storage and transmission of audio content have also been steadily increasing. Therefore, the concept of encoding and decoding for audio content has been enhanced. For example, the so-called "Advanced Audio Coding" (AAC) has been developed, which is described in, for example, the International Standard ISO / IEC 13818-7:2003. In addition, some spatial extensions have been created, such as the so-called "MPEG Surround Sound", which is described in, for example, the International Standard ISO / IEC 23003-1:2007. In addition, additional improvements for encoding and decoding spatial information of audio signals are described in the International Standard ISO / IEC23003-2:2010, which relates to the so-called Spatial Audio Object Coding (SAOC).

[0008] Furthermore, flexible audio coding / decoding concepts are defined in the international standard ISO / IEC 23003-3:2012, which describes the so-called "Unified Speech and Audio Coding" (USAC) concept, which provides the possibility to encode both general audio signals and speech signals with good coding efficiency and to process multi-channel audio signals.

[0009] In MPEG USAC [1], joint stereo coding of two channels is performed using complex prediction with a band limited residual signal or a full band residual signal, MPS 2-1-1 or unified stereo.

[0010] MPEG Surround [2] hierarchically combines OTT and TTT frames to perform joint coding of multi-channel audio with or without sending a residual signal.

[0011] However, it would be desirable to provide even more advanced concepts for efficient encoding and decoding of three-dimensional audio scenes. Summary of the invention

[0012] According to an embodiment of the present invention, an audio decoder for providing at least four audio channel signals based on an encoded representation is created. The audio decoder is configured to use multi-channel decoding to provide a first residual signal and a second residual signal based on a joint encoded representation of the first residual signal and the second residual signal. The audio decoder is also configured to use residual signal-assisted multi-channel decoding to provide a first audio channel signal and a second audio channel signal based on a first down-mix signal and the first residual signal. The audio decoder is also configured to use residual signal-assisted multi-channel decoding to provide a third audio channel signal and a fourth audio channel signal based on a second down-mix signal and the second residual signal.

[0013] Embodiments according to the invention are based on the finding that dependencies between four or even more audio channel signals can be exploited by deriving two residual signals from a jointly encoded representation of the residual signals, each of the two residual signals being used for multi-channel decoding assisted by the residual signals to provide the two or more audio channel signals. In other words, it has been found that there are usually some similarities of the residual signals, such that the bit rate for encoding the residual signals can be reduced by deriving the two residual signals from the jointly encoded representation using multi-channel decoding, which helps improve the audio quality when decoding at least four audio channel signals, which exploits the similarity and / or dependencies between the residual signals.

[0014] In a preferred embodiment, the audio decoder is configured to provide the first down-mix signal and the second down-mix signal based on a joint coded representation of the first down-mix signal and the second down-mix signal using multi-channel decoding. Thus, a hierarchical structure of an audio decoder is created, wherein both the down-mix signal and the residual signal used in the residual signal-assisted multi-channel decoding for providing at least four audio channel signals are derived using separate multi-channel decoding. This concept is particularly effective because the two down-mix signals typically include similarities that can be used in multi-channel encoding / decoding, and because the two residual signals typically also include similarities that can be used in multi-channel encoding / decoding. Thus, good coding efficiency can generally be obtained using this concept.

[0015] In a preferred embodiment, the audio decoder is configured to use prediction-based multi-channel decoding to provide the first residual signal and the second residual signal based on the jointly encoded representation of the first residual signal and the second residual signal. The use of prediction-based multi-channel decoding usually leads to a fairly good quality of the reconstruction of the residual signals. This is for example advantageous if the first residual signal represents the left side of an audio scene and the second residual signal represents the right side of the audio scene, because human hearing is usually quite sensitive to the difference between the left side and the right side of an audio scene.

[0016] In a preferred embodiment, the audio decoder is configured to provide the first residual signal and the second residual signal based on a jointly encoded representation of the first residual signal and the second residual signal using residual signal-assisted multi-channel decoding. It has been found that a particularly good quality of the first residual signal and the second residual signal can be achieved if the first residual signal and the second residual signal are provided using multi-channel decoding that also receives a residual signal (and typically also receives a downmix signal that combines the first residual signal and the second residual signal). Thus, there is a cascade of decoding stages in which two residual signals (a first residual signal for providing a first audio channel signal and a second audio channel signal, and a second residual signal for providing a third audio channel signal and a fourth audio channel signal) are provided based on an input downmix signal and an input residual signal, wherein the input residual signal can also be designated as a common residual signal (of the first residual signal and the second residual signal). Thus, the first residual signal and the second residual signal are in fact "intermediate" residual signals that are derived from the corresponding downmix signal and the corresponding "common" residual signal using multi-channel decoding.

[0017] In a preferred embodiment, the prediction-based multi-channel decoding is configured to estimate prediction parameters describing the contribution of signal components derived using signal components of a previous frame to the residual signals (i.e., the first residual signal and the second residual signal) providing the current frame. The use of the prediction-based multi-channel decoding leads to a particularly good quality of the residual signals (the first residual signal and the second residual signal).

[0018] In a preferred embodiment, the prediction-based multi-channel decoding is configured to obtain the first residual signal and the second residual signal based on a (corresponding) downmix signal and a (corresponding) "common" residual signal, wherein the prediction-based multi-channel decoding is configured to apply the common residual signal with a first sign to obtain the first residual signal and to apply the common residual signal with a second sign to obtain the second residual signal, the second sign being opposite to the first sign. It has been found that such prediction-based multi-channel decoding leads to good efficiency in reconstructing the first residual signal and the second residual signal.

[0019] In a preferred embodiment, the audio decoder is configured to provide the first residual signal and the second residual signal based on a joint coded representation of the first residual signal and the second residual signal using multi-channel decoding operating in a modified discrete cosine transform domain (MDCT domain). It has been found that this concept can be implemented in an efficient manner because the audio decoding that can be used to provide the joint coded representation of the first residual signal and the second residual signal preferably operates in the MDCT domain. Therefore, intermediate conversion can be avoided by applying multi-channel decoding that provides the first residual signal and the second residual signal in the MDCT domain.

[0020] In a preferred embodiment, the audio decoder is configured to provide the first residual signal and the second residual signal based on a joint coded representation of the first residual signal and the second residual signal using USAC complex stereo prediction (e.g., as mentioned in the USAC standard cited above). It has been found that this USAC complex stereo prediction leads to good decoding results for the first residual signal and the second residual signal. In addition, the use of USAC complex stereo prediction for the decoding of the first residual signal and the second residual signal also makes it possible to simply implement the concept using decoding blocks already available in Unified Speech and Audio Coding (USAC). Therefore, the Unified Speech and Audio Codec can be easily reconfigured to perform the decoding concepts discussed herein.

[0021] In a preferred embodiment, the audio decoder is configured to provide the first audio channel signal and the second audio channel signal based on the first down-mix signal and the first residual signal using parameter-based, residual signal-assisted multi-channel decoding. Similarly, the audio decoder is configured to provide the third audio channel signal and the fourth audio channel signal based on the second down-mix signal and the second residual signal using parameter-based, residual signal-assisted multi-channel decoding. It has been found that the multi-channel decoding is extremely suitable for audio channel signal derivation based on the first down-mix signal, the first residual signal, the second down-mix signal and the second residual signal. Furthermore, it has been found that the parameter-based, residual signal-assisted multi-channel decoding can be implemented with less effort using processing blocks that are already present in typical multi-channel audio decoders.

[0022] In a preferred embodiment, the parameter-based, residual signal-assisted multi-channel decoding is configured to estimate one or more parameters describing a required correlation between two channels and / or an order difference between two channels in order to provide two or more audio channel signals based on a respective downmix signal and a respective corresponding residual signal. It has been found that this parameter-based, residual signal-assisted multi-channel decoding is well suited for a second stage of a cascaded multi-channel decoding (wherein, preferably, a prediction-based multi-channel decoding is used to provide the first downmix signal and the second downmix signal and the first residual signal and the second residual signal).

[0023] In a preferred embodiment, the audio decoder is configured to provide the first audio channel signal and the second audio channel signal based on the first down-mix signal and the first residual signal using residual signal-assisted multi-channel decoding operating in the QMF domain. Similarly, the audio decoder is preferably configured to provide the third audio channel signal and the fourth audio channel signal based on the second down-mix signal and the second residual signal using residual signal-assisted multi-channel decoding operating in the QMF domain. Thus, the second stage of layered multi-channel decoding is operated in the QMF domain, which is extremely suitable for typical post-processing and is also typically performed in the QMF domain so that intermediate conversions can be avoided.

[0024] In a preferred embodiment, the audio decoder is configured to provide the first audio channel signal and the second audio channel signal based on the first down-mix signal and the first residual signal using MPEG surround 2-1-2 decoding or unified stereo decoding. Similarly, the audio decoder is preferably configured to provide the third audio channel signal and the fourth audio channel signal based on the second down-mix signal and the second residual signal using MPEG surround 2-1-2 decoding or unified stereo decoding. It has been found that this decoding concept is particularly suitable for the second stage of layered decoding.

[0025] In a preferred embodiment, the first residual signal and the second residual signal are associated with different horizontal positions (or equivalently, azimuthal positions) of the audio scene. It has been found that separating the residual signals associated with different horizontal positions (or azimuthal positions) in a first stage of layered multichannel processing is particularly advantageous, since a particularly good auditory impression can be obtained if a perceptually important left / right separation is performed in a first stage of layered multichannel decoding.

[0026] In a preferred embodiment, the first audio channel signal and the second audio channel signal are associated with vertically adjacent positions of the audio scene (or equivalently, adjacent height positions of the audio scene). Furthermore, the third audio channel signal and the fourth audio channel signal are preferably associated with vertically adjacent positions of the audio scene (or equivalently, adjacent height positions of the audio scene). It has been found that good decoding results can be achieved if the separation between the upper signal and the lower signal is performed in a second stage of layered audio decoding (which separation typically includes a slightly smaller separation accuracy than in the first stage), because the human auditory system is less sensitive to the vertical position of the audio source when compared to the horizontal position of the audio source.

[0027] In a preferred embodiment, the first audio channel signal and the second audio channel signal are associated with a first horizontal position (or equivalently, azimuthal position) of an audio scene, and the third audio channel signal and the fourth audio channel signal are associated with a second horizontal position (or equivalently, azimuthal position) of the audio scene, the second horizontal position (or equivalently, azimuthal position) being different from the first horizontal position (or equivalently, azimuthal position).

[0028] Preferably, the first residual signal is associated with a left side of the audio scene and the second residual signal is associated with a right side of the audio scene.Thus, a left-right separation is performed in a first stage of layered audio decoding.

[0029] In a preferred embodiment, the first audio channel signal and the second audio channel signal are associated with a left side of an audio scene, and the third audio channel signal and the fourth audio channel signal are associated with a right side of the audio scene.

[0030] In another preferred embodiment, the first audio channel signal is associated with the lower left side of the audio scene, the second audio channel signal is associated with the upper left side of the audio scene, the third audio channel signal is associated with the lower right side of the audio scene, and the fourth audio channel signal is associated with the upper right side of the audio scene. This correlation of the audio channel signals leads to particularly good encoding results.

[0031] In a preferred embodiment, the audio decoder is configured to provide the first down-mix signal and the second down-mix signal based on a joint encoded representation of the first down-mix signal and the second down-mix signal using multi-channel decoding, wherein the first down-mix signal is associated with a left side of an audio scene and the second down-mix signal is associated with a right side of the audio scene. It has been found that multi-channel encoding can be used to encode down-mix signals with good coding efficiency even if the down-mix signals are associated with different sides of the audio scene.

[0032] In a preferred embodiment, the audio decoder is configured to provide the first down-mix signal and the second down-mix signal based on a joint coded representation of the first down-mix signal and the second down-mix signal using prediction-based multi-channel decoding or even using residual signal-assisted, prediction-based multi-channel decoding. It has been found that the use of such multi-channel decoding concepts provides particularly good decoding results. Furthermore, existing decoding functionality may be reused in some audio decoders.

[0033] In a preferred embodiment, the audio decoder is configured to perform a first multi-channel bandwidth extension based on the first audio channel signal and the third audio channel signal. Furthermore, the audio decoder may be configured to perform a second (usually separate) multi-channel bandwidth extension based on the second audio channel signal and the fourth audio channel signal. It has been found to be advantageous to perform a possible bandwidth extension based on two audio channel signals associated with different sides of an audio scene, wherein different residual signals are typically associated with different sides of the audio scene.

[0034] In a preferred embodiment, the audio decoder is configured to perform a first multi-channel bandwidth extension based on the first audio channel signal and the third audio channel signal and one or more bandwidth extension parameters to obtain two or more bandwidth extended audio channel signals associated with a first common horizontal plane (or equivalently, a first common altitude) of the audio scene. Furthermore, the audio decoder is preferably configured to perform a second multi-channel bandwidth extension based on the second audio channel signal and the fourth audio channel signal and one or more bandwidth extension parameters to obtain two or more bandwidth extended audio channel signals associated with a second common horizontal plane (or equivalently, a second common altitude) of the audio scene. It has been found that this decoding scheme results in good audio quality because in this arrangement the multi-channel bandwidth extension can take into account stereo characteristics, which are important for the auditory impression.

[0035] In a preferred embodiment, the joint coded representation of the first residual signal and the second residual signal comprises a channel pair unit, the channel pair unit comprising a downmix of the first residual signal and the second residual signal and a common residual signal of the first residual signal and the second residual signal. It has been found that the encoding of the downmix of the first residual signal and the second residual signal and the common residual signal of the first residual signal and the second residual signal using a channel pair unit is advantageous because the downmix of the first residual signal and the second residual signal and the common residual signal of the first residual signal and the second residual signal generally share a plurality of characteristics. Therefore, the use of a channel pair unit generally reduces the signal transmission overhead and thus enables efficient encoding.

[0036] In another preferred embodiment, the audio decoder is configured to use multi-channel decoding to provide the first down-mix signal and the second down-mix signal based on a joint encoded representation of the first down-mix signal and the second down-mix signal, wherein the joint encoded representation of the first down-mix signal and the second down-mix signal comprises a channel pair unit. The channel pair unit comprises down-mix signals of the first down-mix signal and the second down-mix signal and a common residual signal of the first down-mix signal and the second down-mix signal. This embodiment is based on the same considerations as the above-described embodiment.

[0037] According to another embodiment of the present invention, an audio encoder for providing an encoded representation based on at least four audio channel signals is created. The audio encoder is configured to jointly encode at least a first audio channel signal and a second audio channel signal using multi-channel encoding assisted by a residual signal to obtain a first down-mix signal and a first residual signal. The audio encoder is configured to jointly encode at least a third audio channel signal and a fourth audio channel signal using multi-channel encoding assisted by a residual signal to obtain a second down-mix signal and a second residual signal. In addition, the audio encoder is configured to jointly encode the first residual signal and the second residual signal using multi-channel encoding to obtain a jointly encoded representation of the residual signals. This audio encoder is based on the same considerations as the audio decoder described above.

[0038] Furthermore, the optional improvements of this audio encoder and the preferred configuration of the audio encoder are substantially parallel to the improvements and preferred configuration of the audio decoder discussed above. Therefore, reference is made to the above discussion.

[0039] According to another embodiment of the invention, a method for providing at least four audio channel signals based on an encoded representation is created, which method substantially performs the functions of the audio encoder described above and which method may be supplemented by any of the features and functions discussed above.

[0040] According to another embodiment of the invention a method for providing an encoded representation based on at least four audio channel signals is created, which method substantially implements the functionality of the audio decoder described above.

[0041] According to a further exemplary embodiment of the invention, a computer program is created for executing the above-mentioned method. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Embodiments according to the present invention will now be described with reference to the accompanying drawings, in which:

[0043] Figure 1 A schematic block diagram of an audio encoder according to an embodiment of the present invention is shown;

[0044] Figure 2 A schematic block diagram of an audio decoder according to an embodiment of the present invention is shown;

[0045] Figure 3 A schematic block diagram of an audio decoder according to another embodiment of the present invention is shown;

[0046] Figure 4 A schematic block diagram of an audio encoder according to an embodiment of the present invention is shown;

[0047] Figure 5 A schematic block diagram of an audio decoder according to an embodiment of the present invention is shown;

[0048] Fig. 6A and Figure 6B A schematic block diagram of an audio decoder according to another embodiment of the present invention is shown;

[0049] Figure 7 A flow chart showing a method for providing an encoded representation based on at least four audio channel signals according to an embodiment of the present invention is shown;

[0050] Figure 8 A flow chart showing a method for providing at least four audio channel signals based on an encoded representation according to an embodiment of the present invention is shown;

[0051] Fig. 9 A flow chart showing a method for providing an encoded representation based on at least four audio channel signals according to an embodiment of the present invention; and

[0052] Fig.10 A flow chart showing a method for providing at least four audio channel signals based on an encoded representation according to an embodiment of the present invention is shown;

[0053] Fig.11 A schematic block diagram of an audio encoder according to an embodiment of the present invention is shown;

[0054] Fig.12 A schematic block diagram of an audio encoder according to another embodiment of the present invention is shown;

[0055] Fig.13 A schematic block diagram showing an audio decoder according to an embodiment of the present invention;

[0056] Fig.14A The syntax representation of the bit stream is shown, which can be used according to Fig.13 Used with the audio encoder of

[0057] Fig. 14B A tabular representation showing different values ​​of the parameter qceIndex is shown;

[0058] Fig.15 shows a schematic block diagram of a 3D audio encoder in which the concepts according to the present invention may be used;

[0059] Fig.16 A schematic block diagram showing a 3D audio decoder in which the concepts according to the present invention may be used; and

[0060] Fig.17 A schematic block diagram of a format converter is shown.

[0061] Fig.18 shows a diagrammatic representation of the topology of a quad channel element (QCE) according to an embodiment of the present invention;

[0062] Fig.19 A schematic block diagram of an audio decoder according to an embodiment of the present invention is shown;

[0063] Fig. 20 A detailed schematic block diagram of a QCE decoder according to an embodiment of the present invention is shown; and

[0064] Fig.21 A detailed schematic block diagram of a four-channel encoder according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0065] 1. According to Figure 1 Audio Codec

[0066] Figure 1A schematic block diagram of an audio encoder is shown, which is designated in its entirety at 100. The audio encoder 100 is configured to provide an encoded representation based on at least four audio channel signals. The audio encoder 100 is configured to receive a first audio channel signal 110, a second audio channel signal 112, a third audio channel signal 114 and a fourth audio channel signal 116. Furthermore, the audio encoder 100 is configured to provide an encoded representation of a first downmix signal 120 and an encoded representation of a second downmix signal 122, and a jointly encoded representation 130 of a residual signal. The audio encoder 100 comprises a residual signal-assisted multi-channel encoder 140, which is configured to jointly encode the first audio channel signal 110 and the second audio channel signal 112 using residual signal-assisted multi-channel coding to obtain the first downmix signal 120 and the first residual signal 142. The audio signal encoder 100 further comprises a residual signal assisted multi-channel encoder 150 configured to jointly encode at least the third audio channel signal 114 and the fourth audio channel signal 116 using residual signal assisted multi-channel coding to obtain a second downmix signal 122 and a second residual signal 152. The audio decoder 100 further comprises a multi-channel encoder 160 configured to jointly encode the first residual signal 142 and the second residual signal 152 using multi-channel coding to obtain a jointly encoded representation 130 of the residual signals 142, 152.

[0067] With regard to the functionality of the audio encoder 100, it should be noted that the audio encoder 100 performs layered coding, wherein a first audio channel signal 110 and a second audio channel signal 112 are jointly encoded using a residual signal-assisted multi-channel encoding 140, wherein both a first downmix signal 120 and a first residual signal 142 are provided. The first residual signal 142 may, for example, describe the difference between the first audio channel signal 110 and the second audio channel signal 112, and / or may describe some or any signal features that cannot be represented by the first downmix signal 120 and optional parameters that may be provided by the residual signal-assisted multi-channel encoder 140. In other words, the first residual signal 142 may be a refined residual signal that takes into account a decoding result that may be obtained based on the first downmix signal 120 and any possible parameters that may be provided by the residual signal-assisted multi-channel encoder 140. For example, the first residual signal 142 may take into account at least a partial waveform reconstruction of the first audio channel signal 110 and the second audio channel signal 112 at the audio decoder side when compared to a pure reconstruction of high-order signal characteristics (like, for example, correlation characteristics, covariance characteristics, order difference characteristics, etc.). Similarly, the residual signal-assisted multi-channel encoder 150 provides both the second downmix signal 122 and the second residual signal 152 based on the third audio channel signal 114 and the fourth audio channel signal 116, so that the second residual signal takes into account a refinement of the signal reconstruction of the third audio channel signal 114 and the fourth audio channel signal 116 at the side of the audio decoder. The second residual signal 152 may thus serve the same function as the first residual signal 142. However, if the audio channel signals 110, 112, 114, 116 include some correlation, the first residual signal 142 and the second residual signal 152 are typically also correlated to some extent. Therefore, the joint encoding of the first residual signal 142 and the second residual signal 152 using the multi-channel encoder 160 generally includes high efficiency, because multi-channel encoding of related signals generally reduces the bit rate by exploiting dependencies. Therefore, the first residual signal 142 and the second residual signal 152 can be encoded with good accuracy while keeping the bit rate of the joint encoded representation 130 of the residual signals reasonably small.

[0068] In short, according to Figure 1 Embodiments of the invention provide layered multi-channel coding, wherein a good reproduction quality can be achieved by using the residual signal assisted multi-channel encoders 140, 150, and wherein the bit rate requirements can be kept moderate by jointly encoding the first residual signal 142 and the second residual signal 152.

[0069] Another optional improvement of the audio encoder 100 is possible. Figure 4 , Fig.11 and Fig.12Some of these improvements are described.However, it should be noted that the audio encoder 100 may also be adapted in parallel with the audio decoder described herein, wherein the functionality of the audio encoder is generally the inverse of that of the audio decoder.

[0070] 2. According to Figure 2 Audio Codec

[0071] Figure 2 A schematic block diagram of an audio decoder, generally designated 200 , is shown.

[0072] The audio decoder 200 is configured to receive an encoded representation comprising a jointly encoded representation 210 of a first residual signal and a second residual signal. The audio decoder 200 also receives representations of a first downmix signal 212 and a second downmix signal 214. The audio decoder 200 is configured to provide a first audio channel signal 220, a second audio channel signal 222, a third audio channel signal 224 and a fourth audio channel signal 226.

[0073] The audio decoder 200 comprises a multi-channel decoder 230 configured to provide a first residual signal 232 and a second residual signal 234 based on a jointly encoded representation 210 of the first residual signal 232 and the second residual signal 234. The audio decoder 200 further comprises a (first) residual signal-assisted multi-channel decoder 240 configured to provide a first audio channel signal 220 and a second audio channel signal 222 based on the first downmix signal 212 and the first residual signal 232 using multi-channel decoding. The audio decoder 200 further comprises a (second) residual signal-assisted multi-channel decoder 250 configured to provide a third audio channel signal 224 and a fourth audio channel signal 226 based on the second downmix signal 214 and the second residual signal 234.

[0074] With regard to the functionality of the audio decoder 200, it should be noted that the audio signal decoder 200 provides a first audio channel signal 220 and a second audio channel signal 222 based on a multi-channel decoding 240 assisted by a (first) common residual signal, wherein the decoding quality of the multi-channel decoding (when compared to a non-residual signal assisted decoding) is improved by a first residual signal 232. In other words, the first downmix signal 212 provides "coarse" information about the first audio channel signal 220 and the second audio channel signal 222, wherein, for example, the difference between the first audio channel signal 220 and the second audio channel signal 222 may be described by (optional) parameters and by the first residual signal 232, which (optional) parameters may be received by the residual signal assisted multi-channel decoder 240. Thus, the first residual signal 232 may, for example, allow for a partial waveform reconstruction of the first audio channel signal 220 and the second audio channel signal 222.

[0075] Similarly, the (second) residual signal assisted multi-channel decoder 250 provides a third audio channel signal 224 and a fourth audio channel signal 226 based on the second downmix signal 214, wherein the second downmix signal 214 may, for example, “roughly” describe the third audio channel signal 224 and the fourth audio channel signal 226. Furthermore, the difference between the third audio channel signal 224 and the fourth audio channel signal 226 may, for example, be described by (optional) parameters and by the second residual signal 234, which (optional) parameters may be received by the (second) residual signal assisted multi-channel decoder 250. Thus, the estimation of the second residual signal 234 may, for example, take into account a partial waveform reconstruction of the third audio channel signal 224 and the fourth audio channel signal 226. Thus, the second residual signal 234 may take into account an enhancement of the reconstruction quality of the third audio channel signal 224 and the fourth audio channel signal 226.

[0076] However, the first residual signal 232 and the second residual signal 234 are derived from the joint encoded representation 210 of the first residual signal and the second residual signal. Such multi-channel decoding performed by the multi-channel decoder 230 allows for high decoding efficiency because the first audio channel signal 220, the second audio channel signal 222, the third audio channel signal 224, and the fourth audio channel signal 226 are typically similar or "correlated". Therefore, the first residual signal 232 and the second residual signal 234 are also typically similar or "correlated", which can be exploited by deriving the first residual signal 232 and the second residual signal 234 from the joint encoded representation 210 using multi-channel decoding.

[0077] Hence, it is possible to obtain a high decoding quality with a moderate bit rate by decoding the residual signals based on the joint encoded representation 210 of the residual signals 232, 234 and by using each of the residual signals for decoding of two or more audio channel signals.

[0078] In summary, the audio decoder 200 allows for high coding efficiency by providing high quality audio channel signals 220 , 222 , 224 , 226 .

[0079] It should be noted that reference will be made to Figure 3 , Figure 5 , Fig. 6A , Figure 6B and Fig.13 hereinafter is used to describe additional features and functions that may be optionally implemented in the audio decoder 200. However, it should be noted that the audio encoder 200 may include the above-mentioned advantages without any additional modifications.

[0080] 3. According to Figure 3 Audio Codec

[0081] Figure 3 A schematic block diagram of an audio decoder according to another embodiment of the present invention is shown. Figure 3 The audio decoders of are all designated by 300. The audio decoder 300 is similar to the audio decoder according to Figure 2 The audio decoder 300 is similar to the audio decoder 200, so that the above explanation also applies. However, the audio decoder 300 is supplemented with additional features and functions when compared with the audio decoder 200, as will be explained below.

[0082] The audio decoder 300 is configured to receive a jointly encoded representation 310 of a first residual signal and a second residual signal. Furthermore, the audio decoder 300 is configured to receive a jointly encoded representation 360 of a first downmix signal and a second downmix signal. Furthermore, the audio decoder 300 is configured to provide a first audio channel signal 320, a second audio channel signal 322, a third audio channel signal 324 and a fourth audio channel signal 326. The audio decoder 300 comprises a multi-channel decoder 330, which is configured to receive the jointly encoded representation 310 of the first residual signal and the second residual signal and to provide, based on the joint encoded representation, a first residual signal 332 and a second residual signal 334. The audio decoder 300 further comprises a (first) residual signal assisted multi-channel decoding 340, which receives the first residual signal 332 and the first downmix signal 312 and provides the first audio channel signal 320 and the second audio channel signal 322. The audio decoder 300 further comprises a (second) residual signal assisted multi-channel decoding 350 which is configured to receive the second residual signal 334 and the second downmix signal 314 and to provide a third audio channel signal 324 and a fourth audio channel signal 326 .

[0083] The audio decoder 300 further comprises a further multi-channel decoder 370 which is configured to receive the jointly encoded representation 360 of the first downmix signal and the second downmix signal and to provide the first downmix signal 312 and the second downmix signal 314 based on the jointly encoded representation.

[0084] In the following, some other specific details of the audio decoder 300 will be described. However, it should be noted that the actual audio decoder does not need to implement all of these additional features and functions in combination. Instead, the features and functions described below can be added to the audio decoder 200 (or any other audio decoder) individually to gradually improve the audio decoder 200 (or any other audio decoder).

[0085] In a preferred embodiment, the audio decoder 300 receives a joint coded representation 310 of the first residual signal and the second residual signal, wherein the joint coded representation 310 may include a downmix signal of the first residual signal 332 and the second residual signal 334, and a common residual signal of the first residual signal 332 and the second residual signal 334. In addition, the joint coded representation 310 may, for example, include one or more prediction parameters. Thus, the multi-channel decoder 330 may be a multi-channel decoder assisted by a predicted residual signal. For example, the multi-channel decoder 330 may be a USAC complex stereo prediction as described in the "Complex Stereo Prediction" section of the international standard ISO / IEC 23003-3:2012, for example. For example, the multi-channel decoder 330 may be configured to estimate prediction parameters that describe the contribution of signal components derived using signal components of a previous frame to providing the first residual signal 332 and the second residual signal 334 of the current frame. Furthermore, the multi-channel decoder 330 may be configured to apply the common residual signal (which is included in the joint coded representation 310) with a first sign to obtain a first residual signal 332, and to apply the common residual signal (which is included in the joint coded representation 310) with a second sign opposite to the first sign to obtain a second residual signal 334. Thus, the common residual signal may at least partially describe the difference between the first residual signal 332 and the second residual signal 334. However, the multi-channel decoder 330 may estimate the downmix signal, the common residual signal, and one or more prediction parameters (which are all included in the joint coded representation 310) to obtain the first residual signal 332 and the second residual signal 334, as described in the international standard ISO / IEC 23003-3:2012 cited above. Furthermore, it should be noted that the first residual signal 332 may be associated with a first horizontal position (or azimuth position), such as a left horizontal position, and the second residual signal 334 may be associated with a second horizontal position (or azimuth position), such as a right horizontal position, of the audio scene.

[0086] The joint coded representation 360 of the first down-mix signal and the second down-mix signal preferably includes down-mix signals of the first down-mix signal and the second down-mix signal, a common residual signal of the first down-mix signal and the second down-mix signal, and one or more prediction parameters. In other words, there is a "common" down-mix signal to which the first down-mix signal 312 and the second down-mix signal 314 are down-mixed, and there is a "common" residual signal that can at least partially describe the difference between the first down-mix signal 312 and the second down-mix signal 314. The multi-channel decoder 370 is preferably a multi-channel decoder assisted by a predicted residual signal, such as a USAC complex stereo prediction decoder. In other words, the multi-channel decoder 370 providing the first down-mix signal 312 and the second down-mix signal 314 can be substantially the same as the multi-channel decoder 330 providing the first residual signal 332 and the second residual signal 334, so that the above explanation and references are also applicable. Furthermore, it should be noted that the first downmix signal 312 is preferably associated with a first horizontal position or azimuth position of the audio scene (e.g., a left horizontal position or azimuth position), and the second downmix signal 314 is preferably associated with a second horizontal position or azimuth position of the audio scene (e.g., a right horizontal position or azimuth position). Thus, the first downmix signal 312 and the first residual signal 332 may be associated with the same first horizontal position or azimuth position (e.g., a left horizontal position), and the second downmix signal 314 and the second residual signal 334 may be associated with the same second horizontal position or azimuth position (e.g., a right horizontal position). Thus, both the multi-channel decoder 370 and the multi-channel decoder 330 may perform horizontal division (or horizontal separation or horizontal distribution).

[0087] The residual signal assisted multi-channel decoder 340 may preferably be parameter-based and may thus receive one or more parameters 342 describing a desired correlation between two channels (e.g. between a first audio channel signal 320 and a second audio channel signal 322) and / or a level difference between the two channels. For example, the residual signal assisted multi-channel decoding 340 may be based on MPEG surround coding with residual signal extension (as described, for example, in ISO / IEC 23003-1:2007), or a "unified stereo decoding" decoder (as described, for example, in ISO / IEC 23003-3, chapter 7.11 (decoder) and annex B.21 (description of encoder and definition of the term "unified stereo"). Thus, the residual signal assisted multi-channel decoder 340 may provide a first audio channel signal 320 and a second audio channel signal 322, wherein the first audio channel signal 320 and the second audio channel signal 322 are associated with vertically adjacent positions of the audio scene. For example, the first audio channel signal may be associated with the lower left position of the audio scene, and the second audio channel signal may be associated with the upper left position of the audio scene (so that the first audio channel signal 320 and the second audio channel signal 322 are associated with the same horizontal position or azimuth position of the audio scene, or with azimuth positions that are no more than 30 degrees apart, for example). In other words, the residual signal-assisted multi-channel decoder 340 may perform vertical division (or distribution, or separation).

[0088] The functionality of the residual signal assisted multi-channel decoder 350 may be the same as that of the residual signal assisted multi-channel decoder 340, wherein the third audio channel signal may, for example, be associated with a lower right position of the audio scene, and the fourth audio channel signal may, for example, be associated with an upper right position of the audio scene. In other words, the third audio channel signal and the fourth audio channel signal may be associated with vertically adjacent positions of the audio scene, and may be associated with the same horizontal position or azimuth position of the audio scene, wherein the residual signal assisted multi-channel decoder 350 performs vertical division (or separation, or distribution).

[0089] In summary, according to Figure 3The audio decoder 300 of WO 2014 / 036698 performs layered audio decoding, wherein left-right division is performed in the first stage (multi-channel decoder 330, multi-channel decoder 370), and wherein upper-lower division is performed in the second stage (residual signal assisted multi-channel decoder 340, 350). In addition, the residual signals 332, 334 are encoded using the joint coded representation 310, and the downmix signals 312, 314 are encoded (using the joint coded representation 360). Thus, the correlation between the different channels is used for both the encoding (and decoding) of the downmix signals 312, 314 and the encoding (and decoding) of the residual signals 332, 334. Thus, a high coding efficiency is achieved, and the correlation between the signals is also exploited.

[0090] 4. According to Figure 4 Audio Codec

[0091] Figure 4 FIG. 4 is a schematic block diagram of an audio encoder according to another embodiment of the present invention. Figure 4 The audio encoder of FIG. 400 is all designated by 400. The audio encoder 400 is configured to receive four audio channel signals, namely a first audio channel signal 410, a second audio channel signal 412, a third audio channel signal 414 and a fourth audio channel signal 416. Furthermore, the audio encoder 400 is configured to provide an encoded representation based on the audio channel signals 410, 412, 414 and 416, wherein the encoded representation comprises a jointly encoded representation 420 of the two downmix signals, and an encoded representation of a first set 422 of common bandwidth extension parameters and a second set 424 of common bandwidth extension parameters. The audio encoder 400 comprises a first bandwidth extension parameter extractor 430, which is configured to obtain the first set 422 of common bandwidth extraction parameters based on the first audio channel signal 410 and the third audio channel signal 414. The audio encoder 400 further comprises a second bandwidth extension parameter extractor 440 configured to obtain a second set 424 of common bandwidth extension parameters based on the second audio channel signal 412 and the fourth audio channel signal 416 .

[0092] Furthermore, the audio encoder 400 comprises a (first) multi-channel encoder 450, which is configured to jointly encode at least a first audio channel signal 410 and a second audio channel signal 412 using multi-channel coding to obtain a first downmix signal 452. Furthermore, the audio encoder 400 further comprises a (second) multi-channel encoder 460, which is configured to jointly encode at least a third audio channel signal 414 and a fourth audio channel signal 416 using multi-channel coding to obtain a second downmix signal 462. Furthermore, the audio encoder 400 further comprises a (third) multi-channel encoder 470, which is configured to jointly encode the first downmix signal 452 and the second downmix signal 462 using multi-channel coding to obtain a jointly encoded representation 420 of the downmix signals.

[0093] With regard to the functionality of the audio encoder 400, it should be noted that the audio encoder 400 performs layered multi-channel encoding, wherein a first audio channel signal 410 and a second audio channel signal 412 are combined in a first stage, and a third audio channel signal 414 and a fourth audio channel signal 416 are also combined in a first stage, to thereby obtain a first down-mix signal 452 and a second down-mix signal 462. The first down-mix signal 452 and the second down-mix signal 462 are then jointly encoded in a second stage. However, it should be noted that the first bandwidth extension parameter extractor 430 provides a first set 422 of common bandwidth extraction parameters based on the audio channel signals 410, 414 processed by different multi-channel encoders 450, 460 in the first stage of the layered multi-channel encoding. Similarly, the second bandwidth extension parameter extractor 440 provides a second set 424 of common bandwidth extraction parameters based on different audio channel signals 412, 416 processed by different multi-channel encoders 450, 460 in the first processing stage. This particular processing order brings the following advantage: The sets 422, 424 of bandwidth extension parameters are based on channels that are combined only in the second stage of hierarchical encoding (i.e., in the multi-channel encoder 470). This is advantageous because it is desirable to combine such audio channels in the first stage of hierarchical encoding, the relationship of which is not extremely relevant with respect to the sound source position perception. On the contrary, it is recommended that the relationship between the first down-mix signal and the second down-mix signal mainly determines the sound source position perception, because the relationship between the first down-mix signal 452 and the second down-mix signal 462 can be better maintained than the relationship between the corresponding audio channel signals 410, 412, 414, 416. In other words, it has been found that it is desirable that the first set 422 of common bandwidth extension parameters is based on two audio channels (audio channel signals) that contribute to the difference of the downmix signals 452, 462, and the second set 424 of common bandwidth extension parameters is provided based on the audio channel signals 412, 416 that also contribute to the difference of the downmix signals 452, 462, which is achieved by the processing of the audio channel signals in the layered multi-channel encoding described above. Therefore, the first set 422 of common bandwidth extension parameters is based on a similar channel relationship when compared to the channel relationship between the first downmix signal 452 and the second downmix signal 462, wherein the channel relationship between the first downmix signal and the second downmix signal usually dominates in the spatial impression produced on the audio decoder side. Therefore, the provision of the first set 422 of bandwidth extension parameters and the provision of the second set 424 of bandwidth extension parameters are extremely suitable for the spatial auditory impression produced on the audio decoder side.

[0094] 5. According to Figure 5 Audio Codec

[0095] Figure 5FIG. 4 is a schematic block diagram of an audio decoder according to another embodiment of the present invention. Figure 5 The audio decoders are all specified with 500.

[0096] The audio decoder 500 is configured to receive a jointly encoded representation 510 of a first downmix signal and a second downmix signal. Furthermore, the audio decoder 500 is configured to provide a first bandwidth extended channel signal 520, a second bandwidth extended channel signal 522, a third bandwidth extended channel signal 524 and a fourth bandwidth extended channel signal 526.

[0097] The audio decoder 500 comprises a (first) multi-channel decoder 530 configured to use multi-channel decoding to provide a first downmix signal 532 and a second downmix signal 534 based on a joint encoded representation 510 of the first downmix signal and the second downmix signal. The audio decoder 500 further comprises a (second) multi-channel decoder 540 configured to use multi-channel decoding to provide at least a first audio channel signal 542 and a second audio channel signal 544 based on the first downmix signal 532. The audio decoder 500 further comprises a (third) multi-channel decoder 550 configured to use multi-channel decoding to provide at least a third audio channel signal 556 and a fourth audio channel signal 558 based on the second downmix signal 544. Furthermore, the audio decoder 500 comprises a (first) multi-channel bandwidth extension 560, which is configured to perform a multi-channel bandwidth extension based on the first audio channel signal 542 and the third audio channel signal 556 to obtain a first bandwidth extended channel signal 520 and a third bandwidth extended channel signal 524. Furthermore, the audio decoder comprises a (second) multi-channel bandwidth extension 570, which is configured to perform a multi-channel bandwidth extension based on the second audio channel signal 544 and the fourth audio channel signal 558 to obtain a second bandwidth extended channel signal 522 and a fourth bandwidth extended channel signal 526.

[0098] With regard to the functionality of the audio decoder 500, it should be noted that the audio decoder 500 performs layered multi-channel decoding, wherein the division between the first downmix signal 532 and the second downmix signal 534 is performed in a first stage of the layered decoding, and in a second stage of the layered decoding, the first audio channel signal 542 and the second audio channel signal 544 are derived from the first downmix signal 532, and in the second stage of the layered decoding, the third audio channel signal 556 and the fourth audio channel signal 558 are derived from the second downmix signal 550. However, both the first multi-channel bandwidth extension 560 and the second multi-channel bandwidth extension 570 each receive one audio channel signal derived from the first downmix signal 532, and one audio channel signal derived from the second downmix signal 534. Since a good channel separation is usually achieved by the (first) multi-channel decoding 530 (performed as the first stage of layered multi-channel decoding), when compared with the second stage of layered decoding, it can be seen that each multi-channel bandwidth extension 560, 570 receives a well-separated input signal (because the input signal originates from the well-channel-separated first down-mix signal 532 and the second down-mix signal 534). Thus, the multi-channel bandwidth extension 560, 570 can take into account the stereo characteristics, which are important for the auditory impression and which are well represented by the relationship between the first down-mix signal 532 and the second down-mix signal 534, and the multi-channel bandwidth extension can therefore provide a good auditory impression.

[0099] In other words, the "crossover" structure of the audio decoders allows for a good multi-channel bandwidth extension, which takes into account the stereo relationship between the channels, wherein each of the multi-channel bandwidth extension stages 560, 570 receives input signals from both (second stage) multi-channel decoders 540, 550.

[0100] However, it should be noted that the audio decoder 500 may be implemented as described herein with respect to Figure 2 , Figure 3 , according to 6 and Fig.13 The audio decoder 500 may be supplemented by any of the features and functions described in the audio decoder, wherein it is possible to introduce corresponding features into the audio decoder 500 to gradually improve the performance of the audio decoder.

[0101] 6. According to Fig. 6A and Figure 6B Audio Codec

[0102] Fig. 6A and Figure 6B FIG. 4 is a schematic block diagram of an audio decoder according to another embodiment of the present invention. Fig. 6A and Figure 6B All audio decoders are specified with 600. Fig. 6A and Figure 6B The audio decoder 600 is similar to the one according to Figure 5 The audio decoder 600 is also described in detail with reference to FIG. 5 , so that the above explanations also apply. However, the audio decoder 600 has been supplemented by some features and functionalities which can also be introduced into the audio decoder 500 for improvement, either individually or in combination.

[0103] The audio decoder 600 is configured to receive a joint coded representation 610 of a first downmix signal and a second downmix signal, and to provide a first bandwidth extended signal 620, a second bandwidth extended signal 622, a third bandwidth extended signal 624, and a fourth bandwidth extended signal 626. The audio decoder 600 comprises a multi-channel decoder 630, which is configured to receive a joint coded representation 610 of the first downmix signal and the second downmix signal, and to provide a first downmix signal 632 and a second downmix signal 634 based on the joint coded representation. The audio decoder 600 further comprises a multi-channel decoder 640, which is configured to receive the first downmix signal 632, and to provide a first audio channel signal 542 and a second audio channel signal 544 based on the first downmix signal. The audio decoder 600 further comprises a multi-channel decoder 650, which is configured to receive the second downmix signal 634, and to provide a third audio channel signal 656 and a fourth audio channel signal 658. The audio decoder 600 further comprises a (first) multi-channel bandwidth extension 660, which is configured to receive the first audio channel signal 642 and the third audio channel signal 656 and, based on the first audio channel signal and the third audio channel signal, provide a first bandwidth-extended channel signal 620 and a third bandwidth-extended channel signal 624. Furthermore, a (second) multi-channel bandwidth extension 670 receives the second audio channel signal 644 and the fourth audio channel signal 658 and, based on the second audio channel signal and the fourth audio channel signal, provides a second bandwidth-extended channel signal 622 and a fourth bandwidth-extended channel signal 626.

[0104] The audio decoder 600 further comprises a further multi-channel decoder 680 configured to receive a jointly encoded representation 682 of the first residual signal and the second residual signal and to provide, based on the jointly encoded representation, a first residual signal 684 for use by the multi-channel decoder 640 and a second residual signal 686 for use by the multi-channel decoder 650.

[0105] The multi-channel decoder 630 is preferably a multi-channel decoder assisted by a residual signal based on prediction. For example, the multi-channel decoder 630 may be substantially the same as the multi-channel decoder 370 described above. For example, the multi-channel decoder 630 may be a USAC complex stereo prediction decoder as described above and as described in the USAC standard cited above. Thus, the joint coded representation 610 of the first down-mix signal and the second down-mix signal may, for example, include a (common) down-mix signal of the first down-mix signal and the second down-mix signal, a (common) residual signal of the first down-mix signal and the second down-mix signal, and one or more prediction parameters, which are estimated by the multi-channel decoder 630.

[0106] Furthermore, it should be noted that the first downmix signal 632 may, for example, be associated with a first horizontal position or azimuth position of the audio scene (e.g., a left horizontal position), and the second downmix signal 634 may, for example, be associated with a second horizontal position or azimuth position of the audio scene (e.g., a right horizontal position).

[0107] In addition, the multi-channel decoder 680 may, for example, be a multi-channel decoder associated with a predicted residual signal. The multi-channel decoder 680 may be substantially the same as the multi-channel decoder 330 described above. For example, the multi-channel decoder 680 may be a USAC complex stereo prediction decoder, as mentioned above. Therefore, the joint coded representation 682 of the first residual signal and the second residual signal may include a (common) down-mix signal of the first residual signal and the second residual signal, a (common) residual signal of the first residual signal and the second residual signal, and one or more prediction parameters, which are estimated by the multi-channel decoder 680. In addition, it should be noted that the first residual signal 684 may be associated with a first horizontal position or azimuth position of the audio scene (e.g., a left horizontal position), and the second residual signal 686 may be associated with a second horizontal position or azimuth position of the audio scene (e.g., a right horizontal position).

[0108] The multi-channel decoder 640 may for example be a parameter-based multi-channel decoding, similar to for example MPEG surround multi-channel decoding as described above and as described in the referenced standards. However, in the presence of the (optional) multi-channel decoder 680 and the (optional) first residual signal 684, the multi-channel decoder 640 may be a parameter-based, residual signal-assisted multi-channel decoder, similar to for example a unified stereo decoder. Thus, the multi-channel decoder 640 may be substantially the same as the multi-channel decoder 340 described above, and the multi-channel decoder 640 may for example receive the parameters 342 described above.

[0109] Similarly, the multi-channel decoder 650 may be substantially identical to the multi-channel decoder 640. Thus, the multi-channel decoder 650 may be, for example, parameter-based and optionally residual signal-assisted (in case the optional multi-channel decoder 680 is present).

[0110] Furthermore, it should be noted that the first audio channel signal 642 and the second audio channel signal 644 are preferably associated with vertically adjacent spatial positions of the audio scene. For example, the first audio channel signal 642 is associated with the lower left position of the audio scene, and the second audio channel signal 644 is associated with the upper left position of the audio scene. Thus, the multi-channel decoder 640 performs a vertical division (or separation, or distribution) of the audio content described by the first downmix signal 632 (and, optionally, by the first residual signal 684). Similarly, the third audio channel signal 656 and the fourth audio channel signal 658 are associated with vertically adjacent positions of the audio scene, and are preferably associated with the same horizontal position or azimuth position of the audio scene. For example, the third audio channel signal 656 is preferably associated with the lower right position of the audio scene, and the fourth audio channel signal 658 is preferably associated with the upper right position of the audio scene. Thus, the multi-channel decoder 650 performs a vertical division (or separation, or distribution) of the audio content described by the second downmix signal 634 (and, optionally, by the second residual signal 686).

[0111] However, the first multi-channel bandwidth extension 660 receives the first audio channel signal 642 and the third audio channel 656, which are associated with the lower left position and the lower right position of the audio scene. Therefore, the first multi-channel bandwidth extension 660 performs multi-channel bandwidth extension based on two audio channel signals associated with the same horizontal plane (e.g., lower horizontal plane) or height of the audio scene and different sides (left / right) of the audio scene. Therefore, when performing bandwidth extension, the multi-channel bandwidth extension can take into account stereo characteristics (e.g., human stereo perception). Similarly, the second multi-channel bandwidth extension 670 can also take into account stereo characteristics because the second multi-channel bandwidth extension operates on audio channel signals at the same horizontal plane (e.g., upper horizontal plane) or height of the audio scene but at different horizontal positions (different sides) (left / right).

[0112] To summarize further, the layered audio decoder 600 includes the following structure: left / right division (or separation, or distribution) is performed in the first stage (multi-channel decoding 630, 680), vertical division (separation or distribution) is performed in the second stage (multi-channel decoding 640, 650), and multi-channel bandwidth extension operates on a pair of left / right signals (multi-channel bandwidth extension 660, 670). This "crossing" of the decoding path allows the left / right separation, which is particularly important for the auditory impression (for example, more important than the upper / lower division), to be performed in the first processing stage of the layered audio decoder, and the multi-channel bandwidth extension can also be performed on a pair of left and right audio channel signals, which in turn leads to a particularly good auditory impression. The upper / lower division is performed as an intermediate stage between the left and right separation and the multi-channel bandwidth extension, which makes it possible to derive four audio channel signals (or bandwidth-extended channel signals) without significantly degrading the auditory impression.

[0113] 7. According to Figure 7 Methods

[0114] Figure 7 A flow chart of a method 700 for providing an encoded representation based on at least four audio channel signals is shown.

[0115] The method 700 comprises jointly encoding 710 at least a first audio channel signal and a second audio channel signal using residual signal-assisted multi-channel coding to obtain a first down-mix signal and a first residual signal. The method further comprises jointly encoding 720 at least a third audio channel signal and a fourth audio channel signal using residual signal-assisted multi-channel coding to obtain a second down-mix signal and a second residual signal. The method further comprises jointly encoding 730 the first residual signal and the second residual signal using multi-channel coding to obtain an encoded representation of the residual signal. However, it should be noted that the method 700 may be supplemented by any of the features and functionalities described herein with respect to the audio encoder and the audio decoder.

[0116] 8. According to Figure 8 Methods

[0117] Figure 8 A flow chart of a method 800 for providing at least four audio channel signals based on an encoded representation is shown.

[0118] The method 800 includes using multi-channel decoding to provide 810 a first residual signal and a second residual signal based on a jointly encoded representation of the first residual signal and the second residual signal. The method 800 also includes using residual signal-assisted multi-channel decoding to provide 820 a first audio channel signal and a second audio channel signal based on the first down-mix signal and the first residual signal. The method also includes using residual signal-assisted multi-channel decoding to provide 830 a third audio channel signal and a fourth audio channel signal based on the second down-mix signal and the second residual signal.

[0119] Furthermore, it should be noted that the method 800 may be supplemented by any of the features and functionalities described herein with respect to the audio decoder and the audio encoder.

[0120] 9. According to Fig. 9 Methods

[0121] Fig. 9 A flow chart of a method 900 for providing an encoded representation based on at least four audio channel signals is shown.

[0122] The method 900 comprises obtaining 910 a first set of common bandwidth extension parameters based on a first audio channel signal and a third audio channel signal. The method 900 also comprises obtaining 920 a second set of common bandwidth extension parameters based on a second audio channel signal and a fourth audio channel signal. The method further comprises jointly encoding at least the first audio channel signal and the second audio channel signal using multi-channel coding to obtain a first down-mix signal, and jointly encoding 940 at least the third audio channel signal and the fourth audio channel signal using multi-channel coding to obtain a second down-mix signal. The method further comprises jointly encoding 950 the first down-mix signal and the second down-mix signal using multi-channel coding to obtain an encoded representation of the down-mix signal.

[0123] It should be noted that some of the steps of method 900 not including specific interdependencies may be performed in any order or in parallel. Furthermore, it should be noted that method 900 may be supplemented by any of the features and functionalities described herein with respect to the audio encoder and the audio decoder.

[0124] 10. According to Fig.10 Methods

[0125] Fig.10 A flow chart of a method 1000 for providing at least four audio channel signals based on an encoded representation is shown.

[0126] The method 1000 comprises: using multi-channel decoding, providing 1010 a first down-mix signal and a second down-mix signal based on a jointly encoded representation of the first down-mix signal and the second down-mix signal; using multi-channel decoding, providing 1020 at least a first audio channel signal and a second audio channel signal based on the first down-mix signal; using multi-channel decoding, providing 1030 at least a third audio channel signal and a fourth audio channel signal based on the second down-mix signal; performing 1040 multi-channel bandwidth extension based on the first audio channel signal and the third audio channel signal to obtain a first bandwidth-extended channel signal and a third bandwidth-extended channel signal; and performing 1050 multi-channel bandwidth extension based on the second audio channel signal and the fourth audio channel signal to obtain a second bandwidth-extended channel signal and a fourth bandwidth-extended channel signal.

[0127] It should be noted that some of the steps of the method 1000 may be performed in any order or in parallel. Furthermore, it should be noted that the method 1000 may be supplemented by any of the features and functionalities described herein with respect to the audio encoder and the audio decoder.

[0128] 11. According to Fig.11 , Fig.12 and Fig.13 Example

[0129] In the following, some additional embodiments and underlying considerations according to the present invention will be described.

[0130] Fig.11 1 shows a schematic block diagram of an audio encoder 1100 according to an embodiment of the present invention. The audio encoder 1100 is configured to receive a left bottom channel signal 1110 , a left top channel signal 1112 , a right bottom channel signal 1114 and a right top channel signal 1116 .

[0131] The audio encoder 1100 comprises a first multi-channel audio encoder (or encoding) 1120, which is an MPEG Surround 2-1-2 audio encoder (or encoding) or a unified stereo audio encoder (or encoding), and which receives a left bottom channel signal 1110 and a left top channel signal 1112. The first multi-channel audio encoder 1120 provides a left downmix signal 1122 and (optionally) a left residual signal 1124. Furthermore, the audio encoder 1100 comprises a second multi-channel encoder (or encoding) 1130, which is an MPEG Surround 2-1-2 encoder (or encoding) or a unified stereo encoder (or encoding), and which receives a right bottom channel signal 1114 and a right top channel signal 1116. The second multi-channel audio encoder 1130 provides a right downmix signal 1132 and (optionally) a right residual signal 1134. The audio encoder 1100 further comprises a stereo encoder (or encoding) 1140, which receives a left downmix signal 1122 and a right downmix signal 1132. Furthermore, the first stereo encoding 1140, which is a complex predictive stereo encoding, receives psychoacoustic model information 1142 from a psychoacoustic model. For example, the psychoacoustic model information 1142 may describe psychoacoustic correlations of different frequency bands or subbands, psychoacoustic masking effects, etc. The stereo encoding 1140 provides a channel pair unit (CPE) "downmix", which is designated by 1144 and describes the left downmix signal 1122 and the right downmix signal 1132 in a jointly encoded form. Furthermore, the audio encoder 1100 optionally comprises a second stereo encoder (or encoding) 1150, which is configured to receive an optional left residual signal 1124 and an optional right residual signal 1134, as well as the psychoacoustic model information 1142. The second stereo encoding 1150, being a complex predictive stereo encoding, is configured to provide a channel pair element (CPE) "residual" representing the left residual signal 1124 and the right residual signal 1134 in a jointly coded form.

[0132] The encoder 1100 (as well as other audio encoders described herein) is based on the idea of ​​exploiting horizontal and vertical signal dependencies by hierarchically combining available USAC stereo tools (i.e., coding concepts available in USAC coding). Vertically adjacent channel pairs are combined using MPEG Surround 2-1-2 or unified stereo (designated by 1120 and 1130) with band-limited residual signals or full-band residual signals (designated by 1124 and 1134). The output of each vertical channel pair is a downmix signal 1122, 1132, and for unified stereo is a residual signal 1124, 1134. In order to meet the perceptual requirement of binaural unmasking, both downmix signals 1122, 1132 are horizontally combined and jointly encoded by using complex prediction in the MDCT domain (encoder 1140), which includes the possibility of left-right coding and mid-side coding. The same method can be applied to the horizontally combined residual signals 1124, 1134. This concept is described in detail in the text of the encoder 1100. Fig.11 Shown in.

[0133] refer to Fig.11 The hierarchical structure of the interpretation can be achieved by enabling two stereo tools (e.g., two USAC stereo tools) and re-sorting the channels between the two. Thus, no additional pre-processing / post-processing steps are necessary, and the bitstream syntax for sending the tool's payload remains unchanged (e.g., substantially unchanged when compared to the USAC standard). This idea leads to Fig.12 The encoder structure shown in .

[0134] Fig.12 1 shows a schematic block diagram of an audio encoder 1200 according to an embodiment of the present invention. The audio encoder 1200 is configured to receive a first channel signal 1210, a second channel signal 1212, a third channel signal 1214, and a fourth channel signal 1216. The audio encoder 1200 is configured to provide a bit stream 1220 for a first channel pair unit and a bit stream 1222 for a second channel pair unit.

[0135] The audio encoder 1200 comprises a first multi-channel encoder 1230, which is an MPEG surround 2-1-2 encoder or a unified stereo encoder, and which receives a first channel signal 1210 and a second channel signal 1212. Furthermore, the first multi-channel encoder 1230 provides a first downmix signal 1232, an MPEG surround payload 1236, and (optionally) a first residual signal 1234. The audio encoder 1200 further comprises a second multi-channel encoder 1240, which is an MPEG surround 2-1-2 encoder or a unified stereo encoder, and which receives a third channel signal 1214 and a fourth channel signal 1216. The second multi-channel encoder 1240 provides a first downmix signal 1242, an MPEG surround payload 1246, and (optionally) a second residual signal 1244.

[0136] The audio encoder 1200 also includes a first stereo encoding 1250, which is a complex prediction stereo encoding. The first stereo encoding 1250 receives a first downmix signal 1232 and a second downmix signal 1242. The first stereo encoding 1250 provides a joint coding representation 1252 of the first downmix signal 1232 and the second downmix signal 1242, wherein the joint coding representation 1252 may include a (common) downmix signal (of the first downmix signal 1232 and the second downmix signal 1242) and a representation of a common residual signal (of the first downmix signal 1232 and the second downmix signal 1242). In addition, the (first) complex prediction stereo encoding 1250 provides a complex prediction payload 1254, which generally includes one or more complex prediction coefficients. In addition, the audio encoder 1200 also includes a second stereo encoding 1260, which is a complex prediction stereo encoding. The second stereo encoding 1260 receives the first residual signal 1234 and the second residual signal 1244 (or a zero input value if there is no residual signal provided by the multi-channel encoders 1230, 1240). The second stereo encoding 1260 provides a joint encoded representation 1262 of the first residual signal 1234 and the second residual signal 1244, which may, for example, include a (common) downmix signal (of the first residual signal 1234 and the second residual signal 1244) and a common residual signal (of the first residual signal 1234 and the second residual signal 1244). In addition, the complex prediction stereo encoding 1260 provides a complex prediction payload 1264, which typically includes one or more prediction coefficients.

[0137] In addition, the audio encoder 1200 includes a psychoacoustic model 1270 that provides information for controlling the first complex predictive stereo encoding 1250 and the second complex predictive stereo encoding 1260. For example, the information provided by the psychoacoustic model 1270 may describe which frequency bands or frequency bins have high psychoacoustic relevance and should be encoded with high precision. However, it should be noted that using the information provided by the psychoacoustic model 1270 is optional.

[0138] Furthermore, the audio encoder 1200 comprises a first encoder and multiplexer 1280 which receives the joint coded representation 1252 from the first complex predictive stereo encoding 1250, receives the complex prediction payload 1254 from the first complex predictive stereo encoding 1250 and receives the MPEG surround payload 1236 from the first multi-channel audio encoder 1230. Furthermore, the first encoding and multiplexing 1280 may receive information from the psychoacoustic model 1270 which describes which coding precision should be applied to which frequency bands or subbands, e.g. taking into account psychoacoustic masking effects etc. Thus, the first encoding and multiplexing 1280 provides a first channel pair unit bitstream 1220.

[0139] Furthermore, the audio encoder 1200 includes a second encoding and multiplexing 1290 configured to receive a joint coded representation 1262 provided by the second complex predictive stereo encoding 1260, a complex prediction payload 1264 provided by the second complex predictive stereo encoding 1260, and an MPEG surround payload 1246 provided by the second multi-channel audio encoder 1240. Furthermore, the second encoding and multiplexing 1290 may receive information from the psychoacoustic model 1270. Thus, the second encoding and multiplexing 1290 provides a second channel pair unit bitstream 1222.

[0140] With regard to the functions of the audio encoder 1200, reference is made to the above explanations and also to the explanations regarding the Figure 2 , Figure 3 , Figure 5 and Fig. 6A and Figure 6B An explanation of the audio codec.

[0141] Furthermore, it should be noted that this concept can be extended to the use of multiple MPEG Surround audio frames for joint coding of horizontally related channels, vertically related channels or other geometrically related channels and combining the downmix signal and the residual signal into a complex predicted stereo pair, taking into account their geometric and perceptual properties. This leads to a generalized decoder structure.

[0142] In the following, the implementation of a four-channel unit will be described. In a three-dimensional audio coding system, a hierarchical combination of four channels to form a four-channel unit (QCE) is used. The QCE consists of two USAC channel pair units (CPE) (or provides two USAC channel pair units, or receives two USAC channel pair units). Vertical channel pairs are combined using MPS 2-1-2 or unified stereo. The downmixed channels are jointly coded in the first channel pair unit CPE. If residual coding is applied, the residual signal is jointly coded in the second channel pair unit CPE, otherwise the signal in the second CPE is set to zero. The two channel pair units CPE use complex prediction for joint stereo coding, including the possibility of left and right coding and mid-side coding. In order to retain the perceptual stereo properties of the high-frequency part of the signal, stereo SBR (spectral bandwidth replication) is applied between the upper left / upper right channel pair and the lower left / lower right channel pair by an additional re-sorting step before applying SBR.

[0143] Will refer to Fig.13 Describe possible decoder structures, Fig.13 13 shows a schematic block diagram of an audio decoder according to an embodiment of the present invention. The audio decoder 1300 is configured to receive a first bit stream 1310 representing a first channel pair unit and a second bit stream 1312 representing a second channel pair unit. However, the first bit stream 1310 and the second bit stream 1312 may be included in a common overall bit stream.

[0144] The audio decoder 1300 is configured to provide a first bandwidth-extended channel signal 1320, a second bandwidth-extended channel signal 1322, a third bandwidth-extended channel signal 1324, and a fourth bandwidth-extended channel signal 1326. The first bandwidth-extended channel signal 1320 may, for example, represent a lower left position of an audio scene, the second bandwidth-extended channel signal 1322 may, for example, represent an upper left position of an audio scene; the third bandwidth-extended channel signal 1324 may, for example, be associated with a lower right position of an audio scene; and the fourth bandwidth-extended channel signal 1326 may, for example, be associated with an upper right position of an audio scene.

[0145] The audio decoder 1300 includes a first bitstream decoding 1330 configured to receive a bitstream 1310 for a first channel pair unit and to provide, based on the bitstream, a jointly coded representation of two downmix signals, a complex prediction payload 1334, an MPEG surround payload 1336, and a spectral bandwidth replication payload 1338. The audio decoder 1300 also includes a first complex predictive stereo decoding 1340 configured to receive the jointly coded representation 1332 and the complex prediction payload 1334 and to provide, based on the jointly coded representation and the complex prediction payload, a first downmix signal 1342 and a second downmix signal 1344. Similarly, the audio decoder 1300 includes a second bitstream decoding 1350 configured to receive the bitstream 1312 for the second channel unit and to provide, based on the bitstream, a joint coded representation 1352 of two residual signals, a complex prediction payload 1354, an MPEG surround payload 1356, and a spectral bandwidth replication bit load 1358. The audio decoder also includes a second complex predictive stereo decoding 1360 that provides, based on the joint coded representation 1352 and the complex prediction payload 1354, a first residual signal 1362 and a second residual signal 1364.

[0146] Furthermore, the audio decoder 1300 comprises a first MPEG surround type multi-channel decoding 1370, which is an MPEG surround 2-1-2 decoding or a unified stereo decoding. The first MPEG surround type multi-channel decoding 1370 receives a first downmix signal 1342, a first residual signal 1362 (optional) and an MPEG surround payload 1336, and provides a first audio channel signal 1372 and a second audio channel signal 1374 based on the first downmix signal, the first residual signal and the MPEG surround payload. The audio decoder 1300 further comprises a second MPEG surround type multi-channel decoding 1380, which is an MPEG surround 2-1-2 multi-channel decoding or a unified stereo multi-channel decoding. The second MPEG surround type multi-channel decoding 1380 receives the second downmix signal 1344 and the second residual signal 1364 (optional), and the MPEG surround payload 1356, and provides a third audio channel signal 1382 and a fourth audio channel signal 1384 based on the second downmix signal, the second residual signal and the MPEG surround payload. The audio decoder 1300 also includes a first stereo spectral bandwidth replication 1390, which is configured to receive the first audio channel signal 1372 and the third audio channel signal 1382, and the spectral bandwidth replication payload 1338, and provides a first bandwidth extended channel signal 1320 and a third bandwidth extended channel signal 1324 based on the first audio channel signal, the third audio channel signal and the spectral bandwidth replication payload. In addition, the audio decoder includes a second stereo spectral bandwidth replication 1394, which is configured to receive the second audio channel signal 1374 and the fourth audio channel signal 1384, and the spectral bandwidth replication payload 1358, and provide a second bandwidth-extended channel signal 1322 and a fourth bandwidth-extended channel signal 1326 based on the second audio channel signal, the fourth audio channel signal and the spectral bandwidth replication payload.

[0147] Regarding the functions of the audio decoder 1300, reference is made to the above discussion, and also to the Figure 2 , Figure 3 , Figure 5 and Fig. 6A and Figure 6B A discussion of audio decoders.

[0148] In the following, reference will be made to Fig.14A and Fig. 14B1 to describe an example of a bitstream that can be used for audio encoding / decoding as described herein. It should be noted that the bitstream may, for example, be an extension of the bitstream used in Unified Speech and Audio Coding (USAC), which is described in the above-mentioned standard (ISO / IEC 23003-3:2012). For example, the MPEG surround sound payloads 1236, 1246, 1336, 1356 and the complex prediction payloads 1254, 1264, 1334, 1354 may be sent as conventional channel pair units (i.e., for channel pair units according to the USAC standard). For signaling the use of four channel units QCE, the USAC channel pair configuration may be extended by two bits, such as Fig.14A In other words, two bits specified by "qceIndex" may be added to the USAC bitstream element "UsacChannelPairElementConfig()". Fig. 14B The meaning of the parameter represented by the bit "qceindex" is defined as shown in the table of FIG.

[0149] For example, two channel pair units forming a QCE may be sent as consecutive units, first containing the downmixed channels and the CPE for the MPS payload of the first MPS box, and second containing the residual signal (or the zero audio signal for MPS 2-1-2 encoding) and the CPE for the MPS payload of the second MPS box.

[0150] In other words, there is only a small signaling overhead when compared to the conventional USAC bitstream for transmitting the four channel elements QCE.

[0151] However, naturally also different bitstream formats may be used.

[0152] 12. Encoding / Decoding Environment

[0153] In the following, an audio encoding / decoding environment to which the concept according to the present invention can be applied will be described.

[0154] The 3D audio codec system in which the concepts according to the present invention can be used is based on the MPEG-D USAC codec for decoding of channel and object signals. To improve the efficiency of encoding a large number of objects, the MPEG SAOC technology has been adapted. Three types of renderers perform the tasks of rendering objects to channels, rendering channels to headphones or rendering channels to different speaker settings. When object signals are sent explicitly or encoded using SAOC parameterization, the corresponding object metadata information is compressed and multiplexed into the 3D audio bitstream.

[0155] Fig.15 A schematic block diagram of such an audio encoder is shown, and Fig.16A schematic block diagram of such an audio decoder is shown. In other words, Fig.15 and Fig.16 Different algorithm blocks of a 3D audio system are shown.

[0156] refer to Fig.15 Now I will explain some details. Fig.15 15 shows a schematic block diagram of a 3D audio encoder 1500. The encoder 1500 comprises an optional pre-renderer / mixer 1510, which receives one or more channel signals 1512 and one or more object signals 1514 and provides one or more channel signals 1516 and one or more object signals 1518, 1520 based on the one or more channel signals and the one or more object signals. The audio encoder further comprises a USAC encoder 1530 and (optionally) an SAOC encoder 1540. The SAOC encoder 1540 is configured to provide one or more SAOC transport channels 1542 and SAOC sideband information 1544 based on one or more objects 1520 provided to the SAOC encoder. In addition, the USAC encoder 1530 is configured to receive a channel signal 1516 including channels and pre-rendered objects from a pre-renderer / mixer, receive one or more object signals 1518 from the pre-renderer / mixer, and receive one or more SAOC transmission channels 1542 and SAOC side information 1544, and based on the above, provide an encoded representation 1532. In addition, the audio encoder 1500 also includes an object metadata encoder 1550, which is configured to receive object metadata 1552 (which object metadata may be estimated by the pre-renderer / mixer 1510) and encode the object metadata to obtain encoded object metadata 1554. The encoded metadata is also received by the USAC encoder 1530 and used to provide the encoded representation 1532.

[0157] Some details regarding the various components of the audio encoder 1500 are described below.

[0158] Reference now Fig.16 , an audio decoder 1600 will be described. The audio decoder 1600 is configured to receive an encoded representation 1610 and to provide a multi-channel speaker signal 1612, a headphone signal 1614 and / or a speaker signal 1616 in an alternative format (eg, a 5.1 format) based on the encoded representation.

[0159] The audio decoder 1600 includes a USAC decoder 1620 and provides one or more channel signals 1622, one or more pre-rendered object signals 1624, one or more object signals 1626, one or more SAOC transmission channels 1628, SAOC side information 1630, and compressed object metadata information 1632 based on the encoded representation 1610. The audio decoder 1600 also includes an object renderer 1640, which is configured to provide one or more rendered object signals 1642 based on the object signals 1626 and the object metadata information 1644, wherein the object metadata information 1644 is provided by an object metadata decoder 1650 based on the compressed object metadata information 1632. The audio decoder 1600 also includes (optionally) an SAOC decoder 1660, which is configured to receive the SAOC transmission channels 1628 and the SAOC side information 1630, and to provide one or more rendered object signals 1662 based on the SAOC transmission channels and the SAOC side information. The audio decoder 1600 further comprises a mixer 1670 configured to receive the channel signal 1622, the pre-rendered object signal 1624, the rendered object signal 1642, and the rendered object signal 1662, and based thereon provide a plurality of mixed channel signals 1672, which may, for example, constitute the multi-channel speaker signal 1612. The audio decoder 1600 may, for example, further comprise a binaural rendering 1680, which is configured to receive the mixed channel signal 1672 and, based thereon, provide a headphone signal 1614. Furthermore, the audio decoder 1600 may comprise a format conversion 1690, which is configured to receive the mixed channel signal 1672 and the reproduction layout information 1692, and, based thereon, provide a speaker signal 1616 for an alternative speaker arrangement.

[0160] In the following, some details regarding the components of the audio encoder 1500 and the audio decoder 1600 will be described.

[0161] Pre-renderer / Mixer

[0162] The pre-renderer / mixer 1510 is optionally used to convert the channel plus object input scene into a channel scene before encoding. Functionally, the pre-renderer / mixer may be the same as the object renderer / mixer described below. Pre-rendering of objects can, for example, ensure a certain signal entropy at the encoder input, which is basically independent of the number of object signals that are valid at the same time. In the pre-rendering of objects, no object metadata is required to be sent. Discreet object signals are rendered to the channel layout configured for use by the encoder. The weight of the object for each channel is obtained from the associated object metadata (OAM) 1552.

[0163] USAC core codec

[0164] The core codecs 1530, 1620 for speaker channel signals, discreet object signals, object downmix signals and pre-rendered signals are based on MPEG-D USAC technology. The core codec handles the encoding of a large number of signals by creating channel and object mapping information based on geometric information and semantic information assigned to input channels and objects. The mapping information describes how the input channels and objects are mapped to USAC channel units (CPE, SCE, LFE) and how the corresponding information is sent to the decoder. All additional payloads (such as SAOC data or object metadata) have passed through extension units and have been considered in the encoder rate control.

[0165] Objects may be encoded in different ways, depending on the rate / distortion requirements placed on the renderer and the interactivity requirements. The following object encoding variants are possible:

[0166] 1. Pre-rendered objects: Object signals are pre-rendered and mixed into 22.2 channel signals before encoding. The subsequent encoding chain refers to 22.2 channel signals.

[0167] 2. Discreet object wave form: The object is supplied to the encoder as a mono wave form. In addition to the channel signal, the encoder uses a mono unit SCE to deliver the object. The decoded object is rendered and mixed at the receiver side. Compressed object metadata information is sent along the side to the receiver / renderer.

[0168] 3. Parametric object wave form: Object properties and their relationships to each other are described by SAOC parameters. Downmix of object signals is encoded using USAC. Parameter information is sent along the side. The number of downmix channels is selected depending on the number of objects and the overall data rate. Compressed object metadata information is sent to the SAOC renderer.

[0169] SAOC

[0170] The SAOC encoder 1540 and SAOC decoder 1660 for object signals are based on MPEG SAOC technology. The system can recreate, modify and render many audio objects based on a smaller number of transmission channels and additional parameter data (object level difference OLD, inter-object correlation IOC, down-mixing gain DMG). The additional parameter data shows a data rate significantly reduced than the data rate required for sending all objects separately, making encoding extremely effective. The SAOC encoder uses object / channel signals (such as monophonic waveforms) as input, and outputs parameter information (the parameter information is encapsulated in 3D audio bitstreams 1532, 1610) and SAOC transmission channels (encoded and sent using monophonic units).

[0171] The SAOC decoder 1600 reconstructs the object / channel signals from the decoded SAOC transport channels 1628 and parameter information 1630 and generates an output audio scene based on the reproduction layout, the decompressed object metadata information and optionally based on user interaction information.

[0172] Object Metadata Codec

[0173] For each object, associated metadata specifying the object's geometric position and volume in 3D space is efficiently encoded through quantization of the object's properties in time and space.The compressed object metadata cOAM 1554, 1632 is sent to the receiver as sideband information.

[0174] Object Renderer / Mixer

[0175] The object renderer uses compressed object metadata to generate object waveforms according to a given reproduction format. Each object is rendered to certain output channels according to its metadata. The output of this box comes from the sum of partial results. If channel-based content and discreet object / parameter objects are decoded, the channel-based waveforms and the rendered object waveforms are mixed before outputting the generated waveforms (or before feeding the generated waveforms to a post-processor module such as a binaural renderer or speaker renderer module).

[0176] Binaural Renderer

[0177] The binaural renderer module 1680 generates a binaural downmix of the multi-channel audio material such that each input channel is represented by a virtual sound source. The processing is performed frame by frame in the QMF domain. The binauralization is based on measured binaural spatial impulse responses.

[0178] Speaker Renderer / Format Conversion

[0179] The speaker renderer 1690 converts between the transmit channel configuration and the desired reproduction format. The speaker renderer is therefore referred to as a "format converter" hereinafter. The format converter performs the conversion to a lower number of output channels, i.e., it creates a downmix. The system automatically generates an optimal downmix matrix for a given combination of input format and output format and applies the matrix in the downmix process. The format converter takes into account standard speaker configurations and takes into account random configurations with non-standard speaker positions.

[0180] Fig.1717 shows a schematic block diagram of a format converter. As can be seen in the figure, the format converter 1700 receives a mixer output signal 1710, such as a mixed channel signal 1672, and provides a speaker signal 1712, such as a speaker signal 1616. The format converter includes a downmixing configurator 1730 and a downmixing process 1720 in the QMF domain, wherein the downmixing configurator provides configuration information for the downmixing process 1720 based on the mixer output layout information 1732 and the reproduction layout information 1734.

[0181] Furthermore, it should be noted that the concepts described above, such as the audio encoder 100, the audio decoder 200 or 300, the audio encoder 400, the audio decoder 500 or 600, the methods 700, 800, 900 or 1000, the audio encoder 1100 or 1200, and the audio decoder 1300, may be used within the audio encoder 1500 and / or within the audio decoder 1600. For example, the previously mentioned audio encoder / decoder may be used for encoding or decoding of channel signals associated with different spatial positions.

[0182] 13. Alternative Embodiments

[0183] In the following, some additional embodiments will be described.

[0184] Reference Figures 18 to 21 , additional embodiments according to the present invention will be explained.

[0185] It should be noted that a so-called "quadrature channel element" (QCE) may be seen as a tool of an audio decoder which may be used, for example, to decode three-dimensional audio content.

[0186] In other words, a quad channel element (QCE) is a four channel joint coding method for more efficient coding of horizontally and vertically distributed channels. A QCE consists of two consecutive CPEs and is formed by hierarchically combining a joint stereo tool with the possibility of a complex stereo prediction tool in the horizontal direction and a stereo tool based on MPEG surround in the vertical direction. This is achieved by enabling both stereo tools and swapping the output channels between the applied tools. Stereo SBR is performed in the horizontal direction to preserve the left-right relationship of the high frequencies.

[0187] Fig.18 The topology of the QCE is shown. It should be noted that Fig.18 The QCE is very similar to Fig.11 QCE, so that reference can be made to the above explanation. However, it should be noted that Fig.18In the QCE of , it is not necessary to use a psychoacoustic model when performing complex stereo prediction (optionally, although such use is of course possible). Furthermore, it can be seen that a first stereo spectral bandwidth replication (stereo SBR) is performed based on the lower left channel and the lower right channel, and a second stereo spectral bandwidth replication (stereo SBR) is performed based on the upper left channel and the upper right channel.

[0188] Hereinafter, some terms and definitions will be provided, which may be applicable in some embodiments.

[0189] The data unit qceIndex indicates the QCE mode of the CPE. For the meaning of the bitstream variable qceIndex, refer to Fig. 14B It should be noted that qceIndex describes whether the two subsequent elements of type UsacChannelPairElement() are to be treated as quad-channel elements (QCE). Fig. 14B The different QCE modes are given in . qceIndex should be the same for two subsequent units forming a QCE.

[0190] In the following, some helper units will be defined, which can be used in some implementations according to the present invention:

[0191] cplx_out_dmx_L[ ] The first channel of the first CPE after complex predictive stereo decoding

[0192] cplx_out_dmx_R[ ] Second channel of the first CPE after complex predictive stereo decoding

[0193] cplx_out_res_L[ ] Second CPE after complex predictive stereo decoding (zero if qceIndex=1)

[0194] cplx_out_res_R[ ] Second channel of the second CPE after complex predictive stereo decoding (zero if qceIndex=1)

[0195] mps_out_L_1[ ] The first output channel of the first MPS frame

[0196] mps_out_L_2[ ] Second output channel of the first MPS frame

[0197] mps_out_R_1[ ] The first output channel of the second MPS frame

[0198] mps_out_R_2[ ] Second output channel of the second MPS frame

[0199] sbr_out_L_1[ ] First output channel of the first stereo SBR box

[0200] sbr_out_R_1[ ] Second output channel of the first stereo SBR box

[0201] sbr_out_L_2[ ] First output channel of the second stereo SBR box

[0202] sbr_out_R_2[ ] Second output channel of the second stereo SBR box

[0203] Hereinafter, a decoding process performed in an embodiment according to the present invention will be explained.

[0204] The syntax element (or bitstream element, or data element) qceIndex in UsacChannelPairElementConfig() indicates whether the CPE belongs to a QCE and whether residual coding is used. In the case where qceIndex is not equal to 0, the current CPE forms a QCE with its subsequent element, which should be a CPE with the same qceIndex. Stereo SBR is always used for QCE, so the syntax item stereoConfigIndex should be 3 and bsStereoSbr should be 1.

[0205] In case qceIndex == 1, only payloads for MPEG surround and SBR and no related audio signal data are contained in the second CPE and the syntax element bsResidualCoding is set to 0.

[0206] The presence of a residual signal in the second CPE is indicated by qceIndex == 2. In this case, the syntax element bsResidualCoding is set to 1.

[0207] However, some different and possibly simplified signaling schemes may also be used.

[0208] The decoding of joint stereo with the possibility of complex stereo prediction is performed as described in ISO / IEC 23003-3, section 7.7. The resulting output of the first CPE is the MPS downmix signals cplx_out_dmx_L[ ] and cplx_out_dmx_R[ ]. If residual coding is used (i.e., qceIndex == 2), the output of the second CPE is the MPS residual signals cplx_out_res_L[ ], cplx_out_res_R[ ], and if no residual signal is sent (i.e., qceIndex == 1), a zero signal is inserted.

[0209] Before applying MPEG surround decoding, the second channel of the first component (cplx_out_dmx_R[ ]) and the first channel of the second component (cplx_out_res_L[ ]) are swapped.

[0210] The decoding of MPEG Surround is performed as described in ISO / IEC 23003-3 section 7.11. If residual coding is used, however, in some embodiments the decoding may be modified compared to conventional MPEG Surround decoding. The decoding of MPEG Surround without residual using SBR as defined in ISO / IEC 23003-3 section 7.11.2.7 (FIG. 23) is modified so that stereo SBR is also used for bsResidualCoding == 1, resulting in Fig.19 The decoder diagram shown in . Fig.19 A schematic block diagram of an audio encoder for bsResidualCoding == 0 and bsStereoSbr == 1 is shown.

[0211] like Fig.19 As can be seen in the figure, the USAC core decoder 2010 provides a downmix signal (DMX) 2012 to an MPS (MPEG Surround) decoder 2020, which provides a first decoded audio signal 2022 and a second decoded audio signal 2024. A stereo SBR decoder 2030 receives the first decoded audio signal 2022 and the second decoded audio signal 2024, and provides a left bandwidth extended audio signal 2032 and a right bandwidth extended audio signal 2034 based on the first decoded audio signal and the second decoded audio signal.

[0212] Before applying stereo SBR, the second channel of the first component (mps_out_L_2[ ]) and the first channel of the second component (mps_out_R_1[ ]) are swapped to allow left-right stereo SBR. After applying stereo SBR, the second output channel of the first component (sbr_out_R_1[ ]) and the first channel of the second component (sbr_out_L_2[ ]) are swapped again to restore the input channel order.

[0213] exist Fig. 20 The QCE decoder structure is illustrated in Fig. 20 A QCE decoder schematic is shown.

[0214] It should be noted that Fig. 20 The schematic diagram is very similar to Fig.13 , so that reference can also be made to the above explanations. In addition, it should be noted that Fig. 20Some signal indications have been added in , where reference is made to the definitions in this section. Furthermore, the final re-sorting of the channels is shown, which is performed after stereo SBR.

[0215] Fig.21 2 shows a schematic block diagram of a four-channel encoder 2200 according to an embodiment of the present invention. Fig.21 A four-channel encoder (four-channel unit) which can be considered as a core encoder tool is illustrated in .

[0216] The four-channel encoder 2200 includes a first stereo SBR 2210, which receives a first left channel input signal 2212 and a second left channel input signal 2214, and provides a first SBR payload 2215, a first left channel SBR output signal 2216, and a first right channel SBR output signal 2218 based on the first left channel input signal and the second left channel input signal. In addition, the four-channel encoder 2200 includes a second stereo SBR, which receives a second left channel input signal 2222 and a second right channel input signal 2224, and provides a first SBR payload 2225, a first left channel SBR output signal 2226, and a first right channel SBR output signal 2228 based on the second left channel input signal and the second right channel input signal.

[0217] The four-channel encoder 2200 includes a first MPEG surround type (MPS 2-1-2 or unified stereo) multi-channel encoder 2230, which receives a first left channel SBR output signal 2216 and a second left channel SBR output signal 2226, and the first MPEG surround type (MPS 2-1-2 or unified stereo) multi-channel encoder provides a first MPS payload 2232, a left channel MPEG surround downmix signal 2234 and (optionally) a left channel MPEG surround residual signal 2236 based on the first left channel SBR output signal and the second left channel SBR output signal. The four-channel encoder 2200 also includes a second MPEG surround type (MPS 2-1-2 or unified stereo) multi-channel encoder 2240, which receives a first right channel SBR output signal 2218 and a second right channel SBR output signal 2228, and the second MPEG surround type (MPS 2-1-2 or unified stereo) multi-channel encoder provides a first MPS payload 2242, a right channel MPEG surround downmix signal 2244 and (optionally) a right channel MPEG surround residual signal 2246 based on the first right channel SBR output signal and the second right channel SBR output signal.

[0218] The four-channel encoder 2200 includes a first complex predictive stereo encoding 2250, which receives a left-channel MPEG surround sound downmix signal 2234 and a right-channel MPEG surround sound downmix signal 2244, and provides a complex prediction payload 2252 and a joint encoded representation 2254 of the left-channel MPEG surround sound downmix signal 2234 and the right-channel MPEG surround sound downmix signal 2244 based on the left-channel MPEG surround sound downmix signal and the right-channel MPEG surround sound downmix signal. The four-channel encoder 2200 includes a second complex predictive stereo encoding 2260, which receives a left-channel MPEG surround sound residual signal 2236 and a right-channel MPEG surround sound residual signal 2246, and provides a complex prediction payload 2262 and a joint encoded representation 2264 of the left-channel MPEG surround sound downmix signal 2236 and the right-channel MPEG surround sound downmix signal 2246 based on the left-channel MPEG surround sound residual signal and the right-channel MPEG surround sound residual signal.

[0219] The four-channel encoder also includes a first bitstream code 2270 that receives the joint coded representation 2254, the complex prediction payload 2252, the MPS payload 2232, and the SBR payload 2215, and provides a bitstream portion representing the first channel pair unit based on the above items. The four-channel encoder also includes a second bitstream code 2280 that receives the joint coded representation 2264, the complex prediction payload 2262, the MPS payload 2242, and the SBR payload 2225, and provides a bitstream portion representing the first channel pair unit based on the above items.

[0220] 14. Alternative Implementation Plans

[0221] Although some schemes have been described in the context of equipment, it is obvious that these schemes also represent the description of the corresponding method, wherein the frame or device corresponds to the feature of the method step or the method step. Similarly, in the context of the method step, the scheme also represents the description of the corresponding frame or item or feature of the corresponding equipment. Some or all of the method steps can be performed by (using) hardware devices, such as, for example, microprocessors, programmable computers or electronic circuits. In some embodiments, one or more of the most important method steps can be performed by this device.

[0222] The inventive encoded audio signal may be stored on a digital storage medium or may be transmitted via a transmission medium such as a wireless transmission medium or a wired transmission medium, such as the Internet.

[0223] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or in software. The implementation may be performed using a digital storage medium, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM or flash memory, on which electronically readable control signals are stored, which cooperate (or are capable of cooperating) with a programmable computer system so that the corresponding method can be performed. Thus, the digital storage medium may be computer readable.

[0224] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0225] Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product is executed on a computer. The program code may, for example, be stored on a machine readable carrier.

[0226] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0227] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0228] A further embodiment of the inventive method is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded on the data carrier, the computer program for performing one of the methods described herein. The data carrier, the digital storage medium or the recorded medium are typically tangible and / or non-transitory.

[0229] A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein.The data stream or the sequence of signals may, for example, be configured to be transferred via a data communication connection, for example via the Internet.

[0230] A further embodiment comprises a processing means, for example a computer or a programmable logic device, configured or adapted to perform one of the methods described herein.

[0231] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0232] Another embodiment according to the invention comprises a device or system configured to transfer (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may, for example, be a computer, a mobile device, a storage device, etc. The device or system may, for example, comprise a file server for transferring the computer program to the receiver.

[0233] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware device.

[0234] The embodiments described above are merely illustrative of the principles of the present invention. It will be appreciated that modifications and variations of the arrangements and details described herein will be apparent to those skilled in the art. Therefore, it is intended that the invention be limited only by the scope of the forthcoming patent claims and not by the specific details presented by way of description and explanation of the embodiments herein.

[0235] 15. Conclusion

[0236] In the following, some conclusions are provided.

[0237] The embodiment according to the present invention is based on the following considerations: To account for the signal dependency between vertically distributed channels and horizontally distributed channels, the four channels can be jointly encoded by hierarchically combining joint stereo coding tools. For example, MPS 2-1-2 with band-limited residual coding or full-band residual coding and / or unified stereo are used to combine vertical channel pairs. In order to meet the perceptual requirements for binaural unmasking, the output downmix is ​​jointly encoded, for example by using complex prediction in the MDCT domain, which includes the possibility of left-right coding and mid-side coding. If a residual signal exists, the residual signal is combined horizontally using the same method.

[0238] Furthermore, it should be noted that embodiments according to the present invention overcome some or all of the disadvantages of the prior art. Embodiments according to the present invention are suitable for 3D audio scenarios, where the speaker channels are distributed in layers of several heights, resulting in horizontal channel pairs and vertical channel pairs. It has been found that joint coding of only two channels as defined in USAC is not sufficient to take into account the spatial and perceptual relationships between the channels. However, embodiments according to the present invention overcome this problem.

[0239] Furthermore, conventional MPEG surround is applied in additional pre-processing / post-processing steps, so that the residual signals are sent separately without the possibility of joint stereo coding, for example, to explore the dependency between the left and right fundamental residual signals. In contrast, embodiments according to the present invention allow for efficient encoding / decoding by exploiting such dependencies.

[0240] In further summary, a device, method or computer program for encoding and decoding as described herein is created according to an embodiment of the present invention.

[0241] 16. Additional Notes

[0242] An example of the foregoing embodiment will be described below as a supplementary note. However, the present invention illustrated in the foregoing exemplary embodiment and modifications thereto are not limited to the following.

[0243] Example 1. An audio decoder (200; 300; 600; 1300; 1600; 2000) for providing at least four audio channel signals (220, 222, 224, 226; 320, 322, 324, 326; 620, 622, 624, 626; 1320, 1322, 1324, 1326) based on an encoded representation (210; 310, 360; 610, 682; 1310, 1312; 1610),

[0244] wherein the audio decoder is configured to provide the first residual signal (232; 332; 684; 1362) and the second residual signal (234; 334; 686; 1364) based on a joint coded representation (210; 310; 682; 1312) of the first residual signal and the second residual signal using multi-channel decoding (230; 330; 680; 1360);

[0245] wherein the audio decoder is configured to provide a first audio channel signal (220; 320; 642; 1372) and a second audio channel signal (222; 322; 644; 1374) based on a first downmix signal (212; 312; 632; 1342) and the first residual signal using residual signal-assisted multi-channel decoding (240; 340; 640; 1370); and

[0246] The audio decoder is configured to use residual signal-assisted multi-channel decoding (250; 350; 650; 1380) to provide a third audio channel signal (224; 324; 656; 1382) and a fourth audio channel signal (226; 326; 658; 1384) based on a second downmix signal (214; 314; 634; 1344) and the second residual signal.

[0247] Example 2. An audio decoder according to Example 1, wherein the audio decoder is configured to: use multi-channel decoding (370; 630; 1340) to provide the first downmix signal (212; 312; 632; 1342) and the second downmix signal (214; 314; 634; 1344) based on a joint encoded representation (360; 610; 1310) of the first downmix signal and the second downmix signal.

[0248] Example 3. An audio decoder according to Example 1 or 2, wherein the audio decoder is configured to: provide the first residual signal and the second residual signal based on a joint encoded representation of the first residual signal and the second residual signal using prediction-based multi-channel decoding.

[0249] Example 4. An audio decoder according to any one of Examples 1 to 3, wherein the audio decoder is configured to: use residual signal-assisted multi-channel decoding to provide the first residual signal and the second residual signal based on a joint encoded representation of the first residual signal and the second residual signal.

[0250] Example 5. The audio decoder of Example 3, wherein the prediction based multi-channel decoding is configured to estimate prediction parameters describing a contribution of signal components derived using signal components of a previous frame to providing a residual signal for a current frame.

[0251] Example 6. An audio decoder according to any one of Examples 3 to 5, wherein the prediction-based multi-channel decoding is configured to obtain the first residual signal and the second residual signal based on a downmix signal of the first residual signal and the second residual signal and based on a common residual signal of the first residual signal and the second residual signal.

[0252] Example 7. An audio decoder according to Example 6, wherein the prediction-based multi-channel decoding is configured to: apply the common residual signal with a first symbol to obtain the first residual signal, and apply the common residual signal with a second symbol to obtain the second residual signal, and the second symbol is opposite to the first symbol.

[0253] Example 8. An audio decoder according to any one of Examples 1 to 7, wherein the audio decoder is configured to: use multi-channel decoding operating in the MDCT domain to provide the first residual signal and the second residual signal based on a joint encoded representation of the first residual signal and the second residual signal.

[0254] Example 9. An audio decoder according to any one of Examples 1 to 8, wherein the audio decoder is configured to provide the first residual signal and the second residual signal based on a joint encoded representation of the first residual signal and the second residual signal using USAC complex stereo prediction.

[0255] Example 10. An audio decoder according to any one of Examples 1 to 9,

[0256] wherein the audio decoder is configured to provide the first audio channel signal and the second audio channel signal based on the first downmix signal and the first residual signal using parameter-based, residual signal-assisted multi-channel decoding; and

[0257] Therein, the audio decoder is configured to provide the third audio channel signal and the fourth audio channel signal based on the second downmix signal and the second residual signal using parameter-based, residual signal-assisted multi-channel decoding.

[0258] Example 11. An audio decoder according to Example 10, wherein the parameter-based, residual signal-assisted multi-channel decoding is configured to: estimate one or more parameters describing the required correlation between two channels and / or the order difference between two channels to provide the two or more audio channel signals based on a corresponding one of the downmix signals and a corresponding one of the residual signals.

[0259] Example 12. An audio decoder according to any one of Examples 1 to 11, wherein the audio decoder is configured to: provide the first audio channel signal and the second audio channel signal based on the first downmix signal and the first residual signal using residual signal-assisted multi-channel decoding operating in the QMF domain; and

[0260] The audio decoder is configured to provide the third audio channel signal and the fourth audio channel signal based on the second downmix signal and the second residual signal using residual signal assisted multi-channel decoding operating in a QMF domain.

[0261] Example 13. An audio decoder according to any one of Examples 1 to 12, wherein the audio decoder is configured to: provide the first audio channel signal and the second audio channel signal based on the first downmix signal and the first residual signal using MPEG Surround 2-1-2 decoding or unified stereo decoding; and

[0262] The audio decoder is configured to provide the third audio channel signal and the fourth audio channel signal based on the second downmix signal and the second residual signal using MPEG Surround 2-1-2 decoding or unified stereo decoding.

[0263] Example 14. The audio decoder according to any one of Examples 1 to 13, wherein the first residual signal and the second residual signal are associated with different horizontal positions of an audio scene or with different azimuthal positions of the audio scene.

[0264] Example 15. An audio decoder according to any one of Examples 1 to 14, wherein the first audio channel signal and the second audio channel signal are associated with vertically adjacent positions of an audio scene, and

[0265] The third audio channel signal and the fourth audio channel signal are associated with vertically adjacent positions of the audio scene.

[0266] Example 16. An audio decoder according to any one of Examples 1 to 15, wherein the first audio channel signal and the second audio channel signal are associated with a first horizontal position or an azimuth position of an audio scene, and

[0267] The third audio channel signal and the fourth audio channel signal are associated with a second horizontal position or an azimuth position of the audio scene, the second horizontal position or the azimuth position being different from the first horizontal position or the first azimuth position.

[0268] Example 17. The audio decoder of any one of Examples 1 to 16, wherein the first residual signal is associated with a left side of the audio scene and the second residual signal is associated with a right side of the audio scene.

[0269] Example 18. The audio encoder according to Example 17,

[0270] wherein the first audio channel signal and the second audio channel signal are associated with the left side of the audio scene, and

[0271] The third audio channel signal and the fourth audio channel signal are associated with a right side of the audio scene.

[0272] Example 19. The audio decoder of Example 18, wherein the first audio channel signal is associated with a lower left position of the audio scene,

[0273] the second audio channel signal is associated with an upper left position of the audio scene,

[0274] The third audio channel signal is associated with a lower right position of the audio scene, and

[0275] The fourth audio channel signal is associated with an upper right position of the audio scene.

[0276] Example 20. An audio decoder according to any one of Examples 1 to 19, wherein the audio decoder is configured to: use multi-channel decoding to provide the first down-mix signal and the second down-mix signal based on a joint encoded representation of the first down-mix signal and the second down-mix signal, the first down-mix signal being associated with a left side of an audio scene, and the second down-mix signal being associated with a right side of the audio scene.

[0277] Example 21. An audio decoder according to any one of Examples 1 to 20, wherein the audio decoder is configured to: use prediction-based multi-channel decoding to provide the first down-mix signal and the second down-mix signal based on a joint encoded representation of the first down-mix signal and the second down-mix signal.

[0278] Example 22. An audio decoder according to any one of Examples 1 to 21, wherein the audio decoder is configured to use residual signal-assisted, prediction-based multi-channel decoding to provide the first down-mix signal and the second down-mix signal based on a joint encoded representation of the first down-mix signal and the second down-mix signal.

[0279] Example 23. An audio decoder according to any one of Examples 1 to 22, wherein the audio decoder is configured to: perform a first multi-channel bandwidth extension (660; 1390) based on the first audio channel signal and the third audio channel signal, and

[0280] The audio decoder is configured to perform a second multi-channel bandwidth extension based on the second audio channel signal and the fourth audio channel signal (670; 1394).

[0281] Example 24. An audio decoder according to Example 23, wherein the audio decoder is configured to: perform the first multi-channel bandwidth extension based on the first audio channel signal and the third audio channel signal and one or more bandwidth extension parameters (1338) to obtain two or more bandwidth extended audio channel signals (620, 624; 1320, 1324) associated with a first common horizontal plane or a first common height of the audio scene, and

[0282] The audio decoder is configured to perform the second multi-channel bandwidth extension based on the second audio channel signal and the fourth audio channel signal and one or more bandwidth extension parameters (1358) to obtain two or more bandwidth extended audio channel signals (622, 626; 1322, 1326) associated with a second common horizontal plane or a second common height of the audio scene.

[0283] Example 25. An audio decoder according to any one of Examples 1 to 24, wherein the joint encoded representation of the first residual signal and the second residual signal includes a channel pair unit, and the channel pair unit includes a downmix signal of the first residual signal and the second residual signal and a common residual signal of the first residual signal and the second residual signal.

[0284] Example 26. An audio decoder according to any one of Examples 1 to 25, wherein the audio decoder is configured to: use multi-channel decoding to provide the first down-mix signal and the second down-mix signal based on a joint encoded representation of the first down-mix signal and the second down-mix signal,

[0285] The jointly encoded representation of the first downmix signal and the second downmix signal comprises a channel pair unit, the channel pair unit comprising a downmix signal of the first downmix signal and the second downmix signal and a common residual signal of the first downmix signal and the second downmix signal.

[0286] Example 27. An audio encoder (100; 1100; 1200; 1500; 2100) for providing an encoded representation (130; 1144, 1154; 1220, 1222; 2272, 2282) based on at least four audio channel signals (110, 112, 114, 116; 1110, 1112, 1114, 1116; 1210, 1212, 1214, 1216; 2216, 2226, 2218, 2228),

[0287] wherein the audio encoder is configured to jointly encode at least a first audio channel signal and a second audio channel signal using residual signal-assisted multi-channel encoding (140; 1120; 1230; 2230) to obtain a first downmix signal (120; 1122; 1232; 2234) and a first residual signal (142; 1124; 1234; 2236); and

[0288] wherein the audio encoder is configured to jointly encode at least a third audio channel signal and a fourth audio channel signal using residual signal-assisted multi-channel encoding (150; 1130; 1240; 2240) to obtain a second downmix signal (122; 1132; 1242; 2244) and a second residual signal (152; 1134; 1244; 2246); and

[0289] The audio encoder is configured to jointly encode the first residual signal and the second residual signal using multi-channel coding (160; 1150; 1260; 2260) to obtain a jointly encoded representation (130; 1154; 1262; 2264) of the residual signals.

[0290] Example 28. An audio encoder according to Example 27, wherein the audio encoder is configured to: jointly encode the first downmix signal and the second downmix signal using multi-channel coding (1140; 1250; 2250) to obtain a jointly encoded representation of the downmix signals (1144; 1252; 2254).

[0291] Example 29. The audio encoder of Example 28, wherein the audio encoder is configured to: jointly encode the first residual signal and the second residual signal using prediction-based multi-channel coding, and

[0292] The audio encoder is configured to jointly encode the first downmix signal and the second downmix signal using prediction based multi-channel coding.

[0293] Example 30. An audio encoder according to any one of Examples 27 to 29, wherein the audio encoder is configured to: jointly encode at least the first audio channel signal and the second audio channel signal using parameter-based, residual signal-assisted multi-channel coding, and

[0294] The audio encoder is configured to jointly encode at least the third audio channel signal and the fourth audio channel signal using parameter-based, residual signal-assisted multi-channel coding.

[0295] Example 31. An audio encoder according to any one of Examples 27 to 30, wherein the first audio channel signal and the second audio channel signal are associated with vertically adjacent positions of an audio scene, and

[0296] The third audio channel signal and the fourth audio channel signal are associated with vertically adjacent positions of the audio scene.

[0297] Example 32. An audio encoder according to any one of Examples 27 to 31, wherein the first audio channel signal and the second audio channel signal are associated with a first horizontal position or an azimuth position of an audio scene, and

[0298] The third audio channel signal and the fourth audio channel signal are associated with a second horizontal position or azimuth position of the audio scene, the second horizontal position or azimuth position being different from the first horizontal position or azimuth position.

[0299] Example 33. An audio encoder according to any one of Examples 27 to 32, wherein the first residual signal is associated with a left side of an audio scene and the second residual signal is associated with a right side of the audio scene.

[0300] Example 34. The audio encoder of Example 33,

[0301] wherein the first audio channel signal and the second audio channel signal are associated with the left side of the audio scene, and

[0302] The third audio channel signal and the fourth audio channel signal are associated with the right side of the audio scene.

[0303] Example 35. The audio decoder of Example 34, wherein the first audio channel signal is associated with a lower left position of the audio scene,

[0304] the second audio channel signal is associated with an upper left position of the audio scene,

[0305] The third audio channel signal is associated with a lower right position of the audio scene, and

[0306] The fourth audio channel signal is associated with an upper right position of the audio scene.

[0307] Example 36. An audio encoder according to any one of Examples 27 to 35, wherein the audio encoder is configured to: jointly encode the first downmix signal and the second downmix signal using multi-channel coding to obtain a jointly encoded representation of the downmix signals, the first downmix signal being associated with a left side of an audio scene, and the second downmix signal being associated with a right side of the audio scene.

[0308] Example 37. A method (800) for providing at least four audio channel signals based on an encoded representation, the method comprising:

[0309] Using multi-channel decoding, providing (810) the first residual signal and the second residual signal based on a jointly encoded representation of the first residual signal and the second residual signal;

[0310] providing (820) a first audio channel signal and a second audio channel signal based on a first downmix signal and the first residual signal using residual signal assisted multi-channel decoding; and

[0311] Using residual signal assisted multi-channel decoding, a third audio channel signal and a fourth audio channel signal are provided (830) based on the second downmix signal and the second residual signal.

[0312] Example 38. A method (700) for providing an encoded representation based on at least four audio channel signals, the method comprising:

[0313] Jointly encoding at least a first audio channel signal and a second audio channel signal using residual signal-assisted multi-channel coding (710) to obtain a first downmix signal and a first residual signal;

[0314] jointly encoding at least the third audio channel signal and the fourth audio channel signal using residual signal assisted multi-channel coding (720) to obtain a second downmix signal and a second residual signal; and

[0315] The first residual signal and the second residual signal are jointly encoded using multi-channel coding (730) to obtain an encoded representation of the residual signals.

[0316] Example 39. A computer program for performing the method according to Example 37 or 38 when the computer program is executed on a computer.

[0317] References:

[0318] [1] ISO / IEC 23003-3: 2012 - Information Technology – MPEG AudioTechnologies, Part 3: Unified Speech and Audio Coding;

[0319] [2] ISO / IEC 23003-1: 2007 - Information Technology – MPEG AudioTechnologies, Part 1: MPEG Surround.

Claims

1. An audio decoder (200; 300; 600; 1300; 1600; 2000) for providing at least four audio channel signals (220, 222, 224, 226; 320, 322, 324, 326; 620, 622, 624, 626; 1320, 1322, 1324, 1326) based on an encoded representation (210; 310, 360; 610, 682; 1310, 1312; 1610), in, The audio decoder is configured to provide the first residual signal (232; 332; 684; 1362) and the second residual signal (234; 334; 686; 1364) based on a joint encoded representation (210; 310; 682; 1312) of the first residual signal and the second residual signal using multi-channel decoding (230; 330; 680; 1360); wherein the audio decoder is configured to provide a first audio channel signal (220; 320; 642; 1372) and a second audio channel signal (222; 322; 644; 1374) based on a first downmix signal (212; 312; 632; 1342) and the first residual signal using residual signal-assisted multi-channel decoding (240; 340; 640; 1370); and The audio decoder is configured to use residual signal-assisted multi-channel decoding (250; 350; 650; 1380) to provide a third audio channel signal (224; 324; 656; 1382) and a fourth audio channel signal (226; 326; 658; 1384) based on a second downmix signal (214; 314; 634; 1344) and the second residual signal.

2. The audio decoder according to claim 1, wherein: The audio decoder is configured to provide the first downmix signal (212; 312; 632; 1342) and the second downmix signal (214; 314; 634; 1344) based on a joint encoded representation (360; 610; 1310) of the first downmix signal and the second downmix signal using multi-channel decoding (370; 630; 1340).

3. The audio decoder according to claim 1, wherein: The audio decoder is configured to provide the first residual signal and the second residual signal based on a jointly encoded representation of the first residual signal and the second residual signal using prediction based multi-channel decoding.

4. The audio decoder according to claim 1, wherein: The audio decoder is configured to provide the first residual signal and the second residual signal based on a jointly encoded representation of the first residual signal and the second residual signal using residual signal-assisted multi-channel decoding.

5. The audio decoder according to claim 3, wherein: The prediction based multi-channel decoding is configured to estimate prediction parameters describing the contribution of signal components derived using signal components of a previous frame to providing a residual signal for a current frame.

6. The audio decoder according to claim 3, wherein: The prediction-based multi-channel decoding is configured to obtain the first residual signal and the second residual signal based on a downmix signal of the first residual signal and the second residual signal and based on a common residual signal of the first residual signal and the second residual signal.

7. The audio decoder according to claim 6, wherein: The prediction-based multi-channel decoding is configured to apply the common residual signal with a first sign to obtain the first residual signal and to apply the common residual signal with a second sign to obtain the second residual signal, the second sign being opposite to the first sign.

8. The audio decoder according to claim 1, wherein: The audio decoder is configured to provide the first residual signal and the second residual signal based on a jointly encoded representation of the first residual signal and the second residual signal using multi-channel decoding operating in the MDCT domain.

9. The audio decoder according to claim 1, wherein: The audio decoder is configured to provide the first residual signal and the second residual signal based on a jointly encoded representation of the first residual signal and the second residual signal using USAC complex stereo prediction.

10. The audio decoder according to claim 1, in, The audio decoder is configured to provide the first audio channel signal and the second audio channel signal based on the first downmix signal and the first residual signal using parameter-based, residual signal-assisted multi-channel decoding; as well as Therein, the audio decoder is configured to provide the third audio channel signal and the fourth audio channel signal based on the second downmix signal and the second residual signal using parameter-based, residual signal-assisted multi-channel decoding.

11. The audio decoder according to claim 10, wherein: The parameter-based, residual signal-assisted multi-channel decoding is configured to estimate one or more parameters describing a required correlation between two channels and / or an order difference between two channels to provide corresponding two or more audio channel signals based on a corresponding one of the first down-mix signal and the second down-mix signal and a corresponding one of the first residual signal and the second residual signal.

12. The audio decoder according to claim 1, wherein: The audio decoder is configured to provide the first audio channel signal and the second audio channel signal based on the first downmix signal and the first residual signal using residual signal assisted multi-channel decoding operating in the QMF domain; as well as The audio decoder is configured to provide the third audio channel signal and the fourth audio channel signal based on the second downmix signal and the second residual signal using residual signal assisted multi-channel decoding operating in a QMF domain.

13. The audio decoder according to claim 1, wherein: The audio decoder is configured to provide the first audio channel signal and the second audio channel signal based on the first downmix signal and the first residual signal using MPEG Surround 2-1-2 decoding or unified stereo decoding; as well as The audio decoder is configured to provide the third audio channel signal and the fourth audio channel signal based on the second downmix signal and the second residual signal using MPEG Surround 2-1-2 decoding or unified stereo decoding.

14. The audio decoder according to claim 1, wherein: The first residual signal and the second residual signal are associated with different horizontal positions of the audio scene or with different azimuth positions of the audio scene.

15. The audio decoder according to claim 1, wherein: The first audio channel signal and the second audio channel signal are associated with vertically adjacent positions of an audio scene, and The third audio channel signal and the fourth audio channel signal are associated with vertically adjacent positions of the audio scene.

16. The audio decoder of claim 1, wherein: The first audio channel signal and the second audio channel signal are associated with a first horizontal position or an azimuth position of an audio scene, and The third audio channel signal and the fourth audio channel signal are associated with a second horizontal position or azimuth position of the audio scene, the second horizontal position or azimuth position being different from the first horizontal position or azimuth position.

17. The audio decoder of claim 1, wherein: The first residual signal is associated with a left side of the audio scene and the second residual signal is associated with a right side of the audio scene.

18. The audio decoder according to claim 17, in, The first audio channel signal and the second audio channel signal are associated with a left side of the audio scene, and The third audio channel signal and the fourth audio channel signal are associated with a right side of the audio scene.

19. The audio decoder of claim 18, wherein: The first audio channel signal is associated with a lower left position of the audio scene, the second audio channel signal is associated with an upper left position of the audio scene, The third audio channel signal is associated with a lower right position of the audio scene, and The fourth audio channel signal is associated with an upper right position of the audio scene.

20. The audio decoder of claim 1, wherein: The audio decoder is configured to provide the first downmix signal and the second downmix signal based on a joint encoded representation of the first downmix signal and the second downmix signal using multi-channel decoding, the first downmix signal being associated with a left side of an audio scene and the second downmix signal being associated with a right side of the audio scene.

21. The audio decoder of claim 1, wherein: The audio decoder is configured to provide the first downmix signal and the second downmix signal based on a jointly encoded representation of the first downmix signal and the second downmix signal using prediction based multi-channel decoding.

22. The audio decoder of claim 1, wherein: The audio decoder is configured to provide the first downmix signal and the second downmix signal based on a jointly encoded representation of the first downmix signal and the second downmix signal using residual signal assisted prediction based multi-channel decoding.

23. The audio decoder of claim 1, wherein: The audio decoder is configured to: perform a first multi-channel bandwidth extension based on the first audio channel signal and the third audio channel signal (660; 1390), and The audio decoder is configured to perform a second multi-channel bandwidth extension based on the second audio channel signal and the fourth audio channel signal (670; 1394).

24. The audio decoder of claim 23, wherein: The audio decoder is configured to: perform the first multi-channel bandwidth extension based on the first audio channel signal and the third audio channel signal and one or more bandwidth extension parameters (1338) to obtain two or more bandwidth extended audio channel signals (620, 624; 1320, 1324) associated with a first common horizontal plane or a first common height of the audio scene, and The audio decoder is configured to perform the second multi-channel bandwidth extension based on the second audio channel signal and the fourth audio channel signal and one or more bandwidth extension parameters (1358) to obtain two or more bandwidth extended audio channel signals (622, 626; 1322, 1326) associated with a second common horizontal plane or a second common height of the audio scene.

25. The audio decoder of claim 1, wherein: The jointly encoded representation of the first residual signal and the second residual signal comprises a channel pair unit, the channel pair unit comprising a downmix signal of the first residual signal and the second residual signal and a common residual signal of the first residual signal and the second residual signal.

26. The audio decoder of claim 1, wherein: The audio decoder is configured to provide the first downmix signal and the second downmix signal based on a joint encoded representation of the first downmix signal and the second downmix signal using multi-channel decoding, The jointly encoded representation of the first downmix signal and the second downmix signal comprises a channel pair unit, the channel pair unit comprising a downmix signal of the first downmix signal and the second downmix signal and a common residual signal of the first downmix signal and the second downmix signal.

27. An audio encoder (100; 1100; 1200; 1500; 2100) for providing an encoded representation (130; 1144, 1154; 1220, 1222; 2272, 2282) based on at least four audio channel signals (110, 112, 114, 116; 1110, 1112, 1114, 1116; 1210, 1212, 1214, 1216; 2216, 2226, 2218, 2228), in, The audio encoder is configured to jointly encode at least a first audio channel signal and a second audio channel signal using residual signal-assisted multi-channel coding (140; 1120; 1230; 2230) to obtain a first downmix signal (120; 1122; 1232; 2234) and a first residual signal (142; 1124; 1234; 2236); and in, The audio encoder is configured to jointly encode at least a third audio channel signal and a fourth audio channel signal using residual signal-assisted multi-channel coding (150; 1130; 1240; 2240) to obtain a second downmix signal (122; 1132; 1242; 2244) and a second residual signal (152; 1134; 1244; 2246); and The audio encoder is configured to jointly encode the first residual signal and the second residual signal using multi-channel coding (160; 1150; 1260; 2260) to obtain a jointly encoded representation (130; 1154; 1262; 2264) of the first residual signal and the second residual signal.

28. The audio encoder according to claim 27, wherein The audio encoder is configured to jointly encode the first downmix signal and the second downmix signal using multi-channel coding (1140; 1250; 2250) to obtain a jointly encoded representation (1144; 1252; 2254) of the first downmix signal and the second downmix signal.

29. The audio encoder according to claim 28, wherein The audio encoder is configured to: jointly encode the first residual signal and the second residual signal using prediction-based multi-channel coding, and The audio encoder is configured to jointly encode the first downmix signal and the second downmix signal using prediction based multi-channel coding.

30. The audio encoder of claim 27, wherein: The audio encoder is configured to jointly encode at least the first audio channel signal and the second audio channel signal using parameter-based, residual signal-assisted multi-channel coding, and The audio encoder is configured to jointly encode at least the third audio channel signal and the fourth audio channel signal using parameter-based, residual signal-assisted multi-channel coding.

31. The audio encoder of claim 27, wherein: The first audio channel signal and the second audio channel signal are associated with vertically adjacent positions of an audio scene, and The third audio channel signal and the fourth audio channel signal are associated with vertically adjacent positions of the audio scene.

32. The audio encoder of claim 27, wherein: The first audio channel signal and the second audio channel signal are associated with a first horizontal position or an azimuth position of an audio scene, and The third audio channel signal and the fourth audio channel signal are associated with a second horizontal position or azimuth position of the audio scene, the second horizontal position or azimuth position being different from the first horizontal position or azimuth position.

33. The audio encoder of claim 27, wherein: The first residual signal is associated with a left side of an audio scene and the second residual signal is associated with a right side of the audio scene.

34. The audio encoder according to claim 33, in, The first audio channel signal and the second audio channel signal are associated with a left side of the audio scene, and The third audio channel signal and the fourth audio channel signal are associated with the right side of the audio scene.

35. The audio encoder of claim 34, wherein: The first audio channel signal is associated with a lower left position of the audio scene, the second audio channel signal is associated with an upper left position of the audio scene, The third audio channel signal is associated with a lower right position of the audio scene, and The fourth audio channel signal is associated with an upper right position of the audio scene.

36. The audio encoder of claim 27, wherein: The audio encoder is configured to jointly encode the first downmix signal and the second downmix signal using multi-channel coding to obtain a jointly encoded representation of the first downmix signal and the second downmix signal, the first downmix signal being associated with a left side of an audio scene and the second downmix signal being associated with a right side of the audio scene.

37. A method (800) for providing at least four audio channel signals based on an encoded representation, the method comprising: Using multi-channel decoding, providing (810) the first residual signal and the second residual signal based on a jointly encoded representation of the first residual signal and the second residual signal; providing (820) a first audio channel signal and a second audio channel signal based on the first downmix signal and the first residual signal using residual signal assisted multi-channel decoding; as well as Using residual signal assisted multi-channel decoding, a third audio channel signal and a fourth audio channel signal are provided (830) based on the second downmix signal and the second residual signal.

38. A method (700) for providing an encoded representation based on at least four audio channel signals, the method comprising: Jointly encoding at least a first audio channel signal and a second audio channel signal using residual signal-assisted multi-channel coding (710) to obtain a first downmix signal and a first residual signal; Jointly encoding at least the third audio channel signal and the fourth audio channel signal using residual signal assisted multi-channel coding (720) to obtain a second downmix signal and a second residual signal; as well as The first residual signal and the second residual signal are jointly encoded using multi-channel coding (730) to obtain encoded representations of the first residual signal and the second residual signal.

39. A computer program product comprising a computer program for performing the method according to claim 37 or 38 when said computer program is executed on a computer.

40. An audio decoder (200; 300; 600; 1300; 1600; 2000) for providing at least four audio channel signals (220, 222, 224, 226; 320, 322, 324, 326; 620, 622, 624, 626; 1320, 1322, 1324, 1326) based on an encoded representation (210; 310, 360; 610, 682; 1310, 1312; 1610), in, The audio decoder is configured to provide the first residual signal (232; 332; 684; 1362) and the second residual signal (234; 334; 686; 1364) based on a joint encoded representation (210; 310; 682; 1312) of the first residual signal and the second residual signal using multi-channel decoding (230; 330; 680; 1360); wherein the audio decoder is configured to provide a first audio channel signal (220; 320; 642; 1372) and a second audio channel signal (222; 322; 644; 1374) based on a first downmix signal (212; 312; 632; 1342) and the first residual signal using residual signal-assisted multi-channel decoding (240; 340; 640; 1370); and wherein the audio decoder is configured to provide a third audio channel signal (224; 324; 656; 1382) and a fourth audio channel signal (226; 326; 658; 1384) based on a second downmix signal (214; 314; 634; 1344) and the second residual signal using residual signal-assisted multi-channel decoding (250; 350; 650; 1380); The audio decoder is configured to: perform a first multi-channel bandwidth extension based on the first audio channel signal and the third audio channel signal (660; 1390); and The audio decoder is configured to: perform a second multi-channel bandwidth extension based on the second audio channel signal and the fourth audio channel signal (670; 1394); The audio decoder is configured to: perform the first multi-channel bandwidth extension based on the first audio channel signal and the third audio channel signal and one or more bandwidth extension parameters (1338) to obtain two or more bandwidth extended audio channel signals (620, 624; 1320, 1324) associated with a first common horizontal plane or a first common height of the audio scene, and The audio decoder is configured to perform the second multi-channel bandwidth extension based on the second audio channel signal and the fourth audio channel signal and one or more bandwidth extension parameters (1358) to obtain two or more bandwidth extended audio channel signals (622, 626; 1322, 1326) associated with a second common horizontal plane or a second common height of the audio scene.

41. A method (800) for providing at least four audio channel signals based on an encoded representation, the method comprising: Using multi-channel decoding, providing (810) the first residual signal and the second residual signal based on a jointly encoded representation of the first residual signal and the second residual signal; providing (820) a first audio channel signal and a second audio channel signal based on the first downmix signal and the first residual signal using residual signal assisted multi-channel decoding; as well as providing (830) a third audio channel signal and a fourth audio channel signal based on a second downmix signal and the second residual signal using residual signal assisted multi-channel decoding; The method comprises: performing a first multi-channel bandwidth extension based on the first audio channel signal and the third audio channel signal (660; 1390), and The method comprises: performing a second multi-channel bandwidth extension based on the second audio channel signal and the fourth audio channel signal (670; 1394); wherein the first multi-channel bandwidth extension is performed based on the first audio channel signal and the third audio channel signal and one or more bandwidth extension parameters (1338) to obtain two or more bandwidth extended audio channel signals (620, 624; 1320, 1324) associated with a first common horizontal plane or a first common height of the audio scene, and The second multi-channel bandwidth extension is performed based on the second audio channel signal and the fourth audio channel signal and one or more bandwidth extension parameters (1358) to obtain two or more bandwidth-extended audio channel signals (622, 626; 1322, 1326) associated with a second common horizontal plane or a second common height of the audio scene.

42. A computer program product comprising a computer program for performing the method according to claim 41 when said computer program is executed on a computer.

Images