Method and device for encoding and / or decoding immersive audio signals

Abstract

The present document describes a method (700) for encoding a multi-channel input signal (201). The method (700) comprises determining (701) a plurality of downmix channel signals (203) from the multi-channel input signal (201) and carrying out (702) energy compression on the plurality of downmix channel signals (203) to provide a plurality of compressed channel signals (404). Furthermore, the method (700) comprises determining (703) joint decoding metadata (205) based on the plurality of compressed channel signals (404) and based on the multi-channel input signal (201), wherein the joint decoding metadata (205) is such that it allows upmixing the plurality of compressed channel signals (404) to an approximation of the multi-channel input signal (201). Furthermore, the method (700) comprises encoding (704) the plurality of compressed channel signals (404) and the joint decoding metadata (205).

Description

[0001] CROSS REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 693,246, filed on July 2, 2018, which is hereby incorporated by reference. Technical Field

[0003] The present document relates to immersive audio signals which may include sound field representation signals, in particular to high-fidelity stereophonic reproduction signals. In particular, the present document relates to providing encoders and corresponding decoders which enable transmission and / or storage of immersive audio signals in a bit-rate efficient manner and / or with high perceptual quality. Background Art

[0004] An Ambisonics signal may be used to describe the sounds or sound field within a listening environment of a listener at a listening position. The Ambisonics signal may be considered a multi-channel audio signal, where each channel corresponds to a specific directional pattern of the sound field at the listening position of the listener. The Ambisonics signal may be described using a three-dimensional (3D) Cartesian coordinate system, where the origin of the coordinate system corresponds to the listening position, the x-axis points forward, the y-axis points to the left, and the z-axis points upward.

[0005] By increasing the number of audio signals or channels and by increasing the number of corresponding directivity patterns (and corresponding panning functions), the accuracy of describing the sound field can be increased. For example, a first-order Ambisonics signal comprises 4 channels or waveforms, namely: a W channel indicating the omnidirectional component of the sound field; an X channel describing the sound field with a dipole directivity pattern corresponding to the x-axis; a Y channel describing the sound field with a dipole directivity pattern corresponding to the y-axis; and a Z channel describing the sound field with a dipole directivity pattern corresponding to the z-axis. A second-order Ambisonics signal comprises 9 channels, including the 4 channels of the first-order Ambisonics signal (also known as B-format) plus 5 additional channels of different directivity patterns. In general, an L-order Ambisonics signal comprises (L+1) 2 channels, the (L+1) 2 The channels contain the (L-1)-order Ambisonics signal L 2 channels plus additional directional patterns [(L+1) 2 -L 2 ] additional channels (when using the 3D Ambisonics format). The L-order Ambisonics signal in the case of L>1 may be referred to as a Higher-Order Ambisonics (HOA) signal.

[0006] The HOA signal may be used to describe a 3D sound field independently of the arrangement of speakers used to render the HOA signal. Example arrangements of speakers include one or more arrangements of headphones or loudspeakers or a virtual reality rendering environment. Therefore, it may be beneficial to provide the HOA signal to an audio renderer to allow the audio renderer to flexibly adapt to different arrangements of speakers.

[0007] A sound field representation (SR) signal (e.g. an Ambisonics signal) may be complemented with an audio object and / or a multichannel (test bed) signal to provide an immersive audio (IA) signal. The present document addresses the technical problem of transmitting and / or storing IA signals at high perceptual quality in a bandwidth efficient manner. The technical problem is solved by independent technical solutions. Preferred embodiments are described in the appended claims. Summary of the invention

[0008] According to an aspect, a method for encoding a multi-channel input signal is described. The multi-channel input signal may be part of an immersive audio (IA) signal. The multi-channel input signal may include a sound field representation (SR) signal, specifically a first-order or higher-order Ambisonics signal. The method comprises determining a plurality of downmix channel signals from the multi-channel input signal. Furthermore, the method comprises performing energy compression on the plurality of downmix channel signals to provide a plurality of compressed channel signals. In addition, the method comprises determining joint decoding metadata, specifically spatial audio resolution reconstruction (SPAR) metadata, based on the plurality of compressed channel signals and based on the multi-channel input signal, wherein the joint decoding metadata is such that it allows upmixing the plurality of compressed channel signals to an approximation of the multi-channel input signal. The method further comprises encoding the plurality of compressed channel signals and the joint decoding metadata.

[0009] According to another aspect, a method of determining a reconstructed multi-channel signal from coded audio data indicating a plurality of reconstructed channel signals and from coded metadata indicating joint coding metadata is described. The method comprises decoding the coded audio data to provide the plurality of reconstructed channel signals and decoding the coded metadata to provide the joint coding metadata. Furthermore, the method comprises determining the reconstructed multi-channel signal from the plurality of reconstructed channel signals using the joint coding metadata.

[0010] According to another aspect, a software program is described. The software program may be adapted to be executed on a processor and, when executed on the processor, performs the method steps outlined in this document.

[0011] According to another aspect, a storage medium is described. The storage medium may include a software program adapted to be executed on a processor and which, when executed on the processor, performs the method steps outlined in this document.

[0012] According to another aspect, a computer program product is described. The computer program may comprise executable instructions for carrying out the method steps outlined in this document when executed on a computer.

[0013] According to another aspect, an encoding unit or an encoding device for encoding a multi-channel input signal and / or an immersive audio (IA) signal is described. The encoding unit is configured to determine a plurality of downmix channel signals from the multi-channel input signal. Furthermore, the encoding unit is configured to perform energy compression on the plurality of downmix channel signals to provide a plurality of compressed channel signals. In addition, the encoding unit is configured to determine joint decoding metadata based on the plurality of compressed channel signals and based on the multi-channel input signal, wherein the joint decoding metadata allows upmixing the plurality of compressed channel signals to an approximation of the multi-channel input signal. The encoding unit is further configured to encode the plurality of compressed channel signals and the joint decoding metadata.

[0014] According to another aspect, a decoding unit or a decoding device for determining a reconstructed multi-channel signal from coded audio data indicating a plurality of reconstructed channel signals and from coded metadata indicating joint decoding metadata is described. The decoding unit is configured to decode the coded audio data to provide the plurality of reconstructed channel signals and to decode the coded metadata to provide the joint decoding metadata. Furthermore, the decoding unit is configured to determine the reconstructed multi-channel signal from the plurality of reconstructed channel signals using the joint decoding metadata.

[0015] It should be noted that the methods, devices, and systems (including preferred embodiments thereof) outlined in this patent application can be used alone or in combination with other methods, devices, and systems disclosed in this document. In addition, all aspects of the methods, devices, and systems outlined in this patent application can be arbitrarily combined. Specifically, the features of the technical solutions can be arbitrarily combined with each other. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The present invention is explained exemplarily below with reference to the accompanying drawings, in which

[0017] Figure 1 Demonstrating an example decoding system;

[0018] Figure 2 An example encoding unit for encoding an immersive audio signal is presented;

[0019] Figure 3 Another exemplary decoding unit for decoding an immersive audio signal is shown;

[0020] Figure 4An exemplary encoding unit and decoding unit for encoding and decoding an immersive audio signal are shown;

[0021] Figure 5 Shows an example encoding unit and decoding unit with mode switching;

[0022] Figure 6 Demonstrate an example refactoring module;

[0023] Figure 7 A flowchart showing an example method for encoding an immersive audio signal; and

[0024] Figure 8 A flow chart showing an example method for decoding data indicative of an immersive audio signal. DETAILED DESCRIPTION

[0025] As outlined above, the present document relates to efficient decoding of immersive audio (IA) signals (e.g., first-order Ambisonics (FOA) or HOA signals, multi-channel and / or object audio signals), where in particular FOA or HOA signals are more commonly referred to herein as soundfield representation (SR) signals.

[0026] As outlined in the introductory section, an SR signal may include a relatively high number of channels or waveforms, where different channels are associated with different panning functions and / or different directivity patterns. For example, an L-order 3D FOA or HOA signal includes (L+1) 2 The SR signal can be represented in various formats.

[0027] The sound field can be considered to consist of one or more sound wave events emanating from arbitrary directions around the listening position. Thus, the location of the one or more sound wave events can be defined on the surface of a sphere (with the listening position or reference position at the center of the sphere).

[0028] Sound field formats, such as FOA or higher-order Ambisonics (HOA), are defined to allow the sound field to be presented via an arbitrary speaker arrangement, i.e., an arbitrary presentation system. However, presentation systems, such as Dolby Atmos systems, are typically limited in that the possible altitudes of the speakers are fixed to a defined number of planes, such as the ear-height (horizontal) plane, the ceiling plane or upper plane, and / or the floor plane or lower plane. Therefore, the concept of an ideal spherical sound field can be modified to a sound field consisting of sound wave objects in different rings located at various heights on the surface of a sphere, similar to the stacked rings that make up a honeycomb.

[0029] like Figure 1, the audio coding system 100 includes an encoding unit 110 and a decoding unit 120. The encoding unit 110 may be configured to generate a bitstream 101 for transmission to the decoding unit 120 based on an input signal 111, wherein the input signal 111 may include an immersive audio signal (e.g., for virtual reality (VR) applications). The immersive audio signal may include an SR signal, a multi-channel (test bed) signal, and / or a plurality of objects (each object including an object signal and object metadata). The decoding unit 120 may be configured to provide an output signal 121 based on the bitstream 101, wherein the output signal 121 may include a reconstructed immersive audio signal.

[0030] Figure 2 An exemplary encoding unit 110, 200 is illustrated. The encoding unit 200 may be configured to encode an input signal 111, wherein the input signal 111 may be an immersive audio (IA) input signal 111. The IA input signal 111 may include a multi-channel input signal 201. The multi-channel input signal 201 may include an SR signal and one or more object signals. Furthermore, object metadata 202 of the plurality of object signals may be provided as part of the IA input signal 111. The IA input signal 111 may be provided by a content capture engine, wherein the content capture engine may be configured to derive objects and / or SR signals from (complex) VR content.

[0031] The encoding unit 200 comprises a downmixing module 210 configured to downmix a multi-channel input signal 201 to a plurality of downmixed channel signals 203. The plurality of downmixed channel signals 203 may correspond to SR signals, in particular to first order Ambisonics (FOA) signals. The downmixing may be performed in the subband domain or in the QMF domain (e.g. using 10 or more subbands).

[0032] The encoding unit 200 further comprises a joint decoding module 230, in particular a SPAR module, which is configured to determine joint decoding metadata 205, in particular SPAR spatial audio resolution reconstruction metadata, which is configured to reconstruct the multi-channel input signal 201 from the plurality of downmix channel signals 203. The joint decoding module 230 may be configured to determine the joint decoding metadata 205 in the subband domain.

[0033] To determine the joint decoding metadata 205, the plurality of downmix channel signals 203 may be transformed into the subband domain and / or the plurality of downmix channel signals 203 may be processed in the subband domain. Furthermore, the multi-channel input signal 201 may be transformed into the subband domain. Subsequently, the joint decoding metadata 205 may be determined for the subbands, in particular such that an approximation of the subband signals of the multi-channel input signal 201 is obtained by upmixing the subband signals of the plurality of downmix channel signals 203 using the joint decoding metadata 205. The joint decoding metadata 205 for different subbands may be inserted into the bitstream 101 for transmission to the corresponding decoding unit 120.

[0034] In addition, the encoding unit 200 may include a decoding module 240 configured to perform waveform encoding on the plurality of downmix channel signals 203, thereby providing decoded audio data 206. Each of the downmix channel signals 203 may be encoded using a mono waveform encoder, such as 3GPP EVS encoding, to achieve efficient encoding. Other examples of encoding the plurality of downmix channel signals 203 are MPEG AAC, MPEG HE-AAC and other MPEG audio codecs, 3GPP codecs, Dolby Digital / Dolby Digital Plus (AC-3, eAC-3), Opus, LC-3, and similar codecs. As another example, the decoding tools included in the AC-4 codec may also be configured to perform the operations of the encoding unit 200.

[0035] Furthermore, the coding module 240 may be configured to perform entropy encoding on the joint coding metadata (ie, SPAR metadata) 205 and the object metadata 202, thereby providing coded metadata 207. The coded audio data 206 and the coded metadata 207 may be inserted into the bitstream 101.

[0036] Figure 3 An example decoding unit 120, 350 is shown. The decoding unit 120, 350 may include a receiver that receives a bitstream 101 that may include coded audio data 206 and coded metadata 207. The decoding unit 120, 350 may include a processor and / or a demultiplexer that demultiplexes the coded audio data 206 and the coded metadata 207 from the bitstream 101. The decoding unit 350 includes a decoding module 360 ​​configured to derive a plurality of reconstructed channel signals 314 from the coded audio data 206. The decoding module 360 ​​may further be configured to derive the joint coded metadata 205 and the object metadata 202 from the coded metadata 207.

[0037] In addition, the decoding unit 350 comprises a reconstruction module 370 configured to derive a reconstructed multi-channel signal 311 from the joint decoding metadata 205 and from the plurality of reconstructed channel signals 314. The joint decoding metadata 205 may convey time-varying elements and / or frequency-varying elements of an upmix matrix allowing the multi-channel signal 311 to be reconstructed from the plurality of reconstructed channel signals 314. The upmix process may be performed in the QMF (quadrature mirror filter) subband domain. Alternatively, another time / frequency transform, in particular a transform based on an FFT (Fast Fourier Transform) may be used to implement the upmix process. In general, a transform that achieves frequency selective analysis and (upmix) processing may be applied. The upmix process may also include a decorrelator capable of improving the reconstruction of the covariance of the reconstructed multi-channel signal 311, wherein the decorrelator may be controlled by the additional joint decoding metadata 205.

[0038] The reconstructed multi-channel signal 311 may include a signal referred to as a reconstructed SR signal and include one or more reconstructed object signals. The reconstructed multi-channel signal 311 and the object metadata may form a reconstructed IA signal 121. The reconstructed IA signal 121 may be used for speaker rendering 330, for headphone rendering 331 and / or for SR rendering 332.

[0039] Figure 4 The diagram illustrates the encoding unit 200 and the decoding unit 350. The encoding unit 200 includes Figure 2 Furthermore, the encoding unit 200 includes an energy compression module 420 configured to concentrate the energy of the plurality of downmix channel signals 203 to one or more downmix channel signals 203. The energy compression module 420 may transform the downmix channel signals 203 to provide a plurality of compressed channel signals 404. The transformation may be performed such that one or more of the compressed channel signals 404 has less energy than the corresponding one or more downmix channel signals 203.

[0040] For example, the plurality of downmix channel signals 203 may include a W channel signal, an X channel signal, a Y channel signal, and a Z channel signal. The plurality of compressed channel signals 404 may include a W channel signal, an X' channel signal, a Y' channel signal, and a Z' channel signal. The X' channel signal, the Y' channel signal, and the Z' channel signal may be determined such that the X' channel signal has less energy than the X channel signal, such that the Y' channel signal has less energy than the Y channel signal, and / or such that the Z' channel signal has less energy than the Z channel signal.

[0041] The energy compression module 420 may be configured to perform energy compression using a prediction operation. Specifically, a first subset of the plurality of downmix channel signals 203 (e.g., X channel signals, Y channel signals, and Z channel signals) may be predicted from a second subset of the plurality of downmix channel signals 203 (e.g., W channel signals). The energy compression may include subtracting a scaled version of one of the downmix channel signals 203 (e.g., W channel signals) from the other downmix channel signals 203 (e.g., X channel signals, Y channel signals, and / or Z channel signals). The scaling factor may be determined such that the energy of the other downmix channel signals 203 is reduced, in particular minimized.

[0042] By performing energy compression, the efficiency of encoding the plurality of compressed channel signals 404 can be improved compared to encoding the plurality of downmix channel signals 203. The encoding unit 200 is configured to implicitly insert metadata for performing the inversion of the energy compression operation into the joint decoding metadata 205. Therefore, efficient encoding of the IA input signal 111 is achieved.

[0043] As outlined above, the decoding unit comprises a reconstruction module 370 . Figure 6 An exemplary reconstruction module 370 is illustrated. The reconstruction module 370 takes as input the plurality of reconstructed channel signals 314 (e.g., which may form a first-order Ambisonics signal). A first mixer 611 may be configured to upmix the plurality of reconstructed channel signals 314 (e.g., four channel signals) into an increased number of signals (e.g., 11 signals, which represent a second-order Ambisonics signal and two object signals). The first mixer 611 depends on the joint decoding metadata 205.

[0044] The reconstruction module 370 may include decorrelators 601, 602 configured to generate two signals from the W channel signal, which are processed in a second mixer 612 to generate an increased number of signals (e.g., 11 signals). The second mixer 612 depends on the joint decoding metadata 205. The output of the first mixer 611 is added to the output of the second mixer 612 to provide a reconstructed multi-channel signal 311.

[0045] As indicated above, the joint coding or SPAR metadata 205 may consist of data representing coefficients of upmix matrices used by the first mixer 611 and by the second mixer 612. The mixers 611, 612 may operate in the subband domain, in particular, in the QMF domain. In this case, the joint coding or SPAR metadata 205 includes data representing coefficients of upmix matrices used by the first mixer 611 and by the second mixer 612 for a plurality of different subbands, for example, 10 or more subbands.

[0046] Figure 5An encoding unit 200 is shown, which comprises two branches for encoding a multi-channel input signal 201 and for encoding object metadata 202 (the multi-channel input signal 201 and the object metadata 202 form the IA input signal 111). The upper branch corresponds to Figure 4 In the lower branch, the joint decoding unit 230 is modified to determine metadata 205 that allows reconstruction of the plurality of downmix channel signals 203 from the plurality of compressed channel signals 404. Thus, the metadata 205 indicates prediction factors (in particular one or more scaling factors) that have been used to generate the plurality of compressed channel signals 404 from the plurality of downmix channel signals 203. In a variant, the metadata 205 may be provided directly from the energy compression module 220 (without using the joint decoding module 230).

[0047] Figure 5 The encoding unit 200 includes a mode switching module 500 configured to switch between a first mode (corresponding to an upper branch) and a second mode (corresponding to a lower branch). The first mode can be used to provide high perceptual quality at an increased bit rate, and the second mode can be used to provide reduced perceptual quality at a reduced bit rate. The mode switching module 500 can be configured to switch between the first mode and the second mode depending on the state of the transmission network.

[0048] also, Figure 5 The corresponding decoding unit 350 is shown configured to perform decoding according to the first mode (upper branch) and according to the second mode (lower branch). The mode switching module 550 can be configured to determine which mode the encoding unit 200 has used (e.g., frame by frame). If the first mode has been used, the reconstructed multi-channel signal 311 and the object metadata 202 (e.g., Figure 4 On the other hand, if the second mode has been used, the decoding unit 350 may determine a plurality of reconstructed downmix channel signals 513 (corresponding to the plurality of downmix channel signals 203).

[0049] Thus, an encoding unit 200 is described that includes a downmix module 210 configured to process object and HOA input signals 111 to generate an output signal 203 with a reduced number of channels, such as a first order Ambisonics (FOA) signal. A SPAR encoding module 230 generates metadata (i.e., SPAR metadata) 205 indicating how the original input 111, 201 (e.g., object signal plus HOA) can be regenerated from the FOA signal 203. A set of EVS encoders 240 can take the 4-channel FOA signal 203 and can form encoded audio data 206 for insertion into the bitstream 101, which is then decoded by a set of EVS decoders 360 to form a four-channel FOA signal 314. The SPAR metadata 205 can be provided as (entropy) encoded metadata 207 within the bitstream 101 to the decoder 360. Subsequently, a reconstruction module 370 reproduces the output 121 consisting of audio objects and HOA signals.

[0050] The WXYZ energy compression transform (in module 420) may modify the low resolution signal 203 produced by the downmix module 210, which may generate an output signal 404 having reduced inter-channel correlation compared to the output of the downmix module 210. The purpose of the energy compression filter 420 is to reduce the energy in the XYZ channels so that the W channel can be encoded at a higher bit rate and the low energy X'Y'Z' channels can be encoded at a lower bit rate. This more efficiently masks decoding artifacts, thereby improving audio quality.

[0051] In addition or in lieu of performing prediction, energy compression may utilize a Karhonen Loeve Transform (KLT), a principal component analysis (PCA) transform, and / or a singular value decomposition (SVD) transform. Specifically, an energy compression filter 420 may be used, which includes a whitening filter, a KLT, a PCA transform, and / or an SVD transform. The whitening filter may be implemented using the above-described prediction scheme. Specifically, the energy compression filter 420 may include a combination of a whitening filter and a KLT transform, a PCA transform, and / or an SVD transform, wherein the latter is arranged in series with the whitening filter. The KLT transform, the PCA transform, and / or the SVD transform may be applied to the X channel, the Y channel, the Z channel, and in particular to the prediction residual.

[0052] Figure 7A flow chart of an exemplary method 700 for encoding a multi-channel input signal 201 is shown. Specifically, the method 700 is intended to encode an IA signal comprising the multi-channel input signal 201. The multi-channel input signal 201 may comprise a sound field representation (SR) signal. Specifically, the multi-channel input signal 201 may comprise a combination of a SR signal (e.g. a HOA signal, in particular a second-order Ambisonics signal) and one or more (in particular two) object signals of one or more audio objects 303.

[0053] The method 700 comprises determining 701 a plurality of downmix channel signals 203 from a multichannel input signal 201. The plurality of downmix channel signals 203 may comprise a reduced number of channels compared to the multichannel input signal 201. As indicated above, the multichannel input signal 201 may comprise: an SR signal, in particular an L-order Ambisonics signal, where L≥1, and one or more object signals of one or more audio objects 303. The plurality of downmix channel signals 203 may be determined by downmixing the multichannel input signal 201 to a SR signal, in particular an K-order Ambisonics signal, where L≥K. Hence, the plurality of downmix channel signals 203 may be SR signals, in particular K-order Ambisonics signals.

[0054] Specifically, determining 701 the plurality of downmixed channel signals 203 may include mixing one or more object signals of one or more audio objects 303 (the multi-channel input signal 201) to the SR signal (or a downmixed version of the SR signal) of the multi-channel input signal 201. The mixing (particularly the panning) may be performed in dependence on object metadata 202 of the one or more audio objects 303, wherein the object metadata 202 of the audio objects 303 indicates a spatial position of the audio objects 303. The downmixed SR signal may include removing [(L+1)2-L2] additional channels from the L-order SR signal, thereby providing a (L-1)-order SR signal.

[0055] In a preferred example, the plurality of downmixed channel signals 203 form a first order Ambisonics signal, in particular in B format or in A format. The SR signal of the multi-channel input signal 201 may be a second order (or higher order) Ambisonics signal.

[0056] Furthermore, the method 700 comprises carrying out 702 energy compression on the plurality of downmix channel signals 203 to provide a plurality of compressed channel signals 404. The number of channels of the plurality of downmix channel signals 203 may be the same as the number of channels of the plurality of compressed channel signals 404. Specifically, the plurality of compressed channel signals 404 may form or may be in a first-order Ambisonics signal format, in particular in a B-format or in an A-format.

[0057] Energy compression may be performed such that inter-channel correlation between different channel signals 203 is reduced. Specifically, the plurality of compressed channel signals 404 may exhibit less inter-channel correlation than the plurality of downmix channel signals 203. Alternatively or additionally, energy compression may be performed such that the energy of the compressed channel signal is lower than or equal to the energy of the corresponding downmix channel signal. This condition may be satisfied for each channel.

[0058] Performing 702 energy compression may include predicting a first downmix channel signal 203 (e.g., an X channel, a Y channel, or a Z channel) from a second downmix channel signal (e.g., a W channel) to provide a first predicted channel signal. The first predicted channel signal may be subtracted from the first downmix channel signal 203 (or vice versa) to provide a first compressed channel signal 404.

[0059] Predicting the first downmix channel signal 203 from the second downmix channel signal 203 may include determining a scaling factor for scaling the second downmix channel signal 203. The scaling factor may be determined such that the energy of the first compressed channel signal 404 is reduced compared to the energy of the first downmix channel signal 203 and / or such that the energy of the first compressed channel signal 404 is minimized. The first predicted channel signal may then correspond to the second downmix channel signal 203 scaled according to the scaling factor. Different scaling factors may be determined for different channels.

[0060] Specifically (in the case of a first-order Ambisonics signal), carrying out 702 the energy compression may include predicting an X-channel signal, a Y-channel signal, and a Z-channel signal from the W-channel signal of the plurality of downmix channel signals 203 to provide a predicted X-channel signal, a predicted Y-channel signal, and a predicted Z-channel signal, respectively. The predicted X-channel signal may be subtracted from the X-channel signal (or vice versa) to determine an X′-channel signal of the plurality of compressed channel signals 404. The predicted Y-channel signal may be subtracted from the Y-channel signal (or vice versa) to determine the Y′-channel signal of the plurality of compressed channel signals 404. The predicted Z-channel signal may be subtracted from the Z-channel signal (or vice versa) to determine the Z′-channel signal of the plurality of compressed channel signals 404. Furthermore, the W-channel signal of the plurality of downmix channel signals 203 may be used as the W-channel signal of the plurality of compressed channel signals 404.

[0061] Therefore, the energy of all channels (except one, ie, the W channel) may be reduced, thereby enabling efficient encoding of the plurality of compressed channel signals 404.

[0062] The method 700 may further include determining 703 joint coding metadata (also referred to herein as SPAR metadata) 205 based on the plurality of compressed channel signals 404 and based on the multi-channel input signal 201. The joint coding metadata 205 may be determined such that the joint coding metadata 205 allows upmixing the plurality of compressed channel signals 404 to an approximation of the multi-channel input signal 201. By utilizing the plurality of compressed channel signals 404 to determine the joint coding metadata, the process of inverse energy compression is automatically included in the joint coding metadata 205 (without providing additional metadata for the inverse energy compression operation).

[0063] The joint decoding metadata 205 may comprise upmix data, in particular comprising one or more upmix matrices, enabling upmixing of the plurality of compressed channel signals 404 to an approximation of the multi-channel input signal 201. The approximation of the multi-channel input signal 201 comprises the same number of channels as the multi-channel input signal 201. Furthermore, the joint decoding metadata 205 may comprise decorrelation data enabling reconstruction of the covariance of the multi-channel input signal 201.

[0064] The joint decoding metadata 205 may be determined for a plurality of different subbands (eg for 10 or more subbands, in particular in the QMF domain) of the multi-channel input signal 201. By providing the joint decoding metadata 205 for different subbands (ie in different frequency bands), an accurate upmixing operation may be performed.

[0065] In addition, the method 700 includes encoding 704 the plurality of compressed channel signals 404 and the joint decoding metadata 205 (also referred to as SPAR metadata). Encoding 704 the plurality of compressed channel signals 404 may include performing waveform encoding (especially performing EVS encoding) on ​​each of the plurality of compressed channel signals 404, especially using a mono encoder to perform waveform encoding on each compressed channel signal 404. Alternatively or additionally, the joint decoding metadata 205 may be encoded using an entropy encoder. As indicated above, the multi-channel input signal 201 may include one or more object signals of one or more audio objects 303. In these cases, the method 700 may include encoding the object metadata 202 of the one or more audio objects 303, especially using an entropy encoder.

[0066] The method 700 allows encoding a multi-channel input signal 201 , which may be indicative of an SR signal and / or one or more audio object signals, in a bit rate efficient manner, while enabling a decoder to reconstruct the multi-channel input signal 201 with a high perceptual quality.

[0067] Determining the joint coding metadata 205 based on the plurality of compressed channel signals 404 and based on the multi-channel input signal 201 may correspond to a first mode of encoding the multi-channel input signal 201 .

[0068] Alternatively or in addition to using prediction, performing 702 energy compression may include applying Karhunen-Loewen transform, principal component analysis transform and / or singular value decomposition transform to at least some of the plurality of downmix channel signals 203. Thereby, the coding efficiency of the plurality of compressed channel signals 404 may be further increased.

[0069] Specifically, Karhunen-Loewen transform, principal component analysis transform and / or singular value decomposition transform may be applied to the compressed channel signal 404 corresponding to the prediction residual that has been derived based on the second downmix channel signal 203 (in particular based on the W channel signal). In other words, Karhunen-Loewen transform, principal component analysis transform and / or singular value decomposition transform may be applied to the prediction residual.

[0070] As indicated above, in the context of prediction, an X' channel signal, a Y' channel signal, and a Z' channel signal may be derived based on the W channel signal of the plurality of downmix channel signals 203 to form an Ambisonics signal. Specifically, the X' channel signal may correspond to an X channel signal minus a prediction of the X channel signal, the X' channel signal being based on the W channel signal. Likewise, the Y' channel signal may correspond to a Y channel signal minus a prediction of the Y channel signal, the Y' channel signal being based on the W channel signal. Likewise, the Z' channel signal may correspond to a Z channel signal minus a prediction of the Z channel signal, the Z' channel signal being based on the W channel signal. The plurality of compressed channel signals 404 may be determined based on the W channel signal, the X' channel signal, the Y' channel signal, and the Z' channel signal, or may correspond to the W channel signal, the X' channel signal, the Y' channel signal, and the Z' channel signal.

[0071] To further improve the coding efficiency of the plurality of compressed channel signals 404, Karhunen-Rossler transform, principal component analysis transform, and / or singular value decomposition transform may be applied to the X' channel signal, the Y' channel signal, and the Z' channel signal to provide an X" channel signal, a Y" channel signal, and a Z" channel signal. Then, the plurality of compressed channel signals 404 may be determined based on the W channel signal, the X" channel signal, the Y" channel signal, and the Z" channel signal.

[0072] In the second mode, the joint coding metadata 205 may be determined based on the plurality of compressed channel signals 404 and based on the plurality of downmix channel signals 203. The joint coding metadata 205 may be determined such that the joint coding metadata 205 allows reconstruction of the plurality of downmix channel signals 203 from the plurality of compressed channel signals 404. Specifically, the joint coding metadata 205 may be determined such that the joint coding metadata 205 (only) reverses or inverts the energy compression operation (without performing an upmix operation). The second mode may be used to reduce the bit rate (with reduced perceptual quality).

[0073] As indicated above, the multi-channel input signal 201 may include an SR signal and one or more object signals. The first mode and the second mode may allow the SR signal to be reconstructed (based on the plurality of compressed channel signals 404). Thus, the overall listening experience of the listener may be maintained (even when the second mode is used).

[0074] The multi-channel input signal 201 may include a sequence of frames. The processing described in this document may be performed frame by frame for each frame in the sequence of frames. Specifically, the method 700 may include determining whether to use the first mode or the second mode for each frame in the sequence of frames. Thereby, the encoding may be adapted to rapidly changing conditions of the transmission network.

[0075] The method 700 may include generating a bitstream 101 based on the coded audio data 206 derived by encoding 704 the plurality of compressed channel signals 404 and based on the coded metadata 207 derived by encoding 704 the joint decoding metadata 205. Furthermore, the method 700 may include inserting an indication indicating whether the second mode or the first mode has been used into the bitstream 101. The indication may be inserted frame by frame. Thus, the corresponding decoding unit 350 is enabled to adapt decoding in a reliable manner.

[0076] Figure 8 A flowchart is shown of an example method 800 for determining a reconstructed multi-channel signal 311 from coded audio data 206 indicative of a plurality of reconstructed channel signals 314 and from coded metadata 207 indicative of joint coding metadata 205. The method 800 may include extracting the coded audio data 206 and the coded metadata 207 from the bitstream 101.

[0077] Furthermore, the method 800 may comprise decoding 801 the coded audio data 206 to provide the plurality of reconstructed channel signals 314 and decoding the coded metadata 207 to provide the joint coded metadata 205. In a preferred example, the plurality of reconstructed channel signals 203 form a first-order Ambisonics signal, in particular in a B-format or in an A-format.

[0078] Decoding 801 the coded audio data 206 may include waveform decoding each of the plurality of reconstructed channel signals 314, in particular using a mono decoder such as an EVS decoder to waveform decode each reconstructed channel signal 314. The coded metadata 207 may be decoded using an entropy decoder.

[0079] Furthermore, the method 800 includes determining 802 a reconstructed multi-channel signal 311 from the plurality of reconstructed channel signals 314 using the joint decoding metadata 205, wherein the reconstructed multi-channel signal 311 may include a reconstructed sound field representation (SR) signal. Specifically, the reconstructed multi-channel signal 311 corresponds to an approximation or reconstruction of the multi-channel input signal 201. The reconstructed multi-channel signal 311 and the object metadata 202 may together form a reconstructed immersive audio (IA) signal 121.

[0080] In addition, the method 800 may include rendering the reconstructed multi-channel signal 311 (typically in combination with the object metadata 202). The rendering may be performed using headphone rendering, speaker rendering, and / or sound field rendering. Thus, spatial audio content may be flexibly rendered (particularly for VR applications).

[0081] As indicated above, the joint decoding metadata 205 may include upmix data, in particular one or more upmix matrices, enabling upmixing of the plurality of reconstructed channel signals 404 into the reconstructed multi-channel signal 311. Furthermore, the joint decoding metadata 205 may include decorrelation data enabling generation of the reconstructed multi-channel signal 311 with a predetermined covariance. The joint decoding metadata 205 may include different metadata for different subbands of the reconstructed multi-channel signal 311. Thus, an accurate reconstruction of the multi-channel input signal 201 may be achieved.

[0082] At the corresponding encoder 200, energy compression may have been applied to the plurality of downmix channel signals 304. Energy compression may have been performed using prediction and / or using a Karhunen-Loewe transform, a principal component analysis transform, and / or a singular value decomposition transform. The joint decoding metadata 205 may be such that it implicitly performs an inverse of the energy compression operation in addition to the upmix. Specifically, the joint decoding metadata 205 may be such that it implicitly performs an inverse of the prediction operation and / or an inverse of a Karhunen-Loewe transform, a principal component analysis transform, and / or a singular value decomposition transform in addition.

[0083] In other words, the joint decoding metadata 205 may be configured to enable upmixing of the plurality of reconstructed channel signals 404 into the reconstructed multi-channel signal 311 and to (implicitly) perform an inverse energy compression operation on the plurality of reconstructed channel signals 314. Specifically, the joint decoding metadata 205 may be configured to (implicitly) perform an inverse prediction operation (an inverse of the prediction operation performed by the encoder 200) on at least some of the plurality of reconstructed channel signals 314. Alternatively or additionally, the joint decoding metadata 205 may be configured to perform an inverse of a Karhunen-Loewen transform, a principal component analysis transform, and / or a singular value decomposition transform (an inverse of the transform performed by the encoder 200) on at least some of the plurality of reconstructed channel signals 314. Thus, a particularly efficient decoding scheme may be provided.

[0084] The reconstructed multi-channel signal 311 may include one or more reconstructed object signals (in addition to SR signals such as FOA or HOA signals) of the one or more audio objects 303. The method 800 may include decoding the object metadata 202 of the one or more audio objects 303 from the coded metadata 207, in particular using an entropy decoder. Thus, the one or more objects 303 may be accurately presented.

[0085] As indicated above, the plurality of reconstructed channel signals 314 may form an SR signal, in particular a K-order Ambisonics signal, where K≥1 (in particular K=1). On the other hand, the reconstructed multi-channel signal 311 may include: a reconstructed SR signal, in particular an L-order Ambisonics signal, where L≥K (in particular L=K or L=K+1); and one or more (e.g. n=2) reconstructed object signals of one or more audio objects 303. The reconstructed multi-channel signal 311 may be determined by upmixing the plurality of reconstructed channel signals 314 using the joint decoding metadata 205, thereby providing a reconstructed multi-channel signal 311 with a large number of spatial acoustic events.

[0086] As indicated above, using upmixing may correspond to a first mode (achieving high perceptual quality). In the first mode, the joint object metadata 205 includes upmix data for achieving an upmix operation. In a second mode, the reconstructed multi-channel signal 311 may include the same number of channels as the plurality of reconstructed channel signals 314 (so that no upmix operation is required).

[0087] In the second mode, the joint decoding metadata 205 may include prediction data (e.g., one or more scaling factors) configured to redistribute energy among the different reconstructed channel signals 314. Furthermore, in the second mode, determining 802 the reconstructed multi-channel signal 311 may include using the prediction data to redistribute energy among the different reconstructed channel signals 314. Specifically, the inverse of the energy compression operation described above may be performed using the joint decoding metadata 205. Thus, the plurality of downmix channel signals 203 may be efficiently and accurately reconstructed.

[0088] As outlined above, the energy compression operations performed during encoding may include applying a Karhunen-Loewen transform, a principal component analysis transform, and / or a singular value decomposition transform to at least some of the plurality of downmix channel signals 203. The joint decoding metadata 205 may include transform data that enables the decoder 350 to perform an inversion of the Karhunen-Loewen transform, the principal component analysis transform, and / or the singular value decomposition transform. In other words, the transform data indicates an inversion of the Karhunen-Loewen transform, the principal component analysis transform, and / or the singular value decomposition transform applied to at least some of the plurality of reconstructed channel signals 314 used to determine the reconstructed multi-channel signal 311. Thus, the plurality of downmix channel signals 203 may be efficiently and accurately reconstructed.

[0089] As indicated above, the reconstructed multi-channel input signal 311 may comprise a sequence of frames. The method 800 may comprise determining for each frame in the sequence of frames whether the second mode is to be used. To this end, an indication may be extracted from the bitstream 101 indicating whether the second mode is to be used.

[0090] Various exemplary embodiments may be implemented in hardware or special purpose circuits, software, logic or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software executable by a controller, microprocessor or other computing device. In general, it should be understood that the present invention also encompasses devices suitable for implementing the methods described above, such as devices having a memory and a processor coupled to the memory (spatial renderer), wherein the processor is configured to execute instructions and implement methods according to embodiments of the present invention.

[0091] Although various aspects of example embodiments of the invention may be illustrated and described in block diagram form, flow chart form, or using some other pictorial representation, it should be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented by, by way of non-limiting example, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controllers, or other computing devices, or some combination thereof.

[0092] In addition, the various blocks shown in the flowchart may be viewed as method steps and / or operations derived from the operation of computer program code, and / or as a plurality of coupled logic circuit elements constructed to perform the associated functions. For example, embodiments of the present invention include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, wherein the computer program contains program code configured to perform the method described above.

[0093] In the context of the present invention, a machine-readable medium may be any tangible medium that may contain or store a program used by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium would include an electrical connection with one or more wirings, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0094] Computer program code for implementing the method of the present invention may be written in any combination of one or more programming languages. Such computer program code may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor of the computer or other programmable data processing device, causes the implementation of the functions / operations specified in the flow chart and / or block diagram. The program code may be executed partially entirely on the computer, partially on the computer, partially on the computer and partially on a remote computer, or entirely on a remote computer or server as a separate software package.

[0095] In addition, although the operations are depicted in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or in a sequential order or that all of the illustrated operations be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Similarly, although the above discussion contains several specific implementation details, these should not be considered to limit the scope of any invention or the content that can be claimed, but rather to illustrate features that may be unique to a particular embodiment of a particular invention. Certain features described in this specification in the context of a separate embodiment may also be implemented in a single embodiment in combination. Conversely, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination.

[0096] It should be noted that the description and drawings illustrate only the principles of the proposed method and apparatus. It should therefore be understood that a person skilled in the art will be able to envision various arrangements that, although not explicitly described or shown herein, embody the principles of the present invention and are included in the spirit and scope of the present invention. In addition, all examples described herein are in principle expressly intended to achieve only illustrative purposes to help the reader understand the principles of the proposed method and apparatus and the concepts that the inventors have contributed to advance this technology, and should be considered as not limited to such specifically described examples and conditions. In addition, all statements describing the principles, aspects, and embodiments of the present invention herein and specific examples of the present invention are intended to encompass their equivalent forms.

Claims

1. A method for encoding a multi-channel input Ambisonics signal, wherein the method comprises: - determining a plurality of downmix channel signals from said multi-channel input Ambisonics signal; - performing energy compression on the plurality of downmix channel signals to provide a plurality of compressed channel signals, wherein the plurality of downmix channel signals include a Y channel signal, at least one of an X channel signal and a Z channel signal, and a W channel signal, wherein the plurality of compressed channel signals include at least one of a corresponding Y' channel signal, an X' channel signal and a Z' channel signal, and the W channel signal, and wherein the X' channel signal, the Y' channel signal and the Z' channel signal are determined such that the Y' channel signal has less energy than the Y channel signal, such that the X' channel signal has less energy than the X channel signal, and such that the Z' channel signal has less energy than the Z channel signal; - determining audio reconstruction metadata based on the plurality of compressed channel signals and based on the multi-channel input Ambisonics signal; wherein the audio reconstruction metadata causes a receiving device to upmix the plurality of compressed channel signals to an approximation of the multi-channel input Ambisonics signal; and - encoding the plurality of compressed channel signals and the audio reconstruction metadata.

2. The method of claim 1, wherein performing energy compression comprises - predicting the first downmix channel signal from the second downmix channel signal to provide a first predicted channel signal; and - subtracting the first predicted channel signal from the first downmix channel signal to provide a first compressed channel signal.

3. The method according to claim 2, wherein - predicting the first downmix channel signal from the second downmix channel signal comprises determining a scaling factor for scaling the second downmix channel signal; and - the first predicted channel signal corresponds to the second downmix channel signal scaled according to the scaling factor.

4. The method of claim 3, wherein the scaling factor is determined such that At least one of the following (1) or (2) is true: (1) the energy of the first compressed channel signal is reduced compared to the energy of the first downmix channel signal; (2) the energy of the first compressed channel signal is minimized.

5. The method of claim 2, wherein performing energy compression comprises - determining a number of compressed channel signals based on a prediction from said second downmix channel signal; and - Applying one of a Karhunen-Loewe transform, a principal component analysis transform or a singular value decomposition transform to the plurality of compressed channel signals.

6. The method according to claim 1, wherein At least one of the following (1) or (2) is true: (1) the plurality of downmix channel signals are first-order Ambisonics signals in B format or in A format; (2) the plurality of compressed channel signals are represented in a first-order Ambisonics signal format, in B format, or in A format.

7. The method of claim 6, wherein performing energy compression comprises - predicting the X channel signal, the Y channel signal and the Z channel signal from the W channel signal of the plurality of downmix channel signals to provide a predicted X channel signal, a predicted Y channel signal and a predicted Z channel signal; - subtracting the predicted X channel signal from the X channel signal to determine the X' channel signal; - subtracting the predicted Y channel signal from the Y channel signal to determine the Y' channel signal; - subtracting the predicted Z channel signal from the Z channel signal to determine the Z' channel signal; and - determining the plurality of compressed channel signals based on the W channel signal, the X′ channel signal, the Y′ channel signal and the Z′ channel signal.

8. The method of claim 7, wherein performing energy compression comprises - applying one of Karhunen-Loewen transform, principal component analysis transform or singular value decomposition transform to the X' channel signal, the Y' channel signal and the Z' channel signal to provide an X" channel signal, a Y" channel signal and a Z" channel signal; and - determining the plurality of compressed channel signals based on the W channel signal, the X" channel signal, the Y" channel signal and the Z" channel signal.

9. The method of claim 1, wherein effecting energy compression comprises applying one of a Karhunen-Loewe transform, a principal component analysis transform, or a singular value decomposition transform to at least some of the plurality of downmix channel signals.

10. The method of claim 1, wherein the audio reconstruction metadata comprises - at least one of upmix data, an upmix matrix enabling said upmixing of said plurality of compressed channel signals to an approximation of said multi-channel input Ambisonics signal thereby comprising the same number of channels as said multi-channel input Ambisonics signal; or - decorrelation data enabling reconstruction of the covariance of said multi-channel input Ambisonics signal.

11. The method of claim 1 , wherein the audio reconstruction metadata is determined for a plurality of different subbands of the multi-channel input Ambisonics signal.

12. The method of claim 1, wherein encoding the plurality of compressed channel signals comprises performing waveform encoding on each of the plurality of compressed channel signals, waveform encoding being performed on each compressed channel signal using a mono encoder.

13. The method of claim 1, wherein the audio reconstruction metadata is encoded using an entropy encoder.

14. The method according to claim 1, wherein - the multi-channel input Ambisonics signal comprises one or more object signals of one or more audio objects; and - The method comprises encoding object metadata of the one or more audio objects using an entropy encoder.

15. The method according to claim 1, wherein - the multi-channel input Ambisonics signal comprises: a sound field representation signal, an L-order Ambisonics signal, and one or more object signals of one or more audio objects, wherein the sound field representation signal is referred to as an SR signal, L≥1; and - determining the plurality of downmix channel signals by downmixing the multi-channel input Ambisonics signal to a SR signal, in particular to a K-order Ambisonics signal, wherein L≥K.

16. The method according to claim 15, wherein - determining the plurality of downmix channel signals comprises mixing the one or more object signals of the one or more audio objects to the SR signal of the multi-channel input Ambisonics signal in dependence on object metadata of the one or more audio objects; and - said object metadata of an audio object indicates a spatial position of said audio object.

17. The method according to claim 1, wherein - the method comprises determining that the multi-channel input Ambisonics signal is to be encoded using a second mode; and - in the second mode, the audio reconstruction metadata is determined based on the plurality of compressed channel signals and based on the plurality of downmix channel signals, such that the audio reconstruction metadata allows reconstruction of the plurality of downmix channel signals from the plurality of compressed channel signals.

18. The method according to claim 17, wherein - determining based on the plurality of compressed channel signals and based on the multi-channel input Ambisonics signal that the audio reconstruction metadata corresponds to a first mode; - said multi-channel input Ambisonics signal comprises a sequence of frames; and - The method comprises determining for each frame in the sequence of frames whether to use the first mode or the second mode.

19. The method according to claim 17, wherein the method comprises - generating a bitstream based on coded audio data derived by encoding the plurality of compressed channel signals and based on coded metadata derived by encoding the audio reconstruction metadata; and - inserting into the bitstream an indication indicating whether the second mode has been used.

20. An encoding device for encoding a multi-channel input Ambisonics signal, wherein the encoding device is configured to - determining a plurality of downmix channel signals from said multi-channel input Ambisonics signal; - performing energy compression on the plurality of downmix channel signals to provide a plurality of compressed channel signals, wherein the plurality of downmix channel signals include a Y channel signal, at least one of an X channel signal and a Z channel signal, and a W channel signal, wherein the plurality of compressed channel signals include at least one of a corresponding Y' channel signal, an X' channel signal and a Z' channel signal, and the W channel signal, and wherein the Y' channel signal, the X' channel signal and the Z' channel signal are determined such that the Y' channel signal has less energy than the Y channel signal, such that the X' channel signal has less energy than the X channel signal, and such that the Z' channel signal has less energy than the Z channel signal; - determining audio reconstruction metadata based on the plurality of compressed channel signals and based on the multi-channel input Ambisonics signal; wherein the audio reconstruction metadata causes a receiving device to upmix the plurality of compressed channel signals to an approximation of the multi-channel input Ambisonics signal; and - encoding the plurality of compressed channel signals and the audio reconstruction metadata.

Images