Method and device for generating bitstream comprising immersive audio signal

Abstract

To provide a method and device for generating or decoding a bitstream comprising an immersive audio signal.SOLUTION: A method for generating a bitstream including a sequence of superframes for a sequence of frames of an immersive audio signal includes the steps of: inserting, repeatedly for a sequence of superframes, coded audio data for one or more frames of one or more downmix channel signals derived from the immersive audio signal into a data field of the superframe; and inserting metadata for reconstructing the one or more frames of the immersive audio signal from the encoded audio data into a metadata field of the superframe.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 693,246, filed July 2, 2018, the contents of which are incorporated herein by reference.

[0002] Technical Field This document relates to immersive audio signals, which may include sound field representation signals, in particular Ambisonics signals. In particular, this document relates to generating and decoding bitstreams that include immersive audio signals. [Background technology]

[0003] A sound or sound field in a listening environment of a listener at a listening position may be described using an Ambisonics signal. An Ambisonics signal can be viewed as a multi-channel audio signal, where each channel corresponds to a particular directional pattern of the sound field at the listener's listening position. An Ambisonics signal may be described using a three-dimensional (3D) Cartesian coordinate system, where the origin of the coordinate system corresponds to the listening position, the x-axis points forward, the y-axis points to the left, and the z-axis points upward.

[0004] By increasing the number of audio signals or channels and the corresponding number of directivity patterns (and corresponding panning functions), the accuracy of the sound field description can be increased. As an example, a first-order Ambisonics signal contains four channels or waveforms: a W channel describing the omnidirectional component of the sound field, an X channel describing the sound field with a dipole directivity pattern corresponding to the x-axis, a Y channel describing the sound field with a dipole directivity pattern corresponding to the y-axis, and a Z channel describing the sound field with a dipole directivity pattern corresponding to the z-axis. A second-order Ambisonics signal has nine channels, including the four channels of the first-order Ambisonics signal (also called the B format) and five additional channels for different directivity patterns. In general, an L-order Ambisonics signal is the L-order of an (L-1)-order Ambisonics signal. 2channels and [(L+1) 2 -L 2 ] additional channels (L+1) 2 channels (when using the 3D Ambisonics format). An L-th order Ambisonics signal, for L>1, is sometimes called a Higher Order Ambisonics (HOA) signal.

[0005] The HOA signal may be used to describe a 3D sound field independent of the arrangement of speakers used to render the HOA signal. Examples of speaker arrangements include headphones, or one or more arrangements of loudspeakers, or a virtual reality rendering environment. Thus, it may be beneficial to provide the HOA signal to an audio renderer to allow the audio rendering to flexibly adapt to different arrangements of speakers. Summary of the Invention [Problem to be solved by the invention]

[0006] A soundfield representation (SR) signal, such as an Ambisonics signal, may be complemented with audio objects and / or multi-channel signals to provide an immersive audio (IA) signal. This document addresses the technical problem of transmitting and / or storing IA signals with high perceptual quality in a bandwidth-efficient manner. In particular, this document addresses the technical problem of providing an efficient bitstream representing the IA signal. Such technical problem is solved by the independent claims. Preferred examples are given in the dependent claims. [Means for solving the problem]

[0007] According to an aspect, a method for generating a bitstream is described, the bitstream comprising a sequence of superframes for a sequence of frames of an immersive audio signal. The method comprises iterating over said sequence of superframes and inserting encoded audio data for one or more frames of one or more downmix channel signals derived from the immersive audio signal into a data field of the superframe. Furthermore, the method comprises inserting metadata for reconstructing one or more frames of the immersive audio signal from the encoded audio data, in particular encoded metadata, into a metadata field of the superframe.

[0008] According to another aspect, a method for deriving data relating to an immersive audio signal from a bitstream is described. The bitstream includes a superframe sequence for a sequence of frames of the immersive audio signal. The method includes iterating over the sequence of superframes and extracting from a data field of the superframe encoded audio data for one or more frames of one or more downmix channel signals derived from the immersive audio signal. Furthermore, the method includes extracting from a metadata field of the superframe metadata for reconstructing one or more frames of the immersive audio signal from the encoded audio data.

[0009] According to a further aspect, a software program is described which may be adapted for execution on a processor and, when executed on the processor, to perform the method steps outlined herein.

[0010] According to another aspect, a storage medium is described that may include a software program adapted for execution on a processor and, when executed on the processor, to perform the method steps outlined herein.

[0011] According to a further aspect, a computer program product is described that may include executable instructions for performing the method steps outlined herein when executed on a computer.

[0012] According to a further aspect, a superframe of a bitstream is described. The bitstream comprises a sequence of superframes for a sequence of frames of an immersive audio signal. The superframe comprises a data field for encoded audio data for one or more (in particular multiple) frames of one or more downmix channel signals derived from the immersive audio signal. Furthermore, the superframe comprises a (single) metadata field for metadata adapted to reconstruct one or more (in particular multiple) frames of the immersive audio signal from the encoded audio data.

[0013] According to another aspect, an encoding device is described that is configured to generate a bitstream, the bitstream comprising a sequence of superframes for a sequence of frames of an immersive audio signal, the encoding device being configured to iterate over said sequence of superframes, to insert encoded audio data for one or more (in particular multiple) frames of one or more downmix channel signals derived from the immersive audio signal into a data field of the superframe, and to insert metadata for reconstructing one or more (in particular multiple) frames of the immersive audio signal from the encoded audio data into a metadata field of the superframe.

[0014] According to a further aspect, a decoding device configured to derive data relating to an immersive audio signal from a bitstream, the bitstream including a sequence of superframes for a sequence of frames of the immersive audio signal, the decoding device being configured to iterate over the sequence of superframes to extract from a data field of the superframes encoded audio data for one or more (in particular multiple) frames of one or more downmix channel signals derived from the immersive audio signal; and to extract from a metadata field of the superframes metadata for reconstructing the one or more (in particular multiple) frames of the immersive audio signal from the encoded audio data.

[0015] It should be noted that the methods, devices and systems outlined in this patent application, including preferred embodiments thereof, may be used independently or in combination with other methods, devices and systems disclosed herein. Furthermore, all aspects of the methods, devices and systems outlined in this patent application may be combined in any manner. In particular, the features of the claims may be combined with each other in any manner. [Brief description of the drawings]

[0016] The invention is described below, by way of example, with reference to the accompanying drawings, in which: [Figure 1] 1 shows an example of an encoding system. [Diagram 2] 3 illustrates an exemplary encoding unit for encoding an immersive audio signal. [Diagram 3] 2 shows another exemplary decoding unit for decoding an immersive audio signal; [Figure 4] 1 illustrates an exemplary superframe structure for an immersive audio signal, and in particular for encoded data indicative of an immersive audio signal. [Diagram 5]1 shows a flowchart of an exemplary method for generating a bitstream including a sequence of superframes indicative of an immersive audio signal. [Figure 6] 1 shows a flowchart of an exemplary method for extracting information from a bitstream including a sequence of superframes indicative of an immersive signal. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] As outlined above, this document relates to efficient encoding of immersive audio signals, such as HOA signals, multi-channel and / or object audio signals, where the HOA signals in particular are referred to herein more generally as soundfield representation (SR) signals. Additionally, this document relates to the storage or transmission of immersive audio (IA) signals over a transmission network in a bitstream.

[0018] As outlined in the introduction, SR signals may contain a relatively large number of channels or waveforms, with different channels relating to different panning functions and / or different directional patterns. As an example, a 3D First Order Ambisonics (FOA) or HOA signal of order L may have a total of (L+1) 2 A First Order Ambisonics (FOA) signal is an Ambisonics signal of order L=1 that contains four channels. SR signals can be represented in a variety of different formats.

[0019] The sound field can be considered as consisting of one or more sound events emanating from any direction around the listening position, and as a result, the position of said one or more sound events may be defined on the surface of a sphere (with the listening position or reference position being at the center of the sphere).

[0020] Sound field formats like FOA or Higher Order Ambisonics (HOA) are defined in a way that allows the sound field to be rendered with any speaker arrangement (i.e. any rendering system). However, rendering systems (such as Dolby Atmos systems) are typically constrained in the sense that the possible heights of the speakers are fixed to a defined number of planes (e.g. a (horizontal) plane at ear height, a ceiling or upper plane, and / or a floor or lower plane). Thus, the concept of an ideal spherical sound field can be modified to a sound field composed of sound objects located in different rings (similar to the stacked rings that make up a beehive) at various heights on the surface of a sphere.

[0021] As shown in Fig. 1, the audio coding system 100 comprises an encoding unit 110 and a decoding unit 120. The encoding unit 110 may be configured to generate a bitstream 101 for transmission to the decoding unit 120 based on an input signal 111, which may include or may be an immersive audio signal (e.g., used for a virtual reality (VR) application). The immersive audio signal 111 may include an SR signal, a multi-channel signal, and / or multiple objects (each object including an object signal and object metadata). The decoding unit 120 may be configured to provide an output signal 121 based on the bitstream 101, which may include or may be a reconstructed immersive audio signal.

[0022] FIG. 2 shows an example of an encoding unit 110, 200. The encoding unit 200 may be configured to encode an input signal 111, which may be an immersive audio (IA) signal 111. The IA signal 111 may comprise a multi-channel input signal 201. The multi-channel input signal 201 may comprise an SR signal and one or more object signals. Furthermore, object metadata 202 for said plurality of object signals may be provided as part of the IA signal 111. The IA signal 111 may be provided by a content ingestion engine, which may be configured to derive the objects and / or the SR signal from (composite) IA content, such as VR content, which may comprise an SR signal, one or more multi-channel signals, and / or one or more objects.

[0023] The encoding unit 200 comprises a downmix module 210 configured to downmix a multi-channel input signal 201 into a plurality of downmix channel signals 203. Said plurality of downmix channel signals 203 may correspond to SR signals, in particular First Order Ambisonics (FOA) signals. The downmix may be performed in the subband domain or the QMF domain (e.g., using 10 or more subbands). The encoding unit 200 further comprises a joint encoding module 230 (in particular, a SPAR module) configured to determine joint encoding metadata 205 (in particular, SPAR (Spatial Audio Resolution Reconstruction) metadata) configured to reconstruct the multi-channel input signal 201 from the multiple downmix channel signals 203. The joint encoding module 230 may be configured to determine the joint encoding metadata 205 in the subband domain. In one example, a spatial audio reconstruction (SPAR) tool is a coding tool for improved coding of a relatively large number of audio channels and objects. To gain coding efficiency, the tool supports reconstructing audio channels and objects from a smaller number of joint input audio channels and low-overhead side information.

[0024] To determine the joint coding or SPAR metadata 205, the multiple downmix channel signals 203 may be transformed into and / or processed in the subband domain. Furthermore, the multi-channel input signal 201 may be transformed into the subband domain. The joint coding or SPAR metadata 205 may then be determined for each subband, in particular by upmixing the subband signals 203 of the multiple downmix channel signals 203 using the joint coding or SPAR metadata 205 to obtain an approximation of the subband signals of the multi-channel input signal 201. The joint coding or SPAR metadata 205 for the various subbands may be inserted into the bitstream 101 for transmission to the corresponding decoding unit 120.

[0025] Further, the encoding unit 200 may include an encoding module 240 configured to perform waveform encoding of the multiple downmix channel signals 203, thereby providing encoded audio data 206. Each of the downmix channel signals 203 may be encoded using a mono waveform encoder (e.g., 3GPP EVS encoding), thereby enabling efficient encoding. Further examples of encoding the multiple downmix channel signals 203 are MPEG AAC, MPEG HE-AAC and other MPEG audio codecs, 3GPP codecs, Dolby Digital / Dolby Digital Plus (AC-3, eAC-3), Opus, LC-3 and other similar codecs. As a further example, encoding tools included in the AC-4 codec may be configured to perform the operations of the encoding unit 200.

[0026] Further, the encoding module 240 may be configured to perform entropy encoding of the jointly encoded metadata (i.e., SPAR metadata) 205 and the object metadata 202, thereby providing encoded metadata 207. The encoded audio data 206 and the encoded metadata 207 may be inserted into a bitstream 101. The bitstream 101 may exhibit a superframe structure as described herein. The method 500 described herein may be performed by the encoding module 240.

[0027] FIG. 3 illustrates an example of a decoding unit 120, 350. The decoding unit 120, 350 may include a receiver that receives a bitstream 101 that may include the encoded audio data 206 and the encoded metadata 207. The decoding unit 120, 350 may include a processor and / or a demultiplexer that demultiplexes the encoded audio data 206 and the encoded metadata 207 from the bitstream 101. The decoding unit 350 has a decoding module 360 ​​configured to derive a plurality of reconstructed channel signals 314 from the encoded audio data 206. The decoding module 360 ​​may be further configured to derive joint coding or SPAR metadata 205 and / or object metadata 202 from the encoded metadata 207. The method 600 described herein may be performed by the decoding module 360. Furthermore, the decoding unit 350 comprises a reconstruction module 370 configured to derive a reconstructed multi-channel signal 311 from the joint coding or SPAR metadata 205 and from the plurality of reconstructed channel signals 314. The joint coding or SPAR metadata 205 may convey time- and / or frequency-varying elements of an upmix matrix allowing to reconstruct the multi-channel signal 311 from the plurality of reconstructed channel signals 314. The upmix process may be performed in the QMF (Quadrature Mirror Filter) subband domain. Alternatively, another time / frequency transform may be used to perform the upmix process, in particular a transform based on the FFT (Fast Fourier Transform). In general, a transform allowing a frequency-selective analysis and (upmix) processing may be applied. The upmix process may also include a decorrelator allowing an improved reconstruction of the covariance of the reconstructed multi-channel signal 311, the decorrelator being controlled by the additional joint coding or SPAR metadata 205.

[0028] The reconstructed multi-channel signal 311 may include a reconstructed SR signal and one or more reconstructed object signals. The reconstructed multi-channel signal 311 and the object metadata may form the output signal 121 (also known as a reconstructed IA signal 121). The reconstructed IA signal 121 may be used for speaker rendering 331, headphone rendering 332, and / or rendering 333 of VR content, for example, that relies on the SR representation.

[0029] Therefore, the IA input signal 111 a downmix signal comprising a plurality of downmix channel signals 203; where the downmix signal 203 may be a sound field representation (SR) signal; and Metadata 202, 205 including SPAR or joint coding metadata 205 and / or object metadata 202 for one or more objects An encoding unit 110, 200 configured to encode the image in the image is described.

[0030] The metadata 202, 205, in particular the SPAR metadata 205, may indicate a different time resolution than the downmix signal. In particular, the metadata 202, 205 may be used for multiple frames (e.g., two frames) of the downmix signal. In view of this, a superframe may be defined for the bitstream 101. A superframe comprises multiple frames of the downmix signal and metadata 202, 205 for the multiple frames of the SR downmix signal.

[0031] 4 illustrates an exemplary superframe 400. The superframe 400 may include a base header (BH) field 401 and / or a configuration information (CI) field 402 that may include data valid for the entire superframe 400. Furthermore, the superframe 400 includes signal data fields 411, 412, 421, 422 for the encoded audio data 206 for one or more (particularly a plurality of) frames of a downmix signal. In particular, for each downmix channel signal 203, one or more (particularly a plurality of) signal data fields 411, 412, 421, 422 may be provided, e.g. signal data fields 411, 421 for two frames of the first downmix channel signal 203 and signal data fields 412, 422 for two frames of the Nth downmix channel signal 203. The signal data fields 411, 412, 421, 422 are also referred to herein as EVS bit fields (eg, an EVS encoder is used to encode the downmix channel signal 203).

[0032] Further, the superframe 400 includes a metadata field (MDF) 403. The metadata field 403 may be configured to provide SPAR or joint coding metadata 205 and / or prediction coefficients (PC). Thus, the metadata field 403 may be a SPAR bit field or a PC bit field (depending on the coding mode being used). Additionally, the superframe 400 may include a frame extender (FE) field 404.

[0033] Thus, the superframe 400 may include signaling elements configured as follows: indicating the one or more (EVS) codec modes used for the (EVS) encoding of the N downmix channel signals; the default may be N=4 channels, i.e. there are 4 (EVS) codec downmix channel signals W, X', Y', Z'; Indicates the selected operation mode of the metadata-assisted (EVS) codec; · Metadata · Show bitrate; · Provides the possibility to signal potential future expansions.

[0034] One or more signaling elements (e.g., CI field 402) may only be provided conditionally in-band within the superframe 400. If optional or conditional signaling elements are provided, the signaling elements may be dynamically adapted and / or included within the superframe 400. One or more signaling elements may be kept static and / or provided only once, for example, as an out-of-band message. One or more signaling elements may be semi-dynamic, in which case the one or more signaling elements are provided in-band only in selected superframes 400.

[0035] The superframe 400 may be designed to enable one or more of the following features: Full decoding and rendering of metadata-assisted EVS encoded superframes. Metadata assisted partial mono decoding of EVS encoded superframes. Low-complexity extraction of superframe size information from a sequence of concatenated superframes without the need to decode the superframes, for example to put the superframe size information into a secondary format (e.g., ISOBMFF, ISO Base Media File Format) that provides or requires this superframe size information. Low computational complexity bitrate determination without the need to decode superframe data. Low-complexity feed-forward and skip of superframes without the need to decode the superframe data. Low computational complexity feedback without the need to decode superframe data (especially in the case of constant bitrate operation). Simple resynchronization and superframe skipping in case of bit errors in the arithmetic and / or entropy coded EVS and / or metadata bitstream parts. Editable superframe, which allows replacing metadata or EVS data frames.

[0036] The encoded bit superframe 400 of a metadata-assisted EVS codec may correspond to an encoding stride of 40 ms (e.g., containing two frames of 20 ms). It consists of the following basic bit fields: Base Header field (BH) 401: This field includes a Configuration field Presence Indicator (CPI), a MetaData field size Adjustment indicator (MDA), and an Extension Indicator (EI). The CPI may indicate whether a Configuration Information (CI) field is provided in the current superframe 400. The MDA may signal the difference between the signaled maximum metadata frame size and the actual metadata frame size. The EI may signal whether the superframe 400 is extended by a Frame Extender (FE) 404. Configuration Information Field (CI) 402: This field may carry signaling information regarding the configuration of the EVS, SPAR, and Predictive Coefficient coding tools used, such as frame type (coding mode), bit rate, and other configuration parameters as described in this document. EVS bit fields 411, 421, 412, 422: Each field may carry the bits of a single EVS frame (in particular, without the EVS payload header) as specified in 3GPP TS26.445: "Codec for Enhanced Voice Services (EVS); Detailed algorithmic description", Section 7, which is incorporated herein by reference. · SPAR Bits Field (SPAR) 403: This field may carry the bits of a single SPAR metadata frame, possibly zero padded at the end to make it byte aligned. · Predictive Coefficient Bits Field (PC) 403: This field may carry the bits of a single predictive coefficient metadata frame, possibly zero padded at the end to make it byte aligned. Frame Extender (FE) 404: This field may be defined for future use and may carry extension data. Except for the size element contained in the FE, other data carried by the FE may be reserved for future use (RFU).

[0037] All basic bit fields are byte-aligned and may finally be zero-padded at the end to their defined size if necessary.

[0038] The above basic fields may be included in a (single) superframe 400 in the following sequential order: A base header (BH) 401, - Configuration Field Presence Indicator (CPI), -Metadata Field Sizing Indicator (MDA) and -Extended Indicator (EI) Including BH. One optional configuration information field (CI) 402. The presence of the CI field 402 may be signaled by the CPI. N EVS encoded downmix channel signals S1,…,S N of data. There are two successive frames for each downmix channel signal, which can be carried by 2×N basic EVS bit fields 411, 421, 412, 422 (referred to herein as EVS(.)). In default operation with four downmix channel signals, there are eight successive EVS bit fields 411, 421, 412, 422, representing two frames of downmix channel signals W, X', Y', Z'. One Metadata Frame (MDF) field 403 for SPAR or prediction coefficients. This means that -One basic SPAR bit field, or - A basic PC bit field. · One optional Frame Extender (FE) 404. The presence of a FE field may be indicated by EI.

[0039] Table 1 shows an exemplary structure of a superframe 400. [Table 1]

[0040] In the default situation, there are four EVS encoded downmix channel signals. The superframe structure for the default situation is shown in Table 2. [Table 2]

[0041] Further details regarding the various basic bit fields are provided below.

[0042] The Base Header (BH) field 401 may carry a Configuration field Presence Indicator (CPI), a MetaData field size Adjustment indicator (MDA), and an Extension Indicator (EI). This byte field may always be the first element of the superframe 400.

[0043] The structure of the BH field 401 is shown in Table 3. [Table 3]

[0044] The Configuration Field Present Indicator (CPI) may be a single bit used to signal the presence of a Configuration Information (CI) field in the current superframe 400. The CPI may have the following meanings: CPI='0': This indicates that no configuration information field is provided in the current superframe 400. It should be noted that the configuration information may instead be provided as static out-of-band information or from the most recent previously received superframe 400 that carries a configuration information field 402. CPI='1': indicates that a configuration information field is provided in the current superframe 400. The configuration information provided in the CI field 402 is valid for this superframe 400 and future superframes 400 until the next superframe 400 carrying a configuration information field 402 is provided.

[0045] A metadata field size adjustment indicator (MDA) may be provided immediately following the CPI bits. This 6-bit indicator may signal the difference between the length of the MDF 403 signaled by the MDR element (defined below) and the actual size of the MDF 403. The indicated difference may be derived from a lookup shown in Table 4 using the MDA as an index. The set of adjustment values ​​in Table 4 are specified in Matlab style: start-value:step-size:end-value. The non-constant adjustment parameter step size shown in Table 4 may be designed according to an approximate model of the distribution of the metadata total entropy code length. This allows to minimize the number of unused bits in the MDF 403 and thus minimize the transmission overhead. [Table 4]

[0046] Depending on the maximum MDF size, the adjustment value represents 1 or 2 byte units. For maximum MDF sizes up to 275 bytes, the adjustment value represents 1 byte units, otherwise it represents 2 byte units.

[0047] The MDA indicator may be followed by a single Extension Indicator bit (EI), which, if set to 1, indicates that a Frame Extender (FE) element is added after the current superframe 400.

[0048] The optionally provided configuration information (CI) field 402 may carry the following signaling elements as shown in Table 5. The CI field 402 may consist of or contain 8 bytes of data (for two EVS frames per downmix channel signal and N=4 downmix channels). [Table 5]

[0049] Table 6 shows the optional configuration information field 402 for the default case with four EVS encoded downmix channel signals, in which the CI field consists of 9 bytes of data. [Table 6]

[0050] The indicator for the number N of EVS coded downmix channel signals (NI) may be a 3-bit element encoding the number N of EVS coded downmix channel signals. N is obtained from the indicator NI by incrementing the number represented by the 3-bit element by 1. To achieve the default operation with four EVS downmix channel signals, the NI element may be set to 3 ('011').

[0051] The Metadata Type indication (MDT) bit can have the following meanings: · MDT = '0': indicates that the MDF carries the PC bit field. · MDT = '1': indicates that the MDF carries the SPAR bit field.

[0052] The MetaData Coding configuration field (MDC) may contain configuration information of either the prediction coefficient tool or the SPAR coding tool used, depending on the indication of the MDT bits. The MDC field may be an 11-bit element of the CI field 402. The meaning of the bits may depend on the MDT bits of the CI field 402. Depending on the value of the MDT bits, the MDC bits may have the following meanings: MDT='0': When the MDT bit is 0, the three MSBs of the MDC encode the configuration parameters of the prediction coefficient coding scheme. The remaining 8 bits of the MDC are unused and are zero-padded. The structure and contents of the MDC field in this case are shown in Table 7a. · MDT='1': If the MDT bit is 1, then the 11 MDC bits encode the SPAR codec configuration shown in Table 7 b. The HOA order may be calculated by incrementing hoa_order_idx by one. [Table 7]

[0053] The MetaData Bit rate signaling field (MDR) may contain 5 bits and may be used to encode the maximum size of the MDF. The maximum MDF size may be obtained by a table lookup using Table 8, where the MDR value is an index into Table 8. Table 8 further indicates the (maximum) metadata bit rate in kbps. In Table 8, the actual MDF size is signaled as the maximum MDF size minus the adjustment number / value indicated by the MDA (from the BH field 401). This allows signaling of the actual MDF size with fine resolution (typically byte resolution). It should also be noted that unused bits in the MDF may be zero padded, which may occur if the actual MDF size provides more space than is needed for the encoded metadata. [Table 8]

[0054] The Band Number field (BND) may be a 3-bit number and may indicate the number of sub-bands used in the metadata encoding. The band number is derived from the BND value by a lookup in Table 9. In default operation, the BND field may be set to 5 ('101'), indicating 12 sub-bands. [Table 9]

[0055] The Reserved bit (RES) may be reserved for future use. The default behavior is that this bit is set to '0' and may be ignored by the receiver.

[0056] The EVS FT field (FT-x,y) may represent the EVS frame type (FT) applied for the encoding of the y-th frame of the x-th downmix channel signal, where x=1…N and y=1,2. EVS frame types may be defined in 3GPP TS 26.445, section A2.2.1.2, which is incorporated herein by reference. It should be noted that the last EVS FT field in the CI field 402 may be followed by up to seven zero padding bits to ensure octet alignment. If the last EVS FT field ends octet aligned, no zero padding bits are added after it. The zero padding bits are ignored by the receiver.

[0057] The basic EVS bit fields 411, 421, 412, 422 may be defined as specified in 3GPP TS 26.445, Section 7, which is incorporated herein by reference, for the EVS coding mode used, respectively. As specified in the cited documents, no additional signaling bits are defined as part of the basic EVS frame fields to indicate the bit rate or EVS operating mode. This information may be part of the optional CI field 402 of the current or previous superframe 400, or may be provided out-of-band.

[0058] The detailed allocation of coefficients for SPAR metadata is shown in Table 10. Table 10 shows the order of bits inserted into the frame. Note that the most significant bit (MSB) of each parameter is always inserted first. Since each field is dynamically quantized, the bit allocation is variable. [Table 10]

[0059] The detailed allocation of coefficients of the PC metadata is shown in Table 11. Table 11 shows the order of bits inserted in the superframe 400. Note that the most significant bit (MSB) of each parameter is always inserted first. Since each field is dynamically quantized, the bit allocation is variable. [Table 11]

[0060] The Frame Extender (FE) 404 typically carries in the first two bytes a 16-bit unsigned integer indicating the size of the FE field 404 in bytes. This element is referred to as the FE size. Thus, the FE size number is 2 or greater. The content and meaning of the remaining FE data portion of the FE field 404 may be reserved for future use. In default behavior, the FE size element may be parsed and the FE data element may be skipped and ignored. The structure and content of the FE field 404 is shown in Table 12. [Table 12]

[0061] Thus, a superframe structure is described that enables signaling of configuration information for a metadata-enabled EVS codec, the superframe structure enabling a receiver to decode metadata-enabled EVS codec data.

[0062] At a general level, metadata-assisted EVS codecs are multi-mode and / or multi-rate coding systems. The underlying EVS codec may be configured to operate with a number of different coding modes and / or bitrates. Furthermore, spatial metadata codecs may offer a variety of different coding modes and / or bitrates. Spatial metadata codecs use entropy coding, which typically results in a non-constant bitrate. This means that the actual bitrate used is typically lower than a given target bitrate. For some frames this bitrate undershoot may be smaller and for some other frames it may be larger.

[0063] The exact coding mode and bitrate used by the encoder 110 may be provided so that the decoder 120 can properly decode the transmitted bitstream 101. For the entropy coded parts of the metadata, the exact bitrate used may not be required since the Huffman codes used are commaless and uniquely decodable. Nevertheless, the receiver of the bitstream 101 may be provided with the number of bits used for the coding of a frame (or superframe 400). This may be desirable, for example, if the decoder 120 needs to skip some received frames without needing to decode these frames. In this document we have described a superframe structure that supports the following features: Full frame decoding. - Decode only what is needed for mono playback. · Extraction of length information from frames 400 into a secondary format (ISOBMFF) that provides and / or requires this information. In concatenating frames 400, decode only partway through, effectively skipping the first few frames. Finding the start of the next frame if there is a bit error (resynchronization). · Quickly and efficiently determine bitrate without the need to decode frames. Editing frames (replacing metadata or parts of EVS frames). Fast feed-forward operation without frame decoding. Support for efficient transport of constant and variable length codec data payloads.

[0064] Furthermore, a superframe structure is described which contains all the necessary signalling elements for: Indicates the EVS codec mode (including bitrate) used for EVS encoding of the N downmix channel signals. The default may be N=4, which means there are 4 EVS codec downmix channels W, X', Y', Z'. Supports metadata indicating the selected operating mode of the codec. · Indicate metadata bitrates in high resolution without significant signaling overhead. · Provides the possibility to signal potential future expansions.

[0065] Some signaling elements of the superframe 400 may not change often during an encoding session, or may even be static. Some other signaling elements, such as the metadata bitrate, may change from superframe to superframe. For that reason, certain signaling elements are only provided conditionally in-band within the superframe 400 (such as the CI field 402). If they are provided, these signaling elements can be dynamically adapted from superframe to superframe. There is also the possibility to keep these signaling elements static and provide them only once, for example as an out-of-band message. Signaling elements may also be semi-dynamic, in which case they are only provided in-band in certain superframes.

[0066] With regard to signaling metadata bitrates, the main challenge is that the number of bits (or bytes) required per superframe 400 can vary within a relatively large range. Signaling only the maximum possible number of bits per frame can result in a relatively large number of bits remaining unused if the entropy code is much shorter than the maximum length. On the other hand, providing a direct signaling element to indicate the number of bits (or bytes) actually used within the superframe 400 would require a relatively large number of signaling bits. In this paper, a scheme is described that covers a relatively large range of possible metadata bitrates while keeping the number of signaling bits for the number of bits (or bytes) actually used within the superframe 400 to a minimum.

[0067] From a system perspective, the superframe 400 of the metadata-assisted EVS codec is generated at the encoding headend. This may be a server in the network with access to unencoded immersive or VR (Virtual Reality) audio data. It may also be a mobile phone capturing the immersive audio signal. The encoded frame 400 may be downloaded to a receiving terminal or inserted into a file transmitted according to a streaming protocol such as DASH (Dynamic Adaptive Streaming over HTTP) or RTSP / RTP (Real-Time Streaming Protocol / Real-time Transport Protocol). If the encoded superframe 400 is stored in a file, the superframe 400 may be inserted into a file formatted according to ISOBMFF. If certain configuration information is static and not transmitted as part of the superframe 400, it may instead be provided from the encoding end to the decoding end by out-of-band means such as the session description protocol (SDP).

[0068] The scheme outlined in this paper may use the EVS codec as the underlying codec and may provide multi-mode / multi-rate messages (frame types) in-band in the superframe 400 or out-of-band, for example using SDP. This may be combined with a multi-mode immersive metadata encoding framework, which can be efficiently configured using a set of configuration parameters that can also be transmitted in-band or out-of-band. Furthermore, the multi-mode immersive metadata encoding may be combined with a scheme that allows for an associated maximum bitrate (or number of bits in a frame / superframe) in-band or out-of-band.

[0069] The superframe structure described herein signals the actual used metadata field size as a maximum number (which is optionally signaled out-of-band) minus an adjustment parameter, for which an indication is transmitted as part of each superframe 400. The encoding of the adjustment parameter is preferably performed with a non-constant step size, which allows covering an increased range of possible adjustments with a reduced number of signaling bits for the adjustment parameter. Furthermore, the non-constant adjustment parameter step size may be designed using an approximate model of the distribution of the total entropy code length of the metadata. This allows minimizing the number of unused bits in the metadata field, thus minimizing the transmission overhead. Furthermore, the overhead for the metadata bitrate (size) may be signaled while minimizing the number of unused bits in the metadata field. Thus, the overall transmission bitrate is reduced.

[0070] The configuration information (CI) in the CI field 402 may relate to a selected EVS frame type for EVS encoding of the four downmix channel signals W, X', Y', Z'. The configuration information may further relate to (i) a selected operation mode of metadata-assisted EVS code, FOA or HIQ, (ii) a bit rate of SPAR metadata in case of HIQ operation, and (iii) a bit rate of prediction coefficients metadata in case of FOA operation. The configuration information indication can be (1) dynamic and provided in-band with the payload; (2) semi-dynamic and provided in-band with the previous payload; or (3) static and provided out-of-band as a hex string together with the codec attribute of the DASH adaptation set.

[0071] The FOA (First Order Ambisonics) mode is a low bitrate operating mode (e.g., operating at about 128 kbps) that relies on prediction coefficient metadata. FOA typically exhibits relatively limited quality due to its relatively low spatial resolution. The HIQ (High Immersive Quality) mode is a medium to high rate operating mode (e.g., operating at 128-512 kbps). It relies on SPAR metadata and can provide very high immersive quality with the goal of reconstructing the original SR signal.

[0072] Fig. 5 shows a method 500 for generating a bitstream 101, the bitstream 101 comprising a sequence of superframes 400 for a sequence of (basic) frames of an immersive audio signal 111. The immersive audio (IA) signal 111 may comprise a soundfield representation (SR) signal that may describe a sound field at a reference position. The reference position may be a listening position of a listener and / or a capture position of a microphone. The SR signal may comprise multiple channels (or waveforms) for multiple different arrival directions of the sound field at the reference position. Alternatively or additionally, the IA signal 111 may comprise one or more audio objects and / or multi-channel signals.

[0073] The IA signal 111, in particular the SR signal contained within the IA signal, may include or be an L-order Ambisonics signal, where L is equal to or greater than 1. Alternatively or additionally, the SR signal may exhibit a beehive (BH) format, with multiple directions of arrival arranged in multiple different rings on a sphere around a reference position. The multiple rings may include a central ring, an upper ring, a lower ring, and / or a zenith. Alternatively or additionally, the SR signal may exhibit an intermediate spatial format called ISF, in particular the ISF format defined within the Dolby Atmos technology.

[0074] Thus, IA signal 111 may include multiple different channels. Each channel included in IA signal 111 typically includes a sequence of audio samples for a sequence of time points or for a sequence of frames. In other words, a "signal" as described herein typically includes a sequence of audio samples for a corresponding sequence of time points or frames (e.g., a temporal distance of 20 ms or less).

[0075] The method 500 may include extracting one or more audio objects from the IA signal 111. An audio object typically includes an object signal (with a sequence of audio samples for a corresponding time instant or sequence of frames). Additionally, an audio object typically includes object metadata 202 that indicates a location of the audio object. The location of an audio object may change over time, such that the object metadata 202 for an audio object may indicate a sequence of locations for a sequence of time instants or frames.

[0076] Further, the method 500 may include determining a residual signal based on the IA signal 111 and based on the one or more audio objects. The residual signal may describe the original IA signal from which the one or more audio objects 103, 303 have been extracted and / or removed. The residual signal may be an SR signal included within the IA signal 111. Alternatively or additionally, the residual signal may include or may be a multi-channel audio signal and / or a bed of audio signals. Alternatively or additionally, the residual signal may include multiple audio objects at fixed object locations and / or positions (e.g. audio objects assigned to specific speakers of a defined speaker arrangement).

[0077] Further, the method 500 may include generating and / or providing a downmix signal based on the IA signal 111 (e.g., using the downmix module 210). The number of channels of the downmix signal is typically smaller than the number of channels of the IA signal 111. Furthermore, the method 500 may include determining joint coding or SPAR metadata 205 enabling upmixing the downmix signal (i.e., the one or more downmix channel signals 203) into a signal for one or more reconstructed audio objects for the corresponding one or more audio objects. Furthermore, the joint coding or SPAR metadata 205 may enable upmixing the downmix signal into a reconstructed residual signal for the corresponding residual signal.

[0078] A downmix signal comprising one or more downmix channel signals 203, SPAR metadata 205 and object metadata 202 may be inserted into the bitstream 101. In particular, the method 500 may include performing waveform coding of the downmix signal to provide encoded audio data 206 for a sequence of frames of the one or more downmix channel signals 203. The waveform coding may be performed, for example, using Enhanced Voice Services (EVS) coding. Furthermore, the method 500 may include performing entropy coding of the SPAR metadata 205 and / or the object metadata 202 of one or more audio objects to provide the (encoded) metadata 207 to be inserted into the bitstream 101.

[0079] The method 500 may include repeating for a sequence of superframes 400 and inserting (501) encoded audio data 206 for one or more (in particular a plurality of) frames (e.g., two or more frames) of one or more downmix channel signals 203 derived from the immersive audio signal 111 into data fields 411, 421, 412, 422 of the superframes 400. A (basic) frame of the downmix channel signal 203 may span 20 ms of the downmix channel signal 203. The superframe 400 may span a multiple of the length of the (basic) frame, e.g., 40 ms.

[0080] Furthermore, the method 500 may include inserting (502) metadata 202, 205 (particularly the encoded metadata 207) for reconstructing one or more (particularly multiple) frames of the immersive audio signal 111 from the encoded audio data 206 into a (single) metadata field 403 of the super-frame 400. Thus, the super-frame 400 may provide metadata 202, 205 for one or more (particularly multiple) frames of the one or more downmix channel signals 203, thereby enabling an efficient transmission of the IA signal 111.

[0081] In particular, the frames of the downmix channel signal 203 may be generated using a multi-mode and / or multi-rate speech or audio codec. Furthermore, the metadata 202, 205 may be generated using a multi-mode and / or multi-rate immersive metadata encoding scheme. Configuration information indicating the operation of the multi-mode and / or multi-rate speech or audio codec (used for the downmix channel signal 203) and / or the operation of the multi-mode and / or multi-rate immersive metadata encoding scheme may be included in the configuration information field 402 of the (current) super-frame 400, in the configuration information field 402 of a previous super-frame 400 in the sequence of super-frames 400, or may be conveyed using an out-of-band signaling scheme. As a result of this, an efficient and flexible scheme for encoding the immersive audio signal 111 may be provided.

[0082] The super-frame 400 may include encoded audio data 206 associated with multiple downmix channel signals 203. The encoded audio data 206 for the frames of the first downmix channel signal 203 may be generated using a first instance of a multi-mode and / or multi-rate speech or audio codec. Furthermore, the encoded audio data 206 for the frames of the second downmix channel signal 203 may be generated using a second instance of a multi-mode and / or multi-rate speech or audio codec, the first and second instances of the multi-mode and / or multi-rate speech or audio codec being different. Configuration information (included in the current super-frame 400, in the previous super-frame 400, or conveyed out-of-band) may indicate the operation of the first and second instances of the multi-mode and / or multi-rate speech or audio codec (in particular each instance). This may further increase flexibility and efficiency for encoding the immersive audio signal 111.

[0083] In other words, the method 500 may include inserting encoded audio data 206 into one or more first data fields 411, 421 and one or more second data fields 412, 422 of the superframe 400 for one or more frames of the first downmix channel signal 203 and the second downmix channel signal 203 derived from the immersive audio signal 111, respectively. The first downmix channel signal 203 may be encoded using a first (audio or speech) encoder and the second downmix channel signal may be encoded using a second (audio or speech) encoder. The first and second encoders may be different or may be operated using different configurations. Furthermore, the method 500 may include providing configuration information regarding the first and second encoders within the superframe 400, within a previous superframe 400 of the sequence of superframes 400, or using an out-of-band signaling scheme. This may provide greater flexibility and efficiency for encoding the immersive audio signal 111.

[0084] The method 500 may include inserting a header field 401 into the superframe 400. The header field 401 may indicate a size of the metadata field 403 of the superframe 400, such that the size of the superframe 400 can be flexibly adapted to different lengths of the (entropy coded and / or losslessly coded) metadata 207.

[0085] The metadata field 403 may indicate a maximum possible size (which may for example be indicated in the optional configuration information field 402 of the superframe 400). The header field 401 may indicate an adjustment value, and the size of the metadata field 403 of the superframe 400 may correspond to the maximum possible size minus the adjustment value, thereby allowing the size of the metadata field 403 to be signaled in an accurate and efficient manner. The header field 401 may include a size indicator (e.g. said adjustment value) for the size of the metadata field 403. The size indicator may indicate different resolutions or step sizes (in terms of size intervals) for different size ranges of the size of the metadata field 403. The resolution and / or step size of the size indicator may depend on the statistical size distribution of the (entropy coded) metadata. By providing a size indicator with variable resolution, the bitrate efficiency for signaling the size of the metadata field 403 may be improved.

[0086] The header field 401 may indicate whether the superframe 400 includes a configuration information field 402. In other words, the header field 401 may indicate the presence of the configuration information field 402. The configuration information field 402 may be inserted into the superframe 400 only if necessary (e.g., if the configuration of the encoder of the IA signal 111 is changed). As a result, the bitrate efficiency of a sequence of superframes 400 may be improved.

[0087] The header field 401 may indicate that the configuration information field 402 is not present in the current superframe 400. The method 500 may include communicating the configuration information in a previous superframe 400 of the sequence of superframes 400 or using out-of-band signaling techniques. As a result, the configuration information (which is at least temporarily static) may be transmitted in an efficient manner.

[0088] Alternatively or additionally, the header field 401 may indicate whether the superframe 400 includes an extension field 404 for additional information regarding the immersive audio signal 111. As a result of this, the superframe structure can be adapted in a flexible manner to future extensions.

[0089] The method 500 may include inserting (if necessary) a configuration information field 402 into the superframe 400. The configuration information field 402 may indicate the number of downmix channel signals 203 included in the data fields 411, 421, 412, 422 of the superframe 400. Alternatively or additionally, the configuration information field 402 may indicate the order of the sound field representation signal included in the IA signal 111. As a result of this, various different types of IA signals 111 (having various different types of SR signals) may be encoded and transmitted.

[0090] The configuration information field 402 may indicate the maximum possible size of the metadata field 403. Alternatively or additionally, the configuration information field 402 may indicate the frame type and / or coding mode used to code each of the one or more downmix channel signals 203. Providing this information allows the use of different coding schemes for encoding the IA signal 111.

[0091] The encoded audio data 206 of the frames of the downmix channel signal 203 may be generated using a multi-mode and / or multi-rate speech or audio codec. Alternatively or additionally, the (encoded) metadata 207 may be generated using a multi-mode and / or multi-rate immersive metadata encoding scheme. As a result of this, the IA signal 111 may be encoded at a relatively low data rate with a relatively high quality.

[0092] The superframes 400 of the sequence of superframes 400 may constitute at least a part of a data element that is transmitted using a transmission protocol, in particular DASH, RTSP or RTP, or stored in a file according to a storage format, in particular ISOBMFF. In other words, the bitstream 101 comprising the sequence of superframes 400 may use one or more data elements of a transmission protocol or storage format, thereby enabling the bitstream 101 to be transmitted or stored in an efficient and reliable manner.

[0093] 6 shows a flow chart of an exemplary method 600 for deriving data relating to an immersive audio signal 111 from a bitstream 101. The bitstream 101 comprises a sequence of superframes 400 for a sequence of frames of the immersive audio signal 111. In a preferred example, multiple (basic) frames of the IA signal 111 are included within a single superframe 400. It should be noted that all features described in the context of the method 500 for generating the bitstream 101 are applicable in an analogous manner for the method 600 for deriving data from the bitstream 101.

[0094] The IA signal 111 may include an SR signal, a multi-channel signal, and / or one or more audio objects. Aspects and / or features described in the context of the method 500 and / or in the context of the encoding device 110 are applicable in an analogous and / or complementary manner to the method 600 and / or the decoding device 120 (and vice versa).

[0095] The method 600 may include iterating over a sequence of superframes 400 and extracting (601) from the data fields 411, 421, 412, 422 of the superframes 400 encoded audio data 206 for one or more (particularly multiple) frames of one or more downmix channel signals 203 derived from the immersive audio signal 111. Furthermore, the method 600 includes extracting (602) from the metadata field 403 of the superframe 400 (encoded) metadata 207 for reconstructing one or more (particularly multiple) frames of the immersive audio signal 111 from the encoded audio data 206.

[0096] The method 600 may include deriving one or more reconstructed audio objects from the encoded audio data 206 and metadata 207 (in particular the object metadata 202). As mentioned above, an audio object typically includes an object signal and object metadata 202 indicating the (time-varying) position of the audio object. Furthermore, the method 600 may include deriving a reconstructed residual signal from the encoded audio data 206 and metadata 202, 205. The one or more reconstructed audio objects and the reconstructed residual signal may describe and / or indicate the IA signal 111. In particular, data (e.g., the order of the SR signal contained within the IA signal 111) may be extracted from the bitstream 101, which allows for the determination of a reconstructed IA signal 121, where the reconstructed IA signal 121 is an approximation of the original IA signal 111.

[0097] As mentioned above, the method 600 for deriving data relating to an immersive audio signal 111 from a bitstream 101 may include corresponding features to the method 500 for generating a bitstream 101. In particular, the method 600 may include extracting a header field 401 from a given superframe 400. A size of a metadata field 403 of the given superframe 400 may be derived from the header field 401.

[0098] The size of the metadata field 403 may be indicated as outlined in the context of the method 500. The metadata field 403 may indicate a maximum possible size, the header field 401 may indicate an adjustment value, and the size of the metadata field 403 of the superframe 400 may correspond to the maximum possible size minus the adjustment value. In particular, the header field 401 may include a size indicator for the size of the metadata field 403, and the size indicator may indicate different resolutions for different size ranges of the size of the metadata field 403. As a result of this, the size of the metadata field 403 may be signaled in a bitrate efficient manner.

[0099] The method 600 may include determining, based on the header field 401, whether the superframe 400 includes a configuration information field 402 and / or whether the configuration information field 402 is present in the superframe 400. If the configuration information field 402 is not present, configuration information provided in a previous superframe 400 and / or provided out-of-band may be used to process one or more frames of the one or more downmix channel signals 203 included in the superframe 400. If the configuration information field 402 is present, the configuration information included in the superframe 400 may be used to process one or more frames of the one or more downmix channel signals 203 included in the superframe 400.

[0100] Additionally, the method 600 may include determining, based on the header field 401, whether the superframe 400 includes an extension field 404 for additional information regarding the immersive audio signal 111, thereby providing an efficient and flexible means for transmitting information within the bitstream 101.

[0101] The method 600 may include extracting the configuration information field 402 from the super-frame 400. Additionally, the method 600 may include determining a number of downmix channel signals 203 represented by the data fields 411, 421, 412, 422 of the super-frame 400 based on the configuration information field 402, thereby enabling accurate processing of one or more frames of the one or more downmix channel signals 203 included within the super-frame 400.

[0102] Further, the method 600 may include determining a maximum possible size of the metadata field 403 based on the configuration information field 402 .

[0103] Additionally, the method 600 may include determining an order of the immersive audio signal 111 to enable accurate reconstruction of the IA signal 111 based on the configuration information field 402 .

[0104] The method 600 may also include determining a frame type and / or coding mode used to encode each of the one or more downmix channel signals based on the configuration information field 402, thereby enabling correct processing of one or more frames of the one or more downmix channel signals 203 included within the super-frame 400.

[0105] Various exemplary embodiments of the invention may be implemented in hardware or special purpose circuits, software, logic, or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. In general, it is understood that the present disclosure encompasses an apparatus suitable for performing the above-described methods, such as an apparatus having a memory and a processor coupled to the memory (a spatial renderer), where the processor is configured to execute instructions and perform the methods according to embodiments of the present disclosure.

[0106] Although various aspects of the exemplary embodiments of the invention have been illustrated and described as block diagrams, flowcharts, or using some other pictorial representations, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in, by way of non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controllers, or other computing devices, or some combination thereof.

[0107] Additionally, the various blocks illustrated in the flowcharts may be viewed as method steps and / or operations resulting from computer program code operations and / or as multiple coupled logic circuit elements configured to perform the associated functions. For example, embodiments of the present invention include a computer program product including a computer program tangibly embodied on a machine-readable medium, the computer program including program code configured to perform the methods described above.

[0108] In the context of this disclosure, a machine-readable medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples of machine-readable storage media include electrical connections having one or more wires, portable computer diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), optical fibers, portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0109] The computer program codes for carrying out the methods of the present invention may be written in any combination of one or more programming languages. These computer program codes may be provided to a processor of a general purpose computer, a special purpose computer, or other programmable data processing apparatus, and when executed by the processor of the computer or other programmable data processing apparatus, the program codes cause the processor to perform the functions / operations specified in the flowcharts and / or block diagrams. The program codes may be executed on the computer, partly on the computer, as a stand-alone software package, partly on the computer, partly on a remote computer, or entirely on a remote computer or server.

[0110] Furthermore, although operations are depicted in a particular order, this should not be understood as requiring such operations to be performed in the particular order depicted or sequentially, or that all of the depicted operations be performed to achieve a desired result. In certain situations, multitasking and parallel processing may be advantageous. Similarly, while some specific implementation details have been included in the above discussion, these should not be construed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features that may be specific to specific embodiments of specific inventions. Certain features that are described in the context of separate embodiments herein may be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may be implemented in multiple embodiments separately or in any suitable subcombination.

[0111] It should be noted that the specification and drawings merely illustrate the principles of the proposed method and apparatus. Thus, it will be understood that those skilled in the art can devise various configurations that, although not explicitly described or shown herein, embody the principles of the present invention and are included within its spirit and scope. Furthermore, all examples described herein are expressly intended for educational purposes only to aid the reader in understanding the principles of the proposed method and apparatus, as well as concepts contributed by the inventors to advance the art, and are to be construed as being limited to the examples and conditions specifically described as such. Furthermore, all statements herein describing principles, aspects, and embodiments of the present invention, as well as specific examples thereof, are intended to encompass equivalents thereof.

[0112] Several aspects will be described. [Aspect 1] A method (500) for generating a bitstream (101), the bitstream (101) comprising a sequence of superframes (400) for a sequence of frames of an immersive audio signal (111), the method (500) comprising the steps of repeatedly: inserting (501) encoded audio data (206) for one or more frames of one or more downmix channel signals (203) derived from the immersive audio signal (111) into a data field (411, 421, 412, 422) of a superframe (400); and inserting (502) metadata (202, 205) for reconstructing one or more frames of the immersive audio signal (111) from the encoded audio data (206) into a metadata field (403) of the superframe (400). method. [Aspect 2] the method (500) comprising inserting a header field (401) into the superframe (400); the header field (401) indicates the size of the metadata field (403) of the superframe (400); The method according to embodiment 1. [Aspect 3] said metadata field (403) indicates the maximum possible size; said header field (401) indicating an adjustment value; the size of the metadata field (403) of the superframe (400) corresponds to the maximum possible size minus the adjustment value; The method according to embodiment 2. Aspect 4 The header field (401) includes a size indicator for the metadata field (403); the size indicators indicating different resolutions for different size ranges of the size of the metadata field (403); The method according to embodiment 2 or 3. Aspect 5 the metadata (202, 205) for reconstructing the one or more frames of the immersive audio signal (111) indicates a statistical size distribution of sizes of the metadata (202, 205); the resolution of the size indicator depends on the size distribution of the metadata (202, 205); The method according to embodiment 4. Aspect 6 the method (500) comprising inserting a header field (401) into the superframe (400); The header field (401) indicates whether the superframe (400) includes a configuration information field (402); The header field (401) indicates the presence of a configuration information field (402), 6. The method according to any one of embodiments 1 to 5. Aspect 7 The method (500) includes inserting a configuration information field (402) into the superframe (400); the configuration information field (402) indicates the number of downmix channel signals (203) represented by the data fields (411, 421, 412, 422) of the superframe (400); 7. The method according to any one of embodiments 1 to 6. Aspect 8 The method (500) includes inserting a configuration information field (402) into the superframe (400); The configuration information field (402) indicates the maximum possible size of the metadata field (403), 8. The method according to any one of embodiments 1 to 7. Aspect 9 The method (500) includes inserting a configuration information field (402) into the superframe (400); The configuration information field (402) indicates the order of the sound field representation signal contained in the immersive audio signal (111), 9. The method according to any one of embodiments 1 to 8. Aspect 10 The method (500) includes inserting a configuration information field (402) into the superframe (400); the configuration information field (402) indicating a frame type and / or a coding mode used to code each of the one or more downmix channel signals (203), 10. The method according to any one of embodiments 1 to 9. Aspect 11 The method (500) includes inserting a header field (401) into the superframe (400); a header field (401) indicating whether the superframe (400) includes an extension field (404) for additional information regarding the immersive audio signal (111); 11. The method according to any one of embodiments 1 to 10. Aspect 12 4. The method of any one of aspects 1 to 3, wherein a superframe (400) includes two or more frames of the one or more downmix channel signals (203). Aspect 13 the encoded audio data (206) for a frame of the downmix channel signal (203) is generated using a multi-mode and / or multi-rate speech or audio codec; and / or the metadata (202, 205) is generated using a multi-mode and / or multi-rate immersive metadata encoding scheme; 13. The method according to any one of embodiments 1 to 12. Aspect 14 14. The method according to any one of aspects 1 to 13, wherein the encoded audio data (206) of a frame of a downmix channel signal (203) is encoded using an enhanced audio services encoder. Aspect 15 15. The method according to any one of the preceding aspects, wherein the superframe (400) constitutes at least a part of a data element transmitted using a transmission protocol, in particular DASH, RTSP or RTP, or stored in a file according to a storage format, in particular ISOBMFF. Aspect 16 the header field (401) indicates that a configuration information field (402) is not present; The method (500) includes communicating configuration information in a previous superframe (400) of the sequence of superframes (400) or using out-of-band signaling, 16. The method according to any one of embodiments 1 to 15. Aspect 17 The method comprises: inserting encoded audio data (206) for one or more frames of a first downmix channel signal (203) and a second downmix channel signal (203) derived from the immersive audio signal (111) into one or more first data fields (411, 421) and one or more second data fields (412, 422) of the superframe (400), respectively, where the first downmix channel signal (203) is encoded using a first encoder and the second downmix channel signal (203) is encoded using a second encoder; providing configuration information for the first and second encoders within the superframe (400), within a previous superframe (400) of the sequence of superframes (400), or using out-of-band signaling; 17. The method according to any one of embodiments 1 to 16. Aspect 18 The method comprises: extracting one or more audio objects from the immersive audio signal (111), referred to as the IA signal, the audio objects comprising an object signal and object metadata (202) indicating the location of the audio objects; determining a residual signal (201) based on said IA signal (111) and based on said one or more audio objects; providing a downmix signal based on the IA signal (111), in particular such that the number of downmix channel signals (203) of the downmix signal is less than the number of channel signals of the IA signal (111); determining joint coding metadata (205) for enabling upmixing the downmix signal into one or more reconstructed audio object signals corresponding to the one or more audio objects and / or a reconstructed residual signal (311) corresponding to the residual signal (201); performing waveform coding of the downmix signal to provide coded audio data (206) for a sequence of frames of the one or more downmix channel signals (203); performing entropy coding of the jointly coded metadata (205) and the object metadata (202) of the one or more audio objects to provide the metadata (202, 205) to be inserted into the metadata field (403) of the sequence of superframes (400), 18. The method according to any one of embodiments 1 to 17. Aspect 19 A superframe (400) of a bitstream (101), the bitstream (101) comprising a sequence of superframes (400) for a sequence of frames of an immersive audio signal (111), the superframe (400) comprising: data fields (411, 421, 412, 422) of encoded audio data (206) for one or more frames of one or more downmix channel signals (203) derived from said immersive audio signal (111); a single metadata field (403) for metadata (202, 205) adapted to reconstruct one or more frames of the immersive audio signal (111) from the encoded audio data (206), Superframe. Aspect 20 A method (600) for deriving data relating to an immersive audio signal (111) from a bitstream (101), the bitstream (101) comprising a sequence of superframes (400) for a sequence of frames of the immersive audio signal (111), the method (600) comprising: extracting (601) from a data field (411, 421, 412, 422) of a superframe (400) encoded audio data (206) for one or more frames of one or more downmix channel signals (203) derived from said immersive audio signal (111); extracting (602) from a metadata field (403) of the superframe (400) metadata (202, 205) for reconstructing one or more frames of the immersive audio signal (111) from the encoded audio data (206), method. Aspect 21 deriving one or more reconstructed audio objects from the encoded audio data (206) and the metadata (202, 205), the audio objects including an object signal and object metadata (202) indicating a position of the audio objects; - deriving a reconstructed residual signal (311) from said encoded audio data (206) and said metadata (202, 205), said one or more reconstructed audio objects and said reconstructed residual signal (311) describing said immersive audio signal (111), The method according to embodiment 20. Aspect 22 The method comprises: extracting a header field (401) from the superframe (400); deriving a size of the metadata field (403) of the superframe (400) from the header field (401), 22. The method according to embodiment 20 or 21. Aspect 23 indicating the maximum possible size of said metadata field (403); said header field (401) indicating an adjustment value; the size of the metadata field (403) of the superframe (400) corresponds to the maximum possible size minus the adjustment value; The method according to embodiment 22. Aspect 24 the header field (401) includes a size indicator for the size of the metadata field (403); the size indicator indicates different resolutions for different size ranges of the size of the metadata field (403); 24. The method of embodiment 22 or 23. Aspect 25 The method comprises: extracting a header field (401) from the superframe (400); determining whether the superframe (400) includes a configuration information field (402) based on the header field (401); determining whether a configuration information field (402) is present in the superframe (400) based on the header field (401); 25. The method according to any one of embodiments 20 to 24. Aspect 26 The method comprises: extracting a configuration information field (402) from the superframe (400); determining the number of downmix channel signals (203) represented by the data fields (411, 421, 412, 422) of the superframe (400) based on the configuration information field (402), 26. The method according to any one of embodiments 20 to 25. Aspect 27 The method comprises: extracting a configuration information field (402) from the superframe (400); determining the maximum possible size of the metadata field (403) based on the configuration information field (402), 27. The method according to any one of embodiments 20 to 26. Aspect 28 The method comprises: extracting a configuration information field (402) from the superframe (400); determining an order of a sound field representation signal included in the immersive audio signal (111) based on the configuration information field (402), 28. The method according to any one of embodiments 20 to 27. Aspect 29 The method comprises: extracting a configuration information field (402) from the superframe (400); determining a frame type and / or a coding mode to be used for coding each of the one or more downmix channel signals (203) based on the configuration information field (402), 29. The method according to any one of embodiments 20 to 28. Aspect 30 The method comprises: extracting a header field (401) from the superframe (400); determining based on the header field (401) whether the superframe (400) includes an extension field (404) for additional information regarding the immersive audio signal (111), 30. The method according to any one of embodiments 20 to 29. Aspect 31 An encoding device (110) configured to generate a bitstream (101), the bitstream (101) comprising a sequence of superframes (400) for a sequence of frames of an immersive audio signal (111), the encoding device (110) repeatedly performs, for the sequence of superframes (400): inserting encoded audio data (206) for one or more frames of one or more downmix channel signals (203) derived from said immersive audio signal (111) into a data field (411, 421, 412, 422) of a superframe (400); and inserting into a metadata field (403) of the superframe (400) metadata (202, 205) for reconstructing one or more frames of the immersive audio signal (111) from the encoded audio data (206). Encoding device. Aspect 32 A decoding device (120) configured to derive data relating to an immersive audio signal (111) from a bitstream (101), the bitstream (101) comprising a sequence of superframes (400) for a sequence of frames of the immersive audio signal (111), the decoding device (120) repeating for the sequence of superframes (400), extracting from a data field (411, 421, 412, 422) of a superframe (400) encoded audio data (206) for one or more frames of one or more downmix channel signals (203) derived from said immersive audio signal; extracting, from a metadata field (403) of the superframe (400), metadata (202, 205) for reconstructing one or more frames of the immersive audio signal (111) from the encoded audio data (206), Decoding device.

Claims

1. A method (500) for generating a bitstream (101), the bitstream (101) comprising a sequence of superframes (400) for a sequence of frames of an immersive audio signal (111), the method (500) comprising, for the sequence of superframes (400), repeatedly: inserting (501) encoded audio data (206) for one or more frames of one or more downmix channel signals (203) derived from the immersive audio signal (111) into a data field (411, 421, 412, 422) of a superframe (400), wherein the encoded audio data (206) for the frames of the downmix channel signals (203) are generated using a multi-mode and / or multi-rate speech or audio codec; and inserting (502) metadata (202, 205) for reconstructing one or more frames of the immersive audio signal (111) from the encoded audio data (206) into a metadata field (403) of the superframe (400). method.

2. the method (500) comprising inserting a header field (401) into the superframe (400); the header field (401) indicates the size of the metadata field (403) of the superframe (400); Optionally, - the metadata field (403) indicates the maximum possible size; - the header field (401) indicates an adjustment value; the size of the metadata field (403) of the superframe (400) corresponds to the maximum possible size minus the adjustment value; The method of claim 1.

3. The header field (401) includes a size indicator for the metadata field (403); the size indicators indicate different resolutions for different size ranges of the size of the metadata field (403); Optionally, the metadata (202, 205) for reconstructing the one or more frames of the immersive audio signal (111) indicates a statistical size distribution of sizes of the metadata (202, 205); the resolution of the size indicator depends on the size distribution of the metadata (202, 205); The method of claim 2.

4. the method (500) comprising inserting a header field (401) into the superframe (400); - the header field (401) indicates whether the superframe (400) includes a configuration information field (402); - the header field (401) indicates the presence of a configuration information field (402), and / or the header field (401) indicates whether the superframe (400) includes an extension field (404) for additional information regarding the immersive audio signal (111); 4. The method according to any one of claims 1 to 3.

5. The method (500) includes inserting a configuration information field (402) into the superframe (400); the configuration information field (402) indicates the number of downmix channel signals (203) represented by the data fields (411, 421, 412, 422) of the superframe (400); and / or The configuration information field (402) indicates the maximum possible size of the metadata field (403), and / or the configuration information field (402) indicates the order of the sound field representation signal contained in the immersive audio signal (111); and / or the configuration information field (402) indicates the frame type and / or coding mode used to code each of the one or more downmix channel signals (203), 5. The method according to any one of claims 1 to 4.

6. 6. The method of claim 1, wherein the encoded audio data (206) of a frame of a downmix channel signal (203) is encoded using an Enhanced Voice Services encoder.

7. 7. The method according to claim 1, wherein the superframe (400) constitutes at least part of a data element transmitted using a transmission protocol, in particular DASH, RTSP or RTP, or stored in a file according to a storage format, in particular ISOBMFF.

8. - the header field (401) indicates that the configuration information field (402) is not present; the method (500) includes communicating configuration information in a previous superframe (400) of the sequence of superframes (400) or using out-of-band signaling; 8. The method according to any one of claims 1 to 7.

9. The method comprises: - inserting encoded audio data (206) for one or more frames of a first downmix channel signal (203) and a second downmix channel signal (203) derived from the immersive audio signal (111) into one or more first data fields (411, 421) and one or more second data fields (412, 422) of the superframe (400), respectively, where the first downmix channel signal (203) is encoded using a first encoder and the second downmix channel signal (203) is encoded using a second encoder; providing configuration information for the first and second encoders within the superframe (400), within a previous superframe (400) of the sequence of superframes (400), or using out-of-band signaling; 9. The method according to any one of claims 1 to 8.

10. The method comprises: - extracting one or more audio objects from the immersive audio signal (111), called the IA signal, the audio object comprising an object signal and object metadata (202) indicating the position of the audio object; determining a residual signal (201) based on said IA signal (111) and based on said one or more audio objects; providing a downmix signal based on said IA signal (111), such that the number of downmix channel signals (203) of said downmix signal is less than the number of channel signals of said IA signal (111); - determining joint encoding metadata (205) for enabling upmixing the downmix signal to one or more reconstructed audio object signals corresponding to the one or more audio objects and / or a reconstructed residual signal (311) corresponding to the residual signal (201); performing waveform coding of the downmix signal to provide encoded audio data (206) for a sequence of frames of the one or more downmix channel signals (203); performing entropy coding of the jointly coded metadata (205) and the object metadata (202) of the one or more audio objects to provide the metadata (202, 205) to be inserted into the metadata field (403) of the sequence of superframes (400), 10. The method according to any one of claims 1 to 9.

11. A superframe (400) of a bitstream (101), the bitstream (101) comprising a sequence of superframes (400) for a sequence of frames of an immersive audio signal (111), the superframe (400) comprising: data fields (411, 421, 412, 422) for encoded audio data (206) for one or more frames of one or more downmix channel signals (203) derived from said immersive audio signal (111), wherein said encoded audio data (206) for a frame of a downmix channel signal (203) is generated using a multi-mode and / or multi-rate speech or audio codec; a single metadata field (403) for metadata (202, 205) adapted to reconstruct one or more frames of the immersive audio signal (111) from the encoded audio data (206), Superframe.

12. An encoding device (110) configured to generate a bitstream (101), the bitstream (101) comprising a sequence of superframes (400) for a sequence of frames of an immersive audio signal (111), the encoding device (110) repeatedly performs, for the sequence of superframes (400): inserting encoded audio data (206) for one or more frames of one or more downmix channel signals (203) derived from the immersive audio signal (111) into a data field (411, 421, 412, 422) of a superframe (400), wherein the encoded audio data (206) for the frames of the downmix channel signals (203) are generated using a multi-mode and / or multi-rate speech or audio codec; and inserting into a metadata field (403) of the superframe (400) metadata (202, 205) for reconstructing one or more frames of the immersive audio signal (111) from the encoded audio data (206). Encoding device.

Images