Method and apparatus for generating a bitstream containing an immersive audio signal

Abstract

To provide a method and device for generating or decoding a bitstream comprising an immersive audio signal.SOLUTION: A method for generating a bitstream including a sequence of superframes for a sequence of frames of an immersive audio signal includes the steps of: inserting, repeatedly for a sequence of superframes, coded audio data for one or more frames of one or more downmix channel signals derived from the immersive audio signal into a data field of the superframe; and inserting metadata for reconstructing the one or more frames of the immersive audio signal from the encoded audio data into a metadata field of the superframe.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 693,246, filed Jul. 2, 2018. The content of the application is hereby incorporated by reference.

[0002] Technical Field This document relates to immersive audio signals that may include sound field representation signals, particularly ambisonics signals. In particular, this document relates to generating and decoding a bitstream that includes immersive audio signals.

Background Art

[0003] The sound or sound field within a listener's listening environment at a listening position can be described using ambisonics signals. Ambisonics signals can be viewed as multi - channel audio signals, where each channel corresponds to a specific directivity pattern of the sound field at the listener's listening position. Ambisonics signals may be described using a three - dimensional (3D) Cartesian coordinate system, with the origin of the coordinate system corresponding to the listening position, the x - axis pointing forward, the y - axis pointing left, and the z - axis pointing upward.

[0004] By increasing the number of audio signals or channels and the number of corresponding directivity patterns (and corresponding panning functions), the description accuracy of the sound field can be improved. As an example, first - order ambisonics signals include four channels or waveforms, namely, the W channel indicating the omnidirectional component of the sound field, the X channel describing the sound field with a dipole directivity pattern corresponding to the x - axis, the Y channel describing the sound field with a dipole directivity pattern corresponding to the y - axis, and the Z channel describing the sound field with a dipole directivity pattern corresponding to the z - axis. Second - order ambisonics signals have nine channels, including the four channels of the first - order ambisonics signal (also called B - format) and five additional channels for different directivity patterns. Generally, an L - order ambisonics signal has L channels of the (L - 1) - order ambisonics signal 2Individual channels and [(L+1) for additional directional patterns] 2 -L 2 Includes ] additional channels (L+1) 2 It has 1 channel (when using the 3D ambisonic format). An L-order ambisonic signal for L>1 is sometimes called a high-order ambisonic (HOA) signal.

[0005] The HOA signal can be used to describe a 3D sound field independently of the speaker arrangement used to render the HOA signal. Examples of speaker arrangements include one or more arrangements of headphones or loudspeakers, or a virtual reality rendering environment. Therefore, it may be beneficial to provide the audio renderer with the HOA signal so that the audio rendering can flexibly adapt to different speaker arrangements. [Overview of the project] [Problems that the invention aims to solve]

[0006] Soundfield representation (SR) signals, such as ambisonic signals, may be complemented with audio objects and / or multi-channel signals to provide immersive audio (IA) signals. This paper addresses the technical problem of transmitting and / or storing IA signals with high perceptual quality in a bandwidth-efficient manner. In particular, this paper addresses the technical problem of providing an efficient bitstream representing IA signals. Such technical problems are solved by the independent claims. Preferred examples are described in the dependent claims. [Means for solving the problem]

[0007] In one aspect, a method for generating a bitstream is described. The bitstream includes a sequence of superframes for a sequence of frames of an immersive audio signal. The method includes iterating over the sequence of superframes and inserting encoded audio data for one or more frames of one or more downmix channel signals derived from the immersive audio signal into the data fields of the superframes. Furthermore, the method includes inserting metadata, in particular encoded metadata, for reconstructing one or more frames of the immersive audio signal from the encoded audio data into the metadata fields of the superframes.

[0008] Another aspect describes a method for deriving data relating to an immersive audio signal from a bitstream. The bitstream includes a superframe sequence of frames of the immersive audio signal. The method involves iterating over the sequence of superframes and extracting encoded audio data from the data field of the superframes for one or more frames of one or more downmix channel signals derived from the immersive audio signal. Furthermore, the method includes extracting metadata from the metadata field of the superframes for reconstructing one or more frames of the immersive audio signal from the encoded audio data.

[0009] Further details include the description of a software program. This software program may be adapted for execution on a processor, and, when executed on a processor, to perform the method steps outlined in this paper.

[0010] In another respect, a storage medium is described. The storage medium may contain a software program adapted for execution on a processor, and when executed on a processor, to perform the method steps outlined in this paper.

[0011] Further details describe computer program products. These computer programs may contain executable instructions for performing the method steps outlined in this paper when executed on a computer.

[0012] Further details describe the superframes of the bitstream. The bitstream contains a sequence of superframes for a sequence of frames of the immersive audio signal. The superframes contain data fields for encoded audio data for one or more (in particular multiple) frames of one or more downmix channel signals derived from the immersive audio signal. In addition, the superframes contain a (single) metadata field for metadata adapted to reconstruct one or more (in particular multiple) frames of the immersive audio signal from the encoded audio data.

[0013] In another aspect, an encoding device configured to generate a bitstream is described. The bitstream includes a sequence of superframes for a sequence of frames of an immersive audio signal. The encoding device is configured to iterate over the sequence of superframes and insert encoded audio data for one or more frames of one or more downmix channel signals derived from the immersive audio signal into the data fields of the superframes; and to insert metadata for reconstructing one or more frames of the immersive audio signal from the encoded audio data into the metadata fields of the superframes.

[0014] In a further aspect, a decoding device configured to derive data relating to an immersive audio signal from a bitstream, wherein the bitstream includes a sequence of superframes relating to a sequence of frames of an immersive audio signal. The decoding device is configured to iterate over the sequence of superframes and extract encoded audio data from the data field of the superframe for one or more frames of one or more downmix channel signals derived from the immersive audio signal; and to extract metadata from the metadata field of the superframe for reconstructing one or more frames of the immersive audio signal from the encoded audio data.

[0015] It should be noted that the methods, apparatus, and systems, including preferred embodiments thereof, outlined in this patent application may be used independently or in combination with other methods, apparatus, and systems disclosed herein. Furthermore, all aspects of the methods, apparatus, and systems outlined in this patent application may be combined in any way. In particular, the features of the claims may be combined with each other in any way. [Brief explanation of the drawing]

[0016] The present invention is described below in an illustrative manner with reference to the accompanying drawings. [Figure 1] An example of an encoding system is shown. [Figure 2] An exemplary encoding unit for encoding immersive audio signals is shown. [Figure 3] This shows another exemplary decoding unit for decoding immersive audio signals; [Figure 4] This shows an exemplary superframe structure for immersive audio signals, particularly for encoded data representing immersive audio signals. [Figure 5]A flowchart of an exemplary method for generating a bitstream including a sequence of superframes indicative of an immersive audio signal is shown. [Figure 6] A flowchart of an exemplary method for extracting information from a bitstream including a sequence of superframes indicative of an immersive signal is shown. **DETAILED DESCRIPTION**

[0017] As outlined above, this document relates to the efficient encoding of immersive audio signals such as HOA signals, multi-channel and / or object audio signals. Here, in particular, HOA signals are more generally referred to herein as soundfield representation (SR) signals. Further, this document relates to the storage or transmission of immersive audio (IA) signals through a transmission network within a bitstream.

[0018] As outlined in the introduction, SR signals may contain a relatively large number of channels or waveforms, and different channels are related to different panning functions and / or different directivity patterns. As an example, an Lth order 3D first order ambisonics (FOA) or HOA signal has (L + 1) 2 channels. A first order ambisonics (FOA) signal is a first order ambisonics signal with 4 channels. SR signals can be represented in a variety of different formats.

[0019] A sound field can be considered to be composed of one or more sound events emitted from any direction around the listening position. As a result, the position of the one or more sound events may be defined on the surface of a sphere (with the listening position or reference position at the center of the sphere).

[0020] Sound field formats such as FOA or higher order ambisonics (HOA) are defined in a way that enables the rendering of a sound field with any speaker arrangement (i.e., any rendering system). However, a rendering system (such as a Dolby Atmos system) is typically constrained in that the possible heights of the speakers are fixed to a defined number of planes (e.g., the (horizontal) plane at ear height, the ceiling or upper plane and / or the floor or lower plane). Thus, the concept of an ideal spherical sound field can be modified to a sound field composed of sound objects located within different rings (similar to stacked rings forming a honeycomb) at different heights on the surface of the sphere.

[0021] As shown in FIG. 1, an audio encoding system 100 includes an encoding unit 110 and a decoding unit 120. The encoding unit 110 may be configured to generate a bitstream 101 for transmission to the decoding unit 120 based on an input signal 111, which may include an immersive audio signal (e.g., used for a virtual reality (VR) application), or may be an immersive audio signal. The immersive audio signal 111 may include an SR signal, a multi-channel signal, and / or a plurality of objects (each object including an object signal and object metadata). The decoding unit 120 may be configured to provide an output signal 121 based on the bitstream 101, which may include a reconstructed immersive audio signal, or may be a reconstructed immersive audio signal.

[0022] Figure 2 shows examples of encoding units 110 and 200. Encoding unit 200 may be configured to encode an input signal 111, which may be an immersive audio (IA) signal 111. The IA signal 111 may include a multi-channel input signal 201. The multi-channel input signal 201 may include an SR signal and one or more object signals. Furthermore, object metadata 202 for the multiple object signals may be provided as part of the IA signal 111. The IA signal 111 may also be provided by a content consumption engine, which may be configured to derive objects and / or SR signals from (composite) IA content, such as VR content, which may include an SR signal, one or more multi-channel signals, and / or one or more objects.

[0023] The encoding unit 200 has a downmix module 210 configured to downmix a multi-channel input signal 201 into a plurality of downmix channel signals 203. The plurality of downmix channel signals 203 may correspond to SR signals, in particular to primary ambisonics (FOA) signals. Downmixing may be performed in the subband region or the QMF region (for example, using 10 or more subbands). The encoding unit 200 further includes a joint encoding module 230 (in particular, an SPAR module) configured to determine joint encoding metadata 205 (in particular, SPAR (Spatial Audio Resolution Reconstruction) metadata) configured to reconstruct a multichannel input signal 201 from multiple downmixed channel signals 203. The joint encoding module 230 may be configured to determine the joint encoding metadata 205 in the subband region. In one example, the spatial audio reconstruction (SPAR) tool is an encoding tool for improved encoding of a relatively large number of audio channels and objects. To gain encoding efficiency, the tool supports reconstructing audio channels and objects from fewer joint input audio channels and low-overhead side information.

[0024] To determine the combined coding or SPAR metadata 205, multiple downmix channel signals 203 may be converted to a subband region and / or processed within the subband region. Furthermore, the multi-channel input signal 201 may be converted to a subband region. Subsequently, the combined coding or SPAR metadata 205 may be determined for each subband, and in particular, an approximation of the subband signal of the multi-channel input signal 201 can be obtained by upmixing the subband signals 203 of the multiple downmix channel signals 203 using the combined coding or SPAR metadata 205. The combined coding or SPAR metadata 205 for various subbands may be inserted into the bitstream 101 for transmission to the corresponding decoding unit 120.

[0025] Furthermore, the encoding unit 200 may have an encoding module 240 configured to perform waveform encoding of multiple downmix channel signals 203 and thereby provide encoded audio data 206. Each of the downmix channel signals 203 may be encoded using a mono waveform encoder (e.g., 3GPP® EVS encoding), thereby enabling efficient encoding. Further examples of encoding multiple downmix channel signals 203 include MPEG AAC, MPEG HE-AAC and other MPEG audio codecs, 3GPP codecs, Dolby Digital / Dolby Digital Plus (AC-3, eAC-3), Opus, LC-3 and other similar codecs. As a further example, the encoding tools included in the AC-4 codec may be configured to perform the operation of the encoding unit 200.

[0026] Furthermore, the encoding module 240 may be configured to perform entropy encoding of the joint encoded metadata (i.e., SPAR metadata) 205 and object metadata 202, thereby providing encoded metadata 207. The encoded audio data 206 and encoded metadata 207 may be inserted into the bitstream 101. The bitstream 101 may exhibit the superframe structure described in this paper. The method 500 described in this paper may be performed by the encoding module 240.

[0027] Figure 3 shows examples of decoding units 120 and 350. Decoding units 120 and 350 may include a receiver that receives a bitstream 101 which may contain encoded audio data 206 and encoded metadata 207. Decoding units 120 and 350 may include a processor and / or demultiplexer that multiplexes and demultiplexes the encoded audio data 206 and encoded metadata 207 from the bitstream 101. Decoding unit 350 has a decoding module 360 ​​configured to derive a plurality of reconstructed channel signals 314 from the encoded audio data 206. The decoding module 360 ​​may further be configured to derive jointly encoded or SPAR metadata 205 and / or object metadata 202 from the encoded metadata 207. Method 600 described herein may be performed by the decoding module 360. Furthermore, the decoding unit 350 has a reconfiguration module 370 configured to derive a reconfigured multichannel signal 311 from the congruent coding or SPAR metadata 205 and from a plurality of reconfigured channel signals 314. The congruent coding or SPAR metadata 205 may convey time and / or frequency-varying elements of an upmix matrix that enables the reconfiguration of the multichannel signal 311 from the plurality of reconfigured channel signals 314. The upmix process may be performed in the QMF (quadrature mirror filter) subband region. Alternatively, another time / frequency transform, in particular a transform based on the FFT (fast Fourier transform), may be used to perform the upmix process. Generally, transforms that enable frequency-selective analysis and (upmix) processing may be applied. The upmix process may also include a decorrelator that enables improved reconstruction of the covariance of the reconfigured multichannel signal 311, and the decorrelator may be controlled by additional congruent coding or SPAR metadata 205.

[0028] The reconstructed multichannel signal 311 may include a reconstructed SR signal and one or more reconstructed object signals. The reconstructed multichannel signal 311 and object metadata may form an output signal 121 (also known as a reconstructed IA signal 121). The reconstructed IA signal 121 may be used for speaker rendering 331, headphone rendering 332, and / or, for example, rendering 333 of VR content that relies on SR representation.

[0029] Therefore, the IA input signal 111 • A downmix signal containing multiple downmix channel signals 203; where the downmix signal 203 may be a sound field representation (SR) signal; and Metadata 202, 205 containing SPAR or jointly coded metadata 205 and / or object metadata 202 for one or more objects Encoding units 110 and 200 configured to encode are described.

[0030] Metadata 202, 205, and in particular SPAR metadata 205, may exhibit a different time resolution than the downmix signal. In particular, metadata 202, 205 may be used for multiple frames (e.g., two frames) of the downmix signal. In view of this, a superframe may be defined for bitstream 101. The superframe includes multiple frames of the downmix signal and metadata 202, 205 for the multiple frames of the SR downmix signal.

[0031] Figure 4 shows an exemplary superframe 400. The superframe 400 may include a base header (BH) field 401 and / or a configuration information (CI) field 402 which may contain data valid for the entire superframe 400. Furthermore, the superframe 400 includes signal data fields 411, 412, 421, 422 for encoded audio data 206 for one or more (particularly more) frames of the downmix signal. In particular, one or more (particularly more) signal data fields 411, 412, 421, 422 may be provided for each downmix channel signal 203, for example, signal data fields 411, 421 for two frames of a first downmix channel signal 203 and signal data fields 412, 422 for two frames of an Nth downmix channel signal 203. The signal data fields 411, 412, 421, and 422 are also referred to herein as EVS bit fields (for example, used by an EVS encoder to encode the downmix channel signal 203).

[0032] Furthermore, the superframe 400 includes a metadata field (MDF) 403. The metadata field 403 may be configured to provide SPAR or combined coded metadata 205 and / or prediction coefficients (PC). Thus, the metadata field 403 can be an SPAR bit field or a PC bit field (depending on the coding mode used). In addition, the superframe 400 may include a frame extender (FE) field 404.

[0033] Therefore, the superframe 400 may include signal transmission elements configured as follows: • Indicates one or more (EVS) codec modes used for (EVS) encoding of N downmix channel signals; the default may be N=4 channels, i.e., there are 4 (EVS) codec downmix channel signals W, X', Y', Z'; • Indicates the selected operating mode of the metadata-assisted (EVS) codec; • Shows metadata and bitrate; • Provides the potential for signal transmission of potential future expansions.

[0034] One or more signaling elements (e.g., CI field 402) may be provided conditionally in-band within a superframe 400. If an optional or conditional signaling element is provided, it may be dynamically adapted and / or included within the superframe 400. One or more signaling elements may be statically held and / or provided only once, for example, as an out-of-band message. One or more signaling elements may be semi-dynamic, in which case they are provided in-band only within selected superframes 400.

[0035] The Superframe 400 may be designed to enable one or more of the following features: • Complete decoding and rendering of metadata-assisted EVS encoded superframes. • Partial mono-decoding of metadata-assisted EVS-encoded superframes. - Low-computational extraction of superframe size information from a concatenated sequence of superframes, without the need to decode the superframes. For example, to put the superframe size information into a secondary format that provides or requires this superframe size information (e.g., ISOBMFF, an ISO-based media file format). • Low-computation bitrate determination that does not require decoding of superframe data. • Superframe: Low-computation feedforward and skipping of Superframe data, eliminating the need to decode the data. • Low-computational-intensive feedback that does not require decoding of superframe data (especially in constant bitrate operation). - Simple resynchronization and superframe skipping in case of bit errors in arithmetic-encoded and / or entropy-encoded EVS and / or metadata bitstream portions. • Editable superframe. This allows replacing metadata or EVS data frames.

[0036] A metadata-assisted EVS codec encoded bit superframe 400 can accommodate an encoding stride of 40ms (e.g., two frames of 20ms each). It consists of the following basic bit fields: • Base Header Field (BH) 401: This field includes a Configuration Field Presence Indicator (CPI), a MetaData Field Size Adjustment Indicator (MDA), and an Extension Indicator (EI). The CPI may indicate whether the Configuration Information (CI) field is supplied in the current superframe 400. The MDA may signal the difference between the signaled maximum metadata frame size and the actual metadata frame size. The EI may signal whether the superframe 400 is extended by a Frame Extender (FE) 404. • Configuration Information Field (CI) 402: This field may carry signal-transmission information regarding the configuration of the EVS, SPAR, and Predictive Coefficient coding tools used, such as the frame type (coding mode), bitrate, and other configuration parameters described in this paper. • EVS bit fields 411, 421, 412, 422: Each field may carry bits of a single EVS frame, as specified in 3GPP TS26.445: "Codec for Enhanced Voice Services (EVS); Detailed algorithmic description", Section 7 (in particular, without the EVS payload header). This document is incorporated herein by reference. • SPAR bit field (SPAR) 403: This field may carry the bits of a single SPAR metadata frame, and may be zero-padded at the end for byte alignment. • Predictive Coefficient bit field (PC) 403: This field may carry bits from a single predictive coefficient metadata frame, and may be zero-padded at the end for byte alignment. • Frame Extender (FE) 404: This field may be defined for future use and may carry extension data. Except for the size element contained in the FE, other data carried by the FE may be reserved for future use (RFU).

[0037] All basic bit fields are byte-aligned and, if necessary, may be zero-padded to the end until they reach their defined size.

[0038] The above basic fields may be included within a (single) superframe 400 in the following sequence order. The superframe is • One base header (BH) 401, - Configuration Field Presence Indicator (CPI), - Metadata Field Size Adjustment Indicator (MDA) and - Extended Indicator (EI) BH includes. • One optional configuration information field (CI) 402. The presence of the CI field 402 may be signaled by the CPI. • N EVS-encoded downmix channel signals S1, ..., S N The data for each downmix channel signal consists of two successive frames. These can be carried by 2 × N basic EVS bit fields 411, 421, 412, 422 (hereinafter referred to as EVS(.)). In the default operation with four downmix channel signals, there are eight successive EVS bit fields 411, 421, 412, 422, representing two frames for each downmix channel signal W, X', Y', Z'. • One metadata frame (MDF) field 403 for SPAR or prediction coefficient. Therefore, this is - One basic SPAR bit field, or -It is a single basic PC bit field. • One optional frame extender (FE) 404. The presence of an FE field may be indicated by an EI.

[0039] Table 1 shows an exemplary structure of the Super Frame 400. [Table 1]

[0040] In the default scenario, there are four EVS-encoded downmix channel signals. The superframe structure for the default scenario is shown in Table 2. [Table 2]

[0041] Further details regarding the various basic bit fields are provided below.

[0042] The Base Header (BH) field 401 may carry a Configuration field Presence Indicator (CPI), a MetaData field size Adjustment indicator (MDA), and an Extension Indicator (EI). This byte field may always be the first element of the superframe 400.

[0043] The structure of BH field 401 is shown in Table 3. [Table 3]

[0044] The Configuration Field Presence Indicator (CPI) may be a single bit used to signal the presence of a Configuration Information (CI) field within the current Superframe 400. The CPI may have the following meanings: • CPI='0': This indicates that the configuration information field is not provided in the current superframe 400. It should be noted that instead, the configuration information may be provided as static out-of-band information, or from the most recent previously received superframe 400 that carries the configuration information field 402. • CPI='1': Indicates that the configuration information field is provided in the current superframe 400. The configuration information provided in the CI field 402 is valid for this superframe 400 and future superframes 400 until the next superframe 400 carrying the configuration information field 402 is provided.

[0045] A metadata field size adjustment indicator (MDA) may be provided immediately after the CPI bit. This 6-bit indicator may signal the difference between the length of the MDF 403, which is signaled by the MDR element (defined below), and the actual size of the MDF 403. The difference shown can be derived from the lookups shown in Table 4, using the MDA as an index. The set of adjustment values ​​in Table 4 are specified in Matlab style:start-value:step-size:end-value. The non-constant adjustment parameter step sizes shown in Table 4 can be designed according to an approximate model of the distribution of the total entropy code length of the metadata. This allows for minimizing the number of unused bits in the MDF 403 and thus minimizing transmission overhead. [Table 4]

[0046] Depending on the maximum MDF size, the adjustment value is expressed in units of 1 or 2 bytes. For maximum MDF sizes up to 275 bytes, the adjustment value is expressed in units of 1 byte; otherwise, it is expressed in units of 2 bytes.

[0047] The MDA indicator may be followed by a single Extension Indicator bit (EI). If this bit is set to 1, a Frame Extender (FE) element is added after the current Superframe 400.

[0048] The optional configuration information (CI) field 402 may carry the following signal transmission elements, as shown in Table 5. The CI field 402 may consist of 8 bytes of data, or may contain 8 bytes of data (for two EVS frames per downmix channel signal and N=4 downmix channels). [Table 5]

[0049] Table 6 shows the optional configuration information field 402 for the default case with four EVS-encoded downmix channel signals. In this case, the CI field consists of 9 bytes of data. [Table 6]

[0050] An indicator (NI) for the number N of EVS-encoded downmix channel signals may be a 3-bit element that encodes the number N of EVS-encoded downmix channel signals. N is obtained from the indicator NI by incrementing the number represented by the 3-bit element by 1. To achieve the default operation with four EVS downmix channel signals, the NI element may be set to 3 ('011').

[0051] The Metadata Type Indication (MDT) bit can have the following meanings: • MDT='0': Indicates that the MDF carries the PC bit field. • MDT='1': Indicates that the MDF carries the SPAR bit field.

[0052] The MetaData Coding configuration field (MDC) may contain configuration information for either the prediction coefficient tool or the SPAR coding tool used, depending on the indication of the MDT bit. The MDC field may also be an 11-bit element of CI field 402. The meaning of its bits may depend on the MDT bit of CI field 402. Depending on the value of the MDT bit, the MDC bit may have the following meanings: • MDT='0': When the MDT bit is 0, the three MSBs of the MDC encode the configuration parameters of the prediction coefficient coding scheme. The remaining 8 bits of the MDC are not used and are zero-padded. The structure and contents of the MDC field in this case are shown in Table 7a. • MDT='1': If the MDT bit is 1, 11 MDC bits encode the SPAR codec configuration shown in Table 7b. The HOA order can be calculated by incrementing hoa_order_idx by 1. [Table 7]

[0053] The MetaData Bit Rate signaling field (MDR) may contain 5 bits and may be used to encode the maximum size of the MDF. The maximum MDF size may be obtained by a table lookup using Table 8, where the MDR value is the index in Table 8. Furthermore, Table 8 shows the (maximum) metadata bitrate in kbps. In Table 8, the actual MDF size is signaled as the maximum MDF size minus the adjustment number / value shown by MDA (from BH field 401). This allows for signaling of the actual MDF size with fine resolution (typically byte resolution). It should also be noted that unused bits in the MDF may be zero-padding, which may occur if the actual MDF size provides more space than is required for the encoded metadata. [Table 8]

[0054] The Band Number field (BND) may be a 3-bit number and may indicate the number of subbands used in metadata encoding. The Band Number is derived from the BND value by a lookup in Table 9. By default, the BND field may be set to 5 ('101'), which indicates 12 subbands. [Table 9]

[0055] The reserved bit (RES) may be used to reserve a bit for future use. By default, this bit is set to '0' and may be ignored by the recipient.

[0056] The EVS FT field (FT-x,y) may represent the EVS frame type (FT) applied for encoding the y-th frame of the x-th downmix channel signal, where x=1…N and y=1,2. The EVS frame type may also be defined in 3GPP TS 26.445, section A2.2.1.2, which is incorporated here by reference. It should be noted that the last EVS FT field in CI field 402 may be followed by up to seven zero-padding bits to ensure octet alignment. If the last EVS FT field ends with octet alignment, no zero-padding bits are added afterward. The zero-padding bits are ignored by the receiver.

[0057] The basic EVS bit fields 411, 421, 412, and 422 may be defined for each EVS coding mode used, as specified in 3GPP TS 26.445, Section 7 (which is incorporated here by reference). No additional signal-transmitting bits are defined as part of the basic EVS frame fields to indicate the bitrate or EVS operating mode, as specified in the cited literature. This information may be part of the optional CI field 402 of the current or previous superframe 400, or it may be provided out of band.

[0058] Table 10 shows the detailed allocation of coefficients for SPAR metadata. Table 10 shows the order in which bits are inserted into the frame. Note that the most significant bit (MSB) of each parameter is always inserted first. Since each field is dynamically quantized, the bit allocation is variable. [Table 10]

[0059] Table 11 shows the detailed allocation of PC metadata coefficients. Table 11 shows the order of bits inserted within the superframe 400. Note that the most significant bit (MSB) of each parameter is always inserted first. Since each field is dynamically quantized, the bit allocation is variable. [Table 11]

[0060] A frame extender (FE) 404 typically carries a 16-bit unsigned integer in its first two bytes that indicates the size of the FE field 404 in bytes. This element is called the FE size. Therefore, the FE size number is 2 or greater. The content and meaning of the remaining FE data portion of the FE field 404 may be reserved for future use. By default, the FE size element may be parsed, while the FE data element may be skipped and ignored. The structure and content of the FE field 404 are shown in Table 12. [Table 12]

[0061] Therefore, a superframe structure is described that enables the signal transmission of metadata-assisted EVS codec configuration information. The superframe structure allows the receiver to decode metadata-assisted EVS codec data.

[0062] At a general level, metadata-assisted EVS codecs are multimode and / or multirate coding systems. The underlying EVS codec may be configured to operate in a number of different coding modes and / or bitrates. Furthermore, spatial metadata codecs may offer a variety of different coding modes and / or bitrates. Spatial metadata codecs typically use entropy coding, which results in a non-constant bitrate. This means that the bitrate actually used is typically lower than the given target bitrate. For some frames, this bitrate undershoot may be smaller, and for some other frames, it may be larger.

[0063] The exact encoding mode and bitrate used by the encoder 110 may be provided so that the decoder 120 can properly decode the transmitted bitstream 101. For the entropy-encoded portion of the metadata, the exact bitrate used may not be required, since the Huffman coding used is commaless and uniquely decodeable. Nevertheless, the receiver of bitstream 101 may provide the number of bits used for encoding the frames (or superframes 400). This is desirable, for example, if the decoder 120 needs to skip some received frames without needing to decode them. This paper describes a superframe structure that supports the following features: • Full-frame decoding. • Decoding only the parts necessary for mono playback. • Extract length information from frame 400 and convert this information into a secondary format (ISOBMFF) that provides and / or requires it. • When concatenating 400 frames, decode only in the middle, efficiently skipping the first few frames. • If a bit error occurs, find the start of the next frame (resynchronize). • Determines the bitrate quickly and efficiently without needing to decode the frame. • Editing frames (replacing metadata or parts of EVS frames). • High-speed feedforward operation without decoding frames. • Support for efficient delivery of codec data payloads of fixed and variable length.

[0064] Furthermore, the superframe structure is described to include all the necessary signal transmission elements for the following: This shows the EVS codec modes (including bitrate) used for EVS coding of N downmix channel signals. The default may be N=4, which means there are four EVS codec downmix channels W, X', Y', and Z'. • Indicates the selected operating mode of the supporting metadata codec. • Displays metadata bitrate at high resolution without significant signal transmission overhead. • Provides the potential for signal transmission of potential future expansions.

[0065] Some signaling elements of a superframe 400 may not change frequently during an encoding session, or may even be static. Several other signaling elements, such as metadata bitrate, may change from superframe to superframe. For this reason, certain signaling elements are only conditionally provided in-band within a superframe 400 (e.g., CI field 402). When provided, these signaling elements can be dynamically adapted per superframe. Alternatively, these signaling elements may be kept static and provided only once, for example, as an out-of-band message. Signaling elements may also be semi-dynamic, in which case they are only provided in-band in certain superframes.

[0066] Regarding the signaling of metadata bitrate, a major challenge is that the number of bits (or bytes) required per superframe 400 can vary within a relatively large range. If only the maximum possible number of bits per frame is signaled, a relatively large number of bits may remain unused if the entropy code is considerably shorter than its maximum length. On the other hand, providing a direct signaling element to indicate the number of bits (or bytes) actually used within a superframe 400 would require a relatively large number of signaling bits. This paper describes a method that minimizes the number of signaling bits for the number of bits (or bytes) actually used within a superframe 400 while covering a relatively large range of possible metadata bitrates.

[0067] From a system perspective, the metadata-assisted EVS codec superframe 400 is generated at the encoding headend. This may be a server in a network with access to unencoded immersive or VR (Virtual Reality) audio data. It may also be a mobile phone capturing the immersive audio signal. The encoded frame 400 can be downloaded to a receiving terminal or inserted into a file transmitted according to a streaming protocol such as DASH (Dynamic Adaptive Streaming over HTTP) or RTSP / RTP (Real-Time Streaming Protocol / Real-time Transport Protocol). If the encoded superframe 400 is stored in a file, it may be inserted into a file formatted according to ISOBMFF. If certain configuration information is static and not transmitted as part of the superframe 400, it may instead be provided from the encoding end to the decoding end by out-of-band means such as a session description protocol (SDP).

[0068] The scheme outlined in this paper may use the EVS codec as the underlying codec and may provide multimode / multirate messages (frame types) in-band or out-of-band using, for example, SDP in Superframe 400. This may be combined with a multimode immersive metadata coding framework, which can be efficiently configured using a set of configuration parameters that can also be transmitted in-band or out-of-band. Furthermore, the multimode immersive metadata coding may be combined with a scheme that allows the relevant maximum bitrate (or number of bits in a frame / superframe) in-band or out-of-band.

[0069] The superframe structure described in this paper transmits the actual metadata field size used as the maximum number (which is optionally transmitted out of band) minus the tuning parameters. The tuning parameters are transmitted as part of each superframe 400. The encoding of the tuning parameters is preferably performed with a non-constant step size, allowing the reduced number of transmitted bits for the tuning parameters to cover an increased range of possible adjustments. Furthermore, the non-constant tuning parameter step size may be designed using an approximate model of the distribution of the total entropy code length of the metadata. This minimizes the number of unused bits in the metadata field, thus minimizing transmission overhead. Additionally, the overhead for the metadata bitrate (size) can be transmitted while minimizing the number of unused bits in the metadata field. Therefore, the overall transmission bitrate is reduced.

[0070] The configuration information (CI) in CI field 402 may relate to the selected EVS frame type for EVS coding of the four downmix channel signals W, X', Y', and Z'. The configuration information may further relate to (i) the selected operating mode of metadata-assisted EVS code, FOA, or HIQ; (ii) the bitrate of SPAR metadata in the case of HIQ operation; and (iii) the bitrate of prediction coefficient metadata in the case of FOA operation. The configuration information may be (1) dynamic and provided in-band with the payload; (2) semi-dynamic and provided in-band with the previous payload; or (3) static and provided out-of-band as a hexadecimal string with the codec attributes of the DASH adaptive set.

[0071] FOA (First Order Ambisonics) mode is a low-bitrate operating mode that relies on prediction coefficient metadata (e.g., operating at approximately 128kbps). Due to its relatively low spatial resolution, FOA typically exhibits relatively limited quality. HIQ (High Immersive Quality) mode is a medium to high-bitrate operating mode (e.g., operating at 128–512kbps). It relies on SPAR metadata and aims to reconstruct the original SR signal, providing very high immersive quality.

[0072] Figure 5 shows a method 500 for generating a bitstream 101, which includes a sequence of superframes 400 for a sequence of (basic) frames of an immersive audio signal 111. The immersive audio (IA) signal 111 may include a soundfield representation (SR) signal that can describe the sound field at a reference position. The reference position may be the listener's listening position and / or the microphone's capture position. The SR signal may include multiple channels (or waveforms) for multiple different directions of arrival of the sound field at the reference position. Alternatively or additionally, the IA signal 111 may include one or more audio objects and / or multichannel signals.

[0073] The IA signal 111, in particular the SR signal contained within the IA signal, may include an L-order ambisonic signal, or it may be an L-order ambisonic signal, where L is 1 or greater. Alternatively or additionally, the SR signal may represent a beehive (BH) format having multiple directions of arrival arranged in multiple different rings on a sphere around a reference position. The multiple rings may include a central ring, an upper ring, a lower ring, and / or a zenith. Alternatively or additionally, the SR signal may represent an intermediate spatial format called ISF, in particular the ISF format as defined within Dolby Atmos technology.

[0074] Therefore, the IA signal 111 may contain multiple different channels. Each channel contained within the IA signal 111 typically contains a sequence of audio samples for a sequence of time points or for a sequence of frames. In other words, the “signal” described in this paper typically contains a sequence of audio samples for a sequence of time points or frames (for example, a temporal distance of 20 ms or less).

[0075] Method 500 may include extracting one or more audio objects from the IA signal 111. An audio object typically includes an object signal (having a sequence of audio samples for a corresponding sequence of time points or frames). Furthermore, an audio object typically includes object metadata 202 indicating the location of the audio object. The location of the audio object may change over time, so the object metadata 202 of the audio object may indicate a sequence of locations for a sequence of time points or frames.

[0076] Furthermore, method 500 may include determining a residual signal based on the IA signal 111 and on one or more audio objects. The residual signal may describe the original IA signal from which one or more audio objects 103, 303 have been extracted and / or removed. The residual signal may also be an SR signal contained within the IA signal 111. Alternatively or additionally, the residual signal may include, or may be, a multi-channel audio signal and / or a bed of audio signals. Alternatively or additionally, the residual signal may include a plurality of audio objects at fixed object locations and / or positions (for example, audio objects assigned to specific speakers in a defined speaker arrangement).

[0077] Furthermore, Method 500 may include generating and / or providing a downmix signal based on the IA signal 111 (for example, using a downmix module 210). The number of channels in the downmix signal is typically smaller than the number of channels in the IA signal 111. Furthermore, Method 500 may include determining a joint coding or SPAR metadata 205 that enables upmixing the downmix signal (i.e., one or more downmix channel signals 203) into signals for one or more reconfigured audio objects for one or more corresponding audio objects. Furthermore, the joint coding or SPAR metadata 205 may enable upmixing the downmix signal into reconfigured residual signals for corresponding residual signals.

[0078] A downmix signal including one or more downmix channel signals 203, SPAR metadata 205, and object metadata 202 may be inserted into the bitstream 101. In particular, method 500 may include performing waveform coding on the downmix signal to provide encoded audio data 206 for a sequence of frames of one or more downmix channel signals 203. Waveform coding may be performed using, for example, Enhanced Voice Services (EVS) coding. Furthermore, method 500 may include performing entropy coding on the SPAR metadata 205 and / or object metadata 202 for one or more audio objects to provide (encoded) metadata 207 to be inserted into the bitstream 101.

[0079] Method 500 may also include repeating for a sequence of superframes 400 the encoded audio data 206 for one or more (in particular multiple) frames (e.g., two or more frames) of one or more downmix channel signals 203 derived from the immersive audio signal 111 into the data fields 411, 421, 412, 422 of the superframe 400 (501). The (basic) frame of the downmix channel signal 203 may span 20 ms of the downmix channel signal 203. The superframe 400 may span a multiple of the length of the (basic) frame, for example, 40 ms.

[0080] Furthermore, method 500 may include inserting metadata 202, 205 (particularly encoded metadata 207) for reconstructing one or more (particularly multiple) frames of the immersive audio signal 111 from the encoded audio data 206 into a (single) metadata field 403 of the superframe 400 (502). Thus, the superframe 400 can provide metadata 202, 205 for one or more (particularly multiple) frames of one or more downmix channel signals 203, thereby enabling efficient transmission of the IA signal 111.

[0081] In particular, the frames of the downmix channel signal 203 may be generated using a multimode and / or multirate speech or audio codec. Furthermore, the metadata 202, 205 may be generated using a multimode and / or multirate immersive metadata encoding scheme. Configuration information indicating the operation of the multimode and / or multirate speech or audio codec (used for the downmix channel signal 203) and / or the operation of the multimode and / or multirate immersive metadata encoding scheme may be contained in the configuration information field 402 of the (current) superframe 400, in the sequence of superframes 400, in the configuration information field 402 of the previous superframe 400, or may be transmitted using an out-of-band signaling scheme. As a result, an efficient and flexible method for encoding the immersive audio signal 111 may be provided.

[0082] The superframe 400 may include encoded audio data 206 associated with multiple downmix channel signals 203. The encoded audio data 206 for the frames of the first downmix channel signal 203 may be generated using a first instance of a multimode and / or multirate speech or audio codec. Furthermore, the encoded audio data 206 for the frames of the second downmix channel signal 203 may be generated using a second instance of a multimode and / or multirate speech or audio codec, and the first and second instances of the multimode and / or multirate speech or audio codec may be different. Configuration information (contained within the current superframe 400, within the previous superframe 400, or transmitted out of band) may indicate the operation of the first and second instances of the multimode and / or multirate speech or audio codec (in particular, each instance). This can further improve the flexibility and efficiency for encoding the immersive audio signal 111.

[0083] In other words, Method 500 may include inserting encoded audio data 206 for one or more frames of a first downmix channel signal 203 and a second downmix channel signal 203 derived from an immersive audio signal 111 into one or more first data fields 411, 421 and one or more second data fields 412, 422 of a superframe 400, respectively. The first downmix channel signal 203 may be encoded using a first (audio or speech) encoder, and the second downmix channel signal may be encoded using a second (audio or speech) encoder. The first and second encoders may be different or operated using different configurations. Furthermore, Method 500 may include providing configuration information relating to the first and second encoders within the superframe 400, in a sequence of superframes 400, in a preceding superframe 400, or using an out-of-band signaling scheme. This can further improve the flexibility and efficiency of encoding the immersive audio signal 111.

[0084] Method 500 may include inserting a header field 401 into the superframe 400. The header field 401 may indicate the size of the metadata field 403 of the superframe 400, thereby allowing the size of the superframe 400 to flexibly accommodate various lengths of the metadata 207 (entropy-encoded and / or lossless-encoded).

[0085] The metadata field 403 may indicate the maximum possible size (which may, for example, be indicated in an optional configuration information field 402 of the superframe 400). The header field 401 may indicate an adjustment value, and the size of the metadata field 403 of the superframe 400 may correspond to the maximum possible size minus the adjustment value, thereby enabling the size of the metadata field 403 to be signaled accurately and efficiently. The header field 401 may include a size indicator (e.g., the adjustment value) for the size of the metadata field 403. The size indicator may indicate different resolutions or step sizes (with respect to size intervals) for different size ranges of the size of the metadata field 403. The resolution and / or step size of the size indicator may depend on the statistical size distribution of the (entropy-encoded) metadata. By providing a size indicator with variable resolution, the bitrate efficiency for signaling the size of the metadata field 403 can be improved.

[0086] The header field 401 may indicate whether the superframe 400 includes a configuration information field 402. In other words, the header field 401 may indicate the presence of a configuration information field 402. The configuration information field 402 may be inserted into the superframe 400 only when necessary (for example, when the encoder configuration of the IA signal 111 is changed). As a result, the bitrate efficiency of the sequence of the superframe 400 may be improved.

[0087] The header field 401 may indicate that the configuration information field 402 is not present in the current superframe 400. Method 500 may include transmitting the configuration information within a previous superframe 400 in the sequence of superframes 400, or using an out-of-band signaling scheme. As a result, the configuration information (which is at least temporarily static) can be transmitted in an efficient manner.

[0088] Alternatively or additionally, the header field 401 may indicate whether the superframe 400 includes an extension field 404 for additional information regarding the immersive audio signal 111. As a result, the superframe structure can be adapted in a flexible manner to future expansions.

[0089] Method 500 may include (if necessary) inserting a configuration information field 402 into the superframe 400. The configuration information field 402 may indicate the number of downmix channel signals 203 contained within the data fields 411, 421, 412, and 422 of the superframe 400. Alternatively or additionally, the configuration information field 402 may indicate the order of the sound field representation signals contained within the IA signal 111. As a result, various different types of IA signals 111 (having various different types of SR signals) can be encoded and transmitted.

[0090] The configuration information field 402 may indicate the maximum possible size of the metadata field 403. Alternatively or additionally, the configuration information field 402 may indicate the frame type and / or encoding mode used to encode each of the one or more downmix channel signals 203. Providing this information allows for the use of different encoding schemes for encoding the IA signal 111.

[0091] The encoded audio data 206 of the frames of the downmix channel signal 203 may be generated using a multimode and / or multirate speech or audio codec. Alternatively or additionally, the (encoded) metadata 207 may be generated using a multimode and / or multirate immersive metadata encoding scheme. As a result, the IA signal 111 can be encoded at a relatively low data rate with relatively high quality.

[0092] A sequence of superframes 400 may constitute at least a portion of data elements that are transmitted using a transmission protocol, in particular DASH, RTSP, or RTP, or stored in a file according to a storage format, in particular ISOBMFF. In other words, the bitstream 101 containing the sequence of superframes 400 may use one or more data elements of a transmission protocol or storage format. This enables the bitstream 101 to be transmitted or stored in an efficient and reliable manner.

[0093] Figure 6 shows a flowchart of an exemplary method 600 for deriving data relating to an immersive audio signal 111 from a bitstream 101. The bitstream 101 contains a sequence of superframes 400 for a sequence of frames of the immersive audio signal 111. In one preferred example, multiple (basic) frames of the IA signal 111 are contained within a single superframe 400. It should be noted that all features described in the context of method 500 for generating the bitstream 101 are similarly applicable to method 600 for deriving data from the bitstream 101.

[0094] The IA signal 111 may include an SR signal, a multi-channel signal, and / or one or more audio objects. Aspects and / or features described in the context of Method 500 and / or the context of the encoding device 110 are applicable to Method 600 and / or the decoding device 120 in a similar and / or complementary manner (and vice versa).

[0095] Method 600 may include (601) iterating over a sequence of superframes 400 and extracting encoded audio data 206 for one or more (in particular multiple) frames of one or more downmix channel signals 203 derived from the immersive audio signal 111 from the data fields 411, 421, 412, 422 of the superframes 400. Furthermore, Method 600 includes (602) extracting (encoded) metadata 207 from the metadata field 403 of the superframes 400 for reconstructing one or more (in particular multiple) frames of the immersive audio signal 111 from the encoded audio data 206.

[0096] Method 600 may include deriving one or more reconstructed audio objects from the encoded audio data 206 and metadata 207 (in particular, object metadata 202). As described above, an audio object typically includes an object signal and object metadata 202 indicating the (time-varying) position of the audio object. Furthermore, Method 600 may include deriving a reconstructed residual signal from the encoded audio data 206 and metadata 202, 205. The one or more reconstructed audio objects and the reconstructed residual signal may describe and / or represent the IA signal 111. In particular, data (e.g., the order of the SR signals contained within the IA signal 111) may be extracted from the bitstream 101, which enables the determination of a reconstructed IA signal 121, where the reconstructed IA signal 121 is an approximation of the original IA signal 111.

[0097] As described above, method 600 for deriving data relating to the immersive audio signal 111 from the bitstream 101 may include corresponding features to method 500 for generating the bitstream 101. In particular, method 600 may include extracting a header field 401 from a given superframe 400. The size of the metadata field 403 of the given superframe 400 may be derived from the header field 401.

[0098] The size of the metadata field 403 may be indicated as outlined in the context of Method 500. The metadata field 403 may indicate the maximum possible size, and the header field 401 may indicate an adjustment value, and the size of the metadata field 403 in the superframe 400 may correspond to the maximum possible size minus the adjustment value. In particular, the header field 401 may include a size indicator for the size of the metadata field 403, and the size indicator may indicate different resolutions for different size ranges of the size of the metadata field 403. As a result, the size of the metadata field 403 can be signaled in a bitrate-efficient manner.

[0099] Method 600 may include determining, based on the header field 401, whether the superframe 400 includes a configuration information field 402 and / or whether the configuration information field 402 is present in the superframe 400. If the configuration information field 402 is not present, configuration information provided in a previous superframe 400 and / or provided out of band may be used to process one or more frames of one or more downmix channel signals 203 contained in the superframe 400. If the configuration information field 402 is present, configuration information contained in the superframe 400 may be used to process one or more frames of one or more downmix channel signals 203 contained in the superframe 400.

[0100] Furthermore, method 600 may include determining, based on the header field 401, whether the superframe 400 includes an extension field 404 for additional information relating to the immersive audio signal 111, thereby providing an efficient and flexible means for transmitting information within the bitstream 101.

[0101] Method 600 may include extracting a configuration information field 402 from the superframe 400. Furthermore, Method 600 may include determining the number of downmix channel signals 203 represented by the data fields 411, 421, 412, and 422 of the superframe 400 based on the configuration information field 402, thereby enabling precise processing of one or more frames of one or more downmix channel signals 203 contained within the superframe 400.

[0102] Furthermore, method 600 may include determining the maximum possible size of the metadata field 403 based on the configuration information field 402.

[0103] Furthermore, method 600 may include determining the order of the immersive audio signals 111 based on the configuration information field 402 in order to enable the precise reconstruction of the IA signals 111.

[0104] Method 600 may also include determining, based on the configuration information field 402, the frame type and / or encoding mode used to encode each of the one or more downmix channel signals, thereby enabling precise processing of one or more frames of the one or more downmix channel signals 203 contained within the superframe 400.

[0105] Various exemplary embodiments of the present invention may be implemented in hardware or special-purpose circuits, software, logic, or any combination thereof. Some aspects may be implemented in hardware, while others may be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. Generally, it is understood that the present disclosure encompasses devices suitable for performing the methods described above, such as a device having memory and a processor coupled to the memory (a spatial renderer), wherein the processor is configured to execute instructions and perform the methods according to embodiments of the present disclosure.

[0106] While various aspects of exemplary embodiments of the present invention are illustrated and described using block diagrams, flowcharts, or other pictorial representations, it will be understood that the blocks, apparatus, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, special-purpose circuits or logic, general-purpose hardware or controllers, or other computing devices, or any combination thereof, as examples without limitation.

[0107] Furthermore, the various blocks shown in the flowchart can be viewed as method steps and / or actions resulting from the operation of computer program code and / or as a plurality of coupled logic circuit elements constructed to perform related functions. For example, embodiments of the present invention include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program comprising program code configured to perform the methods described above.

[0108] In the context of this disclosure, a machine-readable medium can be any tangible medium that contains or can store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any preferred combination thereof. More specific examples of machine-readable storage media include one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or electrical connections having any preferred combination thereof.

[0109] Computer program code for carrying out the method of the present invention may be written in any combination of one or more programming languages. These computer program codes may be provided to the processor of a general-purpose computer, a dedicated computer, or other programmable data processing device, and when the program code is executed by the computer processor or other programmable data processing device, it causes the functions / operations specified in the flowchart and / or block diagram to be performed. The program code may be executed on a computer, partially on a computer, as a standalone software package, partially on a computer, partially on a remote computer, or entirely on a remote computer or server.

[0110] Furthermore, although the operations are described in a specific order, this should not be understood as requiring that such operations be performed in a specific illustrated order or sequentially, or that all illustrated operations be performed to achieve a desired result. In certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limitations on the scope of any invention or claim, but rather as descriptions of features that may be specific to specific embodiments of a particular invention. Certain features described in this specification in the context of separate embodiments may be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may be implemented separately or in any preferred subcombination in multiple embodiments.

[0111] It should be noted that the specification and drawings merely illustrate the principles of the proposed method and apparatus. Therefore, those skilled in the art will understand that various configurations embodying the principles of the present invention and falling within its spirit and scope can be devised, even if not explicitly described or illustrated herein. Furthermore, all examples described herein are primarily intended solely for educational purposes, to assist the reader in understanding the principles of the proposed method and apparatus, and the concepts to which the inventors have contributed to advance the art, and should be interpreted without limitation to the examples and conditions thus specifically described. Moreover, all statements herein describing the principles, aspects, and embodiments of the present invention, and specific examples thereof, are intended to encompass their equivalents.

[0112] Several aspects are described below. [Aspect 1] A method (500) for generating a bitstream (101), wherein the bitstream (101) includes a sequence of superframes (400) for a sequence of frames of an immersive audio signal (111), and the method (500) repeats the superframes (400) for the sequence: Step (501) of inserting encoded audio data (206) for one or more frames of one or more downmixed channel signals (203) derived from the immersive audio signal (111) into the data fields (411, 421, 412, 422) of the superframe (400); The process includes the step (502) of inserting metadata (202, 205) for reconstructing one or more frames of the immersive audio signal (111) from the encoded audio data (206) into the metadata field (403) of the superframe (400), method. [Aspect 2] The method (500) includes inserting a header field (401) into the superframe (400); The header field (401) indicates the size of the metadata field (403) of the superframe (400). The method described in Embodiment 1. [Aspect 3] The metadata field (403) indicates the maximum possible size; The header field (401) above indicates the adjustment value; The size of the metadata field (403) of the superframe (400) corresponds to the maximum possible size minus the adjustment value. The method described in Embodiment 2. [Aspect 4] The header field (401) includes a size indicator for the metadata field (403); The size indicator shows different resolutions for different size ranges of the metadata field (403). The method according to embodiment 2 or 3. [Aspect 5] The metadata (202,205) for reconstructing one or more frames of the immersive audio signal (111) shows a statistical size distribution of the size of the metadata (202,205); The resolution of the size indicator depends on the size distribution of the metadata (202,205). The method according to aspect 4. [Aspect 6] The method (500) includes inserting a header field (401) into the superframe (400); The header field (401) indicates whether the superframe (400) includes the configuration information field (402); The header field (401) indicates the presence of the configuration information field (402). The method described in any one of the embodiments 1 to 5. [Aspect 7] The method (500) includes inserting the configuration information field (402) into the superframe (400); The configuration information field (402) indicates the number of downmix channel signals (203) represented by the data fields (411, 421, 412, 422) of the superframe (400). The method according to any one of the embodiments 1 to 6. [Aspect 8] The method (500) includes inserting the configuration information field (402) into the superframe (400); The configuration information field (402) indicates the maximum possible size of the metadata field (403). The method described in any one of the embodiments 1 to 7. [Aspect 9] The method (500) includes inserting the configuration information field (402) into the superframe (400); The configuration information field (402) indicates the order of the sound field representation signal contained within the immersive audio signal (111). The method described in any one of embodiments 1 to 8. [Aspect 10] The method (500) includes inserting the configuration information field (402) into the superframe (400); The configuration information field (402) indicates the frame type and / or encoding mode used to encode each of the one or more downmix channel signals (203). The method according to any one of the embodiments 1 to 9. [Aspect 11] The method (500) includes inserting a header field (401) into the superframe (400); The header field (401) indicates whether the superframe (400) includes an extension field (404) for additional information relating to the immersive audio signal (111). The method according to any one of embodiments 1 to 10. [Aspect 12] The method according to any one of embodiments 1 to 3, wherein the superframe (400) includes two or more frames of the one or more downmix channel signals (203). [Aspect 13] The encoded audio data (206) of the frame of the downmix channel signal (203) is generated using a multimode and / or multirate utterance or audio codec; and / or The metadata (202,205) is generated using a multi-mode and / or multi-rate immersive metadata encoding scheme. The method according to any one of the embodiments 1 to 12. [Aspect 14] The method according to any one of embodiments 1 to 13, wherein the encoded audio data (206) of the frame of the downmix channel signal (203) is encoded using an enhanced audio service encoder. [Aspect 15] The method according to any one of embodiments 1 to 14, wherein the superframe (400) constitutes at least a portion of data elements that are transmitted using a transmission protocol, in particular DASH, RTSP, or RTP, or stored in a file according to a storage format, in particular ISOBMFF. [Aspect 16] The header field (401) indicates that the configuration information field (402) does not exist; The method (500) includes transmitting configuration information in a previous superframe (400) in the sequence of superframes (400), or using an out-of-band signaling method. The method described in any one of the embodiments 1 to 15. [Aspect 17] The said method, - A step of inserting encoded audio data (206) for one or more frames of a first downmix channel signal (203) and a second downmix channel signal (203) derived from the immersive audio signal (111) into one or more first data fields (411, 421) and one or more second data fields (412, 422) of the superframe (400), wherein the first downmix channel signal (203) is encoded using a first encoder and the second downmix channel signal (203) is encoded using a second encoder; The steps include providing configuration information relating to the first encoder and the second encoder within the superframe (400), within a previous superframe (400) in the sequence of the superframe (400), or using an out-of-band signaling scheme. The method described in any one of embodiments 1 to 16. [Aspect 18] The said method, A step of extracting one or more audio objects from the immersive audio signal (111), called an IA signal, wherein the audio object includes an object signal and object metadata (202) indicating the location of the audio object; The steps include: determining the residual signal (201) based on the IA signal (111) and based on one or more audio objects; - A downmix signal is provided based on the IA signal (111), in particular, such that the number of downmix channel signals (203) in the downmix signal is less than the number of channel signals in the IA signal (111); - A step of determining joint coded metadata (205) to enable upmixing the downmix signal into one or more reconfigured audio object signals corresponding to the one or more audio objects and / or reconfigured residual signals (311) corresponding to the residual signal (201); The steps include: performing waveform coding of the downmix signal to provide coded audio data (206) for a sequence of frames of the one or more downmix channel signals (203); The process includes the step of performing entropy coding on the joint coded metadata (205) and the object metadata (202) of one or more audio objects to provide the metadata (202, 205) to be inserted into the metadata field (403) of the sequence of superframes (400), The method described in any one of the embodiments 1 to 17. [Aspect 19] A superframe (400) of a bitstream (101), wherein the bitstream (101) includes a sequence of superframes (400) for a sequence of frames of an immersive audio signal (111), and the superframe (400) is - Data fields (411, 421, 412, 422) of encoded audio data (206) for one or more frames of one or more downmixed channel signals (203) derived from the immersive audio signal (111); - A single metadata field (403) for metadata (202,205) adapted to reconstruct one or more frames of the immersive audio signal (111) from the encoded audio data (206), Superframe. [Aspect 20] A method (600) for deriving data relating to an immersive audio signal (111) from a bitstream (101), wherein the bitstream (101) includes a sequence of superframes (400) relating to a sequence of frames of the immersive audio signal (111), and the method (600) iterates over the sequence of superframes (400). Step (601) of extracting encoded audio data (206) from the data fields (411, 421, 412, 422) of the superframe (400) for one or more frames of one or more downmix channel signals (203) derived from the immersive audio signal (111); - The process includes performing the step (602) of extracting metadata (202, 205) from the metadata field (403) of the superframe (400) for reconstructing one or more frames of the immersive audio signal (111) from the encoded audio data (206), method. [Aspect 21] The step of deriving one or more reconstructed audio objects from the encoded audio data (206) and the metadata (202, 205), wherein the audio object includes an object signal and object metadata (202) indicating the location of the audio object; - A step of deriving a reconstructed residual signal (311) from the encoded audio data (206) and the metadata (202, 205), wherein one or more reconstructed audio objects and the reconstructed residual signal (311) describe the immersive audio signal (111). The method according to embodiment 20. [Aspect 22] This method is - The step of extracting the header field (401) from the superframe (400); The step of deriving the size of the metadata field (403) of the superframe (400) from the header field (401) is included. The method according to embodiment 20 or 21. [Aspect 23] The metadata field (403) indicates the maximum possible size; The header field (401) indicates the adjustment value; The size of the metadata field (403) of the superframe (400) corresponds to the maximum possible size minus the adjustment value. The method described in Embodiment 22. [Aspect 24] The header field (401) includes a size indicator for the size of the metadata field (403); The size indicator shows different resolutions for different size ranges of the metadata field (403). The method according to embodiment 22 or 23. [Aspect 25] The said method, - The step of extracting the header field (401) from the superframe (400); - A step of determining whether the superframe (400) includes a configuration information field (402) based on the header field (401); The step includes determining whether a configuration information field (402) exists within the superframe (400) based on the header field (401), The method according to any one of embodiments 20 to 24. [Aspect 26] The said method, The step of extracting the configuration information field (402) from the superframe (400); The step of determining the number of downmix channel signals (203) represented by the data fields (411, 421, 412, 422) of the superframe (400) based on the configuration information field (402), The method according to any one of embodiments 20 to 25. [Aspect 27] The said method, The step of extracting the configuration information field (402) from the superframe (400); The step includes determining the maximum possible size of the metadata field (403) based on the configuration information field (402), The method according to any one of embodiments 20 to 26. [Aspect 28] The said method, The step of extracting the configuration information field (402) from the superframe (400); The steps include determining the order of the sound field representation signal contained in the immersive audio signal (111) based on the configuration information field (402), The method according to any one of embodiments 20 to 27. [Aspect 29] The said method, The step of extracting the configuration information field (402) from the superframe (400); The steps include determining, based on the configuration information field (402), the frame type and / or encoding mode used to encode each of the one or more downmix channel signals (203), The method according to any one of embodiments 20 to 28. [Aspect 30] The said method, - The step of extracting the header field (401) from the superframe (400); The step includes determining, based on the header field (401), whether the superframe (400) includes an extension field (404) for additional information relating to the immersive audio signal (111), The method according to any one of embodiments 20 to 29. [Aspect 31] An encoding device (110) configured to generate a bitstream (101), the bitstream (101) comprising a sequence of superframes (400) of a sequence of frames of an immersive audio signal (111), wherein the encoding device (110) repeatedly: The steps include: inserting encoded audio data (206) for one or more frames of one or more downmix channel signals (203) derived from the immersive audio signal (111) into the data fields (411, 421, 412, 422) of the superframe (400); The system is configured to perform the steps of inserting metadata (202,205) for reconstructing one or more frames of the immersive audio signal (111) from the encoded audio data (206) into the metadata field (403) of the superframe (400), Encoding device. [Aspect 32] A decoding device (120) configured to derive data relating to an immersive audio signal (111) from a bitstream (101), wherein the bitstream (101) includes a sequence of superframes (400) relating to a sequence of frames of the immersive audio signal (111), and the decoding device (120) repeatedly processes the sequence of superframes (400) - A step of extracting encoded audio data (206) for one or more frames of one or more downmix channel signals (203) derived from the immersive audio signal from the data fields (411, 421, 412, 422) of the superframe (400); The system is configured to perform the steps of: extracting metadata (202, 205) from the metadata field (403) of the superframe (400) for reconstructing one or more frames of the immersive audio signal (111) from the encoded audio data (206); Decoding device.

Claims

1. A method (500) for generating a bitstream (101), wherein the bitstream (101) includes a sequence of superframes (400) for a sequence of frames of an immersive audio signal (111), and the method (500) repeatedly: Step (501) of inserting encoded audio data (206) for two or more frames of one or more downmix channel signals (203) derived from the immersive audio signal (111) into the data fields (411, 421, 412, 422) of the superframe (400), wherein the encoded audio data (206) for the frames of the downmix channel signal (203) is generated using multimode and / or multirate speech or audio codecs; The process includes the step (502) of inserting metadata (202,205) for reconstructing two or more frames of the immersive audio signal (111) from the encoded audio data (206) into the metadata field (403) of the superframe (400), method.

2. A non-temporary computer-readable medium storing instructions that cause an operation to be performed when executed by one or more processors, wherein the operation is: Steps include: inserting encoded audio data (206) for two or more frames of one or more downmix channel signals (203) derived from an immersive audio signal (111) into the data fields (411, 421, 412, 422) of a superframe (400), wherein the encoded audio data (206) for the frames of the downmix channel signals (203) is generated using multimode and / or multirate speech or audio codecs; The process includes inserting metadata (202,205) into the metadata field (403) of the superframe (400) for reconstructing two or more frames of the immersive audio signal (111) from the encoded audio data (206), A non-temporary computer-readable medium.

3. An encoding device (110) configured to generate a bitstream (101), the bitstream (101) comprising a sequence of superframes (400) of a sequence of frames of an immersive audio signal (111), wherein the encoding device (110) repeatedly: Steps include: inserting encoded audio data (206) for two or more frames of one or more downmix channel signals (203) derived from the immersive audio signal (111) into the data fields (411, 421, 412, 422) of the superframe (400), wherein the encoded audio data (206) for the frames of the downmix channel signal (203) is generated using multimode and / or multirate speech or audio codecs; The system is configured to perform the steps of inserting metadata (202,205) for reconstructing two or more frames of the immersive audio signal (111) from the encoded audio data (206) into the metadata field (403) of the superframe (400), Encoding device.

Images