IVAS bitstream encoding and decoding

Abstract

Encoding / decoding an Immersive Audio Services (IVAS) bitstream includes encoding / decoding a coding mode indicator in a common header (CH) section of the IVAS bitstream, encoding / decoding a mode header or a tool header in a tool header (TH) section of the bitstream, where the TH section follows the CH section, encoding / decoding a metadata payload in a metadata payload (MDP) section of the bitstream, where the MDP section follows the CH section, and encoding / decoding an Enhanced Voice Services (EVS) payload in an EVS payload (EP) section of the bitstream, where the EP section follows the CH section, at an encoder side, storing or streaming the encoded bitstream, and at a decoder side, controlling an audio decoder based on the coding mode, the tool header, the EVS payload, and the metadata payload or storing a representation thereof.

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims priority to U.S. Provisional Application No. 62 / 881,541, filed August 1, 2019, U.S. Provisional Application No. 62 / 927,894, filed October 30, 2019, U.S. Provisional Application No. 63 / 037,721, filed June 11, 2020, and U.S. Provisional Application No. 63 / 057,666, filed July 28, 2020. The entire disclosures of these U.S. provisional applications are incorporated herein by reference.

[0002] This disclosure relates generally to encoding and decoding audio bitstreams. [Background technology]

[0003] Audio and video encoder / decoder ("codec") standards development has focused in recent years on the development of immersive voice and audio service (IVAS) codecs. IVAS is expected to support a variety of audio service capabilities, including, but not limited to, mono-to-stereo upmixing and full immersive audio encoding, decoding, and rendering. IVAS is intended to be supported by a wide range of devices, endpoints, and network nodes, including, but not limited to, mobile and smart phones, electronic tablets, personal computers, conference phones, conference rooms, virtual reality (VR) and augmented reality (AR) devices, home theater devices, and other suitable devices. These devices, endpoints, and network nodes may have a variety of acoustic interfaces for sound capture and rendering. Summary of the Invention

[0004] An embodiment for encoding and decoding an IVAS bitstream is disclosed.

[0005] In some embodiments, a method for generating a bitstream of an audio signal includes determining, using an Immersive Audio Services (IVAS) encoder, a coding mode indicator or a coding tool indicator, the coding mode indicator or the coding tool indicator indicating a coding mode or a coding tool of the audio signal; encoding, using the IVAS encoder, the coding mode indicator or the coding tool indicator in a common header (CH) section of an IVAS bitstream; determining, using the IVAS encoder, a mode header or a tool header; and encoding, using the IVAS encoder, the mode header or the tool header in a tool header (TH) section of the IVAS bitstream. using the IVAS encoder to determine a metadata payload including spatial metadata, the TH section following the CH section, and encoding the metadata payload using the IVAS encoder into a metadata payload (MDP) section of the IVAS bitstream, the MDP section following the CH section; using the IVAS encoder to determine an Enhanced Voice Services (EVS) payload, the EVS payload including EVS encoded bits for each channel or each downmix channel of the audio signal, and encoding the EVS payload using the IVAS encoder into an EVS payload (EP) section of the IVAS bitstream, the EP section following the CH section.

[0006] In some embodiments, the IVAS bitstream is stored on a non-transitory computer-readable medium. In other embodiments, the IVAS bitstream is streamed to a downstream device, and the encoding mode or encoding tool indicator, the mode or tool header, the metadata payload, and the EVS payload are extracted and decoded from the CH section, the TH section, the MDP section, and the EP section of the IVAS bitstream, respectively, for use in reconstructing the audio signal in the downstream device or another device.

[0007] In some embodiments, a method of decoding a bitstream of an audio signal includes using an Immersive Audio Services (IVAS) decoder to extract and decode a coding mode indicator or a coding tool indicator in a common header (CH) section of an IVAS bitstream, the coding mode indicator or the coding tool indicator indicating a coding mode or coding tool of the audio signal, and using the IVAS decoder to extract and decode a mode header or a tool header in a tool header (TH) section of the IVAS bitstream, the TH section indicating a coding mode or coding tool of the audio signal. using the IVAS decoder to extract and decode a metadata payload from a metadata payload (MDP) section of the IVAS bitstream, the MDP section following the CH section, the metadata payload including spatial metadata; and using the IVAS decoder to extract and decode an Enhanced Voice Services (EVS) payload from an EVS payload (EP) section of the IVAS bitstream, the EP section following the CH section, the EVS payload including EVS encoded bits for each channel or each downmix channel of the audio signal.

[0008] In some embodiments, an audio decoder of a downstream device for use in reconstructing the audio signal in the downstream device or another device is controlled based on the encoding mode indicator or the encoding tool indicator, the mode header or the tool header, the EVS payload, and the metadata payload. In other embodiments, a representation of the encoding mode indicator or the encoding tool indicator, the mode header or the tool header, the EVS payload, and the metadata payload is stored on a non-transitory computer-readable medium.

[0009] In some embodiments, the bitrate of each EVS encoded channel or each downmix channel is determined by the total available bits of the EVS, the SPAR bitrate distribution control table and a bitrate distribution algorithm.

[0010] In some embodiments, the CH is a multi-bit data structure, one value of the multi-bit data structure corresponds to a spatial reconstruction (SPAR) coding mode and another value of the data structure corresponds to another coding mode.

[0011] In some embodiments, the method further includes storing or reading, respectively, an index offset in the TH section of the IVAS bitstream for calculating a row index of a spatial reconstruction (SPAR) bitrate distribution control table.

[0012] In some embodiments, the method further includes storing or reading, respectively, a quantization strategy indicator, a bitstream encoding strategy indicator, and the quantized and encoded real and imaginary parts of the set of coefficients in or from the MDP section of the IVAS bitstream.

[0013] In some embodiments, the set of coefficients includes predicted coefficients, direct coefficients, diagonal real coefficients and lower triangular complex coefficients.

[0014] In some embodiments, the prediction coefficients are variable bit length based on entropy coding, and the direct coefficients, the diagonal real coefficients and the lower triangular complex coefficients are variable bit length based on downmix construction and entropy coding.

[0015] In some embodiments, the quantization strategy indicator is a multi-bit data structure that indicates a quantization strategy.

[0016] In some embodiments, the bitstream encoding strategy indicator is a multi-bit data structure indicating the number of bands and the non-differential entropy encoding scheme or the time-differential entropy encoding scheme of the spatial metadata.

[0017] In some embodiments, the quantization of the coefficients is according to an EVS bitrate distribution control strategy that includes metadata quantization and EVS bitrate distribution.

[0018] In some embodiments, the above method further includes storing or reading an EVS payload of an EVS instance in the EP section of the IVAS bitstream, respectively, in accordance with 3rd Generation Partnership Project (3GPP) Technical Specification (TS) 26.445.

[0019] In some embodiments, the method includes: determining a bitrate from the IVAS bitstream; reading an index offset from a spatial reconstruction (SPAR) tool header (TH) section of the IVAS bitstream; determining a table row index for the SPAR bitrate distribution control table using the index offset; reading quantization strategy bits and encoding strategy bits from a metadata payload (MDP) section in the IVAS bitstream; and determining a bitrate from the IVAS bitstream based on the quantization strategy bits and the encoding strategy bits. and determining an Enhanced Voice Services (EVS) bitrate for each channel in the IVAS bitstream using all available EVS bits and a SPAR bitrate distribution control table; reading EVS encoded bits from the EP section of the IVAS bitstream based on the EVS bitrates; decoding the EVS bits; decoding the spatial metadata; and generating a first order Ambisonics (FoA) output using the decoded EVS bits and the decoded spatial metadata.

[0020] Other embodiments disclosed herein relate to systems, devices, and computer-readable media. Details of the disclosed embodiments are set forth in the accompanying drawings and the following description. Other features, objects, and advantages will be apparent from the following description, drawings, and claims.

[0021] Certain implementations disclosed herein provide one or more of the following advantages: The disclosed IVAS bitstream format is an efficient and robust bitstream format that supports a variety of audio service capabilities, including, but not limited to, mono-to-stereo upmix and fully immersive audio encoding, decoding and rendering. In some implementations, the IVAS bitstream format supports complex advance coupling (CACPL) for analyzing and downmixing stereo audio signals. In other implementations, the IVAS bitstream format supports spatial reconstruction (SPAR) for analyzing and downmixing first order Ambisonics (FoA) audio signals.

[0022] In the figures, a particular arrangement or ordering of graphical elements, such as elements representing devices, units, instruction blocks, and data elements, is shown for ease of explanation. However, it should be understood by those skilled in the art that the particular ordering or arrangement of these graphical elements in the figures does not imply that any particular order or sequence of processing is required, nor that separation of processes is required. Furthermore, the inclusion of a graphical element in a drawing does not imply that such element is required in all embodiments, nor does it imply that features represented by such element cannot be included in or combined with other elements in some embodiments.

[0023] Furthermore, in the drawings, when a connecting element such as a solid or dashed line or an arrow is used to indicate a connection, relationship, or association between two or more other diagrammatic elements, it is not intended to imply that the connection, relationship, or association may not exist in the absence of such connecting element. In other words, some connections, relationships, or associations between elements are not shown in the drawings so as not to obscure the present disclosure. In addition, for ease of illustration, a single connecting element is used to represent multiple connections, relationships, or associations between elements. For example, when a connecting element represents communication of signals, data, or instructions, it should be understood by those skilled in the art that such an element represents one or more signal paths for implementing the communication, as appropriate. [Brief description of the drawings]

[0024] [Figure 1] FIG. 1 illustrates an IVAS system according to one embodiment.

[0025] [Diagram 2] FIG. 1 is a block diagram of a system for encoding and decoding an IVAS bitstream, according to one embodiment.

[0026] [Diagram 3] A block diagram of a FoA coder / decoder ("codec") that encodes and decodes an IVAS bitstream in FoA format according to one embodiment.

[0027] [Figure 4A] FIG. 1 is a flow diagram of an IVAS encoding process according to one embodiment.

[0028] [Figure 4B] FIG. 1 is a flow diagram of an IVAS encoding process using an alternative IVAS format, according to one embodiment.

[0029] [Figure 5A] FIG. 1 is a flow diagram of an IVAS decoding process according to one embodiment.

[0030] [Figure 5B] FIG. 13 is a flow diagram of an IVAS decoding process using an alternative IVAS format, according to one embodiment.

[0031] [Figure 6] FIG. 1 is a flow diagram of an IVAS SPAR encoding process according to one embodiment.

[0032] [Figure 7] FIG. 1 is a flow diagram of an IVAS SPAR decoding process according to one embodiment.

[0033] [Figure 8] FIG. 2 is a block diagram of an exemplary device architecture according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0034] The use of the same reference numbers in the various drawings indicates similar elements.

[0035] In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments described. It will be apparent to those skilled in the art that the various embodiments described may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments. Several features are described below that can each be used independently or in some combination with other features.

[0036] nomenclature As used herein, the terms "include" and variations thereof should be interpreted as open-ended terms meaning "including, but not limited to." The term "or" should be interpreted as "and / or" unless the context clearly indicates otherwise. The term "based on" should be interpreted as "based at least in part on." The terms "one exemplary embodiment" and "an exemplary embodiment" should be interpreted as "at least one exemplary embodiment." The term "another embodiment" should be interpreted as "at least one other embodiment." The terms "determined," "determines," "determining" should be interpreted as "obtain," "receive," "calculate," "calculate," "estimate," "predict," or "derive." Additionally, in the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0037] Overview of the IVAS system FIG. 1 illustrates an IVAS system 100 according to one or more embodiments. In some embodiments, various devices communicate through a call server 102 configured to receive audio signals from public switched telephone network (PSTN) devices or public land mobile network (PLMN) devices, as indicated, for example, by PSTN / other PLMN 104. The IVAS system 100 supports legacy devices 106 that render and capture audio only mono, including but not limited to devices that support enhanced voice service (EVS), multi-rate wideband (AMR-WB), and adaptive multi-rate narrowband (AMR-NB). The IVAS system 100 also supports user equipment (UE) 108, 114 that captures and renders stereo audio signals, or a UE 110 that captures mono signals and renders them binaurally into a multi-channel signal. The IVAS system 100 also supports immersive and stereo signals to be captured and rendered by video conferencing room systems 116, 118, respectively. The IVAS system 100 also supports stereo capture and immersive rendering of stereo audio signals for home theater systems, as well as mono capture and immersive rendering of audio signals for virtual reality (VR) gear 122 and immersive content ingest 124.

[0038] Example IVAS Encoding / Decoding System 2 is a block diagram of a system 200 for encoding and decoding an IVAS bitstream, according to one or more embodiments. To perform the encoding, the IVAS encoder includes a spatial analysis downmix unit 202 that receives audio data 201. This audio data includes, but is not limited to, mono signals, stereo signals, binaural signals, spatial audio signals (e.g., multi-channel spatial audio objects), FoA, higher order Ambisonics (HoA), and any other audio data. In some embodiments, the spatial analysis downmix unit 202 implements CACPL to analyze / downmix stereo audio signals and / or SPAR to analyze / downmix FoA audio signals. In other embodiments, the spatial analysis downmix unit 202 implements other formats.

[0039] The output of the spatial analysis downmix unit 202 includes spatial metadata and 1-4 channels of audio. The spatial metadata is input to a quantization and entropy coding unit 203, which quantizes and entropy codes the spatial data. In some implementations, the quantization can include a fine quantization strategy, a medium quantization strategy, a coarse quantization strategy, and a very coarse quantization strategy, and the entropy coding can include Huffman coding or arithmetic coding. The Enhanced Voice Services (EVS) coding unit 206 encodes the 1-4 channels of audio into one or more EVS bitstreams.

[0040] In some embodiments, the EVS encoding unit 206 complies with 3GPP TS26.445 and provides a wide range of features, such as improved quality and coding efficiency for narrowband (EVS-NB) and wideband (EVS-WB) voice services, improved quality using ultra-wideband (EVS-SWB) voice, improved quality of mixed content and music in conversational applications, robustness to packet loss and delay jitter, and backward compatibility to AMR-WB codec. In some embodiments, the EVS encoding unit 206 includes a pre-processing mode selection unit that selects either a speech coder that encodes the speech signal and a perceptual coder that encodes the audio signal at a specified bit rate based on the mode / bitrate control unit 207. In some embodiments, the speech encoder is an improved variant of algebraic code-excited linear prediction (ACELP) with specialized LP-based modes for different speech classes. In some embodiments, the audio encoder is a modified discrete cosine transform (MDCT) encoder for enhanced efficiency at low delay / low bitrate and designed to provide seamless and reliable switching between the speech encoder and the audio encoder.

[0041] In some embodiments, the IVAS decoder includes a quantization entropy decoding unit 204 configured to recover the spatial metadata and an EVS decoder(s) configured to recover the 1-4 channel audio signals. The recovered spatial metadata and audio signals are input to a spatial synthesis / rendering unit 209, which synthesizes / renders the audio signals using the spatial metadata for playback on various audio systems 210.

[0042] Example IVAS / SPAR codec FIG. 3 is a block diagram of a FoA codec 300 for encoding and decoding FoA in SPAR format according to some embodiments. The FoA codec 300 includes a SPAR FoA encoder 301, an EVS encoder 305, a SPAR FoA decoder 306, and an EVS decoder 307. The FoA codec 300 converts the FoA input signal into a set of downmix channels and parameters that are used to regenerate the input signal in the decoders 306, 307. The downmix signal can vary between one channel and four channels, and the parameters include prediction coefficients (PR), interprediction coefficients (C), and decorrelation coefficients (P). It should be noted that SPAR is a process used to reconstruct an audio signal from a downmix of the audio signal using PR, C, and P parameters, as described in more detail below.

[0043] Note that the exemplary implementation shown in Figure 3 assumes a passive W channel and depicts a nominal two-channel downmix in which the W channel is sent unchanged to the decoder 306 along with a single prediction channel Y'. In other implementations, W can be an active channel. An active W channel allows some mixing of the X, Y, and Z channels into the W channel as follows:

number

[0044] As will be explained in more detail below, the C coefficients allow the X and some of the Z channels to be reconstructed from Y', while the remaining channels are reconstructed by a decorrelated version of the W channel, as will be explained in more detail below.

[0045] In some embodiments, the SPAR FoA Encoder 301 includes a Passive / Active Predictor unit 302, a Remix unit 303, and an Extraction / Downmix Selection unit 304. The Passive / Active Predictor receives the FoA channels in 4-channel B-format (W, Y, Z, X) and calculates the prediction channel (W or W', Y', Z', X'). Note that the W channel is an omnidirectional polar pattern that includes all sounds in a sphere coming from all directions with equal gain and phase, X is a figure-8 bidirectional polar pattern pointing forward, Y is a figure-8 bidirectional polar pattern pointing left, and Z is a figure-8 bidirectional polar pattern pointing up.

[0046] The extraction / downmix selection unit 304 extracts the SPAR FoA metadata from the metadata payload section of the IVAS bitstream, as described in more detail below. The passive / active predictor unit 302 and the remix unit 303 use the SPAR FoA metadata to generate remixed FoA channels (W or W', A', B', C'), which are input to the EVS encoder 305 to be encoded into an EVS bitstream, which is encapsulated within the IVAS bitstream sent to the decoder 306. Note that in this example, the Ambisonic B format channels are placed in the AmbiX format, although other formats such as the Furse-Malham (FuMa) format (W, X, Y, Z) can be used as well.

[0047] Referring to the SPAR FoA decoder 306, the EVS bitstream is decoded by the EVS decoder 307, resulting in N (e.g., N=4) downmix channels. In some embodiments, the SPAR FoA decoder 306 performs the inverse of the operations performed by the SPAR encoder 301. For example, the remixed FoA channels (W or W', A', B', C') are recovered from the N downmix channels using the SPAR FoA spatial metadata. The remixed SPAR FoA channels are input to the inverse mixer 311, which recovers the predicted SPAR FoA channels (W or W', Y', Z', X'). The predicted SPAR FoA channels are then input to the inverse predictor 312, which recovers the original unmixed SPAR FoA channels (W, Y, Z, X). In this two-channel example, the decorrelator blocks 309a (dec1)...309n (dec D Note that the sparse-spaced ensemble (SPAR) is used to generate a decorrelated version of the W channel using a time-domain or frequency-domain decorrelator. The decorrelated channel is then used in combination with the SPAR FoA metadata to fully or parametrically reconstruct the X and Z channels.

[0048] In some embodiments, depending on the number of downmix channels, one of the FoA inputs (W channel) is sent intact to the SPAR FoA decoder 306, and one to three of the other channels (Y, Z, and X) are sent to the SPAR FoA decoder 306 as residuals or fully parametrically. The PR coefficients, which remain the same regardless of the number of downmix channels N, are used to minimize the predictable energy in the residual downmix channel. The C coefficients are used to further help regenerate the fully parameterized channel from the residual. Thus, the C coefficients are not needed in the one and four channel downmix cases where there are no residual or parameterized channels to predict. The P coefficients are used to fill in the remaining energy not accounted for by the PR and C coefficients. The number of P coefficients depends on the number N of downmix channels in each band. In some embodiments, the SPAR PR coefficients (passive W only) are calculated as follows:

[0049] Step 1. Predict all side signals (Y, Z, X) from the main W signal using equation [1].

number

number

[0050] Step 2. Remix the W signal with the predicted (Y', Z', X') signals (in that order from most acoustically related to least acoustically related), where "remix" means rearranged or recombined signals based on some methodology.

number

[0051] One implementation of remix is ​​a reordering of the input signal into W, Y', X', Z', assuming that audio cues from left and right are more acoustically relevant than front and back, and front and back cues are more acoustically relevant than top and bottom cues.

[0052] Step 3. Compute the covariance of the 4-channel posterior prediction and remix and downmix as shown in equations [4] and [5].

number

number

[0053] As an example of a WABC downmix with channels 1-4, d and u represent the following channels as shown in Table I: [Table 1]

[0054] The main targets for SPAR FoA metadata calculation are R dd quantity, R ud Amount and R uu The quantity is Rdd quantity, R ud Amount and R uu From this quantity, the system determines whether it is possible to inter-predict the remaining portion of the fully parametric channel from the residual channel that is transmitted to the decoder. In some implementations, the required extra C coefficients are given by the following equation:

number

[0055] Thus, the C parameter has the shape (1×2) for a 3-channel downmix and (2×1) for a 2-channel downmix.

[0056] Step 4. Calculate the residual energy in the parameterized channels that must be reconstructed by the decorrelator. uu is the actual energy R uu (posterior prediction) and the regenerated mutual prediction energy Reg uu is the difference between

number

number

number

[0057] P is also a covariance matrix and therefore Hermitian symmetric, so only parameters from the upper or lower triangle need to be transmitted to the decoder 306. The diagonal entries are real numbers, while the off-diagonal elements may be complex numbers.

[0058] Example encoding / decoding of IVAS bitstream As described with reference to Figures 2 and 3, the IVAS bitstream(s) are encoded and decoded by the IVAS codec. In some embodiments, the IVAS encoder determines and encodes a coding tool indicator and a sampling rate indicator in a common header (CH) section of the IVAS bitstream. In some embodiments, the coding tool indicator comprises a value corresponding to the coding tool, and the sampling rate indicator comprises a value indicating the sampling rate. The IVAS encoder determines and encodes an EVS payload in an EVS payload (EP) section of the bitstream. The EP section follows the CH section. The IVAS encoder determines and encodes a metadata payload in a metadata payload (MDP) section of the bitstream. In some embodiments, the MDP section follows the CH section. In other embodiments, the MDP section follows the EP section of the bitstream, or the EP section follows the MDP section of the bitstream. In some embodiments, the IVAS encoder stores the bitstream on a non-transitory computer-readable medium or streams the bitstream to a downstream device. In another embodiment, the IVAS encoder has the device architecture shown in FIG.

[0059] In some embodiments, the IVAS decoder receives an IVAS bitstream and extracts and decodes audio data encoded in the IVAS format by the IVAS encoder. The IVAS decoder extracts and decodes an encoding tool indicator and a sampling rate indicator in the CH section of the IVAS bitstream. The IVAS decoder extracts and decodes an EVS payload in the EP section of the bitstream. The EP section follows the CH section. The IVAS decoder extracts and decodes a metadata payload in the MDP section of the bitstream. The MDP section follows the CH section. In other embodiments, the MDP section follows the EP section of the bitstream or the EP section follows the MDP section of the bitstream. In some embodiments, the IVAS system controls the audio decoder based on the encoding tool, the sampling rate, the EVS payload, and the metadata payload. In other embodiments, the IVAS system stores a representation of the encoding tool, the sampling rate, the EVS payload, and the metadata payload on a non-transitory computer-readable medium. In some embodiments, the IVAS decoder has a device architecture shown in FIG. 8.

[0060] In some embodiments, the IVAS coding tool indicator is a multi-bit data structure. In other embodiments, the IVAS coding tool indicator is a 3-bit data structure, where a first value of the 3-bit data structure corresponds to a multi-mono coding tool, a second value of the 3-bit data structure corresponds to a CACPL coding tool, and a third value of the 3-bit data structure corresponds to another coding tool. In other embodiments, the IVAS coding tool indicator is a 2-bit data structure indicating one to four IVAS coding tools or a 1-bit data structure indicating one or two IVAS coding tools. In other embodiments, the IVAS coding tool indicator includes more than two bits to indicate various IVAS coding tools.

[0061] In some embodiments, the input sampling rate indicator is a multi-bit data structure indicating various input sampling rates. In some embodiments, the input sampling rate indicator is a 2-bit data structure, where a first value of the 2-bit data structure indicates an 8 kHz sampling rate, a second value of the 2-bit data structure indicates a 16 kHz sampling rate, a third value of the 2-bit data structure indicates a 32 kHz sampling rate, and a fourth value of the 2-bit data structure indicates a 48 kHz sampling rate. In other embodiments, the input sampling rate indicator is a 1-bit data structure indicating one or two sampling rates. In other embodiments, the input sampling rate indicator includes three or more bits indicating various sampling rates.

[0062] In some embodiments, the system is 3rd Generation Partnership Project (3GPP) rd The number of EVS channels, i.e., an EVS channel number indicator; a bitrate (BR) extraction mode indicator; EVS BR data; and the EVS payload for all channels are stored in or read from the EP section of the bitstream, as described in the IEEE 802.11 generation partnership project (ESP) technical specification (TS) 26.445, in that order.

[0063] In another embodiment, the system stores or reads the EVS channel number indicator in the EP section of the bitstream.

[0064] In another embodiment, the system stores or reads a bitrate (BR) extraction mode indicator in or from the EP section of the bitstream.

[0065] In another embodiment, the system stores or reads the EVS BR data in the EP section of the bitstream.

[0066] In another embodiment, the system stores or reads the EVS payloads of all channels in the EP section of the bitstream, in that order, as described in 3rd Generation Partnership Project (3GPP) Technical Specification (TS) 26.445.

[0067] In some embodiments, the IVAS system stores or reads from the MDP section of the data stream a coding technique indicator; a number of bands indicator; an indicator indicative of the delay configuration of the filter bank; an indicator of the quantization strategy; an entropy coder indicator; a probability model type indicator; a real coefficient part; an imaginary coefficient part; and one or more coefficients.

[0068] In another embodiment, the IVAS system stores or reads the encoding technique indicator in the MDP section of the data stream.

[0069] In another embodiment, the IVAS system stores or reads the band number indicator in the MDP section of the data stream.

[0070] In another embodiment, the IVAS system stores or reads an indicator indicative of the delay configuration of the filter bank in the MDP section of the data stream.

[0071] In another embodiment, the IVAS system stores or reads an indicator of the quantization strategy in the MDP section of the data stream.

[0072] In another embodiment, the IVAS system stores or reads the entropy coder indicator in the MDP section of the data stream.

[0073] In another embodiment, the IVAS system stores or reads the probabilistic model type indicator in the MDP section of the data stream.

[0074] In another embodiment, the IVAS system stores or reads the real coefficient parts in or from the MDP section of the data stream.In another embodiment, the IVAS system stores or reads the imaginary coefficient parts in or from the MDP section of the data stream.

[0075] In other embodiments, the IVAS system stores or reads one or more coefficients in an MDP section of the data stream.

[0076] Some examples of IVAS bitstream formats are given below:

[0077] Example IVAS bitstream format-3 subformat In some embodiments, the IVAS bitstream format includes three subdivisions as follows: [Table 2]

[0078] In some embodiments, the parameters within each field in each subdivision and their respective bit allocations are shown below. [Table 3] [Table 4] [Table 5] JPEG0007605826000018.jpg49128

[0079] An advantage of the above-described IVAS bitstream format embodiment is that it efficiently and compactly encodes data supporting various audio service capabilities, including, but not limited to, mono to stereo upmix and fully immersive audio encoding, decoding and rendering. The embodiment is also supported by a wide range of devices, endpoints, and network nodes, including, but not limited to, mobile and smart phones, electronic tablets, personal computers, conference phones, conference rooms, virtual reality (VR) and augmented reality (AR) devices, home theater devices, and other suitable devices, each of which may have a variety of acoustic interfaces for sound capture and rendering. The IVAS bitstream format is extensible so that it can easily evolve with IVAS standards and technologies.

[0080] Example IVAS bitstream format-4 subformat The following description of a further embodiment focuses on the differences between this further embodiment and the previously described embodiment. Therefore, features common to both embodiments may be omitted from the following description, and in that case, it should be assumed that the features of the previously described embodiment are implemented or at least can be implemented in this further embodiment (unless the following description requires otherwise). In addition, when a feature is extracted from an embodiment disclosed below and added to a claim, the feature may not be related or closely related to other features of that embodiment.

[0081] In another embodiment, the IVAS bitstream includes four subdivisions as follows: [Table 6]

[0082] In some embodiments, the IVAS encoder determines and encodes a coding tool indicator in a common header (CH) section of the IVAS bitstream. The coding tool indicator comprises a value corresponding to the coding tool. The IVAS encoder determines and encodes a row index into an IVAS bitrate distribution control table in a common spatial coding tool header (CTH) section of the IVAS bitstream. The CTH section follows the CH section. The IVAS encoder determines and encodes an EVS payload in an EVS payload (EP) section of the IVAS bitstream. The EP section follows the CH section. The IVAS encoder determines and encodes a metadata payload in a metadata payload (MDP) section of the IVAS bitstream. The MDP section follows the CH section.

[0083] In some embodiments, the EP section is located before or after the MDP section depending on one or more parameters. In some embodiments, the one or more parameters include a backward compatible mode of mono downmix of a multi-channel input with a nominal bit rate mode, as described in 3GPP TS26.445.

[0084] In some embodiments, the IVAS system stores the IVAS bitstream on a non-transitory computer-readable medium. In other embodiments, the IVAS system streams the bitstream to a downstream device. In some embodiments, the IVAS encoder has the device architecture shown in FIG. 8.

[0085] In some embodiments, the IVAS decoder receives an IVAS bitstream and extracts and decodes audio data encoded in the IVAS format by the IVAS encoder. The IVAS decoder extracts and decodes an encoding tool indicator in a CH section of the IVAS bitstream. The IVAS decoder extracts and decodes an index to an IVAS bitrate distribution control table. The IVAS decoder extracts and decodes an EVS payload in an EP section of the IVAS bitstream, the EP section following the CH section. The IVAS decoder extracts and decodes metadata payload in an MDP section of the IVAS bitstream, the MDP section following the CH section.

[0086] In some embodiments, the EP section is located before or after the MDP section depending on one or more parameters. In some embodiments, the one or more parameters include a backward compatible mode of mono downmix of a multi-channel input with a nominal bit rate mode, as described in 3GPP TS26.445.

[0087] In some embodiments, the IVAS system controls the audio decoder based on the encoding tools, the index to the IVAS bitrate distribution control table, the EVS payload, and the metadata payload. In other embodiments, the IVAS system stores a representation of the encoding tools, the index to the IVAS bitrate distribution control table, the EVS payload, and the metadata payload on a non-transitory computer-readable medium. In some embodiments, the IVAS decoder has the device architecture shown in FIG. 8. [Table 7] [Table 8]

[0088] Metadata Payload (MDP): An advantage of the IVAS bitrate distribution control table is that it records information about the spatial coding modes so that it is not necessary to include this information in the MDP section. [Table 9]

[0089] EVS Payload (EP): This section of the payload contains EVS encoded bits for one or more audio downmix channels. In some embodiments, the total number of bits in this section is JPEG0007605826000023.jpg539, where N (e.g., N=4) is the number of audio downmix channels required to be encoded, and EVS BR(i) is the calculated EVS bitrate of the ith audio downmix channel, and stride secs is the input stride length in seconds.

[0090] In some embodiments, each table entry in the IVAS Bitrate Distribution Control Table has enough information to extract the bitrate of each EVS instance from the total bits allocated for the EVS. This structure provides the advantage that no additional header information is required in the EVS payload to extract the bits for each EVS instance. [Table 10]

[0091] In some embodiments, the parameters in the IVAS bitrate distribution control table have the following values: [Table 11]

[0092] An example IVAS bitrate distribution control table is as follows: [Table 12] JPEG0007605826000027.jpg192139JPEG0007605826000028.jpg189139JPEG0007605826000029.jpg171140

[0093] Example Decoding of IVAS Bitstream In one embodiment, the steps for decoding an IVAS bitstream are as follows.

[0094] Step 1: Bitstream length and stride secs Calculate the IVAS operating bit rate based on this.

[0095] Step 2: Read out a fixed length CH section that indicates the spatial coding tool.

[0096] Step 3: Based on the IVAS operating bit rate, determine the length of the CTH field by looking up the number of entries of the IVAS operating bit rate (calculated in step 1) in the IVAS bit rate distribution control table.

[0097] Step 3: Once the length of the CTH field is known, read the index offset within the CTH field.

[0098] Step 5: The index offset and the IVAS operating bit rate are used to determine the actual IVAS bit rate distribution control table index.

[0099] Step 6: Read all information about EVS bitrate distribution and mono downmix backward compatibility from the indexed table entry.

[0100] Step 7: If the mono downmix backward compatibility mode is ON, first pass the remaining IVAS bits to the EVS decoder, calculate the bit length of each EVS instance based on its EVS bitrate distribution, read the EVS bits of each EVS instance, decode the EVS bits using the corresponding EVS decoder, and decode the spatial metadata in the MDP section.

[0101] Step 8: If the mono downmix backward compatibility mode is OFF, decode the spatial metadata in the MDP section, calculate the bit length of each EVS instance based on its EVS bitrate distribution, and read and decode the EVS bits of each EVS instance from the EP section of the IVAS bitstream.

[0102] Step 9: Use the decoded EVS output and spatial metadata to configure the input audio format, e.g. stereo (CACPL) or FoA (SPAR), etc.

[0103] An advantage of the above-described IVAS bitstream format embodiment is that it efficiently and compactly encodes data supporting various audio service capabilities, including, but not limited to, mono to stereo upmix and fully immersive audio encoding, decoding and rendering. The embodiment is also supported by a wide range of devices, endpoints, and network nodes, including, but not limited to, mobile and smart phones, electronic tablets, personal computers, conference phones, conference rooms, virtual reality (VR) and augmented reality (AR) devices, home theater devices, and other suitable devices, each of which may have a variety of acoustic interfaces for sound capture and rendering. The IVAS bitstream format is extensible so that it can easily evolve with IVAS standards and technologies.

[0104] Example IVAS SPAR encoding / decoding The following description of a further embodiment focuses on the differences between this further embodiment and the previously described embodiment. Therefore, features common to both embodiments may be omitted from the following description, and in that case, it should be assumed that the features of the previously described embodiment are implemented or at least can be implemented in this further embodiment (unless the following description requires otherwise). In addition, when a feature is extracted from an embodiment disclosed below and added to a claim, the feature may not be related or closely related to other features of that embodiment.

[0105] In some embodiments, the IVAS SPAR encoder determines and encodes a coding mode / tool ​​indicator in a common header (CH) section of the IVAS bitstream. The coding mode / tool ​​indicator has a value corresponding to the coding mode / tool. The IVAS bitstream determines and encodes a mode header / tool ​​header in a tool header (TH) section of the IVAS bitstream, where the TH section follows the CH section. The IVAS SPAR encoder determines and encodes a metadata payload in a metadata payload (MDP) section of the IVAS bitstream, where the MDP section follows the CH section. The IVAS SPAR encoder determines and encodes an enhanced voice service (EVS) payload in an EVS payload (EP) section of the IVAS bitstream, where the EP section follows the CH section. In some embodiments, the IVAS system stores the bitstream on a non-transitory computer-readable medium. In other embodiments, the IVAS system streams the bitstream to a downstream device. In some embodiments, the IVAS SPAR encoder has a device architecture described with reference to FIG. 8.

[0106] In some embodiments, the EP section follows the MDP section. Note that having the EP section follow the MDP section of the IVAS bitstream ensures efficient bit packing, and allowing the number of MDP and EP bits to vary (according to a bitrate distribution algorithm) ensures utilization of all available bits in the IVAS bitrate budget.

[0107] In some embodiments, the IVAS SPAR decoder extracts and decodes an IVAS bitstream encoded in the IVAS SPAR format. The IVAS SPAR decoder extracts and decodes a coding mode / tool ​​indicator in a CH section of the bitstream. The coding mode / tool ​​indicator has a value corresponding to the coding mode / tool. The IVAS SPAR decoder extracts and decodes a mode header / tool ​​header in a tool header (TH) section of the bitstream. The TH section follows the CH section. The IVAS SPAR decoder extracts and decodes a metadata payload in an MDP section of the bitstream. The MDP section follows the CH section. The IVAS SPAR decoder decodes an EVS payload in an EP section of the bitstream. The EP section follows the CH section.

[0108] In some embodiments, the IVAS system controls the audio decoder based on the encoding mode, the tool header, the EVS payload, and the metadata payload. In other embodiments, the IVAS system stores a representation of the encoding mode, the tool header, the EVS payload, and the metadata payload on a non-transitory computer-readable medium. In some embodiments, the IVAS SPAR decoder has a device architecture as described with reference to FIG.

[0109] In some embodiments, CH comprises a 3-bit data structure, one of the values ​​of the 3-bit data structure corresponds to the SPAR coding mode, and the rest of the values ​​correspond to other coding modes. The 3-bit data structure is advantageous because it allows for a compact code that can indicate up to eight coding modes. In other embodiments, CH comprises fewer than 3 bits. In other embodiments, CH comprises more than 3 bits.

[0110] In some embodiments, the IVAS system stores or reads a row index in the TH section of the IVAS bitstream that points to a row in the SPAR bitrate distribution control table. For example, the row index can be calculated based on the number of rows corresponding to the IVAS operating bitrate as follows: x=ceil(log2(number of rows corresponding to the IVAS bitrate)). Thus, the length of the TH section is variable.

[0111] In some embodiments, the system stores or reads a quantization strategy indicator; a coding strategy indicator; and the quantized and coded real and imaginary parts of one or more coefficients in an MDP section of the IVAS bitstream.

[0112] In another embodiment, the system stores or reads the quantization strategy indicator in the MDP section of the IVAS bitstream.

[0113] In another embodiment, the system stores or reads the encoding strategy indicator in the MDP section of the IVAS bitstream.

[0114] In another embodiment, the system stores or reads quantized and coded real and imaginary parts of one or more coefficients in or from the MDP section of the IVAS bitstream.

[0115] In some embodiments, the one or more coefficients include, but are not limited to, prediction coefficients, inter-prediction coefficients (or direct coefficients), real (diagonal) decorrelator coefficients, and complex (non-diagonal) decorrelator coefficients.

[0116] In some embodiments, more or fewer coefficients are stored in and read from the MDP section of the IVAS bitstream.

[0117] In some embodiments, the IVAS system stores or reads the EVS payload of all channels in the EP section of the IVAS bitstream according to 3GPP TS26.445.

[0118] An example IVAS bitstream using SPAR formatting is shown below: The IVAS bitstream contains four subdivisions as follows: [Table 13]

[0119] Common Header (CH): In some embodiments, the IVAS common header (CH) is formatted as follows: [Table 14]

[0120] Tool Header (TH): In some embodiments, the SPAR Tool Header (TH) is an index offset into the SPAR Bitrate Distribution Control Table. [Table 15]

[0121] An example embodiment of the SPAR bitrate distribution control table is shown below. Each IVAS bitrate can support one or more values ​​of Bandwidth (BW), Downmix configuration (dmx channel, dmx string), Active W, Complex flag, Transition mode value, EVS bitrate setting, Metadata quantization level setting and Decorrelator ducking flag. In this example embodiment, there is only one entry per bitrate, so the number of bits in the SPAR TH section is 0. The acronyms used in the following table are defined as follows: PR: prediction coefficient, C: Interprediction coefficients (or direct coefficients), P r : real (diagonal) decorrelator coefficients, P c : complex (off-diagonal) decorrelator coefficients.

[0122] An example SPAR bitrate distribution control table is as follows: [Table 16]

[0123] Metadata Payload (MDP): An example metadata payload (MDP) is as follows: [Table 17]

[0124] EVS Payload (EP): In some embodiments, the quantization of the metadata and the calculation of the actual EVS bitrate of each downmix channel is performed using an EVS bitrate distribution control strategy, one exemplary implementation of which is described below.

[0125] Example EVS bitrate distribution control strategies In some embodiments, the EVS bitrate distribution control strategy includes two sections: metadata quantization and EVS bitrate distribution.

[0126] Metadata Quantization. There are two defined thresholds in this section: the target parameter bitrate threshold (MDtar) and the maximum target bitrate threshold (MDmax).

[0127] Step 1: For each frame, parameters are quantized in a non-temporal differential manner and coded using an entropy coder. In some embodiments, an arithmetic coder is used. In other embodiments, a Huffman encoder is used. If the parameter bit rate estimate is less than MDtar, any extra available bits are fed to the audio encoder to increase the bit rate of the audio essence.

[0128] Step 2: If step 1 fails, a subset of the parameter values ​​in the frame are quantized and subtracted from the quantized parameter values ​​in the previous frame, and the differential quantized parameter values ​​are coded using an entropy coder. If the parameter bit rate estimate is less than MDtar, any extra available bits are fed to the audio encoder to increase the bit rate of the audio essence.

[0129] Step 3: If step 2 fails, the bitrate of the quantized parameters is calculated without entropy.

[0130] Step 4: The results of steps 1, 2, and 3 are compared with MDmax. If the minimum of steps 1, 2, and 3 is within MDmax, the remaining bits are coded and provided to the audio coder.

[0131] Step 5: If step 4 fails, the parameters are quantized more coarsely and the above steps are repeated as the first fallback strategy (Fallback 1).

[0132] Step 6: If step 5 fails, the parameters are quantized using a quantization scheme guaranteed to be within MDmax as a second fallback strategy (Fallback 2). After all the above iterations, the metadata bitrate is guaranteed to be within MDmax and the encoder generates actual metadata bits, i.e. Metadata_actual_bits (MDact).

[0133] EVS Bitrate Distribution (EVSbd). For the purposes of this section, the following definitions apply: EVStar: EVS target bits, the desired bits for each EVS instance. EVSact: EVS actual bits, the sum of the actual bits available for all EVS instances. EVSmin: EVS minimum bits, the minimum bits for each EVS instance. The EVS bitrate must not fall below the value indicated by these bits. EVSmax: EVS maximum bits, the maximum bits for each EVS instance. The EVS bitrate must not exceed the value indicated by these bits. EVS W: An EVS instance that encodes the W channel. EVS Y: An EVS instance that encodes the Y channel. EVS X: An EVS instance encoding the X channel. EVS Z: An EVS instance that encodes the Z channel. EVSact=IVAS_bits-header_bits-MDact

[0134] If EVSact is less than the sum of EVStar of all EVS instances, then bits are taken from the EVS instances in the following order (Z, X, Y, W): Maximum bits that can be taken from any channel = EVStar(ch) - EVSmin(ch).

[0135] If EVSact is greater than the sum of EVStar of all EVS instances, then all additional bits are allocated to the downmix channels in the following order (W, Y, X, Z): Maximum additional bits that can be added to any channel = EVSmax(ch) - EVStar(ch).

[0136] The EVSbd scheme described above calculates the actual EVS bitrates of all channels, i.e. EWa, EYa, EXa, EZa for W, Y, X and Z channels, respectively. After each channel is encoded by a separate EVS instance with EWa, EYa, EXa and EZa bitrates, all the EVS bits are concatenated and packed together. The advantage of this configuration is that no additional header is needed to indicate the EVS bitrate for any channel.

[0137] In some embodiments, the EP section is as follows: [Table 18]

[0138] Example SPAR Decoder Bitstream Unpacking In some embodiments, the steps of SPAR decoder bitstream unpacking are described as follows.

[0139] Step 1: Calculate the IVAS bit rate from the length of the receive bit buffer.

[0140] Step 2: Parse the SPAR TH section based on the number of entries of the IVAS bitrate in the SPAR bitrate distribution control table and extract the index offset, where the index offset is determined by the IVAS operating bitrate.

[0141] Step 3: Use the index offset to determine the actual table row index of the SPAR bitrate distribution control table, and read all columns of the SPAR bitrate distribution control table row pointed to by the actual table row index.

[0142] Step 4: Read the quantization strategy bits and encoding strategy bits from the MDP section of the IVAS bitstream, and unquantize the SPAR spatial metadata in the MPD section based on the indicated quantization strategy and encoding strategy.

[0143] Step 5: Based on the total EVS bitrate (the remaining bits read from the IVAS bitstream), determine the actual EVS bitrate of each channel per the EVS bitrate distribution (EVSbd) mentioned above.

[0144] Step 6: Read the encoded EVS bits from the EP section of the IVAS bitstream based on the actual EVS bitrate, and decode each channel of the FoA audio signal using the respective EVS instance.

[0145] Step 7: Use the decoded EVS output and spatial metadata to construct the FoA(SPAR) audio signal.

[0146] An advantage of the above-described embodiment of the IVAS bitstream format is that it efficiently and compactly encodes data supporting various audio service capabilities, including, but not limited to, mono to stereo upmix and fully immersive audio encoding, decoding and rendering (e.g., FoA encoding). This embodiment is also supported by a wide range of devices, endpoints, and network nodes, including, but not limited to, mobile and smart phones, electronic tablets, personal computers, conference phones, conference rooms, virtual reality (VR) and augmented reality (AR) devices, home theater devices, and other suitable devices, each of which may have a variety of acoustic interfaces for sound capture and rendering. The IVAS bitstream format is extensible so that it can easily evolve with IVAS standards and technologies.

[0147] Example Process - IVAS Bitstream in CACPL Format 4A is a flow diagram of an IVAS encoding process 400 according to one embodiment. The process 400 can be implemented using a device architecture such as that described with reference to FIG.

[0148] The process 400 includes determining an encoding tool indicator and a sampling rate indicator using an IVAS encoder and encoding the encoding tool indicator and the sampling rate indicator into a common header (CH) section of an IVAS bitstream (401). In some embodiments, the tool indicator has a value corresponding to the encoding tool and the sampling rate indicator has a value indicative of the sampling rate.

[0149] The process 400 further includes determining an Enhanced Voice Services (EVS) payload using an IVAS encoder and encoding the Enhanced Voice Services (EVS) payload into an EVS payload (EP) section of the IVAS bitstream (402). In some embodiments, the EP section follows the CH section.

[0150] The process 400 further includes determining a metadata payload in the metadata payload using an IVAS encoder and encoding the metadata payload into a metadata payload (MDP) section of the IVAS bitstream (403). In some embodiments, the MDP section follows the CH section. In some embodiments, the EP section follows the MDP section of the bitstream.

[0151] The process 400 further includes storing the IVAS bitstream on a non-transitory computer-readable medium or streaming the IVAS bitstream to a downstream device (404).

[0152] 4B is a flow diagram of an IVAS encoding process 405 using an alternative IVAS format, according to one embodiment. The process 405 may include a device architecture as described with reference to FIG.

[0153] The process 405 includes determining a coding tool indicator using an IVAS encoder and encoding the coding tool indicator into a common header (CH) section of the IVAS bitstream (406). In some embodiments, the tool indicator has a value corresponding to the coding tool.

[0154] The process 405 further includes encoding (407) a representation of the IVAS bitrate distribution control table into a common spatial coding tool header (CTH) section of the IVAS bitstream using an IVAS encoder.

[0155] The process 405 further includes determining a metadata payload using an IVAS encoder and encoding the metadata payload into a metadata payload (MDP) section of the IVAS bitstream (408). In some embodiments, the MDP section follows the CH section of the IVAS bitstream.

[0156] The process 405 further includes determining an Enhanced Voice Services (EVS) payload using an IVAS encoder and encoding (409) the Enhanced Voice Services (EVS) payload into an EVS Payload (EP) section of the IVAS bitstream. In some embodiments, the EP section follows a CH section of the IVAS bitstream. In some embodiments, an MDP section follows the EP section of the IVAS bitstream.

[0157] The process 405 further includes storing the IVAS bitstream on a storage device or streaming the IVAS bitstream to a downstream device (410).

[0158] 5A is a flow diagram of an IVAS decoding process 500 according to one embodiment. The process 500 can be implemented using a device architecture such as that described with reference to FIG.

[0159] The process 500 includes extracting and decoding, using an IVAS decoder, a coding tool indicator and a sampling rate indicator from a common header (CH) section of an IVAS bitstream (501). In some embodiments, the tool indicator has a value corresponding to a coding tool and the sampling rate indicator has a value indicative of a sampling rate.

[0160] The process 500 further includes extracting and decoding (502) an Enhanced Voice Services (EVS) payload from an EVS Payload (EP) section of the IVAS bitstream using an IVAS decoder. In some embodiments, the EP section follows the CH section of the IVAS bitstream.

[0161] The process 500 further includes extracting and decoding (503) the metadata payload from a metadata payload (MDP) section of the bitstream using an IVAS decoder. In some embodiments, the MDP section follows a CH section of the IVAS bitstream. In some embodiments, an EP section follows the MDP section of the IVAS bitstream.

[0162] The process 500 further includes controlling an audio decoder based on the encoding tools, the sampling rate, the EVS payload, and the metadata payload or storing (504) a representation of the encoding tools, the sampling rate, the EVS payload, and the metadata payload on a non-transitory computer-readable medium.

[0163] 5B is a flow diagram of an IVAS decoding process 505 using an alternative format according to one embodiment. The process 505 can be implemented using a device architecture such as that described with reference to FIG.

[0164] The process 505 includes extracting and decoding, using an IVAS decoder, a coding tool indicator in a common header (CH) section of the IVAS bitstream (506). In some embodiments, the tool indicator has a value corresponding to the coding tool.

[0165] The process 505 further includes extracting and decoding (507) using an IVAS decoder a representation of the IVAS bitrate distribution control table within a common spatial coding tool header (CTH) section of the IVAS bitstream.

[0166] The process 505 further includes decoding (508) a metadata payload in a metadata payload (MDP) section of the IVAS bitstream using an IVAS decoder. In some embodiments, the MDP section follows the CH section of the IVAS bitstream.

[0167] The process 505 further includes decoding (509) an Enhanced Voice Services (EVS) payload in an EVS Payload (EP) section of the IVAS bitstream using an IVAS decoder. In some embodiments, the EP section follows a CH section of the IVAS bitstream. In some embodiments, an MDP section follows the EP section of the IVAS bitstream.

[0168] The process 505 further includes controlling an audio decoder based on the representation of the encoding tool indicator, the IVAS bitrate distribution control table, the metadata payload, and the EVS payload, or storing (510) the representation of the encoding tool indicator, the IVAS bitrate distribution control table, the metadata payload, and the EVS payload on a storage device.

[0169] Example Process-SPAR Format IVAS Bitstream 6 is a flow diagram of an IVAS SPAR encoding process 600 according to one embodiment. The process 600 can be implemented using a device architecture such as that described with reference to FIG.

[0170] The process 600 includes decoding a coding mode / coding tool indicator using an IVAS encoder and encoding the coding mode / coding tool indicator into a common header (CH) section of an IVAS bitstream (601).

[0171] The process 600 further includes obtaining and encoding (602) a representation of the SPAR bitrate distribution control table using an IVAS encoder into a mode header / tool ​​header in a tool header (TH) section of the IVAS bitstream, where the TH section follows the CH section.

[0172] The process 600 further includes determining, using an IVAS encoder, a metadata payload and encoding the metadata payload into a metadata payload (MDP) section of the IVAS bitstream (603). In some embodiments, the MDP section follows the CH section of the IVAS bitstream.

[0173] In some embodiments, the MDP section includes a quantization strategy indicator; an encoding strategy indicator; and the quantized and encoded real and imaginary parts of one or more coefficients.In some embodiments, the one or more coefficients include, but are not limited to, prediction coefficients, inter-prediction coefficients (or direct coefficients), real (diagonal) decorrelator coefficients and complex (non-diagonal) decorrelator coefficients.In some embodiments, more or less coefficients are stored in the MDP section of the IVAS bitstream and read from the MDP section of the IVAS bitstream.

[0174] The process 600 further includes using an IVAS encoder to determine an Enhanced Voice Services (EVS) payload and encoding the EVS payload into an EVS payload (EP) section of the IVAS bitstream (604). In some embodiments, the EP section of the IVAS bitstream includes EVS payloads for all channels in accordance with 3GPP TS26.445. In some embodiments, the EP section follows the CH section of the IVAS bitstream. In some embodiments, the EP section follows the MDP section. It should be noted that having the EP section follow the MDP section of the IVAS bitstream ensures efficient bit packing, and allowing the number of MDP and EP bits to vary (according to a bitrate distribution algorithm) ensures utilization of all available bits in the IVAS bitrate budget.

[0175] The process 600 further includes storing the bitstream on a non-transitory computer-readable medium or streaming the bitstream to a downstream device (605).

[0176] 7 is a flow diagram of an IVAS SPAR decoding process 700 according to one embodiment. The process 700 may be implemented using a device architecture such as that described with reference to FIG.

[0177] The process 700 includes extracting and decoding (701) a coding mode indicator in a common header (CH) section of an IVAS bitstream using an IVAS decoder.

[0178] The process 700 includes extracting and decoding (702) a representation of the SPAR bitrate distribution control table in a mode header / tool ​​header in a tool header (TH) section of the IVAS bitstream using an IVAS decoder. In some embodiments, the TH section follows the CH section.

[0179] The process 700 further includes extracting and decoding (703) a metadata payload from a metadata payload (MDP) section of the IVAS bitstream using an IVAS decoder. In some embodiments, the MDP section follows the CH section of the IVAS bitstream.

[0180] The process 700 further includes extracting and decoding (704) an Enhanced Voice Services (EVS) payload from an EVS Payload (EP) section of the IVAS bitstream using an IVAS decoder. In some embodiments, the EP section follows a CH section. In some embodiments, the EP section follows an MDP section. Note that having the EP section follow an MDP section of the IVAS bitstream ensures efficient bit packing, and allowing the number of MDP and EP bits to vary (according to a bitrate distribution algorithm) ensures utilization of all available bits in the IVAS bitrate budget.

[0181] The process 700 further includes controlling an audio decoder based on the representation of the encoding mode indicator, the SPAR bitrate distribution control table, the EVS payload, and the metadata payload or storing (705) the representation of the encoding mode indicator, the SPAR bitrate distribution control table, the EVS payload, and the metadata payload on a non-transitory computer-readable medium.

[0182] Example System Architecture 8 illustrates a block diagram of an example system 800 suitable for implementing example embodiments of the present disclosure. The system 800 includes one or more server computers or any client devices, including, but not limited to, any of the devices shown in FIG. 1, such as the call server 102, legacy devices 106, user equipment 108, 114, conference room systems 116, 118, home theater systems, VR gear 122, and immersive content ingest 124. The system 800 includes any consumer device, including, but not limited to, a smartphone, a tablet computer, a wearable computer, a vehicle computer, a game console, a surround system, and a kiosk.

[0183] As shown, the system 800 includes a central processing unit (CPU) 801 capable of executing various processes according to programs stored, for example, in a read only memory (ROM) 802 or loaded, for example, from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 also stores data required by the CPU 801 when executing various processes, as needed. The CPU 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0184] The following components are connected to the I / O interface 805: an input unit 806, which may include a keyboard, a mouse, etc.; an output unit 807, which may include a display, such as a liquid crystal display (LCD), and one or more speakers; a storage unit 808, which may include a hard disk or another suitable storage device; and a communication unit 809, which may include a network interface card, such as a network card (e.g., wired or wireless).

[0185] In some embodiments, the input unit 806 includes one or more microphones in different positions (depending on the host device) that enable capture of audio signals in various formats (e.g., mono, stereo, spatial, immersive, and other suitable formats).

[0186] In some embodiments, the output unit 807 includes a system having a variety of speakers. As shown in FIG. 1, the output unit 807 can render audio signals in a variety of formats (e.g., mono, stereo, immersive, binaural, and other suitable formats) (depending on the capabilities of the host device).

[0187] The communication unit 809 is configured to communicate with other devices (e.g., via a network). The drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a flash drive, or another suitable removable medium, is attached to the drive 810 so that a computer program read therefrom is installed in the storage unit 808 as needed. Those skilled in the art will understand that although the system 800 is described as including the above-mentioned components, in practical applications, some of these components can be added, removed, and / or replaced, and all such changes or modifications are all within the scope of the present disclosure.

[0188] Other embodiments In one embodiment, a method for generating a bitstream of an audio signal includes: determining, using an IVAS encoder, a coding tool indicator and a sampling rate indicator, where the coding tool indicator has a value corresponding to a coding tool and the sampling rate indicator has a value indicating a sampling rate; encoding, using the IVAS encoder, the coding tool indicator and the sampling rate indicator into a common header (CH) section of an IVAS bitstream; determining, using the IVAS encoder, an Enhanced Voice Services (EVS) payload; encoding, using the IVAS encoder, the EVS payload into an EVS payload (EP) section of the IVAS bitstream, where the EP section follows the CH section; determining, using the IVAS encoder, a metadata payload; encoding, using the IVAS encoder, the metadata payload into a metadata payload (MDP) section of the IVAS bitstream, where the MDP section follows the CH section; and storing the IVAS bitstream on a non-transitory computer readable medium or streaming the IVAS bitstream to a downstream device.

[0189] In one embodiment, a method for decoding a bitstream of an audio signal includes: extracting and decoding, using an IVAS decoder, an encoding tool indicator and a sampling rate indicator from a CH section of the IVAS bitstream, where the tool indicator has a value corresponding to the encoding tool and the sampling rate indicator has a value indicating the sampling rate; extracting and decoding, using the IVAS decoder, an EVS payload from an EP section of the bitstream, where the EP section follows the CH section; decoding, using the IVAS decoder, a metadata payload from an MDP section of the bitstream, where the MDP section follows the CH section; and controlling an audio decoder based on the encoding tool, the sampling rate, the EVS payload, and the metadata payload or storing representations of the encoding tool, the sampling rate, the EVS payload, and the metadata payload on a non-transitory computer-readable medium.

[0190] In one embodiment, an MDP section follows an EP section in the bitstream, or an EP section follows an MDP section in the bitstream.

[0191] In one embodiment, the IVAS coding tool indicator is a 3-bit data structure, a first value of the 3-bit data structure corresponds to a multi-mono coding tool, a second value of the 3-bit data structure corresponds to a Composite Advanced Coupling (CACPL) coding tool, and a third value of the 3-bit data structure corresponds to another coding tool.

[0192] In one embodiment, the input sampling rate indicator is a 2-bit data structure, a first value of the 2-bit data structure indicating an 8 kHz sampling rate, a second value of the 2-bit data structure indicating a 16 kHz sampling rate, a third value of the 2-bit data structure indicating a 32 kHz sampling rate, and a fourth value of the 2-bit data structure indicating a 48 kHz sampling rate.

[0193] In one embodiment, the method includes storing or retrieving an EVS channel number indicator, a bit rate (BR) extraction mode indicator, EVS BR data, and an EVS payload in or from an EP section of the bitstream, respectively.

[0194] In one embodiment, the method includes storing or retrieving, in an MDP section of the data stream, a coding technique indicator, a number of bands indicator, an indicator indicative of a delay configuration of the filter bank, an indicator of a quantization strategy, an entropy coder indicator, a probability model type indicator, a real coefficient part, an imaginary coefficient part, and one or more coefficients, respectively.

[0195] In one embodiment, a method for generating a bitstream of an audio signal includes determining, using an IVAS encoder, a coding tool indicator, the tool indicator having a value corresponding to a coding tool; encoding, using the IVAS encoder, the coding tool indicator into a common header (CH) section of the IVAS bitstream; determining, using the IVAS encoder, a representation of an index to an IVAS bitrate distribution control table; and encoding, using the IVAS encoder, the representation of the index to the IVAS bitrate distribution control table into a common spatial coding tool header (CTH) section of the IVAS bitstream, the CTH section following the CH section. using the IVAS encoder to determine a metadata payload; using the IVAS encoder to encode the metadata payload into a metadata payload (MDP) section of the IVAS bitstream, where the MDP section follows the CTH section; using the IVAS encoder to determine an Enhanced Voice Services (EVS) payload; using the IVAS encoder to encode the EVS payload into an EVS payload (EP) section of the IVAS bitstream, where the EP section follows the CTH section; and storing the bitstream on a non-transitory computer readable medium or streaming the bitstream to a downstream device.

[0196] In one embodiment, a method for decoding a bitstream of an audio signal includes receiving the bitstream by an IVAS decoder; calculating an IVAS operating bitrate based on a length and stride of the bitstream; reading an indicator of a spatial coding tool from a common header (CH) section of the bitstream; determining a length of a common spatial coding tool header (CTH) section of the bitstream based on the IVAS operating bitrate, which includes looking up a number of entries in an IVAS bitrate distribution control table in the CTH section that correspond to the IVAS operating bitrate; once the length of the CTH section is determined and an index to the IVAS bitrate distribution control table is determined, reading a value in the CTH section; reading information about an Enhanced Voice Services (EVS) bitrate distribution from an entry in the IVAS bitrate distribution control table that corresponds to the index to the IVAS bitrate distribution control table; and providing the information about the EVS bitrate distribution to an EVS decoder.

[0197] In one embodiment, any of the above methods includes reading an indicator of mono downmix backward compatibility with 3GPP TS26.445 from an entry in the IVAS bitrate distribution control table.

[0198] In one embodiment, the method includes determining that a mono downmix backward compatibility indicator is in an ON mode; in response to the ON mode, providing a remaining portion of the bitstream to an EVS decoder; then calculating a respective bitlength of each EVS instance from the remaining portion of the bitstream based on an EVS bitrate distribution; reading EVS bits for each EVS instance based on the corresponding bitlength; providing the EVS bits to the EVS decoder as a first portion, and providing the remaining portion of the bitstream to an MDP decoder for decoding the spatial metadata.

[0199] In one embodiment, the method includes determining that a mono downmix backward compatibility indicator is in an OFF mode; in response to the OFF mode, providing a remainder of the bitstream to an MDP decoder to decode the spatial metadata; then calculating a respective bit length of each EVS instance from the remainder of the bitstream based on an EVS bitrate distribution; reading EVS bits for each EVS instance based on the corresponding bit length; and providing the EVS bits to the EVS decoder as a first portion.

[0200] In one embodiment, a system comprises one or more computer processors; and a non-transitory computer-readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the operations of any one of the method claims above.

[0201] In one embodiment, a non-transitory computer readable medium stores instructions that, when executed by one or more processors, cause the one or more processors to perform the operations of any one of the above method claims.

[0202] According to an exemplary embodiment of the present disclosure, the above-described process can be implemented as a computer software program or on a computer-readable storage medium. For example, an embodiment of the present disclosure includes a computer program product including a computer program tangibly embodied on a machine-readable medium, the computer program including program code for performing the method. In such an embodiment, the computer program can be downloaded and implemented via a communication unit 809 from a network and / or installed from a removable medium 811, as shown in FIG. 8.

[0203] In general, various exemplary embodiments of the present disclosure can be implemented in hardware or dedicated circuits (e.g., control circuitry), software, logic, or any combination thereof. For example, the above-mentioned units can be executed by control circuitry (e.g., a CPU in combination with other components of FIG. 8), which can then perform the operations described in the present disclosure. Some aspects can be implemented in hardware, while other aspects can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device (e.g., control circuitry). Although various aspects of the exemplary embodiments of the present disclosure are illustrated and described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, apparatus, systems, techniques, or methods described herein can be implemented in hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof, as non-limiting examples.

[0204] Additionally, the various blocks illustrated in the flowcharts may be considered as multiple coupled logic circuit elements configured to perform method steps and / or operations and / or related function(s) resulting from the operation of computer program code. For example, embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine-readable medium, the computer program including program code configured to perform a method as described above.

[0205] In the context of this disclosure, a machine-readable medium can be any tangible medium that can contain or store a program used by or in connection with an instruction execution system, an instruction execution apparatus, or an instruction execution device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be non-transitory and can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. More specific examples of machine-readable storage media include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0206] The computer program codes for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These computer program codes can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus having control circuitry, such that the program codes, when executed by a processor of the computer or other programmable data processing apparatus, cause the implementation of the functions / operations specified in the flowcharts and / or block diagrams. The program codes can be executed entirely on the computer or partially on the computer as a stand-alone software package, partly on the computer and partly on a remote computer, entirely on a remote computer or remote server, or distributed across one or more remote computers and / or remote servers.

[0207] Although the specification contains many specific implementation details, these should not be construed as limitations on the scope of what may be claimed, but rather as descriptions of features that may be specific to a particular embodiment. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as operating in a certain combination and may even be initially claimed as such, one or more features from the claimed combination may in some cases be deleted from the combination, and the claimed combination may be subject to subcombinations or variations of the subcombination. The logic flows depicted in the figures do not require the particular order depicted or sequential order to achieve desirable results. In addition, other steps may be provided or deleted from the described flows, and other components may be added or deleted from the described systems. Accordingly, other implementations are within the scope of the appended claims.

Claims

1. 1. A computer-implemented method for generating a bitstream of an audio signal, comprising the steps of: determining, using an immersive voice audio services (IVAS) encoder, an encoding mode indicator or an encoding tool indicator, the encoding mode indicator or the encoding tool indicator indicating an encoding mode or an encoding tool of the audio signal; encoding the coding mode indicator or the coding tool indicator into a common header (CH) section of an IVAS bitstream using the IVAS encoder; determining a mode header or a tool header using the IVAS encoder; encoding the mode header or the tool header into a tool header (TH) section of the IVAS bitstream using the IVAS encoder, the TH section following the CH section; determining a metadata payload including spatial metadata using the IVAS encoder; encoding the metadata payload into a metadata payload (MDP) section of the IVAS bitstream using the IVAS encoder, the MDP section following the CH section; determining an Enhanced Voice Services (EVS) payload using the IVAS encoder, the EVS payload including EVS encoded bits for each channel or each downmix channel of the audio signal; encoding the EVS payload into an EVS payload (EP) section of the IVAS bitstream using the IVAS encoder, the EP section following the CH section; A method comprising:

2. storing the IVAS bitstream on a non-transitory computer readable medium or streaming the IVAS bitstream to a downstream device; 2. The method of claim 1, wherein the coding mode or coding tool indicator, the mode or tool header, the metadata payload and the EVS payload are extracted and decoded from the CH section, the TH section, the MDP section and the EP section of the IVAS bitstream, respectively, for use in reconstructing the audio signal in the downstream device or another device.

3. 3. The method of claim 1, further comprising storing an index offset in the TH section of the IVAS bitstream for calculating a row index of a spatial reconstruction (SPAR) bitrate distribution control table.

4. Quantization Strategy Indicator, a bitstream encoding strategy indicator; quantized and coded real and imaginary parts of the set of coefficients; 4. The method of claim 1, further comprising: storing each of the first and second IVAS bitstreams in the MDP section.

5. 5. The method of claim 1, further comprising storing an EVS payload of an EVS instance in the EP section of the bitstream in accordance with 3rd Generation Partnership Project (3GPP) Technical Specification (TS) 26.

445.

6. 1. A computer-implemented method for decoding a bitstream of an audio signal, comprising the steps of: extracting and decoding, using an immersive voice and audio service (IVAS) decoder, a coding mode indicator or a coding tool indicator in a common header (CH) section of an IVAS bitstream, the coding mode indicator or the coding tool indicator indicating a coding mode or a coding tool of the audio signal; extracting and decoding a mode header or a tool header in a tool header (TH) section of the IVAS bitstream using the IVAS decoder, the TH section following the CH section; extracting and decoding a metadata payload from a metadata payload (MDP) section of the IVAS bitstream using the IVAS decoder, the MDP section following the CH section, contains spatial metadata; extracting and decoding an Enhanced Voice Services (EVS) payload from an EVS payload (EP) section of the IVAS bitstream using the IVAS decoder, the EP section following the CH section, the EVS payload including EVS encoded bits for each channel or downmix channel of the audio signal; A method comprising:

7. 7. The method of claim 6, further comprising: controlling an audio decoder of the downstream device for use in reconstructing the audio signal in a downstream device or another device based on the encoding mode or tool indicator, the mode or tool header, the EVS payload, and the metadata payload, or storing a representation of the encoding mode or tool indicator, the mode header or tool header, the EVS payload, and the metadata payload on a non-transitory computer-readable medium.

8. The method of claim 6 or 7, comprising reading an index offset from the TH section of the IVAS bitstream for calculating a row index of a spatial reconstruction (SPAR) bitrate distribution control table.

9. A method for determining a quantization strategy indicator, comprising: a bitstream encoding strategy indicator; quantized and coded real and imaginary parts of the set of coefficients; from the MDP section of the IVAS bitstream, respectively.

10. A method according to any one of claims 6 to 9, comprising reading an EVS payload of an EVS instance from the EP section of the bitstream in accordance with 3rd Generation Partnership Project (3GPP) Technical Specification (TS) 26.

445.

11. determining a bit rate from the IVAS bitstream; reading an index offset from a spatial reconstruction (SPAR) tool header (TH) section of the IVAS bitstream; determining a table row index for the SPAR bitrate distribution control table using the index offset; reading quantization strategy bits and encoding strategy bits from a metadata payload (MDP) section in the IVAS bitstream; dequantizing SPAR spatial metadata in the MDP section of the IVAS bitstream based on the quantization strategy bits and the encoding strategy bits; determining an Enhanced Voice Services (EVS) bit rate for each channel in said IVAS bit stream using all available EVS bits, a SPAR bit rate distribution control table and a bit rate distribution algorithm; reading EVS encoded bits from the EP section of the IVAS bitstream based on the EVS bitrate; decoding the EVS bits; and Decoding the spatial metadata; and generating a first order Ambisonics (FoA) output using the decoded EVS bits and the decoded spatial metadata; The method of any one of claims 6 to 10, further comprising:

12. 12. The method of claim 1, wherein the CH is a multi-bit data structure, one value of the multi-bit data structure corresponding to a spatial reconstruction (SPAR) coding mode and another value of the data structure corresponding to another coding mode.

13. 9. The method of claim 3 or 8, wherein the EP section follows the MDP section to ensure efficient bit packing, and the number of bits in the MDP section of the IVAS bitstream and the number of bits of the EP section of the IVAS bitstream vary according to the SPAR bitrate distribution control table and bitrate distribution algorithm to ensure utilization of all available bits in an IVAS bitrate budget.

14. The method according to any one of claims 1 to 13, wherein the bit rate of each EVS encoded channel or each downmix channel is determined by the total available bits of the EVS, a bit rate distribution control table and a bit rate distribution algorithm.

15. The method according to claim 4 or 9, wherein the set of coefficients comprises predicted coefficients, direct coefficients, diagonal real coefficients and lower triangular complex coefficients.

16. The method of claim 15, wherein the prediction coefficients are of variable bit length based on entropy coding, and the direct coefficients, the diagonal real coefficients and the lower triangular complex coefficients are of variable bit length based on downmix construction and entropy coding.

17. The method of claim 4 or 9, wherein the quantization strategy indicator is a multi-bit data structure indicating a quantization strategy.

18. The method of claim 4 or 9, wherein the bitstream encoding strategy indicator is a multi-bit data structure indicating a number of bands and a non-differential entropy or a temporal differential entropy encoding scheme of the spatial metadata.

19. The method according to claim 4 or 9, wherein the quantization of the coefficients is in accordance with an EVS bitrate distribution control strategy comprising metadata quantization and EVS bitrate distribution.

20. one or more processors; A non-transitory computer-readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the operations of the method of any one of claims 1 to 19; A system comprising:

21. A non-transitory computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the operations of the method of any one of claims 1 to 19.

Images