Encoding and decoding IVAS bitstream

Abstract

To provide a method for encoding / decoding an immersive voice audio service (IVAS) bitstream.SOLUTION: An encoding method includes: encoding a coding mode indicator in a common header (CH) section of an IVAS bitstream; encoding a mode header or a tool header in a tool header (TH) section following the CH section of the bitstream; encoding a metadata payload in a metadata payload (MDP) section following the CH section of the bitstream; encoding an enhanced voice service (EVS) payload in an EVS payload (EP) section following the CH section; and storing or streaming the encoded bitstream.SELECTED DRAWING: Figure 4B

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims priority to U.S. Provisional Application No. 62 / 881,541, filed August 1, 2019, U.S. Provisional Application No. 62 / 927,894, filed October 30, 2019, U.S. Provisional Application No. 63 / 037,721, filed June 11, 2020, and U.S. Provisional Application No. 63 / 057,666, filed July 28, 2020. The entire disclosures of these U.S. provisional applications are incorporated herein by reference.

[0002] This disclosure relates generally to encoding and decoding audio bitstreams. [Background technology]

[0003] Standards development for audio and video encoder / decoders ("codecs") has recently focused on the development of immersive voice and audio services (IVAS). The focus is on codec development. IVAS is expected to support a variety of audio service capabilities, including but not limited to mono to stereo upmixing and fully immersive audio encoding, decoding and rendering. IVAS is intended to be supported by a wide range of devices, endpoints and network nodes, including mobile and smart phones, electronic tablets, personal computers, conference phones, conference rooms, virtual reality (VR) devices, and more. And augmented reality (AR) devices, home theater devices, These devices, endpoints and network nodes may have a variety of acoustic interfaces for capturing and rendering sound. Summary of the Invention

[0004] An embodiment for encoding and decoding an IVAS bitstream is disclosed.

[0005] In some embodiments, a method for generating a bitstream of an audio signal includes determining, using an Immersive Audio Services (IVAS) encoder, a coding mode indicator or a coding tool indicator, the coding mode indicator or the coding tool indicator indicating a coding mode or a coding tool of the audio signal, encoding, using the IVAS encoder, the coding mode indicator or the coding tool indicator into a common header (CH) section of an IVAS bitstream, determining, using the IVAS encoder, a mode header or a tool header, and encoding, using the IVAS encoder, the mode header or the tool header into the IVAS bitstream. using the IVAS encoder to determine a metadata payload including spatial metadata; encoding the metadata payload using the IVAS encoder into a metadata payload (MDP) section of the IVAS bitstream, the MDP section following the CH section; using the IVAS encoder to determine an Enhanced Voice Services (EVS) payload, the EVS payload including EVS coded bits for each channel or each downmix channel of the audio signal; and encoding the S bitstream into an EVS payload (EP) section of the S bitstream, the EP section following the CH section.

[0006] In some embodiments, the IVAS bitstream is stored on a non-transitory computer-readable medium. In other embodiments, the IVAS bitstream is streamed to a downstream device, and the encoding mode or encoding tool indicator, the mode or tool header, the metadata payload, and the EVS payload are extracted and decoded from the CH section, the TH section, the MDP section, and the EP section of the IVAS bitstream, respectively, for use in reconstructing the audio signal in the downstream device or another device.

[0007] In some embodiments, a method of decoding a bitstream of an audio signal includes using an Immersive Audio Services (IVAS) decoder to extract and decode a coding mode indicator or a coding tool indicator in a common header (CH) section of an IVAS bitstream, the coding mode indicator or the coding tool indicator indicating a coding mode or coding tool of the audio signal, and using the IVAS decoder to extract and decode a mode header or a tool header in a tool header (TH) section of the IVAS bitstream, the TH section indicating a coding mode or coding tool of the audio signal. using the IVAS decoder to extract and decode a metadata payload from a metadata payload (MDP) section of the IVAS bitstream, the MDP section following the CH section, the metadata payload including spatial metadata; and using the IVAS decoder to extract and decode an Enhanced Voice Services (EVS) payload from an EVS payload (EP) section of the IVAS bitstream, the EP section following the CH section, the EVS payload including EVS encoded bits for each channel or each downmix channel of the audio signal.

[0008] In some embodiments, an audio decoder of a downstream device for use in reconstructing the audio signal in the downstream device or another device is controlled based on the encoding mode indicator or the encoding tool indicator, the mode header or the tool header, the EVS payload, and the metadata payload. In other embodiments, a representation of the encoding mode indicator or the encoding tool indicator, the mode header or the tool header, the EVS payload, and the metadata payload is stored on a non-transitory computer-readable medium.

[0009] In some embodiments, the bitrate of each EVS encoded channel or each downmix channel is determined by the total available bits of the EVS, the SPAR bitrate distribution control table and a bitrate distribution algorithm.

[0010] In some embodiments, the CH is a multi-bit data structure, one value of the multi-bit data structure corresponds to a spatial reconstruction (SPAR) coding mode and another value of the data structure corresponds to another coding mode.

[0011] In some embodiments, the method further includes storing or reading, respectively, an index offset in the TH section of the IVAS bitstream for calculating a row index of a spatial reconstruction (SPAR) bitrate distribution control table.

[0012] In some embodiments, the method further includes storing or reading, respectively, a quantization strategy indicator, a bitstream encoding strategy indicator, and the quantized and encoded real and imaginary parts of the set of coefficients in or from the MDP section of the IVAS bitstream.

[0013] In some embodiments, the set of coefficients includes predicted coefficients, direct coefficients, diagonal real coefficients and lower triangular complex coefficients.

[0014] In some embodiments, the prediction coefficients are variable bit length based on entropy coding, and the direct coefficients, the diagonal real coefficients and the lower triangular complex coefficients are variable bit length based on downmix construction and entropy coding.

[0015] In some embodiments, the quantization strategy indicator is a multi-bit data structure that indicates a quantization strategy.

[0016] In some embodiments, the bitstream encoding strategy indicator is a multi-bit data indicating the number of bands and a non-differential entropy encoding scheme or a time-differential entropy encoding scheme of the spatial metadata. It is a data structure.

[0017] In some embodiments, the quantization of the coefficients is according to an EVS bitrate distribution control strategy that includes metadata quantization and EVS bitrate distribution.

[0018] In some embodiments, the above method further includes storing or reading an EVS payload of an EVS instance in the EP section of the IVAS bitstream, respectively, in accordance with 3rd Generation Partnership Project (3GPP) Technical Specification (TS) 26.445.

[0019] In some embodiments, the method includes: determining a bitrate from the IVAS bitstream; reading an index offset from a spatial reconstruction (SPAR) tool header (TH) section of the IVAS bitstream; determining a table row index for the SPAR bitrate distribution control table using the index offset; reading quantization strategy bits and encoding strategy bits from a metadata payload (MDP) section in the IVAS bitstream; and determining a bitrate from the IVAS bitstream based on the quantization strategy bits and the encoding strategy bits. and determining an Enhanced Voice Services (EVS) bitrate for each channel in the IVAS bitstream using all available EVS bits and a SPAR bitrate distribution control table; reading EVS encoded bits from the EP section of the IVAS bitstream based on the EVS bitrates; decoding the EVS bits; decoding the spatial metadata; and generating a first order Ambisonics (FoA) output using the decoded EVS bits and the decoded spatial metadata.

[0020] Other embodiments disclosed herein relate to systems, devices, and computer-readable media. Details of the disclosed embodiments are set forth in the accompanying drawings and the following description. Other features, objects, and advantages will be apparent from the following description, drawings, and claims.

[0021] Certain implementations disclosed herein provide one or more of the following advantages: The disclosed IVAS bitstream format is an efficient and robust bitstream format that supports a variety of audio service capabilities, including, but not limited to, mono-to-stereo upmix and fully immersive audio encoding, decoding and rendering. In some implementations, the IVAS bitstream format supports complex advance coupling (CACPL) for analyzing and downmixing stereo audio signals. In other implementations, the IVAS bitstream format supports spatial reconstruction (SPAR) for analyzing and downmixing first order Ambisonics (FoA) audio signals.

[0022] In the figures, a particular arrangement or ordering of graphical elements, such as elements representing devices, units, instruction blocks, and data elements, is shown for ease of explanation. However, it should be understood by those skilled in the art that the particular ordering or arrangement of these graphical elements in the figures does not imply that any particular order or sequence of processing is required, nor that separation of processes is required. Furthermore, the inclusion of a graphical element in a drawing does not imply that such element is required in all embodiments, nor does it imply that features represented by such element cannot be included in or combined with other elements in some embodiments.

[0023] Furthermore, in the drawings, when a connecting element such as a solid or dashed line or an arrow is used to indicate a connection, relationship, or association between two or more other diagrammatic elements, it is not intended to imply that the connection, relationship, or association may not exist in the absence of such connecting element. In other words, some connections, relationships, or associations between elements are not shown in the drawings so as not to obscure the present disclosure. In addition, for ease of illustration, a single connecting element is used to represent multiple connections, relationships, or associations between elements. For example, when a connecting element represents communication of signals, data, or instructions, it should be understood by those skilled in the art that such an element represents one or more signal paths for implementing the communication, as appropriate. [Brief description of the drawings]

[0024] [Figure 1] FIG. 1 illustrates an IVAS system according to one embodiment.

[0025] [Diagram 2] FIG. 1 is a block diagram of a system for encoding and decoding an IVAS bitstream, according to one embodiment.

[0026] [Diagram 3] A block diagram of a FoA coder / decoder ("codec") that encodes and decodes an IVAS bitstream in FoA format according to one embodiment.

[0027] [Figure 4A] FIG. 1 is a flow diagram of an IVAS encoding process according to one embodiment.

[0028] [Figure 4B] FIG. 1 is a flow diagram of an IVAS encoding process using an alternative IVAS format, according to one embodiment.

[0029] [Figure 5A] FIG. 1 is a flow diagram of an IVAS decoding process according to one embodiment.

[0030] [Figure 5B] FIG. 13 is a flow diagram of an IVAS decoding process using an alternative IVAS format, according to one embodiment.

[0031] [Figure 6] FIG. 1 is a flow diagram of an IVAS SPAR encoding process according to one embodiment.

[0032] [Figure 7] FIG. 1 is a flow diagram of an IVAS SPAR decoding process according to one embodiment.

[0033] [Figure 8] FIG. 2 is a block diagram of an exemplary device architecture according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0034] The use of the same reference numbers in the various drawings indicates similar elements.

[0035] In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments described. It will be apparent to those skilled in the art that the various embodiments described may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments. Several features are described below that can each be used independently or in some combination with other features.

[0036] nomenclature As used herein, the terms "include" and "comprises" and variations thereof The terms "determined" and "seeking" are to be interpreted as open-ended terms meaning "including, but not limited to." The term "or" is to be interpreted as "and / or" unless the context clearly indicates otherwise. The term "based on" is to be interpreted as "based at least in part on." The terms "one exemplary embodiment" and "an exemplary embodiment" are to be interpreted as "at least one exemplary embodiment." The term "another embodiment" is to be interpreted as "at least one alternative embodiment." The terms "determined" and "seeking" / "determines" and "determining" are to be interpreted as "obtaining," "receiving," "receiving" and "receiving" and are to be interpreted as "based at least in part on." In addition, in the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0037] Overview of the IVAS system 1 illustrates an IVAS system 100 according to one or more embodiments. In some embodiments, various devices communicate with a public switched telephone network (PSTN), for example, denoted by PSTN / other PLMN 104. The IVAS system 100 communicates with the call server 102 through a call server 102 configured to receive audio signals from a device or public land mobile network (PLMN) device. The IVAS system 100 supports legacy devices 106 that render and capture audio in mono only, including but not limited to devices that support enhanced voice service (EVS), multi-rate wideband (AMR-WB), and adaptive multi-rate narrowband (AMR-NB). The IVAS system 100 also supports user equipment (UE) 108, 114 capturing and rendering stereo audio signals, or UE 110 capturing mono signals and binaurally rendering them into a multi-channel signal. The IVAS system 100 also supports immersive and stereo signals captured and rendered by video conferencing room systems 116, 118, respectively. The IVAS system 100 also supports stereo capture and immersive rendering of stereo audio signals for home theater systems, as well as mono capture and immersive rendering of audio signals for virtual reality (VR) gear 122 and immersive content ingest 124.

[0038] Example IVAS Encoding / Decoding System 2 is a block diagram of a system 200 for encoding and decoding an IVAS bitstream, according to one or more embodiments. To perform the encoding, the IVAS encoder includes a spatial analysis downmix unit 202 that receives audio data 201. This audio data may include mono signals, stereo signals, binaural signals, spatial audio signals (e.g., multi-channel spatial audio objects), FoA, higher order Ambisonics (HoA), and any other audio data. In some embodiments, the spatial analysis downmix unit 202 implements CACPL for analyzing / downmixing a stereo audio signal and / or SPAR for analyzing / downmixing a FoA audio signal. In other embodiments, the spatial analysis downmix unit 202 implements other formats.

[0039] The output of the spatial analysis downmix unit 202 includes spatial metadata and 1-4 channels of audio. The spatial metadata is input to a quantization and entropy coding unit 203, which quantizes and entropy codes the spatial data. In some implementations, the quantization can include a fine quantization strategy, a medium quantization strategy, a coarse quantization strategy, and a very coarse quantization strategy, and the entropy coding can include Huffman coding or arithmetic coding. The Enhanced Voice Services (EVS) coding unit 206 encodes the 1-4 channels of audio into one or more EVS bitstreams.

[0040] In some embodiments, the EVS encoding unit 206 complies with 3GPP TS26.445 and provides a wide range of features, such as improved quality and coding efficiency for narrowband (EVS-NB) and wideband (EVS-WB) voice services, improved quality using ultra-wideband (EVS-SWB) voice, improved quality of mixed content and music in conversational applications, robustness to packet loss and delay jitter, and backward compatibility to AMR-WB codec. In some embodiments, the EVS encoding unit 206 includes a pre-processing mode selection unit that selects either a speech coder that encodes the speech signal and a perceptual coder that encodes the audio signal at a specified bit rate based on the mode / bitrate control unit 207. In some embodiments, the speech encoder is an improved variant of algebraic code-excited linear prediction (ACELP) with specialized LP-based modes for different speech classes. In some embodiments, the audio encoder is a modified discrete cosine transform (MDCT) encoder for enhanced efficiency at low delay / low bitrate and designed to provide seamless and reliable switching between the speech encoder and the audio encoder.

[0041] In some embodiments, the IVAS decoder includes a quantization entropy decoding unit 204 configured to recover the spatial metadata and an EVS decoder(s) configured to recover the 1-4 channel audio signals. The recovered spatial metadata and audio signals are input to a spatial synthesis / rendering unit 209, which synthesizes / renders the audio signals using the spatial metadata for playback on various audio systems 210.

[0042] Example IVAS / SPAR codec 3 is a block diagram of a FoA codec 300 for encoding and decoding FoA in SPAR format according to some embodiments. The FoA codec 300 includes a SPAR FoA encoder 301, an EVS encoder 305, a SPAR FoA decoder 306, and an EVS decoder 307. The FoA codec 300 converts the FoA input signal into a set of downmix channels and parameters that are used to regenerate the input signal in the decoders 306, 307. The downmix signal can vary between 1 channel to 4 channels, and the parameters include a prediction coefficient (PR), a cross-prediction coefficient (C), and a decorrelation coefficient (P). It should be noted that SPAR is a process used to reconstruct an audio signal from a downmix of the audio signal using the PR, C and P parameters, as described in further detail below.

[0043] Note that the exemplary implementation shown in Figure 3 assumes a passive W channel and depicts a nominal two-channel downmix in which the W channel is sent unchanged to the decoder 306 along with a single prediction channel Y'. In other implementations, W can be an active channel. An active W channel allows some mixing of the X, Y, and Z channels into the W channel as follows:

number

[0044] As will be explained in more detail below, the C coefficients allow the X and some of the Z channels to be reconstructed from Y', while the remaining channels are reconstructed by a decorrelated version of the W channel, as will be explained in more detail below.

[0045] In some embodiments, the SPAR FoA Encoder 301 includes a Passive / Active Predictor unit 302, a Remix unit 303, and an Extraction / Downmix Selection unit 304. The Passive / Active Predictor receives the FoA channels in 4-channel B-format (W, Y, Z, X) and calculates the prediction channel (W or W', Y', Z', X'). Note that the W channel is an omnidirectional polar pattern that includes all sounds in a sphere coming from all directions with equal gain and phase, X is a figure-8 bidirectional polar pattern pointing forward, Y is a figure-8 bidirectional polar pattern pointing left, and Z is a figure-8 bidirectional polar pattern pointing up.

[0046] The Extraction / Downmix Selection unit 304 extracts the SPAR FoA metadata from the metadata payload section of the IVAS bitstream, as described in more detail below. The Passive / Active Predictor unit 302 and Remix unit 303 use the SPAR FoA metadata to generate remixed FoA channels (W or W', A', B', C') that are input to the EVS Encoder 305 for encoding into an EVS bitstream, which is encapsulated within the IVAS bitstream that is sent to the Decoder 306. In this example, the Ambisonic B format channel is encoded into the AmbiX Note that the RGB values ​​are arranged in the format, although other formats such as the Furse-Malham (FuMa) format (W, X, Y, Z) can be used as well.

[0047] Referring to the SPAR FoA decoder 306, the EVS bitstream is decoded by the EVS decoder 307, resulting in N (e.g., N=4) downmix channels. In some embodiments, the SPAR FoA decoder 306 performs the inverse of the operations performed by the SPAR encoder 301. For example, the remixed FoA channels (W or W', A', B', C') are recovered from the N downmix channels using the SPAR FoA spatial metadata. The remixed SPAR FoA channels are input to the inverse mixer 311, which generates the predicted SPAR FoA. The FoA channel (W or W', Y', Z', X') is recovered. The predicted SPAR FoA channel is then input to an inverse predictor 312 to recover the original unmixed SPAR FoA channel (W, Y, Z, X). In this two-channel example, the decorrelator blocks 309a (dec1)...309n (dec D Note that the sparse-spaced ensemble (SPAR) is used to generate a decorrelated version of the W channel using a time-domain or frequency-domain decorrelator. The decorrelated channel is then used in combination with the SPAR FoA metadata to fully or parametrically reconstruct the X and Z channels.

[0048] In some implementations, depending on the number of downmix channels, one of the FoA inputs (W channel) is sent intact to the SPAR FoA decoder 306, and one to three of the other channels (Y, Z, and X) are sent to the SPAR FoA decoder 306 as residuals or fully parametrically. The PR coefficients, which remain the same regardless of the number of downmix channels N, are used to minimize the predictable energy in the residual downmix channel. The C coefficients are used to further help regenerate the fully parameterized channel from the residual. Thus, the C coefficients are not needed in the one and four channel downmix cases where there are no residual or parameterized channels to predict. The P coefficients are used to fill in the remaining energy not accounted for by the PR and C coefficients. The number of P coefficients depends on the number N of downmix channels in each band. In some implementations, the SPAR PR coefficients (passive W only) are calculated as follows:

[0049] Step 1. Predict all side signals (Y, Z, X) from the main W signal using equation [1].

number

number

[0050] Step 2. Remix the W signal with the predicted (Y', Z', X') signals (in that order from most acoustically related to least acoustically related), where "remix" means rearranged or recombined signals based on some methodology.

number

[0051] One implementation of remix is ​​a reordering of the input signal into W, Y', X', Z', assuming that audio cues from left and right are more acoustically relevant than front and back, and front and back cues are more acoustically relevant than top and bottom cues.

[0052] Step 3. Compute the covariance of the 4-channel posterior prediction and remix and downmix as shown in equations [4] and [5].

number

number

[0053] As an example of a WABC downmix with channels 1-4, d and u represent the following channels as shown in Table I: [Table 1]

[0054] The main targets for SPAR FoA metadata calculation are R dd quantity, R ud Amount and R uu The quantity is Rdd quantity, R ud Amount and R uu From this quantity, the system determines whether it is possible to inter-predict the remaining portion of the fully parametric channel from the residual channel that is transmitted to the decoder. In some implementations, the required extra C coefficients are given by the following equation:

number

[0055] Thus, the C parameter has the shape (1×2) for a 3-channel downmix and (2×1) for a 2-channel downmix.

[0056] Step 4. Calculate the residual energy in the parameterized channels that must be reconstructed by the decorrelator. uu is the actual energy R uu (posterior prediction) and the regenerated mutual prediction energy Reg uu is the difference between

number

number

number

[0057] P is also a covariance matrix and therefore Hermitian symmetric, so only parameters from the upper or lower triangle need to be transmitted to the decoder 306. The diagonal entries are real numbers, while the off-diagonal elements may be complex numbers.

[0058] Example encoding / decoding of IVAS bitstream As described with reference to Figures 2 and 3, the IVAS bitstream(s) are encoded and decoded by the IVAS codec. In some embodiments, the IVAS encoder determines and encodes an encoding tool indicator and a sampling rate indicator into a common header (CH) section of the IVAS bitstream. The encoder has a value corresponding to the encoding tool, and the sampling rate indicator has a value indicating the sampling rate. The IVAS encoder determines the EVS payload and encodes it into the EVS payload (EP) section of the bitstream. The EP section follows the CH section. The IVAS encoder determines and encodes a metadata payload into the metadata payload (MDP) section of the bitstream. In some embodiments, the MDP section follows the CH section. In other embodiments, the MDP section follows the EP section of the bitstream, or the EP section follows the MDP section of the bitstream. In some embodiments, the IVAS encoder stores the bitstream on a non-transitory computer-readable medium or streams the bitstream to a downstream device. In other embodiments, the IVAS encoder has a device architecture as shown in FIG. 8.

[0059] In some embodiments, the IVAS decoder receives an IVAS bitstream and extracts and decodes audio data encoded in the IVAS format by the IVAS encoder. The IVAS decoder extracts and decodes an encoding tool indicator and a sampling rate indicator in the CH section of the IVAS bitstream. The IVAS decoder extracts and decodes an EVS payload in the EP section of the bitstream. The EP section follows the CH section. The IVAS decoder extracts and decodes a metadata payload in the MDP section of the bitstream. The MDP section follows the CH section. In other embodiments, the MDP section follows the EP section of the bitstream or the EP section follows the MDP section of the bitstream. In some embodiments, the IVAS system controls the audio decoder based on the encoding tool, the sampling rate, the EVS payload, and the metadata payload. In other embodiments, the IVAS system stores a representation of the encoding tool, the sampling rate, the EVS payload, and the metadata payload on a non-transitory computer-readable medium. In some embodiments, the IVAS decoder has a device architecture shown in FIG. 8.

[0060] In some embodiments, the IVAS coding tool indicator is a multi-bit data structure. In other embodiments, the IVAS coding tool indicator is a 3-bit data structure, where a first value of the 3-bit data structure corresponds to a multi-mono coding tool, a second value of the 3-bit data structure corresponds to a CACPL coding tool, and a third value of the 3-bit data structure corresponds to another coding tool. In other embodiments, the IVAS coding tool indicator is a 2-bit data structure indicating one to four IVAS coding tools or a 1-bit data structure indicating one or two IVAS coding tools. In other embodiments, the IVAS coding tool indicator includes more than two bits to indicate various IVAS coding tools.

[0061] In some embodiments, the input sampling rate indicator is a multi-bit data structure indicating various input sampling rates. In some embodiments, the input sampling rate indicator is a 2-bit data structure, where a first value of the 2-bit data structure indicates an 8 kHz sampling rate, a second value of the 2-bit data structure indicates a 16 kHz sampling rate, a third value of the 2-bit data structure indicates a 32 kHz sampling rate, and a fourth value of the 2-bit data structure indicates a 48 kHz sampling rate. In other embodiments, the input sampling rate indicator is a 1-bit data structure indicating one or two sampling rates. In other embodiments, the input sampling rate indicator includes three or more bits indicating various sampling rates.

[0062] In some embodiments, the system is 3rd Generation Partnership Project (3GPP) rd As stated in the Generation Partnership Project (TS) technical specification 26.445 in this order, the number of EVS channels, i.e. EVS channel count indicator; bitrate (BR) extraction mode indicator the EVS BR data; and the EVS payload of all channels are stored in or read from the EP section of the bitstream.

[0063] In another embodiment, the system stores or reads the EVS channel number indicator in the EP section of the bitstream.

[0064] In another embodiment, the system stores or reads a bitrate (BR) extraction mode indicator in or from the EP section of the bitstream.

[0065] In another embodiment, the system stores or reads the EVS BR data in the EP section of the bitstream.

[0066] In another embodiment, the system stores or reads the EVS payloads of all channels in the EP section of the bitstream, in that order, as described in 3rd Generation Partnership Project (3GPP) Technical Specification (TS) 26.445.

[0067] In some embodiments, the IVAS system stores or reads from the MDP section of the data stream a coding technique indicator; a number of bands indicator; an indicator indicative of the delay configuration of the filter bank; an indicator of the quantization strategy; an entropy coder indicator; a probability model type indicator; a real coefficient part; an imaginary coefficient part; and one or more coefficients.

[0068] In another embodiment, the IVAS system stores or reads the encoding technique indicator in the MDP section of the data stream.

[0069] In another embodiment, the IVAS system stores or reads the band number indicator in the MDP section of the data stream.

[0070] In another embodiment, the IVAS system includes an interface that indicates the delay configuration of the filter bank. The indicator is stored in or read from the MDP section of the data stream.

[0071] In another embodiment, the IVAS system stores or reads an indicator of the quantization strategy in the MDP section of the data stream.

[0072] In another embodiment, the IVAS system stores or reads the entropy coder indicator in the MDP section of the data stream.

[0073] In another embodiment, the IVAS system stores or reads the probabilistic model type indicator in the MDP section of the data stream.

[0074] In another embodiment, the IVAS system stores or reads the real coefficient parts in or from the MDP section of the data stream.In another embodiment, the IVAS system stores or reads the imaginary coefficient parts in or from the MDP section of the data stream.

[0075] In other embodiments, the IVAS system stores or reads one or more coefficients in an MDP section of the data stream.

[0076] Some examples of IVAS bitstream formats are given below:

[0077] Example IVAS bitstream format-3 subformat In some embodiments, the IVAS bitstream format includes three subdivisions as follows: [Table 2]

[0078] In some embodiments, the parameters within each field in each subdivision and their respective bit allocations are shown below. [Table 3] [Table 4] [Table 5] JPEG2025063033000019.jpg49128

[0079] An advantage of the above-described IVAS bitstream format embodiment is that it efficiently and compactly encodes data supporting a variety of audio service capabilities, including mono-to-stereo upmix and fully immersive audio encoding, decoding and rendering. The embodiments are also supported by a wide range of devices, endpoints, and network nodes, including, but not limited to, mobile and smart phones, electronic tablets, personal computers, conference phones, conference rooms, virtual reality (VR) and augmented reality (AR) devices, home theater devices, and other suitable devices, each of which may have a variety of acoustic interfaces for capturing and rendering sound. The IVAS bitstream format is extensible so that it can easily evolve with IVAS standards and technologies.

[0080] Example IVAS bitstream format-4 subformat The following description of a further embodiment focuses on the differences between this further embodiment and the previously described embodiment. Therefore, features common to both embodiments may be omitted from the following description, and in that case, it should be assumed that the features of the previously described embodiment are implemented or at least can be implemented in this further embodiment (unless the following description requires otherwise). In addition, when a feature is extracted from an embodiment disclosed below and added to a claim, the feature may not be related or closely related to other features of that embodiment.

[0081] In another embodiment, the IVAS bitstream includes four subdivisions as follows: [Table 6]

[0082] In some embodiments, the IVAS encoder determines a coding tool indicator and encodes it in a common header (CTH) section of the IVAS bitstream. The coding tool indicator comprises a value corresponding to the coding tool. The IVAS encoder determines a row index into an IVAS bitrate distribution control table and encodes it in a common spatial coding tool header (CTH) section of the IVAS bitstream. The CTH section is encoded in the CTH header section. The CTH section follows the CH section. The IVAS encoder takes the EVS payload and encodes it into the EVS Payload (EP) section of the IVAS bitstream, which follows the CH section. The IVAS encoder takes the metadata payload and encodes it into the Metadata Payload (MDP) section of the IVAS bitstream, which follows the CH section.

[0083] In some embodiments, the EP section is located before or after the MDP section depending on one or more parameters. In some embodiments, the one or more parameters include a backward compatible mode of mono downmix of a multi-channel input with a nominal bit rate mode, as described in 3GPP TS26.445.

[0084] In some embodiments, the IVAS system stores the IVAS bitstream on a non-transitory computer readable medium. , and streams the bitstream to a downstream device. In some embodiments, the IVAS encoder has the device architecture shown in FIG.

[0085] In some embodiments, the IVAS decoder receives an IVAS bitstream and extracts and decodes audio data encoded in the IVAS format by the IVAS encoder. The IVAS decoder extracts and decodes an encoding tool indicator in a CH section of the IVAS bitstream. The IVAS decoder extracts and decodes an index to an IVAS bitrate distribution control table. The IVAS decoder extracts and decodes an EVS payload in an EP section of the IVAS bitstream, the EP section following the CH section. The IVAS decoder extracts and decodes metadata payload in an MDP section of the IVAS bitstream, the MDP section following the CH section.

[0086] In some embodiments, the EP section is located before or after the MDP section depending on one or more parameters. In some embodiments, the one or more parameters include a backward compatible mode of mono downmix of a multi-channel input with a nominal bit rate mode, as described in 3GPP TS26.445.

[0087] In some embodiments, the IVAS system controls the audio decoder based on the encoding tools, the index to the IVAS bitrate distribution control table, the EVS payload, and the metadata payload. In other embodiments, the IVAS system stores a representation of the encoding tools, the index to the IVAS bitrate distribution control table, the EVS payload, and the metadata payload on a non-transitory computer-readable medium. In some embodiments, the IVAS decoder has the device architecture shown in FIG. 8. [Table 7] [Table 8]

[0088] Metadata Payload (MDP): An advantage of the IVAS bitrate distribution control table is that it records information about the spatial coding modes so that this information does not need to be included in the MDP section. [Table 9]

[0089] EVS Payload (EP): This section of the payload contains EVS encoded bits for one or more audio downmix channels. In some embodiments, the total number of bits in this section is JPEG2025063033000024.jpg539, where N (e.g., N=4) is the number of audio downmix channels required to be encoded, and EVS BR(i) is the calculated EVS bitrate of the ith audio downmix channel, and stride secs is the input stride length in seconds.

[0090] In some embodiments, each table entry in the IVAS Bitrate Distribution Control Table has enough information to extract the bitrate of each EVS instance from the total bits allocated for the EVS. This structure provides the advantage that no additional header information is required in the EVS payload to extract the bits for each EVS instance. [Table 10]

[0091] In some embodiments, the parameters in the IVAS bitrate distribution control table have the following values: [Table 11]

[0092] An example IVAS bitrate distribution control table is as follows: [Table 12] JPEG2025063033000028.jpg192139JPEG2025063033000029.jpg189139JPEG2025063033000030.jpg171140

[0093] Example Decoding of IVAS Bitstream In one embodiment, the steps for decoding an IVAS bitstream are as follows.

[0094] Step 1: Bitstream length and stride secs Calculate the IVAS operating bit rate based on this.

[0095] Step 2: Read out a fixed length CH section that indicates the spatial coding tool.

[0096] Step 3: Based on the IVAS operating bit rate, determine the length of the CTH field by looking up the number of entries of the IVAS operating bit rate (calculated in step 1) in the IVAS bit rate distribution control table.

[0097] Step 3: Once the length of the CTH field is known, read the index offset within the CTH field.

[0098] Step 5: The index offset and the IVAS operating bit rate are used to determine the actual IVAS bit rate distribution control table index.

[0099] Step 6: Read all information about EVS bitrate distribution and mono downmix backward compatibility from the indexed table entry.

[0100] Step 7: If the mono downmix backward compatibility mode is ON, first pass the remaining IVAS bits to the EVS decoder, calculate the bit length of each EVS instance based on its EVS bitrate distribution, read the EVS bits of each EVS instance, decode the EVS bits using the corresponding EVS decoder, and decode the spatial metadata in the MDP section.

[0101] Step 8: If the mono downmix backward compatibility mode is OFF, decode the spatial metadata in the MDP section, calculate the bit length of each EVS instance based on its EVS bitrate distribution, and read and decode the EVS bits of each EVS instance from the EP section of the IVAS bitstream.

[0102] Step 9: Use the decoded EVS output and spatial metadata to configure the input audio format, e.g., stereo (CACPL) or FoA (SPAR), etc.

[0103] An advantage of the above-described IVAS bitstream format embodiment is that it efficiently and compactly encodes data supporting various audio service capabilities, including, but not limited to, mono to stereo upmix and fully immersive audio encoding, decoding and rendering. The embodiment is also supported by a wide range of devices, endpoints, and network nodes, including, but not limited to, mobile and smart phones, electronic tablets, personal computers, conference phones, conference rooms, virtual reality (VR) and augmented reality (AR) devices, home theater devices, and other suitable devices, each of which may have a variety of acoustic interfaces for sound capture and rendering. The IVAS bitstream format is extensible so that it can easily evolve with IVAS standards and technologies.

[0104] Example IVAS SPAR encoding / decoding The following description of a further embodiment focuses on the differences between this further embodiment and the previously described embodiment. Therefore, features common to both embodiments may be omitted from the following description, and in that case, it should be assumed that the features of the previously described embodiment are implemented or at least can be implemented in this further embodiment (unless the following description requires otherwise). In addition, when a feature is extracted from an embodiment disclosed below and added to a claim, the feature may not be related or closely related to other features of that embodiment.

[0105] In some embodiments, the IVAS SPAR encoder determines and encodes a coding mode / tool ​​indicator in a common header (CH) section of the IVAS bitstream. The coding mode / tool ​​indicator has a value corresponding to the coding mode / tool. The IVAS bitstream determines and encodes a mode header / tool ​​header in a tool header (TH) section of the IVAS bitstream, where the TH section follows the CH section. The IVAS SPAR encoder determines and encodes a metadata payload (M The IVAS SPAR encoder determines an Enhanced Voice Services (EVS) payload and encodes it into an EVS Payload (EP) section of the IVAS bitstream, where the EP section follows the CH section. In some embodiments, the IVAS system stores the bitstream on a non-transitory computer-readable medium. In other embodiments, the IVAS system streams the bitstream to a downstream device. In some embodiments, the IVAS SPAR encoder has a device architecture described with reference to FIG. 8.

[0106] In some embodiments, the EP section follows the MDP section. Note that having the EP section follow the MDP section of the IVAS bitstream ensures efficient bit packing, and allowing the number of MDP and EP bits to vary (according to a bitrate distribution algorithm) ensures utilization of all available bits in the IVAS bitrate budget.

[0107] In some embodiments, the IVAS SPAR decoder extracts and decodes an IVAS bitstream encoded in the IVAS SPAR format. The SPAR decoder extracts and decodes the coding mode / tool ​​indicators in the CH section of the bitstream. The coding mode / tool ​​indicators have values ​​that correspond to the coding mode / tool. The IVAS SPAR decoder extracts and decodes the mode header / tool ​​headers in the Tool Header (TH) section of the bitstream. The TH section follows the CH section. The IVAS SPAR decoder extracts and decodes the metadata payload in the MDP section of the bitstream. The MDP section follows the CH section. The IVAS SPAR decoder decodes the EVS payload in the EP section of the bitstream. The EP section follows the CH section.

[0108] In some embodiments, the IVAS system controls the audio decoder based on the encoding mode, the tool header, the EVS payload, and the metadata payload. In other embodiments, the IVAS system stores a representation of the encoding mode, the tool header, the EVS payload, and the metadata payload on a non-transitory computer-readable medium. In some embodiments, the IVAS SPAR decoder has a device architecture as described with reference to FIG.

[0109] In some embodiments, CH comprises a 3-bit data structure, one of the values ​​of the 3-bit data structure corresponds to the SPAR coding mode, and the rest of the values ​​correspond to other coding modes. The 3-bit data structure is advantageous because it allows for a compact code that can indicate up to eight coding modes. In other embodiments, CH comprises fewer than 3 bits. In other embodiments, CH comprises more than 3 bits.

[0110] In some embodiments, the IVAS system stores or reads a row index in the TH section of the IVAS bitstream that points to a row in the SPAR bitrate distribution control table. For example, the row index can be calculated based on the number of rows corresponding to the IVAS operating bitrate as follows: x=ceil(log2(number of rows corresponding to the IVAS bitrate)). Thus, the length of the TH section is variable.

[0111] In some embodiments, the system includes a quantization strategy indicator; a coding strategy indicator; and a quantized and coded implementation of one or more coefficients. The digital and imaginary parts are stored in or read from the MDP section of the IVAS bitstream.

[0112] In another embodiment, the system stores or reads the quantization strategy indicator in the MDP section of the IVAS bitstream.

[0113] In another embodiment, the system stores or reads the encoding strategy indicator in the MDP section of the IVAS bitstream.

[0114] In another embodiment, the system stores or reads quantized and coded real and imaginary parts of one or more coefficients in or from the MDP section of the IVAS bitstream.

[0115] In some embodiments, the one or more coefficients include, but are not limited to, prediction coefficients, inter-prediction coefficients (or direct coefficients), real (diagonal) decorrelator coefficients, and complex (off-diagonal) decorrelator coefficients.

[0116] In some embodiments, more or fewer coefficients are stored in and read from the MDP section of the IVAS bitstream.

[0117] In some embodiments, the IVAS system stores or reads the EVS payload of all channels in the EP section of the IVAS bitstream according to 3GPP TS26.445.

[0118] An example IVAS bitstream using SPAR formatting is shown below: The IVAS bitstream contains four subdivisions as follows: [Table 13]

[0119] Common Header (CH): In some embodiments, the IVAS common header (CH) is formatted as follows: [Table 14]

[0120] Tool Header (TH): In some embodiments, the SPAR Tool Header (TH) is an index offset into the SPAR Bitrate Distribution Control Table. [Table 15]

[0121] An example implementation of the SPAR bitrate distribution control table is shown below. Each IVAS bitrate is determined by the bandwidth (BW), downmix configuration (dmx channels, mx string), active W, complex flag, transition mode value, EVS bitrate setting, metadata quantization level setting, and decorrelator ducking flag. More than one value can be supported. In this exemplary embodiment, there is only one entry per bit rate, so the number of bits in the SPAR TH section is 0. The acronyms used in the table below are defined as follows: PR: prediction coefficient, C: Interprediction coefficients (or direct coefficients), P r : real (diagonal) decorrelator coefficients, P c : complex (off-diagonal) decorrelator coefficients.

[0122] An example SPAR bitrate distribution control table is as follows: [Table 16]

[0123] Metadata Payload (MDP): An example metadata payload (MDP) is as follows: [Table 17]

[0124] EVS Payload (EP): In some embodiments, the quantization of the metadata and the calculation of the actual EVS bitrate of each downmix channel is performed using an EVS bitrate distribution control strategy, one exemplary implementation of which is described below.

[0125] Example EVS bitrate distribution control strategies In some embodiments, the EVS bitrate distribution control strategy includes two sections: metadata quantization and EVS bitrate distribution.

[0126] Metadata Quantization. There are two defined thresholds in this section: the target parameter bitrate threshold (MDtar) and the maximum target bitrate threshold (MDmax).

[0127] Step 1: For each frame, parameters are quantized in a non-temporal differential manner and coded using an entropy coder. In some embodiments, an arithmetic coder is used. In other embodiments, a Huffman encoder is used. If the parameter bit rate estimate is less than MDtar, any extra available bits are fed to the audio encoder to increase the bit rate of the audio essence.

[0128] Step 2: If step 1 fails, a subset of the parameter values ​​in the frame are quantized and subtracted from the quantized parameter values ​​in the previous frame, and the differential quantized parameter values ​​are coded using an entropy coder. If the parameter bit rate estimate is less than MDtar, any extra available bits are fed to the audio encoder to increase the bit rate of the audio essence.

[0129] Step 3: If step 2 fails, the bitrate of the quantized parameters is calculated without entropy.

[0130] Step 4: The results of steps 1, 2, and 3 are compared with MDmax. If the minimum of steps 1, 2, and 3 is within MDmax, the remaining bits are coded and provided to the audio coder.

[0131] Step 5: If step 4 fails, the parameters are quantized more coarsely and the above steps are repeated as the first fallback strategy (Fallback 1).

[0132] Step 6: If step 5 fails, the parameters are quantized using a quantization scheme guaranteed to be within MDmax as a second fallback strategy (Fallback 2). After all the above iterations, the metadata bitrate is guaranteed to be within MDmax and the encoder generates actual metadata bits, i.e. Metadata_actual_bits (MDact).

[0133] EVS Bitrate Distribution (EVSbd). For the purposes of this section, the following definitions apply: EVStar: EVS target bits, the desired bits for each EVS instance. EVSact: EVS actual bits, the sum of the actual bits available for all EVS instances. EVSmin: EVS minimum bits, the minimum bits for each EVS instance. The EVS bitrate must not fall below the value indicated by these bits. EVSmax: EVS maximum bits, the maximum bits for each EVS instance. The EVS bitrate must not exceed the value indicated by these bits. EVS W: An EVS instance that encodes the W channel. EVS Y: An EVS instance that encodes the Y channel. EVS X: An EVS instance encoding the X channel. EVS Z: An EVS instance that encodes the Z channel. EVSact=IVAS_bits-header_bits-MDact

[0134] If EVSact is less than the sum of EVStar of all EVS instances, then bits are taken from the EVS instances in the following order (Z, X, Y, W): Maximum bits that can be taken from any channel = EVStar(ch) - EVSmin(ch).

[0135] If EVSact is greater than the sum of EVStar of all EVS instances, then all additional bits are allocated to the downmix channels in the following order (W, Y, X, Z): Maximum additional bits that can be added to any channel = EVSmax(ch) - EVStar(ch).

[0136] The EVSbd scheme described above calculates the actual EVS bitrates of all channels, i.e. EWa, EYa, EXa, EZa for W, Y, X and Z channels, respectively. After each channel is encoded by a separate EVS instance with EWa, EYa, EXa and EZa bitrates, all the EVS bits are concatenated and packed together. The advantage of this configuration is that no additional header is needed to indicate the EVS bitrate for any channel.

[0137] In some embodiments, the EP section is as follows: [Table 18]

[0138] Example SPAR Decoder Bitstream Unpacking In some embodiments, the steps of SPAR decoder bitstream unpacking are described as follows.

[0139] Step 1: Calculate the IVAS bit rate from the length of the receive bit buffer.

[0140] Step 2: Parse the SPAR TH section based on the number of entries of the IVAS bitrate in the SPAR bitrate distribution control table and extract the index offset, where the index offset is determined by the IVAS operating bitrate.

[0141] Step 3: Use the index offset to determine the actual table row index of the SPAR bitrate distribution control table, and read all columns of the SPAR bitrate distribution control table row pointed to by the actual table row index.

[0142] Step 4: Read the quantization strategy bits and encoding strategy bits from the MDP section of the IVAS bitstream, and unquantize the SPAR spatial metadata in the MPD section based on the indicated quantization strategy and encoding strategy.

[0143] Step 5: Total EVS bitrate (remaining bitrate read from IVAS bitstream) Based on the above-mentioned EVS bitrate distribution (EVSbd), the actual EVS bitrate of each channel is calculated.

[0144] Step 6: Read the encoded EVS bits from the EP section of the IVAS bitstream based on the actual EVS bitrate, and decode each channel of the FoA audio signal using the respective EVS instance.

[0145] Step 7: Use the decoded EVS output and spatial metadata to construct the FoA(SPAR) audio signal.

[0146] An advantage of the above-described embodiment of the IVAS bitstream format is that it efficiently and compactly encodes data supporting various audio service capabilities, including, but not limited to, mono to stereo upmix and fully immersive audio encoding, decoding and rendering (e.g., FoA encoding). This embodiment is also supported by a wide range of devices, endpoints, and network nodes, including, but not limited to, mobile and smart phones, electronic tablets, personal computers, conference phones, conference rooms, virtual reality (VR) and augmented reality (AR) devices, home theater devices, and other suitable devices, each of which may have a variety of acoustic interfaces for sound capture and rendering. The IVAS bitstream format is extensible so that it can easily evolve with IVAS standards and technologies.

[0147] Example Process - IVAS Bitstream in CACPL Format 4A is a flow diagram of an IVAS encoding process 400 according to one embodiment. The process 400 can be implemented using a device architecture such as that described with reference to FIG.

[0148] The process 400 includes determining an encoding tool indicator and a sampling rate indicator using an IVAS encoder and encoding the encoding tool indicator and the sampling rate indicator into a common header (CH) section of an IVAS bitstream (401). In some embodiments, the tool indicator has a value corresponding to the encoding tool and the sampling rate indicator has a value indicative of the sampling rate.

[0149] The process 400 further includes determining an Enhanced Voice Services (EVS) payload using an IVAS encoder and encoding the Enhanced Voice Services (EVS) payload into an EVS payload (EP) section of the IVAS bitstream (402). In some embodiments, the EP section follows the CH section.

[0150] The process 400 further includes determining a metadata payload in the metadata payload using an IVAS encoder and encoding the metadata payload into a metadata payload (MDP) section of the IVAS bitstream (403). In some embodiments, the MDP section follows the CH section. In some embodiments, the EP section follows the MDP section of the bitstream.

[0151] The process 400 further includes storing the IVAS bitstream on a non-transitory computer-readable medium or streaming the IVAS bitstream to a downstream device (404).

[0152] 4B is a flow diagram of an IVAS encoding process 405 using an alternative IVAS format, according to one embodiment. The process 405 may include a device architecture as described with reference to FIG.

[0153] The process 405 includes determining a coding tool indicator using an IVAS encoder and encoding the coding tool indicator into a common header (CH) section of the IVAS bitstream (406). In some embodiments, the tool indicator has a value corresponding to the coding tool.

[0154] The process 405 further includes encoding (407) a representation of the IVAS bitrate distribution control table into a common spatial coding tool header (CTH) section of the IVAS bitstream using an IVAS encoder.

[0155] The process 405 further includes determining a metadata payload using an IVAS encoder and encoding the metadata payload into a metadata payload (MDP) section of the IVAS bitstream (408). In some embodiments, the MDP section follows the CH section of the IVAS bitstream.

[0156] The process 405 further includes determining an Enhanced Voice Services (EVS) payload using an IVAS encoder and encoding (409) the Enhanced Voice Services (EVS) payload into an EVS Payload (EP) section of the IVAS bitstream. In some embodiments, the EP section follows a CH section of the IVAS bitstream. In some embodiments, an MDP section follows the EP section of the IVAS bitstream.

[0157] The process 405 further includes storing the IVAS bitstream on a storage device or streaming the IVAS bitstream to a downstream device (410).

[0158] 5A is a flow diagram of an IVAS decoding process 500 according to one embodiment. The process 500 can be implemented using a device architecture such as that described with reference to FIG.

[0159] The process 500 includes extracting and decoding, using an IVAS decoder, a coding tool indicator and a sampling rate indicator from a common header (CH) section of an IVAS bitstream (501). In some embodiments, the tool indicator has a value corresponding to a coding tool and the sampling rate indicator has a value indicative of a sampling rate.

[0160] The process 500 further includes extracting and decoding (502) an Enhanced Voice Services (EVS) payload from an EVS Payload (EP) section of the IVAS bitstream using an IVAS decoder. In some embodiments, the EP section follows the CH section of the IVAS bitstream.

[0161] The process 500 further includes extracting and decoding (503) the metadata payload from a metadata payload (MDP) section of the bitstream using an IVAS decoder. In some embodiments, the MDP section follows a CH section of the IVAS bitstream. In some embodiments, an EP section follows the MDP section of the IVAS bitstream.

[0162] The process 500 further includes controlling an audio decoder based on the encoding tools, the sampling rate, the EVS payload, and the metadata payload or storing (504) a representation of the encoding tools, the sampling rate, the EVS payload, and the metadata payload on a non-transitory computer-readable medium.

[0163] 5B is a flow diagram of an IVAS decoding process 505 using an alternative format according to one embodiment. The process 505 can be implemented using a device architecture such as that described with reference to FIG.

[0164] The process 505 includes extracting and decoding, using an IVAS decoder, a coding tool indicator in a common header (CH) section of the IVAS bitstream (506). In some embodiments, the tool indicator has a value corresponding to the coding tool.

[0165] The process 505 further includes extracting and decoding (507) using an IVAS decoder a representation of the IVAS bitrate distribution control table within a common spatial coding tool header (CTH) section of the IVAS bitstream.

[0166] The process 505 further includes decoding (508) a metadata payload in a metadata payload (MDP) section of the IVAS bitstream using an IVAS decoder. In some embodiments, the MDP section follows the CH section of the IVAS bitstream.

[0167] The process 505 further includes decoding (509) an Enhanced Voice Services (EVS) payload in an EVS Payload (EP) section of the IVAS bitstream using an IVAS decoder. In some embodiments, the EP section follows a CH section of the IVAS bitstream. In some embodiments, an MDP section follows the EP section of the IVAS bitstream.

[0168] The process 505 further includes controlling an audio decoder based on the representation of the encoding tool indicator, the IVAS bitrate distribution control table, the metadata payload, and the EVS payload, or storing (510) the representation of the encoding tool indicator, the IVAS bitrate distribution control table, the metadata payload, and the EVS payload on a storage device.

[0169] Example Process-SPAR Format IVAS Bitstream 6 is a flow diagram of an IVAS SPAR encoding process 600 according to one embodiment. The process 600 can be implemented using a device architecture such as that described with reference to FIG.

[0170] The process 600 includes decoding, using an IVAS encoder, a coding mode / coding tool indicator and encoding the coding mode / coding tool indicator into a common header (CH) section of an IVAS bitstream (601).

[0171] The process 600 further includes obtaining and encoding (602) a representation of the SPAR bitrate distribution control table using an IVAS encoder into a mode header / tool ​​header in a tool header (TH) section of the IVAS bitstream, where the TH section follows the CH section.

[0172] The process 600 further includes determining, using an IVAS encoder, a metadata payload and encoding the metadata payload into a metadata payload (MDP) section of the IVAS bitstream (603). In some embodiments, the MDP section follows the CH section of the IVAS bitstream.

[0173] In some embodiments, the MDP section includes a quantization strategy indicator; an encoding strategy indicator; and the quantized and encoded real and imaginary parts of one or more coefficients.In some embodiments, the one or more coefficients include, but are not limited to, prediction coefficients, inter-prediction coefficients (or direct coefficients), real (diagonal) decorrelator coefficients and complex (non-diagonal) decorrelator coefficients.In some embodiments, more or less coefficients are stored in the MDP section of the IVAS bitstream and read from the MDP section of the IVAS bitstream.

[0174] The process 600 further includes using an IVAS encoder to determine an Enhanced Voice Services (EVS) payload and encoding the EVS payload into an EVS payload (EP) section of the IVAS bitstream (604). In some embodiments, the EP section of the IVAS bitstream includes EVS payloads for all channels in accordance with 3GPP TS26.445. In some embodiments, the EP section follows the CH section of the IVAS bitstream. In some embodiments, the EP section follows the MDP section. It should be noted that having the EP section follow the MDP section of the IVAS bitstream ensures efficient bit packing, and allowing the number of MDP and EP bits to vary (according to a bitrate distribution algorithm) ensures utilization of all available bits in the IVAS bitrate budget.

[0175] The process 600 further includes storing the bitstream on a non-transitory computer-readable medium or streaming the bitstream to a downstream device (605).

[0176] 7 is a flow diagram of an IVAS SPAR decoding process 700 according to one embodiment. The process 700 can be implemented using a device architecture such as that described with reference to FIG.

[0177] The process 700 includes extracting and decoding (701) a coding mode indicator in a common header (CH) section of an IVAS bitstream using an IVAS decoder.

[0178] The process 700 includes extracting and decoding (702) a representation of the SPAR bitrate distribution control table in a mode header / tool ​​header in a tool header (TH) section of the IVAS bitstream using an IVAS decoder. In some embodiments, the TH section follows the CH section.

[0179] The process 700 further includes extracting and decoding (703) a metadata payload from a metadata payload (MDP) section of the IVAS bitstream using an IVAS decoder. In some embodiments, the MDP section follows the CH section of the IVAS bitstream.

[0180] The process 700 uses an IVAS decoder to extract the Enhanced Voice Services (EVS) payload from the EVS Payload (EP) section of the IVAS bitstream. The method further includes decoding (704) the EP section according to the IVAS bitstream. In some embodiments, the EP section follows the CH section. In some embodiments, the EP section follows the MDP section. Note that by having the EP section follow the MDP section of the IVAS bitstream, efficient bit packing is ensured, and by allowing the number of MDP and EP bits to vary (according to a bitrate distribution algorithm), utilization of all available bits in the IVAS bitrate budget is ensured.

[0181] The process 700 further includes controlling an audio decoder based on the representation of the encoding mode indicator, the SPAR bitrate distribution control table, the EVS payload, and the metadata payload or storing (705) the representation of the encoding mode indicator, the SPAR bitrate distribution control table, the EVS payload, and the metadata payload on a non-transitory computer-readable medium.

[0182] Example System Architecture 8 illustrates a block diagram of an example system 800 suitable for implementing example embodiments of the present disclosure. The system 800 includes one or more server computers or any client devices, including, but not limited to, any of the devices shown in FIG. 1, such as the call server 102, legacy devices 106, user equipment 108, 114, conference room systems 116, 118, home theater systems, VR gear 122, and immersive content ingest 124. The system 800 includes any consumer device, including, but not limited to, a smartphone, a tablet computer, a wearable computer, a vehicle computer, a game console, a surround system, and a kiosk.

[0183] As shown, the system 800 includes a central processing unit (CPU) 801 capable of executing various processes according to programs stored, for example, in a read only memory (ROM) 802 or loaded, for example, from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 includes a number of memory areas that allow the CPU 801 to execute various processes. Data required for executing the program is also stored as necessary. The CPU 801, the ROM 802, and the RAM 803 are connected to one another via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0184] The following components are connected to the I / O interface 805: an input unit 806, which may include a keyboard, a mouse, etc.; an output unit 807, which may include a display, such as a liquid crystal display (LCD), and one or more speakers; a storage unit 808, which may include a hard disk or another suitable storage device; and a communication unit 809, which may include a network interface card, such as a network card (e.g., wired or wireless).

[0185] In some embodiments, the input unit 806 includes one or more microphones in different positions (depending on the host device) that enable capture of audio signals in various formats (e.g., mono, stereo, spatial, immersive, and other suitable formats).

[0186] In some embodiments, the output unit 807 includes a system with a variable number of speakers. As shown in FIG. 1, the output unit 807 can output audio in a variety of formats (e.g., mono, stereo, immersive, binaural, and so on) (depending on the capabilities of the host device). It is possible to render audio signals in a variety of formats (such as 32-bit, 64-bit, 8-bit, 128-bit, and other suitable formats).

[0187] The communication unit 809 is configured to communicate with other devices (e.g., via a network). The drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a flash drive, or another suitable removable medium, is attached to the drive 810 so that a computer program read therefrom is installed in the storage unit 808 as needed. Those skilled in the art will understand that although the system 800 is described as including the above-mentioned components, in practical applications, some of these components can be added, removed, and / or replaced, and all such changes or modifications are all within the scope of the present disclosure.

[0188] Other embodiments In one embodiment, a method for generating a bitstream of an audio signal includes: determining, using an IVAS encoder, a coding tool indicator and a sampling rate indicator, where the coding tool indicator has a value corresponding to a coding tool and the sampling rate indicator has a value indicating a sampling rate; encoding, using the IVAS encoder, the coding tool indicator and the sampling rate indicator into a common header (CH) section of an IVAS bitstream; determining, using the IVAS encoder, an Enhanced Voice Services (EVS) payload; encoding, using the IVAS encoder, the EVS payload into an EVS payload (EP) section of the IVAS bitstream, where the EP section follows the CH section; determining, using the IVAS encoder, a metadata payload; encoding, using the IVAS encoder, the metadata payload into a metadata payload (MDP) section of the IVAS bitstream, where the MDP section follows the CH section; and storing the IVAS bitstream on a non-transitory computer readable medium or streaming the IVAS bitstream to a downstream device.

[0189] In one embodiment, a method for decoding a bitstream of an audio signal includes: extracting and decoding, using an IVAS decoder, an encoding tool indicator and a sampling rate indicator from a CH section of the IVAS bitstream, where the tool indicator has a value corresponding to the encoding tool and the sampling rate indicator has a value indicating the sampling rate; extracting and decoding, using the IVAS decoder, an EVS payload from an EP section of the bitstream, where the EP section follows the CH section; decoding, using the IVAS decoder, a metadata payload from an MDP section of the bitstream, where the MDP section follows the CH section; and controlling an audio decoder based on the encoding tool, the sampling rate, the EVS payload, and the metadata payload or storing representations of the encoding tool, the sampling rate, the EVS payload, and the metadata payload on a non-transitory computer-readable medium.

[0190] In one embodiment, an MDP section follows an EP section in the bitstream, or an EP section follows an MDP section in the bitstream.

[0191] In one embodiment, the IVAS coding tool indicator is a 3-bit data structure, a first value of the 3-bit data structure corresponds to a multi-mono coding tool, a second value of the 3-bit data structure corresponds to a Composite Advanced Coupling (CACPL) coding tool, and a third value of the 3-bit data structure corresponds to another coding tool.

[0192] In one embodiment, the input sampling rate indicator is a 2-bit data structure, a first value of the 2-bit data structure indicating an 8 kHz sampling rate, a second value of the 2-bit data structure indicating a 16 kHz sampling rate, a third value of the 2-bit data structure indicating a 32 kHz sampling rate, and a fourth value of the 2-bit data structure indicating a 48 kHz sampling rate.

[0193] In one embodiment, the method includes storing or retrieving an EVS channel number indicator, a bit rate (BR) extraction mode indicator, EVS BR data, and an EVS payload in or from an EP section of the bitstream, respectively.

[0194] In one embodiment, the method includes storing or retrieving, in an MDP section of the data stream, a coding technique indicator, a number of bands indicator, an indicator indicative of a delay configuration of the filter bank, an indicator of a quantization strategy, an entropy coder indicator, a probability model type indicator, a real coefficient part, an imaginary coefficient part, and one or more coefficients, respectively.

[0195] In one embodiment, a method for generating a bitstream of an audio signal includes determining, using an IVAS encoder, a coding tool indicator, the tool indicator having a value corresponding to a coding tool; encoding, using the IVAS encoder, the coding tool indicator into a common header (CH) section of the IVAS bitstream; determining, using the IVAS encoder, a representation of an index to an IVAS bitrate distribution control table; and encoding, using the IVAS encoder, the representation of the index to the IVAS bitrate distribution control table into a common spatial coding tool header (CTH) section of the IVAS bitstream, the CTH section following the CH section. using the IVAS encoder to determine a metadata payload; using the IVAS encoder to encode the metadata payload into a metadata payload (MDP) section of the IVAS bitstream, where the MDP section follows the CTH section; using the IVAS encoder to determine an Enhanced Voice Services (EVS) payload; using the IVAS encoder to encode the EVS payload into an EVS payload (EP) section of the IVAS bitstream, where the EP section follows the CTH section; and storing the bitstream on a non-transitory computer readable medium or streaming the bitstream to a downstream device.

[0196] In one embodiment, a method for decoding a bitstream of an audio signal includes receiving the bitstream by an IVAS decoder; calculating an IVAS operating bitrate based on a length and stride of the bitstream; reading an indicator of a spatial coding tool from a common header (CH) section of the bitstream; determining a length of a common spatial coding tool header (CTH) section of the bitstream based on the IVAS operating bitrate, which includes looking up a number of entries in an IVAS bitrate distribution control table in the CTH section that correspond to the IVAS operating bitrate; once the length of the CTH section is determined and an index to the IVAS bitrate distribution control table is determined, reading a value in the CTH section; reading information about an Enhanced Voice Services (EVS) bitrate distribution from an entry in the IVAS bitrate distribution control table that corresponds to the index to the IVAS bitrate distribution control table; and providing the information about the EVS bitrate distribution to an EVS decoder.

[0197] In one embodiment, any of the above methods includes reading an indicator of mono downmix backward compatibility with 3GPP TS26.445 from an entry in the IVAS bitrate distribution control table.

[0198] In one embodiment, the method includes determining that a mono downmix backward compatibility indicator is in an ON mode; in response to the ON mode, providing a remaining portion of the bitstream to an EVS decoder; then calculating a respective bitlength of each EVS instance from the remaining portion of the bitstream based on an EVS bitrate distribution; reading EVS bits for each EVS instance based on the corresponding bitlength; providing the EVS bits to the EVS decoder as a first portion, and providing the remaining portion of the bitstream to an MDP decoder for decoding the spatial metadata.

[0199] In one embodiment, the method includes determining that a mono downmix backward compatibility indicator is in an OFF mode; in response to the OFF mode, providing a remainder of the bitstream to an MDP decoder to decode the spatial metadata; then calculating a respective bit length of each EVS instance from the remainder of the bitstream based on an EVS bitrate distribution; reading EVS bits for each EVS instance based on the corresponding bit length; and providing the EVS bits to the EVS decoder as a first portion.

[0200] In one embodiment, a system comprises one or more computer processors; and a non-transitory computer-readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the operations of any one of the method claims above.

[0201] In one embodiment, a non-transitory computer readable medium stores instructions that, when executed by one or more processors, cause the one or more processors to perform the operations of any one of the above method claims.

[0202] According to an exemplary embodiment of the present disclosure, the above-described process can be implemented as a computer software program or on a computer-readable storage medium. For example, an embodiment of the present disclosure includes a computer program product including a computer program tangibly embodied on a machine-readable medium, the computer program including program code for performing the method. In such an embodiment, the computer program can be downloaded and implemented via a communication unit 809 from a network and / or installed from a removable medium 811, as shown in FIG. 8.

[0203] In general, various exemplary embodiments of the present disclosure may be implemented in hardware or dedicated circuitry (e.g., control circuitry), software, logic, or any combination thereof. For example, the units described above may be implemented by control circuitry (e.g., a CPU in combination with other components of FIG. 8), which may then perform the operations described in the present disclosure. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device (e.g., control circuitry). Various aspects of the exemplary embodiments of the present disclosure may be illustrated and described as block diagrams, flow charts, or using some other graphical representation, but the blocks, apparatus, systems, techniques, or methods described herein may be implemented in any manner that is not intended to be limiting. It will be understood that the methods may be implemented in, by way of non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing device, or some combination thereof.

[0204] Additionally, the various blocks illustrated in the flowcharts may be considered as multiple coupled logic circuit elements configured to perform method steps and / or operations and / or related function(s) resulting from the operation of computer program code. For example, embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine-readable medium, the computer program including program code configured to perform a method as described above.

[0205] In the context of this disclosure, a machine-readable medium can be any tangible medium that can contain or store a program used by or in connection with an instruction execution system, an instruction execution apparatus, or an instruction execution device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be non-transitory and can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. More specific examples of machine-readable storage media include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0206] The computer program codes for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These computer program codes can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus having control circuitry, such that the program codes, when executed by a processor of the computer or other programmable data processing apparatus, cause the implementation of the functions / operations specified in the flowcharts and / or block diagrams. The program codes can be executed entirely on the computer or partially on the computer as a stand-alone software package, partly on the computer and partly on a remote computer, entirely on a remote computer or remote server, or distributed across one or more remote computers and / or remote servers.

[0207] Although the specification contains many specific implementation details, these should not be construed as limitations on the scope of what may be claimed, but rather as descriptions of features that may be specific to a particular embodiment. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as operating in a certain combination, and may even be initially claimed as such, one or more features from a claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be subject to subcombinations or variations of the subcombination. The logic flow depicted in the figures may be implemented in the specific order depicted in order to achieve desired results. The following order is not required. Additionally, other steps may be provided or removed from the described flows, and other components may be added or removed from the described systems. Accordingly, other implementations are within the scope of the following claims.

Claims

1. 1. A method for decoding a bitstream of an audio signal, comprising the steps of: determining a frame length of the bitstream; determining a table row index for an immersive voice and audio services (IVAS) bit rate distribution control table based on the determined frame length; accessing the IVAS bitrate distribution control table at the table row index to identify a number of channels and a quantization strategy indicator for configuring an IVAS decoder; extracting and decoding, using an immersive voice and audio service (IVAS) decoder, a coding mode indicator or a coding tool indicator in a common header (CH) section of an IVAS bitstream, the coding mode indicator or the coding tool indicator indicating a coding mode or a coding tool of the audio signal; extracting and decoding mode or tool headers within a tool header (TH) section of the IVAS bitstream using the IVAS decoder, the TH section including data related to the IVAS bitrate distribution control table; extracting and decoding a metadata payload from a metadata payload (MDP) section of the IVAS bitstream using the IVAS decoder, the metadata payload including spatial metadata; determining a quantization level from a set of quantization levels based on the quantization level indicator; dequantizing the spatial metadata in accordance with the determined quantization level; extracting and decoding a payload from a payload section of the bitstream using the IVAS decoder, the payload including coded bits for each channel or each downmix channel of the audio signal; reconstructing or rendering the audio signal for playback on a device based on the coding mode or coding tool indicator, the mode or tool header, the dequantized spatial metadata, and the payload; A method comprising:

2. 2. The method of claim 1, further comprising: controlling an audio decoder of the device for use in reconstructing or rendering the audio signal at the device based on the encoding mode or tool indicator, the mode or tool header, the payload, and the metadata payload, or storing a representation of the encoding mode or tool indicator, the mode or tool header, the payload, and the metadata payload on a non-transitory computer-readable medium.

3. 2. The method of claim 1, wherein the CH is a multi-bit data structure, one value of the multi-bit data structure corresponding to a spatial reconstruction (SPAR) coding mode and another value of the data structure corresponding to another coding mode.

4. 2. The method of claim 1, comprising storing or reading, respectively, an index offset for calculating the table row index of the IVAS bitrate distribution control table in the TH section of the IVAS bitstream.

5. a bitstream encoding strategy indicator; quantized and coded real and imaginary parts of the set of coefficients; 2. The method of claim 1, further comprising storing or reading, respectively, the MDP section of the IVAS bitstream.

6. 2. The method of claim 1, wherein a number of bits in the MDP section of the IVAS bitstream and a number of bits in the payload section of the IVAS bitstream are varied according to the IVAS bitrate distribution control table and a bitrate distribution algorithm to ensure utilization of all available bits in an IVAS bitrate budget.

7. The method of claim 1 , wherein the bit rate of each encoded channel or each downmix channel is determined by a total number of bits available for encoding, the IVAS bit rate distribution control table and a bit rate distribution algorithm.

8. The method of claim 5 , wherein the set of coefficients includes predicted coefficients, direct coefficients, diagonal real coefficients and lower triangular complex coefficients.

9. The method of claim 8 , wherein the prediction coefficients are of variable bit length based on entropy coding, and the direct coefficients, the diagonal real coefficients and the lower triangular complex coefficients are of variable bit length based on downmix construction and entropy coding.

10. The method of claim 1 , wherein the quantization strategy indicator is a multi-bit data structure that indicates the quantization strategy.

11. The method of claim 5 , wherein the bitstream encoding strategy indicator is a multi-bit data structure indicating a number of bands and a non-differential entropy or a temporal differential entropy encoding scheme of the spatial metadata.

12. The method of claim 5 , wherein the coefficients are quantized in an encoder according to an IVAS bitrate distribution control strategy, which includes metadata quantization and IVAS bitrate distribution.

13. determining a bit rate from the IVAS bitstream; determining an index offset based on the TH section of the IVAS bitstream; determining a table row index for the IVAS bitrate distribution control table using the index offset; obtaining quantization strategy bits and encoding strategy bits from the MDP section in the IVAS bitstream; dequantizing the spatial metadata in the MDP section of the IVAS bitstream based on the quantization strategy bits and the encoding strategy bits; determining a bit rate for each channel in the IVAS bitstream using a total number of bits available for encoding, the IVAS bit rate distribution control table, and a bit rate distribution algorithm; obtaining coded bits from the payload section of the IVAS bitstream based on the bit rate; decoding said bits; and Decoding the spatial metadata; and generating an Ambisonics output using the decoded bits and the decoded spatial metadata; and The method of claim 1 further comprising:

14. one or more processors; A non-transitory computer-readable medium storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the operations of the method of claim 1; A system comprising:

15. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the operations of the method of claim 1.

Images