Encoding and decoding of IVAS bitstreams

Abstract

This provides a method for encoding / decoding Immersive Audio Services (IVAS) bitstreams. [Solution] The method includes encoding / decoding an encoding mode indicator in the common header section of an IVAS bitstream; encoding / decoding a mode header or tool header in the tool header section of the bitstream; encoding / decoding a metadata payload in the metadata payload section of the bitstream; encoding / decoding an Extensible Voice Services (EVS) payload in the EVS payload section of the bitstream; storing or streaming the encoded bitstream on the encoder side; and controlling an audio decoder on the decoder side based on the encoding mode, tool header, EVS payload, and metadata payload.

Description

[Technical Field]

[0001] [Cross-reference of related applications] This application claims priority to U.S. Provisional Application No. 62 / 881,541 filed on 1 August 2019, U.S. Provisional Application No. 62 / 927,894 filed on 30 October 2019, U.S. Provisional Application No. 63 / 037,721 filed on 11 June 2020, and U.S. Provisional Application No. 63 / 057,666 filed on 28 July 2020. The full disclosures of these U.S. Provisional Applications are incorporated herein for reference.

[0002] This disclosure, in general, relates to the encoding and decoding of audio bitstreams. [Background technology]

[0003] The development of standards for audio and video encoders / decoders ("codecs") has recently been driven by the increasing demand for immersive voice and audio services (IVAS). The focus is on codec development. IVAS is expected to support a variety of audio service capabilities. These audio service capabilities include, but are not limited to, mono-to-stereo upmixing and fully immersive audio encoding, decoding, and rendering. IVAS is intended to be supported by a wide range of devices, endpoints, and network nodes. These wide range of devices, endpoints, and networks include mobile phones and smartphones, electronic tablets, personal computers, conference phones, conference rooms, and virtual reality (VR) devices. and augmented reality (AR) devices, home theater devices, Other suitable devices may be included, but are not limited to, these. These devices, endpoints, and network nodes may have various acoustic interfaces for sound capture and rendering. [Overview of the project]

[0004] Embodiments for encoding and decoding IVAS bitstreams are disclosed.

[0005] In some embodiments, a method for generating an audio signal bitstream is to use an Immersive Audio Services (IVAS) encoder to determine an encoding mode indicator or encoding tool indicator, wherein the encoding mode indicator or encoding tool indicator indicates the encoding mode or encoding tool of the audio signal; to use the IVAS encoder to encode the encoding mode indicator or encoding tool indicator within a common header (CH) section of the IVAS bitstream; to use the IVAS encoder to determine a mode header or tool header; and to use the IVAS encoder to encode the mode header or tool header into the IVAS bitstream. Encoding within a tool header (TH) section of the bitstream, the TH section following the CH section; using the IVAS encoder to obtain a metadata payload containing spatial metadata; using the IVAS encoder to encode the metadata payload within a metadata payload (MDP) section of the IVAS bitstream, the MDP section following the CH section; using the IVAS encoder to obtain an enhanced voice services (EVS) payload, the EVS payload containing EVS encoded bits for each channel or each downmix channel of the audio signal; and using the IVAS encoder to encode the EVS payload into the IVA This includes encoding within an EVS payload (EP) section of an S bitstream, wherein the EP section follows the CH section.

[0006] In some embodiments, the IVAS bitstream is stored on a non-temporary computer-readable medium. In other embodiments, the IVAS bitstream is streamed to a downstream device, and the encoding mode or encoding tool indicator, the mode header or tool header, the metadata payload and the EVS payload are extracted and decoded from the CH section, TH section, MDP section and EP section of the IVAS bitstream, respectively, for use in reconstructing the audio signal in the downstream device or another device.

[0007] In some embodiments, a method for decoding an audio signal bitstream includes using an Immersive Audio Services (IVAS) decoder to extract and decode an encoding mode indicator or encoding tool indicator within the Common Header (CH) section of the IVAS bitstream, wherein the encoding mode indicator or encoding tool indicator indicates the encoding mode or encoding tool of the audio signal, and using the IVAS decoder to extract and decode a mode header or tool header within the Tool Header (TH) section of the IVAS bitstream, wherein the TH section is within the CH section The process includes: following the CH section; extracting and decoding a metadata payload from the metadata payload (MDP) section of the IVAS bitstream using the IVAS decoder, wherein the MDP section follows the CH section and the metadata payload includes spatial metadata; and extracting and decoding an enhanced voice services (EVS) payload from the EVS payload (EP) section of the IVAS bitstream using the IVAS decoder, wherein the EP section follows the CH section and the EVS payload includes EVS encoded bits for each channel or each downmix channel of the audio signal.

[0008] In some embodiments, the audio decoder of a downstream device, used for reconstructing the audio signal in a downstream device or another device, is controlled based on the encoding mode indicator or the encoding tool indicator, the mode header or the tool header, the EVS payload, and the metadata payload. In other embodiments, representations of the encoding mode indicator or the encoding tool indicator, the mode header or the tool header, the EVS payload, and the metadata payload are stored on a non-temporary computer-readable medium.

[0009] In some embodiments, the bitrate of each EVS coding channel or each downmix channel is determined by all available bits of the EVS, a SPAR bitrate distribution control table, and a bitrate distribution algorithm.

[0010] In some embodiments, the CH is a multibit data structure, where one value of the multibit data structure corresponds to a spatial reconstruction (SPAR) coding mode, and other values ​​of the data structure correspond to other coding modes.

[0011] In some embodiments, the method further includes storing or reading index offsets for calculating row indices in the Spatial Reconstruction (SPAR) bitrate distribution control table in the TH section of the IVAS bitstream, respectively.

[0012] In some embodiments, the method further includes storing a quantization strategy indicator, a bitstream coding strategy indicator, and the quantized and coded real and imaginary parts of a set of coefficients in or reading from the MDP section of the IVAS bitstream, respectively.

[0013] In some embodiments, the set of coefficients includes a prediction coefficient, a direct coefficient, a diagonal real coefficient, and a lower triangular complex coefficient.

[0014] In some embodiments, the prediction coefficient is a variable bit length based on entropy coding, and the direct coefficient, the diagonal real coefficient, and the lower triangular complex coefficient are a variable bit length based on a downmix configuration and entropy coding.

[0015] In some embodiments, the quantization strategy indicator is a multi-bit data structure that represents a quantization strategy.

[0016] In some embodiments, the bitstream coding strategy indicator shows the bandwidth of the spatial metadata and a multi-bit data indicating a non-differential entropy coding scheme or a time-differential entropy coding scheme. It is a data structure.

[0017] In some embodiments, the quantization of the coefficients follows an EVS bitrate distribution control strategy that includes metadata quantization and an EVS bitrate distribution.

[0018] In some embodiments, the method described above further includes storing the EVS payload of an EVS instance in the EP section of the IVAS bitstream, respectively, or reading it from the EP section of the IVAS bitstream, in accordance with the Third Generation Partnership Project (3GPP) Technical Specification (TS) 26.445.

[0019] In some embodiments, the method includes determining a bitrate from the IVAS bitstream, reading an index offset from a spatial reconstruction (SPAR) tool header (TH) section of the IVAS bitstream, using the index offset to determine a table row index of the SPAR bitrate distribution control table, reading quantization strategy bits and coding strategy bits from a metadata payload (MDP) section in the IVAS bitstream, dequantizing SPAR spatial metadata within the MDP section of the IVAS bitstream based on the quantization strategy bits and the coding strategy bits, determining an extended voice service (EVS) bitrate for each channel in the IVAS bitstream using all available EVS bits and the SPAR bitrate distribution control table, reading EVS encoded bits from the EP section of the IVAS bitstream based on the EVS bitrate, decoding the EVS bits, decoding the spatial metadata, and generating a first-order ambisonics (FoA) output using the decoded EVS bits and the decoded spatial metadata.

[0020] Other embodiments disclosed herein relate to systems, apparatuses, and computer-readable media. Details of the disclosed embodiments are set forth in the accompanying drawings and the following description. Other features, objectives, and advantages will be apparent from the following description, drawings, and claims.

[0021] The specific embodiments disclosed herein provide one or more of the following advantages. The disclosed IVAS bitstream format is an efficient and robust bitstream format that supports various audio service capabilities. These audio service capabilities include, but are not limited to, upmixing from monoral to stereo and immersive audio encoding, decoding, and rendering. In some embodiments, the IVAS bitstream format supports complex advance coupling (CACPL) for analyzing and downmixing stereo audio signals. In other embodiments, the IVAS bitstream format supports spatial reconstruction (SPAR) for analyzing and downmixing first order Ambisonics (FoA) audio signals.

[0022] In the drawings, a particular arrangement or ordering of graphical elements, such as elements representing devices, units, instruction blocks, and data elements, is shown for ease of explanation. However, it should be understood by those skilled in the art that a particular ordering or arrangement of these graphical elements in the drawings does not imply that any particular order or sequence is required for processing or that process separation is required. Further, the inclusion of graphical elements in the drawings does not imply that such elements are required in all embodiments, nor does it imply that the features represented by such elements cannot be included in or combined with other elements in some embodiments.

[0023] Furthermore, where connecting elements such as solid or dashed lines or arrows are used in drawings to indicate connections, relationships, or associations between two or more other graphic elements, the absence of such connecting elements does not imply that such connections, relationships, or associations cannot exist. In other words, some connections, relationships, or associations between elements are not shown in the drawings in order to avoid obscuring the disclosure. In addition, for the sake of illustration, a single connecting element is used to represent multiple connections, relationships, or associations between elements. For example, if a connecting element represents the communication of signals, data, or commands, it should be understood by those skilled in the art that such an element may, as necessary, represent one or more signal paths for carrying out the communication. [Brief explanation of the drawing]

[0024] [Figure 1] This figure shows an IVAS system according to one embodiment.

[0025] [Figure 2] This is a block diagram of a system for encoding and decoding an IVAS bitstream according to one embodiment.

[0026] [Figure 3] This is a block diagram of a FoA coder / decoder ("codec") that encodes and decodes an IVAS bitstream in FoA format according to one embodiment.

[0027] [Figure 4A] This is a flowchart of the IVAS coding process according to one embodiment.

[0028] [Figure 4B] This is a flowchart of an IVAS encoding process using an alternative IVAS format according to one embodiment.

[0029] [Figure 5A] This is a flowchart of the IVAS decoding process according to one embodiment.

[0030] [Figure 5B] This is a flowchart of an IVAS decoding process using an alternative IVAS format according to one embodiment.

[0031] [Figure 6] This is a flowchart of the IVAS SPAR coding process according to one embodiment.

[0032] [Figure 7] This is a flowchart of the IVAS SPAR decoding process according to one embodiment.

[0033] [Figure 8] This is a block diagram of an example device architecture according to one embodiment. [Modes for carrying out the invention]

[0034] The same reference numeral used in various drawings indicates the same element.

[0035] In the following detailed description, a great many specific details are given in order to provide a full understanding of the various embodiments described. It will be apparent to those skilled in the art that the various embodiments described can be carried out without these specific details. Otherwise, known methods, procedures, components, and circuits are not described in detail so as not to unnecessarily obscure the aspects of the embodiments. Several features that can be used independently of each other or in any combination of other features are described below.

[0036] nomenclature The terms "include" and their variations as used herein The term "includes, but is not limited to" should be interpreted as an open-ended term. The term "or" should be interpreted as "and / or" unless the context clearly indicates another meaning. The term "based on" should be interpreted as "at least partially based on." The terms "one exemplary embodiment" and "one exemplary embodiment" should be interpreted as "at least one exemplary embodiment." The term "another embodiment" should be interpreted as "at least one other embodiment." The terms "determined," "determines," and "determining" should be interpreted as "to obtain" and "to receive." It should be interpreted as “believe,” “calculate,” “calculate,” “estimate,” “predict,” or “derive.” In addition, in the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art to which this disclosure belongs.

[0037] Overview of the IVAS System Figure 1 shows an IVAS system 100 according to one or more embodiments. In some embodiments, various devices are used, for example, in a public switched telephone network (PSTN) represented by a PSTN / PLMN 104. Communication is conducted through a call server 102 configured to receive audio signals from a device or a public land mobile network (PLMN) device. The IVAS system 100 supports legacy devices 106 that render and capture audio only in mono. These legacy devices include, but are not limited to, devices that support enhanced voice service (EVS), multi-rate wideband (AMR-WB), and adaptive multi-rate narrowband (AMR-NB). No. The IVAS system 100 also supports user equipment (UE) 108, 114 that capture and render stereo audio signals, or UE 110 that captures mono signals and renders them binaurally into multi-channel signals. The IVAS system 100 also supports immersive and stereo signals captured and rendered by video conferencing systems 116, 118, respectively. The IVAS system 100 also supports stereo capture and immersive rendering of stereo audio signals for home theater systems, as well as mono capture and immersive rendering of audio signals for virtual reality (VR) gear 122 and immersive content ingest 124.

[0038] Exemplary IVAS coding / decoding system Figure 2 is a block diagram of a system 200 for encoding and decoding an IVAS bitstream according to one or more embodiments. To perform encoding, the IVAS encoder includes a spatial analysis downmix unit 202 that receives audio data 201. This audio data includes monaural signals, stereo signals, binaural signals, spatial audio signals (e.g., multi-channel spatial audio objects), FoA, higher-order ambisonics (HoA), and other arbitrary audio data. However, it is not limited to these. In some embodiments, the spatial analysis downmix unit 202 implements CACPL for analyzing / downmixing stereo audio signals and / or SPAR for analyzing / downmixing FoA audio signals. In other embodiments, the spatial analysis downmix unit 202 implements other formats.

[0039] The output of the spatial analysis downmix unit 202 includes spatial metadata and 1-4 channels of audio. The spatial metadata is input to the quantization-entropy coding unit 203, which quantizes and entropy codes the spatial data. In some embodiments, the quantization may include fine-grained, medium-grained, coarse, and very coarse quantization strategies, and the entropy coding may include Huffman coding or arithmetic coding. The Extended Voice Services (EVS) coding unit 206 codes the 1-4 channels of audio into one or more EVS bitstreams.

[0040] In some embodiments, the EVS coding unit 206 conforms to 3GPP TS26.445 and provides a wide range of features, such as improved quality and coding efficiency for narrowband (EVS-NB) and wideband (EVS-WB) voice services, improved quality using ultra-wideband (EVS-SWB) voice, improved quality of mixed content and music in conversational applications, robustness to packet loss and delay jitter, and backward compatibility with the AMR-WB codec. In some embodiments, the EVS coding unit 206 includes a preprocessing mode selection unit that selects either a voice coder that encodes a voice signal at a bitrate specified based on the mode / bitrate control unit 207 or a perceptual coder that encodes an audio signal. In some embodiments, the voice encoder is an improved variant of algebraic code-excited linear prediction (ACELP) extended with LP-based modes specialized for different voice classes. In some embodiments, the audio encoder is a modified discrete cosine transform (MDCT) encoder that is highly efficient at low latency / low bitrate and is designed to provide seamless and reliable switching between the speech encoder and the audio encoder.

[0041] In some embodiments, the IVAS decoder includes a quantization entropy decoding unit 204 configured to recover spatial metadata and one or more EVS decoders configured to recover 1-4 channel audio signals. The recovered spatial metadata and audio signals are input to a spatial synthesis / rendering unit 209, which uses the spatial metadata to synthesize / render the audio signals for playback on various audio systems 210.

[0042] Example IVAS / SPAR codec Figure 3 is a block diagram of a FoA codec 300 that encodes and decodes FoA in SPAR format according to several embodiments. The FoA codec 300 includes a SPAR FoA encoder 301, an EVS encoder 305, a SPAR FoA decoder 306, and an EVS decoder 307. The FoA codec 300 converts the FoA input signal into a set of downmix channels and parameters used to regenerate the input signal in decoders 306, 307. The downmix signal can vary between 1 and 4 channels, and the parameters include a prediction coefficient (PR), a cross-prediction coefficient (C), and a decorrelation coefficient (P). It should be noted that SPAR is a process used to reconstruct an audio signal from a downmixed audio signal, using the PR, C, and P parameters, as will be explained in more detail below.

[0043] Note that the exemplary embodiment shown in Figure 3 assumes a passive W channel and depicts a nominal two-channel downmix in which the W channel is sent to the decoder 306 unchanged along with a single predicted channel Y'. In other embodiments, W can be an active channel. An active W channel allows for a certain mix (combination) of the X, Y, and Z channels into the W channel, as follows:

number

[0044] As will be explained in more detail below, the C coefficient allows some of the X and Z channels to be reconstructed from Y', and the remaining channels are reconstructed by the uncorrelated W channels, as will be explained in more detail below.

[0045] In some embodiments, the SPAR FoA encoder 301 includes a passive / active predictor unit 302, a remix unit 303, and an extract / downmix selection unit 304. The passive / active predictor receives FoA channels in 4-channel B format (W, Y, Z, X) and calculates the predicted channels (W or W', Y', Z', X'). Note that the W channel is an omnidirectional polar pattern containing all sounds in the sphere coming from all directions with equal gain and phase, X is a figure-eight bidirectional polar pattern pointing forward, Y is a figure-eight bidirectional polar pattern pointing left, and Z is a figure-eight bidirectional polar pattern pointing upward.

[0046] The extraction / downmix selection unit 304 extracts SPAR FoA metadata from the metadata payload section of the IVAS bitstream, as will be described in more detail below. The passive / active predictor unit 302 and remix unit 303 use the SPAR FoA metadata to generate remixed FoA channels (W or W', A', B', C'), which are input to the EVS encoder 305 and encoded into an EVS bitstream, which is then encapsulated within the IVAS bitstream sent to the decoder 306. In this example, the Ambisonic B format channel is AmbiX Please note that the elements are arranged in the specified format. However, other formats such as the Furse-Malham (FuMa) format (W, X, Y, Z) can also be used.

[0047] Referring to the SPAR FoA decoder 306, the EVS bitstream is decoded by the EVS decoder 307, resulting in N (e.g., N=4) downmix channels. In some embodiments, the SPAR FoA decoder 306 performs the inverse of the operation performed by the SPAR encoder 301. For example, the remixed FoA channels (W or W', A', B', C') are recovered from the N downmix channels using SPAR FoA spatial metadata. The remixed SPAR FoA channels are input to the inverse mixer 311, and the predicted SPAR The FoA channel (W or W', Y', Z', X') is recovered. The predicted SPAR FoA channel is then input to the inverse predictor 312, and the original unmixed SPAR FoA channel (W, Y, Z, X) is recovered. In this two-channel example, decorator blocks 309a (dec1)...309n (dec D Note that this is used to generate an uncorrelated W channel using a time-domain or frequency-domain decorator. The uncorrelated channel is used in conjunction with the SPAR FoA metadata to reconstruct the X and Z channels completely or parameterically.

[0048] In some embodiments, depending on the number of downmix channels, one of the FoA inputs (channel W) is transmitted to the SPAR FoA decoder 306 in its complete state, and one to three of the other channels (Y, Z, and X) are transmitted to the SPAR FoA decoder 306 as residuals or entirely parameterized. The PR coefficient, which remains the same regardless of the number of downmix channels N, is used to minimize the predictable energy in the downmix channels of the residuals. The C coefficient is used to further help regenerate the fully parameterized channels from the residuals. Therefore, the C coefficient is not required in the cases of one and four-channel downmix where there are no residual or parameterized channels to predict. The P coefficient is used to fill in the remaining energy not taken into account by the PR and C coefficients. The number of P coefficients depends on the number of downmix channels N in each band. In some embodiments, the SPAR PR coefficient (passive W only) is calculated as follows:

[0049] Step 1. Predict all side signals (Y, Z, X) from the main W signal using equation [1].

number

number

[0050] Step 2. Remix the W signal and the predicted (Y’, Z’, X’) signals (in this order, the most acoustically related to the least acoustically related). Here, “remix” means a permuted or recombined signal based on a certain methodology. [Number] [3]

[0051] One embodiment of the remix is the permutation of the input signals to W, Y’, X’, Z’ when it is assumed that the audio cues from left and right are more acoustically related than front and back, and the cues from front and back are more acoustically related than the cues from top and bottom.

[0052] Step 3. As shown in equations [4] and [5], calculate the covariance of the 4-channel post-prediction and remix, and downmix it. [Number] [4] [Number] [5] Here, d represents the extra downmix channels beyond W (i.e., the channels from the second channel to the Ndmx-th channel), and u represents the channels that need to be fully regenerated (i.e., the channels from the (Ndmx + 1)-th channel to the fourth channel).

[0053] As an example of the WABC downmix with 1 - 4 channels, d and u represent the following channels shown in Table I. [Table 1]

[0054] The main targets for the calculation of SPAR FoA metadata are the R dd quantity, the R ud quantity, and the R uu quantity. Rdd Amount, R ud Quantity and R uu From the quantity, the system determines whether the remainder of the fully parametric channel can be mutually predicted from the residual channel sent to the decoder. In some embodiments, the required extra C coefficient is given by the following equation.

number

[0055] Therefore, the C parameter has a shape (1×2) in the case of a 3-channel downmix and a shape (2×1) in the case of a 2-channel downmix.

[0056] Step 4. Calculate the remaining energy in the parameterized channel that must be reconstructed by the decorator. Residual energy in the upmix channel Res uu This is the actual energy R uu (Post-prediction) and regenerated mutual prediction energy Reg uu It is the difference between [the two values].

number

number

number

[0057] P is also a covariance matrix and is therefore Hermitian symmetric; thus, only parameters from the upper or lower triangular regions need to be sent to the decoder 306. Diagonal entries are real numbers, while off-diagonal elements may be complex numbers.

[0058] Encoding / decoding of an example IVAS bitstream As illustrated with reference to Figures 2 and 3, the IVAS bitstream(s) are encoded and decoded by the IVAS codec. In some embodiments, the IVAS encoder obtains and encodes the encoding tool indicator and sampling rate indicator within the common header (CH) section of the IVAS bitstream. The encoder has values ​​corresponding to the encoding tool, and the sampling rate indicator has a value indicating the sampling rate. The IVAS encoder obtains the EVS payload and encodes it within the EVS payload (EP: EVS payload) section of the bitstream. The EP section follows the CH section. The IVAS encoder requests the metadata payload and encodes it within the metadata payload (MDP) section of the bitstream. In some embodiments, the MDP section follows the CH section. In other embodiments, the MDP section follows the EP section of the bitstream, or the EP section follows the MDP section of the bitstream. In some embodiments, the IVAS encoder stores the bitstream on a non-temporary computer-readable medium or streams the bitstream to a downstream device. In other embodiments, the IVAS encoder has the device architecture shown in Figure 8.

[0059] In some embodiments, the IVAS decoder receives an IVAS bitstream and extracts and decodes the audio data encoded in IVAS format by the IVAS encoder. The IVAS decoder extracts and decodes the coding tool indicator and sampling rate indicator within the CH section of the IVAS bitstream. The IVAS decoder extracts and decodes the EVS payload within the EP section of the bitstream. The EP section follows the CH section. The IVAS decoder extracts and decodes the metadata payload within the MDP section of the bitstream. The MDP section follows the CH section. In other embodiments, the MDP section follows the EP section of the bitstream, or the EP section follows the MDP section of the bitstream. In some embodiments, the IVAS system controls the audio decoder based on the coding tool, sampling rate, EVS payload, and metadata payload. In other embodiments, the IVAS system stores representations of the coding tool, sampling rate, EVS payload, and metadata payload on a non-temporary computer-readable medium. In some embodiments, the IVAS decoder has the device architecture shown in Figure 8.

[0060] In some embodiments, the IVAS coding tool indicator is a multi-bit data structure. In other embodiments, the IVAS coding tool indicator is a 3-bit data structure, where the first value of the 3-bit data structure corresponds to a multi-mono coding tool, the second value of the 3-bit data structure corresponds to a CACPL coding tool, and the third value of the 3-bit data structure corresponds to another coding tool. In other embodiments, the IVAS coding tool indicator is a 2-bit data structure indicating one to four IVAS coding tools or a 1-bit data structure indicating one or two IVAS coding tools. In other embodiments, the IVAS coding tool indicator contains three or more bits to indicate various IVAS coding tools.

[0061] In some embodiments, the input sampling rate indicator is a multi-bit data structure indicating various input sampling rates. In some embodiments, the input sampling rate indicator is a 2-bit data structure, where the first value of the 2-bit data structure indicates an 8kHz sampling rate, the second value of the 2-bit data structure indicates a 16kHz sampling rate, the third value of the 2-bit data structure indicates a 32kHz sampling rate, and the fourth value of the 2-bit data structure indicates a 48kHz sampling rate. In other embodiments, the input sampling rate indicator is a 1-bit data structure indicating one or two sampling rates. In other embodiments, the input sampling rate indicator includes three or more bits indicating various sampling rates.

[0062] In some embodiments, the system is part of the Third Generation Partnership Project (3GPP:3 rd As described in this order in the technical specification (TS) 26.445 of the generation partnership project, the number of EVS channels, i.e. EVS Channel Count Indicator; Bitrate (BR) Extraction Mode Indicator The EVS BR data and EVS payloads for all channels are stored in or read from the EP section of the bitstream.

[0063] In another embodiment, the system stores or reads an EVS channel count indicator from the EP section of the bitstream.

[0064] In another embodiment, the system stores or reads a bitrate (BR) extraction mode indicator from the EP section of the bitstream.

[0065] In another embodiment, the system stores or reads EVS BR data from the EP section of the bitstream.

[0066] In other embodiments, the system stores or reads the EVS payloads for all channels into or from the EP section of the bitstream, as described in this order in the Third Generation Partnership Project (3GPP) Technical Specification (TS) 26.445.

[0067] In some embodiments, the IVAS system stores or reads from the MDP section of the data stream an encoding technique indicator; a bandwidth indicator; an indicator showing the delay configuration of the filter bank; an indicator of the quantization strategy; an entropy coder indicator; a probabilistic model type indicator; a real part of a coefficient; an imaginary part of a coefficient; and one or more coefficients.

[0068] In another embodiment, the IVAS system stores or reads coding technique indicators from the MDP section of the data stream.

[0069] In another embodiment, the IVAS system stores or reads a bandwidth indicator from the MDP section of the data stream.

[0070] In another embodiment, the IVAS system shows an input indicating a delay configuration of the filter bank. Store the DDP section of the data stream or read it from the MDP section of the data stream.

[0071] In another embodiment, the IVAS system stores or reads indicators of the quantization strategy from the MDP section of the data stream.

[0072] In another embodiment, the IVAS system stores or reads an entropy coder indicator from the MDP section of the data stream.

[0073] In another embodiment, the IVAS system stores or reads probabilistic model type indicators from the MDP section of the data stream.

[0074] In other embodiments, the IVAS system stores the real part of the coefficients in the MDP section of the data stream or reads them from the MDP section of the data stream. In other embodiments, the IVAS system stores the imaginary part of the coefficients in the MDP section of the data stream or reads them from the MDP section of the data stream.

[0075] In another embodiment, the IVAS system stores or reads one or more coefficients from the MDP section of the data stream.

[0076] Some examples of the IVAS bitstream format are shown below.

[0077] Example IVAS bitstream format - 3 subdivision format In some embodiments, the IVAS bitstream format includes three subdivisions, as follows: [Table 2]

[0078] In some embodiments, the parameters within each field in each subdivision and their respective bit assignments are shown below. [Table 3] [Table 4] [Table 5] JPEG2026102670000019.jpg49128

[0079] The advantage of the above-described embodiment of the IVAS bitstream format is that this embodiment efficiently and compactly encodes data that supports various audio service capabilities. These audio service capabilities include mono-to-stereo upmixing as well as fully immersive audio encoding, decoding, and rendering. This embodiment is also supported by a wide range of devices, endpoints, and network nodes. These wide range of devices include, but are not limited to, mobile phones and smartphones, electronic tablets, personal computers, conference phones, conference rooms, virtual reality (VR) and augmented reality (AR) devices, home theater devices, and other suitable devices, each of which may have various acoustic interfaces for sound capture and rendering. The IVAS bitstream format is extensible so that it can be easily developed along with the IVAS standard and technology.

[0080] Example IVAS bitstream format - 4 subdivision format The following description of further embodiments focuses on the differences between these further embodiments and the embodiments described above. Therefore, features common to both embodiments may be omitted from the following description, and where omitted, it should be assumed that the features of the embodiments described above are implemented, or at least can be implemented, in these further embodiments (unless the following description requests otherwise). In addition, when a feature is taken from an embodiment disclosed below and added to a claim, that feature may not be related to or closely related to other features of that embodiment.

[0081] In another embodiment, the IVAS bitstream includes four subdivisions, as follows: [Table 6]

[0082] In some embodiments, the IVAS encoder retrieves and encodes a coding tool indicator within the common header (CH) section of the IVAS bitstream. The coding tool indicator has a value corresponding to the coding tool. The IVAS encoder retrieves a row index to the IVAS bitrate distribution control table and encodes it within the common spatial coding tool header (CTH) of the IVAS bitstream. Encode within the header section. The CTH section follows the CH section. The IVAS encoder obtains the EVS payload and encodes it within the EVS Payload (EP) section of the IVAS bitstream. The EP section follows the CH section. The IVAS encoder obtains the metadata payload and encodes it within the Metadata Payload (MDP) section of the IVAS bitstream. The MDP section follows the CH section.

[0083] In some embodiments, the EP section is located before or after the MDP section, depending on one or more parameters. In some embodiments, as described in 3GPP TS26.445, one or more parameters include a backward-compatible mode for the monaural downmix of a multi-channel input with a nominal bitrate mode.

[0084] In some embodiments, the IVAS system stores the IVAS bitstream on a non-temporary computer-readable medium. In other embodiments, the IVAS system The bitstream is streamed to downstream devices. In some embodiments, the IVAS encoder has the device architecture shown in Figure 8.

[0085] In some embodiments, the IVAS decoder receives an IVAS bitstream and extracts and decodes the audio data encoded in IVAS format by the IVAS encoder. The IVAS decoder extracts and decodes the encoding tool indicators within the CH section of the IVAS bitstream. The IVAS decoder extracts and decodes the index to the IVAS bitrate distribution control table. The IVAS decoder extracts and decodes the EVS payload within the EP section of the IVAS bitstream. The EP section follows the CH section. The IVAS decoder extracts and decodes the metadata payload within the MDP section of the IVAS bitstream. The MDP section follows the CH section.

[0086] In some embodiments, the EP section is located before or after the MDP section, depending on one or more parameters. In some embodiments, as described in 3GPP TS26.445, one or more parameters include a backward-compatible mode for the monaural downmix of multi-channel inputs with nominal bitrate mode.

[0087] In some embodiments, the IVAS system controls the audio decoder based on the encoding tool, the index to the IVAS bitrate distribution control table, the EVS payload, and the metadata payload. In other embodiments, the IVAS system stores representations of the encoding tool, the index to the IVAS bitrate distribution control table, the EVS payload, and the metadata payload on a non-temporary computer-readable medium. In some embodiments, the IVAS decoder has the device architecture shown in Figure 8. [Table 7] [Table 8]

[0088] Metadata Payload (MDP): The advantage of the IVAS bitrate distribution control table is that it records information about the spatial coding mode, eliminating the need to include information about the spatial coding mode in the MDP section. [Table 9]

[0089] EVS Payload (EP): This section of the payload includes EVS encoded bits for one or more audio downmix channels. In some embodiments, the total number of bits in this section is This can be given by JPEG2026102670000024.jpg539, where N (for example, N=4) is the number of audio downmix channels required to encode, and EVS BR(i) This is the calculated EVS bitrate of the i-th audio downmix channel, and stride secs This is the input stride length in seconds.

[0090] In some embodiments, each table entry in the IVAS bitrate distribution control table contains enough information to extract the bitrate for each EVS instance from all the bits allocated for the EVS. This structure has the advantage that no additional header information is required in the EVS payload to extract the bits for each EVS instance. [Table 10]

[0091] In some embodiments, the parameters in the IVAS bitrate distribution control table have the following values. [Table 11]

[0092] The example IVAS bitrate distribution control table is as follows: [Table 12] JPEG2026102670000028.jpg192139JPEG2026102670000029.jpg189139JPEG2026102670000030.jpg171140

[0093] Decryption of an example IVAS bitstream In one embodiment, the steps for decoding the IVAS bitstream are as follows:

[0094] Step 1: Bitstream length and stride secs The IVAS operating bitrate is calculated based on this.

[0095] Step 2: Read out a fixed-length CH section that represents the spatial coding tool.

[0096] Step 3: Based on the IVAS operating bitrate, determine the length of the CTH field by finding the number of entries for the IVAS operating bitrate (calculated in Step 1) in the IVAS bitrate distribution control table.

[0097] Step 3: Once the length of the CTH field is known, read the index offset within the CTH field.

[0098] Step 5: Use the index offset and IVAS operating bitrate to determine the actual IVAS bitrate distribution control table index.

[0099] Step 6: Read all information about EVS bitrate distribution and monaural downmix backward compatibility from the indexed table entries.

[0100] Step 7: If the mono downmix backward compatibility mode is ON, first pass the remaining IVAS bits to the EVS decoder, calculate the bit length of each EVS instance based on its EVS bitrate distribution, read the EVS bits of each EVS instance, decode the EVS bits using the corresponding EVS decoder, and decode the spatial metadata in the MDP section.

[0101] Step 8: If the mono downmix backward compatibility mode is OFF, decode the spatial metadata in the MDP section, calculate the bit length of each EVS instance based on its EVS bitrate distribution, and read and decode the EVS bits of each EVS instance from the EP section of the IVAS bitstream.

[0102] Step 9: Use the decoded EVS output and spatial metadata to configure the input audio format, e.g., stereo (CACPL) or FoA (SPAR).

[0103] The advantage of the above-described embodiment of the IVAS bitstream format is that it efficiently and compactly encodes data supporting a variety of audio service capabilities. These audio service capabilities include, but are not limited to, mono-to-stereo upmixing and fully immersive audio encoding, decoding, and rendering. This embodiment is also supported by a wide range of devices, endpoints, and network nodes. These wide range of devices include, but are not limited to, mobile phones and smartphones, electronic tablets, personal computers, conference phones, conference rooms, virtual reality (VR) and augmented reality (AR) devices, home theater devices, and other suitable devices, each of which may have various acoustic interfaces for sound capture and rendering. The IVAS bitstream format is extensible so that it can be easily developed along with the IVAS standard and technology.

[0104] Example of IVAS SPAR encoding / decoding The following description of further embodiments focuses on the differences between these further embodiments and the embodiments described above. Therefore, features common to both embodiments may be omitted from the following description, and where omitted, it should be assumed that the features of the embodiments described above are implemented, or at least can be implemented, in these further embodiments (unless the following description requests otherwise). In addition, when a feature is taken from an embodiment disclosed below and added to a claim, that feature may not be related to or closely related to other features of that embodiment.

[0105] In some embodiments, the IVAS SPAR encoder determines the encoding mode / tool ​​indicator and encodes it within the common header (CH) section of the IVAS bitstream. The encoding mode / tool ​​indicator has a value corresponding to the encoding mode / tool. The IVAS bitstream determines the mode header / tool ​​header and encodes it within the tool header (TH) section of the IVAS bitstream, where the TH section follows the CH section. The IVAS SPAR encoder determines the metadata payload and encodes it within the metadata payload (M) of the IVAS bitstream. The IVAS SPAR encoder encodes the data within the DP section, where the MDP section follows the CH section. The IVAS SPAR encoder obtains the Extended Voice Services (EVS) payload and encodes it within the EVS Payload (EP) section of the IVAS bitstream, where the EP section follows the CH section. In some embodiments, the IVAS system stores the bitstream on a non-temporary computer-readable medium. In other embodiments, the IVAS system streams the bitstream to downstream devices. In some embodiments, the IVAS SPAR encoder has a device architecture described with reference to Figure 8.

[0106] In some embodiments, the EP section follows the MDP section. Note that by placing the EP section after the MDP section of the IVAS bitstream, efficient bit packing is ensured, and by allowing the number of MDP bits and EP bits to vary (according to the bitrate distribution algorithm), the utilization of all available bits in the IVAS bitrate budget is ensured.

[0107] In some embodiments, the IVAS SPAR decoder extracts and decodes an IVAS bitstream encoded in the IVAS SPAR format. The SPAR decoder extracts and decodes the encoding mode / tool ​​indicator within the CH section of the bitstream. The encoding mode / tool ​​indicator has a value corresponding to the encoding mode / tool. The IVAS SPAR decoder extracts and decodes the mode header / tool ​​header within the tool header (TH) section of the bitstream. The TH section follows the CH section. The IVAS SPAR decoder extracts and decodes the metadata payload within the MDP section of the bitstream. The MDP section follows the CH section. The IVAS SPAR decoder decodes the EVS payload within the EP section of the bitstream. The EP section follows the CH section.

[0108] In some embodiments, the IVAS system controls the audio decoder based on the encoding mode, tool header, EVS payload, and metadata payload. In other embodiments, the IVAS system stores representations of the encoding mode, tool header, EVS payload, and metadata payload on a non-temporary computer-readable medium. In some embodiments, the IVAS SPAR decoder has a device architecture described with reference to Figure 8.

[0109] In some embodiments, CH includes a 3-bit data structure, where one of the values ​​in the 3-bit data structure corresponds to an SPAR coding mode, and the remaining values ​​correspond to other coding modes. The 3-bit data structure is advantageous because it allows for a compact code that can represent up to eight coding modes. In other embodiments, CH includes fewer than 3 bits. In other embodiments, CH includes more than 3 bits.

[0110] In some embodiments, the IVAS system stores or reads from the TH section of the IVAS bitstream a row index that points to a row in the SPAR bitrate distribution control table. For example, the row index can be calculated based on the number of rows corresponding to the IVAS operating bitrate, i.e., x = ceil(log2(number of rows corresponding to the IVAS bitrate)). Thus, the length of the TH section is variable.

[0111] In some embodiments, the system includes a quantization strategy indicator; an encoding strategy indicator; and a quantized and encoded real of one or more coefficients. The part and imaginary part are stored in or read from the MDP section of the IVAS bitstream.

[0112] In another embodiment, the system stores or reads the quantization strategy indicator from the MDP section of the IVAS bitstream.

[0113] In another embodiment, the system stores or reads the coding strategy indicator from the MDP section of the IVAS bitstream.

[0114] In another embodiment, the system stores or reads from the MDP section of the IVAS bitstream the quantized and encoded real and imaginary parts of one or more coefficients.

[0115] In some embodiments, one or more coefficients include, but are not limited to, prediction coefficients, cross-prediction coefficients (or direct coefficients), real (diagonal) decorator coefficients, and complex (off-diagonal) decorator coefficients.

[0116] In some embodiments, more or fewer coefficients are stored in the MDP section of the IVAS bitstream and read from the MDP section of the IVAS bitstream.

[0117] In some embodiments, the IVAS system stores or reads the EVS payloads for all channels in the EP section of the IVAS bitstream in accordance with 3GPP TS26.445.

[0118] An example IVAS bitstream using SPAR formatting is shown below. The IVAS bitstream contains four subdivisions as follows: [Table 13]

[0119] Common Header (CH): In some embodiments, the IVAS common header (CH) is formatted as follows: [Table 14]

[0120] Tool Header (TH): In some embodiments, the SPAR tool header (TH) is an index offset to the SPAR bitrate distribution control table. [Table 15]

[0121] An example embodiment of the SPAR bitrate distribution control table is shown below. Each IVAS bitrate is defined by the bandwidth (BW), downmix configuration (dmx channel, d mx string), active W, complex flag, transition mode value, EVS bitrate setting, metadata quantization level setting and decorator ducking flag 1 It can support more than one value. In this example embodiment, there is only one entry per bitrate, so the number of bits in the SPAR TH section is 0. The acronyms used in the following table are defined as follows: PR: Prediction coefficient, C: Mutual prediction coefficient (or direct coefficient), P r : Real (diagonal) decorator coefficients, P c : Complex (off-diagonal) decorator coefficients.

[0122] An example of an SPAR bitrate distribution control table is as follows: [Table 16]

[0123] Metadata Payload (MDP): An example metadata payload (MDP) is as follows: [Table 17]

[0124] EVS Payload (EP): In some embodiments, the quantization and calculation of the metadata for the actual EVS bitrate of each downmix channel is performed using an EVS bitrate distribution control strategy. An example embodiment of the EVS bitrate distribution control strategy is described below.

[0125] Example EVS bitrate distribution control strategy In some embodiments, the EVS bitrate distribution control strategy includes two sections: metadata quantization and EVS bitrate distribution.

[0126] Metadata quantization. This section contains two defined thresholds: the target parameter bitrate threshold (MDtar) and the maximum target bitrate threshold (MDmax).

[0127] Step 1: For each frame, the parameters are quantized in a non-time-difference manner and encoded using an entropy coder. In some embodiments, an arithmetic coder is used. In other embodiments, a Huffman encoder is used. If the parameter bitrate estimate is less than MDtar, any extra available bits are supplied to the audio encoder to increase the bitrate of the audio essence.

[0128] Step 2: If Step 1 fails, a subset of the parameter values ​​in the frame is quantized and subtracted from the quantized parameter values ​​in the preceding frame, and the difference quantized parameter values ​​are encoded using an entropy coder. If the parameter bitrate estimate is less than MDtar, any extra available bits are supplied to the audio encoder to increase the bitrate of the audio essence.

[0129] Step 3: If Step 2 fails, the bitrate of the quantized parameters is calculated without entropy.

[0130] Step 4: The results of Steps 1, 2, and 3 are compared to MDmax. If the minimum values ​​of Steps 1, 2, and 3 are within MDmax, the remaining bits are encoded and provided to the audio coder.

[0131] Step 5: If Step 4 fails, the parameters are quantized more coarsely, and the above steps are repeated as the first fallback strategy (fallback 1).

[0132] Step 6: If Step 5 fails, the parameters are quantized using a quantization scheme guaranteed to fit within MDmax as a second fallback strategy (fallback 2). After all the iterations described above, the metadata bitrate is guaranteed to fit within MDmax, and the encoder generates the actual metadata bits, i.e., Metadata_actual_bits (MDact).

[0133] EVS bitrate distribution (EVSbd). The following definitions apply to this section. EVStar: EVS target bits, desired bits for each EVS instance. EVSact: EVS Actual Bits, the total number of actual bits available to all EVS instances. EVSmin: The minimum EVS bit, the minimum bit for each EVS instance. The EVS bitrate must not fall below the value indicated by these bits. EVSmax: Maximum EVS bits, the maximum number of bits for each EVS instance. The EVS bitrate must not exceed the value indicated by these bits. An EVS instance that encodes the EVS W:W channel. An EVS instance that encodes an EVS Y:Y channel. An EVS instance that encodes an EVS X:X channel. An EVS instance that encodes the EVS Z:Z channel. EVSact=IVAS_bits-header_bits-MDact

[0134] If EVSact is less than the sum of EVStar across all EVS instances, bits are taken from the EVS instances in the following order (Z, X, Y, W). The maximum number of bits that can be taken from any channel is EVStar(ch) - EVSmin(ch).

[0135] If EVSact is greater than the sum of EVStar across all EVS instances, all additional bits are allocated to the downmix channels in the following order (W, Y, X, Z). The maximum number of additional bits that can be added to any channel is EVSmax(ch) - EVStar(ch).

[0136] The EVSbd method described above calculates the actual EVS bitrates for all channels, i.e., EWa, EYa, EXa, and EZa for the W, Y, X, and Z channels, respectively. After each channel is encoded by a separate EVS instance using the EWa, EYa, EXa, and EZa bitrates, all EVS bits are concatenated and packed together. The advantage of this configuration is that no additional header is required to indicate the EVS bitrate for any channel.

[0137] In some embodiments, the EP section is as follows: [Table 18]

[0138] Example of SPAR decoder bitstream unpacking In some embodiments, the SPAR decoder bitstream unpacking step is described as follows.

[0139] Step 1: Determine the IVAS bitrate from the length of the received bit buffer.

[0140] Step 2: Parse the SPAR TH section based on the number of entries for IVAS bitrate in the SPAR bitrate distribution control table and extract the index offset, where this index offset is determined by the IVAS operating bitrate.

[0141] Step 3: Use the index offset to find the actual table row index of the SPAR bitrate distribution control table, and read all columns of the SPAR bitrate distribution control table row pointed to by this actual table row index.

[0142] Step 4: Read the quantization strategy bits and coding strategy bits from the MPD section of the IVAS bitstream, and unquantize the SPAR space metadata in the MPD section based on the indicated quantization and coding strategies.

[0143] Step 5: Total EVS bitrate (the remaining bitrate read from the IVAS bitstream) Based on the bits, the actual EVS bitrate for each channel is determined for each EVS bitrate distribution (EVSbd) described above.

[0144] Step 6: Read the encoded EVS bits from the EP section of the IVAS bitstream based on the actual EVS bitrate, and decode each channel of the FoA audio signal using each EVS instance.

[0145] Step 7: Use the decoded EVS output and spatial metadata to construct the FoA (SPAR) audio signal.

[0146] The advantage of the above-described embodiment of the IVAS bitstream format is that it efficiently and compactly encodes data supporting a variety of audio service capabilities. These audio service capabilities include, but are not limited to, mono-to-stereo upmixing and fully immersive audio encoding, decoding, and rendering (e.g., FoA encoding). This embodiment is also supported by a wide range of devices, endpoints, and network nodes. These wide range of devices include, but are not limited to, mobile phones and smartphones, electronic tablets, personal computers, conference phones, conference rooms, virtual reality (VR) and augmented reality (AR) devices, home theater devices, and other suitable devices, each of which may have various acoustic interfaces for sound capture and rendering. The IVAS bitstream format is extensible so that it can be easily developed along with the IVAS standard and technology.

[0147] Example process - IVAS bitstream in CACPL format Figure 4A is a flowchart of an IVAS coding process 400 according to one embodiment. Process 400 can be implemented using a device architecture as described with reference to Figure 8.

[0148] Process 400 includes using an IVAS encoder to determine an encoding tool indicator and a sampling rate indicator, and encoding the encoding tool indicator and the sampling rate indicator within the common header (CH) section of the IVAS bitstream (401). In some embodiments, the tool indicator has a value corresponding to the encoding tool, and the sampling rate indicator has a value indicating the sampling rate.

[0149] Process 400 further includes using an IVAS encoder to obtain an Enhanced Voice Services (EVS) payload and encoding the Enhanced Voice Services (EVS) payload within an EVS Payload (EP) section of the IVAS bitstream (402). In some embodiments, the EP section follows the CH section.

[0150] Process 400 further includes using an IVAS encoder to determine the metadata payload in the metadata payload and encoding the metadata payload within the metadata payload (MDP) section of the IVAS bitstream (403). In some embodiments, the MDP section follows the CH section. In some embodiments, the EP section follows the MDP section of the bitstream.

[0151] Process 400 further includes storing the IVAS bitstream on a non-temporary computer-readable medium or streaming the IVAS bitstream to a downstream device (404).

[0152] Figure 4B is a flowchart of an IVAS encoding process 405 using an alternative IVAS format according to one embodiment. Process 405 may include a device architecture as described with reference to Figure 8.

[0153] Process 405 includes using an IVAS encoder to determine an encoding tool indicator and encoding the encoding tool indicator within the common header (CH) section of the IVAS bitstream (406). In some embodiments, the tool indicator has a value corresponding to the encoding tool.

[0154] Process 405 further includes using an IVAS encoder to encode a representation of the IVAS bitrate distribution control table into the Common Space Coding Tool Header (CTH) section of the IVAS bitstream (407).

[0155] Process 405 further includes using an IVAS encoder to obtain a metadata payload and encoding the metadata payload within a metadata payload (MDP) section of the IVAS bitstream (408). In some embodiments, the MDP section follows the CH section of the IVAS bitstream.

[0156] Process 405 further includes using an IVAS encoder to obtain an Enhanced Voice Services (EVS) payload and encoding the Enhanced Voice Services (EVS) payload within an EVS Payload (EP) section of the IVAS bitstream (409). In some embodiments, the EP section follows the CH section of the IVAS bitstream. In some embodiments, the MDP section follows the EP section of the IVAS bitstream.

[0157] Process 405 further includes storing the IVAS bitstream on a storage device or streaming the IVAS bitstream to a downstream device (410).

[0158] Figure 5A is a flowchart of an IVAS decoding process 500 according to one embodiment. Process 500 can be implemented using a device architecture as described with reference to Figure 8.

[0159] Process 500 includes using an IVAS decoder to extract and decode the coding tool indicator and the sampling rate indicator from the common header (CH) section of the IVAS bitstream (501). In some embodiments, the tool indicator has a value corresponding to the coding tool, and the sampling rate indicator has a value indicating the sampling rate.

[0160] Process 500 further includes using an IVAS decoder to extract and decode the Extended Voice Services (EVS) payload from the EVS Payload (EP) section of the IVAS bitstream (502). In some embodiments, the EP section follows the CH section of the IVAS bitstream.

[0161] Process 500 further includes using an IVAS decoder to extract and decode the metadata payload from the metadata payload (MDP) section of the bitstream (503). In some embodiments, the MDP section follows the CH section of the IVAS bitstream. In some embodiments, the EP section follows the MDP section of the IVAS bitstream.

[0162] Process 500 further includes controlling an audio decoder based on an encoding tool, sampling rate, EVS payload, and metadata payload, or storing a representation of the encoding tool, sampling rate, EVS payload, and metadata payload on a non-temporary computer-readable medium (504).

[0163] Figure 5B is a flowchart of an IVAS decoding process 505 using an alternative format according to one embodiment. Process 505 can be implemented using a device architecture as described with reference to Figure 8.

[0164] Process 505 includes using an IVAS decoder to extract and decode the coding tool indicator within the common header (CH) section of the IVAS bitstream (506). In some embodiments, the tool indicator has a value corresponding to the coding tool.

[0165] Process 505 further includes using an IVAS decoder to extract and decode a representation of the IVAS bitrate distribution control table within the Common Space Coding Tool Header (CTH) section of the IVAS bitstream (507).

[0166] Process 505 further includes using an IVAS decoder to decode the metadata payload within the metadata payload (MDP) section of the IVAS bitstream (508). In some embodiments, the MDP section follows the CH section of the IVAS bitstream.

[0167] Process 505 further includes decoding the Extended Voice Services (EVS) payload within the EVS Payload (EP) section of the IVAS bitstream using an IVAS decoder (509). In some embodiments, the EP section follows the CH section of the IVAS bitstream. In some embodiments, the MDP section follows the EP section of the IVAS bitstream.

[0168] Process 505 further includes controlling the audio decoder based on representations of the coding tool indicator, the IVAS bitrate distribution control table, the metadata payload, and the EVS payload, or storing representations of the coding tool indicator, the IVAS bitrate distribution control table, the metadata payload, and the EVS payload on a storage device (510).

[0169] Example process - IVAS bitstream in SPAR format Figure 6 is a flowchart of an IVAS SPAR coding process 600 according to one embodiment. Process 600 can be implemented using a device architecture as described with reference to Figure 8.

[0170] Process 600 includes using an IVAS encoder to decode the encoding mode / encoding tool indicator and encoding the encoding mode / encoding tool indicator within the common header (CH) section of the IVAS bitstream (601).

[0171] Process 600 further includes using an IVAS encoder to obtain a representation of the SPAR bitrate distribution control table and encoding it within the mode header / tool ​​header in the tool header (TH) section of the IVAS bitstream (602), where the TH section follows the CH section.

[0172] Process 600 further includes using an IVAS encoder to obtain a metadata payload and encoding the metadata payload within a metadata payload (MDP) section of the IVAS bitstream (603). In some embodiments, the MDP section follows the CH section of the IVAS bitstream.

[0173] In some embodiments, the MDP section includes a quantization strategy indicator; an encoding strategy indicator; and the quantized and encoded real and imaginary parts of one or more coefficients. In some embodiments, the one or more coefficients include, but are not limited to, prediction coefficients, cross-prediction coefficients (or direct coefficients), real (diagonal) decorator coefficients, and complex (off-diagonal) decorator coefficients. In some embodiments, more or fewer coefficients are stored in and read from the MDP section of the IVAS bitstream.

[0174] Process 600 further includes using an IVAS encoder to determine the Extended Voice Services (EVS) payload and encoding the EVS payload within the EVS Payload (EP) section of the IVAS bitstream (604). In some embodiments, the EP section of the IVAS bitstream contains the EVS payload for all channels in accordance with 3GPP TS26.445. In some embodiments, the EP section follows the CH section of the IVAS bitstream. In some embodiments, the EP section follows the MDP section. Note that by placing the EP section after the MDP section of the IVAS bitstream, efficient bit packing is ensured, and by allowing the number of MDP bits and EP bits to vary (according to the bitrate distribution algorithm), the utilization of all available bits in the IVAS bitrate budget is ensured.

[0175] Process 600 further includes storing the bitstream on a non-temporary computer-readable medium or streaming the bitstream to a downstream device (605).

[0176] Figure 7 is a flowchart of an IVAS SPAR decoding process 700 according to one embodiment. The process 700 can be implemented using a device architecture as described with reference to Figure 8.

[0177] Process 700 includes using an IVAS decoder to extract and decode the coding mode indicator within the common header (CH) section of the IVAS bitstream (701).

[0178] Process 700 includes using an IVAS decoder to extract and decode the representation of the SPAR bitrate distribution control table in the mode header / tool ​​header within the tool header (TH) section of the IVAS bitstream (702). In some embodiments, the TH section follows the CH section.

[0179] Process 700 further includes using an IVAS decoder to extract and decode the metadata payload from the metadata payload (MDP) section of the IVAS bitstream (703). In some embodiments, the MDP section follows the CH section of the IVAS bitstream.

[0180] Process 700 uses the IVAS decoder to extract the Enhanced Voice Services (EVS) payload from the EVS Payload (EP) section of the IVAS bitstream. This further includes decoding (704). In some embodiments, the EP section follows the CH section. In some embodiments, the EP section follows the MDP section. Note that by placing the EP section after the MDP section of the IVAS bitstream, efficient bit packing is ensured, and by allowing the number of MDP bits and EP bits to vary (according to the bitrate distribution algorithm), the use of all available bits in the IVAS bitrate budget is ensured.

[0181] Process 700 further includes controlling an audio decoder based on representations of an encoding mode indicator, a SPAR bitrate distribution control table, an EVS payload, and a metadata payload, or storing representations of an encoding mode indicator, a SPAR bitrate distribution control table, an EVS payload, and a metadata payload on a non-temporary computer-readable medium (705).

[0182] Example system architecture Figure 8 shows a block diagram of an exemplary system 800 suitable for carrying out an exemplary embodiment of the present disclosure. System 800 includes one or more server computers or any client devices. These server computers or client devices include, but are not limited to, any of the devices shown in Figure 1, such as the call server 102, legacy device 106, user equipment 108, 114, conference room systems 116, 118, home theater system, VR gear 122, and immersive content ingest 124. System 800 also includes any consumer devices, which include, but are not limited to, smartphones, tablet computers, wearable computers, vehicle computers, game consoles, surround sound systems, and kiosks.

[0183] As shown in the figure, the system 800 includes a central processing unit (CPU) 801 capable of executing various processes according to, for example, a program stored in a read-only memory (ROM) 802, or a program loaded from a memory unit 808 into a random access memory (RAM) 803. The RAM 803 is used by the CPU 801 to execute various processes. Data required when performing operations is also stored as needed. The CPU 801, ROM 802, and RAM 803 are connected to each other via the bus 804. The input / output (I / O) interface 805 is also connected to the bus 804.

[0184] The following components, namely, an input unit 806 which may include a keyboard, mouse, etc.; an output unit 807 which may include a display such as a liquid crystal display (LCD) and one or more speakers; a storage unit 808 which may include a hard disk or another suitable storage device; and a communication unit 809 which may include a network interface card such as a network card (e.g., wired or wireless), are connected to the I / O interface 805.

[0185] In some embodiments, the input unit 806 includes one or more microphones located at different positions (depending on the host device) that enable the capture of audio signals in various formats (e.g., mono, stereo, spatial, immersive, and other suitable formats).

[0186] In some embodiments, the output unit 807 includes a system having a varying number of speakers. As shown in Figure 1, the output unit 807 can output various formats (e.g., mono, stereo, immersive, binaural, etc.) depending on the capabilities of the host device. It can render audio signals in other suitable formats.

[0187] The communication unit 809 is configured to communicate with other devices (for example, via a network). The drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, optical disk, magneto-optical disk, flash drive, or other suitable removable medium, is mounted on the drive 810 so that computer programs read from there are installed in the storage unit 808 as needed. Those skilled in the art will understand that although the system 800 is described as including the components described above, in actual use it is possible to add, remove, and / or replace some of these components, and all such changes or modifications are all within the scope of this disclosure.

[0188] Other Embodiments In one embodiment, a method for generating an audio bitstream includes: using an IVAS encoder to determine an encoding tool indicator and a sampling rate indicator, wherein the encoding tool indicator has a value corresponding to an encoding tool and the sampling rate indicator has a value indicating the sampling rate; using an IVAS encoder to encode the encoding tool indicator and the sampling rate indicator within a common header (CH) section of an IVAS bitstream; using an IVAS encoder to determine an enhanced voice services (EVS) payload; using an IVAS encoder to encode the EVS payload within an EVS payload (EP) section of an IVAS bitstream, wherein the EP section follows the CH section; using an IVAS encoder to determine a metadata payload; using an IVAS encoder to encode the metadata payload within a metadata payload (MDP) section of an IVAS bitstream, wherein the MDP section follows the CH section; and storing the IVAS bitstream on a non-temporary computer-readable medium or streaming the IVAS bitstream to a downstream device.

[0189] In one embodiment, a method for decoding an audio signal bitstream includes: using an IVAS decoder to extract and decode an encoding tool indicator and a sampling rate indicator from the CH section of the IVAS bitstream, wherein the tool indicator has a value corresponding to the encoding tool and the sampling rate indicator has a value indicating the sampling rate; using an IVAS decoder to extract and decode an EVS payload from the EP section of the bitstream, wherein the EP section follows the CH section; using an IVAS decoder to decode a metadata payload from the MDP section of the bitstream, wherein the MDP section follows the CH section; and controlling the audio decoder based on the encoding tool, sampling rate, EVS payload, and metadata payload, or storing representations of the encoding tool, sampling rate, EVS payload, and metadata payload on a non-temporary computer-readable medium.

[0190] In one embodiment, the MDP section follows the EP section of the bitstream, or the EP section follows the MDP section of the bitstream.

[0191] In one embodiment, the IVAS coding tool indicator is a 3-bit data structure, where the first value of the 3-bit data structure corresponds to a multi-mono coding tool, the second value of the 3-bit data structure corresponds to a composite advanced coupling (CACPL) coding tool, and the third value of the 3-bit data structure corresponds to another coding tool.

[0192] In one embodiment, the input sampling rate indicator is a 2-bit data structure, where the first value of the 2-bit data structure indicates an 8kHz sampling rate, the second value of the 2-bit data structure indicates a 16kHz sampling rate, the third value of the 2-bit data structure indicates a 32kHz sampling rate, and the fourth value of the 2-bit data structure indicates a 48kHz sampling rate.

[0193] In one embodiment, the method includes storing an EVS channel count indicator, a bitrate (BR) extraction mode indicator, EVS BR data, and an EVS payload in or reading from the EP section of the bitstream.

[0194] In one embodiment, the method includes storing an encoding technique indicator, a bandwidth indicator, an indicator showing the delay configuration of the filter bank, an indicator of the quantization strategy, an entropy coder indicator, a probabilistic model type indicator, the real part of a coefficient, the imaginary part of a coefficient, and one or more coefficients in or reading from the MDP section of the data stream.

[0195] In one embodiment, a method for generating an audio bitstream is to use an IVAS encoder to determine an encoding tool indicator, the tool indicator having a value corresponding to an encoding tool; to use an IVAS encoder to encode the encoding tool indicator within a common header (CH) section of the IVAS bitstream; to use an IVAS encoder to determine a representation of an index in an IVAS bitrate distribution control table; and to use an IVAS encoder to encode the representation of an index in an IVAS bitrate distribution control table within a common space encoding tool header (CTH) section of the IVAS bitstream, the CTH section being located after the CH section. The process includes: following; obtaining a metadata payload using an IVAS encoder; encoding the metadata payload within a metadata payload (MDP) section of an IVAS bitstream, the MDP section following the CTH section; obtaining an enhanced voice services (EVS) payload using an IVAS encoder; encoding the EVS payload within an EVS payload (EP) section of an IVAS bitstream, the EP section following the CTH section; and storing the bitstream on a non-temporary computer-readable medium or streaming the bitstream to a downstream device.

[0196] In one embodiment, a method for decoding an audio bitstream includes: receiving the bitstream with an IVAS decoder; calculating the IVAS operating bitrate based on the length and stride of the bitstream; reading indicators for spatial coding tools from the common header (CH) section of the bitstream; determining the length of the common spatial coding tool header (CTH) section of the bitstream based on the IVAS operating bitrate, which includes determining the number of entries in the IVAS bitrate distribution control table in the CTH section corresponding to the IVAS operating bitrate; once the length of the CTH section is determined and the index in the IVAS bitrate distribution control table is determined, reading the values ​​in the CTH section; reading information about the Extended Voice Services (EVS) bitrate distribution from the entries in the IVAS bitrate distribution control table corresponding to the index in the IVAS bitrate distribution control table; and providing information about the EVS bitrate distribution to the EVS decoder.

[0197] In one embodiment, any of the above methods includes reading an indicator of monaural downmix backward compatibility with 3GPP TS26.445 from an entry in the IVAS bitrate distribution control table.

[0198] In one embodiment, the method includes determining that a monaural downmix backward compatibility indicator is in ON mode; providing the remainder of the bitstream to an EVS decoder in response to the ON mode; then calculating the respective bit length of each EVS instance from the remainder of the bitstream based on the EVS bitrate distribution; reading out the EVS bits of each EVS instance based on the corresponding bit length; providing the EVS bits to the EVS decoder as a first part; and providing the remainder of the bitstream to an MDP decoder to decode the spatial metadata.

[0199] In one embodiment, the method includes determining that the monaural downmix backward compatibility indicator is in OFF mode; providing the remainder of the bitstream to an MDP decoder in response to the OFF mode to decode spatial metadata; then calculating the respective bit length of each EVS instance from the remainder of the bitstream based on the EVS bitrate distribution; reading out the EVS bits of each EVS instance based on the corresponding bit length; and providing the EVS bits to an EVS decoder as a first part.

[0200] In one embodiment, the system comprises one or more computer processors; and a non-temporary computer-readable medium that stores instructions that, when executed by the one or more processors, cause the one or more processors to perform any one of the operations described in the method claims.

[0201] In one embodiment, a non-temporary computer-readable medium stores instructions that, when executed by one or more processors, cause one or more processors to perform any one of the operations described in the method claims.

[0202] According to exemplary embodiments of the present disclosure, the processes described above can be implemented as a computer software program or on a computer-readable storage medium. For example, embodiments of the present disclosure include a computer program product which includes a computer program tangibly embodied on a machine-readable medium, the computer program which includes program code that performs the method. In such embodiments, the computer program can be downloaded and implemented from a network via a communication unit 809 and / or installed from a removable medium 811, as shown in Figure 8.

[0203] In general, various exemplary embodiments of the Disclosure can be implemented in hardware or dedicated circuitry (e.g., control circuits), software, logic, or any combination thereof. For example, the unit described above can be implemented by control circuits (e.g., a CPU in combination with the other components in Figure 8), and thus these control circuits can perform the operations described in the Disclosure. Some embodiments can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device (e.g., control circuits). Various embodiments of the exemplary embodiments of the Disclosure are illustrated and described using block diagrams, flowcharts, or other graphic representations, but the blocks, apparatus, systems, techniques or It will be understood that, in non-limiting examples, the method can be implemented using hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or any combination thereof.

[0204] In addition, the various blocks shown in the flowchart can be considered as a plurality of coupled logic circuit elements configured to perform method steps and / or operations and / or associated functions (one or more) resulting from the operation of the computer program code. For example, embodiments of the present disclosure include a computer program product including a computer program tangibly embodied on a machine-readable medium, the computer program including program code configured to perform the methods described above.

[0205] In the context of this disclosure, a machine-readable medium can be any tangible medium capable of containing or storing a program used by or in connection with an instruction execution system, instruction execution apparatus, or instruction execution device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may be non-transient and may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. More specific examples of machine-readable storage media include electrical connections with one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0206] Computer program code for performing the methods of this disclosure can be written in any combination of one or more programming languages. This computer program code can be provided to a general-purpose computer, a dedicated computer, or a processor of another programmable data processing device having control circuits, such that when the program code is executed by the processor of the computer or other programmable data processing device, it causes the execution of functions / operations specified in flowcharts and / or block diagrams. The program code can be executed as a standalone software package, either entirely or partially on a computer, partially on a computer and partially on a remote computer, entirely on a remote computer or remote server, or distributed across one or more remote computers and / or remote servers.

[0207] This specification includes many specific details of implementation, but these should not be construed as limitations on the scope of what can be claimed, but rather as descriptions of features that may be specific to a particular embodiment. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately or in any suitable subcombination in multiple embodiments. Furthermore, features described above as operating in a particular combination may even be initially described in the claims as such, but in some cases one or more features from a claimed combination may be removed from that combination, and the claimed combination may cover subcombinations or variations of subcombinations. The logical flow shown in the figure may also be performed in a specific order shown in order to achieve the desired result. The following order is not mandatory. In addition, other steps may be added to the described flow, steps may be deleted, and other components may be added to or removed from the described system. Therefore, other embodiments are within the scope of the appended claims.

Claims

1. A method for generating an audio signal bitstream, Determining an encoding mode indicator or encoding tool indicator that indicates the encoding mode or encoding tool for the aforementioned audio signal, Encoding the encoding mode indicator or encoding tool indicator into the bitstream, Based on the operating bitrate, determine the table row index of the bitrate distribution control table, Accessing the bitrate distribution control table using the determined table row index, wherein the bitrate distribution control table indicates the number of downmix channels and the set of quantization levels, Determining a metadata payload that includes spatial metadata, wherein determining the metadata payload includes quantizing the spatial metadata based on a set of quantization levels indicated, Encoding the metadata payload into the bitstream, Determining the payload, wherein the payload includes encoded bits for each of the indicated downmix channels of the audio signal, Encoding the payload into the bitstream, A method that includes this.

2. One or more processors, A system comprising: a computer-readable medium that, when executed by one or more processors, stores instructions causing one or more processors to perform the operation according to claim 1.

3. A computer program, when executed by one or more processors, that includes instructions causing the one or more processors to perform the operation according to claim 1.

Images