Method and apparatus for discontinuous transmission in object-based audio codec

Abstract

A method and apparatus for discontinuous transmission (DTX) of audio objects in an object-based audio codec, the audio objects comprising respective audio streams, the apparatus comprising an analyzer for generating audio streams of voice or signal activity information about the audio objects. The DTX controller detects a DTX signal segment of the audio object and an SID frame within the DTX signal segment in response to activity information regarding the audio object. The DTX controller (a) updates a global SID counter for the inactive frames, and (b) signals SID frames within the DTX signal segment depending on the value of the global SID counter. The encoder encodes the SID frame. In an apparatus for decoding audio objects during a discontinuous transmission (DTX) operation, the audio objects each comprise an audio stream having metadata (MD), the metadata (MD) comprising at least one MD parameter, a metadata decoder for decoding the metadata and adjusting a value of the MD parameter to reduce a difference in the MD parameter between frames, and an audio stream decoder for decoding the audio streams.

Description

Technical Field

[0001] The present disclosure relates to audio coding and decoding (code), and more particularly but not exclusively to methods and apparatuses for discontinuous transmission (DTX) in object-based audio codecs.

[0002] In the present disclosure and the appended claims:

[0003] (a) The term "audio" may relate to speech, music, and any other sounds.

[0004] (b) The term "multichannel" may relate to two or more channels.

[0005] (c) The term "stereo" is an abbreviation of "stereophonic".

[0006] (d) The term "mono" is an abbreviation of "monophonic".

[0007] (e) The term "object-based audio" aims to represent an auditory scene as a collection of individual elements (also referred to as audio objects). Additionally, "object-based audio" may include, for example, speech, music, and any other sounds including general audio sounds.

[0008] (f) The term "audio object" aims to specify an audio stream with associated metadata. For example, in the present disclosure, an "audio object" is referred to as an independent audio stream with metadata (ISM).

[0009] (g) The term "audio stream" aims to represent an audio waveform as a bitstream, such as speech, music, and / or any other sounds including general audio sounds, and may consist of one channel (mono), although multichannel including two channels (stereo) may also be considered.

[0010] (h) The term "metadata" aims to represent a collection of information that describes, for example, audio streams and artistic intentions used to translate original or coded audio objects into a reproduction system. Metadata typically describes spatial attributes of each individual audio object, such as position, orientation, volume, width, etc. As a non-limiting example, in the context of the present disclosure, two collections of metadata are considered:

[0011] - Input metadata: An unquantified metadata representation used as input to the codec; the present disclosure is not limited to a specific format of input metadata; and

[0012] - Coded metadata: Quantified and coded metadata that forms part of the bitstream transmitted from the encoder to the decoder.

[0013] (i) The term "audio format" is intended to specify a method for achieving an immersive audio experience.

[0014] (j) The term "reproduction system" is intended to specify an element in a decoder that is capable of rendering audio objects on the reproduction side using the transmitted metadata and artistic intent, such as, but not exclusively, audio objects in a 3D (three-dimensional) audio space around a listener. Rendering can be performed on a target speaker layout (such as 5.1 surround) or headphones, and the metadata can be dynamically modified, for example, in response to feedback from a head tracking device. Other types of rendering are conceivable. Background Art

[0015] Discontinuous transmission (DTX) is used in mobile communication systems to turn off the radio transmitter during pauses in speech or general audio. The use of DTX saves power in the mobile station and increases the time between battery recharges. It also reduces the general interference level, thereby improving the transmission quality. However, during pauses in speech or general audio, if the channel is completely cut off, the background noise that is usually transmitted together with the speech or general audio also disappears. As a result, an unnatural sounding audio signal (silence) is produced at the receiving end of the communication.

[0016] Instead of completely turning off the transmission during pauses in speech or general audio, many techniques have been developed in which parameters characterizing the background noise are generated and transmitted in a silence insertion descriptor (SID) frame bitstream at a low bit rate. These parameters, commonly referred to as comfort noise (CN) parameters, can then be used on the receiver side (decoder) to regenerate as much of the background noise as possible on the transmitter side (encoder), which is related to the spectral and temporal content of the background noise. The process of regenerating the background noise is referred to as comfort noise generation (CNG).

[0017] Historically, session telephony has been implemented with mono handsets that have only one transducer to output sound to only one ear of the user. Therefore, the SID of a mono codec can achieve a low bit rate. In the past decade, users have started using their portable handsets in combination with headphones to receive sound through both of their ears, mainly for listening to music but sometimes also for listening to speech. However, when using a portable handset to send and receive session speech, the content is still mono, but is presented to both ears of the user when using headphones.

[0018] Implementing a codec for Enhanced Voice Services (EVS) using the 3GPP (3rd Generation Partnership Project) voice codec standard, as described in reference [1] (the entire content of which is incorporated herein by reference), the quality of encoding and decoding audio sounds (such as speech, music, and any other sounds transmitted and received via a portable handset) has been significantly improved. The next natural step is to transmit stereo information so that the receiver is as close as possible to the real-life audio scenario captured at the other end of the communication link.

[0019] Furthermore, in the past few years, the generation, recording, representation, encoding / decoding, transmission, and reproduction of audio have been evolving towards enhanced, interactive, and immersive experiences for the listener. An immersive experience can be described as a state of being deeply engaged or involved in an audio scenario, for example, when sounds come from all directions. In immersive audio (also known as 3D (three-dimensional) audio), sound images are reproduced in all three dimensions around the listener, taking into account a wide range of sound characteristics, such as timbre, directionality, reverberation, transparency, and (auditory) spaciousness of accuracy. Immersive audio is produced for specific audio playback or reproduction systems, such as speaker-based systems, integrated reproduction systems (soundbars), or headphones. Then, the interactivity of an audio reproduction system can include, for example, the ability to adjust the sound level, change the position of the sound, or select a different language for reproduction.

[0020] There are three basic methods (also referred to below as audio formats) to achieve an immersive audio experience.

[0021] The first method is channel-based audio, where multiple spaced-apart microphones are used to capture sounds from different directions, and one microphone corresponds to one audio channel in a specific speaker layout. Each recorded channel is provided to a speaker at a specific location. Examples of channel-based audio include, for example, stereo, 5.1 surround sound, 5.1+4, etc.

[0022] The second method is scene-based audio (SBA), which represents the desired sound field in a localized space as a function of time through a combination of dimensional components. The signal representing scene-based audio is independent of the sound source location, and the sound field must be transformed into the selected loudspeaker layout at the rendering reproduction system. An example of scene-based audio is high-fidelity stereo sound reproduction.

[0023] The last third immersive audio method is object-based audio, which represents an auditory scene as a collection of individual audio elements (such as a singer, drums, guitar), accompanied by information about, for example, their position in the audio scene, so that they can be rendered to their intended positions at the reproduction system. This gives object-based audio a great deal of flexibility and interactivity, as each object remains discrete and can be manipulated individually.

[0024] In addition to the basic methods, new multichannel codec technologies are being developed, such as, for example, Metadata-Assisted Spatial Audio (MASA) as described in reference [5], the entire content of reference [5] being incorporated herein by reference. In the MASA method, MASA metadata (e.g., direction, energy ratio, extended coherence, distance, surround coherence, all in several time-frequency time slots) is generated in a MASA analyzer, which is quantized, coded and passed into a bitstream, while the MASA audio channels are treated as (multi)mono or (multi)stereo transport signals coded by a core encoder. At the MASA decoder, the MASA metadata then guides the decoding and rendering process to reconstruct the output spatial sound.

[0025] Each of the above audio methods for achieving an immersive experience has advantages and disadvantages. Thus, it is common that, instead of only one audio method, several audio methods are combined in a complex audio system to create an immersive auditory scene. An example could be an audio system that combines scene-based audio (SBA) or MASA with object-based audio (e.g., SBA or MASA with several discrete audio objects).

[0026] In recent years, 3GPP has started working on developing a 3D audio codec for immersive services called IVAS (Immersive Voice and Audio Service) based on the EVS codec as described in reference [1], the entire content of reference [2] being incorporated herein by reference. The IVAS codec is a multichannel codec where the bitrate generally requires to be higher as the number of coded and transmitted channels increases.

[0027] Therefore, DTX operation in a multichannel codec needs to address the trade-off between (a) keeping the low SID bitrate and (b) using a large number of channels to be represented. For example, if each channel is to be represented by its own SID, the total codec SID bitrate will be too high. Thus, effective DTX methods and SID coding are needed. SUMMARY OF THE INVENTION

[0028] According to a first aspect, the present disclosure relates to a method for discontinuous transmission (DTX) of audio objects in an object-based audio codec, the audio objects comprising respective audio streams, the method comprising: analyzing the audio streams to generate speech or signal activity information about the audio objects; in response to the activity information about the audio objects, detecting DTX signal segments of the audio objects and SID frames within the DTX signal segments, wherein the segment and frame detection comprises (a) updating a global SID counter for inactive frames, and (b) signaling the detected SID frames within the DTX signal segments depending on the value of the global SID counter; and encoding the signaled, detected SID frames using SID frame coding.

[0029] According to another aspect, the present disclosure relates to a device for discontinuous transmission (DTX) of audio objects in an object-based audio codec, the audio objects comprising respective audio streams, the device comprising: an analyzer of the audio streams for generating speech or signal activity information about the audio objects; a DTX controller for detecting DTX signal segments of the audio objects and SID frames within the DTX signal segments in response to the activity information about the audio objects, wherein the DTX controller (a) updates a global SID counter for inactive frames, and (b) signals the detected SID frames within the DTX signal segments depending on the value of the global SID counter; and an encoder for the signaled, detected SID frames using SID frame coding.

[0030] According to another aspect, the present disclosure describes a method for decoding an audio object during a discontinuous transmission (DTX) operation, each audio object comprising an audio stream having metadata (MD), the MD comprising at least one MD parameter, the method comprising: decoding the metadata, including adjusting the value of the MD parameter to reduce the difference in the MD parameter between frames; and decoding the audio stream.

[0031] According to a fourth aspect, the present disclosure discloses a device for decoding an audio object during a discontinuous transmission (DTX) operation, each audio object comprising an audio stream having metadata (MD), the MD comprising at least one MD parameter, the device comprising: a metadata decoder for decoding the metadata, wherein the metadata decoder adjusts the value of the MD parameter to reduce the difference in the MD parameter between frames; and an audio stream decoder for decoding the audio stream.

[0032] The foregoing and other objects, advantages, and features of (a) methods and apparatus for discontinuous transmission (DTX) of audio objects in an object-based audio codec and (b) methods and apparatus for decoding audio objects during discontinuous transmission (DTX) operations will become more apparent upon reading the following non-limiting description of its illustrative embodiments given by way of example only with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In the drawings:

[0034] Figure 1 is a schematic block diagram showing both the DTX transmission method and apparatus implemented in an ISM encoder and the corresponding ISM encoding method;

[0035] Figure 2 shows the Figure 1 flowchart of the SID / DTX logic used in the DTX transmission method and apparatus implemented in the ISM encoder and the corresponding ISM encoding method;

[0036] Figure 3 is a non-limiting example of the structure of an SID bitstream in an object-based audio codec;

[0037] Figure 4 is a schematic block diagram showing both an ISM decoder and the corresponding ISM decoding method;

[0038] Figure 5 is a graph of an example of metadata (azimuth) adjustment during DTX operation; and

[0039] Figure 6 is a simplified block diagram of an example configuration of hardware components that form (a) methods and apparatus for discontinuous transmission (DTX) of audio objects in an object-based audio codec, (b) an ISM encoding method and encoder, (c) methods and apparatus for decoding audio objects during discontinuous transmission (DTX) operations, and / or (d) an ISM decoding method and decoder. DETAILED DESCRIPTION

[0040] The present disclosure describes methods and apparatus for discontinuous transmission (DTX) of audio objects in an object-based audio codec and methods and apparatus for decoding audio objects during discontinuous transmission (DTX) operations.

[0041] Non-limiting illustrative embodiments of methods and apparatuses for discontinuous transmission (DTX) of audio objects in an object-based audio codec and methods and apparatuses for decoding audio objects during discontinuous transmission (DTX) operations, as described below, relate to concepts such as identifying inactive signal segments, determining whether to encode and decode SID frames and associated CN parameters, and encoding, quantizing, and recovering metadata in inactive signal segments in an object-based audio codec.

[0042] In the present disclosure, methods and apparatuses for discontinuous transmission (DTX) of audio objects in an object-based audio codec and methods and apparatuses for decoding audio objects during discontinuous transmission (DTX) operations will be described by way of non-limiting examples only, with reference to the IVAS codec framework (or IVAS audio codec), which is referred to throughout the present disclosure. Specifically, the following IVAS formats are used as references: the ISM format, the OMASA format, and the OSBA format. However, incorporating such DTX techniques into any other audio codec that supports object-based audio is also within the scope of the present disclosure.

[0043] 1. Introduction

[0044] As a non-limiting example, the present disclosure contemplates a framework that supports simultaneous encoding and decoding of several audio objects (e.g., up to 4 audio objects), while considering a fixed total codec bitrate, which is constant, for encoding and decoding the audio objects, including the audio stream with its associated metadata. It should be noted that, for example, in the case of non-diegetic content, metadata is not necessarily transmitted for at least some of the audio objects. Non-diegetic sounds in movies, television shows, and other videos are sounds that the characters cannot hear. A soundtrack is an example of a non-diegetic sound because the only member who hears the music is the listener.

[0045] The present disclosure also contemplates a basic non-limiting example of input metadata consisting of two metadata (MD) parameters, namely azimuth and elevation, which are stored per audio frame for each audio object. In this example, an azimuth range of [1800, -1800] and an elevation range of [900, -900] are considered. However, considering only one or more than two (2) MD parameters with various ranges is within the scope of the present disclosure.

[0046] 2. Object-based encoding and decoding

[0047] Figure 1 FIG. 150 is a schematic block diagram showing a method and an apparatus for discontinuous transmission (DTX) of audio objects in an object-based audio codec implemented within an ISM encoding method and a corresponding ISM encoder.

[0048] 2.1 Input buffer

[0049] Reference Figure 1 , the ISM coding method includes operation 151 of the input buffer. To perform operation 151 of the input buffer, the ISM encoder includes an input buffer 101.

[0050] The input buffer 101 buffers N input audio objects 102, that is, N audio streams with their respective associated N metadata. The N input audio objects 102 (including N audio streams and N metadata associated with each of the N audio streams) are buffered for one frame, for example, a frame 20 ms long. As is well known in the field of audio signal processing, an audio signal is sampled at a given sampling frequency and processed in consecutive blocks of these samples called "frames", and each frame is divided into multiple "sub - frames"

[0051] 2.2 Audio stream analysis and front - end pre - processing

[0052] Still referring Figure 1 , the DTX transmission method 150 includes operation 153 of the analysis and front - end pre - processing of N audio streams. To perform operation 153, the DTX transmission device 100 includes an audio stream processor (analyzer) 103, which is used to sequentially analyze and front - end pre - process the buffered N audio streams, and the buffered N audio streams are respectively transmitted from the input buffer 101 to the audio stream processor (analyzer) 103 through N transmission channels 104.

[0053] The analysis and front - end pre - processing operation 153 performed by the audio stream processor (analyzer) 103 may include at least one of the following sub - operations: time - domain transient detection, spectral analysis, long - term prediction analysis, pitch tracking and voiced speech analysis, voice or signal activity detection (VAD / SAD), bandwidth detection, noise estimation, and signal classification (in non - limiting embodiments, it may include (a) core encoder selection between, for example, ACELP core encoder, TCX core encoder, HQ core encoder, etc., (b) signal type classification between, for example, inactive core encoder type, non - speech core encoder type, voiced speech core encoder type, general core encoder type, transitional core encoder type, and audio core encoder type, (c) language / music classification, etc.). The information obtained from the analysis and front - end pre - processing operation 153 is provided to the configuration and decision processor 106 through line 121. Examples of the foregoing sub - operations are described in reference [1] regarding the EVS codec, and thus will not be further described in the present disclosure.

[0054] 2.2.1 Overview of DTX / CNG operation

[0055] As described above, discontinuous transmission (DTX) and comfort noise generation (CNG) are used in method 150 and corresponding device 100 to reduce the transmission bit rate by simulating background noise during inactive signal periods. In the EVS codec described in reference [1], a conventional DTX / CNG scheme is supported for bit rates up to 24.4 kbps. For higher bit rates, the EVS codec supports a less aggressive DTX / CNG scheme that only switches to comfort noise generation (CNG) for low input signal power.

[0056] The reduction in the transmission bit rate during inactive signal periods is achieved by encoding and decoding a parameter referred to as the comfort noise (CN) parameter. The CN parameter can be transmitted at a fixed or adaptive bit rate during the inactive signal period. As an example, in the EVS codec described in reference [1], the default transmission rate for CNG updates (aka SID_UPDATE_RATE) is set to 8 frames.

[0057] When the EVS codec operates in the DTX / CNG scheme, a signal activity detector (SAD) is used to analyze the input audio signal to determine whether the input audio signal is active or inactive (see the SAD decision in subclause 5.1.12 of reference [1]). Based on its analysis, the SAD detector generates a SAD flag f SAD , whose state indicates whether the input audio signal is active (f SAD = 1) or just background noise (f SAD = 0). When the flag f SAD = 1, conventional encoding and decoding are performed as in the default option. When f SAD = 0, the DTX function runs at the encoder that transmits a silence insertion descriptor (SID) frame or a NO_DATA frame. In the following description, frames with an active input audio signal (f SAD = 1) are referred to as "active frames", while frames with only background noise (f SAD = 0) are referred to as "inactive frames".

[0058] The SID frame contains CN parameters for updating the characteristics of the background noise at the decoder, while the NO_DATA frame is empty. The SID frame in EVS is always encoded using 48 bits (which corresponds to a bit rate of 2.4 kbps).

[0059] As indicated above, the multi-channel IVAS codec is based on the EVS codec. Specifically, the variable bitrate version of the EVS encoder is used as the so-called core encoder. The SID bitrate of the core encoder in IVAS is the same as that of EVS, i.e., 2.4 kbps, and the CNG algorithm of the EVS codec is reused as much as possible in the core encoder of the IVAS codec.

[0060] Generally, the number of core encoders in IVAS is from 1 up to the number of input audio channels (i.e., each audio channel is encoded and decoded by an associated core encoder). In the case of the ISM format, the number N of core encoders is usually equal to the number N of independent audio streams (ISMs) with metadata, and it is then referred to as "discrete ISM" encoding. It should be noted that in the case where the number of core encoders is less than the number of audio objects, "parametric ISM" techniques can also be employed. These "parametric ISM" techniques are generally employed at low bitrates where, due to bitrate constraints, it is not possible to encode and decode the active channels of all audio objects.

[0061] Obviously, encoding and decoding all ISMs in the SID frame will result in a bitrate of N times (2.4 kbps + metadata), which will further result in an overly high IVAS SID bitrate. To keep it reasonably low, the IVAS SID bitrate is set to 5.2 kbps, and efficient encoding and decoding of the SID frame are introduced.

[0062] 2.2.2 Classification of Inactive Frames in the ISM Encoder

[0063] As described above, the audio stream processor (analyzer) 103 analyzes the audio streams passed by the respective channels 104 from the input buffer 101, classifies the input audio objects, and generates voice or signal activity detection (VAD / SAD) flags f SAD , one for each audio object. A change from f SAD = 1 to f SAD = 0 indicates the start of an inactive signal segment for a particular audio object. Several VAD / SAD flag variants can be used, such as the "local VAD" flag described in section 5.1.12 of reference [1]. Obviously, the start of an inactive signal segment usually occurs at different times in different audio objects.

[0064] By default, in an object-based audio system, when an inactive signal segment is declared for all audio objects, a DTX signal segment is declared. Then, an additional classification stage is used to classify the inactive signal segments. Additionally, the value of the metadata has an impact on whether a frame is declared as an inactive frame. Figure 2The classification logic of inactive (SID or NO_DATA) frame or active frame is shown in FIG.

[0065] 2.2.3 Global SID Counter, DTX Flag, SID Flag

[0066] Figure 2 It is shown at the same time Figure 1 1. The DTX transmission method 150 and the DTX control operation 250 used in the device 100 and the corresponding DTX controller 200 are implemented in the ISM encoder and the corresponding ISM encoding method.

[0067] refer to Figure 2 To control the SID / NO_DATA frame decision and the start of the DTX signal segment, the DTX controller 200 uses the global SID counter cnt of the inactive frame. SID The global SID counter cnt SID Allows to control the SID update rate and synchronize SID / NO_DATA frames across all audio objects. Therefore, the global SID counter cnt SID This enables efficient tuning of potentially hung (hysteresis) DTX logic, or allows SID update rates other than the default SID update rate of, for example, 8 frames or the SID adaptive update rate.

[0068] In the disclosed logic, the global SID counter cnt SID (parameter "cnt_SID_ISM" in the source code at the end of this disclosure) is therefore superior to the per-audio-object SID counter, and it effectively ensures that a single per-audio-object SID counter is synchronized.

[0069] refer to Figure 2 , the DTX controller 200 receives information obtained from the analysis and preprocessing operation 153 from line 121, including VAD information. The DTX controller 200 then sets the DTX flag flag DTX and SID flag SID Initialized to "0" (block 201).

[0070] By default, when the VAD flag of all audio objects is VAD When the DTX controller 200 detects the DTX signal segment (SID or NO_DATA frame) of the audio object (block 202), the DTX controller 200 detects the DTX signal segment (SID or NO_DATA frame) of the audio object by setting the DTX flag flag. DTX = 1 (parameter "dtx_flag" in the source code at the end of this disclosure) to signal (block 203) the VAD flag flag of all audio objects VAD is equal to 0. This can be expressed using relation (1):

[0071]

[0072] Among them, as described above, flag DTX is the DTX flag. When the VAD flag flag VAD of all audio objects is not equal to 0 (block 202), the DTX controller 200 signals an active frame (block 210) and selects active frame encoding and decoding.

[0073] In addition, the DTX controller 200 uses the SID flag flag SID (parameter "sid_flag" in the source code at the end of this disclosure) to signal the SID frame within the DTX signal segment. (a) When the global SID counter cnt SID is equal to 0 (block 204), the SID flag flag SID is set to 1 (block 205). In this case, the DTX controller 200 signals the SID frame within the DTX signal segment (block 207) and selects the SID frame encoding and decoding of the signaled SID frame; (b) When the global SID counter cnt SID is not equal to 0 (block 204), the SID flag flag SID is set to 0 (block 204). In this case, the DTX controller 200 signals the NO_DATA frame within the DTX signal segment (block 209) and selects the NO_DATA frame encoding and decoding of the signaled NO_DATA frame. This can be expressed using equation (2):

[0074]

[0075] In Figure 2 block 206, the DTX controller 200 resets the SID counter cnt SID to -1 in each active frame and increments the SID counter cnt SID by 1 in each inactive frame until a value corresponding to the SID update rate (in the example of the above embodiment, default is 8 frames). If the value of the counter cnt SID reaches the SID update rate, it is reset to 0, and the DTX controller 200 signals the SID frame within the DTX signal segment (blocks 204, 205, and 207) and selects the SID frame encoding and decoding of the signaled SID frame.

[0076] The DTX controller 200 can change the DTX flag flag DTX by using other classification phases (block 208) based on the core encoder preprocessing values of all audio objects.

[0077] As a non - limiting example, the DTX controller 200 may include logic that, when 1) the average value of the LT (long - term) background noise, noise, over all audio objects mean is higher than a first threshold β1, or 2) the average value of the LT background noise, noise, over all audio objects mean is higher than a second threshold β2 and the change in the LT background noise, noise_var, over all audio objects mean is higher than a third threshold β3, forces active - frame encoding / decoding (flag DTX = 0) and selects active - frame encoding / decoding (block 210). This can be expressed using relation (3):

[0078] flag DTX = 0 if noise mean > β1 or (noise mean > β2 and noise_var mean > β3) (3)

[0079] The thresholds are found experimentally and can be set, for example, as β1 = 50, β2 = 10, and β3 = 2. The introduction of the evaluation of the LT background - noise change is to check the background - noise similarity between all audio objects. The parameter noise mean represents the average value of the long - term background - noise energy values of all audio objects (see paragraph 5.1.11 in reference [1]). Then, the parameter noise_var mean represents the change in the long - term background - noise energy values of all audio objects.

[0080] Also, in the IVAS codec, similar logic to that in the EVS codec is used; for higher bit - rates, the IVAS codec supports a less aggressive DTX / CNG scheme that switches to CNG only for low input - signal power.

[0081] As another non - limiting example of the decision regarding the inactive signal segment, the DTX controller 200 includes logic for evaluating the energy of the background noise in all audio channels 104 (audio streams). Generally, when there is one audio object with high - energy - dominant background noise and other audio objects have low - energy background noise, the DTX flag is set to 1 (flag DTX = 1), otherwise it is set to 0 (flag DTX = 0), which means that DTX is not triggered when there is more than one audio object with high - energy noise (active - frame encoding / decoding (block 210) is selected).

[0082] 2.3 Metadata Analysis, Quantization, and Coding

[0083] ISM encoding method for encoding and decoding object - based audio signals Figure 1 The ISM encoding method also includes operations 155 of metadata analysis, quantization, and encoding / decoding. To perform operation 155, the ISM encoder for encoding and decoding object - based audio signals includes a metadata processor 105.

[0084] The analysis, quantization, and encoding / decoding of metadata in non - DTX operations can be performed, for example, as described in reference [3], the entire content of which is incorporated herein by reference. It should be noted that in the described non - restrictive illustrative embodiments, Figure 1 the metadata processor 105 sequentially quantizes and encodes / decodes the metadata 140 of N audio objects 102 in a loop, and a certain dependency can be adopted between the quantization of audio objects and the metadata parameters of these audio objects.

[0085] As described above, in the present disclosure, in an exemplary embodiment, two metadata parameters, azimuth and elevation (as included in N input metadata), are considered. As a non - restrictive example, the metadata processor 105 includes a quantizer (not shown) using the following metadata parameter indices with the following example resolutions to reduce the number of bits used:

[0086] - Azimuth parameter: The 12 - bit azimuth parameter index from the input metadata file is quantized to an index of B az bits (e.g., B az = 7). Assuming minimum and maximum azimuth limits (- 180 and + 180°), the quantization step of the (B az = 7) - bit uniform scalar quantizer is 2.835°.

[0087] - Elevation parameter: The 12 - bit elevation parameter index from the input metadata file is quantized to an index of B el bits (e.g., B el = 6). Assuming minimum and maximum elevation limits (- 90° and + 90°), the quantization step of the (B el = 6) - bit uniform scalar quantizer is 2.857°.

[0088] Once the azimuth and elevation parameter indices are quantized, they can be encoded / decoded by a metadata encoder (not shown) of the metadata processor 105 using absolute or differential encoding / decoding ( Figure 1Encoding and decoding are performed on 112) therein. As is known in the art, absolute encoding and decoding means that the current value of a parameter is encoded and decoded. Differential encoding and decoding means that the difference between the current value and the previous value of a parameter is encoded and decoded. Since the indices of the azimuth and elevation parameters typically evolve smoothly (i.e., the changes in azimuth or elevation positions can be considered continuous and smooth), differential encoding and decoding are used by default because they generally consume fewer bits compared to absolute encoding and decoding.

[0089] 2.3.1 Metadata Analysis, Quantization, and Encoding / Decoding in SID Frames

[0090] As indicated above, the bit budget in SID frames is relatively low. In the case of the IVAS codec, the SID bit rate is 5.2 kbps, with roughly half of it reserved for the metadata (MD) payload. To transmit as many MD values as possible, some compromises are made.

[0091] This disclosure is based on forcing the MD encoding / decoding of the metadata processor 105 in the SID frame to be an absolute encoding / decoding method. Note that this is different from the MD encoding / decoding in the active frame that uses absolute / differential encoding / decoding. The motivation for using absolute encoding / decoding alone in the SID frame is to avoid potential degradation in long segments of inactive frames due to the loss of SID frames in the case of a noisy channel if differential encoding / decoding is used.

[0092] Then, one possibility to match the bit budget constraint is to reduce the resolution of the MD values. However, this would mean that the MD reconstruction at the decoder would not be precise enough, resulting in subjective quality degradation. Therefore, this disclosure introduces several mechanisms to overcome these constraints.

[0093] First, the DTX controller 200 makes the resolution of the MD values depend on the number of audio objects. Specifically, when the number of audio objects is low, the available bit budget for encoding and decoding the metadata (MD) is relatively large, and the MD resolution thus remains relatively high (e.g., the same as in the active frame). On the other hand, when the number of audio objects being encoded and decoded is high, the resolution of the MD values is relatively low.

[0094] In an exemplary embodiment considering the azimuth and elevation MD parameters, the metadata processor 105 is set to encode:

[0095] a) The azimuth and elevation indices, for example, by means of B in a system with one or two audio objects az = 8 bits and B el = 7 bits,

[0096] b) The azimuth and elevation indices, for example, by means of B in a system with three or four audio objects az= 6 bits and B el = 5 bits,

[0097] whereas the actual number of coded / decoded bits and the resolution are known explicitly from the ISM common signaling (see Figure 1 113 and paragraph 2.8 below).

[0098] Second, in order to keep the SID bit budget as low as possible, savings in the MD bit budget are achieved by means of - calculation flags (one flag per audio object) that indicate that the MD parameters have not changed (or have not changed significantly) since the last frame, and thus the metadata (MD) parameters for a particular audio object are not coded and transmitted. Similarly, this flag serves as an indication that there are no input MD parameters for this particular audio object.

[0099] Return reference Figure 2 , in an example embodiment, the DTX controller 200 calculates an MD on / off flag flag MD (parameter "diff_flag" in the source code at the end of this disclosure) (block 213) for all MD parameter values. For example, the flag flag is calculated for the azimuth MD parameter using the following relation (4) MD,θ :

[0100]

[0101] where θ is the azimuth of the current frame, θ last is the azimuth of the last frame and δ θ is the maximum azimuth difference value, e.g., δ θ = 15. In the same way (see relation (4)), the DTX controller (block 213) calculates the flag for the elevation MD parameter (or any other MD parameter) The final MD on / off flag flag for an audio object MD can be obtained as the logical OR (symbol V in relation (5) below) between all the specific MD flags:

[0102]

[0103] Finally, the flag flag, which represents 1-bit information per audio object MD 214 is inserted into the bitstream in the SID frame 207, immediately after the ISM signaling of the number N of coded / decoded audio objects 301 (see paragraph 2.8.1 below).

[0104] Third, the DTX controller includes a mechanism for estimating the bit budget bits MD for metadata quantization ( Figure 2of blocks 211 and 212). It should be noted that at this stage, only the quantization bit budget of the metadata value is estimated (pre-computed), and the quantization itself may be performed only later. When the estimated bit budget bits MD is higher than the maximum available bit budget bits for MD coding / decoding available (box 212), the flag flag DTX is reset to 0 (box 215), and active frame coding / decoding is performed (box 210). On the other hand, when the estimated MD bit budget bits MD is lower than or equal to the maximum available bit budget bits for MD coding / decoding available (box 212), the flag flag DTX is not changed, and the DTX coding / decoding segment continues. The maximum available bit budget bits for MD coding / decoding available is calculated as the difference between the codec SID bit budget and the bit budget required for other than MD coding / decoding in the SID frame (e.g., SID frame signaling, core encoder SID bit budget, spatial information bit budget, ISM signaling).

[0105] This logic stems from the premise that the CNG frame does not represent important audio details, while the metadata (MD) is considered important for representing the output audio details and is thus transmitted to the decoder. When there are more audio objects, the probability of switching to active frame coding / decoding (box 210) is significantly higher. This logic also becomes more important when more metadata (other than just azimuth and elevation) is present and needs to be coded / decoded.

[0106] 2.4.1 Bit Rate Configuration and Decision for Each Channel

[0107] Reference Figure 1 , the ISM coding method includes operations 156 for the configuration and decision of the bit rate for each transmission channel 104. To perform operation 156, the ISM encoder includes a configuration and decision processor 106 that forms a bit budget allocator.

[0108] The configuration and decision processor 106 (hereinafter referred to as the bit budget allocator 106) can use a bit rate adaptation algorithm to distribute the available bit budget for core coding of N audio streams in N transmission channels 104. Details of the bit rate adaptation algorithm for distributing the available bit budget for core coding can be found in reference [3].

[0109] 2.5 Preprocessing

[0110] Reference Figure 1, the ISM encoding method includes an operation 158 of preprocessing N audio streams transmitted through N transmission channels 104 from a configuration and decision processor 106 (bit budget allocator). To perform operation 158, the ISM encoder includes a preprocessor 108.

[0111] Once the configuration and decision processor 106 (bit budget allocator) has completed the configuration and bit rate allocation among the N audio streams, the preprocessor 108 performs sequential further preprocessing 158 on each of the N audio streams. Such preprocessing 158 may include, for example, further signal classification, further core encoder selection (e.g., selection between an ACELP core, a TCX core, and an HQ core), and other resampling at different internal sampling frequencies Fs adapted to the bit rate to be used for core coding. For example, examples of such preprocessing can be found in reference [1] related to the EVS codec, and thus will not be further described in this disclosure.

[0112] 2.6 Core Coding

[0113] Reference Figure 1 , the ISM encoding method includes an operation 159 of core coding. To perform operation 159, the ISM encoder 100 includes, for example, N core encoders 109 for encoding and decoding the N audio streams to encode and decode the N audio streams transmitted through the N transmission channels 104 from the preprocessor 108, respectively.

[0114] Specifically, in the case of active frame encoding and decoding, N audio streams are encoded using N variable bit rate core encoders 109 (e.g., single core encoders). The bit rate used by each of the N core encoders is the bit rate selected by the configuration and decision processor 106 (bit budget allocator) for the corresponding audio stream. For example, an EVS-based core encoder as described in reference [1] can be used as the core encoder 109.

[0115] In the case of DTX operation (SID frame encoding and decoding), the SID of one audio object is encoded and decoded by one of the core encoders 109, while the processing in the other core encoders 109 can be completed in NO_DATA frame operation (they only update the core encoder state parameters). Alternatively, no processing is performed in the other core encoders 109, that is, preprocessing and core coding are completely skipped in the other core encoders 109.

[0116] 2.7 Encoding and Decoding of Spatial Information

[0117] Reference Figure 1, the ISM encoding method includes an operation 180 for encoding and decoding spatial information, and the ISM encoder includes a corresponding encoding and decoding module 130. The spatial information analyzes the similarity between audio objects and can be based on, for example, inter-channel cues such as inter-channel sound level differences or inter-channel coherence.

[0118] The spatial information is typically estimated and encoded / decoded using parametric ISM encoding / decoding techniques or SID frame encoding / decoding. Examples can be found in the IVAS framework described in reference [4], the entire content of reference [4] being incorporated herein by reference.

[0119] 2.8 SID Bitstream Structure

[0120] Reference Figure 1 , the ISM encoding method includes a multiplexing operation 160. To perform operation 160, the ISM encoder includes a multiplexer 110.

[0121] Figure 3 is shown for a frame the SID bitstream 111 generated by the multiplexer 110 and transmitted from the Figure 1 ISM encoder to the Figure 4 ISM decoder 400. The structure of the SID bitstream 111 can be constructed as shown Figure 3 regardless of whether metadata is present and transmitted.

[0122] Reference Figure 3 , the multiplexer 110 writes the index 302 of the SID format signaling at the start of the bitstream 111, followed by an index 114 of the core encoder SID, while the index of the ISM common signaling 113 from the configuration and decision processor 106 (bit budget allocator), the spatial information 131 from the encoding / decoding module 130, and the metadata 112 from the metadata processor 105 are written from the end of the bitstream 111.

[0123] 2.8.1 ISM Common Signaling in the SID Frame

[0124] The multiplexer writes the ISM common signaling 113 from the end of the bitstream 111. The ISM common signaling in the SID frame is generated by the configuration and decision processor 106 (bit budget allocator) and includes a variable number of bits representing:

[0125] (a) N audio objects: The signaling for the N encoded / decoded audio objects present in the bitstream 111 is in the form of, for example, a unary code 301 with a stop bit (e.g., for N = 3 audio objects, the first 3 bits of the ISM common signaling will be "110" written in reverse order).

[0126] (b) Metadata on / off flag MD, one for each audio object, as described by relation (5) and represented by line 214.

[0127] 2.8.2 Metadata Payload

[0128] In the SID frame, just after the ISM common signaling 113, in reverse order, the following are written into the bitstream 111: a) the spatial information index 131, and finally b) the metadata value 112 quantized as in paragraph 2.3.1 above.

[0129] 2.8.3 Audio Stream Payload

[0130] In the active frame, the multiplexer 110 receives N audio streams 114 encoded and decoded by N core encoders 109 via N transport channels 104, and writes them starting from the beginning of the bitstream 111 just after the IVAS format bit 302 (see Figure 3 ).

[0131] In the SID frame, the multiplexer 110 receives one audio stream 114 encoded and decoded by one core encoder 109 via one transport channel 104, and writes it starting from the beginning of the bitstream 111 just after the IVAS SID format bit 302 (signaling of the SID mode related to the IVAS format) (see Figure 3 ).

[0132] 3. Decoding of Audio Objects

[0133] Figure 4 is a schematic block diagram showing both the ISM decoding method 450 for implementing a method for decoding audio objects during discontinuous transmission (DTX) operations and the corresponding ISM decoder 400 for implementing a device for decoding decoded audio objects during discontinuous transmission (DTX) operations.

[0134] 3.1 Demultiplexing

[0135] Refer to Figure 4 , the ISM decoding method 450 includes the demultiplexing operation 451. To perform operation 451, the ISM decoder 400 includes a demultiplexer 401.

[0136] The demultiplexer 401 receives the bitstream 402 transmitted from the Figure 1 ISM encoder to the Figure 4 ISM decoder 400. Specifically, Figure 4 the bitstream 402 corresponds to the Figure 1 bitstream 111.

[0137] The demultiplexer 401 extracts from the bitstream 402 (a) the N decoded audio streams 114 in the case of active frame coding / decoding, or one audio stream in the case of SID frames, (b) the metadata 112 for the coding / decoding of the N audio objects, (c) the spatial information 131, and (d) the ISM common signaling 113 read from the end of the received bitstream 402.

[0138] 3.2 Metadata dequantization and decoding

[0139] Reference Figure 4 , the ISM decoding method 450 includes an operation 453 of metadata decoding and dequantization. To perform the operation 453, the ISM decoder 400 includes a metadata decoding and dequantization processor (metadata decoder) 403.

[0140] The metadata decoding and dequantization processor (metadata decoder) 403 is provided with the metadata 112 for the coding / decoding of the N transmitted audio objects, the ISM common signaling 113, and the output settings 404 to decode and dequantize the metadata for the audio stream / object with active content. The output settings 404 are command line parameters regarding the number M of decoded audio objects / transmission channels and / or audio formats, which may be equal to or different from the number N of coded audio objects / transmission channels. The metadata decoding and dequantization processor (metadata decoder) 403 generates the decoded metadata 405 for the M audio objects / transmission channels and provides the decoded metadata and information about their respective bit budgets on line 406. Obviously, the decoding and dequantization performed by the processor (metadata decoder) 103 is the inverse of the quantization and coding performed by Figure 1 the metadata processor 105.

[0141] 3.2.1 Metadata adjustment in DTX operation

[0142] At the ISM decoder 400, receiving SID frames at a certain rate (e.g., in IVAS, the default is 8 frames) results in the possibility that the received MD parameter values change in large steps between SID frames. These large steps may cause subjective artifacts; for example, the position of an audio object can suddenly change from one place to another. To avoid these artifacts, the metadata decoding and dequantization processor (metadata decoder) 403 adjusts the MD parameter values at the ISM decoder 400 such that the differences in MD parameter values are reduced between frames. For example, interpolation between the current frame MD parameter value of true decoding and dequantization and the previous frame MD parameter value can be applied in some frames after the SID frame. This causes the MD parameter values to evolve more smoothly, and the smoothing is applied to several CNG frames, or several active frames, or several CNG and active frames following the SID frame.

[0143] As an example, an adjustment or smoothing can be applied to each MD parameter such that the maximum difference (step size) of the MD parameter between two adjacent frames does not exceed (is lower than) a given threshold. Assume the current decoded and dequantized MD parameter value θ of the azimuth angle dec and the maximum smoothing step size of the azimuth angle difference is Δ θ . Then:

[0144]

[0145] where θ true is the quantized azimuth angle (usually the quantized MD parameter value) transmitted in the SID frame, θ last is the azimuth angle in the previous frame (usually the MD parameter value), and is updated to θ last = θ dec at the end of decoding each frame. In the example embodiment, the azimuth angle (usually the MD parameter value) smoothing step size value is Δ θ = 5, corresponding to the parameter "CNG_MD_MAX_DIFF_AZIMUTH" in the source code at the end of this disclosure. In addition, the operation sgn(x) in relation (6) is the mathematical expression

[0146]

[0147] In addition, the maximum number of frames to which the smoothing step is applied can be set to a given threshold D max . For example, in the example embodiment, it is not limited to the inactive segment, but in the example embodiment is limited to the active segment as D max = 5 frames (constant "IVAS_ISM_DTX_HO_MAX" in the source code at the end of this disclosure).

[0148] In addition, when the absolute value of the difference between the current frame quantized MD parameter value θ true and the previous frame decoded MD parameter value θ last is greater than the smoothing step size value Δ θ multiplied by the threshold D max , the smoothing step is skipped in the active frame. For example, in the case of the azimuth angle parameter, this means:

[0149] If |θ true - θ last | > D max · Δ θ then skip smoothing (8)

[0150] Relation (8) also applies to MD parameters other than the azimuth angle. Relation (8) is used when the true value of the MD parameter changes significantly from frame to frame and the number of frames between the last SID frame and the active frame is low (usually lower than D max)Preventing incorrect smooth MD parameter estimation.

[0151] When encoding and decoding two audio objects during DTX operation in an IVAS codec, for the MD azimuth parameter of the first audio object, the smoothing effect can be seen from the Figure 5 curve graph. From top to bottom, for a 1.1 - second segment, there are a) the noise segment of the input audio signal, b) CNG synthesis, c) the total IVAS bit rate indicating active frames (48 kbps), SID frames (5.2 kbps), and NO_DATA frames (0 bps), d) the encoder - input azimuth, e) the decoder (quantized) azimuth in the case of active - frame encoding and decoding (without DTX), f) the decoder azimuth in the CNG segment without smoothing, g) the decoder azimuth in the CNG segment with smoothing.

[0152] 3.3 Configuration and determination regarding bit rate

[0153] Reference Figure 4 , the ISM decoding method 450 includes an operation 457 for the configuration and determination of the bit rate for each channel. To perform operation 457, the ISM decoder 400 includes a configuration and determination processor 407 (bit - budget allocator).

[0154] The bit - budget allocator 407 receives, on line 406, information regarding the respective bit budgets of M decoded metadata to determine the core - decoder bit rate for each audio stream. The bit - budget allocator 407 uses the same process as the Figure 1 bit - budget allocator 106 in

[0155] 3.4 Decoding of spatial information

[0156] Reference Figure 4 , the ISM decoding method 450 includes an operation for the decoding of spatial information 458. The ISM decoder 400 includes a spatial - information decoding module 408 for performing operation 458.

[0157] Operation 458 is responsible for decoding the spatial information 409 used in the parametric ISM technique or SID frames. Decoding 458 is the inverse of the Figure 1 encoding and decoding 180 in

[0158] 3.5 Core decoding

[0159] Still referring to Figure 4, the ISM decoding method 450 includes an operation 460 of core decoding. To perform the operation 460, the ISM decoder 400 includes decoders for N audio streams 410, which includes N core decoders 410, such as N variable bitrate core decoders.

[0160] The N audio streams 114 from the demultiplexer 401 are decoded, for example, sequentially decoded in the N variable bitrate core decoders 410 at their respective core decoder bitrates 411 determined by the bit budget allocator 407. When the number M of decoded audio objects requested by the output setting 404 is less than the number of transmission channels (i.e., M < N), a smaller number of core decoders 410 are used. Similarly, in this case, not all metadata payloads can be decoded.

[0161] In response to the N audio streams 114 from the demultiplexer 401, the core decoder bitrates 411 determined by the bit budget allocator 407, and the output setting 404, the core decoders 410 generate M decoded audio streams 412 on their respective M transmission channels.

[0162] In the case of SID frames, one audio stream 114 from the demultiplexer 401 is decoded and fed into one of the core decoders 410. The other core decoders 410 are fed by the SID and spatial information 409 of the first core encoder.

[0163] 3.6 Audio Channel Rendering

[0164] The ISM decoding method 450 may include an operation 463 of audio channel rendering. To perform the operation 463, the ISM decoder 400 includes a renderer 413 for audio objects.

[0165] Taking into account the output setting 415 indicating the number and content of the output audio channels to be generated, the renderer 413 transforms the M decoded metadata 405 and the M decoded audio streams 412 into a plurality of output audio channels 414. Again, the number of output audio channels 414 may be equal to or different from the number M.

[0166] The renderer 413 can be designed in various different structures to obtain the desired output audio channels. Therefore, the renderer is not further described in this disclosure.

[0167] 4. Example Configurations of Hardware Components

[0168] Figure 6A simplified block diagram of an example configuration of a hardware component that forms (a) a method and apparatus for discontinuous transmission (DTX) of audio objects in an object-based audio codec, (b) an ISM coding method and encoder, (c) a method and apparatus for decoding audio objects during discontinuous transmission (DTX) operations, and / or (d) an ISM decoding method and decoder (collectively referred to hereinafter as "coding and decoding methods and apparatuses").

[0169] The coding and decoding methods and apparatuses may be implemented as part of a mobile terminal, part of a portable media player, or in any similar device. The coding and decoding methods and apparatuses (identified as 600 in Figure 6 include an input terminal 602, an output terminal 603, a processor 601, and a memory 604.

[0170] The input terminal 602 is configured to receive an input signal. The output terminal 603 is configured to supply an output signal. The input terminal 602 and the output terminal 603 may be implemented in a common module, such as a serial input / output device.

[0171] The processor 601 is operably connected to the input terminal 602, the output terminal 603, and the memory 604. The processor 601 is implemented as one or more processors for executing code instructions to support the functions of the various elements and operations of the above-described coding and decoding methods and apparatuses as shown in the drawings and / or as described in the present disclosure.

[0172] The memory 604 may include a non-transitory memory for storing code instructions executable by the processor 601, specifically, a processor-readable memory storing non-transitory instructions that, when executed, cause the processor to implement the elements and operations of the coding and decoding methods and apparatuses. The memory 604 may also include a random access memory or buffer for storing intermediate processing data from the various functions executed by the processor 601.

[0173] Those of ordinary skill in the art will recognize that the description of the coding and decoding methods and apparatuses is illustrative only and is not intended to be limiting in any way. Those of ordinary skill in the art who benefit from the present disclosure will readily envision other embodiments. Additionally, the disclosed coding and decoding methods and apparatuses may be customized to provide valuable solutions to existing needs and problems in coding and decoding audio signals.

[0174] For clarity, not all conventional features of the implementation of the encoding and decoding methods and devices are shown and described. Of course, it should be understood that in the development of any such actual implementation of the encoding and decoding methods and devices, many implementation-specific decisions may need to be made in order to achieve the specific goals of the developer, such as compliance with application, system, network, and business-related constraints, and these specific goals will vary from one implementation to another and from one developer to another. In addition, it should be understood that the development work may be complex and time-consuming, but it is still a routine task of engineering for those of ordinary skill in the audio processing field who benefit from the present disclosure.

[0175] According to the present disclosure, the elements, processing operations, and / or data structures described herein can be implemented using various types of operating systems, computing platforms, network devices, computer programs, and / or general-purpose machines. In addition, those of ordinary skill in the art will recognize that devices of less general nature, such as hardwired devices, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc., can also be used. In cases where a method comprising a series of operations and sub-operations is implemented by a processor, computer, or machine and those operations and sub-operations can be stored as a series of non-transitory code instructions readable by the processor, computer, or machine, they can be stored on a tangible and / or non-transitory medium.

[0176] The elements and processing operations of the encoding and decoding methods and devices as described herein can include software, firmware, hardware, or any combination of software, firmware, or hardware suitable for the purposes described herein.

[0177] In the encoding and decoding methods and devices, the various processing operations and sub-operations can be executed in various orders and some of the processing operations and sub-operations can be optional.

[0178] Although the present disclosure has been described above through non-limiting illustrative embodiments of the present disclosure, these embodiments can be arbitrarily modified within the scope of the appended claims without departing from the spirit and nature of the present disclosure.

[0179] 5. References

[0180] The present disclosure refers to the following references, the entire contents of which are incorporated herein by reference:

[0181] [1] 3GPP TS 26.445, v.17.0.0, "Codec for Enhanced Voice Services (EVS); Detailed Algorithmic Description", April 2022.

[0182] [2]3GPP TS 26.250 v1.0.0, "Codec for Immersive Voice and Audio Services – General overview", September 2023.

[0183] [3]V. Eksler, "Method and System for Coding Metadata in Audio Streams and for Flexible Intra-Object and Inter-Objects Bitrate Adaptation", Patent Publication US2022 / 0238127A1, date July 28, 2022.

[0184] [4]3GPP SA4 contribution S4-231233 "High-level description of the IVAS codec candidate of the IVAS codec candidate of the IVAS codec public collaboration", SA4 meeting #125, August 21 - 25, 2023, https: / / www.3gpp.org / ftp / Meetings_3GPP_Sync / SA4 / Docs / S4-231233.zip

[0185] [5]3GPP SA4 Proposal S4-180462, "On spatial metadata for IVAS spatial audio input format", SA4 meeting #98, April 9 - 13, 2018,

[0186] https: / / www.3gpp.org / ftp / tsg_sa / WG4_CODEC / TSGS4_98 / Docs / S4-180462.zip

[0187] Source code

[0188] Code excerpts implementing this disclosure in the IVAS audio codec framework are as follows.

[0189]

[0190]

[0191]

[0192]

[0193]

[0194]

[0195]

[0196]

[0197]

[0198]

[0199]

[0200]

[0201]

Claims

1. A device for discontinuous transmission DTX of audio objects in an object-based audio codec, the audio objects each comprising an audio stream, the device comprising: An analyzer of the audio stream for generating speech or signal activity information about the audio object; A DTX controller for detecting a DTX signal segment of the audio object and a silence insertion descriptor SID frame within the DTX signal segment in response to the activity information about the audio object, wherein the DTX controller (a) updates a global SID counter for inactive frames, and (b) signals the detected SID frames within the DTX signal segment depending on the value of the global SID counter; And An encoder of the signaled detected SID frames encoded and decoded using the SID frame.

2. The discontinuous transmission device according to claim 1, wherein, The activity information includes an activity detection flag for each audio object, and wherein when the activity detection flag of the audio object is set to a given value, the DTX controller detects a DTX signal segment.

3. The discontinuous transmission device according to claim 1 or 2, wherein, When a DTX signal segment is detected, the DTX controller sets a DTX flag to a given value.

4. The discontinuous transmission device according to any one of claims 1 to 3, wherein, The DTX controller signals the detected SID frames within the DTX signal segment by setting an SID flag to a given value in response to a specific value of the global SID counter.

5. The discontinuous transmission device according to any one of claims 1 to 4, wherein, In response to the global SID counter being equal to "0", the DTX controller signals the detected SID frames.

6. The discontinuous transmission device according to any one of claims 1 to 5, wherein, The DTX controller resets the global SID counter in each active frame.

7. The discontinuous transmission device according to any one of claims 1 to 6, wherein The DTX controller increments the global SID counter in each inactive frame until a value corresponding to the SID update rate.

8. The discontinuous transmission device according to claim 7, wherein, When the global SID counter reaches a value corresponding to the SID update rate, the DTX controller resets the global SID counter.

9. The discontinuous transmission device according to any one of claims 1 to 8, wherein The DTX controller uses an additional classification stage of the audio object to change the DTX flag.

10. The discontinuous transmission device according to claim 9, wherein, The additional classification stage compares an average value of long-term background noise on the audio object and an average value of changes in long-term background noise on the audio object with respective thresholds.

11. The discontinuous transmission device according to claim 9, wherein, The additional classification stage uses the energy of background noise in the audio object.

12. The discontinuous transmission device according to any one of claims 1 to 11, wherein Each of the audio objects includes an audio stream having metadata, and wherein the SID frame encoder includes a metadata encoder for encoding the metadata of the audio object using absolute encoding and decoding.

13. The discontinuous transmission device according to any one of claims 1 to 11, wherein, Each of the audio objects includes an audio stream having metadata, and wherein the DTX controller calculates a metadata MD flag for each audio object, the MD flag indicating that the MD parameters remain unchanged to avoid encoding, decoding, and transmitting the metadata of the audio object by the SID frame encoder, thereby saving the SID bit budget.

14. The discontinuous transmission device according to any one of claims 1 to 11, wherein, Each of the audio objects includes an audio stream having metadata, and wherein the DTX controller estimates a bit budget for quantifying the metadata and compares the estimated bit budget with the bit budget available for quantifying the metadata to select SID frame encoding and decoding or active frame encoding and decoding.

15. The discontinuous transmission device according to claim 14, wherein, When the estimated bit budget is higher than the bit budget available for quantifying the metadata, the DTX controller sets the DTX flag to a first given value and selects active frame coding and decoding to code and decode the metadata.

16. The discontinuous transmission device according to claim 14 or 15, wherein, When the estimated bit budget is lower than the bit budget available for quantifying the metadata, the DTX controller sets the DTX flag to a second given value and selects SID frame coding and decoding to code and decode the metadata.

17. The discontinuous transmission device according to any one of claims 1 to 16, wherein, Each of the audio objects includes an audio stream having metadata MD, and wherein the DTX controller makes the resolution of the MD values depend on the number of audio objects.

18. A device for decoding audio objects during discontinuous transmission DTX operation, each of the audio objects including an audio stream having metadata MD, the MD including at least one MD parameter, the device comprising: a metadata decoder for decoding the metadata, wherein the metadata decoder adjusts the value of the MD parameter to reduce the difference of the MD parameter between frames; and an audio stream decoder for decoding the audio stream.

19. The audio object decoding device according to claim 18, wherein, To reduce the difference in MD parameter values, the metadata decoder smooths the MD parameter by interpolating between the value of the MD parameter in the current frame and the value of the MD parameter in the previous frame.

20. The audio object decoding device according to claim 19, wherein, The metadata decoder smooths the MD parameter in a frame following a silence insertion descriptor SID frame, whereby the value of the MD parameter evolves smoothly.

21. The audio object decoding device according to any one of claims 18 to 20, wherein, The metadata decoder reduces the difference of the MD parameter such that the maximum difference of the MD parameter between two adjacent frames is lower than a given threshold.

22. The audio object decoding device according to claim 19 or 20, wherein, The metadata decoder limits the maximum number of frames to which smoothing is applied to a given threshold.

23. The audio object decoding device according to claim 22, wherein, When the absolute value of the difference between the value of the MD parameter in the current frame and the value of the MD parameter in the previous frame is higher than the smoothing step value multiplied by a given threshold, the metadata decoder skips the smoothing of the MD parameter in the active frame.

24. A method for discontinuous transmission DTX of audio objects in an object-based audio codec, the audio objects including respective audio streams, the method comprising: analyzing the audio stream to generate speech or signal activity information about the audio object; in response to the activity information about the audio object, detecting a DTX signal segment of the audio object and a silence insertion descriptor SID frame within the DTX signal segment, wherein the segment and frame detection includes (a) updating a global SID counter for inactive frames, and (b) signaling the detected SID frames within the DTX signal segment depending on the value of the global SID counter; and using SID frame coding and decoding to encode the detected SID frames signaled.

25. The discontinuous transmission method according to claim 24, wherein, The activity information includes an activity detection flag for each audio object, and wherein the segment and frame detection includes: detecting a DTX signal segment when the activity detection flag of the audio object is set to a given value.

26. The discontinuous transmission method according to claim 24 or 25, wherein, When a DTX signal segment is detected, the segment and frame detection includes setting the DTX flag to a given value.

27. The discontinuous transmission method according to any one of claims 24 to 26, comprising: In response to a specific value of the global SID counter, signal the SID frames detected within the DTX signal segment by setting the SID flag to a given value.

28. The discontinuous transmission method according to any one of claims 24 to 27, comprising: In response to the global SID counter being equal to "0", signal the detected SID frames.

29. The discontinuous transmission method according to any one of claims 24 to 28, including resetting the global SID counter in each active frame.

30. The discontinuous transmission method according to any one of claims 24 to 29, comprising: Increment the global SID counter in each inactive frame until a value corresponding to the SID update rate is reached.

31. The discontinuous transmission method according to claim 30, comprising: When the global SID counter reaches a value corresponding to the SID update rate, reset the global SID counter.

32. The discontinuous transmission method according to any one of claims 24 to 31, comprising: Use an additional classification stage of the audio object to change the DTX flag.

33. The discontinuous transmission method according to claim 32, wherein, The additional classification stage compares the average value of the long-term background noise on the audio object and the average value of the change in the long-term background noise on the audio object with respective thresholds.

34. The discontinuous transmission method according to claim 32, wherein, The additional classification stage uses the energy of the background noise in the audio object.

35. The discontinuous transmission method according to any one of claims 24 to 34, wherein Each of the audio objects includes an audio stream having metadata, and wherein the SID frame encoding includes encoding the metadata of the audio object using absolute encoding and decoding.

36. The discontinuous transmission method according to any one of claims 24 to 34, wherein, Each of the audio objects includes an audio stream having metadata, and wherein the segment and frame detection includes: calculating an MD flag for the metadata of each audio object, the MD flag indicating that the MD parameters remain unchanged to avoid encoding, decoding, and transmitting the metadata of the audio object, thereby saving the SID bit budget.

37. The discontinuous transmission method according to any one of claims 24 to 34, wherein, Each of the audio objects includes an audio stream having metadata, and wherein the segment and frame detection includes: estimating the bit budget for quantifying the metadata and comparing the estimated bit budget with the bit budget available for quantifying the metadata to select SID frame encoding and decoding or active frame encoding and decoding.

38. The discontinuous transmission method according to claim 37, wherein The segment and frame detection includes: when the estimated bit budget is higher than the bit budget available for quantifying the metadata, setting the DTX flag to a first given value and selecting active frame encoding and decoding to encode and decode the metadata.

39. The discontinuous transmission method according to claim 37 or 38, wherein, The segment and frame detection includes: when the estimated bit budget is lower than the bit budget available for quantifying the metadata, setting the DTX flag to a second given value and selecting SID frame encoding and decoding to encode and decode the metadata.

40. The discontinuous transmission method according to any one of claims 24 to 39, wherein, Each of the audio objects includes an audio stream having metadata MD, and wherein the method includes making the resolution of the MD value depend on the number of audio objects.

41. A method for decoding an audio object during discontinuous transmission DTX operation, each of the audio objects including an audio stream having metadata MD, the MD including at least one MD parameter, the method including: Decoding the metadata, including adjusting the value of the MD parameter to reduce the difference in the MD parameter between frames; And Decoding the audio stream.

42. The audio object decoding method according to claim 41, wherein, Decoding the metadata includes: to reduce the difference in the MD parameter values, smoothing the MD parameter by interpolating between the value of the MD parameter in the current frame and the value of the MD parameter in the previous frame.

43. The audio object decoding method according to claim 42, wherein, Decoding the metadata includes: smoothing the MD parameter in a frame following a silence insertion descriptor SID frame, whereby the value of the MD parameter evolves smoothly.

44. The audio object decoding method according to any one of claims 41 to 43, wherein decoding the metadata includes reducing the difference of the MD parameter such that the maximum difference of the MD parameter between two adjacent frames is lower than a given threshold.

45. The audio object decoding method according to claim 42 or 43, wherein, Decoding the metadata includes: limiting the maximum number of frames to which smoothing is applied to a given threshold.

46. The audio object decoding method according to claim 45, wherein, Decoding the metadata includes: skipping the smoothing of the MD parameter in an active frame when the absolute value of the difference between the value of the MD parameter in the current frame and the value of the MD parameter in the previous frame is higher than the smoothing step value multiplied by a given threshold.

47. A device for discontinuous transmission DTX of an audio object in an object-based audio codec, the audio object including respective audio streams, the device including: at least one processor; and a memory coupled to the processor and storing non-transitory instructions that, when executed, cause the processor to implement: an analyzer of the audio stream for generating speech or signal activity information about the audio object; a DTX controller for detecting a DTX signal segment of the audio object and a silence insertion descriptor SID frame within the DTX signal segment in response to the activity information about the audio object, wherein the DTX controller (a) updates a global SID counter for non-active frames, and (b) signals the detected SID frame within the DTX signal segment depending on the value of the global SID counter; and an encoder of the signaled, detected SID frame encoded using the SID frame.

48. A device for discontinuous transmission DTX of an audio object in an object-based audio codec, the audio object including respective audio streams, the device including: at least one processor; and a memory coupled to the processor and storing non-transitory instructions that, when executed, cause the processor to: analyze the audio stream to generate speech or signal activity information about the audio object; detect a DTX signal segment of the audio object and a silence insertion descriptor SID frame within the DTX signal segment in response to the activity information about the audio object, including (a) updating a global SID counter for non-active frames, and (b) signaling the detected SID frame within the DTX signal segment depending on the value of the global SID counter; and an encoder of the signaled, detected SID frame encoded using the SID frame.

49. A device for decoding an audio object during discontinuous transmission DTX operation, each of the audio objects including an audio stream having metadata MD, the MD including at least one MD parameter, the device including: at least one processor; and a memory coupled to the processor and storing non-transitory instructions that, when executed, cause the processor to implement: A metadata decoder for decoding the metadata, wherein the metadata decoder adjusts the value of the MD parameter to reduce the difference in the MD parameter between frames; and An audio stream decoder for decoding the audio stream.

50. A device for decoding an audio object during a discontinuous transmission DTX operation, the audio object each comprising an audio stream having metadata MD, the MD comprising at least one MD parameter, the device comprising: At least one processor; and A memory coupled to the processor and storing non-transitory instructions which, when executed, cause the processor to: Decode the metadata, including adjusting the value of the MD parameter to reduce the difference in the MD parameter between frames; and Decode the audio stream.

Images