Methods, apparatus, and systems for generating, transporting, and processing immediate regeneration frames (IPFs).

Abstract

To provide an audio decoder for decoding a bitstream of encoded audio data.SOLUTION: A bitstream of encoded audio data represents a sequence of audio sample values and includes a plurality of frames. Each frame has an associated encoded audio sample value. An audio decoder includes: a determination unit for determining whether a frame of the bitstream of encoded audio data is an immediate playout frame having an encoded audio sample value associated with a current frame and additional information; and an initialization unit for initializing the decoder when the frame is the immediate playout frame. Initialization includes decoding the encoded audio sample value configured by additional information before decoding the encoded audio sample value associated with the current frame.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] This disclosure generally relates to audio encoders, encoding methods, audio decoders, and decoding methods, including a method for decoding a bitstream of encoded audio data in which the bitstream of encoded audio data represents a sequence of audio sample values ​​and has a plurality of frames, each frame having an associated encoded audio sample value, and a method for generating a bitstream of encoded audio data that includes immediately playable frames. This disclosure further relates to apparatus for generating immediately playable frames in a bitstream of encoded audio data, or for removing immediately playable frames from a bitstream of encoded audio data.

[0002] While several embodiments are described herein with reference to their disclosures, it is clear that this disclosure is not limited to such articulations but is applicable to a broader scope. [Background technology]

[0003] Currently, MPEG-4 audio, as standardized in ISO / IEC 14496-3, Coding of audio-visual objects-Part 3: Audio, for generating, transporting and processing Immediate Playout Frames (IPF), has a shortcoming. IPFs can provide information to a special frame that allows the decoder to be initialized immediately, so that playback occurs instantly upon switching to the data stream containing that special frame. In other words, an IPF is a frame that, upon its reception, allows the decoder to immediately play back the correct sample from the first sample encoded in that IPF. This is because the IPF contains all the information necessary to do so. Thus, an IPF represents an independently decodeable frame that can be decoded using only the information contained within it.

[0004] Encoded audio typically appears as data frames or chunks. In relation to audio standardized in MPEG-4, frames / chunks are sometimes known as granules, and the encoded chunks / frames are called access units (AUs), while the decoded chunks are called composition units (CUs). In transport systems, audio signals may only be accessible and addressable at the level of these coded chunks (access units).

[0005] In adaptive streaming, when audio switches to different settings (for example, different bitrates such as the bitrate set within the adaptive settings in MPEG-DASH), the decoder plays the audio sample accurately from the beginning by using an AU representing the corresponding time segment of the audio program. n And, AU n Additional AU preceding n-1 , AU n-2 ...AU and configuration data must be supplied. Otherwise, due to different coding settings (e.g., windowing data, SBR-related data, PS-related data), the decoder will not be able to process the AU. n Decrypting only this does not guarantee that the correct output will be produced. Therefore, the first AU that should be decrypted with the new settings n This includes the new configuration data and all pre-roll data (AU) required to initialize the decoder with the new settings. n AU represents an earlier time segment n-x It must carry (in the form of ). This can be done using an Immediate Playout Frame (IPF) as defined in the MPEG-H 3D audio standard or the MPEG-D USAC standard. [Overview of the project]

[0006] In view of the above, the object of the present invention is therefore to provide an audio decoder and decoding method, as well as an audio encoder, an audio encoder system, an apparatus, and an encoding method capable of processing IPF with MPEG-4 audio.

[0007] An audio decoder is provided for decoding a bitstream of encoded audio data, wherein the bitstream of encoded audio data represents a sequence of audio sample values ​​and comprises a plurality of frames, each frame having associated encoded audio sample values.

[0008] An audio decoder may have a determination unit configured to determine whether a frame of a bitstream of encoded audio data is an immediate playback frame having an encoded audio sample value associated with the current frame and additional information, the additional information having encoded audio sample values ​​of a number of frames preceding the immediate playback frame, the encoded audio sample values ​​of the preceding frames may be encoded using the same codec settings as the current frame, and the number of preceding frames corresponding to the pre-roll frame may correspond to the number of frames required by the decoder to constitute the whole signal such that whenever an immediate playback frame is decoded, it is in a position to output a valid audio sample value associated with the current frame.

[0009] The decoder may have an initialization unit configured to initialize the decoder when the determination unit determines that the frame is an immediate playback frame, and initializing the decoder may involve decoding the encoded audio sample value composed of additional information before decoding the encoded audio sample value associated with the current frame, and the initialization unit may be configured to switch the audio decoder from the current codec setting to a different codec setting when the determination unit determines that the frame is an immediate playback frame and the audio sample value of the current frame is encoded using a different codec setting, and the decoder may be configured to decode the current frame using the current codec setting and discard the additional information when the determination unit determines that the frame is an immediate playback frame and the audio sample value of the current frame is encoded using the current codec setting.

[0010] In some embodiments, the additional information may further include information about the codec settings used to encode the audio sample values ​​associated with the current frame, and the determination unit may further be configured to determine whether the codec settings in the additional information differ from the current codec settings.

[0011] In some embodiments, the immediate playback frame may have additional information as an extension payload, and the determination unit may be configured to evaluate the extension payload of the immediate playback frame.

[0012] In some embodiments, the bitstream of encoded audio data may be an MPEG-4 audio bitstream.

[0013] In some embodiments, additional information may be carried via an MPEG-4 audio bitstream extension mechanism, which is either a Data Stream Element (DSE) or an extension payload element.

[0014] In some embodiments, either a data stream element (DSE) or an extended payload element may be placed at a predefined location within the MPEG-4 audio bitstream and / or may have a specific instance tag indicating that the payload of the data stream element (DSE) or extended payload element is additional information.

[0015] Extended payload elements may be included, for example, at various locations in the MPEG-4 audio bitstream syntax. This, in turn, makes it possible to use the instant playback frame function even with MPEG-4 audio.

[0016] In some embodiments, the extended payload element may be contained within the fill element (ID_FIL).

[0017] In some embodiments, the additional information may further include a unique identifier, which may optionally be used to detect different codec settings.

[0018] In some embodiments, the decoder may further include a crossfader configured to perform crossfading between an output sample value obtained by flushing the decoder with a previous codec setting and an output sample value obtained by decoding an encoded audio sample value associated with the current frame.

[0019] In some embodiments, the earliest frame among a number of frames included in the additional information may not be encoded or entropy encoded differentially in time with respect to any frame before that earliest frame, and the instant playback frame may not be encoded or entropy encoded differentially in time with respect to any frame before the earliest frame among a number of frames preceding the instant playback frame or with respect to any frame before the instant playback frame.

[0020] According to a second aspect of the present disclosure, there is provided a method of decoding a bitstream of encoded audio data, the bitstream of encoded audio data representing a succession of audio sample values and having a plurality of frames, each frame having associated encoded audio sample values.

[0021] The method may include determining whether a frame of the bitstream of encoded audio data is an instant playback frame having encoded audio sample values associated with the current frame and additional information, the additional information may have encoded audio sample values of a number of frames preceding the instant playback frame, the encoded audio sample values of the preceding frames may be encoded using the same codec settings as the instant playback frame, and the number of preceding frames corresponding to the preroll frames may correspond to the number of frames required by the decoder to compose the full signal such that, whenever the instant playback frame is decoded, it is at a position to output valid audio sample values associated with the current frame.

[0022] The method may further include initializing the decoder if the frame is determined to be an instant playback frame, and initializing may include decoding the encoded audio sample values constituted by the additional information before decoding the encoded audio sample values associated with the current frame.

[0023] The method may further comprise the step of switching the decoder from the current codec setting to a different codec setting when it is determined that the frame is an immediate playback frame and when the audio sample values of the immediate playback frame are encoded using different codec settings.

[0024] The method may comprise the step of discarding additional information when decoding an immediate playback frame using the current codec setting and when it is determined that the frame is an immediate playback frame and when the audio sample values of the immediate playback frame are encoded using the current codec setting.

[0025] When configured as proposed, the method enables switching of an Audio Object Type (AOT) as defined in ISO / IEC 14496-3, for example, in combination with continuously generating accurate output samples, without introducing gaps of silence in the audio output.

[0026] In some embodiments, the additional information may further comprise information regarding the codec setting used to encode the audio sample values associated with the current frame, and the method may further comprise the step of determining whether the codec setting of the additional information is different from the current codec setting used to encode the audio sample values associated with the frames in the bitstream preceding the immediate playback frame.

[0027] In some embodiments, the bitstream of the encoded audio data may be an MPEG-4 audio bitstream.

[0028] In some embodiments, the additional information may be carried via an MPEG-4 audio bitstream extension mechanism that is either a Data Stream Element (ID_DSE) or an Extended Payload Element.

[0029] In some embodiments, either a data stream element (ID_DSE) or an extended payload element may be placed at a predefined location within the MPEG-4 audio bitstream and / or may have a specific instance tag indicating that the payload of the data stream element (ID_DSE) or extended payload element is additional information.

[0030] In some embodiments, the extended payload element may be contained within the filling element (ID_FIL).

[0031] In some embodiments, the additional information may further include unique identifiers, which may optionally be used to detect different codec settings.

[0032] In some embodiments, the bitstream of encoded audio data may have a first number of frames encoded using a first codec setting, followed by a second number of frames encoded using a second codec setting, the first of the second number of frames being an immediate playback frame.

[0033] An audio encoder is provided that generates a bitstream of encoded audio data having immediate playback frames, wherein the bitstream of encoded audio data represents a sequence of audio sample values ​​and has a plurality of frames, each frame having associated encoded audio sample values.

[0034] The audio encoder may have a core encoder configured to encode uncompressed audio sample values ​​associated with multiple frames using a predefined codec configuration.

[0035] The audio encoder may further have a buffer configured to store encoded audio sample values ​​of a number of preceding frames of the current frame among a group of frames, using a predefined codec setting.

[0036] The audio encoder may have an embedding section configured to write an immediate playback frame to the current frame among a plurality of frames, and the immediate playback frame may have an encoded audio sample value associated with the current frame and additional information corresponding to the encoded audio sample values ​​of a number of preceding frames of the current frame.

[0037] In some embodiments, the embedded portion may be further configured to include additional information regarding predefined codec settings.

[0038] In some embodiments, the embedded portion may be further configured to include additional information in the immediate playback frame.

[0039] In some embodiments, the generated bitstream of the encoded audio data may be an MPEG-4 audio bitstream.

[0040] In some embodiments, the embedding section may be further configured to embed additional information into the bitstream via an MPEG-4 audio bitstream extension mechanism, which is either a data stream element (ID_DSE) or an extended payload element.

[0041] In some embodiments, the embedding portion may be further configured to position either a data stream element (ID_DSE) or an extended payload element at a predefined location within the MPEG-4 audio bitstream, and / or to assign a specific instance tag indicating that the payload of the data stream element (ID_DSE) or extended payload element is additional information.

[0042] In some embodiments, the embedding portion may be further configured to embed an extended payload element within the filling element (ID_FIL).

[0043] In some embodiments, the embedding portion may be further configured to include a unique identifier as additional information, and optionally, the unique identifier may convey a predefined codec setting.

[0044] In some embodiments, the audio encoder may be further configured not to encode the earliest frame among a number of frames included in the additional information in a time-differential manner or entropically encode it with respect to any frame preceding that earliest frame, and the audio encoder may be further configured not to encode the immediately played frame in a time-differential manner or entropically encode it with respect to any frame preceding the immediately played frame or with respect to any frame preceding the immediately played frame.

[0045] A system is provided having two or more audio encoders for generating a plurality of bitstreams of encoded audio data, each having an immediate playback frame, wherein each bitstream of encoded audio data represents a sequence of audio sample values ​​and has a plurality of frames, each frame having associated encoded audio sample values.

[0046] In some embodiments, the predefined sampling rate may be the same for each of the core encoders of two or more audio encoders. Thus, resampling and further delay handling in the decoder can be avoided.

[0047] In some embodiments, the system may further include a delay alignment unit for aligning multiple bitstreams. This thus enables seamless switching in the decoder by compensating for different encoder delays.

[0048] A fifth aspect of the present disclosure is provided, a method for generating a bitstream of encoded audio data having immediate playback frames using an audio encoder, wherein the bitstream of encoded audio data represents a sequence of audio sample values ​​and has a plurality of frames, each frame having a related encoded audio sample value.

[0049] The method may include the step of encoding uncompressed audio sample values ​​associated with multiple frames using a core encoder with predefined codec settings.

[0050] The method may further include the step of using a buffer to store encoded audio sample values ​​of a number of preceding frames of the current frame among a group of frames, using predefined codec settings.

[0051] The method may also include the step of writing an immediate playback frame to the current frame of a plurality of frames by an embedding unit, the immediate playback frame may have an encoded audio sample value associated with the current frame and additional information corresponding to the encoded audio sample values ​​of a number of preceding frames of the current frame.

[0052] In some embodiments, the additional information may further include information regarding predefined codec settings.

[0053] In some embodiments, the immediate playback frame may further contain additional information.

[0054] In some embodiments, the generated bitstream of the encoded audio data may be an MPEG-4 audio bitstream.

[0055] In some embodiments, additional information may be embedded in the bitstream by an embedding unit via an MPEG-4 audio bitstream extension mechanism, which is either a data stream element (ID_DSE) or an extended payload element.

[0056] In some embodiments, either a data stream element (ID_DSE) or an extended payload element may be placed by the embedding unit at a predefined location within the MPEG-4 audio bitstream and / or may be assigned a specific instance tag indicating that the payload of the data stream element (ID_DSE) or extended payload element is additional information.

[0057] In some embodiments, the extended payload element may be embedded within the filling element (ID_FIL) by an embedding portion.

[0058] In some embodiments, the additional information may further include a unique identifier, which may optionally convey a predefined codec setting.

[0059] In some embodiments, the audio encoder does not need to encode the earliest frame among the many frames included in the additional information in a time-differential manner or entropically with respect to any frame preceding that earliest frame, and the audio encoder does not need to encode the immediately played frame in a time-differential manner or entropically with respect to any frame preceding the earliest frame among the many frames preceding the immediately played frame, or with respect to any frame preceding the immediately played frame.

[0060] A sixth aspect of the present disclosure is provided for generating an immediate playback frame in a bitstream of encoded audio data, or for removing an immediate playback frame from a bitstream of encoded audio data, wherein the bitstream of encoded audio data represents a sequence of audio sample values ​​and has a plurality of frames, each frame having associated encoded audio sample values.

[0061] The device may have a receiving unit configured to receive a bitstream of encoded audio data, the bitstream of encoded audio data representing a sequence of audio sample values ​​and having multiple frames, each frame having associated encoded audio sample values.

[0062] The device may also have an embedding section configured to write an immediate playback frame to the current frame among a plurality of frames, and the immediate playback frame may have an encoded audio sample value associated with the current frame and additional information corresponding to the encoded audio sample values ​​of a number of preceding frames of the current frame.

[0063] When configured as proposed, the device allows for the generation of individual, immediate playback frames within any of the already existing bitstreams of encoded audio data, i.e., before distribution if required.

[0064] In some embodiments, the device may further have a buffer configured to store encoded audio sample values ​​of a number of preceding frames of the current frame among a plurality of frames.

[0065] In some embodiments, the embedding portion may be further configured to remove additional information from the immediately played frame that corresponds to encoded audio sample values ​​of a number of preceding frames of the current frame.

[0066] Therefore, this makes it possible to individually remove immediate playback frames from the bitstream of encoded audio data, for example, when the audio encoder generates only immediate playback frames.

[0067] A non-temporary digital storage medium storing a computer program is provided, the computer program being a method for decoding a bitstream of encoded audio data when executed by a computer or processor, wherein the bitstream of encoded audio data represents a sequence of audio sample values ​​and has a plurality of frames, each frame having associated encoded audio sample values.

[0068] A non-temporary digital storage medium is provided that stores a computer program, the computer program, when executed by a computer or processor, performs a method of generating a bitstream of encoded audio data having immediately playable frames by an audio encoder, wherein the bitstream of encoded audio data represents a sequence of audio sample values ​​and has a plurality of frames, each frame having an associated encoded audio sample value.

[0069] An exemplary embodiment of this disclosure will now be described, merely as an example, with reference to the attached drawings. [Brief explanation of the drawing]

[0070] [Figure 1] This shows an example of an immediate playback frame in an MPEG-4 audio bitstream of encoded audio data. [Figure 2]An example of a method for decoding a bitstream of encoded audio data, wherein the bitstream of encoded audio data represents a sequence of audio sample values ​​and has multiple frames, each frame having associated encoded audio sample values. [Figure 3] A method for decoding a bitstream of encoded audio data, wherein the bitstream of encoded audio data represents a sequence of audio sample values ​​and has multiple frames, each frame having a related encoded audio sample value. [Figure 4] An example of an audio decoder for decoding a bitstream of encoded audio data, wherein the bitstream of encoded audio data represents a sequence of audio sample values ​​and has multiple frames, each frame having associated encoded audio sample values. [Figure 5] This illustrates an example of an audio encoder that generates a bitstream of encoded audio data, including random access points (Immediate Playout Frames, IPFs). [Figure 6] This example illustrates how to generate a bitstream of encoded audio data that includes random access points (Immediate Playout Frames, IPFs). [Figure 7] This illustrates an example of an audio encoder system that generates multiple bitstreams of encoded audio data, each having time-aligned random access points (Immediate Playout Frames, IPFs). [Figure 8]Examples of apparatuses for generating random access points (Immediate Playout Frames, IPF) in a bitstream of coded audio data or for removing random access points (Immediate Playout Frames, IPF) from a bitstream of coded audio data are presented. [Figure 9] Examples of devices comprising a processor for executing a computer program stored on a non - transient digital storage medium are presented.

Best Mode for Carrying Out the Invention

[0071] The present invention is related to the generation (encoding), transport (bitstream), and processing (decoding) of IPF in MPEG - 4 audio in relation to a data stream including audio standardized according to other standard specifications such as, for example, the MPEG - H 3D audio standard. Here, and hereinafter, the MPEG - 4 audio bitstream refers to a bitstream conforming to the standard specification set forth in ISO / IEC 14496 - 3, Coding of audio - visual objects - Part 3: Audio, and all future editions, revisions and amendments thereto (hereinafter referred to as "MPEG - 4 audio"). In order to enable the IPF function in MPEG - 4 audio, there are several options for generating and carrying the AU n preceding AU and the setup data in time as part of the same payload packet used for n the AU. This is done to enable an accurate output of audio samples from the first sample obtained by decoding the AU n .

[0072] The bitstream of encoded audio data may contain a sequence of audio sample values, e.g., payload packets. The bitstream of encoded audio data may further have multiple frames. Each frame may have associated encoded audio sample values. That is, each payload packet may belong to its own frame or AU.

[0073] In one example, an audio payload packet may follow the following syntax as defined in ISO / IEC 14496-3:

number

[0074] The audio payload packets described above may conform to current and future versions of MPEG standards, such as the MPEG-4 audio standard. In embodiments, the bitstream of encoded audio data may be an MPEG-4 audio bitstream (i.e., an MPEG-4 audio standard-compliant bitstream).

[0075] A frame in the bitstream of encoded audio data may be an immediate playback frame (random access point, special frame) having an encoded audio sample value associated with the current frame and additional information. The additional information may include encoded audio sample values ​​of a number of frames preceding the immediate playback frame, and the encoded audio sample values ​​of the preceding frames may be encoded using the same codec settings as the current frame. The number of preceding frames corresponding to pre-roll frames may correspond to the number of frames required by the decoder to constitute the full signal, such that whenever the immediate playback frame is decoded, it is in a position to output a valid audio sample value associated with the current frame. The full signal may be constituted, for example, during decoder startup or restart. The immediate playback frame may be, for example, the first frame when the decoder is started.

[0076] In embodiments, additional information may be carried via an MPEG-4 audio bitstream extension mechanism, which may be either a data stream element (ID_DSE) or an extension payload element. The extension payload element may be included, for example, at various locations in the MPEG-4 audio bitstream syntax, for example, at various levels. In embodiments, the extension payload element may be included within a fill element (ID_FIL).

[0077] Thus, additional information may be carried via the MPEG-4 audio bitstream extension mechanism based on, for example, one of the following options: Option 1:

number

number

number

[0078] Data stream elements (conveyed by id_syn_ele, equivalent to ID_DSE) or fill elements (conveyed by id_syn_ele, equivalent to ID_FIL), or their equivalents, as defined in ISO / IEC 14496-3 and / or future standards, may be used to carry an extended payload. The extended payload may be used to further extend the information carried in such payload packets without breaking compatibility with legacy decoders.

[0079] Therefore, in an MPEG-4 audio context, either the data stream element (ID_DSE) or the extended payload element that may be contained within the fill element (ID_FIL) is an AU. n (i.e., the current frame, the immediately replayed frame) within the same payload packet as the AU n It may be used to carry AU and configuration information representing the previous time segment. This may be further used to apply processing that enables the use of IPF functionality even in MPEG-4 audio. Similarly, as seen in MPEG-D USAC, an extension mechanism (usacExtElement) may be used to carry the AudioPreRoll() payload, although there are some differences.

[0080] In some embodiments, extension elements (i.e., extension payload elements) may be inserted into the AU, and the element type is communicated in the bitstream for each element and AU. In one example, the extension element may be the first element in the payload packet preceding the first audio element.

[0081] Each audio preroll element defined below may be identified by a Universally Unique Identifier (UUID). The UUID field may be used to notify the audio encoder of a stream setting change and to detect it by the audio decoder. If the UUID field has changed from the previous frame or the initial state (e.g., when the decoder is first started), the stream setting may have changed, and the preroll payload should be evaluated to ensure accurate decoding. If the UUID has not changed from the previous frame, the decoder may always skip audio_preroll_element() and proceed with decoding.

[0082] For example, if a UUID does not exist, the decoder may compare the AudioSpecificConfig of audio_preroll_elment() with the current decoder setting to detect a stream setting switch.

[0083] The 'flags' field is used as an 8-bit set that can be used to convey additional information to the decoder. This may be information about whether a crossfade should be applied, or the type of crossfade (e.g., linear, logarithmic). In the following example, one bit is used to indicate whether either the pre-roll payload or the UUID is present in the bitstream.

[0084] The stream configuration must be known to the decoder before any AU processes it. In MPEG-4 audio, the decoder configuration is carried in the AudioSpecificConfig element. The decoder configuration and UUDI are part of the pre-roll payload. Furthermore, the pre-roll payload has a configuration-dependent number of MPEG-4 audio AUs (raw_data_block).

[0085] An Immediate Playback Frame (IPF) in MPEG-4 may have exactly one audio_preroll_element() payload, as described below, and one or more audio element streams (e.g., single_channel_element()) as defined in ISO / IEC 14496-3. The audio element streams are associated with the current timestamp. The preroll payload may be carried in one of the extended payload mechanisms of MPEG-4 audio.

[0086] Figure 1 shows two pre-roll frames (AU n-1 , AU n-2 )2, 3 and the corresponding stream setting 4 (AudioSpecificConfig) and stream identifier 5 (UUID) are included in the immediate playback frame (AU n ) indicates.

[0087] Audio preroll elements may be defined based on the following:

number

[0088] In one example, the audio preroll element (e.g., audio_preroll_element()) is byte-aligned so that it can be transmitted without further byte alignment by both the extended payload element (e.g., within the fill element) and the data stream element.

[0089] In one example, an IPF relates to the preroll element as follows: The IPF contains both the current AU and additional AUs (i.e., a number of preceding frames) required for accurate decoding. The additional AUs are packaged as part of the preroll element, and then it is packaged into raw_data_block() (via ID_DSE or ID_FIL). Such a raw_data_block can be an IPF.

[0090] IPF may be encoded in various ways. In one example, the preroll frame must be independently decodeable, and if SBR is used, for example, an SBR header must be present. In an embodiment, audio_preroll_elment() may be encapsulated in a data stream element. For example, audio_preroll_elment() may be encapsulated in a data stream element based on the following syntax:

number

[0091] A convention may be used to identify the data stream element carrying audio_preroll_elment(). In an embodiment, ID_DSE may be placed at a predefined position in the stream and / or may have a specific instance tag that indicates the payload is audio_preroll_elment().

[0092] In other embodiments, audio_preroll_elment() may be encapsulated in an extended payload element within the pre-fill element. For example, audio_preroll_elment() may be encapsulated based on the following syntax:

number

[0093] As seen in the example above, rules may be used to identify the type of extended payload element, for example, if EXT_DATA_ELEMENT is carried at a specific predefined location within the payload packet, then the payload is audio_preroll_elment(). Thus, in embodiments, extended payload elements may be placed at a predefined location within the stream and / or may have a specific instance tag that indicates the payload is audio_preroll_elment().

[0094] In one example, an extension payload according to the present invention may be communicated using a new extension payload type, for example, extension_type=EXT_AUDIO_PRE_ROLL=1010b.

number

[0095] In one example, a data stream element (ID_DSE) or extended payload element (e.g., within a fill element (ID_FIL)) carrying an audio_preroll_element() may be embedded in the bitstream before any audio elements within the same payload packet. Some examples of embedding an IPF payload in this way include:

number

[0096] Aspects of the present invention relate to the decoding of IPF. Referring to the example in Figure 2, the decoding process may include step S101 of receiving a bitstream of encoded audio data. The bitstream of encoded audio data may represent a sequence of audio sample values ​​and may have multiple frames, each frame may have associated encoded audio sample values.

[0097] The method may further include step S102 of determining whether a frame of the encoded audio data bitstream is an immediate playback frame. An immediate playback frame may have an encoded audio sample value associated with the current frame and additional information. The additional information may have encoded audio sample values ​​of a number of frames preceding the immediate playback frame, the encoded audio sample values ​​of the preceding frames being encoded using the same codec settings as the immediate playback frame. The number of preceding frames corresponding to a pre-roll frame may correspond to the number of frames required by the decoder to constitute the entire signal, such that whenever an immediate playback frame is decoded, it is in a position to output a valid audio sample value associated with the current frame. The entire signal may be constituted, for example, during decoder startup or restart. An immediate playback frame may be, for example, the first frame at decoder startup.

[0098] The method may include step S103 of initializing the decoder when it is determined that the frame is an immediate playback frame. Initialization may involve decoding an encoded audio sample value configured with additional information before decoding the encoded audio sample value associated with the current frame.

[0099] Referring to the example in Figure 3, the audio decoder may switch from the current codec setting to a different codec setting if it is determined that the frame is an immediate playback frame and the audio sample values ​​of the immediate playback frame are encoded using a different codec setting. The immediate playback frame may be decoded using the current codec setting, and additional information may be discarded if it is determined that the frame is an immediate playback frame and the audio sample values ​​of the immediate playback frame are encoded using the current codec setting.

[0100] In one example, the decoding process may utilize elements from IPF decoding in MPEG-D USAC. Referring again to the example in Figure 3, the decoding process may proceed in detail as follows:

[0101] ●If a payload exists and the UUID has changed from the previous frame (determined in block S104), ●If a payload exists but a UUID does not exist (determined in block S105), 1. Read the new stream configuration, i.e., AudioSpecificConfig() from audio_preroll_element() (S106). 2. The decoder state is flushed, and the result is stored in a buffer, for example, using a "NULL" access unit (S107). ■Store the results in a buffer (crossfade buffer A). 3. Reset (reinitialize) the decoder (S108). 4. Decode n_preroll_frames in audio_preroll_element and discard the output (S109). 5. Decode the next audio element in the bitstream (e.g., SCE / CPE / LFE) and store the result in a buffer (crossfade buffer B). ■When notified by the flag, a crossfade is applied between crossfade buffer A and the decoder output, and the result is written to the output PCM buffer (S110). The result of the crossfade constitutes the composition unit of this frame. ■Otherwise, the decoder output is written directly to the PCM buffer. 6. Continue to the next frame. ●Other cases 1. Skip audio_preroll_element() and decode the frame (S111). 2. Write the result to the PCM buffer (S112).

[0102] The decryption process is: ● If no payload exists, decode each frame and continue with the next frame. It may also include the following.

[0103] In one example, this process, combined with the sequential generation of accurate output samples, could enable switching of audio object types (AOTs) as defined in ISO / IEC 14496-3, without introducing silence gaps in the audio output. In one example, the AOTs that can be switched may include AOT2 (AAC), AOT5 (SBR), AOT29 (PS), and other compatible types.

[0104] Referring to the example in Figure 4, an aspect of the present invention relates to an audio decoder for decoding an IPF. The audio decoder 100 may have a determination unit 101. The determination unit 101 may be configured to determine whether a frame of the bitstream of encoded audio data is an immediate playback frame having an encoded audio sample value associated with the current frame and additional information. The additional information may include encoded audio sample values ​​of a number of frames preceding the immediate playback frame. The encoded audio sample values ​​of the preceding frames may be encoded using the same codec settings as the current frame. The number of preceding frames corresponding to a pre-roll frame may correspond to the number of frames required by the decoder 100 to constitute the entire signal such that, whenever an immediate playback frame is decoded, it is in a position to output a valid audio sample value associated with the current frame. The entire signal may be constituted, for example, during startup or restart of the decoder 100. The immediate playback frame may be, for example, the first frame when the decoder 100 is started.

[0105] The audio decoder 100 may have an initialization unit 102. The initialization unit 102 may be configured to initialize the decoder 100 when the determination unit 101 determines that the frame is an immediate playback frame. Initializing the decoder 100 may involve decoding the encoded audio sample value, which is composed of additional information, before decoding the encoded audio sample value associated with the current frame. The initialization unit 102 may be further configured to switch the audio decoder 100 from the current codec setting to a different codec setting when the determination unit 101 determines that the frame is an immediate playback frame and the audio sample value of the current frame is encoded using a different codec setting. The decoder 100 may then decode the current frame using the current codec setting and discard the additional information when the determination unit 101 determines that the frame is an immediate playback frame and the audio sample value of the current frame is encoded using the current codec setting.

[0106] Referring to the example in Figure 5, an aspect of the present invention relates to an audio encoder that generates a bitstream of encoded audio data including instant playback frames (random access points), wherein the bitstream of encoded audio data represents a sequence of audio sample values ​​and has multiple frames, each frame having associated encoded audio sample values.

[0107] The audio encoder 200 may have a core encoder 202 configured to encode uncompressed audio sample values ​​associated with multiple frames using a predefined codec setting. Using a predefined codec setting may include, for example, using a predetermined sampling rate. In one example, the core encoder 202 may encode the uncompressed audio samples in accordance with the decoding of the MPEG-4 audio standard.

[0108] The audio encoder 200 may further have a buffer 223 configured to store encoded audio sample values ​​of many preceding frames of the current frame among a plurality of frames encoded using a predefined codec setting (indicated by a dashed line).

[0109] For example, when encoding frame N, the associated previous frames N-1, N-2, ... may always be buffered / stored. If instructed to write the IPF to frame N (for example, every 2 seconds to enable dynamic switching), the stored associated previous frames N-1, N-2, ... may then be taken and packaged into the current frame N.

[0110] The audio encoder 200 may have an embedding unit 204 configured to write an immediate playback frame to the current frame among a plurality of frames, and the immediate playback frame may have an encoded audio sample value associated with the current frame and additional information corresponding to the encoded audio sample values ​​of a number of preceding frames of the current frame.

[0111] In the example in Figure 5, the embedding section 204 is defined as part of the audio encoder 200, but alternatively or additionally, the embedding section 204 may also be individually implemented to write an immediate playback frame to the current frame of any of the bitstreams of encoded audio data, or to convert an immediate playback frame in the bitstream of encoded audio data into a “normal” frame by removing additional information from the immediate playback frame. In this case, the embedding section 204 may, but may not be, part of the encoder chain.

[0112] In one embodiment, the embedding section 204 may be further configured to include information regarding predefined codec settings as additional information. In this case, the additional information may supply information regarding predefined codec settings to the decoder. In another embodiment, the embedding section 204 may be further configured to include the additional information in the immediate playback frame. In this way, the additional information can be delivered to the decoder in the bitstream.

[0113] In this embodiment, the generated bitstream of the encoded audio data may be an MPEG-4 audio bitstream.

[0114] In some embodiments, the embedding section 204 may be further configured to embed additional information into the bitstream (for example, for transport) via an MPEG-4 audio bitstream extension mechanism which may be either a data stream element (ID_DSE) or an extended payload element.

[0115] In an embodiment, the embedding unit 204 may be further configured to place either a data stream element (ID_DSE) or an extended payload element at a predefined location within the MPEG-4 audio bitstream, and / or to assign a specific instance tag indicating that the payload of the data stream element (ID_DSE) or extended payload element is additional information. The predefined location may correspond to a first location within the MPEG-4 audio bitstream, i.e., the beginning of the frame, which may carry the decoder settings that may be required to decode frame n (raw_data_block()[n] in Figure 1). Thus, the decoder may infer that if the first element in the frame is an ID_DSE or an extended payload element (which may be contained within an ID_FIL element), this element carries pre-roll data (preceding frame, pre-roll frame).

[0116] In some embodiments, the embedding section 204 may be further configured to include a unique identifier as additional information. Optionally, the unique identifier may indicate a predefined codec setting. The predefined codec setting may then be used by the decoder to decode frame n, as described above. Based on the unique identifier, the decoder may be able to identify additional information in the bitstream and parse the bitstream accordingly.

[0117] In the embodiment, the audio encoder 200 may be further configured not to encode the earliest frame among a number of frames included in the additional information in a time-differential manner or entropically encode it with respect to any frame preceding that earliest frame, and the audio encoder 200 may be further configured not to encode the immediately played frame in a time-differential manner or entropically encode it with respect to any frame preceding the earliest frame among a number of frames preceding the immediately played frame, or with respect to any frame preceding the immediately played frame.

[0118] Referring to the example in Figure 6, an aspect of the present invention relates to a method for generating a bitstream of encoded audio data, including instant playback frames (random access points), using an audio encoder, wherein the bitstream of encoded audio data represents a sequence of audio sample values ​​and has multiple frames, each frame having associated encoded audio sample values.

[0119] The method may include step S201, in which a core encoder encodes uncompressed audio sample values ​​associated with multiple frames using a predefined codec setting. Using a predefined codec setting may include, for example, using a predetermined sampling rate. The method may further include step S202, in which a buffer stores encoded audio sample values ​​of a number of preceding frames of the current frame among the multiple frames encoded using the predefined codec setting.

[0120] The method may include step S203 of writing an immediate playback frame to the current frame among a plurality of frames by an embedding unit, wherein the immediate playback frame has an encoded audio sample value associated with the current frame and additional information corresponding to the encoded audio sample values ​​of a plurality of preceding frames of the current frame.

[0121] In the embodiment, the additional information may further include information regarding predefined codec settings. These predefined codec settings may be used by the decoder in the decoding process described in detail earlier.

[0122] In some embodiments, the immediate playback frame may further include additional information. In this way, the additional information may be carried in the bitstream.

[0123] In this embodiment, the generated bitstream of the encoded audio data may be an MPEG-4 audio bitstream.

[0124] In embodiments, additional information may be embedded in the bitstream (e.g., for transport) by an embedding unit via an MPEG-4 audio bitstream extension mechanism, which is either a data stream element (ID_DSE) or an extended payload element. The extended payload element may be included, for example, at various locations in the MPEG-4 audio bitstream syntax. In embodiments, the extended payload element may be embedded (e.g., for transport) within a fill element (ID_FIL) by an embedding unit.

[0125] As previously detailed, in the embodiment, either a data stream element (ID_DSE) or an extended payload element may be placed by the embedding unit at a predefined location within the MPEG-4 audio bitstream and / or may be assigned a specific instance tag indicating that the payload of the data stream element (ID_DSE) or extended payload element is additional information. The predefined location may always be the first location within the frame, in that it may carry the decoder settings that may be required to decode the current frame.

[0126] In the embodiment, the additional information may further include a unique identifier. Optionally, the unique identifier may indicate a predefined codec setting.

[0127] In this embodiment, the earliest frame among the many frames included in the additional information does not need to be time-differentially encoded or entropically encoded by the audio encoder with respect to any frame preceding that earliest frame, and the immediately played frame does not need to be time-differentially encoded or entropically encoded by the audio encoder with respect to any frame preceding that earliest frame among the many frames preceding the immediately played frame, or with respect to any frame preceding the immediately played frame.

[0128] Referring to the example in Figure 7, an aspect of the present invention relates to a system having two or more audio encoders that generate multiple bitstreams of encoded audio data, each having an immediate playback frame (time-aligned random access point), where each bitstream of encoded audio data represents a sequence of audio sample values ​​and has multiple frames, each frame having associated encoded audio sample values. The number of audio encoders in the system is not limited, while the example in Figure 7 shows a system having two audio encoders. The system may run two audio encoders in parallel for the same content, having different settings, for example, with respect to the bitrate, but inserting immediate playback frames at the same frequency. Both bitstreams thus generated may be stored, for example, on a web server. A client may start playing stream A (for example, at a high bitrate). At some point, the client may decide to switch to a lower bitrate stream B, and thus request the same content, but at a different bitrate. When the first segment of stream B arrives at the decoder, such a segment may always start from the IPF (this may be communicated, for example, via a manifest file following MPEG-DASH). This allows the decoder to output the correct audio from the start.

[0129] Referring again to the example in Figure 7, two audio encoders are shown in parallel, and each audio encoder has a core encoder 202 (encoder core #1, encoder core #2), a buffer (not shown), and an embedded section 204 (IPF insertion).

[0130] In the embodiment, the predetermined sampling rate may be the same for each of the core encoders 202. Otherwise, resampling and additional delay handling may be required on the decoder side. However, the core encoders 202 may be configured to run at different frame rates (e.g., AAC-LC 1024; HE-AAC 2048). Furthermore, the core encoder configuration may require a different number of pre-roll frames p. It may be required that both frames np and p are independently decodeable, i.e., do not depend on information from previous frames (for HE-AAC, they may include an SBR header). After core coding, the decoding time of the IPF may be aligned across different streams.

[0131] In embodiments, the system may further include delay alignment units 201 (delay #1, delay #2) for aligning multiple bitstreams. For seamless switching in the decoder, the decoding times of immediate playback frames (IPFs) may need to be aligned. The delay alignment stage may delay the input PCM samples (uncompressed audio sample values, input audio sample values) to compensate for different encoder / decoder delays.

[0132] Referring to the example in Figure 8, aspects of the present invention relate to a device for generating instant playback frames (random access points) in a bitstream of encoded audio data, or for removing instant playback frames (random access points) from a bitstream of encoded audio data, wherein the bitstream of encoded audio data represents a sequence of audio sample values ​​and has multiple frames, each frame having associated encoded audio sample values.

[0133] The device 300 may have a receiving unit 301 configured to receive a bitstream of encoded audio data, the bitstream of encoded audio data representing a sequence of audio sample values ​​and having multiple frames, each frame having associated encoded audio sample values.

[0134] The device 300 may also have an embedding unit 302 configured to write an immediate playback frame to the current frame among a plurality of frames, the immediate playback frame having an encoded audio sample value associated with the current frame and additional information corresponding to an encoded number of preceding frames of the current frame. In generating an IPF, the embedding unit 302 may operate according to the principle described above in relation to Figures 5 to 7. This operation is sometimes said to correspond to converting a “normal” (non-IPF) frame to an IPF.

[0135] IPF insertion can undertake the actual copying and packaging. By separating core coding and IPF insertion, it may be possible to store a bitstream without IPF and embed the IPF only when necessary before distribution, similar to the above-mentioned device. When configured for seamless switching, each IPF may need to include a stream configuration ASC.

[0136] In one embodiment, the device 300 may further have a buffer configured to store encoded audio sample values ​​of a number of preceding frames of the current frame among a plurality of frames.

[0137] In some embodiments, the embedding section 302 may be further configured to remove additional information from the immediately played frame that corresponds to the encoded audio sample values ​​of a number of preceding frames of the current frame. This is sometimes said to correspond to converting the IPF into a “normal” frame. For example, an audio encoder may generate only the IPF, which may then be later removed by the embedding section depending on the constraints that each channel / service may have.

[0138] Referring hereto to the example in Figure 9, aspects of the present invention may include a non-temporary digital storage medium storing a computer program, wherein the computer program performs the methods described herein when executed by a computer or processor. Figure 9 shows an example of a device 400 equipped with a processor 401 capable of executing the computer program. Alternatively, device 400 may represent each computer.

[0139] The methods and systems described herein may be implemented as software, firmware, and / or hardware. Certain components may be implemented, for example, as software running on a digital signal processor or microprocessor. Other components may be implemented, for example, as hardware and / or as application-specific integrated circuits. Signals produced by the methods and systems described herein may be stored in a medium such as random-access memory or optical storage medium. They may be transmitted over a network such as a radio network, satellite network, wireless network, or wireline network, for example, the Internet. Typical devices using the methods, apparatus, and systems described herein are portable electronic devices or other consumer devices used to store and / or render audio signals.

[0140] It should be noted that the specification and drawings / figures merely represent the principles of the proposed methods, systems, and apparatus. Those skilled in the art will be able to implement various arrangements that embody the principles of the invention and are included within its spirit and scope, even if not expressly described or illustrated herein. Furthermore, all examples and embodiments described herein are expressly intended primarily for illustrative purposes only, to help the reader understand the principles of the proposed methods. Moreover, all descriptions herein providing the principles, aspects, and embodiments of the invention, as well as specific examples thereof, are intended to encompass their equivalents.

[0141] [Cross-reference of related applications] This application claims priority to U.S. Provisional Patent Application No. 62 / 720680 (reference number: D18080USP1), filed on 21 August 2018. This U.S. application, on which the priority claim is based, is incorporated herein by reference.

Claims

1. An audio decoder for decoding a bitstream of encoded audio data, wherein the bitstream of encoded audio data represents a sequence of audio sample values ​​and has multiple frames, each frame having associated encoded audio sample values, A determination unit configured to determine whether a frame of the bitstream of the encoded audio data is an immediate playback frame having an encoded audio sample value associated with the current frame and additional information, When the determination unit determines that the frame is an immediately playable frame, an initialization unit is configured to initialize the decoder. It has, The additional information includes information about the codec settings used to encode the audio sample values ​​associated with the current frame, the bitstream of the encoded audio data is an MPEG-4 audio bitstream, the additional information is carried by an MPEG-4 audio bitstream extension mechanism which is an extension_payload element, the extension_payload element includes EXT_AUDIO_PRE_ROLL which identifies the type of the extension_payload element as a new extension payload element type, the extension_payload element is located at a first position in the MPEG-4 audio bitstream, the extension_payload element is contained within a fill element (ID_FIL), The additional information includes encoded audio sample values ​​of a number of frames preceding the immediately played frame, wherein the encoded audio sample values ​​of the preceding frames are encoded using the same codec settings as the current frame. The number of preceding frames corresponding to the pre-roll frame corresponds to the number of frames required by the decoder to constitute the entire signal so that, whenever an immediate playback frame is decoded, it is in a position to output a valid audio sample value associated with the current frame. Initializing the decoder involves decoding the encoded audio sample value configured with the additional information before decoding the encoded audio sample value associated with the current frame. The initialization unit is configured to switch the audio decoder from the current codec setting to the different codec setting when the determination unit determines that the frame is an immediate playback frame, and the audio sample value of the current frame is encoded using a different codec setting. The decoder is configured to decode the current frame using the current codec settings and discard the additional information when the determination unit determines that the frame is an immediately playable frame and the audio sample value of the current frame is encoded using the current codec settings. Audio decoder.

2. A method for decoding a bitstream of encoded audio data, wherein the bitstream of encoded audio data represents a sequence of audio sample values ​​and has a plurality of frames, each frame having an associated encoded audio sample value, The step of determining whether a frame of the encoded audio data bitstream is an immediate playback frame having an encoded audio sample value associated with the current frame and additional information, wherein the additional information includes information about the codec settings used to encode the audio sample value associated with the current frame, the bitstream of the encoded audio data is an MPEG-4 audio bitstream, the additional information is carried by an MPEG-4 audio bitstream extension mechanism which is an extension_payload element, the extension_payload element includes EXT_AUDIO_PRE_ROLL which identifies the type of the extension_payload element as a new extension payload element type, the extension_payload element is located at a first position in the MPEG-4 audio bitstream, and the extension_payload element is contained within a fill element (ID_FIL), The aforementioned additional information includes encoded audio sample values ​​of a number of frames preceding the immediate playback frame, The encoded audio sample values ​​of the preceding frame are encoded using the same codec settings as the immediately played frame. The number of preceding frames corresponding to the pre-roll frame corresponds to the number of frames required by the decoder to constitute the entire signal so that whenever an immediate playback frame is decoded, it is in a position to output a valid audio sample value associated with the current frame, the determination step, The steps include initializing the decoder when it is determined that the frame is an immediate playback frame, wherein the initialization comprises decoding the encoded audio sample value configured with the additional information before decoding the encoded audio sample value associated with the current frame, When it is determined that the frame is an immediate playback frame, and the audio sample values ​​of the immediate playback frame are encoded using a different codec setting, the decoder is switched from the current codec setting to the different codec setting. If it is determined that the frame is an immediate playback frame, and the audio sample value of the immediate playback frame is encoded using the current codec settings, the steps include: decoding the immediate playback frame using the current codec settings and discarding the additional information; A method of having.

Images