Methods, apparatus, and systems for sideloading packetized media streams

Abstract

This disclosure relates to methods, apparatus, and systems for sideloading packetized media streams. [Solution] In one embodiment, the device includes a receiver that receives a bitstream and a splitter that identifies the packet types in the bitstream and separates them into a main stream and an auxiliary stream based on the identification of the packet type values ​​in the bitstream.

Description

Technical Field

[0001] [Related Applications] This application claims priority to the following prior applications: US Provisional Application No. 62 / 634,136 (Reference No.: D18027USP1), filed on February 22, 2018, US Provisional Application No. 62 / 641,098 (Reference No.: D18027USP2), filed on March 9, 2018, EP Application No. 18166319.6 (Reference No.: D18027EP), filed on April 9, 2018, and US Provisional Application No. 62 / 697,536 (Reference No.: D18027USP3), filed on July 13, 2018. These applications are hereby incorporated by reference.

[0002] [Technical Field] The present disclosure relates to providing devices, systems, and methods for packetized media processing.

Brief Description of the Drawings

[0003] [Figure 1] Exemplary Device #1 and Exemplary Device #2 are shown. [Figure 2] An exemplary MPEG-H 3D audio packet stream according to the present invention is shown. [Figure 3] Exemplary Device #1 and Exemplary Device #3 are shown. [Figure 4A] Another example of a primary decoder / renderer according to the present invention is schematically shown. [Figure 4B] Another example of a primary decoder / renderer according to the present invention is schematically shown. [Figure 5A] Yet another example of a primary decoder / renderer according to the present invention is schematically shown. [Figure 5B] Yet another example of a primary decoder / renderer according to the present invention is schematically shown. [Figure 6] Exemplary packet types for achieving the means of the present invention based on MHAS are shown.

Detailed Description of the Invention

[0004] In the installation of consumer electronics (CE) for home use, functionality can extend across several devices (e.g., set-top boxes, TV sets, AVR receivers). These devices are connected via a standardized interface (e.g., HDMI®).

[0005] The first device can receive media streams from broadcast and / or broadband connections. This first device may further possess advanced intelligence (e.g., "smart speaker" functionality). The second device may be dedicated to media decoding, rendering, and presentation to the user.

[0006] Typically, a media program is received by device #1 and transmitted to device #2 for playback. This media program may be known as the "main program." Occasionally, or additionally, different media programs (such as advertisements) received from different transmission channels or media generated by the "smart device" capabilities within device #1 are both usually presented in different media formats and should be inserted into or overlaid on the main media program.

[0007] This can be achieved by decoding both the main and auxiliary programs into a generalized, standardized, uncompressed representation, switching the streams, or mixing / rendering them into a combined representation, and then re-encoding the resulting media stream into the encoded transmission format. This method can be performed on any device, such that all operations are carried out on device #1. However, this may require high computational power, and the rendering stage and intermediate representations may not provide an optimal fit for the actual playback system on device #2.

[0008] Figure 1 shows exemplary apparatus #1 and exemplary apparatus #2. Apparatus #1 may include a primary receiver 101, a secondary receiver / local engine 103, and a packetized stream integrator 102. The packetized stream integrator 102 may integrate a data stream received from the primary receiver 101 (e.g., the main stream) and a data stream received from the secondary receiver / local engine 103 (e.g., an auxiliary data stream). Apparatus #2 may include a stream splitter 201, a primary decoder / renderer 202, and a secondary decoder / renderer 203. The stream splitter 201 may separate streams such as the main stream (which may then be provided to the primary decoder / renderer 202) and the auxiliary stream (which may then be provided to the secondary decoder / renderer 203). Both decoder output signals may be presented to the viewer simultaneously.

[0009] To overcome the aforementioned limitations, device #1 can directly transmit the main media stream it receives to device #2. This mode of device #1 is called "pass-through" mode. However, downstream of this approach, only a standardized interface is specified for transmitting a single data stream for video and audio. Therefore, the second media stream cannot, in principle, be transmitted to device #2 for playback via the interface.

[0010] The present invention provides a method and apparatus for a primary receiver that receives packetized media data, a secondary receiver that receives additional media streams, and a transmitter interface that transmits packetized media data.

[0011] This invention proposes integrating a second auxiliary media stream into a packetized main media stream by the following means.

[0012] Media streams, represented as packetized streams, typically use packet type identifiers to differentiate multiple substreams along with the overall media stream. To transmit additional (media) data unrelated to the main media stream, the first device #1 can encapsulate the additional data in packets formatted according to the main media stream and labeled with a special tag in the packet header. This special tag triggers the receiving device #2 to remove the packet carrying the additional media stream. Optionally, device #2 may then provide the additional media stream to a secondary decoder / renderer 203 instance. Simultaneously, the main stream is received by the primary decoder / renderer 202.

[0013] For example, legacy receiving devices that do not recognize this new tag in the packet header for additional encapsulated media streams are expected to ignore these packets.

[0014] The tags may be provided in any encoded audio data stream environment, such as MPEG-H, AC-4, Dolby Digital+, etc.

[0015] If the additionally inserted data stream exceeds the effective data rate compared to the original media stream, the receiving device must filter the input packetized stream and optionally remove the additional data packets to maintain the receiver buffer model of the downstream connected (legacy) media decoder.

[0016] Furthermore, Figure 3 shows that device #1 may include a primary receiver 101, a secondary receiver / local engine 103, and a packetized stream integration unit 102. The packetized stream integration unit 102 may integrate the data stream received from the primary receiver 101 (e.g., the main stream) and the data stream received from the secondary receiver / local engine 103 (e.g., the auxiliary data stream). Device #3 may include only the primary decoder / renderer 301 and not include a stream splitter. In one example, it may not be possible to provide the auxiliary stream to the secondary decoder / renderer. In this example, the modified primary decoder may internally decode / convert / mix the auxiliary stream together with the main stream.

[0017] <MPEG-Hエコシステム> MPEG-H 3D Audio, compliant with ISO / IEC 23008-3, is encapsulated in the MHAS format. This format utilizes a packetized structure, where each packet consists of a packet header and a packet payload. The payload is arbitrary binary data, and the header specifies the type and length of the payload. (Additionally available labels can be used to distinguish between multiple instances, but are not used here.)

[0018] By assigning a new MHAS packet type to the secondary media stream (exemplified by PACTYP_MEDIA), additional audio data, optionally represented as uncompressed PCM data further specified using the RIFF / WAV format, or as compressed audio data such as MPEG-4 audio according to ISO / IEC 14496-3, or any other encoded representation (e.g., according to ATSCA / 52 or ETSI TS 103 190), can be encapsulated in an MHAS packet and thus integrated into the main MPEG-H 3D Audio stream. Different formats to be encapsulated can be distinguished by different packet types (e.g., PACTYP_PCM, PACTYP_MPEG4AUDIO, ...) or by additional indicators forming the subheader of the MHAS packet, as shown in the example below.

[0019] (Media) data may request configuration data, but since it is not represented as a self-sufficient stream, this data may be encapsulated in the header of the container MHAS packet, or an additional MHAS packet (e.g., PACTYP_MEDIA_CONFIG, or another type of MHAS packet name indicating configuration, such as PACTYP_PCMCONFIG) may be assigned, which may further carry information in the form of additional data. The MHAS packet type may carry configuration information for the PCM payload data in order to supply the configuration information to the decoder. For example, if an MHAS packet type for configuration information (e.g., PACTYP_MEDIA_CONFIG or PACTYP_PCMCONFIG) exists in the bitstream (e.g., after PACTYP_MEDIA_CONFIG), the PCT data configuration information in the form of a data structure (e.g., pcmDataConfig()) may be supplied to the decoder.

[0020] Typically, an MHAS packet type (e.g., PACTYP_PCMDATA) may be used to embed a PCM payload corresponding to a PCM signal defined in a configuration structure and to supply PCM data in the form of a PCM data payload structure to a decoder. When an MHAS packet type (e.g., PACTYP_PCMDATA) is present within a bitstream, a PCM data payload structure (e.g., pcmDataPayload()) may be used during decoding.

[0021] Figure 2 shows an exemplary MPEG-H 3D audio packet stream according to the present invention.

[0022] In one example, the present invention may be based on identification information based on the following syntax changes: 1) Change the syntax of Table 220 - MHASPacketPayload() as follows. [Table 1] 2) Assign an appropriate MHASPacketType to PACTYP_MEDIA in Table 223. Alternatively, the value of MHASPacketType may vary or may depend on a preset value, e.g., a value published by a standardization organization in a final standardized document.

[0023] 3) The following media types should be specified using the mhasMediaDataType list. [Table 2] Figure 2 illustratively shows a packetized main stream including exemplary packets conforming to MPEG-H 3D audio along with 3D audio data (exemplarily, configuration data, e.g., PACTYP_MPEGH3DACFG as metadata, and encoded audio data, e.g., PACTYP_MPEGH3DAFRAME are included).

[0024] Furthermore, Figure 2 illustrates a packetized secondary stream (auxiliary stream) containing an exemplary packet according to the present invention, indicated by header information referencing the exemplary additional packet type PACTYP_MEDIA described above, which includes exemplary compression format, for example, additional audio data (auxiliary audio data / secondary audio data) of MPEG4Audio.

[0025] Alternatively or additionally, additional audio data, which is exemplified here as MPEG-H 3D audio, can be packetized into packets having a header, following the packetization mainstream format which includes a subheader indicating a different encapsulated format as described above.

[0026] According to exemplary embodiments of the present invention, the main stream and auxiliary (secondary) streams can be integrated by a stream integration unit, for example, by a packetized stream integration unit 102.

[0027] The output stream (integrated stream) contains packets related to the encoded audio data of the main stream and packets related to the audio data of the auxiliary stream within a single packetized bitstream in the same format (e.g., the exemplary MPEG-H 3D audio in Figure 2).

[0028] Note that legacy MPEG-H 3D audio decoders do not need to understand newly added packet types (e.g., PACTYP_MEDIA), and such legacy MPEG-H 3D audio decoders may ignore or dump packets that have newly added packet types (e.g., PACTYP_MEDIA) indicated in the header. Such legacy MPEG-H 3D audio decoders can still decode the audio data related to the mainstream, but they will not process any additional auxiliary / secondary audio data.

[0029] To decode and process an integrated stream having a main stream and an auxiliary stream, the decoder device may be modified to include a modified decoder capable of filtering and decoding / processing packets related to auxiliary audio data.

[0030] Figure 4 schematically shows another example of a primary decoder / renderer according to the present invention.

[0031] Figure 4 shows how this is achieved in an MPEG-H 3D Audio decoder, where the media format is PCM data. Exemplarily, the primary decoder / renderer 301 is implemented to include an MPEG-H 3D Audio Core Decoder M1 and an object renderer M2, which is an associated rendering unit, for example, defined according to the MPEG-H 3D Audio (ISO / IEC 23008-3) standard.

[0032] The modified decoder 301 may further filter and remove MHAS packets that have a header indicating a new additional packet type (e.g., PACTYP_MEDIA), and input packets containing auxiliary audio data to the format conversion unit 301c, and then to the sample rate converter (for example, the sample rate converter M3 is located in the decoder architecture downstream of the MPEG-H 3D Audio Core Decoder M1, as defined in accordance with the MPEG-H 3D Audio (ISO / IEC23008-3) standard).

[0033] Therefore, the modified decoder 301 may perform sample rate conversion (e.g., by a sample rate converter M3) and format conversion (e.g., by a format conversion unit 301c1) on the input media data (MHASPacketType==PACTYP_MEDIA) to adapt the media sampling rate and channel layout to the output sample rate and the decoder's channel configuration. Furthermore, the modified decoder may mix the input media data or the sample rate-converted input media data with the audio media data generated by the MPEG-H 3D Audio Core Decoder M1 in a mixer (for example, exemplary, a mixer unit M4 exists in a decoder architecture downstream of the MPEG-H 3D Audio Core Decoder M1 as defined according to the MPEG-H 3D audio (ISO / IEC23008-3) standard).

[0034] The example in Figure 4 illustrates the relationship with auxiliary media data, which includes uncompressed media data such as PCM data. It should be noted that if the uncompressed media data, such as PCM data, further includes, for example, location information of a 3D audio device, or is related to corresponding location metadata, the auxiliary data can be further processed by a location data processing object renderer, such as an object renderer M2, which exists in the decoder architecture downstream of the MPEG-H 3D Audio CoreDecoder M1 as defined in accordance with the MPEG-H 3D Audio (ISO / IEC 23008-3) standard.

[0035] Figure 5 schematically illustrates yet another example of a primary decoder / renderer according to the present invention. Exemplarily, the primary decoder / renderer 301 is implemented to include an MPEG-H 3D Audio Core Decoder M1 and an object renderer M2, an associated rendering unit as defined, for example, according to the MPEG-H 3D Audio (ISO / IEC 23008-3) standard.

[0036] Figure 5 shows how the above is achieved in an MPEG-H 3D Audio decoder, where the media format is encoded data (e.g., MPEG-4 Audio). The modified decoder 301 may decode the input media data (MHASPacketType==PACTYP_MEDIA) using an additional decoder 301c2 (a converter and / or decoder configured to convert and / or decode auxiliary media data) not necessarily specified in ISO / IEC23008-3. Furthermore, the modified decoder may mix these decoded media data with audio media data generated by the MPEG-H 3D Audio Core Decoder M1 as defined in accordance with the MPEG-H 3D audio (ISO / IEC23008-3) standard.

[0037] <Time-alignment of multiple MHAS substreams originating from different sources> In exemplary embodiments of the present invention, additional time-matching units may be provided to provide time-matching for auxiliary stream packets, for example, for multiple MHAS substreams originating from different sources.

[0038] According to section 14.6 of ISO / IEC 23008-3, "It is assumed that substreams are generated by the same encoder and (and therefore) various input streams are perfectly aligned and have no phase offset." In this case, frame alignment may be achieved using MHASPacketLabel numbers. According to the proposed method of the present invention, the above constraint is no longer taken for granted. Due to different frame durations or sampling rates of different codecs, the time offset of consecutive MHAS packets of a secondary stream integrated with the MHAS mainstream changes over time. In each particular time slot, the time offset of the secondary stream relative to the mainstream needs to be signaled, for example, in associated packets of the auxiliary stream indicating the packet type related to metadata associated with the media data contained in the payload of the auxiliary stream packets, as shown in Figure 6.

[0039] Figure 6 shows exemplary packet types for achieving the means of the present invention based on MHAS. For example, here another MHAS packet type is defined to carry appropriate time offset signaling. In order to link this signaling to the corresponding stream, the time offset packet must have the same MHASPacketLabel number assigned as an MHASPacket of type PACTYP_MEDIA that it references. From Figure 6, it is clear that a direct one-to-one relationship of stream data corresponding to one specific time slot of each stream is not guaranteed, but two or more time slots from one stream may correspond in a timely manner to one time slot from another stream.

[0040] Another option for signaling the time offset is to add this time offset to the MHAS packet itself of type PACTYP_MEDIA.

[0041] In view of the foregoing, in some exemplary embodiments of the present invention, the conversion and / or decoding unit of the modified primary decoder / renderer 301 in the example of Figures 4 and / or 5 may include a time alignment unit for time alignment of auxiliary stream data with main stream data, for example, upstream of a mixer such as the mixer unit M4 in the example of Figures 4 and 5, or upstream of a sampler (e.g., sample rate converter M3) in the example of Figures 4 and 5, or as part of the secondary decoder / renderer 203 in the example of Figure 1.

[0042] <Mixing control of the main audio stream and secondary audio stream> Additional data is required to control the mixing of the secondary (auxiliary) audio stream with the main audio stream. In particular, this data may include static or dynamic gain sequences, which are exemplary formed as ISO / IEC 23003-4 DynamicRangeControl data, to process the main stream when the secondary audio stream is played back. This data is typically generated by device #1 and incorporated into the stream in a separate MHAS packet (e.g., with identifier PACTYP_MPEGH_MEDIA_CFG) and added to the secondary stream header, or further added by any other type of stream encapsulation.

[0043] In view of the foregoing, in some exemplary embodiments of the present invention, the conversion and / or decoding unit of the modified decoder 301 in the example of Figures 4 and / or 5 may include a gain adjustment unit that applies static and / or dynamic gain (for example, for volume adjustment), for example, upstream of a mixer such as the mixer unit M4 in the example of Figures 4 and 5, or upstream of a sampler (e.g., sample rate converter M3) in the example of Figures 4 and 5, or as part of the secondary decoder / renderer 203 in the example of Figure 1.

[0044] <Dolbyエコシステム> Furthermore, Dolby AC-4 (ETSI TS103 190) and Dolby Digital and Dolby Digital Plus (ETSI TS102 366) offer the possibility of carrying arbitrary binary data within the MEDF payload. This can be used to carry the same or similar data as defined above (MPEG-H Ecosystem).

[0045] For this purpose, the syntax element emdf_info() defined in ETSI TS103 190, or the syntax element emdf_container() defined in ETSI TS102 366, Annex H, and their underlying elements may be used. To do this, one or more emdf_payload_id definitions can simply be defined. These can be used to identify binary data having the same or similar format as those described above under PACTYP_MEDIA and / or PACTYP_MPEGH_MEDIA_CFG.

[0046] System sound mixing of media streams containing uncompressed / unencoded data may be achieved in the same way as Dolby AC-4 or Dolby Digital / Dolby Digital Plus, as shown in Figures 4 and 5. The difference here is that the distinction of which signal path to use is considered based on the emdf_payload_id value, rather than the MHASPacketType.

[0047] The media stream, main stream, and side data stream solved by the present invention may be of the following types: • Audio streams, both compressed and uncompressed. • Video stream, ·subtitle.

[0048] The present invention may also be applied to video display devices (monitors) in which overlay pictures, videos, or text are transmitted to the main (typically a compressed video stream) via a standardized interface connection.

[0049] F • US20170223429A1,EP3149955A1 ·ISO / IEC23008-3:(MPEG-H 3d Audio, 2nd Edition) The Enumerated Exemplary Embodiments (EEEs) of this Disclosure relate to the following:

[0050] EEE1. A method for processing audio signals, The steps include receiving an integrated packetized media bitstream which includes packets associated with a main stream indicating main audio information and packets associated with a side data stream indicating auxiliary audio information, The steps include identifying the packet type value in the packet header information of the integrated packetized media bitstream, A method comprising the step of separating the integrated packetized media bitstream into a main stream indicating main audio information and a side data stream indicating auxiliary audio information, based on the identification of a packet type value in the header information of the packets of the integrated packetized media bitstream.

[0051] EEE2. The method according to EEE1, further comprising the step of mixing an audio output signal based on an output signal obtained from the main audio information of the main stream and an output signal obtained from the auxiliary audio information of the side data stream.

[0052] EEE3. The output signals from the main and auxiliary audio information are simultaneously output to the listener, as described in EEE2.

[0053] EEE4. The method according to EEE1, further comprising the step of decoding the mainstream with a primary decoder.

[0054] EEE5. The method of EEE1, further comprising the step of decoding the side data stream with a secondary decoder when the side data stream relates to compressed auxiliary audio information.

[0055] EEE6. The method according to EEE1, further comprising the step of converting media data contained in packets of the side data stream by a converter when the side data stream relates to compressed auxiliary audio information.

[0056] EEE7. The compression auxiliary audio information includes MPEG-4 audio data, as described in EEE5 or EEE6.

[0057] EEE8. The method according to EEE1, wherein the side data stream relates to uncompressed auxiliary audio information.

[0058] EEE9. The uncompressed auxiliary audio information includes PCM data, as described in EEE8.

[0059] EEE10. The method according to EEE1, wherein the packet type indicates an internal raw format, in particular a decoder internal raw format.

[0060] EEE11. The method of EEE1, further comprising the step of performing signal processing on the side data stream.

[0061] EEE12. The method according to EEE11, wherein the step of performing signal processing on the side data stream is performed before the step of mixing the audio output signal based on the output signal obtained from the main audio information of the main stream and the output signal obtained from the auxiliary audio information of the side data stream.

[0062] EEE13. The method according to EEE11, wherein the step of performing signal processing on the side data stream includes the step of performing gain adjustment.

[0063] EEE14. Gain adjustment performed based on sexual gain or dynamic gain, as described in EEE13.

[0064] EEE15. The method according to EEE13, further comprising the step of receiving mixing information relating to at least one of the static gain or the dynamic gain.

[0065] EEE16. Gain adjustment is performed in the same manner as in EEE13, for volume adjustment.

[0066] EEE17. The method according to EEE11, wherein the step of performing signal processing on the side data stream includes the step of performing time matching.

[0067] EEE18. The method according to EEE17, wherein the step of performing time alignment is performed to time-align the auxiliary audio information of the side data bitstream with the main audio information of the main bitstream.

[0068] EEE19. The method of EEE17 or EEE18, further comprising the step of receiving time alignment information relating to the time alignment of the auxiliary audio information of the side data bitstream with the main audio information of the main bitstream.

[0069] EEE20. The method according to EEE1, wherein the packet type is defined to carry time offset signaling, particularly for time matching.

[0070] EEE21. The method of EEE1, further comprising the step of receiving an instruction for a time offset of the side data stream relative to the main stream.

[0071] EEE22. The method according to EEE1, wherein the step of performing signal processing on the side data stream includes object rendering when the auxiliary audio information is related to positional information.

[0072] EEE23. The method according to EEE22, wherein object rendering is performed by an object renderer that performs object rendering of the main and side data streams.

[0073] EEE24. The method according to EEE11, wherein the step of performing signal processing on the side data stream includes format conversion.

[0074] EEE25. The method according to EEE1, further comprising the step of decoding the mainstream using a standardized MPEG-H 3D audio decoder.

[0075] EEE26. The method according to EEE25, further comprising the step of mixing media data contained in packets of the side data stream with media data generated by the standardized MPEG-H 3D audio decoder.

[0076] EEE27. The method according to EEE26, wherein the media data contained in the packet of the side data stream is uncompressed data, in particular PCM data, or compressed data, in particular MPEG4 audio data.

[0077] EEE28. The method according to EEE26, wherein the media data contained in the packets of the side data stream is decoded by a decoder that is not standardized for MPEG-H 3D audio.

[0078] EEE29. The method according to EEE1, which includes the integrated packetized media bitstream, including HMAS packets.

[0079] EEE30. A method for processing audio signals, A step of receiving the main stream which indicates the main audio information, The steps include receiving a side data stream indicating auxiliary audio information or generating a side data stream based on auxiliary audio information, A method comprising the steps of: integrating the main stream and the side data stream to generate an integrated packetized media bitstream which includes packets related to the main stream indicating main audio information and packets related to the side data stream indicating auxiliary audio information.

[0080] EEE31. The method of EEE30, further comprising the step of packetizing media data indicating the auxiliary audio information into packets in the format of the integrated packetized media bitstream.

[0081] EEE32. The method according to EEE30, wherein the side data stream relates to compressed auxiliary audio information.

[0082] EEE33. The method according to EEE32, wherein the compression auxiliary audio information includes MPEG-4 audio data.

[0083] EEE34. The method according to EEE30, wherein the side data stream relates to uncompressed auxiliary audio information.

[0084] EEE35. The uncompressed auxiliary audio information includes PCM data, as described in EEE34.

[0085] EEE36. The header information of the packets of the integrated packetized media bitstream is as described in EEE30, indicating the packet type.

[0086] EEE37. The method according to EEE36, wherein the packet type value of a packet associated with the side data stream indicating auxiliary audio information indicates media data associated with the auxiliary audio information.

[0087] EEE38. The method according to EEE30, which includes the integrated packetized media bitstream, including HMAS packets.

[0088] EEE39. Audio signal processing equipment, A receiver configured to receive an integrated packetized media bitstream, which includes packets associated with a main stream indicating main audio information and packets associated with a side data stream indicating auxiliary audio information, A device including a splitter configured to separate the integrated packetized media bitstream into a main stream indicating main audio information and a side data stream indicating auxiliary audio information, based on the identification of a packet type value in the header information of the packets of the integrated packetized media bitstream.

[0089] EEE40. The apparatus according to EEE39, further comprising a mixer configured to mix audio output signals based on an output signal obtained from the main audio information of the main stream and an output signal obtained from the auxiliary audio information of the side data stream.

[0090] EEE41. The apparatus according to EEE39, further comprising a primary decoder configured to decode the mainstream.

[0091] EEE42. The apparatus according to EEE39, further comprising a secondary decoder configured to decode the side data stream when the side data stream relates to compressed auxiliary audio information.

[0092] EEE43. The apparatus according to EEE39, further comprising a converter configured to convert media data contained in packets of the side data stream when the side data stream relates to compressed auxiliary audio information.

[0093] EEE44. The compression auxiliary audio information includes MPEG-4 audio data, as described in EEE42 or EEE43.

[0094] EEE45. The side data stream relates to uncompressed auxiliary audio information, as described in EEE39.

[0095] EEE46. The uncompressed auxiliary audio information includes PCM data, as described in EEE45.

[0096] EEE47. The packet type refers to the internal raw format, in particular the decoder internal raw format, as described in EEE39 for the equipment.

[0097] EEE48. The apparatus according to EEE39, further comprising a gain adjustment unit configured to perform gain adjustment applied to the side data stream.

[0098] EEE49. The apparatus according to EEE39, further comprising a time matching unit configured to perform time matching applied to the side data stream.

[0099] EEE50. The apparatus described in EEE39, further comprising an object renderer configured to perform object rendering applied to the side data stream.

[0100] EEE51. The object renderer is the device described in EEE50, which is included in the primary decoder.

[0101] EEE52. The apparatus according to EEE39, further comprising a format conversion unit configured to perform a format conversion applied to the side data stream.

[0102] EEE53. The apparatus described in EEE39, further comprising a standardized MPEG-H 3D audio decoder for decoding the aforementioned mainstream.

[0103] EEE54. The integrated packetized media bitstream, including HMAS packets, is the device described in EEE39.

[0104] EEE55. Audio signal processing equipment, A receiver configured to receive a mainstream stream indicating main audio information, A local engine configured to generate a side data stream indicating the auxiliary audio information based on the auxiliary audio information, and / or a receiver configured to receive the side data stream, A device including an integrator configured to integrate the main stream and the side data stream to generate an integrated packetized media bitstream that includes packets related to the main stream indicating main audio information and packets related to the side data stream indicating auxiliary audio information.

[0105] EEE56. The apparatus according to EEE55, further comprising a packetization unit configured to packetize media data indicating the auxiliary audio information into packets in the format of the integrated packetized media bitstream.

[0106] EEE57. A system including the equipment described in EEE55 and the equipment described in EEE39. [Explanation of symbols]

[0107] 101 Primary receiver 102 Packetized Stream Integration Unit 103 Secondary Receiver / Local Engine 201 Stream Splitter 202 Primary Decoder / Renderer 203 Secondary Decoder / Renderer

Claims

1. A method for processing a merged packetized media bitstream, the method being: The steps include: extracting the main stream containing MPEG-H 3D audio data encapsulated in MHAS format from the merged packetized media bitstream; A step of extracting an auxiliary stream from the merged packetized media bitstream, which includes additional audio data encapsulated as packets in the MHAS format, wherein the packets have a new MHAS packet type, the format of the packet payload of the packets differs from the MPEG-H 3D format of the MPEG-H 3D audio data, and the new MHAS packet type indicates an internal raw format. A step of receiving mixing information between an auxiliary audio stream and the main stream, wherein the mixing information relates to at least one of static gain or dynamic gain; A step of identifying a time offset signaling indicating the existence of a time offset of the auxiliary stream relative to the main stream, wherein the time offset signaling indicates that the auxiliary stream signal needs to be offset so that it arrives at the mixer aligned with the main stream. The steps include aligning the additional audio data temporally with the MPEG-H 3D audio data based on the aforementioned time offset signaling, The steps include mixing the main stream with the auxiliary audio stream based on the mixing information, A method that includes this.

2. A non-temporary storage medium containing, storing, or recording a digital audio signal mixed according to the method described in claim 1.

3. A non-temporary computer-readable storage medium storing executable instructions that cause a computer to perform the method described in claim 1.

4. An audio decoding device that processes a merged packetized media bitstream, wherein the device A splitter configured to extract a main stream and an auxiliary stream from the merged packetized media bitstream, wherein the main stream includes MPEG-H 3D audio data encapsulated in MHAS format, the auxiliary stream includes additional audio data encapsulated as packets in MHAS format, the packets have a new MHAS packet type, the format of the packet payload of the packets differs from the MPEG-H 3D format of the MPEG-H 3D audio data, and the new MHAS packet type indicates an internally unprocessed format. A receiver that receives mixing information between an auxiliary audio stream and the main stream, wherein the mixing information relates to at least one of static gain or dynamic gain, An identification unit configured to identify a time offset signaling indicating the existence of a time offset of the auxiliary stream relative to the main stream, wherein the time offset signaling indicates that the auxiliary stream signal needs to be offset so that it arrives at the mixer aligned with the main stream; A time alignment unit that temporally aligns the additional audio data with the MPEG-H 3D audio data based on the aforementioned time offset signaling, A mixer that mixes the main stream with the auxiliary audio stream based on the mixing information, Equipment including.

Images