Method, apparatus, and system for bypass loading processing of packed media stream
By encapsulating additional media streams into packets with dedicated tags, the method addresses the challenge of integrating and rendering multiple media streams in consumer electronics, ensuring efficient and optimal playback across various formats.
Patent Information
- Application Number
- JP2024150451
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2018-07-13
- Filing Date
- 2024-09-02
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2039-02-22
AI Technical Summary
Existing consumer electronics systems face challenges in efficiently integrating and rendering multiple media streams, such as main and auxiliary programs, due to the limitations of standardized interfaces that can only convey a single data stream, requiring high computational power and suboptimal adaptation to playback systems.
The method involves encapsulating additional media streams into packets of the main media stream using dedicated tags in the packet header, allowing legacy devices to ignore these packets, and integrating them with the main stream for simultaneous playback, while maintaining compatibility with standardized interfaces.
This approach enables efficient integration and rendering of multiple media streams, including uncompressed and compressed formats, without overburdening computational resources, and ensures optimal adaptation to playback systems.
Smart Images

Figure 2024159963000001_ABST
Abstract
Description
[Technical field]
[0001] [Related Applications] This application claims priority to the following earlier applications: US Provisional Application No. 62 / 634,136 (Reference No. D18027USP1), filed February 22, 2018, US Provisional Application No. 62 / 641,098 (Reference No. D18027USP2), filed March 9, 2018, EP Application No. 18166319.6 (Reference No. D18027EP), filed April 9, 2018, and US Provisional Application No. 62 / 697,536 (Reference No. D18027USP3), filed July 13, 2018, all of which are incorporated herein by reference.
[0002] [Technical field] The present disclosure is directed to providing apparatus, systems, and methods for packetized media processing. [Brief description of the drawings]
[0003] [Figure 1] An exemplary device #1 and an exemplary device #2 are shown. [Diagram 2] 2 illustrates an exemplary MPEG-H 3D audio packet stream in accordance with the present invention. [Diagram 3] An exemplary device #1 and an exemplary device #3 are shown. [Figure 4A] 4 illustrates diagrammatically another example of a primary decoder / renderer according to the invention; [Figure 4B] 4 illustrates diagrammatically another example of a primary decoder / renderer according to the invention; [Figure 5A] 4 illustrates diagrammatically yet another example of a primary decoder / renderer according to the invention; [Figure 5B] 4 illustrates diagrammatically yet another example of a primary decoder / renderer according to the invention; [Figure 6] 1 shows exemplary packet types for implementing the methods of the present invention based on MHAS. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0004] In a home Consumer Electronics (CE) installation, functionality may be spread across several devices (e.g., set-top boxes, TV sets, AVR receivers) where such devices are connected by standardized interfaces (e.g., HDMI®).
[0005] A first device can receive media streams from a broadcast and / or broadband connection. This first device may further have advanced intelligence (e.g., "smart speaker" functionality). A second device may be dedicated to media decoding, rendering, and presentation to a user.
[0006] Typically, a media program is received by device #1 and transmitted to device #2 for playback. This media program may be known as the “main program.” Occasionally, or in addition, different media programs (such as advertisements) received from different transmission channels or media generated by “smart device” capabilities within device #1, both of which are typically presented in different media formats, should be inserted into or overlaid on the main media program.
[0007] This can be achieved by decoding both the main and auxiliary programs into a generalized standard uncompressed representation, switching the streams or mixing / rendering both into a combined representation, and re-encoding the resulting media stream into a coded transmission format. This method can be performed in any device, such as all performed in device #1. However, high computational power may be required, and the rendering stages and intermediate representations may not provide an optimal fit for the actual playback system in device #2.
[0008] FIG. 1 shows an exemplary device #1 and an exemplary device #2. Device #1 may include a primary receiver 101, a secondary receiver / local engine 103, and a packetized stream aggregator 102. The packetized stream aggregator 102 may aggregate a data stream (e.g., a main stream) received from the primary receiver 101 and a data stream (e.g., an auxiliary data stream) received from the secondary receiver / local engine 103. Device #2 may include a stream splitter 201, a primary decoder / renderer 202, and a secondary decoder / renderer 203. The stream splitter 201 may separate streams such as a main stream (which may then be provided to the primary decoder / renderer 202) and an auxiliary stream (which may then be provided to the secondary decoder / renderer 203). Both decoder output signals may be presented to a viewer simultaneously.
[0009] To overcome the above limitations, device #1 can send the incoming main media stream directly to device #2. This mode of device #1 is called the "pass-through" mode. However, downstream of this approach, a standardized interface is only specified to carry one single data stream for video and audio. Therefore, a second media stream cannot inherently be sent over the interface to device #2 for playback.
[0010] The present invention provides methods and apparatus for a primary receiver to receive packetized media data, a secondary receiver to receive additional media streams, and a transmitter interface to transmit the packetized media data.
[0011] The invention proposes to integrate a second auxiliary media stream into a packetized main media stream by the following measures.
[0012] Media streams represented as packetized streams typically use packet type identifiers to differentiate between multiple sub-streams as well as the overall media stream. To convey additional (media) data not related to the main media stream, the first device #1 can encapsulate the additional data in packets formatted according to the main media stream and labeled with a dedicated tag in the packet header. This dedicated tag triggers the receiving device #2 to remove the packets carrying the additional media stream. Optionally, device #2 may then provide the additional media stream to a secondary decoder / renderer 203 instance. At the same time, the main stream is received by the primary decoder / renderer 202.
[0013] In one example, legacy receiving devices that do not recognize this new tag in the packet headers for the additional encapsulated media stream are already expected to ignore these packets.
[0014] The tags may be provided in any encoded audio data stream environment, such as MPEG-H, AC-4, Dolby Digital+, and the like.
[0015] If the additionally inserted data stream exceeds a substantial data rate compared to the original media stream, the receiving device must filter the incoming packetized stream and, optionally, remove the additional data packets to maintain the receiver buffer model of any downstream connected (legacy) media decoders.
[0016] Further, FIG. 3 shows that device #1 may include a primary receiver 101, a secondary receiver / local engine 103, and a packetized stream integrator 102. The packetized stream integrator 102 may integrate the data stream (e.g., main stream) received from the primary receiver 101 and the data stream (e.g., auxiliary data stream) received from the secondary receiver / local engine 103. Device #3 may include only a primary decoder / renderer 301 and may not include a stream splitter. In one example, it may not be possible to provide the auxiliary stream to the secondary decoder / renderer. In this example, the modified primary decoder may internally decode / convert / mix the auxiliary stream with the main stream.
[0017] <MPEG-Hエコシステム> MPEG-H 3d Audio according to ISO / IEC23008-3 is encapsulated in the MHAS format. This format uses a packetized format, where each packet consists of a packet header and a packet payload. The payload is arbitrary binary data, and the header specifies the type and length of the payload. (Additionally available labels can be used to distinguish between multiple instances, but are not used here.)
[0018] By allocating a new MHAS packet type for the secondary media stream (exemplarily called PACTYP_MEDIA), additional audio data, optionally further represented as uncompressed PCM data, specified using the RIFF / WAV format, or compressed audio data, such as MPEG-4 Audio according to ISO / IEC 14496-3, or any other encoded representation (for example according to ATSCA / 52 or ETSI TS 103 190), can be encapsulated in MHAS packets and thus integrated into the main MPEG-H 3D Audio stream. The different formats to be encapsulated can be differentiated by different packet types (for example PACTYP_PCM, PACTYP_MPEG4AUDIO, ...) or by additional indicators forming the sub-header of the MHAS packets, as shown in the following example:
[0019] Since the (media) data may require configuration data but is not represented as a self-contained stream, this data may be encapsulated in the header of the container MHAS packet, or an additional MHAS packet (e.g., PACTYP_MEDIA_CONFIG, or another type of MHAS packet name indicating the configuration such as PACTYP_PCMCONFIG) may be assigned, which may further carry information in the additional data type. The MHAS packet type may carry configuration information for the PCM payload data to provide the configuration information to the decoder. For example, if an MHAS packet type for configuration information (e.g., PACTYP_MEDIA_CONFIG or PACTYP_PCMCONFIG) is present in the bitstream (e.g., after PACTYP_MEDIA_CONFIG), then PCT data configuration information in the form of a data structure (e.g., pcmDataConfig()) may be provided to the decoder.
[0020] Typically, an MHAS packet type (e.g., PACTYP_PCMDATA) may be used to embed a PCM payload corresponding to a PCM signal defined in a configuration structure and to provide PCM data in the form of a PCM data payload structure to a decoder. If an MHAS packet type (e.g., PACTYP_PCMDATA) is present in the bitstream, then a PCM data payload structure (e.g., pcmDataPayload()) may be used during decoding.
[0021] FIG. 2 illustrates an exemplary MPEG-H 3D audio packet stream according to the present invention.
[0022] In one example, the present invention may base the identification information on the following syntax changes: 1) Change the syntax of Table 220-MHASPacketPayload() as follows. [Table 1] 2) Assign an appropriate MHASPacketType to PACTYP_MEDIA in Table 223. Alternatively, the value of MHASPacketType may vary or may depend on a preset value, for example a value published by a standards organization in a final standards document.
[0023] 3) The following media types should be specified using the mhasMediaDataType enumeration: [Table 2] FIG. 2 exemplarily shows a packetized mainstream stream including exemplarily packets according to MPEG-H 3D Audio together with 3D audio data (exemplarily including configuration data, e.g. PACTYP_MPEGH3DACFG as metadata, and encoded audio data, e.g. PACTYP_MPEGH3DAFRAME).
[0024] Furthermore, FIG. 2 exemplarily illustrates a packetized secondary stream (auxiliary stream) including exemplary packets according to the present invention, as indicated by header information referencing the above-mentioned exemplary additional packet type PACTYP_MEDIA, which exemplarily includes additional audio data (auxiliary audio data / secondary audio data) in a compressed format, for example MPEG4 Audio.
[0025] Alternatively or additionally, the additional audio data may be packetized into packets having a header according to the format of the packetized mainstream stream, here exemplarily MPEG-H 3D Audio, including sub-headers indicating the different encapsulated formats as described above.
[0026] According to an exemplary embodiment of the present invention, the main stream and the auxiliary (secondary) streams may be aggregated by a stream aggregater, for example, by the packetized stream aggregater 102 .
[0027] The output stream (integrated stream) contains packets relating to the coded audio data of the main stream and packets relating to the audio data of the auxiliary stream within a single packetized bitstream of the same format (e.g., the exemplary MPEG-H 3D Audio of FIG. 2).
[0028] Note that unmodified (legacy) MPEG-H 3D audio decoders may not understand the newly added packet types (e.g., PACTYP_MEDIA) and such unmodified (legacy) MPEG-H 3D audio decoders may ignore or dump packets that have the newly added packet type (e.g., PACTYP_MEDIA) indicated in the header. Such unmodified (legacy) MPEG-H 3D audio decoders may still decode the audio data associated with the main stream but will not process the additional auxiliary / secondary audio data.
[0029] In order to decode and process an integrated stream having a main stream and an auxiliary stream, the decoder device may be modified to include a modified decoder capable of filtering and decoding / processing packets relating to auxiliary audio data.
[0030] FIG. 4 illustrates diagrammatically another example of a primary decoder / renderer according to the invention.
[0031] 4 shows how this is achieved in an MPEG-H 3D Audio decoder, where the media format is PCM data. Exemplarily, the primary decoder / renderer 301 is implemented to include an MPEG-H 3D Audio Core Decoder M1, and an object renderer M2, e.g., an associated rendering unit as defined according to the MPEG-H 3D Audio (ISO / IEC 23008-3) standard.
[0032] The modified decoder 301 may additionally filter out MHAS packets having a header indicating a new additional packet type (e.g. PACTYP_MEDIA) and input packets having auxiliary audio data to a format conversion unit 301c and then to a sample rate converter (exemplarily, the sample rate converter M3 is present in the decoder architecture downstream of the MPEG-H 3D Audio Core Decoder M1 as defined in accordance with the MPEG-H 3D Audio (ISO / IEC 23008-3) standard).
[0033] Thus, the modified decoder 301 may perform sample rate conversion (e.g., by the sample rate converter M3) and format conversion (e.g., by the format conversion unit 301c1) on the input media data (MHASPacketType==PACTYP_MEDIA) to adapt the media sampling rate and channel layout to the output sample rate and channel configuration of the decoder. Furthermore, the modified decoder may mix the input media data or the sampling rate converted input media data with the audio media data generated by the MPEG-H 3D Audio Core Decoder M1 in a mixer (e.g., exemplarily, a mixer unit M4 is present in the decoder architecture downstream of the MPEG-H 3D Audio Core Decoder M1 as defined according to the MPEG-H 3D Audio (ISO / IEC23008-3) standard).
[0034] The above example of Fig. 4 exemplarily relates to auxiliary media data including uncompressed media data such as PCM data etc. It should be noted that if the uncompressed media data such as PCM data etc. further includes position information of e.g. 3D audio devices or is associated with corresponding position metadata, the auxiliary data can be further processed by a position data processing object renderer such as object renderer M2 present in a decoder architecture downstream of MPEG-H 3D Audio CoreDecoder M1 defined according to the MPEG-H 3D Audio (ISO / IEC23008-3) standard.
[0035] 5 shows a schematic diagram of yet another example of a primary decoder / renderer according to the invention. Illustratively, the primary decoder / renderer 301 is implemented to include an MPEG-H 3D Audio Core Decoder M1 and an Object Renderer M2, e.g. an associated rendering unit as defined according to the MPEG-H 3D Audio (ISO / IEC 23008-3) standard.
[0036] Fig. 5 shows how the above is achieved in an MPEG-H 3D Audio decoder, where the media format is the encoded data (e.g. MPEG-4 Audio). The modified decoder 301 may decode the input media data (MHAS PacketType == PACTYP_MEDIA) using an additional decoder 301c2 (a converter and / or decoder configured to convert and / or decode auxiliary media data) not necessarily specified in ISO / IEC 23008-3. Furthermore, the modified decoder may mix these decoded media data with the audio media data generated by the MPEG-H 3D Audio Core Decoder M1 as defined according to the MPEG-H 3D audio (ISO / IEC 23008-3) standard.
[0037] <Time-alignment of multiple MHAS sub-streams originating from different sources> In an exemplary embodiment according to the invention, an additional time alignment unit may be provided for time alignment of packets of the auxiliary stream, for example to provide time alignment of multiple MHAS sub-streams originating from different sources.
[0038] According to section 14.6 of ISO / IEC 23008-3, "it is assumed that the sub-streams are generated by the same encoder, (and therefore) the various input streams are [...] perfectly aligned and have no phase offset". In this case, frame alignment may be achieved using the MHAS PacketLabel number. According to the proposed method of the present invention, the above mentioned constraint is no longer taken for granted. Due to different frame durations or sampling rates of different codecs, the time offset of consecutive MHAS packets of a secondary stream integrated with the MHAS main stream changes over time. At each specific time slot, the time offset of the secondary stream with respect to the main stream needs to be signaled. For example, in the associated packet of the auxiliary stream indicates the packet type related to the metadata associated with the media data contained in the payload of the packet of the auxiliary stream as shown in FIG. 6.
[0039] FIG. 6 shows an exemplary packet type for achieving the inventive solution based on MHAS. For example, here another MHAS packet type is defined to carry the appropriate time offset signaling. In order to link this signaling to the corresponding stream, the time offset packet needs to have the same MHAS PacketLabel number assigned as the MHAS Packet of type PACTYP_MEDIA to which it refers. From FIG. 6, it is clear that a direct one-to-one relationship of stream data corresponding to one specific time slot of each stream is not guaranteed, but two or more time slots from one stream may correspond to one time slot from another stream in a timely manner.
[0040] Another option for signaling the time offset is to add this time offset to the MHAS packet of type PACTYP_MEDIA itself.
[0041] In view of the above, in some exemplary aspects consistent with the present invention, the conversion and / or decoding unit of the modified primary decoder / renderer 301 of the examples of Figures 4 and / or 5 may include a time alignment unit for time alignment of the auxiliary stream data with the main stream data, for example upstream of a mixer such as mixer unit M4 of the examples of Figures 4 and 5, or upstream of a sampler (e.g., sample rate converter M3) of the examples of Figures 4 and 5, or even as part of the secondary decoder / renderer 203 of the example of Figure 1.
[0042] <Mixing control of main and secondary audio streams> Additional data is needed to control the mixing of the secondary (auxiliary) audio stream with the main audio stream. In particular, this data may include static gain or dynamic gain sequences, exemplarily formed as ISO / IEC 23003-4 Dynamic Range Control data, to handle the main stream when the secondary audio stream is played. These data are normally generated by device #1 and may be embedded in the stream by a separate MHAS packet (e.g. with identifier PACTYP_MPEGH_MEDIA_CFG), further added to the secondary stream header or by any other kind of stream encapsulation.
[0043] In view of the above, in some exemplary aspects consistent with the present invention, the conversion and / or decoding unit of the modified decoder 301 in the examples of Figures 4 and / or 5 may include a gain adjustment unit that applies static and / or dynamic gain (e.g. for volume adjustment), for example upstream of a mixer such as mixer unit M4 in the examples of Figures 4 and 5, or upstream of a sampler (e.g. sample rate converter M3) in the examples of Figures 4 and 5, or even as part of the secondary decoder / renderer 203 in the example of Figure 1.
[0044] <Dolbyエコシステム> Additionally, Dolby AC-4 (ETSI TS103 190) and Dolby Digital and Dolby Digital Plus (ETSI TS102 366) offer the possibility to carry arbitrary binary data in the MEDF payload, which can be used to carry the same or similar data as specified above (MPEG-H Ecosystem).
[0045] For such purposes the syntax element emdf_info() defined in ETSI TS 103 190 or the syntax element emdf_container() defined in ETSI TS 102 366, Annex H, and their underlying elements, may be used. To do this, one may simply define one or more emdf_payload_id definitions, which can be used to identify binary data having the same or similar format as described above under PACTYP_MEDIA and / or PACTYP_MPEGH_MEDIA_CFG.
[0046] System sound mixing of media streams containing uncompressed / unencoded data may be achieved similarly to Dolby AC-4 or Dolby Digital / Dolby Digital Plus as shown in Figures 4 and 5. Here, the difference is that the distinction of which signal path to use is considered based on the emdf_payload_id value, not the MHASPacketType.
[0047] The media streams addressed by the present invention, both main streams and side data streams, may be of the following types: Audio streams, both compressed and uncompressed, Video streams, ·subtitle.
[0048] The invention may also be applied to video presentation devices (monitors) where overlay pictures, videos or texts are to be transmitted over a standardized interface connection to the main (typically compressed video stream).
[0049] References ·US20170223429A1,EP3149955A1 ·ISO / IEC23008-3:(MPEG-H 3d Audio, 2nd Edition) Enumerated exemplary embodiments (EEE) of the present disclosure relate to the following:
[0050] EEE1. A method of audio signal processing, comprising: receiving an aggregated packetized media bitstream, the aggregated media bitstream including packets associated with a main stream representing main audio information and packets associated with a side data stream representing auxiliary audio information; identifying packet type values in header information of packets of the aggregated packetized media bitstream; and separating the aggregated packetized media bitstream into a main stream representing main audio information and a side data stream representing auxiliary audio information based on an identification of packet type values in header information of packets of the aggregated packetized media bitstream.
[0051] EEE2. The method according to EEE1, further comprising the step of mixing an audio output signal based on an output signal derived from the main audio information of the main stream and an output signal derived from the auxiliary audio information of the side data stream.
[0052] EEE3. The method of EEE2, wherein output signals from the main and auxiliary audio information are output simultaneously to a listener.
[0053] EEE4. The method of EEE1, further comprising the step of decoding the main stream by a primary decoder.
[0054] EEE5. The method of EEE1, further comprising the step of decoding said side data stream by a secondary decoder when said side data stream relates to compressed auxiliary audio information.
[0055] EEE6. The method of EEE1, further comprising the step of transforming, by a converter, media data contained in packets of said side data stream when said side data stream relates to compressed auxiliary audio information.
[0056] EEE7. The method of EEE5 or EEE6, wherein the compressed ancillary audio information comprises MPEG-4 audio data.
[0057] EEE8. The method of EEE1, wherein the side data stream relates to uncompressed auxiliary audio information.
[0058] EEE9. The method of EEE8, wherein the uncompressed auxiliary audio information comprises PCM data.
[0059] EEE10. The method according to EEE1, wherein said packet type indicates an internal raw format, in particular a decoder internal raw format.
[0060] EEE11. The method of EEE1, further comprising the step of performing signal processing on the side data stream.
[0061] EEE12. The method according to EEE11, wherein the step of performing signal processing on the side data stream is performed before the step of mixing an audio output signal based on an output signal derived from the main audio information of the main stream and an output signal derived from the auxiliary audio information of the side data stream.
[0062] EEE13. The method of EEE11, wherein performing signal processing on the side data stream includes performing gain adjustment.
[0063] EEE14. The method of EEE13, wherein the gain adjustment is performed based on sexual gain or dynamic gain.
[0064] EEE15. The method of EEE13, further comprising the step of receiving mixing information relating to at least one of the static gain or the dynamic gain.
[0065] EEE16. The method according to EEE13, wherein gain adjustment is performed for volume adjustment.
[0066] EEE17. The method of EEE11, wherein the step of performing signal processing on the side data stream includes the step of performing time alignment.
[0067] EEE18. The method according to EEE17, wherein the step of performing time alignment is performed to time-align the auxiliary audio information of the side data bitstream with the main audio information of the main bitstream.
[0068] EEE19. The method of any one of EEE17 and EEE18, further comprising the step of receiving time alignment information regarding time alignment of the auxiliary audio information of the side data bitstream with the main audio information of the main bitstream.
[0069] EEE20. The method according to EEE1, wherein said packet type is defined for carrying time offset signaling, particularly for time alignment.
[0070] EEE21. The method of EEE1, further comprising the step of receiving an indication of a time offset of the side data stream relative to the main stream.
[0071] EEE22. The method of EEE1, wherein the step of performing signal processing on the side data stream includes object rendering when the auxiliary audio information is associated with position information.
[0072] EEE23. The method of EEE22, wherein object rendering is performed by an object renderer that performs object rendering of the main and side data streams.
[0073] EEE24. The method of EEE11, wherein the step of performing signal processing on the side data stream includes format conversion.
[0074] EEE25. The method of EEE1, further comprising the step of decoding said main stream by a standardized MPEG-H 3D audio decoder.
[0075] EEE26. The method according to EEE25, further comprising the step of mixing media data contained in packets of said side data stream with media data generated by said standardized MPEG-H 3D audio decoder.
[0076] EEE27. The method according to EEE26, wherein the media data comprised in packets of the side data stream is uncompressed data, in particular PCM data, or compressed data, in particular MPEG4 audio data.
[0077] EEE28. The method according to EEE26, wherein the media data comprised in packets of the side data stream are decoded by a decoder not standardized in MPEG-H 3D Audio.
[0078] EEE29. The method of EEE1, wherein the aggregated packetized media bitstream comprises HMAS packets.
[0079] EEE30. A method for processing an audio signal, comprising: receiving a main stream indicative of main audio information; receiving a side data stream indicative of auxiliary audio information or generating a side data stream based on the auxiliary audio information; aggregating the main stream and the side data stream to generate an aggregated packetized media bitstream comprising packets associated with the main stream representing main audio information and packets associated with the side data stream representing auxiliary audio information.
[0080] EEE31. The method of EEE30, further comprising packetizing media data indicative of said auxiliary audio information into packets in the format of said aggregated packetized media bitstream.
[0081] EEE32. The method of EEE30, wherein said side data stream relates to compressed auxiliary audio information.
[0082] EEE33. The method of EEE32, wherein the compressed ancillary audio information comprises MPEG-4 audio data.
[0083] EEE34. The method of EEE30, wherein said side data stream relates to uncompressed auxiliary audio information.
[0084] EEE35. The method of EEE34, wherein the uncompressed auxiliary audio information comprises PCM data.
[0085] EEE36. The method of EEE30, wherein header information of packets of the aggregated packetized media bitstream indicates a packet type.
[0086] EEE37. The method according to EEE36, wherein a packet type value of a packet associated with the side data stream indicating auxiliary audio information indicates media data associated with the auxiliary audio information.
[0087] EEE38. The method of EEE30, wherein the aggregated packetized media bitstream comprises HMAS packets.
[0088] EEE39. Audio signal processing equipment, comprising: a receiver configured to receive an aggregated packetized media bitstream including packets associated with a main stream representing main audio information and packets associated with a side data stream representing auxiliary audio information; and a splitter configured to separate the aggregated packetized media bitstream into a main stream representing main audio information and a side data stream representing auxiliary audio information based on an identification of a packet type value in header information of packets of the aggregated packetized media bitstream.
[0089] EEE40. The apparatus of EEE39, further comprising a mixer configured to mix an audio output signal based on an output signal derived from the main audio information of the main stream and an output signal derived from the auxiliary audio information of the side data stream.
[0090] EEE41. The apparatus of EEE39, further comprising a primary decoder configured to decode the main stream.
[0091] EEE42. The apparatus of EEE39, further comprising a secondary decoder configured to decode the side data stream when the side data stream is associated with compressed auxiliary audio information.
[0092] EEE43. The apparatus of EEE39, further comprising a converter configured to convert media data contained in packets of the side data stream when the side data stream relates to compressed auxiliary audio information.
[0093] EEE44. The apparatus described in EEE42 or EEE43, wherein the compressed ancillary audio information includes MPEG-4 audio data.
[0094] EEE45. The apparatus described in EEE39, wherein the side data stream relates to uncompressed auxiliary audio information.
[0095] EEE46. The apparatus of EEE45, wherein the uncompressed auxiliary audio information comprises PCM data.
[0096] EEE47. The apparatus according to EEE39, wherein the packet type indicates an internal raw format, in particular a decoder internal raw format.
[0097] EEE48. The apparatus of EEE39, further comprising a gain adjustment unit configured to perform gain adjustment applied to the side data stream.
[0098] EEE49. The apparatus of EEE39, further comprising a time alignment unit configured to perform time alignment applied to the side data stream.
[0099] EEE50. The apparatus of EEE39, further comprising an object renderer configured to perform object rendering applied to the side data stream.
[0100] EEE51. The apparatus of EEE50, wherein the object renderer is included in a primary decoder.
[0101] EEE52. The apparatus according to EEE39, further comprising a format conversion unit configured to perform a format conversion applied to the side data stream.
[0102] EEE53. The apparatus according to EEE39, further comprising a standardized MPEG-H 3D audio decoder for decoding said main stream.
[0103] EEE54. The apparatus according to EEE39, wherein the aggregated packetized media bitstream comprises HMAS packets.
[0104] EEE55. Apparatus for processing audio signals, comprising: a receiver configured to receive a main stream indicative of main audio information; a local engine configured to generate, based on the auxiliary audio information, a side data stream indicative of the auxiliary audio information, and / or a receiver configured to receive the side data stream; and an aggregater configured to aggregate the main stream and the side data stream to generate an aggregated packetized media bitstream comprising packets associated with the main stream representing main audio information and packets associated with the side data stream representing auxiliary audio information.
[0105] EEE56. The apparatus of EEE55, further comprising a packetizer configured to packetize media data indicative of said auxiliary audio information into packets in the format of said aggregate packetized media bitstream.
[0106] EEE57. A system including an apparatus described in EEE55 and an apparatus described in EEE39. [Explanation of symbols]
[0107] 101 Primary Receiver 102 Packetized Stream Integration Unit 103 Secondary Receiver / Local Engine 201 Stream Splitter 202 Primary Decoder / Renderer 203 Secondary Decoder / Renderer
Claims
1. 1. A method for processing a merged packetized media bitstream, the method comprising: extracting a main stream from the merged packetized media bitstream, the main stream including MPEG-H 3D audio data encapsulated in MHAS format; extracting from the merged packetized media bitstream an auxiliary stream including additional audio data encapsulated as MHAS formatted packets, the packets having a new MHAS packet type, the packet payload format of the packets being different from an MPEG-H 3D format of the MPEG-H 3D audio data, the new MHAS packet type indicating an internal format; identifying time offset signaling indicating that there is a time offset of the auxiliary stream relative to the main stream, the time offset signaling indicating that the auxiliary stream signal needs to be offset so that it arrives at a mixer aligned with the main stream; time-aligning the additional audio data with the MPEG-H 3D audio data based on the time offset signaling; Decoding the main stream according to the MPEG-H 3D format; The method includes:
2. The method of claim 1 , wherein the MHAS format is compatible with the ISO / IEC 23008-3 standard.
3. 13. A non-transitory computer readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method of claim 1.
4. 1. An audio decoding device for processing a merged packetized media bitstream, comprising: a splitter configured to extract a main stream and an auxiliary stream from the merged packetized media bitstream, the main stream including MPEG-H 3D audio data encapsulated in an MHAS format, the auxiliary stream including additional audio data encapsulated as packets in the MHAS format, the packets having a new MHAS packet type, a packet payload format of the packets differing from an MPEG-H 3D format of the MPEG-H 3D audio data, the new MHAS packet type indicating an internal format; an identification unit configured to identify time offset signaling indicating that there is a time offset of the auxiliary stream relative to the main stream, the time offset signaling indicating that the auxiliary stream signal needs to be offset so that it arrives at a mixer aligned with the main stream; a time alignment unit for time-aligning the additional audio data with the MPEG-H 3D audio data based on the time offset signaling; an audio decoder for decoding the main stream according to the MPEG-H 3D format; Including equipment.
5. 1. A method comprising: receiving a merged packetized media bitstream at an MPEG-H 3D Audio decoder, the merged packetized media bitstream including MHAS format packets associated with a main stream and new MHAS format packets associated with an auxiliary stream; generating audio data based on packets associated with the main stream by an MPEG-H 3D Audio Core decoder; identifying additional audio data based on the new MHAS format packets associated with the auxiliary stream, the identification being based on a packet type of the new MHAS format packets; The method includes:
6. 6. The method of claim 5, further comprising the step of performing a format conversion of the additional audio data by a modification decoder.
7. The method of claim 6 further comprising the step of converting a sample rate of the additional audio data by the modified decoder.
8. The method of claim 6 , further comprising obtaining a sample rate and converting a sample rate of the additional audio data based on the obtained sample rate.
9. 8. The method of claim 7, further comprising the step of mixing the audio data generated by the modified decoder with audio data generated by a standard MPEG-H 3D Audio Core decoder.
10. receiving, at an MPEG-H 3DA compatible decoder, a merged packetized media bitstream comprising MHAS formatted packets associated with a main stream and said MHAS formatted packets associated with an auxiliary stream; generating, by an MPEG-H 3D Audio Core compatible decoder, audio data based on packets associated with the main stream; identifying, by a modification decoder, a packet type encapsulated in a header of the packet, the packet type being a new MHAS packet type; converting and / or decoding additional audio data, the additional audio data being determined based on the packet type, the decoding being not standardized in MPEG-H 3D Audio; performing a format conversion on the additional audio data by the modified decoder; converting the sample rate of the additional audio data by the modified decoder; The method includes:
11. 10. The method of claim 9, further comprising the step of mixing audio data generated by the modified decoder with audio data generated by the standard MPEG-H 3D Audio Core decoder.
12. an MPEG-H 3D Audio Core decoder configured to generate audio data based on packets associated with a main stream of the merged packetized media bitstream having the main MPEG-H 3D audio data encapsulated as MHAS format packets; A modification decoder; the modified decoder comprising: an additional decoder including means for converting and / or decoding additional audio data encapsulated in packets associated with an auxiliary stream of said merged packetized media bitstream based on an identification of a packet type encapsulated in a header of the packet, said packet type being a new MHAS type that cannot be decoded by a standardized MPEG-H 3D Audio Core decoder; a format conversion unit configured to perform a format conversion on the additional audio data encapsulated in the packets; a sample rate converter configured to convert a sample rate of the additional audio data encapsulated in the packets; Including, equipment.
13. 1. A method comprising: receiving, by an MPEG-H 3D Audio compatible decoder, a merged packetized media bitstream, the media bitstream including packets associated with a main stream and packets associated with an auxiliary stream; identifying, by the MPEG-H 3D Audio compatible decoder, packet types encapsulated within headers of packets in the merged packetized media bitstream; splitting the merged packetized media bitstream into the main stream and the auxiliary stream based on the packet type identification; The method includes:
14. An apparatus comprising: a receiver for receiving a merged packetized media bitstream, the merged media bitstream including packets associated with the main stream and packets associated with the auxiliary stream, said receiver comprising an MPEG-H 3D Audio compatible decoder; a processor for identifying, by the MPEG-H 3D Audio compatible decoder, packet types encapsulated in headers of packets in the merged packetized media bitstream; a splitter for splitting the merged packetized media bitstream into the main stream and the auxiliary stream based on the packet type identification; Including equipment.
Citation Information
Patent Citations
Data processing apparatus and transport for user control data for an audio decoder and renderer
JP2017523638A
Method and apparatus for decoding audio bitstream including system data
US20180033443A1
Transmitting device, transmission method, receiving device, and receiving method
WO2016060101A1
