Methods and devices for generation and processing of modified bitstreams

Abstract

To provide a method for generating a bitstream that can simultaneously process earcon / system sound in parallel with media contents.SOLUTION: A method of generating a modified bitstream in a source device having a receiver and an embedder comprises the steps of: receiving, by the receiver, a bitstream containing encoded media data; obtaining, as an output from the embedder, a modified bitstream including the encoded media data and a payload of additional media data, by generating the payload of additional media data by the embedder and embedding the payload into the bitstream; and outputting the modified bitstream to a sink device.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 62 / 887,111, filed August 15, 2019, and European Patent Application No. 19191919.0, filed August 15, 2019, the disclosures of each of which are incorporated herein by reference in their entirety.

[0002] The present disclosure relates generally to a method and a source device for generating a modified bitstream, and more particularly to generating the modified bitstream by embedding a payload of additional media data into a bitstream containing encoded media data. The present disclosure also relates to a method and a sink device for processing the modified bitstream.

[0003] Although certain embodiments are described herein with particular reference to that disclosure, it will be understood that the disclosure is not limited to such fields of use, but is applicable in a broader context. [Background technology]

[0004] The description of any background art throughout this disclosure should in no way be taken as an admission that such art is widely known or forms part of the common general knowledge in the art.

[0005] In recent years, there has been discussion about establishing an AAC family-based codec as part of the Bluetooth (BT) ecosystem. The SIG may define use cases such as, for example, requiring a pass-through of a bitstream received on a mobile device (BT source) to a connected speaker system (BT sink). Since the data format used from services across BT endpoints may be coded audio, there is no easy way to convey earcon and / or system sounds from the mobile device to the BT speakers. Such earcons may include ringtones, mail notification sounds, key-press sounds, etc.

[0006] In currently deployed systems, audio received from a service is typically decoded to the uncompressed domain (PCM) within the device. In the uncompressed domain, the audio can be modified and / or mixed. When pass-through mode of operation is enabled, it may not be possible to easily enable and / or mix system sounds on the BT source device into the experience being transferred from the BT source to the BT sink.

[0007] One possibility to overcome this limitation by still enabling pass-through would be to open an additional link between both devices. However, this setup has the drawback that an additional link and encoder-decoder pair of Bluetooth codecs are required to function. This involves unnecessary additional system complexity and imposes additional computational complexity on both the BT source and sink devices, which may also strain the battery. Summary of the Invention [Problem to be solved by the invention]

[0008] Therefore, there is a conventional need to simultaneously generate and process earcon and / or system sounds in parallel to the media content via a bitstream, particularly via an MPEG-D USAC based bitstream. [Means for solving the problem]

[0009] According to a first aspect of the present disclosure, there is provided a method of generating a modified bitstream by a source device. The source device may include a receiver and an embedder. The method may include a) receiving, by the receiver, a bitstream including coded media data. The method may further include b) generating, by the embedder, a payload of additional media data and embedding the payload in the bitstream to obtain, as output from the embedder, a modified bitstream including the coded media data and a payload of the additional media data. Furthermore, the method may include c) outputting the modified bitstream to a sink device.

[0010] Configured as proposed, the method allows for inserting system sounds or earcons into a bitstream containing encoded media data and passed through by a source device (e.g., a Bluetooth source device such as a mobile phone) to a sink device (e.g., a Bluetooth sink device such as a Bluetooth loudspeaker). This allows for the system sounds / earcons to be passed to the sink device without the need to establish an additional connection between the source device and the sink device, and without the need to decode and re-encode the encoded media data. Furthermore, the method allows for the system sounds / earcons and the encoded media data to be processed simultaneously.

[0011] In some embodiments, the additional media data may be reproduced by a sink device.

[0012] In some embodiments, the additional media data may be reproduced by the source device.

[0013] In some embodiments, the encoded media data in the received bitstream may be passed through from the source device to the sink device via the modified bitstream.

[0014] In some embodiments, step b) may further include generating information about the additional media data and embedding the information in the bitstream.

[0015] In some embodiments, the generated information may include one or more of information about the composition of the additional media data, information about the presence of the additional media data in the modified bitstream, information about a default loudness value, information about a value for delay alignment, information about delay adaptation, information about panning, and information about a type of the additional media data.

[0016] In some embodiments, step b) may further include generating information regarding a relative gain between the encoded media data and the additional media data, and embedding the information regarding the relative gain in the bitstream.

[0017] In some embodiments, the modified bitstream may be an MPEG-D USAC based bitstream. Thus, the system sounds / earcons are When a device is in pass-through mode for media data encoded in the MPEG-4 audio format, the media data can be sent to a sink device as part of the encoded bitstream without the need to be decoded and re-encoded.

[0018] In some embodiments, the generated payload may be embedded in the bitstream by carrying the payload in the (modified) bitstream via a USAC-type bitstream extension mechanism.

[0019] In some embodiments, the generated information and / or information regarding the generated relative gains may be embedded in the bitstream by conveying the generated information and / or information regarding the generated relative gains in the (modified) bitstream via a USAC-type bitstream extension mechanism.

[0020] In some embodiments, the USAC-type bitstream extension mechanism may be a new USAC-type bitstream extension element that includes a unique identifier.

[0021] In some embodiments, in step c), the modified bitstream may be output to a sink device via a Bluetooth connection.

[0022] In some embodiments, the additional media data may be compressed media data or uncompressed media data.

[0023] In some embodiments, the uncompressed media data may be PCM data generated at the source device.

[0024] In some embodiments, the generation of the PCM data may be based on user input.

[0025] In some embodiments, the PCM data may include one or more of earcon data and system sound data.

[0026] In some embodiments, the compressed data may be data in SBC format or an aptX-based format.

[0027] According to a second aspect of the present disclosure, there is provided a method for processing a modified bitstream by a sink device. The sink device may include a receiver and a bitstream parser. The method may include a) receiving, by the receiver, a modified bitstream including a payload of encoded media data and additional media data. The method may include b) parsing, by the bitstream parser, the modified bitstream to obtain a payload of the encoded media data and additional media data. The method may further include the step of c) core-decoding, by a core decoder, the encoded media data to obtain core-decoded media data. The method may further include the step of d) mixing, by a mixer, the core-decoded media data and the additional media data to obtain an output signal. Furthermore, the method may include the step of e) outputting the output signal.

[0028] In some embodiments, the modified bitstream further includes information about the additional media data, and the method may further include, after step a) and before step b), processing the modified bitstream based on the information.

[0029] In some embodiments, the information may include one or more of information about the composition of the additional media data, information about the presence of the additional media data in the modified bitstream, information about a default loudness value, information about a value for delay alignment, information about delay adaptation, information about panning, and information about the type of the additional media data.

[0030] In some embodiments, the modified bitstream may further include information regarding the relative gain between the encoded media data and the additional media data, and the mixing of the core decoded media data and the additional media data in step d) may be based on the information regarding the relative gain.

[0031] In some embodiments, the method may further include processing, by a processing unit, the additional media data prior to mixing the core decoded media data and the additional media data.

[0032] In some embodiments, processing the additional media data may include one or more of resampling, delay adaptation, and loudness processing.

[0033] In some embodiments, the additional media data may be compressed media data and the method may further comprise the step of decoding, by a decoder, the compressed media data to obtain decoded additional media data, and in step d) the decoded additional media data may be mixed with the core decoded media data.

[0034] In some embodiments, the compressed media data may be data in SBC format or an aptX-based format.

[0035] In some embodiments, the additional media data may be uncompressed media data, and in step d) the uncompressed additional media data may be mixed with the core decoded media data.

[0036] In some embodiments, the modified bitstream may be an MPEG-D USAC-based bitstream.

[0037] In some embodiments, the modified bitstream may include a USAC-type bitstream extension mechanism, and the analysis in step b) may be further based on identifying the USAC-type bitstream extension mechanism in the modified bitstream.

[0038] In some embodiments, the USAC-type bitstream extension mechanism may be a new USAC-type bitstream extension element that includes a unique identifier.

[0039] According to a third aspect of the present disclosure, there is provided a source device for generating a modified bitstream. The device may include: a) a receiver configured to receive a bitstream including encoded media data. The device may further include: b) an embedder configured to generate a payload of additional media data and embed the generated payload in the bitstream to obtain a modified bitstream including the encoded media data and the additional media data payload. Furthermore, the device may include: c) a control unit configured to output the modified bitstream to a sink device.

[0040] In some embodiments, the embedder may be further configured to generate information about the additional media data and embed the information in the bitstream.

[0041] In some embodiments, the embedder may be further configured to generate information regarding a relative gain between the encoded media data and the additional media data, and to embed the information regarding the relative gain in the bitstream.

[0042] In some embodiments, the modified bitstream may be an MPEG-D USAC-based bitstream.

[0043] In some embodiments, the additional media data may be compressed media data or uncompressed media data.

[0044] In some embodiments, the uncompressed media data may be PCM data generated at the source device.

[0045] In some embodiments, the generation of the PCM data may be based on user input.

[0046] In some embodiments, the PCM data may include one or more of earcon data and system sound data.

[0047] In some embodiments, the compressed media data may be data in SBC format or an aptX-based format.

[0048] According to a fourth aspect of the present disclosure, there is provided a sink device for processing a modified bitstream. The device may include a) a receiver configured to receive a modified bitstream including a payload of encoded media data and additional media data. The device may further include b) a bitstream parser configured to parse the modified bitstream to obtain a payload of the encoded media data and the additional media data. The device may further include c) a core decoder configured to core-decode the encoded media data to obtain core-decoded media data. The device may further include d) a mixer configured to mix the core-decoded media data and the additional media data to obtain an output signal. Furthermore, the device may comprise a control unit configured e) to output the output signal.

[0049] In some embodiments, the device may further include a processing unit configured to process the additional media data prior to mixing the core decoded media data and the additional media data.

[0050] In some embodiments, the additional media data is uncompressed media data, and the mixer may be configured to mix the core decoded media data and the uncompressed additional media data.

[0051] In some embodiments, the additional media data is compressed media data and the device may further include a decoder for decoded the compressed media data to obtain decoded additional media data, and the mixer may be configured to mix the core decoded media data and the decoded additional media data.

[0052] In some embodiments, the modified bitstream may further include information about the additional media data, and the sink device may modify the bitstream based on that information. The bitstream may be configured to process the encoded bitstream.

[0053] In some embodiments, the modified bitstream may further include information regarding the relative gain between the encoded media data and the additional media data, and the mixer may be further configured to mix the core decoded media data and the additional media data based on the information regarding the relative gain.

[0054] In some embodiments, the core decoder may be an MPEG-D USAC-based decoder.

[0055] According to a fifth aspect of the present disclosure, there is provided a system of a source device for generating a modified bitstream and a sink device for processing the modified bitstream, the source device including one or more processors configured to perform on the source device the method for generating the modified bitstream, and the sink device including one or more processors configured to perform on the sink device the method for processing the modified bitstream.

[0056] According to a sixth aspect of the present disclosure, there is provided a computer program product comprising a computer-readable storage medium having instructions configured, when executed by a device having processing capability, to cause the device to perform a method for generating a modified bitstream on a source device.

[0057] According to a seventh aspect of the present disclosure, there is provided a computer program product comprising a computer-readable storage medium having instructions configured, when executed by a device having processing capability, to cause the device to perform a method for processing a modified bitstream on a sink device. [Brief description of the drawings]

[0058] Example embodiments of the present disclosure will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0059] [Figure 1] FIG. 1 illustrates a flow diagram of an example method for generating a modified bitstream on a source device. [Diagram 2] FIG. 2 illustrates a flow diagram of a further example method for generating a modified bitstream on a source device. [Diagram 3] FIG. 3 illustrates a flow diagram of an example method for processing the modified bitstream on a sink device. [Figure 4] FIG. 4 illustrates a flow diagram of a further example method for processing the modified bitstream on a sink device. [Diagram 5] FIG. 5 illustrates a flow diagram of another further example method for processing the modified bitstream on a sink device. [Figure 6] FIG. 6 illustrates a flow diagram of another further example method for processing the modified bitstream on a sink device. [Figure 7] FIG. 7 illustrates an example of a source device generating a modified bitstream. [Figure 8]FIG. 8 illustrates a further example of a source device generating a modified bitstream. [Figure 9] FIG. 9 illustrates an example of a sink device processing the modified bitstream. [Figure 10] FIG. 10 illustrates a further example of a sink device processing a modified bitstream. [Figure 11] FIG. 11 illustrates another further example of a sink device processing a modified bitstream. [Figure 12] FIG. 12 illustrates another further example of a sink device processing a modified bitstream. [Figure 13] FIG. 13 illustrates an example of a device having one or more processors. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0060] Generation and processing of MPEG-D USAC-based modified bitstreams In the following, a method and device for generating and processing an MPEG-D USAC-based modified bitstream is described. In this specification, an MPEG-D USAC-based bitstream is defined as a bitstream based on ISO / IEC23003-3 MPEG audio technologies - Part 3: Unified MPEG-D speech and audio coding (hereinafter "MPEG-D USAC") and all future editions, versions and amendments thereof. Hereinafter, reference to an MPEG-D USAC based bitstream means a bitstream conforming to one or more requirements set forth in the MPEG-D USAC standard. The above methods and devices may carry and process additional media data, which may be uncompressed or compressed media data, along with the encoded media data that delivers the media content. This may enable earcon data and / or system sounds to be transmitted and processed without additional system and computational complexity burdens at both the source and sink devices.

[0061] Furthermore, it may be possible to control all settings (e.g., system sound levels, DRC, etc.) using the same mechanisms and bitstream variables for both the source and sink devices. This means that the relative gains (or mixing gains) used by the embedder of the source device may be just the same as the control parameters used by the mixer of the sink device.

[0062] Generating the modified bitstream on the source device With reference to the example of FIG. 1, an example of a method for generating a modified bitstream on a source device is illustrated. In step S100, a bitstream including encoded media data (to be core decoded by a sink device as described below) may be received by a receiver of the source device. The bitstream including encoded media data may be received, for example, from one or more media servers. The encoded media data may include encoded speech and audio data. The encoded speech and audio data may conform to the above MPEG-D USAC standard. The bitstream may deliver, for example, audio content to be played by a sink device connected to the source device. The source device is not limited. For example, the source device may be a mobile device including one or more of a mobile phone, a smart watch, a laptop including a convertible, and a tablet PC.

[0063] Referring again to the example of Fig. 1, in step S101, a payload of additional media data (i.e. media data added separately from the encoded media data, e.g. additional audio data) may be generated and embedded in the bitstream by an embedder.

[0064] The additional media data generated in step S101 will be mixed with the media data by the sink device (at the sink device side), which advantageously allows the additional media data to be mixed with the media data at the sink device side.

[0065] The encoded media data and the additional media data are different (e.g., different types) of media data. For example, as described below, the encoded media data and the additional media data may have different formats. The encoded media data and the additional media data include different types of media content. The encoded media data may include primary audio content, e.g., music, audio, etc. The additional media data may include secondary audio content, e.g., earcons, system sounds, keystroke sounds, email notification sounds, etc. The primary audio content and the secondary audio content may be played by a sink device.

[0066] Alternatively, or in addition, the primary and secondary audio content may be played by a source device.

[0067] In one embodiment, the encoded media data may be played by a source device, a sink device, or both a source device and a sink device.

[0068] In one embodiment, the additional media data may be played by the source device, the sink device, or both the source device and the sink device.

[0069] In one embodiment, the additional media data may be compressed media data (which will be decoded by the sink device to obtain the decoded additional media data, as described below) or uncompressed media data. The uncompressed media data may be, but is not limited to, PCM data generated at the source device in one embodiment. The generation of the PCM data may also be, but is not limited to, based on user input in one embodiment. The user input may be, for example, a keystroke. Alternatively or additionally, the generation of the PCM data may be facilitated at the source device independent of user input, for example, based on an email notification. In one embodiment, the PCM data may include one or more of earcon data and system sound data. Also, the format of the compressed media data may belong to, but is not limited to, the Bluetooth ecosystem. In one embodiment, the compressed media data may be data in an SBC format or an aptX-based format.

[0070] Referring again to the example of Fig. 1, in step S102, a modified bitstream including a payload of encoded media data and additional media data may be obtained as output from the embedder. Then, in step S103, the modified bitstream may be output to a sink device.

[0071] The encoded media data in the received bitstream is passed through from the source device to the sink device via the modified bitstream, and the encoded media data received from the source device is embedded in the modified bitstream and output to the sink device.

[0072] Advantageously, generating (at the source device) and outputting the modified bitstream to the sink device streamlines mixing the media data with additional media data at the sink device. A single bitstream is used by the source device to embed the media data and the additional media data and transfer them to the sink device. Thereby, a single channel / connection can be used between the source and the sink device. Furthermore, there are no additional decoders and / or encoders (e.g., for decoding and re-encoding the encoded media data) at the source device. No additional mixers (e.g., for mixing the decoded encoded media data with the additional media data) are used at the source device.

[0073] In one embodiment, the modified bitstream may be output to a sink device via a Bluetooth connection. In other words, the source device may be connected to the sink device via a Bluetooth connection. For example, a mobile phone may have an earphone connected via Bluetooth.

[0074] In general, the proposed techniques and implementations and examples thereof described throughout this disclosure include generating and outputting a modified bitstream to a sink device, for example, to facilitate sending a payload of additional media data (e.g., system sounds / earcons) to a sink device when the source device is in a pass-through mode of operation. In other words, for example, the payload of additional media data is generated and embedded in a received bitstream including the encoded media data to generate a modified bitstream, without the encoded media data being decoded and re-encoded. Similarly, the additional media data can be passed to a sink device without the need to establish an additional connection between the source device and the sink device. As a further advantage, the proposed techniques enable simultaneous processing of additional media data and encoded media data.

[0075] Now, referring to the example of Fig. 2, a further example of a method for generating a modified bitstream on a source device is illustrated. The example of Fig. 2 differs from the example of Fig. 1 detailed above in that in step S101, further information about the additional media data may be generated (e.g., as part of metadata) and embedded in the bitstream (indicated by dashed lines). In one embodiment, the generated information may include one or more of information about the configuration of the additional media data, information about the presence of the additional media data in the modified bitstream, information about a default loudness value, information about a value for delay alignment, information about delay adaptation, information about panning, and information about a type of the additional media data. The generated information may be utilized by the sink device to process the received modified bitstream, as described in more detail below. The information about the configuration of the additional media data may include, for example, information about a sampling rate, which may enable a decision at the sink device as to whether the additional media data may need to be resampled before mixing. Alternatively or additionally, the information about the configuration of the additional media data may further include information about a channel mode that allows the sink device to determine whether the additional media data may need to be downmixed / upmixed before mixing. The information about the default loudness value may include, for example, loudness measured according to one of the methods described in ITU-R BS.1770-4. The measured loudness may enable the sink device to determine at what level the additional media data may be mixed with the core decoded media data. Furthermore, the information about the type of the additional media data may indicate at the sink device whether the additional media data may need to be decoded, for example, whether the additional media data may be uncompressed or compressed media data.

[0076] Alternatively or additionally, step S101 may further include generating information regarding a relative gain between the encoded media data and the additional media data (e.g., the additional media data as part of metadata) and embedding the information regarding the relative gain in the bitstream (indicated by a dashed line). The relative gain between the encoded media data and the additional media data may be used by the sink device to generate an output signal based on the received modified bitstream, as described in more detail below. Here, the source device and the sink device may use the same relative gain value. For example, if the source device is a mobile phone, the optimal system sound mixing setting may be determined based on the estimated loudness ratio of the system sound and the encoded media data content. The relative gain values ​​may be determined automatically. For example, the relative gain values ​​may include mixing gain values ​​and / or attenuation gain values ​​that may be used by a mixer of a sink device. Including this information in the bitstream may ensure that the additional media data can be mixed with the encoded media data in an appropriate manner after decoding and avoid cases where the additional media data is rendered at an inappropriate sound level compared to the encoded media data. In general, including information about the additional media data in the bitstream gives the source device control over the processing (e.g., including rendering) of the additional media data at the sink device. This ensures proper rendering of the additional media data at the sink device.

[0077] The format of the modified bitstream is not limited, but in one embodiment, the modified bitstream may be an MPEG-D USAC-based bitstream (i.e., a bitstream that complies with the MPEG-D USAC standard). With reference to the example of FIG. 1, in one embodiment, the generated payload may be carried in the modified bitstream via a USAC-type bitstream extension mechanism. With reference to FIG. 2, in one embodiment, the generated information and / or information regarding the generated relative gain may be carried in the modified bitstream via a USAC-type bitstream extension mechanism.

[0078] As an example, the following syntax may be used to embed a payload of the generated additional media data, information about the generated additional media data and / or information about the generated relative gain into an MPEG-D USAC based modified bitstream:

[0079] system_sound_info() syntax A top-level syntax element that conveys a payload of generated additional media data, generated information (information about the additional media data), and / or information about the generated relative gain.

[0080] JPEG2025069380000002.jpg70144

[0081] Semantics of system_sound_info() JPEG2025069380000003.jpg23140

[0082] system_sound_config() syntax JPEG2025069380000004.jpg191146

[0083] Semantics of system_sound_config() JPEG2025069380000005.jpg101142

[0084] system_sound_payload() syntax JPEG2025069380000006.jpg119145

[0085] Semantics of system_sound_payload() JPEG2025069380000007.jpg11141

[0086] pcm_data_config() syntax JPEG2025069380000008.jpg105144

[0087] Semantics of pcm_data_config() JPEG2025069380000009.jpg172141

[0088] pcm_data_payload() syntax JPEG2025069380000010.jpg35144NOTE1: "pcm_frame_size" may either be obtained from "pcm_frame_size_index" or set directly in "pcm_data_config" NOTE2: The number of bits per sample (bps) is obtained from "pcm_data_format_index" can

[0089] Semantics of pcm_data_payload() JPEG2025069380000011.jpg24141

[0090] In one embodiment, the USAC type bitstream extension mechanism may be a new USAC type bitstream extension element that includes a unique identifier. In a USAC type bitstream, system sound info is defined as a USAC type extension element defined in ISO / IEC 23003-3. The new usacExtElementType may be named ID_EXT_ELE_SYS_SOUND_INFO.

[0091] To convey only the configuration part, the syntax in Table 17, "Syntax for UsacExtElementConfig()," can be modified as follows: JPEG2025069380000012.jpg41141

[0092] In this case, system_sound_info() may only carry configuration data, i.e. , the field system_sound_config_present may be set to 1, and the field system_sound_payload_present may be set to 0.

[0093] The generated PCM data payload can be transmitted frame by frame in accordance with Table 25 "Syntax of UsacExtElement()" of ISO / IEC 23003-3.

[0094] In one embodiment, the payload of the generated additional media data may be embedded in an array extension element of a USAC type bitstream. The structure system_sound_info defined above may be embedded in the array "usacExtElementSegmentData" of Table 25, and may contain either generated information (information about the additional media data, e.g., configuration) or the generated payload, or may contain both.

[0095] The array extension elements of a USAC type bitstream (e.g., the array element "usacExtElementSegmentData") typically contain coded media data and / or metadata. In this embodiment, the array extension elements carry (contain) a generated payload of additional media data that is unencoded / raw data (e.g., uncompressed additional media data such as PCM data). Thus, in this embodiment of the invention, existing array elements of a USAC type bitstream that normally carry encoded audio data and / or metadata are instead used to carry unencoded / raw media data.

[0096] Using the above techniques, additional media data and information about the additional media data can be easily embedded into existing MPEG-D USAC-based bitstreams that are passed through by source devices.

[0097] Process the modified bitstream on the sink device Now, referring to the example of Fig. 3, an example of a method for processing a modified bitstream on a sink device is illustrated. In step S200, a modified bitstream including a payload of encoded media data (e.g., received by a source device) and additional media data may be received by a receiver of a sink device.

[0098] In step S201, the received modified bitstream is parsed by a bitstream parser to extract the encoded media data and the additional media data. The modified bitstream may be converted into a payload of media data. The format of the modified bitstream is not limited, but the modified bitstream may be an MPEG-D USAC-based bitstream (i.e., a bitstream that complies with the MPEG-D USAC standard). Referring to the example of FIG. 4, in one embodiment, the modified bitstream may include a USAC-type bitstream extension mechanism, and the analysis in step S201 may be further based on identifying the USAC-type bitstream extension mechanism in the modified bitstream. In one embodiment, the USAC-type bitstream extension mechanism may be a new USAC-type bitstream extension element that includes a unique identifier.

[0099] Referring to the example of FIG. 3 or 4, in step S202, the encoded media data may be core decoded by a core decoder.

[0100] In step S203, the core decoded media data may be mixed with the additional media data by a mixer to obtain an output signal.

[0101] Then, in step 204, the obtained output signal may be output by the control unit.

[0102] Now, referring to the example of Fig. 5, a further example of a method for processing the modified bitstream on a sink device is illustrated. In the example of Fig. 5, in one embodiment, the modified bitstream may further include information about additional media data, and processing the modified bitstream by the sink device may be based on the information. In one embodiment, the information may include one or more of information about the configuration of the additional media data, information about the presence of additional media data in the modified bitstream, information about a default loudness value, information about a value for delay alignment, information about delay adaptation, information about panning, and information about a type of the additional media data.

[0103] In the example of FIG. 5, the method may further include a step S202a of processing the additional media data by a processing unit before mixing the core decoded media data and the additional media data. Then, in step S203, the core decoded media data is mixed with the processed additional media data. In one embodiment, processing the additional media data may include one or more of resampling, delay adaptation and loudness processing. Processing the additional media data may be guided by (based on) generated information included in the modified bitstream and provided by the bitstream parser in step S201 (indicated by dashed lines). For example, resampling may be guided by information about the sampling rate of the additional media data that may be included in the information about the composition of the additional media data. Alternatively or additionally, processing the additional media data may include downmixing / upmixing of the additional media data that may be guided by information about the channel mode that may be included in the information about the composition of the additional media data. The noise processing was measured according to one of the methods described in ITU-R BS.1770-4. The user may be guided by information about default loudness values ​​which may include loudness.

[0104] Alternatively or additionally, in one embodiment, the modified bitstream may further include information on the relative gain between the encoded media data and the additional media data, and may be processed, if necessary, before mixing to obtain the processed additional media data (indicated by the dotted line). Mixing the core decoded media data and the additional media data in step S203 may be based on the information on the relative gain. Here, the sink device may use the relative gain value generated in the source device. In other words, both the source device and the sink device may use the same relative gain value. The relative gain value may also be further modified via a user input in the mixing step S203. The relative gain value may include a mixing gain and / or an attenuation gain value. The information on the relative gain may be provided by the bitstream parser in step S201, as indicated by the dashed line. The information on the additional media data and, alternatively or additionally, the information value on the relative gain may be provided in the modified bitstream in the system_sound_config_synchronization_timeout (SST) as described above. It can be embedded in a top-level syntax element.

[0105] In one embodiment, the additional media data may be compressed or uncompressed media data.

[0106] Now, referring to the example of FIG. 6, in one embodiment, the additional media data may be compressed media data, and the method may further comprise a step S205 of decoding the compressed media data by a decoder to obtain decoded additional media data, and in step S203, the decoded additional media data may be mixed with the core decoded media data. If the method further comprises a step S202a of processing the additional media data when the additional media data is compressed media data, the decoded additional media data may be subjected to the processing in step S202a indicated by the dotted line described above, if necessary, before being mixed with the core decoded media data. The format of the compressed media data may belong to, but is not limited to, the Bluetooth ecosystem. In one embodiment, the compressed media data may be data in SBC format or aptX-based format. Also, in the case of compressed media data, the processing in step S202a may be guided by (based on) information about the generated additional media data included in the modified bitstream and provided in step S201, as indicated by the dashed line.

[0107] Alternatively or additionally, the mixing in step S203 of the core decoded media data and the decoded additional media data, which may have been processed before mixing if necessary, may also be based on information about the relative gains mentioned above, which may also be provided in step S201 as indicated by the dashed line.

[0108] Alternatively, in one embodiment, the additional media data may be uncompressed media data, and in step S203, the uncompressed additional media data may be mixed with the core decoded media data according to the example illustrated in Figure 3 or Figure 4. According to the example of Figure 5, the uncompressed media data may also be processed in step S202a, as described above, if necessary, before being mixed with the core decoded media data. In one embodiment, the uncompressed media data may be PCM data generated in the source device. In one embodiment, the PCM data may include one or more of earcon data and system sound data.

[0109] Source Device Now, referring to the example of FIG. 7, an example of a source device generating a modified bitstream is illustrated. The source device 100 may include a receiver 102 configured to receive a bitstream including encoded media data 101. The bitstream may be received, for example, from one or more media servers. The bitstream may deliver, for example, audio content to be played by a sink device connected to the source device 100. The source device 100 is not limited. For example, the source device 100 may be a mobile device including one or more of a mobile phone, a smart watch, a laptop including a convertible, and a tablet PC.

[0110] In one embodiment, the encoded media data may be played by the source device, the sink device, or both the source device and the sink device. In one embodiment, the additional media data may be played by the source device, the sink device, or both the source device and the sink device. The source device 100 may further include an embedder 104 configured to generate a payload of the additional media data 103 and embed the generated payload in a bitstream to obtain a modified bitstream including the payloads of the encoded media data 101 and the additional media data 103. Here, the modified bitstream may also be generated even when the source device operates in a pass-through mode. The source device 100 may be configured to pass-through the encoded media data to the sink device via the modified bitstream. The encoded media data received from the source device is embedded in the modified bitstream and output to the sink device.

[0111] Advantageously, generating (at the source device) and outputting the modified bitstream to the sink device streamlines mixing of the media data with additional media data at the sink device. A single bitstream is used by the source device to embed the media data and the additional media data and transport them to the sink device. Thereby, a single channel / connection can be used between the source and the sink device. Furthermore, there are no additional decoders and / or encoders (e.g., for decoding and re-encoding the encoded media data) at the source device. No additional mixers (e.g., for mixing the decoded encoded media data with the additional media data) are used at the source device.

[0112] The format of the modified bitstream is not limited, but in one embodiment, the modified bitstream may be an MPEG-D USAC-based bitstream (i.e., a bitstream that complies with the MPEG-D USAC standard). In one embodiment, the embedder 104 may be configured to carry the generated payload in the modified bitstream via a USAC-type bitstream extension mechanism, as described above.

[0113] In one embodiment, the additional media data 103 may be compressed or uncompressed media data. The uncompressed media data may be, but is not limited to, PCM data generated in the source device 100 in one embodiment. The generation of the PCM data may be, but is not limited to, based on user input in one embodiment. The user input may be, for example, a keystroke. In this case, the source device 100 may include a user interface. The user interface may be, for example, a touch display. Alternatively or additionally, the source device 100 may further include at least a keyboard. However, the generation of the PCM data may also be facilitated in the source device 100 independently of the user input, for example, based on an email notification. In one embodiment, the generation of the PCM data may be based on a user input ... In this case, the PCM data may include one or more of earcon data and system sound data. The PCM data may be stored in the source device 100.

[0114] The compressed media data may be, but is not limited to, a format of the compressed media data that belongs to the Bluetooth ecosystem. In one embodiment, the compressed media data may be data in an SBC format or an aptX-based format. The compressed media data may be generated in the source device 100. The compressed media data may be stored in the source device 100. Alternatively or additionally, the compressed media data may be received via the receiver 102.

[0115] 8, in one embodiment, the embedder 104 may be further configured to generate information 106 about the additional media data 103 and embed the generated information 106 in the bitstream (indicated by a dashed line). Alternatively, or in addition, in one embodiment, the embedder 104 may be further configured to generate information 106 about a relative gain between the encoded media data 101 and the additional media data 103 and embed the information 106 about the relative gain in the bitstream (also shown by a dashed line). In one embodiment, the embedder 104 may be further configured to convey the generated information 106 and / or the generated information about the relative gain 106 in the modified bitstream via a USAC-based bitstream extension mechanism, as described above.

[0116] The source device 100 may further include a control unit 105 configured to output the modified bitstream to a sink device.

[0117] Sink Device Now, referring to the example of FIG. 9, an example of a sink device that processes a modified bitstream is illustrated. The sink device 200 may include a receiver 201 configured to receive a modified bitstream including a payload of encoded media data and additional media data. The sink device 200 is not limited. The sink device 200 may be a peripheral device of a source device, such as, for example, an earphone. The sink device 200 and the source device may be connected via Bluetooth. The sink device 200 may be configured to play audio content included in the modified bitstream. For example, the sink device 200 may be configured to play the encoded media data or the additional media data, or both the encoded media data and the additional media data.

[0118] Sink device 200 may further include a bitstream parser 202 configured to parse the modified bitstream to obtain a payload of the encoded media data and additional media data. The format of the modified bitstream is not limited, but the modified bitstream may be an MPEG-D USAC-based bitstream (i.e., a bitstream that complies with the MPEG-D USAC standard). As already mentioned above, in one embodiment, bitstream parser 202 may be configured to parse the bitstream based on identifying USAC-type bitstream extension mechanisms in the modified bitstream.

[0119] Sink device 200 may further include a core decoder 203 configured to core-decode the encoded media data to obtain core-decoded media data. In one embodiment, core decoder 203 may be an MPEG-D USAC-based decoder (conforming to the above-mentioned MPEG-D USAC standard).

[0120] The sink device 200 may further include a mixer 204 configured to mix the core decoded media data and the additional media data to obtain an output signal. In one embodiment, the additional media data may be compressed media data or uncompressed media data. In one embodiment, the additional media data may be uncompressed media data, and the mixer 204 may be configured to mix the core decoded media data and the uncompressed additional media data.

[0121] Referring now to the example of FIG. 10, the modified bitstream may further include information about the additional media data, and sink device 200 may be configured to process the modified bitstream based on the information. In one embodiment, sink device 200 may further include a processing unit 206 configured to process the additional media data before mixing the core decoded media data and the additional media data in mixer 204. In one embodiment, processing the additional media data by processing unit 206 may include one or more of resampling, delay adaptation, and loudness processing. Processing the additional media data by processing unit 206 may be guided by (based on) the information about the additional media data described above. The information about the additional media data may be provided to processing unit 206 by bitstream parser 202, as indicated by the dashed line. If the additional media data is uncompressed media data, the uncompressed media data may be processed directly after the modified bitstream is parsed in bitstream parser 202 and before entering mixer 204.

[0122] Alternatively or additionally, in one embodiment, the modified bitstream may further include information regarding the relative gain between the encoded media data and the additional media data, and the mixer 204 may be further configured to mix the core decoded media data and the additional media data, which may have been processed, if necessary, before mixing (indicated by the dotted lines), based on said information regarding the relative gain. The information regarding the relative gain may be provided to the mixer 204 by the bitstream parser 202, as indicated by the dashed lines.

[0123] In one embodiment, the additional media data may be compressed media data. Now, referring to the example of FIG. 11, the sink device 200 may further include a decoder 207 configured to decode the compressed media data to obtain the decoded additional media data, and the mixer 204 may be configured to mix the core decoded media data and the decoded additional media data. Alternatively or additionally, the modified bitstream may further include information regarding a relative gain between the encoded media data and the additional media data, and the mixer 204 may be configured to mix the core decoded media data and the decoded additional media data based on the information regarding the relative gain. The information regarding the relative gain may be provided to the mixer 204 by the bitstream parser 202, as indicated by the dashed line.

[0124] 12, the decoded additional media data may be further processed by processing unit 206, as necessary, before being mixed with the core decoded media data in mixer 204 (indicated by dotted lines). The processing of the decoded additional media data by processing unit 206 may be guided by (based on) information about the additional media data that may be included in the modified bitstream described above. The information about the additional media data may be provided to processing unit 206 by bitstream parser 202, as indicated by dashed lines.

[0125] Referring to the examples of FIGS. 9 to 12, the sink device 200 is configured to output an output signal. The sink device 200 may further include a control unit 205 configured to receive and transmit the output signal. For example, the output signal may be output to a user of the sink device 200 via an earphone.

[0126] The above methods may be implemented individually on the above devices. The above devices capable of performing the above methods may also form the respective systems. The device 300 referred to in this specification may include one or more processors 301, 302, as illustrated by way of example in FIG. 13. Alternatively or additionally, the above methods may also be implemented as a computer program product comprising a computer-readable storage medium having instructions configured to cause the device 300 to perform the method when executed on the device 300 having the processing capabilities 301, 302.

[0127] interpretation Unless otherwise indicated, and as will be apparent from the description that follows, use of terms such as "processing," "calculating," "calculating," "determining," "analyzing," and the like throughout the disclosed description is understood to refer to operations and / or processes of a computer or computing system, or similar electronic computing device, that manipulates and / or converts data expressed as physical quantities, such as quantities of electrons, into other data similarly expressed as physical quantities.

[0128] Similarly, the term "processor" may refer to any device or any part of a device that processes electronic data, e.g., from a register and / or memory, to convert that electronic data into other electronic data that may be stored in a register and / or memory. A "computer" or a "computing machine" or a "computing platform" may include one or more processors.

[0129] The methodology described herein is executable, in an example embodiment, by one or more processors receiving computer-readable (also referred to as machine-readable) code that includes a set of instructions that, when executed by one or more of the processors, performs at least one of the methods described herein. Any processor capable of executing a set of instructions (sequential or otherwise) that specify operations of interest is included. Thus, an example is a typical processing system that includes one or more processors. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system may further include a memory subsystem that includes a main RAM and / or a static RAM, and / or a ROM. A bus subsystem may be included for communication between these components. Furthermore, the processing system may be a distributed processing system with processors coupled by a network. If the processing system requires a display, such a display may be included, e.g., a liquid crystal display (LCD) or a cathode ray tube display (CRT). If manual data entry is required, the processing system also includes an input device, such as one or more of an alphanumeric input unit such as a keyboard, a pointing control device such as a mouse, and the like. The processing system may also include a storage system, such as a disk drive unit. In some configurations, the processing system may include a sound output device, and a network interface device. Thus, the memory subsystem includes a computer-readable carrier medium carrying computer-readable code (e.g., software) that includes a set of instructions that, when executed by one or more processors, causes one or more of the methods described herein to be performed. It should be noted that, when a method includes several elements, e.g., several steps, no ordering of such elements is implied unless specifically stated otherwise. The software may reside in a hard disk, or may reside, completely or at least partially, in RAM and / or in the processor during execution by the computer system. Thus, the memory and the processor also refer to the computer. In one embodiment, the computer-readable carrier medium carries data-readable code. Further, the computer-readable carrier medium may form or be included in a computer program product.

[0130] In alternative example embodiments, one or more processors may operate as stand-alone devices or may be connected, e.g., networked, to other processors in a network configuration, one or more processors may operate in the capacity of a server or user machine in a server-user network environment, or as a peer machine in a peer-to-peer or distributed network environment. One or more processors may form a personal computer (PC), tablet PC, personal digital assistant (PDA), mobile phone, web appliance, network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify operations to be performed by the machine.

[0131] It should be noted that the term "machine" is also intended to include any collection of machines that individually or collectively execute a set (or sets) of instructions for performing any one or more of the methodologies described herein.

[0132] Thus, an example embodiment of each of the methods described herein is in the form of a computer-readable carrier medium carrying a set of instructions, e.g., a computer program for execution on one or more processors, e.g., one or more processors that are part of a web server configuration. Thus, as will be appreciated by those skilled in the art, example embodiments of the present disclosure may be embodied as a method, an apparatus, such as a dedicated device, an apparatus, such as a data processing system, or a computer-readable carrier medium, e.g., a computer program product. The computer-readable carrier medium carries computer-readable code, which when executed on one or more processors, causes the processor to implement the method. Thus, aspects of the present disclosure may take the form of a method, an example entirely hardware embodiment, an example entirely software embodiment, or an example embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a carrier medium (e.g., a computer program product on a computer-readable storage medium) carrying computer-readable program code embodied in the medium.

[0133] The software may further be transmitted and received across a network via a network interface device. Although the carrier medium is a single medium in the example embodiment, the term "carrier medium" should be interpreted to include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) that store one or more sets of instructions. The term "carrier medium" should also be interpreted to include any medium that can store, encode or carry a set of instructions for execution by one or more of the processors, causing the one or more processors to perform any one or more of the methods disclosed herein. Carrier media may take many forms, including, but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical disks, magnetic disks, and magneto-optical disks. Volatile media include dynamic memory, such as main memory. Transmission media include coaxial cables, copper wire, and optical fibers, including the wires that make up a bus subsystem. Transmission media may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications. Thus, for example, the term "carrier medium" can refer to computer products embodied in solid-state memories, optical and magnetic media, media carrying propagated signals detectable by at least one processor or one or more processors and which, when executed, implement a method, and media detectable by at least one of the one or more processors and which, when executed, implement a set of instructions. This is intended to include, but is not limited to, transmission media within a network that carries propagated signals representing instructions.

[0134] It is understood that the steps of the above method are performed, in an example embodiment, by a suitable processor of a processing (e.g., computer) system executing instructions (e.g., computer readable code) stored in storage. It is also understood that the present disclosure is not limited to any particular implementation or programming technique, and that the present disclosure may be implemented using any suitable technique for implementing the functionality described herein. The present disclosure is not limited to any particular programming language or operating system.

[0135] References throughout this disclosure to "one example embodiment," "example embodiments," or "example embodiments" mean that a particular feature, structure, or characteristic described in connection with that example embodiment is included in at least one example embodiment of the disclosure. Thus, the appearances of the phrases "in one example embodiment," "in example embodiments," or "in example embodiments" in various places throughout this disclosure do not necessarily all refer to the same example embodiment. Furthermore, particular features, structures, or characteristics may be combined in any suitable manner, as would be apparent to one of ordinary skill in the art from this disclosure, in one or more example embodiments.

[0136] As used herein, unless otherwise specified, the use of ordinal numbers such as "first," "second," "third," etc. to describe a common object merely indicates that different instances of a similar object are being referenced, and is not intended to imply that the objects so described must be in a given sequence, whether temporally, spatially, in ranking order, or in any other manner.

[0137] In the following claims and in the description of this specification, the terms "comprise" or "include" are open terms meaning at least the element / feature following the term, but not excluding other elements / features. Thus, the term "comprise", when used in the claims, should not be interpreted as being limited to the means or elements or steps (steps) listed thereafter. For example, the scope of the expression "device comprising A and B" should not be limited to a device consisting of only elements A and B. As used herein, the term "comprise" is also open terms meaning at least the element / feature following the term, but not excluding other elements / features. Thus, "comprise" is synonymous with and means "comprise".

[0138] In the above description of example embodiments of the present disclosure, it should be understood that various features of the present disclosure may be grouped together in a single example embodiment, figure, or description thereof in order to simplify the disclosure and aid in understanding one or more of the various inventive aspects. However, this method of disclosure should not be interpreted as reflecting an intention that the claims require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects reside in fewer than all features of a single example embodiment disclosed above. Thus, the claims that follow this specification are expressly incorporated into this specification, with each claim standing on its own as a separate example embodiment of the present disclosure.

[0139] Moreover, although some example embodiments described herein include some features and not others that are included in other example embodiments, combinations of features of different example embodiments are meant to be within the scope of the disclosure and form different example embodiments, as would be understood by one of ordinary skill in the art. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0140] In the description provided herein, numerous specific details are set forth. However, it will be understood that example embodiments of the present disclosure may be practiced without these specific details. In other instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

[0141] Thus, while what is believed to be the best mode of the disclosure has been described, those skilled in the art will recognize that other and further modifications may be made thereto without departing from the spirit of the disclosure, and it is intended to claim all such variations and modifications as being within the scope of the disclosure. For example, any formulas above are merely representative of procedures that may be used. Functions may be added or deleted from the block diagrams, and operations may be interchanged between functional blocks. Steps may be added or deleted to methods within the scope of the disclosure.

[0142] Enumerated example embodiments (EEE) of the present disclosure are listed below. do.

[0143] A method for generating a modified bitstream on a source device, comprising: a) receiving, by a receiver, a bitstream comprising encoded media data; b) generating, by an embedder, a payload of additional media data and embedding said payload in said bitstream to obtain as output from said embedder a modified bitstream comprising said encoded media data and said payload of said additional media data; c) outputting the modified bitstream to a sink device; The method includes:

[0144] A-EEE2. The method of A-EEE1, wherein step b) further comprises generating information about said additional media data and embedding said information in said bitstream.

[0145] A-EEE3. The method of A-EEE2, wherein the generated information includes one or more of information about the composition of the additional media data, information about the presence of the additional media data in the modified bitstream, information about a default loudness value, information about a value for delay alignment, information about delay adaptation, information about panning, and information about the type of the additional media data.

[0146] A-EEE4. A method according to any of A-EEE1 to 3, wherein step b) further comprises generating information relating to a relative gain between the encoded media data and the additional media data, and embedding the information relating to the relative gain in the bitstream.

[0147] A-EEE5. A method according to any one of A-EEE1 to 4, wherein the modified bitstream is an MPEG-D USAC based bitstream.

[0148] A-EEE6. The method of A-EEE5, wherein the generated payload is embedded in the bitstream by conveying the payload in the bitstream via a USAC-type bitstream extension mechanism.

[0149] A-EEE7. Regarding the generated information and / or the generated relative gains The method according to claim 5 or 6, wherein the generated information and / or information regarding the generated relative gain is embedded in the bitstream by conveying the generated information and / or information regarding the generated relative gain in the bitstream via the USAC type bitstream extension mechanism.

[0150] A-EEE8. The method of A-EEE6 or 7, wherein the USAC type bitstream extension mechanism is a new USAC type bitstream extension element including a unique identifier.

[0151] A-EEE9. A method according to any of A-EEE1 to 8, wherein in step c) the modified bitstream is output to the sink device via a Bluetooth connection.

[0152] A-EEE10. A method according to any one of A-EEE1 to 9, wherein the additional media data is compressed media data or uncompressed media data.

[0153] A-EEE11. The method of A-EEE10, wherein the uncompressed media data is PCM data generated at the source device.

[0154] A-EEE12. The method of A-EEE11, wherein the generation of the PCM data is based on user input.

[0155] A-EEE13. The method of A-EEE11 or 12, wherein the PCM data includes one or more of earcon data and system sound data.

[0156] A-EEE14. The method of A-EEE10, wherein the compressed data is data in SBC format or an aptX-based format.

[0157] A method for processing a modified bitstream on a sink device, comprising: a) receiving, by a receiver, a modified bitstream comprising a payload of encoded media data and additional media data; b) parsing, by a bitstream parser, the modified bitstream to obtain the payload of the encoded media data and the additional media data; c) core-decoding, by a core decoder, the encoded media data to obtain core-decoded media data; d) mixing, by a mixer, the core decoded media data and the additional media data to obtain an output signal; e) outputting said output signal; The method includes:

[0158] A-EEE16. The method according to A-EEE15, wherein the modified bitstream further comprises information about the additional media data, and the method further comprises, after step a) and before step b), processing the modified bitstream based on the information.

[0159] A-EEE17. The information includes information about the composition of the additional media data, information about the presence of the additional media data in the modified bitstream, information about a default loudness value, information about a value for delay alignment, information about delay adaptation, information about panning, and information about the additional media data. The method according to claim 1, further comprising:

[0160] A-EEE18. A method according to any of A-EEE15 to 17, wherein the modified bitstream further comprises information about a relative gain between the encoded media data and the additional media data, and the mixing of the core decoded media data and the additional media data in step d) is based on the information about the relative gain.

[0161] A-EEE19. A method according to any of A-EEE15 to 18, further comprising processing the additional media data by a processing unit prior to mixing the core decoded media data and the additional media data.

[0162] A-EEE20. The method of A-EEE19, wherein the step of processing the additional media data includes one or more of resampling, delay adaptation and loudness processing.

[0163] A-EEE21. A method according to any of A-EEE15 to 20, wherein the additional media data is compressed media data and the method further comprises a step of decoding, by a decoder, the compressed media data to obtain decoded additional media data, and in step d), the decoded additional media data is mixed with the core decoded media data.

[0164] A-EEE22. The method of A-EEE21, wherein the compressed media data is data in SBC format or an aptX-based format.

[0165] A-EEE23. A method according to any one of A-EEE15 to 20, wherein the additional media data is uncompressed media data and in step d) the uncompressed additional media data is mixed with the core decoded media data.

[0166] A-EEE24. A method according to any one of A-EEE15 to 23, wherein the modified bitstream is an MPEG-D USAC based bitstream.

[0167] A-EEE25. The method of A-EEE24, wherein the modified bitstream includes a USAC type bitstream extension mechanism and the analysis in step b) is further based on identifying the USAC type bitstream extension mechanism in the modified bitstream.

[0168] A-EEE26. The method of A-EEE25, wherein the USAC-type bitstream extension mechanism is a new USAC-type bitstream extension element that includes a unique identifier.

[0169] A source device generating a modified bitstream, comprising: a) a receiver configured to receive a bitstream comprising encoded media data; b) an embedder configured to generate a payload of additional media data and embed the generated payload in the bitstream to obtain a modified bitstream comprising the encoded media data and the payload of the additional media data; and c) a control unit for outputting the modified bitstream to a sink device; The device that contains

[0170] A-EEE28. The device of A-EEE27, wherein the embedder is further configured to generate information about the additional media data and embed the information in the bitstream.

[0171] A-EEE29. A device as described in A-EEE27 or 28, wherein the embedder is further configured to generate information regarding a relative gain between the encoded media data and the additional media data, and embed the information regarding the relative gain in the bitstream.

[0172] A-EEE30. The device of any of A-EEE27 to 29, wherein the modified bitstream is an MPEG-D USAC based bitstream.

[0173] A-EEE31. A device according to any one of A-EEE27 to 30, wherein the additional media data is compressed media data or uncompressed media data.

[0174] A-EEE32. The device of A-EEE31, wherein the uncompressed media data is PCM data generated in the source device.

[0175] A-EEE33. The device of A-EEE32, wherein the generation of the PCM data is based on user input.

[0176] A-EEE34. The device of A-EEE32 or 33, wherein the PCM data includes one or more of earcon data and system sound data.

[0177] A-EEE35. The device of A-EEE31, wherein the compressed media data is data in SBC format or an aptX-based format.

[0178] A sink device for processing a modified bitstream, comprising: a) a receiver configured to receive a modified bitstream including a payload of encoded media data and additional media data; b) a bitstream parser configured to parse the modified bitstream to obtain the payload of the encoded media data and the additional media data; and c) a core decoder configured to core-decode the encoded media data to obtain core-decoded media data; d) a mixer configured to mix the core decoded media data and the additional media data to obtain an output signal; e) a control unit configured to output said output signal; and The device that contains

[0179] A-EEE37. The device of A-EEE36, further comprising a processing unit configured to process the additional media data prior to mixing the core decoded media data and the additional media data.

[0180] A-EEE38. The device of A-EEE36 or 37, wherein the additional media data is uncompressed media data and the mixer is configured to mix the core decoded media data and the uncompressed additional media data.

[0181] A-EEE39. The additional media data is compressed media data, and the device decodes the compressed media data to obtain the decoded additional media data. 38. The device of claim 37, further comprising a decoder for receiving the core decoded media data and the decoded additional media data, and wherein the mixer is configured to mix the core decoded media data and the decoded additional media data.

[0182] A-EEE40. A device described in any of A-EEE36 to 39, wherein the modified bitstream further includes information about the additional media data, and the sink device is configured to process the modified bitstream based on the information.

[0183] A-EEE41. A device described in any of A-EEE36 to 40, wherein the modified bitstream further includes information regarding a relative gain between the encoded media data and the additional media data, and the mixer is further configured to mix the core decoded media data and the additional media data based on the information regarding the relative gain.

[0184] A-EEE42. A device according to any one of A-EEE36 to 41, wherein the core decoder is an MPEG-D USAC based decoder.

[0185] A-EEE43. A system of a source device that generates a modified bitstream and a sink device that processes the modified bitstream, the source device including one or more processors configured to perform the methods described in A-EEE1-14, and the sink device including one or more processors configured to perform the methods described in A-EEE15-26.

[0186] A-EEE44. A computer program product comprising a computer-readable storage medium having instructions configured, when executed by a device having processing capability, to cause said device to perform a method as set forth in any one of A-EEE1 to A-EEE14.

[0187] A-EEE45. A computer program product comprising a computer readable storage medium having instructions configured, when executed by a device having processing capability, to cause said device to perform a method according to any one of A-EEE15 to A-EEE26.

[0188] B-EEE1. A method for generating a modified bitstream on a source device, comprising: a) receiving, by a receiver, a bitstream comprising encoded media data; b) generating, by an embedder, a payload of additional media data and embedding said payload in said bitstream to obtain as output from said embedder a modified bitstream comprising said encoded media data and said payload of said additional media data; c) outputting the modified bitstream to a sink device; The method includes:

[0189] B-EEE2. Step b) further comprises generating information about the additional media data and embedding said information in the bitstream, and optionally: The method described in B-EEE1, wherein the generated information includes one or more of information about the composition of the additional media data, information about the presence of the additional media data in the modified bitstream, information about a default loudness value, information about a value for delay alignment, information about delay adaptation, information about panning, and information about a type of the additional media data.

[0190] B-EEE3. The method according to B-EEE1, wherein step b) further comprises generating information regarding a relative gain between the encoded media data and the additional media data, and embedding the information regarding the relative gain in the bitstream.

[0191] B-EEE4. The modified bitstream is an MPEG-D USAC-based bitstream; and The method according to claim 1, wherein the generated payload is embedded in the bitstream by conveying the payload in the bitstream via a USAC type bitstream extension mechanism.

[0192] B-EEE5. The modified bitstream is an MPEG-D USAC-based bitstream; and The method described in B-EEE3, wherein the generated information and / or information regarding the generated relative gain is embedded in the bitstream by conveying the generated information and / or information regarding the generated relative gain in the bitstream via the USAC type bitstream extension mechanism.

[0193] B-EEE6. The method of B-EEE4, wherein the USAC-type bitstream extension mechanism is a new USAC-type bitstream extension element that includes a unique identifier.

[0194] B-EEE7. The method of B-EEE1, wherein in step c) the modified bitstream is output to the sink device via a Bluetooth connection.

[0195] B-EEE8. The method of B-EEE1, wherein the additional media data is PCM data generated at the source device.

[0196] B-EEE9. The method of B-EEE8, wherein the PCM data includes one or more of earcon data and system sound data.

[0197] B-EEE10. The method according to B-EEE1, wherein the additional media data is compressed data in SBC format or an aptX-based format.

[0198] B-EEE11. A method for processing a modified bitstream on a sink device, comprising: a) receiving, by a receiver, a modified bitstream comprising a payload of encoded media data and additional media data; b) parsing, by a bitstream parser, the modified bitstream to obtain the payload of the encoded media data and the additional media data; c) core-decoding, by a core decoder, the encoded media data to obtain core-decoded media data; d) mixing, by a mixer, the core decoded media data and the additional media data to obtain an output signal; e) outputting said output signal; The method includes:

[0199] B-EEE12. The modified bitstream further comprises information regarding a relative gain between the encoded media data and the additional media data, and The method according to claim 1, wherein the mixing of the core decoded media data and the additional media data in d) is based on information about the relative gains.

[0200] B-EEE13. The method further comprises the step of processing the additional media data by a processing unit before mixing the core decoded media data and the additional media data, and optionally: The method according to claim 1, wherein the step of processing the additional media data includes one or more of resampling, delay adaptation and loudness processing.

[0201] B-EEE14. The modified bitstream is an MPEG-D USAC-based bitstream, and The method of claim 1, wherein the modified bitstream includes a USAC type bitstream extension mechanism and the analysis in step b) is further based on identifying the USAC type bitstream extension mechanism in the modified bitstream.

[0202] B-EEE15. The method of B-EEE11, wherein the USAC-type bitstream extension mechanism is a new USAC-type bitstream extension element that includes a unique identifier.

[0203] B-EEE16. A source device generating a modified bitstream, comprising: a) a receiver configured to receive a bitstream comprising encoded media data; b) an embedder configured to generate a payload of additional media data and embed the generated payload in the bitstream to obtain a modified bitstream comprising the encoded media data and the payload of the additional media data; and c) a control unit for outputting the modified bitstream to a sink device; The device that contains

[0204] B-EEE17. A sink device for processing a modified bitstream, comprising: a) a receiver configured to receive a modified bitstream including a payload of encoded media data and additional media data; b) a bitstream parser configured to parse the modified bitstream to obtain the payload of the encoded media data and the additional media data; and c) a core decoder configured to core-decode the encoded media data to obtain core-decoded media data; d) a mixer configured to mix the core decoded media data and the additional media data to obtain an output signal; e) a control unit configured to output said output signal; and The device that contains

[0205] B-EEE18. A system of a source device that generates a modified bitstream and a sink device that processes the modified bitstream, the source device including one or more processors configured to perform the method described in B-EEE1 and the sink device including one or more processors configured to perform the method described in B-EEE11.

[0206] B-EEE19. A computer having instructions configured, when executed by a device having processing capability, to cause said device to perform the method described in B-EEE1. A computer program product comprising a readable storage medium.

[0207] B-EEE20. A computer program product comprising a computer-readable storage medium having instructions configured, when executed by a device having processing capability, to cause said device to perform the method recited in B-EEE11.

Claims

1. A method for generating an MPEG-D USAC-based modified bitstream by a source device comprising a receiver and an embedder, comprising: a) receiving, by said receiver, a bitstream comprising encoded media data; b) generating, by the embedder, a payload of additional audio data and embedding the payload in the bitstream to obtain as output from the embedder a modified bitstream comprising the coded media data and the payload of the additional audio data, the generated payload being embedded in the bitstream by carrying the payload in the bitstream via a USAC type bitstream extension mechanism, the USAC type bitstream extension mechanism being a USAC type bitstream extension element or an array extension element comprising a unique identifier, the additional audio data being uncompressed audio data; c) outputting the modified bitstream to a sink device; The method includes:

2. The method of claim 1 , wherein step b) further comprises generating information about the additional audio data and embedding the information in the bitstream.

3. 3. The method of claim 2, wherein the generated information includes one or more of information about a configuration of the additional audio data, information about the presence of the additional audio data in the modified bitstream, information about a default loudness value, information about a value for delay alignment, information about delay adaptation, information about panning, and information about a type of the additional audio data.

4. Step b) is a step of determining whether the encoded media data and the additional audio data are related to each other. The method of claim 1 , further comprising generating information regarding the relative gains and embedding the information regarding the relative gains in the bitstream.

5. 2. The method of claim 1 , wherein the generated information and / or information regarding the generated relative gain is embedded in the bitstream by conveying the generated information and / or information regarding the generated relative gain in the bitstream via the USAC type bitstream extension mechanism.

6. The method of claim 1 , wherein in step c), the modified bitstream is output to the sink device via a Bluetooth connection.

7. A method for processing an MPEG-D USAC-based modified bitstream by a sink device comprising a receiver, a bitstream parser, a core decoder and a mixer, comprising: a) receiving, by the receiver, a modified bitstream comprising coded media data, a payload of additional audio data, and a USAC type bitstream extension mechanism, the USAC type bitstream extension mechanism being a new USAC type bitstream extension element or array extension element including a unique identifier, and the additional audio data being uncompressed audio data; b) parsing, by the bitstream parser, the modified bitstream to obtain the payload of the encoded media data and the additional audio data, the parsing being based on identifying the USAC-type bitstream extension mechanisms in the modified bitstream; c) core-decoding, by the core decoder, the encoded media data to obtain core-decoded media data; d) mixing, by the mixer, the core decoded media data and the additional audio data to obtain an output signal; e) outputting said output signal; The method includes:

8. 8. The method of claim 7, wherein the modified bitstream further comprises information about the additional audio data, the method further comprising, after step a) and before step b), processing the modified bitstream based on the information.

9. 9. The method of claim 8, wherein the information includes one or more of information about a configuration of the additional audio data, information about the presence of the additional audio data in the modified bitstream, information about a default loudness value, information about a value for delay alignment, information about delay adaptation, information about panning, and information about a type of the additional audio data.

10. 8. The method of claim 7, wherein the modified bitstream further comprises information about a relative gain between the encoded media data and the additional audio data, and wherein the mixing of the core decoded media data and the additional audio data in step d) is based on the information about the relative gain.

11. The method of claim 7 , further comprising the step of processing the additional audio data by a processing unit before mixing the core decoded media data and the additional audio data.

12. The method of claim 11 , wherein the step of processing the additional audio data includes one or more of resampling, delay adaptation, and loudness processing.

13. A computer program comprising instructions arranged, when executed by a device having processing capability, to cause said device to carry out a method according to any of claims 1 to 12.

Images