Method and apparatus for generating and processing a modified audio bitstream

Abstract

The present description discloses a method for generating a modified bitstream on a source device, the method comprising the steps of: a) receiving, by a receiver, a bitstream comprising encoded media data; b) generating, by an embedder, a payload of additional media data and embedding the payload in the bitstream to obtain a modified bitstream as the output of the embedder, the modified bitstream comprising the encoded media data and the payload of the additional media data; and c) outputting the modified bitstream to a sink device. A method for processing the modified bitstream on the sink device is further described. In addition, corresponding source and sink devices, a system having the source and sink devices, and a corresponding computer program product are described.

Description

[0001] Cross - reference to related applications

[0002] This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 887,098, filed on August 15, 2019, and European Patent Application No. 19191920.8, filed on August 15, 2019, each of which is hereby incorporated by reference in its entirety. Technical Field

[0003] The present disclosure generally relates to a method and a source device for generating a modified bitstream, and more particularly, to generating a modified bitstream by embedding the payload of additional media data into a bitstream including encoded media data. The present disclosure also relates to a method and a sink device for processing the modified bitstream.

[0004] Although some embodiments will be described herein with particular reference to this disclosure, it should be understood that the present disclosure is not limited to such fields of use and can be applied in a broader context. Background Art

[0005] Any discussion of background art throughout the present disclosure should in no way be taken as an admission that such art is well - known in the art or forms part of the common general knowledge in the art.

[0006] Recently, efforts have been made to establish codecs based on the AAC family as part of the Bluetooth (BT) ecosystem. In this context, the Bluetooth SIG may define use cases that require a bitstream received, for example, on a mobile device (BT source) to be passed through to a connected speaker system (BT sink). Since the data format used for the services of the entire BT endpoint may be encoded audio, there is no simple way to transfer earpiece and / or system sounds from the mobile device to the BT speaker. Such earpiece sounds may include ringtones, mail notification sounds, keystroke sounds, etc.

[0007] In currently deployed systems, the audio received from services is typically decoded in the device into the uncompressed domain (PCM), where it can be modified and / or mixed. If the pass - through operation mode is enabled, it may not be easy on the BT source device to incorporate system sounds into and / or mix system sounds into the experience transmitted from the BT source to the BT sink.

[0008] One possibility to overcome this limitation while still enabling pass - through is to open an additional link between the two devices. However, the disadvantage of this setup is that it requires an additional link and an encoder - decoder pair of the Bluetooth codec to work. This introduces unwanted additional system complexity and adds additional computational complexity to the BT source and sink devices, which may also consume battery.

[0009] Accordingly, there is a current need to generate and process ear markers and / or system sounds simultaneously in parallel with media content via a bitstream, especially a bitstream in MPEG-4 audio format. Summary of the Invention

[0010] According to a first aspect of the present disclosure, a method for generating a modified bitstream by a source device is provided. The source device may include a receiver and an embedder. The method may include step a): receiving, by the receiver, a bitstream including encoded media data. The method may further include step b): generating, by the embedder, a payload of additional media data and embedding the payload into the bitstream to obtain a modified bitstream as an output of the embedder, the modified bitstream including the encoded media data and the payload of the additional media data. And the method may include step c): outputting the modified bitstream to a sink device.

[0011] Configured as proposed, the method allows system sounds or ear markers to be inserted into a bitstream that includes encoded media data and is transmitted by a source device (e.g., a Bluetooth source device such as a mobile phone) to a sink device (e.g., a Bluetooth sink device such as a Bluetooth speaker). This enables the system sound / ear marker to be delivered to the sink device without having to establish an additional connection between the source device and the sink device and without having to decode and re-encode the encoded media data. In addition, the method enables system sound / ear marker and encoded media data to be processed simultaneously.

[0012] In some embodiments, the additional media data may be played back by the sink device.

[0013] In some embodiments, the additional media data may be played back by the source device.

[0014] In some embodiments, the encoded media data in the received bitstream may be transmitted through the source device to the sink device via the modified bitstream.

[0015] In some embodiments, step b) may further include generating information about the additional media data and embedding the information into the bitstream.

[0016] In some embodiments, the generated information may include one or more of information about the configuration of the additional media data, information about the presence of the additional media data in the modified bitstream, information about a default loudness value, information about a delay alignment value, information about delay adaptation, information about panning, and information about the type of the additional media data.

[0017] In some embodiments, step b) may further include generating information related to a relative gain between the encoded media data and the additional media data, and embedding the information related to the relative gain in the bitstream.

[0018] In some embodiments, the modified bitstream may be in the MPEG-4 audio format. Thus, for media data encoded in the MPEG-4 audio format, when the source device is in the pass-through mode, system sound / earmark may be sent to the sink device as part of the encoded bitstream without decoding and re-encoding the media data.

[0019] In some embodiments, the payload may be embedded in the bitstream by transmitting the generated payload in the (modified) bitstream via an MPEG-4 audio bitstream extension mechanism.

[0020] In some embodiments, further, the generated information and / or the generated information related to the relative gain may be embedded in the bitstream by transmitting the generated information and / or the generated information related to the relative gain in the (modified) bitstream via the MPEG-4 audio bitstream extension mechanism.

[0021] In some embodiments, the MPEG-4 audio bitstream extension mechanism may be a data stream element (DSE).

[0022] In some embodiments, the data stream element may be located at a predefined position in the modified bitstream and / or may include a specific instance tag.

[0023] In some embodiments, the MPEG-4 audio bitstream extension mechanism may be a padding element (ID_FIL).

[0024] In some embodiments, the generated payload may be transmitted via a general extension payload element within the padding element.

[0025] In some embodiments, further, the generated information and / or the generated information related to the relative gain may be transmitted via the general extension payload element within the padding element.

[0026] In some embodiments, the generated payload may be transmitted via a new extension payload element within the padding element.

[0027] In some embodiments, further, the generated information and / or the generated information related to the relative gain may be transmitted via the new extension payload element within the padding element.

[0028] In some embodiments, the new extended payload element may include a unique identifier.

[0029] In some embodiments, in step c), the modified bitstream may be output to the host device via a Bluetooth connection.

[0030] In some embodiments, the additional media data may be compressed media data or uncompressed media data.

[0031] In some embodiments, the uncompressed media data may be PCM data generated at the source device.

[0032] In some embodiments, the generation of the PCM data may be based on user input.

[0033] In some embodiments, the PCM data may include one or more of earmark data and system sound data.

[0034] In some embodiments, the compressed data may be in SBC format or in an aptX-based format.

[0035] According to a second aspect of the present disclosure, there is provided a method for a host device to process a modified bitstream. The host device may include a receiver, a bitstream parser, a core decoder, and a mixer. The method may include step a): receiving, by the receiver, a modified bitstream including a payload of encoded media data and additional media data. The method may further include step b): parsing, by the bitstream parser, the modified bitstream into the encoded media data and the payload of the additional media data. The method may further include step c): core decoding, by the core decoder, the encoded media data to obtain core-decoded media data. The method may further include step d): mixing, by the mixer, the core-decoded media data and the additional media data to obtain an output signal. And the method may include step e): outputting the output signal.

[0036] In some embodiments, the modified bitstream may further include information about the additional media data, and the method may further include, after step a) and before step b), a step of processing the modified bitstream based on the information.

[0037] In some embodiments, the information may include one or more of information about the configuration of the additional media data, information about the presence of the additional media data in the modified bitstream, information about a default loudness value, information about a delay alignment value, information about delay adaptation, information about panning, and information about the type of the additional media data.

[0038] In some embodiments, the modified bitstream may further include information related to a relative gain between the encoded media data and the additional media data, and the mixing of the core-decoded media data and the additional media data in step d) may be based on the information related to the relative gain.

[0039] In some embodiments, the method may further include a step of processing the additional media data by a processing unit before mixing the core-decoded media data and the additional media data.

[0040] In some embodiments, processing the additional media data may include one or more of resampling, delay adaptation, and loudness processing.

[0041] In some embodiments, the additional media data may be compressed media data, and the method may further include a step of decoding the compressed media data by a decoder to obtain decoded additional media data, and the decoded additional media data may be mixed with the core-decoded media data in step d).

[0042] In some embodiments, the compressed media data may be in SBC format or in an aptX-based format.

[0043] In some embodiments, the additional media data may be uncompressed media data, and in step d), the uncompressed additional media data may be mixed with the core-decoded media data.

[0044] In some embodiments, the modified bitstream may be in MPEG-4 audio format.

[0045] In some embodiments, the modified bitstream may include an MPEG-4 audio bitstream extension mechanism, and the parsing in step b) may be further based on identifying the MPEG-4 audio bitstream extension mechanism in the modified bitstream.

[0046] In some embodiments, the MPEG-4 audio bitstream extension mechanism may be a data stream element (DSE), and the parsing in step b) may further be based on identifying the position of the data stream element in the modified bitstream and / or identifying a specific instance tag of the data stream element.

[0047] In some embodiments, the MPEG-4 audio bitstream extension mechanism may be a padding element (ID_FIL), and the parsing in step b) may further be based on identifying an extended payload element within the padding element.

[0048] According to a third aspect of the present disclosure, a source device for generating a modified bitstream is provided. The source device may include a) a receiver configured to receive a bitstream including encoded media data. The source device may further include b) an embedder configured to generate a payload of additional media data and embed the generated payload into the bitstream to obtain a modified bitstream, the modified bitstream including the encoded media data and the payload of the additional media data. And the source device may include c) a control unit configured to output the modified bitstream to a sink device.

[0049] In some embodiments, the embedder may further be configured to generate information about the additional media data and embed the information into the bitstream.

[0050] In some embodiments, the embedder may further be configured to generate information related to a relative gain between the encoded media data and the additional media data, and embed the information related to the relative gain into the bitstream.

[0051] In some embodiments, the modified bitstream may adopt the MPEG-4 audio format.

[0052] In some embodiments, the additional media data may be compressed media data or uncompressed media data.

[0053] In some embodiments, the uncompressed media data may be PCM data generated at the source device.

[0054] In some embodiments, the generation of the PCM data may be based on user input.

[0055] In some embodiments, the PCM data may include one or more of earmark data and system sound data.

[0056] In some embodiments, the compressed media data may be in the SBC format or in an aptX-based format.

[0057] According to a fourth aspect of the present disclosure, there is provided a sink device for processing a modified bitstream. The sink device may include a) a receiver configured to receive a modified bitstream including a payload of encoded media data and additional media data. The sink device may further include b) a bitstream parser configured to parse the modified bitstream into the encoded media data and the payload of the additional media data. The sink device may further include c) a core decoder configured to perform core decoding on the encoded media data to obtain core-decoded media data. The sink device may further include d) a mixer configured to mix the core-decoded media data and the additional media data to obtain an output signal. And the sink device may include e) a control unit configured to output the output signal.

[0058] In some embodiments, the sink device may further include a processing unit configured to process the additional media data before mixing the core-decoded media data and the additional media data.

[0059] In some embodiments, the additional media data may be uncompressed media data, and the mixer may be configured to mix the core-decoded media data and the uncompressed additional media data.

[0060] In some embodiments, the additional media data may be compressed media data, and the sink device may further include a decoder for decoding the compressed media data to obtain decoded additional media data, and the mixer may be configured to mix the core-decoded media data and the decoded additional media data.

[0061] In some embodiments, the modified bitstream may further include information about the additional media data, and the sink device may be configured to process the modified bitstream based on the information.

[0062] In some embodiments, the modified bitstream may further include information related to a relative gain between the encoded media data and the additional media data, and the mixer may be further configured to mix the core-decoded media data and the additional media data based on the information related to the relative gain.

[0063] In some embodiments, the core decoder may be an MPEG-4 audio decoder.

[0064] According to a fifth aspect of the present disclosure, there is provided a system having a source device for generating a modified bitstream and a sink device for processing the modified bitstream, wherein the source device includes one or more processors configured to execute a method for generating a modified bitstream on the source device, and wherein the sink device includes one or more processors configured to execute a method for processing the modified bitstream on the sink device.

[0065] According to a sixth aspect of the present disclosure, there is provided a computer program product including a computer-readable storage medium having instructions adapted to cause a device having processing capabilities to execute a method for generating a modified bitstream on a source device when executed by the device.

[0066] According to a seventh aspect of the present disclosure, there is provided a computer program product including a computer-readable storage medium having instructions adapted to cause a device having processing capabilities to execute a method for processing a modified bitstream on a sink device when executed by the device. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Example embodiments of the present disclosure will now be described by way of example only with reference to the accompanying drawings, in which:

[0068] Figure 1 A flowchart illustrating an example of a method for generating a modified bitstream on a source device is shown.

[0069] Figure 2 A flowchart illustrating another example of a method for generating a modified bitstream on a source device is shown.

[0070] Figure 3 A flowchart illustrating an example of a method for processing a modified bitstream on a sink device is shown.

[0071] Figure 4 A flowchart illustrating another example of a method for processing a modified bitstream on a sink device is shown.

[0072] Figure 5 A flowchart illustrating yet another example of a method for processing a modified bitstream on a sink device is shown.

[0073] Figure 6 A flowchart illustrating yet another example of a method for processing a modified bitstream on a sink device is shown.

[0074] Figure 7Illustrates an example of a source device for generating a modified bitstream.

[0075] Figure 8 Illustrates another example of a source device for generating a modified bitstream.

[0076] Figure 9 Illustrates an example of a sink device for processing a modified bitstream.

[0077] Figure 10 Illustrates another example of a sink device for processing a modified bitstream.

[0078] Figure 11 Illustrates yet another example of a sink device for processing a modified bitstream.

[0079] Figure 12 Illustrates yet another example of a sink device for processing a modified bitstream.

[0080] Figure 13 Illustrates an example of a device having one or more processors. Detailed Description

[0081] Generation and Processing of Modified MPEG-4 Audio Bitstreams

[0082] In the following, methods and devices for generating and processing modified MPEG-4 audio bitstreams will be described. In this document, an MPEG-4 audio bitstream may refer to a bitstream that is compatible with the standards specified in ISO / IEC 14496-3, Coding of audio-visual objects - Part 3: Audio [ISO / IEC 14496-3, Coding of audio-visual objects - Part 3: Audio], and all its future versions, revisions, and amendments (hereinafter referred to as "MPEG-4 audio"). With the described methods and devices, additional media data, as uncompressed or compressed media data, can be transmitted and processed together with the encoded media data that conveys media content. This can enable the transmission and processing of ear tags and / or system sound data without increasing system complexity and computational complexity in both the source device and the sink device.

[0083] Additionally, the same mechanisms and bitstream variables can be used to control all settings (e.g., the level of system sound, DRC, etc.) on both the source device and the sink device. This means that the relative gain (or mixing gain) used by the embedder of the source device can be exactly the same as the control parameter used by the mixer of the sink device.

[0084] Generating a Modified Bitstream on the Source Device

[0085] See Figure 1An example illustrates a method for generating a modified bitstream on a source device. In step S100, a bitstream including encoded media data (to be core decoded by a sink device as detailed below) can be received by a receiver of the source device. For example, a bitstream including encoded media data can be received from one or more media servers. The encoded media data can include encoded audio data. The encoded audio data can conform to the MPEG-4 audio standard. For example, the bitstream can convey audio content to be played back by a sink device connected to the source device. The source device is not limited. For example, the source device can be a mobile device, including one or more of a mobile phone, a smartwatch, a laptop computer, and a tablet computer including a convertible tablet. Referring again to Figure 1 An example, in step S101, a payload of additional media data (i.e., other than the encoded media data, such as additional audio data) can be generated by an embedder and embedded in the bitstream.

[0086] The additional media data generated in step S101 will be mixed with the media data by the sink device (on its side). Advantageously, this enables the additional media data to be mixed with the media data on the sink device side.

[0087] The encoded media data and the additional media data are different (e.g., different types) of media data. For example, as explained below, the encoded media data and the additional media data can have different formats. The encoded media data and the additional media data include different types of media content. The encoded media data can include main audio content, such as music, audio, etc. The additional media data can include auxiliary audio content, such as ear tags, system sounds, key sounds, email notification sounds, etc. The main audio content and the auxiliary audio content can be played back by the sink device.

[0088] Alternatively or additionally, the main audio content and the auxiliary audio content can be played back by the source device.

[0089] In an embodiment, the encoded media data can be played back by the source device, the sink device, or both the source device and the sink device.

[0090] In an embodiment, the additional media data can be played back by the source device, the sink device, or both the source device and the sink device.

[0091] In an embodiment, the additional media data may be compressed media data (to be decoded by a sink device as detailed below to obtain decoded additional media data) or uncompressed media data. Although the uncompressed media data is not limited, in an embodiment, the uncompressed media data may be PCM data generated at the source device. Although the generation of the PCM data is also not limited, in an embodiment, the generation of the PCM data may be based on user input. The user input may be, for example, a key press. Alternatively or additionally, the generation of the PCM data may be facilitated at the source device independently of the user input, for example, based on an email notification. In an embodiment, the PCM data may include one or more of earmark data and system sound data. Also, although the format of the compressed media data is not limited, the format may belong to the Bluetooth ecosystem. In an embodiment, the compressed media data may be in the SBC format or in an aptX-based format.

[0092] Referring again to Figure 1 the example of, in step S102, a modified bitstream including the encoded media data and the payload of the additional media data may be obtained as the output of the embedder. In step S103, the modified bitstream may then be output to the sink device.

[0093] The encoded media data in the received bitstream is relayed from the source device to the sink device via the modified bitstream. The encoded media data received from the source device is embedded in the modified bitstream and output to the sink device. Advantageously, generating (at the source device) and outputting the modified bitstream to the sink device simplifies the mixing of the media data with the additional media data on the sink device side. The source device uses a single bitstream to embed the media data and the additional media data and transmit them to the sink device, so that a single channel / connection can be used between the source device and the sink device. Further, there is no additional decoder and / or encoder in the source device to, for example, decode and re-encode the encoded media data. No additional mixer is used in the source device to, for example, mix the decoded encoded media data with the additional media data.

[0094] In an embodiment, the modified bitstream may be output to the sink device via a Bluetooth connection. In other words, the source device may be connected to the sink device via a Bluetooth connection. For example, the headset is connected to the mobile phone via a Bluetooth connection.

[0095] Generally, the proposed techniques and embodiments described throughout this disclosure and their examples related to generating a modified bitstream and outputting it to a sink device facilitate the transmission of the payload of additional media data (e.g., system sound / earmark) to the sink device, for example, when the source device is in a passthrough operation mode. In other words, the payload of additional media data can be generated and embedded into the received bitstream including the encoded media data to generate a modified bitstream without, for example, decoding and re-encoding the encoded media data. Similarly, the additional media data can be delivered to the sink device without establishing an additional connection between the source device and the sink device. As a further advantage, the proposed techniques enable the simultaneous processing of additional media data and encoded media data.

[0096] Now referring to Figure 2 the example of Figure 2 illustrates yet another example of a method for generating a modified bitstream on a source device. Figure 1 The example of

[0097] Alternatively or additionally, step S101 may further include generating information related to the relative gain between the encoded media data and the additional media data (e.g., also part of the metadata), and embedding the information related to the relative gain into the bitstream (as indicated by the dash). As detailed below, the relative gain between the encoded media data and the additional media data can be used by the sink device to generate an output signal based on the received modified bitstream. Here, the source device and the sink device may use the same relative gain value. For example, if the source device is a mobile phone, the optimal system sound mixing setting can be automatically determined based on the ratio of the system sound to the estimated loudness of the encoded media data content. For example, the relative gain value may include a mixing gain value and / or an attenuation gain value that can be used by the mixer of the sink device. By including this information in the bitstream, it can be ensured that the additional media data can be mixed with the encoded media data in an appropriate manner after decoding, and the situation of rendering the additional media data at an inappropriate sound level compared to the encoded media data can be avoided. Generally, by including information about the additional media data in the bitstream, the source device is given control over the processing (e.g., including rendering) of the additional media data at the sink device. This ensures the proper rendering of the additional media data at the sink device.

[0098] Although the format of the modified bitstream is not restricted, in an embodiment, the modified bitstream may adopt the MPEG-4 audio format (i.e., a bitstream compliant with the MPEG-4 audio standard). See Figure 1 For example, in an embodiment, the generated payload may be transmitted in the modified bitstream via the MPEG-4 audio bitstream extension mechanism. See Figure 2 For example, additionally, in an embodiment, further, the generated information and / or the generated information related to the relative gain may be transmitted in the modified bitstream via the MPEG-4 audio bitstream extension mechanism.

[0099] As an example, the following syntax may be used to embed the generated payload of the additional media data, the generated information about the additional media data, and / or the generated information related to the relative gain into the modified bitstream in the MPEG-4 audio format.

[0100] Syntax of system_sound_info()

[0101] Top-level syntax elements for transmitting the generated payload of the additional media data, the generated information (information about the additional media data), and / or the generated information related to the relative gain.

[0102]

[0103] Semantics of system_sound_info()

[0104] Element Description system_sound_config_present If set, system_sound_config exists in the stream system_sound_payload_present If set, system_sound_payload exists in the stream

[0105] Syntax of system_sound_confin()

[0106]

[0107]

[0108] Semantics of system_sound_config()

[0109]

[0110]

[0111] Table 1: Semantics of system_sound_config()

[0112] Syntax of system_sound_payload()

[0113]

[0114]

[0115] Semantics of system_sound_payload()

[0116] Element Description sys_sound_active The system sound is active and should be decoded

[0117] Syntax of pcm_data_config()

[0118]

[0119] Semantics of pcm_data_config()

[0120]

[0121] Table 2: Semantics of pcm_data_config()

[0122] Syntax of pcm_data_payload()

[0123]

[0124] Remark 1: "pcm_frame_size" is based on "pcm_frame_size_index"

[0125] Obtained or directly set in "pcm_data_config"

[0126] Note 2: The number of bits per sample (bps) can be obtained according to

[0127] "pcm_data_format_index"

[0128] The semantics of pcm_data_payload()

[0129]

[0130] Table 3: The semantics of pcm_data_payload()

[0131] The MPEG-4 audio standard provides at least two independent extension mechanisms. The Data Stream Element (ID_DSE) or extension_payload(int) can be used to transmit the payload of the generated additional media data, the generated information, and / or the generated information related to relative gain. In an embodiment, the MPEG-4 audio bitstream extension mechanism can be the Data Stream Element (DSE). The Data Stream Element in the MPEG-4 audio standard is typically used to transmit metadata rather than the payload of media data (audio content). In an embodiment, the Data Stream Element (DSE) in the MPEG-4 audio bitstream is used to transmit the payload of the generated additional media data. Therefore, the existing Data Stream Element in the MPEG-4 bitstream is used to transmit the payload of alternative metadata and / or media data (e.g., ear tags) in addition to metadata. The following gives example syntax.

[0132]

[0133] Element Description element_instance_tag Identifies the number of data stream elements. If multiple DSEs exist, they must be handled carefully cnt The payload size in bytes data_stream_byte Contains system_sound_info

[0134] In an embodiment, the Data Stream Element can be located at a predefined position in the modified bitstream and / or can include a specific instance tag.

[0135] Alternatively, in an embodiment, the MPEG-4 audio bitstream extension mechanism can be the Fill Element (ID_FIL). The MPEG-4 audio defines another extension mechanism using the top-level syntax element extension_payload(int). For inelastic payloads, extension_payload(int) can be contained within the Fill Element (ID_FIL).

[0136]

[0137] In an embodiment, the generated payload may be transmitted via a generic extension payload element within a padding element. Additionally, in an embodiment, further, the generated information (information about additional media data) and / or the generated information related to relative gain may be transmitted via a generic extension payload element within a padding element. For example, for generic data, extension_type EXT_DATA_ELEMENT may be used.

[0138] Alternatively, in an embodiment, the generated payload may be transmitted via a new extension payload element within a padding element. Additionally, in an embodiment, further, the generated information (information about additional media data) and / or the generated information related to relative gain may be transmitted via a new extension payload element within a padding element.

[0139] Padding elements in the MPEG-4 audio standard are typically used to transmit metadata rather than the payload of media data (audio content). In an embodiment, a data stream element (DSE) in the MPEG-4 audio bitstream is used to transmit the payload of the generated additional media data (e.g., as indicated above in the generic extension payload element or in the new extension payload element within the padding element). Thus, existing padding elements in the MPEG-4 bitstream are used to transmit the payload of alternative metadata and / or media data (e.g., ear tags) in addition to metadata.

[0140] A new extension payload type (extension_type) may be defined for the syntax element system_sound_info(). The syntax of 14496-3, Table 4.59: "Syntax of extension_payload" may be extended as follows:

[0141]

[0142]

[0143] In an embodiment, the new extension payload element may include a unique identifier. For example, the unique identifier may be EXT_SYS_SOUND_INFO.

[0144] Using the above techniques, additional media data and information about the additional media data can be easily embedded into an existing MPEG-4 compliant bitstream transmitted by a source device.

[0145] Processing the modified bitstream on the sink device

[0146] Now refer to Figure 3An example is illustrated, which shows an example of a method for processing a modified bitstream on a host device. In step S200, a modified bitstream including an encoded media data (e.g., as received by a source device) and a payload of additional media data can be received by a receiver of the host device.

[0147] In step S201, the received modified bitstream can be parsed by a bitstream parser into the encoded media data and the payload of the additional media data. Although the format of the modified bitstream is not limited, the modified bitstream can adopt an MPEG-4 audio format (i.e., a bitstream compliant with the MPEG-4 audio standard). See Figure 4 An example, in an embodiment, the modified bitstream can include an MPEG-4 audio bitstream extension mechanism, and the parsing in step S201 can be further based on identifying the MPEG-4 audio bitstream extension mechanism in the modified bitstream. In an embodiment, the MPEG-4 audio bitstream extension mechanism can be a data stream element (DSE), and the parsing in step S201 can be further based on identifying the position of the data stream element in the modified bitstream and / or identifying a specific instance tag of the data stream element. Alternatively, in an embodiment, the MPEG-4 audio bitstream extension mechanism can be a padding element (ID_FIL), and the parsing in step S201 can be further based on identifying an extended payload element within the padding element.

[0148] See Figure 3 Or an example of 4, in step S202, the encoded media data can be core decoded by a core decoder.

[0149] In step S203, the core decoded media data can be mixed with the additional media data by a mixer to obtain an output signal.

[0150] In step 204, the obtained output signal can then be output by a control unit.

[0151] Now see Figure 5 An example, which shows another example of a method for processing a modified bitstream on a host device. In Figure 5 The example, in an embodiment, the modified bitstream can further include information about the additional media data, and the host device can process the modified bitstream based on this information. In an embodiment, the information can include one or more of information about the configuration of the additional media data, information about the presence of the additional media data in the modified bitstream, information about a default loudness value, information about a delay alignment value, information about delay adaptation, information about panning, and information about the type of the additional media data.

[0152] InFigure 5 In the example of, the method may further include step S202a of the processing unit processing the additional media data before mixing the core-decoded media data and the additional media data. In step S203, the core-decoded media data is then mixed with the processed additional media data. In an embodiment, processing the additional media data may include one or more of resampling, latency adaptation, and loudness processing. The processing of the additional media data may be guided (as indicated by the dash) by the generated information included (based) in the modified bitstream and provided by the bitstream parser in step S201. For example, resampling may be guided by information about the sampling rate of the additional media data, which may be included in the information about the configuration of the additional media data. Alternatively or additionally, processing the additional media data may include downmixing / upmixing the additional media data, which may be guided by information about the channel mode, and the information about the channel mode may be included in the information about the configuration of the additional media data. For example, loudness processing may be guided by information about a default loudness value, which may include a loudness measured according to one of the methods specified in ITU-R BS.1770-4.

[0153] Alternatively or additionally, in an embodiment, the modified bitstream may further include information related to the relative gain between the encoded media data and the additional media data, and mixing the core-decoded media data and the additional media data in step S203 may be based on the information related to the relative gain, and the additional media data may optionally have been processed before mixing to obtain the processed additional media data (as indicated by the dotted line). Here, the sink device may use the relative gain value generated at the source device. In other words, the source device and the sink device may use the same relative gain value. The relative gain value may also be further modified via user input in the mixing step S203. The relative gain value may include a mixing gain value and / or an attenuation gain value. The information related to the relative gain may be provided by the bitstream parser in step S201, as indicated by the dash. The information about the additional media data and alternatively or additionally the information related to the relative gain value may be embedded in the modified bitstream in the system_sound_config top-level syntax element as described above.

[0154] In an embodiment, the additional media data may be compressed media data or uncompressed media data.

[0155] Now refer to Figure 6Example, in an embodiment, the additional media data may be compressed media data, and the method may further include step S205 of decoding the compressed media data by a decoder to obtain decoded additional media data, and the decoded additional media data may be mixed with the core decoded media data in step S203. If the method further includes step S202a of processing the additional media data, in the case of compressed media data, the decoded additional media data may optionally be processed in step S202a before being mixed with the core decoded media data, as detailed above and indicated by the dashed line. Although the format of the compressed media data is not limited, the format may belong to the Bluetooth ecosystem. In an embodiment, the compressed media data may be in the SBC format or in an aptX-based format. Also in the case of compressed media data, the processing in step S202a may be guided by the information generated (based on) regarding the additional media data included in the modified bitstream and provided in step S201, as indicated by the dash.

[0156] Alternatively or additionally, in step S203, the core decoded media data and the decoded additional media data (which may optionally be processed before mixing) are mixed and may also be based on the information related to the relative gain as detailed above. The information related to the relative gain may also be provided in step S201, as indicated by the dash.

[0157] Alternatively, in an embodiment, the additional media data may be uncompressed media data, and in step S203, it may be mixed with the core decoded media data according to Figure 3 or Figure 4 the example illustrated in. According to Figure 5 the example, the uncompressed additional media data may also optionally be processed in step S202a before being mixed with the core decoded media data, as detailed above. In an embodiment, the uncompressed media data may be PCM data generated at the source device. In an embodiment, the PCM data may include one or more of ear marker data and system sound data.

[0158] Source device

[0159] Now refer to Figure 7Example, illustrating an example of a source device for generating a modified bitstream. The source device 100 may include a receiver 102 configured to receive a bitstream including encoded media data 101. For example, the bitstream may be received from one or more media servers. For example, the bitstream may convey audio content to be played back by a sink device connected to the source device 100. The source device 100 is not limited. For example, the source device 100 may be a mobile device, including one or more of a mobile phone, a smartwatch, a laptop computer, and a tablet computer including a convertible tablet.

[0160] In an embodiment, the encoded media data may be played back by the source device, the sink device, or both the source device and the sink device. In an embodiment, additional media data may be played back by the source device, the sink device, or both the source device and the sink device.

[0161] The source device 100 may further include an embedder 104 configured to generate a payload of the additional media data 103 and embed the generated payload in the bitstream to obtain a modified bitstream that includes the encoded media data 101 and the payload of the additional media data 103. Here, even if the source device operates in a pass-through mode, a modified bitstream can be generated. The source device 100 may be configured to transmit the encoded media data to the sink device via the modified bitstream. The encoded media data received from the source device is embedded in the modified bitstream and output to the sink device.

[0162] Advantageously, generating (at the source device) and outputting the modified bitstream to the sink device simplifies the mixing of media data with additional media data on the sink device side. The source device uses a single bitstream to embed the media data and the additional media data and transmit them to the sink device, so that a single channel / connection can be used between the source device and the sink device. In addition, there is no additional decoder and / or encoder in the source device to, for example, decode and re-encode the encoded media data. No additional mixer is used in the source device to, for example, mix the decoded encoded media data with the additional media data.

[0163] Although the format of the modified bitstream is not limited, in an embodiment, the modified bitstream may adopt the MPEG-4 audio format (i.e., a bitstream compliant with the MPEG-4 audio standard). In an embodiment, as detailed above, the embedder may be configured to transmit the generated payload via the MPEG-4 audio bitstream extension mechanism in the modified bitstream.

[0164] In an embodiment, the additional media data 103 may be compressed media data or uncompressed media data. Although the uncompressed media data is not limited, in an embodiment, the uncompressed media data may be PCM data generated at the source device 100. Although the generation of the PCM data is not limited, in an embodiment, the generation of the PCM data may be based on a user input. The user input may be, for example, a key press. The source device 100 may include a user interface that may be, for example, a touch display. Alternatively or additionally, the source device 100 may further include at least a keyboard. However, the generation of the PCM data may also be facilitated at the source device 100 independent of the user input, for example, based on an email notification. In an embodiment, the PCM data may include one or more of earmark data and system sound data. The PCM data may be stored in the source device 100.

[0165] Although the compressed media data is not limited, the format of the compressed media data may belong to the Bluetooth ecosystem. In an embodiment, the compressed media data may be in the SBC format or in an aptX-based format. The compressed media data may be generated at the source device 100. The compressed media data may be stored in the source device 100. Alternatively or additionally, the compressed media data may be received via the receiver 102.

[0166] Now referring to Figure 8 the example of, in an embodiment, the embedder 104 may further be configured to generate information 106 about the additional media data 103 and embed the generated information 106 into the bitstream (as indicated by the dash). Alternatively or additionally, in an embodiment, the embedder 104 may further be configured to generate information 106 related to the relative gain between the encoded media data 101 and the additional media data 103, and embed the information 106 related to the relative gain into the bitstream (also as indicated by the dash). In an embodiment, as detailed above, the embedder may further be configured to transmit the generated information and / or the generated information related to the relative gain via the MPEG-4 audio bitstream extension mechanism in the modified bitstream.

[0167] The source device 100 may further include a control unit 105 configured to output the modified bitstream to the sink device.

[0168] Sink device

[0169] Now referring to Figure 9An example illustrates an example of a sink device for processing a modified bitstream. The sink device 200 may include a receiver 201 configured to receive a modified bitstream including a payload of encoded media data and additional media data. The sink device 200 is not limited. The sink device 200 may be a peripheral device of a source device, such as a headset. The sink device 200 and the source device may be connected via Bluetooth. The sink device 200 may be configured to play back audio content included in the modified bitstream. For example, the sink device 200 may be configured to play back the encoded media data or the additional media data or both the encoded media data and the additional media data.

[0170] The sink device 200 may further include a bitstream parser 202 configured to parse the modified bitstream into a payload of encoded media data and additional media data. Although the format of the modified bitstream is not limited, the modified bitstream may adopt an MPEG-4 audio format (i.e., a bitstream compliant with the MPEG-4 audio standard). As detailed above, in an embodiment, the bitstream parser 202 may be configured to parse the bitstream based on identifying an MPEG-4 audio bitstream extension mechanism in the modified bitstream.

[0171] The sink device 200 may further include a core decoder 203 configured to perform core decoding on the encoded media data to obtain core-decoded media data. In an embodiment, the core decoder 203 may be an MPEG-4 audio decoder (as detailed above, compliant with the MPEG-4 audio standard).

[0172] The sink device 200 may further include a mixer 204 configured to mix the core-decoded media data and the additional media data to obtain an output signal. In an embodiment, the additional media data may be compressed media data or uncompressed media data. In an embodiment, the additional media data may be uncompressed media data, and the mixer 204 may be configured to mix the core-decoded media data and the uncompressed additional media data.

[0173] Now refer to Figure 10In an example, the modified bitstream may further include information about additional media data, and the sink device 200 may be configured to process the modified bitstream based on this information. In an embodiment, the sink device 200 may further include a processing unit 206 configured to process the additional media data before mixing the core-decoded media data and the additional media data in the mixer 204. In an embodiment, processing the additional media data by the processing unit 206 may include one or more of resampling, delay adaptation, and loudness processing. As detailed above, processing the additional media data by the processing unit 206 may be guided by (based on) the information about the additional media data. As indicated by the dash, the information about the additional media data may be provided by the bitstream parser 202 to the processing unit 206. In the case where the additional media data is uncompressed media data, the uncompressed media data may be processed directly after the modified bitstream has been parsed in the bitstream parser 202 and before entering the mixer 204.

[0174] Alternatively or additionally, in an embodiment, the modified bitstream may further include information related to a relative gain between the encoded media data and the additional media data, and as described above, the mixer 204 may further be configured to mix the core-decoded media data and the additional media data based on the information related to the relative gain, and the additional media data may optionally be processed (as indicated by the dotted line) before mixing. As indicated by the dash, the information related to the relative gain may be provided by the bitstream parser 202 to the mixer 204.

[0175] In an embodiment, the additional media data may be compressed media data. Now referring to Figure 11 In an example, the sink device 200 may further include a decoder 207 configured to decode the compressed media data to obtain decoded additional media data, and the mixer 204 may be configured to mix the core-decoded media data and the decoded additional media data. Alternatively or additionally, the modified bitstream may further include information related to a relative gain between the encoded media data and the additional media data, and as described above, the mixer 204 may further be configured to mix the core-decoded media data and the decoded additional media data based on the information related to the relative gain. As indicated by the dash, the information related to the relative gain may be provided by the bitstream parser 202 to the mixer 204.

[0176] Further referring to Figure 12In an example, the decoded additional media data can further optionally be processed by the processing unit 206 (as indicated by the dashed line) before being mixed with the core-decoded media data in the mixer 204. As detailed above, the processing of the decoded additional media data by the processing unit 206 can be guided by (based on) information about the additional media data that may be included in the modified bitstream. As indicated by the dash, information about the additional media data can be provided by the bitstream parser 202 to the processing unit 206.

[0177] See Figures 9 to 12 In an example, the host device 200 can further include a control unit 205 that is configured to output an output signal. For example, the output signal can be output to a user of the host device 200 via headphones.

[0178] The above method can be implemented one by one on the above device. The above device capable of executing the above method can also form a corresponding system. As mentioned herein, the device 300 can include one or more processors 301, 302, as Figure 13 illustrated in the example of

[0179] Interpret

[0180] Unless otherwise specifically stated, it should be understood from the following discussion that throughout the present disclosure, terms such as "processing", "computing", "calculating", "determining", "analyzing", etc. are used to refer to actions and / or processes of a computer or computing system or similar electronic computing device that manipulate and / or transform data represented as physical (such as electronic) quantities into other data similarly represented as physical quantities.

[0181] In a similar manner, the term "processor" can refer to any device or part of a device that processes electronic data, for example, from registers and / or memory, to transform the electronic data into other electronic data that can, for example, be stored in registers and / or memory. A "computer" or "computer machine" or "computing platform" can include one or more processors.

[0182] In an example embodiment, the methods described herein may be executed by one or more processors that receive computer-readable (also referred to as machine-readable) code containing a set of instructions that, when executed by the one or more processors, perform at least one of the methods described herein. Any processor that includes a set of instructions (sequential or otherwise) capable of executing the specified actions to be taken. Thus, one example is a typical processing system that includes one or more processors. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system may further include a memory subsystem that includes main RAM and / or static RAM and / or ROM. A bus subsystem may be included for communication between components. The processing system may further be a distributed processing system in which the processors are coupled together via a network. If the processing system requires a display, such a display may be included, for example, a liquid crystal display (LCD) or a cathode ray tube (CRT) display. If manual data input is required, the processing system also includes input devices such as one or more of an alphanumeric input unit (such as a keyboard), a pointing control device (such as a mouse), etc. The processing system may also encompass a storage system such as a disk drive unit. The processing system in some configurations may include a sound output device and a network interface device. The memory subsystem thus includes a computer-readable carrier medium carrying computer-readable code (e.g., software) that includes a set of instructions that, when executed by the one or more processors, cause one or more of the methods described herein to be performed. It should be noted that when the method includes several elements (e.g., several steps), no order of these elements is implied unless specifically stated. During the execution of software by a computer system, the software may reside on a hard disk, or it may also reside entirely or at least partially in RAM and / or in the processor. Thus, the memory and the processor also constitute a computer-readable carrier medium carrying computer-readable code. Additionally, the computer-readable carrier medium may form or be included in a computer program product.

[0183] In an alternative example embodiment, one or more processors may operate as a stand-alone device or may be connected to (e.g., networked to) other processors in a networked deployment, and the one or more processors may operate as a server or a user machine in a server-user network environment, or as a peer machine in a peer-to-peer or distributed network environment. One or more processors may form a personal computer (PC), a tablet PC, a personal digital assistant (PDA), a cellular phone, a web appliance, a network router, a switch, or a bridge, or any machine capable of executing a set of instructions (sequential or otherwise) specifying the actions to be taken by the machine.

[0184] It should be noted that the term "machine" should also be considered to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methods discussed herein.

[0185] Accordingly, an example embodiment of each method described herein takes the form of a computer-readable carrier medium carrying a set of instructions, such as a computer program for execution on one or more processors (e.g., one or more processors as part of a web server device). Thus, as will be understood by those skilled in the art, example embodiments of the present disclosure may be embodied as a method, an apparatus such as a dedicated apparatus, an apparatus such as a data processing system, or a computer-readable carrier medium (e.g., a computer program product). The computer-readable carrier medium carries computer-readable code including a set of instructions that, when executed on one or more processors, cause the one or more processors to implement the method. Accordingly, aspects of the present disclosure may take the form of a method, a fully hardware example embodiment, a fully software example embodiment, or an example embodiment combining software and hardware aspects. Additionally, the present disclosure may take the form of a carrier medium (e.g., a computer program product on a computer-readable storage medium) that carries computer-readable program code embodied in the medium.

[0186] Software may be further sent or received via a network interface device over a network. Although in the example embodiments the carrier medium is a single medium, the term "carrier medium" should be considered to include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) that store a set or multiple sets of instructions. The term "carrier medium" should also be considered to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by one or more of the processors and that causes one or more of the processors to execute any one or more of the methods of the present disclosure. The carrier medium may take many forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical discs, magnetic disks, and magneto-optical discs. Volatile media includes dynamic memory, such as main memory. Transmission media includes coaxial cables, copper wire, and fiber optics, including the wires that make up a bus subsystem. Transmission media can also take the form of acoustic or light waves, such as acoustic or light waves generated during radio wave and infrared data communications. For example, the term "carrier medium" should thus be considered to include but not limited to solid-state memory, computer products embodied in optical and magnetic media; media that carry a propagated signal that can be detected by at least one processor or one or more processors and that represents a set of instructions that, when executed, implement the method; and transmission media in a network that carries a propagated signal that can be detected by at least one of one or more processors and that represents the set of instructions.

[0187] It will be understood that, in one example embodiment, the steps of the methods discussed are performed by a suitable processor (or processors) in a processing (e.g., computer) system that executes instructions (computer-readable code) stored in a storage device. It will also be understood that the present disclosure is not limited to any particular implementation or programming technique, and the present disclosure may be implemented using any suitable technique for implementing the functions described herein. The present disclosure is not limited to any particular programming language or operating system.

[0188] References throughout this disclosure to "one example embodiment", "some example embodiments", or "example embodiments" mean that a particular feature, structure, or characteristic described in connection with the example embodiments is included in at least one example embodiment of the present disclosure. Thus, the phrases "in one example embodiment", "in some example embodiments", or "in example embodiments" appearing throughout this disclosure are not necessarily all referring to the same example embodiment. Additionally, in one or more example embodiments, the particular features, structures, or characteristics may be combined in any suitable manner, which will be apparent to those of ordinary skill in the art in light of the present disclosure.

[0189] As used herein, unless otherwise specified, the use of ordinal adjectives "first", "second", "third", etc. to describe a common object merely indicates different instances of like objects and is not intended to imply that the objects so described must be in a given order in time, space, rank, or any other manner.

[0190] In the following claims and the description herein, any one of the terms comprising, comprised of, or which comprises is an open term, meaning that it includes at least the subsequent element / feature, but does not exclude other elements / features. Thus, when the term "comprising" is used in a claim, it should not be construed as limited to the apparatus or elements or steps listed thereafter. For example, the expression for a device including A and B should not be limited to a device that only includes elements A and B. As used herein, any one of the terms including, which includes, or that includes is also an open term, which also means that it includes at least the element / feature after the term, but does not exclude other elements / features. Thus, including is synonymous with comprising and means comprising.

[0191] It should be understood that in the above description of the exemplary embodiments of the present disclosure, various features of the present disclosure are sometimes combined in a single exemplary embodiment / figure or its description in order to simplify the present disclosure and to assist in understanding one or more of the inventive aspects. However, the methods of the present disclosure should not be construed as reflecting an intention that the claims require more features than those expressly recited in each claim. On the contrary, as reflected in the following claims, the inventive aspects lie in less than all of the features of a single previously disclosed exemplary embodiment. Accordingly, the claims following the description are hereby expressly incorporated into this description, where each claim stands on its own as a separate exemplary embodiment of the present disclosure.

[0192] Moreover, although some of the exemplary embodiments described herein include some features included in other exemplary embodiments and do not include other features included in other exemplary embodiments, as will be understood by those skilled in the art, combinations of features of different exemplary embodiments are intended to be within the scope of the present disclosure and to form different exemplary embodiments. For example, in the following claims, any of the exemplary embodiments recited in the claimed exemplary embodiments can be used in any combination.

[0193] In the description provided herein, numerous specific details are set forth. However, it should be understood that the exemplary embodiments of the present disclosure may be practiced without these specific details. In other instances, well-known methods, structures, and techniques have not been shown in detail in order to avoid obscuring an understanding of this description.

[0194] Accordingly, although a mode has been described that is considered to be the best mode of the present disclosure, those skilled in the art will recognize that other and further modifications can be made thereto without departing from the spirit of the present disclosure, and it is intended to claim all such changes and modifications that fall within the scope of the present disclosure. For example, any of the formulas given above merely represent processes that can be used. Functions can be added or removed from the block diagrams, and operations can be interchanged between functional blocks. Steps can be added to or removed from the methods described within the scope of the present disclosure.

[0195] The following lists the exemplary embodiments (EEEs) enumerated in the present disclosure.

[0196] A-EEE 1. A method for generating a modified bitstream on a source device, wherein the method comprises the following steps:

[0197] a) receiving, by a receiver, a bitstream comprising encoded media data;

[0198] b) The embedder generates a payload of additional media data and embeds the payload into the bitstream to obtain a modified bitstream as the output of the embedder, the modified bitstream including the encoded media data and the payload of the additional media data; and

[0199] c) Output the modified bitstream to the destination device.

[0200] A-EEE 2. The method according to A-EEE 1, wherein step b) further includes generating information about the additional media data and embedding the information into the bitstream.

[0201] A-EEE 3. The method according to A-EEE 2, wherein the generated information includes one or more of information about the configuration of the additional media data, information about the presence of the additional media data in the modified bitstream, information about the default loudness value, information about the delay alignment value, information about the delay adaptation, information about the translation, and information about the type of the additional media data.

[0202] A-EEE 4. The method according to any one of A-EEE 1 to 3, wherein step b) further includes generating information related to the relative gain between the encoded media data and the additional media data, and embedding the information related to the relative gain into the bitstream.

[0203] A-EEE 5. The method according to any one of A-EEE 1 to 4, wherein the modified bitstream adopts the MPEG-4 audio format.

[0204] A-EEE 6. The method according to A-EEE 5, wherein the payload is embedded into the bitstream by transmitting the generated payload in the modified bitstream via the MPEG-4 audio bitstream extension mechanism.

[0205] A-EEE 7. The method according to A-EEE 6, wherein further, the generated information and / or the generated information related to the relative gain is embedded into the bitstream by transmitting the generated information and / or the generated information related to the relative gain in the modified bitstream via the MPEG-4 audio bitstream extension mechanism.

[0206] A-EEE 8. The method according to A-EEE 6 or 7, wherein the MPEG-4 audio bitstream extension mechanism is a data stream element (DSE).

[0207] A-EEE 9. The method according to A-EEE 8, wherein the data stream element is located at a predefined position in the modified bitstream and / or includes a specific instance tag.

[0208] 10. The method according to A-EEE 6 or 7, wherein the MPEG-4 audio bitstream extension mechanism is a padding element (ID_FIL).

[0209] A-EEE 11. The method according to A-EEE 10, wherein the generated payload is transmitted via a general extension payload element within the padding element.

[0210] A-EEE 12. The method according to A-EEE 11, wherein further, the generated information and / or the generated information related to the relative gain is transmitted via the general extension payload element within the padding element.

[0211] A-EEE 13. The method according to A-EEE 10, wherein the generated payload is transmitted via a new extension payload element within the padding element.

[0212] A-EEE 14. The method according to A-EEE 13, wherein further, the generated information and / or the generated information related to the relative gain is transmitted via the new extension payload element within the padding element.

[0213] A-EEE 15. The method according to A-EEE 13 or 14, wherein the new extension payload element includes a unique identifier.

[0214] A-EEE 16. The method according to any one of A-EEE 1 to 15, wherein in step c), the modified bitstream is output to the sink device via a Bluetooth connection.

[0215] A-EEE 17. The method according to any one of A-EEE 1 to 16, wherein the additional media data is compressed media data or uncompressed media data.

[0216] A-EEE 18. The method according to A-EEE 17, wherein the uncompressed media data is PCM data generated at the source device.

[0217] A-EEE 19. The method according to A-EEE 18, wherein the generation of the PCM data is based on user input.

[0218] A-EEE 20. The method according to A-EEE 18 or 19, wherein the PCM data includes one or more of ear tag data and system sound data.

[0219] A-EEE 21. The method according to A-EEE 17, wherein the compressed data is in SBC format or in an aptX-based format.

[0220] A-EEE 22. A method for processing a modified bitstream on a host device, wherein the method comprises the steps of:

[0221] a) receiving, by a receiver, a modified bitstream including a payload of encoded media data and additional media data;

[0222] b) parsing, by a bitstream parser, the modified bitstream into the encoded media data and the payload of the additional media data;

[0223] c) core decoding, by a core decoder, the encoded media data to obtain core-decoded media data;

[0224] d) mixing, by a mixer, the core-decoded media data and the additional media data to obtain an output signal; and

[0225] e) outputting the output signal.

[0226] A-EEE 23. The method according to A-EEE 22, wherein the modified bitstream further includes information about the additional media data, and wherein the method further comprises, after step a) and before step b), the step of processing the modified bitstream based on the information.

[0227] A-EEE 24. The method according to A-EEE 23, wherein the information includes one or more of information about the configuration of the additional media data, information about the presence of the additional media data in the modified bitstream, information about a default loudness value, information about a delay alignment value, information about delay adaptation, information about panning, and information about the type of the additional media data.

[0228] A-EEE 25. The method according to any one of A-EEE 22 to 24, wherein the modified bitstream further includes information related to a relative gain between the encoded media data and the additional media data, and wherein, in step d), mixing the core-decoded media data and the additional media data is based on the information related to the relative gain.

[0229] A-EEE 26. The method according to any one of A-EEE 22 to 25, wherein the method further comprises the step of processing the additional media data by a processing unit before mixing the core-decoded media data and the additional media data.

[0230] A-EEE 27. The method according to A-EEE 26, wherein processing the additional media data comprises one or more of resampling, latency adaptation, and loudness processing.

[0231] A-EEE 28. The method according to any one of A-EEE 22 to 27, wherein the additional media data is compressed media data, and the method further comprises the step of decoding the compressed media data by a decoder to obtain decoded additional media data, and wherein the decoded additional media data is mixed with the core-decoded media data in step d).

[0232] A-EEE 29. The method according to A-EEE 28, wherein the compressed media data is in SBC format or in an aptX-based format.

[0233] A-EEE 30. The method according to any one of A-EEE 22 to 27, wherein the additional media data is uncompressed media data, and wherein, in step d), the uncompressed additional media data is mixed with the core-decoded media data.

[0234] A-EEE 31. The method according to any one of A-EEE 22 to 30, wherein the modified bitstream is in MPEG-4 audio format.

[0235] A-EEE 32. The method according to A-EEE 31, wherein the modified bitstream comprises an MPEG-4 audio bitstream extension mechanism, and wherein the parsing in step b) is further based on identifying the MPEG-4 audio bitstream extension mechanism in the modified bitstream.

[0236] A-EEE 33. The method according to A-EEE 32, wherein the MPEG-4 audio bitstream extension mechanism is a data stream element (DSE), and the parsing in step b) is further based on identifying the position of the data stream element in the modified bitstream and / or identifying a specific instance tag of the data stream element.

[0237] A-EEE 34. The method according to A-EEE 32, wherein the MPEG-4 audio bitstream extension mechanism is a padding element (ID_FIL), and the parsing in step b) is further based on identifying the extended payload element within the padding element.

[0238] A-EEE 35. A source device for generating a modified bitstream, wherein the source device comprises:

[0239] a) a receiver configured to receive a bitstream comprising encoded media data;

[0240] b) an embedder configured to generate a payload of additional media data and embed the generated payload into the bitstream to obtain a modified bitstream, the modified bitstream comprising the encoded media data and the payload of the additional media data; and

[0241] c) a control unit configured to output the modified bitstream to a sink device.

[0242] A-EEE 36. The source device according to A-EEE 35, wherein the embedder is further configured to generate information about the additional media data and embed the information into the bitstream.

[0243] A-EEE 37. The source device according to A-EEE 35 or 36, wherein the embedder is further configured to generate information related to a relative gain between the encoded media data and the additional media data, and embed the information related to the relative gain into the bitstream.

[0244] A-EEE 38. The source device according to any one of A-EEE 35 to 37, wherein the modified bitstream is in MPEG-4 audio format.

[0245] A-EEE 39. The source device according to any one of A-EEE 35 to 38, wherein the additional media data is compressed media data or uncompressed media data.

[0246] A-EEE 40. The source device according to A-EEE 39, wherein the uncompressed media data is PCM data generated at the source device.

[0247] A-EEE 41. The source device according to A-EEE 40, wherein the generation of the PCM data is based on user input.

[0248] A-EEE 42. The source device according to A-EEE 40 or 41, wherein the PCM data includes one or more of ear tag data and system sound data.

[0249] A-EEE 43. The source device according to A-EEE 39, wherein the compressed media data is in SBC format or in an aptX-based format.

[0250] A-EEE 44. A sink device for processing a modified bitstream, wherein the sink device comprises:

[0251] a) a receiver configured to receive a modified bitstream comprising a payload of encoded media data and additional media data;

[0252] b) a bitstream parser configured to parse the modified bitstream into the encoded media data and the payload of the additional media data;

[0253] c) a core decoder configured to perform core decoding on the encoded media data to obtain core-decoded media data;

[0254] d) a mixer configured to mix the core-decoded media data and the additional media data to obtain an output signal; and

[0255] e) a control unit configured to output the output signal.

[0256] A-EEE 45. The sink device according to A-EEE 44, wherein the sink device further comprises a processing unit configured to process the additional media data before mixing the core-decoded media data and the additional media data.

[0257] A-EEE 46. The sink device according to A-EEE 44 or 45, wherein the additional media data is uncompressed media data, and wherein the mixer is configured to mix the core-decoded media data and the uncompressed additional media data.

[0258] A-EEE 47. The sink device according to A-EEE 44 or 45, wherein the additional media data is compressed media data, and the sink device further comprises a decoder for decoding the compressed media data to obtain decoded additional media data, and wherein the mixer is configured to mix the core-decoded media data and the decoded additional media data.

[0259] A-EEE 48. The sink device according to any one of A-EEE 44 to 47, wherein the modified bitstream further includes information about the additional media data, and wherein the sink device is configured to process the modified bitstream based on the information.

[0260] A-EEE 49. The sink device according to any one of A-EEE 44 to 48, wherein the modified bitstream further includes information related to a relative gain between the encoded media data and the additional media data, and wherein the mixer is further configured to mix the core-decoded media data and the additional media data based on the information related to the relative gain.

[0261] A-EEE 50. The sink device according to any one of A-EEE 44 to 49, wherein the core decoder is an MPEG-4 audio decoder.

[0262] A-EEE 51. A system having a source device for generating a modified bitstream and a sink device for processing the modified bitstream, wherein the source device includes one or more processors configured to execute the method according to any one of A-EEE 1 to 21, and wherein the sink device includes one or more processors configured to execute the method according to A-EEE 22 to 34.

[0263] A-EEE 52. A computer program product, the computer program product including a computer-readable storage medium having instructions adapted to cause a device having processing capabilities to execute the method according to A-EEE 1 when executed by the device.

[0264] A-EEE 53. A computer program product, the computer program product including a computer-readable storage medium having instructions adapted to cause a device having processing capabilities to execute the method according to A-EEE 22 when executed by the device.

[0265] B-EEE 1. A method for generating a modified bitstream on a source device, wherein the method includes the following steps:

[0266] a) Receiving, by a receiver, a bitstream including encoded media data;

[0267] b) Generating, by an embedder, a payload of additional media data and embedding the payload in the bitstream to obtain a modified bitstream as an output of the embedder, the modified bitstream including the encoded media data and the payload of the additional media data; and

[0268] c) Output the modified bitstream to the sink device.

[0269] B-EEE 2. The method according to B-EEE 1,

[0270] wherein step b) further comprises generating information about the additional media data and embedding the information into the bitstream; and optionally,

[0271] wherein the generated information includes one or more of information about the configuration of the additional media data, information about the presence of the additional media data in the modified bitstream, information about the default loudness value, information about the delay alignment value, information about the delay adaptation, information about the pan, and information about the type of the additional media data.

[0272] B-EEE 3. The method according to B-EEE 1, wherein step b) further comprises generating information related to the relative gain between the encoded media data and the additional media data, and embedding the information related to the relative gain into the bitstream.

[0273] B-EEE 4. The method according to B-EEE 1,

[0274] wherein the modified bitstream adopts the MPEG-4 audio format; and

[0275] wherein the payload is embedded into the bitstream by transmitting the generated payload in the bitstream via the MPEG-4 audio bitstream extension mechanism.

[0276] B-EEE 5. The method according to B-EEE 3,

[0277] wherein the modified bitstream adopts the MPEG-4 audio format; and

[0278] wherein the generated information and / or the generated information related to the relative gain is embedded into the bitstream by transmitting the generated information and / or the generated information related to the relative gain in the bitstream via the MPEG-4 audio bitstream extension mechanism.

[0279] B-EEE 6. The method according to B-EEE 4,

[0280] wherein the MPEG-4 audio bitstream extension mechanism is a data stream element (DSE); or

[0281] Wherein, the MPEG-4 audio bitstream extension mechanism is a padding element (ID_FIL), and wherein the generated payload is optionally transmitted via a general extension payload element or a new extension payload element within the padding element.

[0282] B-EEE 7. The method according to B-EEE 1, wherein, in step c), the modified bitstream is output to the sink device via a Bluetooth connection.

[0283] B-EEE 8. The method according to B-EEE 1, wherein the additional media data is PCM data generated at the source device.

[0284] B-EEE 9. The method according to B-EEE 8, wherein the PCM data includes one or more of earmark data and system sound data.

[0285] B-EEE 10. The method according to B-EEE 1, wherein the additional media data is compressed data in SBC format or in an aptX-based format.

[0286] B-EEE 11. A method for processing a modified bitstream on a sink device, wherein the method comprises the following steps:

[0287] a) Receiving, by a receiver, a modified bitstream including a payload of encoded media data and additional media data;

[0288] b) Parsing, by a bitstream parser, the modified bitstream into the payload of the encoded media data and the additional media data;

[0289] c) Core decoding, by a core decoder, the encoded media data to obtain core-decoded media data;

[0290] d) Mixing, by a mixer, the core-decoded media data and the additional media data to obtain an output signal; and

[0291] e) Outputting the output signal.

[0292] B-EEE 12. The method according to B-EEE 11, wherein the modified bitstream further includes information related to a relative gain between the encoded media data and the additional media data, and wherein, in step d), mixing the core-decoded media data and the additional media data is based on the information related to the relative gain.

[0293] B-EEE 13. The method according to B-EEE 11

[0294] wherein, the method further comprises the step of processing the additional media data by a processing unit before mixing the core-decoded media data and the additional media data; and optionally,

[0295] wherein, processing the additional media data includes one or more of resampling, delay adaptation, and loudness processing.

[0296] B-EEE 14. The method according to B-EEE 11,

[0297] wherein, the modified bitstream is in MPEG-4 audio format; and

[0298] wherein, the modified bitstream includes an MPEG-4 audio bitstream extension mechanism, and wherein the parsing in step b) is further based on identifying the MPEG-4 audio bitstream extension mechanism in the modified bitstream.

[0299] B-EEE 15. The method according to B-EEE 14,

[0300] wherein, the MPEG-4 audio bitstream extension mechanism is a data stream element (DSE), and the parsing in step b) is further based on identifying the position of the data stream element in the modified bitstream and / or identifying a specific instance tag of the data stream element; or

[0301] wherein, the MPEG-4 audio bitstream extension mechanism is a padding element (ID_FIL), and the parsing in step b) is further based on identifying an extended payload element within the padding element.

[0302] B-EEE 16. A source device for generating a modified bitstream, wherein the source device comprises:

[0303] a) a receiver configured to receive a bitstream comprising encoded media data;

[0304] b) an embedder configured to generate a payload of additional media data and embed the generated payload into the bitstream to obtain a modified bitstream, the modified bitstream comprising the encoded media data and the payload of the additional media data; and

[0305] c) a control unit configured to output the modified bitstream to a sink device.

[0306] B-EEE 17. A sink device for processing a modified bitstream, wherein the sink device comprises:

[0307] a) A receiver configured to receive a modified bitstream having a payload including encoded media data and additional media data;

[0308] b) A bitstream parser configured to parse the modified bitstream into the encoded media data and the payload of the additional media data;

[0309] c) A core decoder configured to perform core decoding on the encoded media data to obtain core-decoded media data;

[0310] d) A mixer configured to mix the core-decoded media data and the additional media data to obtain an output signal; and

[0311] e) A control unit configured to output the output signal.

[0312] B-EEE 18. A system having a source device for generating a modified bitstream and a sink device for processing the modified bitstream, wherein the source device includes one or more processors configured to perform the method according to B-EEE 1, and wherein the sink device includes one or more processors configured to perform the method according to B-EEE 11.

[0313] B-EEE 19. A computer program product including a computer-readable storage medium having instructions adapted to cause a device having processing capabilities to perform the method according to B-EEE 1 when executed by the device.

[0314] B-EEE 20. A computer program product including a computer-readable storage medium having instructions adapted to cause a device having processing capabilities to perform the method according to B-EEE 11 when executed by the device.

Claims

1. A method for generating a modified bitstream in MPEG-4 audio format by a source device, the source device including a receiver and an embedder, wherein, The method comprises the following steps: a) receiving, by the receiver, a bitstream comprising encoded media data; b) generating, by the embedder, a payload of additional uncompressed PCM audio data and embedding the payload into the bitstream to obtain a modified bitstream as the output of the embedder, the modified bitstream comprising the encoded media data and the payload of the additional uncompressed PCM audio data, wherein the payload is embedded into the bitstream by transmitting the generated payload in the modified bitstream via an MPEG-4 audio bitstream extension mechanism, and wherein the MPEG-4 audio bitstream extension mechanism is a data stream element (DSE); and c) outputting the modified bitstream to a sink device.

2. The method according to claim 1, wherein, Step b) further comprises generating information about the additional uncompressed PCM audio data and embedding the information into the bitstream.

3. The method according to claim 2, wherein The generated information comprises one or more of information about the configuration of the additional uncompressed PCM audio data, information about the presence of the additional uncompressed PCM audio data in the modified bitstream, information about a default loudness value, information about a delay alignment value, information about delay adaptation, information about panning, and information about the type of the additional uncompressed PCM audio data.

4. The method according to any one of claims 1 to 3, wherein, Step b) further comprises generating information related to a relative gain between the encoded media data and the additional uncompressed PCM audio data and embedding the information related to the relative gain into the bitstream.

5. The method according to claim 4, wherein, Further, the generated information and / or the information related to the relative gain is embedded into the bitstream by transmitting the generated information and / or the information related to the relative gain in the modified bitstream via the MPEG-4 audio bitstream extension mechanism.

6. The method according to any one of claims 1 to 3, wherein The MPEG-4 audio bitstream extension mechanism is a data stream element (DSE), and the data stream element (DSE) is located at a predefined position in the modified bitstream and / or comprises a specific instance tag.

7. The method according to any one of claims 1 to 3, wherein, In step c), the modified bitstream is output to the sink device via a Bluetooth connection.

8. The method according to any one of claims 1 to 3, wherein The generation of the PCM audio data is based on user input.

9. The method according to any one of claims 1 to 3, wherein, The PCM audio data comprises one or more of earmark data and system sound data.

10. A method for a host device to process a modified bitstream in MPEG-4 audio format, the host device including a receiver, a bitstream parser, a core decoder, and a mixer, wherein, The method comprises the following steps: a) receiving, by the receiver, a modified bitstream comprising encoded media data, a payload of additional uncompressed PCM audio data, and an MPEG-4 audio bitstream extension mechanism, wherein the MPEG-4 audio bitstream extension mechanism is a data stream element (DSE); b) parsing, by the bitstream parser, the modified bitstream into the encoded media data and the payload of the additional uncompressed PCM audio data, wherein the parsing is based on identifying the MPEG-4 audio bitstream extension mechanism in the modified bitstream; c) The core decoder core decodes the encoded media data to obtain core decoded media data; d) The mixer mixes the core decoded media data and the additional uncompressed PCM audio data to obtain an output signal; and e) Output the output signal.

11. The method according to claim 10, wherein, The modified bitstream further includes information about the additional uncompressed PCM audio data, and wherein the method further includes, after step a) and before step b), the step of processing the modified bitstream based on the information.

12. The method according to claim 11, wherein, The information includes one or more of information about the configuration of the additional uncompressed PCM audio data, information about the presence of the additional uncompressed PCM audio data in the modified bitstream, information about a default loudness value, information about a delay alignment value, information about delay adaptation, information about panning, and information about the type of the additional uncompressed PCM audio data.

13. The method according to any one of claims 10 to 12, wherein, The modified bitstream further includes information related to a relative gain between the encoded media data and the additional uncompressed PCM audio data, and wherein, in step d), the mixing of the core decoded media data and the additional uncompressed PCM audio data is based on the information related to the relative gain.

14. The method according to any one of claims 10 to 12, wherein, The method further includes the step of processing the additional uncompressed PCM audio data by a processing unit before mixing the core decoded media data and the additional uncompressed PCM audio data.

15. The method according to claim 14, wherein, Processing the additional uncompressed PCM audio data includes one or more of resampling, delay adaptation, and loudness processing.

16. The method according to any one of claims 10 to 12, wherein, The parsing in step b) is further based on identifying the position of the data stream element in the modified bitstream and / or identifying a specific instance tag of the data stream element.

17. A source device for generating a modified bitstream in the MPEG-4 audio format, wherein, The source device includes: a) A receiver configured to receive a bitstream including encoded media data; b) An embedder configured to generate a payload of additional uncompressed PCM audio data and embed the generated payload into the bitstream to obtain a modified bitstream, the modified bitstream including the encoded media data and the payload of the additional uncompressed PCM audio data, wherein the embedder is further configured to embed the payload into the bitstream by transmitting the generated payload in the modified bitstream via an MPEG-4 audio bitstream extension mechanism, wherein the MPEG-4 audio bitstream extension mechanism is a data stream element (DSE); and c) A control unit configured to output the modified bitstream to a sink device.

18. The source device according to claim 17, wherein, The embedder is further configured to generate information about the additional uncompressed PCM audio data and embed the information into the bitstream.

19. The source device according to claim 17 or 18, wherein The embedder is further configured to generate information related to a relative gain between the encoded media data and the additional uncompressed PCM audio data, and embed the information related to the relative gain into the bitstream.

20. The source device according to claim 17 or 18, wherein The generation of the uncompressed PCM audio data is based on user input.

21. The source device according to claim 17 or 18, wherein, The uncompressed PCM audio data includes one or more of earmark data and system sound data.

22. A sink device for processing a modified bitstream in MPEG-4 audio format, wherein, The sink device includes: a) a receiver configured to receive a modified bitstream including encoded media data, a payload of additional uncompressed PCM audio data, and an MPEG-4 audio bitstream extension mechanism, wherein the MPEG-4 audio bitstream extension mechanism is a data stream element (DSE); b) a bitstream parser configured to parse the modified bitstream into the encoded media data and the payload of the additional uncompressed PCM audio data, wherein the bitstream parser is further configured to parse the modified bitstream based on identifying the MPEG-4 audio bitstream extension mechanism in the modified bitstream; c) a core decoder configured to perform core decoding on the encoded media data to obtain core-decoded media data; d) a mixer configured to mix the core-decoded media data and the additional uncompressed PCM audio data to obtain an output signal; and e) a control unit configured to output the output signal.

23. The host device according to claim 22, wherein, The sink device further includes a processing unit configured to process the additional uncompressed PCM audio data before mixing the core-decoded media data and the additional uncompressed PCM audio data.

24. The host device according to any one of claims 22 to 23, wherein, The modified bitstream further includes information about the additional uncompressed PCM audio data, and wherein the sink device is configured to process the modified bitstream based on the information.

25. The host device according to any one of claims 22 to 23, wherein, The modified bitstream further includes information related to a relative gain between the encoded media data and the additional uncompressed PCM audio data, and wherein the mixer is further configured to mix the core-decoded media data and the additional uncompressed PCM audio data based on the information related to the relative gain.

26. The host device according to any one of claims 22 to 23, wherein, The core decoder is an MPEG-4 audio decoder.

27. A system having a source device for generating a modified bitstream in MPEG-4 audio format and a sink device for processing the modified bitstream in MPEG-4 audio format, wherein, The source device includes one or more processors configured to perform the method according to any one of claims 1 to 9, wherein the sink device includes one or more processors configured to perform the method according to claims 10 to 16.

28. A computer program product comprising a computer-readable storage medium having instructions adapted to cause a device having processing capabilities to perform the method according to any one of claims 1 to 16 when executed by the device.

Images