Methods and devices for generating and processing modified bitstreams

Abstract

To provide a method for generating a bitstream that can simultaneously process earcon / system sound in parallel with media contents.SOLUTION: A method of generating a modified bitstream in a source device having a receiver and an embedder comprises the steps of: receiving, by the receiver, a bitstream containing encoded media data; obtaining, as an output from the embedder, a modified bitstream including the encoded media data and a payload of additional media data, by generating the payload of additional media data by the embedder and embedding the payload into the bitstream; and outputting the modified bitstream to a sink device.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] Cross - reference to related applications This application claims priority based on U.S. Provisional Patent Application No. 62 / 887,111 filed on August 15, 2019, and European Patent Application No. 19191919.0 filed on August 15, 2019, and incorporates the entire disclosure content of each application herein by reference.

[0002] Generally speaking, the present disclosure relates to a method for generating a modified bitstream and a source device, and more particularly, to generating a modified bitstream by embedding the payload of additional media data into a bitstream including encoded media data. The present disclosure also relates to a method for processing such a modified bitstream and a sink device.

[0003] Although several embodiments are described herein with particular reference to their disclosure, it is understood that the present disclosure is not limited to such fields of use and is applicable in a broader context.

Background Art

[0004] No description of the background art in the present disclosure as a whole should ever be considered as an admission that such technology is widely known or forms part of common general knowledge in the field.

[0005] In recent years, establishing an AAC - family - based codec as part of the Bluetooth (BT) ecosystem has been considered. In this context, Bluetooth The SIG may define use cases such as requiring a pass-through of a bitstream received on a mobile device (BT source) to a connected speaker system (BT sink). Since the data format used by the service across the entire BT endpoint may be encoded audio, there is no simple way to transmit earcon and / or system sounds from the mobile device to the BT speaker. Such earcon sounds may include ringtones, email notification sounds, key-press sounds, etc.

[0006] In currently used systems, audio received from a service is typically decoded into the uncompressed domain (PCM) within the device. In the uncompressed domain, audio can be modified and / or mixed. When passthrough operation mode is enabled, it may not be readily possible to enable and / or mix system sounds into the experience transferred from the BT source to the BT sink on the BT source device.

[0007] One possibility to overcome this limitation by enabling passthrough would be to establish an additional link between both devices. However, this setup has the drawback of requiring an additional link and encoder-decoder pair of Bluetooth codecs to function. This involves unnecessary additional system complexity and can burden both the BT source and sink devices with additional computational complexity and battery power. [Overview of the Initiative] [Problems that the invention aims to solve]

[0008] Therefore, it has long been necessary to generate and process earphone and / or system sounds simultaneously with media content via a bitstream, particularly via an MPEG-D USAC-based bitstream. [Means for solving the problem]

[0009] A first aspect of this disclosure provides a method for generating a modified bitstream by a source device. The source device may comprise a receiver and an embedder. The method may include a) receiving a bitstream containing encoded media data by the receiver. The method may further include b) obtaining a modified bitstream containing the encoded media data and the payload of the additional media data as output from the embedder by generating a payload of additional media data by the embedder and embedding the payload in the bitstream. The method may further include c) outputting the modified bitstream to a sink device.

[0010] As proposed, this method enables the insertion of system tones or earphones into a bitstream that includes encoded media data and is passed through by a source device (e.g., a Bluetooth source device such as a mobile phone) to a sink device (e.g., a Bluetooth sink device such as a Bluetooth loudspeaker). This allows system tones / earphones to be passed to the sink device without the need to establish further connections between the source and sink devices, and without the need to decode and re-encode the encoded media data. Furthermore, this method enables the simultaneous processing of system tones / earphones and encoded media data.

[0011] In some embodiments, additional media data may be played back by a sink device.

[0012] In some embodiments, additional media data can be played back by the source device.

[0013] In some embodiments, encoded media data in the received bitstream can be passed through from the source device to the sink device via the modified bitstream.

[0014] In some embodiments, step b) may further include generating information about additional media data and embedding such information in the bitstream.

[0015] In some embodiments, the generated information may include one or more of the following: information about the configuration of the additional media data, information about the presence of the additional media data in the modified bitstream, information about the default loudness value, information about the value for delay alignment, information about delay adaptation, information about panning, and information about the type of additional media data.

[0016] In some embodiments, step b) may further include generating information about the relative gain between the encoded media data and the added media data, and embedding the relative gain information in the bitstream.

[0017] In some embodiments, the modified bitstream may be an MPEG-D USAC-based bitstream. Therefore, the system audio / earphones are source When the device is in pass-through mode for media data encoded in the MPEG-4 audio format, the media data can be sent to the sink device as part of the encoded bitstream without needing to decode and re-encode it.

[0018] In some embodiments, the generated payload may be embedded in the bitstream by transporting the payload in the (modified) bitstream via a USAC-type bitstream expansion mechanism.

[0019] In some embodiments, information about the generated information and / or the generated relative gain can be embedded in the bitstream by carrying the information about the generated information and / or the generated relative gain through the USAC type bitstream extension mechanism in the (modified) bitstream.

[0020] In some embodiments, the USAC type bitstream extension mechanism can be a new USAC type bitstream extension element that includes a unique identifier.

[0021] In some embodiments, in step c), the modified bitstream can be output to the sink device via a Bluetooth connection.

[0022] In some embodiments, the additional media data can be compressed media data or uncompressed media data.

[0023] In some embodiments, the uncompressed media data can be PCM data generated at the source device.

[0024] In some embodiments, the generation of the PCM data can be based on user input.

[0025] In some embodiments, the PCM data can include one or more of audio data and system sound data.

[0026] In some embodiments, the compressed data can be data in the SBC format or an aptX-based format.

[0027] According to a second aspect of the present disclosure, a method for processing a modified bitstream by a sink device is provided. The sink device may include a receiver and a bitstream parser. The method may include: a) receiving, by the receiver, a modified bitstream including a payload of encoded media data and additional media data; The method may further include: b) parsing, by the bitstream parser, the modified bitstream to obtain the payload of the encoded media data and the additional media data.

[0028] The method may further include: c) core-decoding, by a core decoder, the encoded media data to obtain core-decoded media data. The method may further include: d) mixing, by a mixer, the core-decoded media data and the additional media data to obtain an output signal. Further, the method may include: e) outputting the output signal.

[0029] In some embodiments, the modified bitstream further includes information about the additional media data, and the method may further include, after step a) and before step b), processing the modified bitstream based on the information.

[0030] In some embodiments, the information may include one or more of information about the composition of the additional media data, information about the presence of the additional media data in the modified bitstream, information about a default loudness value, information about a value for delay alignment, information about delay adaptation, information about panning, and information about the type of the additional media data.

[0030] In some embodiments, the modified bitstream may further include information about the relative gain between the encoded media data and the additional media data, and the mixing of the core-decoded media data and the additional media data in step d) may be based on such relative gain information.

[0031] In some embodiments, the method may further include the step of processing the additional media data by a processing unit before mixing the core-decoded media data and the additional media data.

[0032] In some embodiments, processing additional media data may include one or more of the following: resampling, delay adaptation, and loudness processing.

[0033] In some embodiments, the supplemental media data may be compressed media data, and the method may further include the step of decoding the compressed media data with a decoder to obtain decoded supplemental media data, in step d), the decoded supplemental media data may be mixed with the core-decoded media data.

[0034] In some embodiments, the compressed media data may be in SBC format or aptX-based format.

[0035] In some embodiments, the additional media data may be uncompressed media data, and in step d), the uncompressed additional media data may be mixed with the core-decoded media data.

[0036] In some embodiments, the modified bitstream may be an MPEG-D USAC-based bitstream.

[0037] In some embodiments, the modified bitstream may include a USAC-type bitstream extension mechanism, and the analysis in step b) may further be based on identifying the USAC-type bitstream extension mechanism in the modified bitstream.

[0038] In some embodiments, the USAC-type bitstream extension mechanism may be a new USAC-type bitstream extension element that includes a unique identifier.

[0039] A third aspect of this disclosure provides a source device for generating a modified bitstream. The device may include a) a receiver configured to receive a bitstream containing encoded media data. The device may further include b) an embedder configured to generate an additional media data payload and embed the generated payload into the bitstream to obtain a modified bitstream containing the encoded media data and the additional media data payload. The device may further include c) a control unit that outputs the modified bitstream to a sink device.

[0040] In some embodiments, the embedder may be further configured to generate information about additional media data and embed such information into the bitstream.

[0041] In some embodiments, the embedder may be further configured to generate information about the relative gain between encoded media data and additional media data, and to embed such relative gain information into the bitstream.

[0042] In some embodiments, the modified bitstream may be an MPEG-D USAC-based bitstream.

[0043] In some embodiments, the additional media data may be compressed media data or uncompressed media data.

[0044] In some embodiments, the uncompressed media data may be PCM data generated on the source device.

[0045] In some embodiments, the generation of PCM data may be based on user input.

[0046] In some embodiments, the PCM data may include one or more of the earphone data and system sound data.

[0047] In some embodiments, the compressed media data may be data in SBC format or aptX-based format.

[0048] A fourth aspect of this disclosure provides a sink device for processing a modified bitstream. The device may include a) a receiver configured to receive a modified bitstream, which includes a payload of encoded media data and additional media data. The device may further include b) a bitstream parser configured to parse the modified bitstream to obtain a payload of encoded media data and additional media data. The device may further include c) a core decoder configured to core decode the encoded media data to obtain core-decoded media data. The device may further include d) a mixer configured to mix the core-decoded media data and additional media data to obtain an output signal. Furthermore, the device may include e) a control unit configured to output an output signal.

[0049] In some embodiments, the device may further include a processing unit configured to process the additional media data before mixing the core-decoded media data with the additional media data.

[0050] In some embodiments, the additional media data is uncompressed media data, and the mixer may be configured to mix the core-decoded media data with the uncompressed additional media data.

[0051] In some embodiments, the additional media data is compressed media data, and the device may further include a decoder that decodes the compressed media data to obtain decoded additional media data, and the mixer may be configured to mix the core decoded media data and the decoded additional media data.

[0052] In some embodiments, the modified bitstream may further include information about additional media data, and the sink device modifies based on that information. It can be configured to process the resulting bitstream.

[0053] In some embodiments, the modified bitstream may further include information regarding the relative gain between the encoded media data and the additional media data, and the mixer may be further configured to mix the core-decoded media data and the additional media data based on the relative gain information.

[0054] In some embodiments, the core decoder may be an MPEG-D USAC-based decoder.

[0055] A fifth aspect of this disclosure provides a system comprising a source device that generates a modified bitstream and a sink device that processes the modified bitstream. The source device includes one or more processors configured on the source device to perform a method of generating a modified bitstream. The sink device includes one or more processors configured on the sink device to perform a method of processing the modified bitstream.

[0056] According to a sixth aspect of this disclosure, a computer program product is provided comprising a computer-readable storage medium having instructions configured to cause a device having processing capabilities to perform a method of generating a modified bitstream on a source device.

[0057] According to a seventh aspect of this disclosure, a computer program product is provided comprising a computer-readable storage medium having instructions configured to cause a device having processing capabilities to perform a method of processing a modified bitstream on a sink device. [Brief explanation of the drawing]

[0058] The following describes an example of an embodiment of this disclosure with reference to the attached drawings.

[0059] [Figure 1] Figure 1 illustrates a flowchart of an example of how to generate the modified bitstream on the source device. [Figure 2] Figure 2 illustrates a flowchart of a further example of how to generate the modified bitstream on the source device. [Figure 3] Figure 3 illustrates a flowchart showing an example of how to process the modified bitstream on a sink device. [Figure 4] Figure 4 illustrates a flowchart of a further example of how the modified bitstream is processed on the sink device. [Figure 5] Figure 5 illustrates a flowchart of another further example of how the modified bitstream is processed on the sink device. [Figure 6] Figure 6 illustrates a flowchart of another further example of how the modified bitstream is processed on the sink device. [Figure 7] Figure 7 illustrates an example of a source device that generates a modified bitstream. [Figure 8]Figure 8 illustrates a further example of a source device that generates a modified bitstream. [Figure 9] Figure 9 illustrates an example of a sink device that processes a modified bitstream. [Figure 10] Figure 10 illustrates a further example of a sink device that processes a modified bitstream. [Figure 11] Figure 11 illustrates another further example of a sink device that handles a modified bitstream. [Figure 12] Figure 12 illustrates another further example of a sink device that handles a modified bitstream. [Figure 13] Figure 13 illustrates an example of a device having one or more processors. [Modes for carrying out the invention]

[0060] Generation and processing of modified bitstreams based on MPEG-D USAC. The following describes methods and devices for generating and processing modified MPEG-D USAC-based bitstreams. In this specification, MPEG-D USAC-based bitstreams are defined in ISO / IEC 23003-3 MPEG audio technologies - Part 3: Unified This may refer to a bitstream conforming to the standards described in speech and audio coding (hereinafter, "MPEG-D USAC") and all future editions, versions, and amendments thereof. Hereafter, references to MPEG-D USAC-based bitstreams mean bitstreams that conform to one or more requirements described in the MPEG-D USAC standard. By the methods and devices described above, additional media data, which is uncompressed or compressed media data, can be carried and processed together with encoded media data that delivers media content. This may enable the transmission and processing of earphone data and / or system sound without further burden of system complexity and computational complexity at both the source and sink devices.

[0061] Furthermore, it may be possible to control all settings (e.g., system sound level, DRC, etc.) for both the source and sink devices using the same mechanism and bitstream variables. This means that the relative gain (or mixing gain) used by the embedder of the source device may be exactly the same as the control parameter used by the mixer of the sink device.

[0062] Generate the modified bitstream on the source device. Referring to the example in Figure 1, an example of how to generate a modified bitstream on a source device is illustrated. In step S100, the bitstream containing encoded media data (which will be core-decoded by the sink device as described later) may be received by a receiver on the source device. The bitstream containing encoded media data may be received from, for example, one or more media servers. The encoded media data may include encoded speech and audio data. The encoded speech and audio data may conform to the MPEG-D USAC standard described above. The bitstream may deliver, for example, audio content to be played back by a sink device connected to the source device. The source device is not limited. For example, the source device may be a mobile device including one or more of the following: a mobile phone, a smartwatch, a laptop including a convertible, and a tablet PC.

[0063] Refer again to the example in Figure 1. In step S101, an additional media data payload (i.e., media data added separately from the encoded media data, such as additional audio data) may be generated and embedded in the bitstream by the embedder.

[0064] The additional media data generated in step S101 will be mixed with the media data by the sink device (on the sink device side). Advantageously, this allows the sink device to mix the additional media data with the media data.

[0065] Encoded media data and supplemental media data are different (e.g., different types) media data. For example, as described below, encoded media data and supplemental media data may have different formats. Encoded media data and supplemental media data contain different types of media content. Encoded media data may include primary audio content, such as music and audio. Supplemental media data may include secondary audio content, such as earphone sounds, system sounds, keystroke sounds, and email notification sounds. Primary and secondary audio content can be played back by the sink device.

[0066] Alternatively, or as an addendum, the primary and secondary audio content may be played by the source device.

[0067] In one embodiment, encoded media data can be played back by a source device, a sink device, or both a source device and a sink device.

[0068] In one embodiment, the additional media data may be played back by a source device, a sink device, or both a source device and a sink device.

[0069] In one embodiment, the additional media data may be compressed media data (which will be decoded by the sink device as described below, and the decoded additional media data will be obtained) or uncompressed media data. The uncompressed media data may, in one embodiment, be PCM data generated in the source device. The generation of PCM data may also, in one embodiment, be based on user input. User input may, for example, be a keystroke. Alternatively, or additionally, the generation of PCM data may be facilitated in the source device independently of user input, for example, based on an email notification. In one embodiment, the PCM data may include one or more of earphone data and system sound data. The format of the compressed media data may, in one embodiment, belong to the Bluetooth ecosystem. In one embodiment, the compressed media data may be data in SBC format or aptX-based format.

[0070] Refer again to the example in Figure 1. In step S102, a modified bitstream containing the encoded media data and the additional media data payload may be obtained as the output from the embedder. Next, in step S103, the modified bitstream may be output to the sink device.

[0071] The encoded media data in the received bitstream is passed through from the source device to the sink device via the modified bitstream. The encoded media data received from the source device is embedded in the modified bitstream and output to the sink device.

[0072] Advantageously, generating a modified bitstream (at the source device) and outputting it to the sink device streamlines the process of mixing media data with additional media data at the sink device. A single bitstream is used by the source device to embed the media data and additional media data and transfer them to the sink device. This allows for the use of a single channel / connection between the source and sink devices. Furthermore, there are no additional decoders and / or encoders (e.g., for decoding and re-encoding encoded media data) at the source device. No additional mixers (e.g., for mixing decoded encoded media data with additional media data) are used at the source device.

[0073] In one embodiment, the modified bitstream may be output to a sink device via a Bluetooth connection. In other words, the source device may be connected to the sink device via a Bluetooth connection. For example, there may be earphones connected to a mobile phone via Bluetooth.

[0074] In general, the proposed technology and implementation examples described throughout this disclosure, as well as such examples, involve generating a modified bitstream and outputting it to a sink device, for example, facilitating the transmission of an additional media data payload (e.g., system sound / earphones) to a sink device when the source device is in pass-through operation mode. In other words, for example, without the encoded media data being decoded and re-encoded, the additional media data payload is generated and embedded in the received bitstream containing the encoded media data to generate a modified bitstream. Similarly, the additional media data can be passed to the sink device without the need to establish further connections between the source device and the sink device. As a further advantage, the proposed technology enables simultaneous processing of additional media data and encoded media data.

[0075] Here, referring to the example in Figure 2, we illustrate a further example of how the modified bitstream is generated on the source device. The example in Figure 2 differs from the example in Figure 1 detailed above in that, in step S101, further information about the additional media data may be generated (e.g., as part of metadata) and embedded in the bitstream (indicated by dashed lines). In one embodiment, the generated information may include one or more of the following: information about the configuration of the additional media data, information about the presence of additional media data in the modified bitstream, information about the default loudness value, information about the value for delay alignment, information about delay adaptation, information about panning, and information about the type of additional media data. The generated information may be used by the sink device to process the received modified bitstream, as detailed below. Information about the configuration of the additional media data may include, for example, information about the sampling rate, which may allow the sink device to determine whether the additional media data may need to be resampled before mixing. Alternatively, or as an addition, information about the configuration of the additional media data may further include, for example, information about the channel mode that allows the sink device to determine whether the additional media data may need to be downmixed / upmixed before mixing. Information about the default loudness value may include, for example, the loudness measured according to one of the methods described in ITU-R BS.1770-4. The measured loudness can enable the sink device to determine at what level the additional media data can be mixed with the core-decoded media data. Furthermore, information about the type of additional media data can indicate to the sink device whether the additional media data may need to be decoded, for example, whether the additional media data can be uncompressed or compressed media data.

[0076] Alternatively, or additionally, step S101 may further include generating information about the relative gain between the encoded media data and the additional media data (e.g., additional media data as part of metadata) and embedding the relative gain information in the bitstream (indicated by dashed lines). The relative gain between the encoded media data and the additional media data may be used by a sink device to generate an output signal based on the received modified bitstream, as detailed below. Here, the source device and the sink device may use the same relative gain value. For example, if the source device is a mobile phone, the optimal system sound mixing setting may be based on the estimated loudness ratio of the system sound and the encoded media data content. This can be determined automatically. For example, the relative gain value may include a mixing gain value and / or attenuation gain value that can be used by the mixer of the sink device. Including this information in the bitstream ensures that, after decoding, the additional media data can be mixed with the encoded media data in an appropriate manner, and that cases where the additional media data is rendered at an inappropriate sound level compared to the encoded media data can be avoided. In general, including information about the additional media data in the bitstream gives the source device control over how the sink device handles the additional media data (including rendering). This ensures proper rendering of the additional media data at the sink device.

[0077] The format of the modified bitstream is not limited, but in one embodiment, the modified bitstream may be an MPEG-D USAC-based bitstream (i.e., a bitstream compliant with the MPEG-D USAC standard). Referring to an example in Figure 1, in one embodiment, the generated payload may be carried in the modified bitstream via a USAC-type bitstream extension mechanism. Referring to Figure 2, in one embodiment, the generated information and / or information regarding the generated relative gain may be carried in the modified bitstream via a USAC-type bitstream extension mechanism.

[0078] As an example, the following syntax may be used to embed the payload of the generated additional media data, information about the generated additional media data, and / or information about the generated relative gain into a modified MPEG-D USAC-based bitstream.

[0079] Syntax of system_sound_info() A top-level syntax element that conveys information about the payload of the generated additional media data, the generated information (information about the additional media data), and / or the generated relative gain.

[0080] JPEG0007829756000001.jpg70144

[0081] Semantics of system_sound_info() JPEG0007829756000002.jpg23140

[0082] Syntax of system_sound_config() JPEG0007829756000003.jpg191146

[0083] Semantics of system_sound_config() JPEG0007829756000004.jpg101142

[0084] Syntax of system_sound_payload() JPEG0007829756000005.jpg119145

[0085] Semantics of system_sound_payload() JPEG0007829756000006.jpg11141

[0086] Syntax of pcm_data_config() JPEG0007829756000007.jpg105144

[0087] Semantics of pcm_data_config() JPEG0007829756000008.jpg172141

[0088] Syntax of pcm_data_payload() JPEG0007829756000009.jpg35144 NOTE1: "pcm_frame_size" can be obtained from "pcm_frame_size_index" or set directly in "pcm_data_config". NOTE2: The number of bits / samples (bps) is obtained from "pcm_data_format_index". It is possible

[0089] Semantics of pcm_data_payload() JPEG0007829756000010.jpg24141

[0090] In one embodiment, the USAC bitstream extension mechanism may be a new USAC bitstream extension element that includes a unique identifier. In the USAC bitstream, system sound info is a USAC bitstream extension as defined in ISO / IEC 23003-3. It can be transported by a tension element. A new usacExtElementType may be named ID_EXT_ELE_SYS_SOUND_INFO.

[0091] To transport only the components, the syntax in Table 17, "Syntax of UsacExtElementConfig()", can be corrected as follows: JPEG0007829756000011.jpg41141

[0092] In this case, system_sound_info() may only carry configuration data. The field system_sound_config_present may be set to 1, and the field system_sound_payload_present may be set to 0.

[0093] The generated PCM data payload can be transmitted frame by frame according to Table 25 of ISO / IEC 23003-3, "Syntax of UsacExtElement()".

[0094] In one embodiment, the payload of the generated additional media data may be embedded in the array extension element of the USAC type bitstream. The structure system_sound_info defined above may be embedded in the array "usacExtElementSegmentData" in Table 25, which may include either the generated information (information about the additional media data, e.g., configuration) or the generated payload, or both.

[0095] The array extension elements of a USAC type bitstream (e.g., array element "usacExtElementSegmentData") typically contain encoded media data and / or metadata. In this embodiment, the array extension element carries (includes) the generated payload of additional media data (e.g., uncompressed additional media data such as PCM data), which is unencoded / raw data. Thus, in this embodiment of the present invention, existing array elements of a USAC-type bitstream that would normally carry encoded audio data and / or metadata are instead used to carry unencoded / raw media data.

[0096] Using the above technology, additional media data and information about that additional media data can be easily embedded into an existing MPEG-D USAC-based bitstream that is passed through by the source device.

[0097] Process the modified bitstream on the sink device. Here, referring to the example in Figure 3, we illustrate an example of how to process a modified bitstream on a sink device. In step S200, the modified bitstream, which includes the payload of encoded media data (e.g., received by the source device) and additional media data, may be received by the receiver of the sink device.

[0098] In step S201, the received modified bitstream is parsed by a bitstream parser, and the encoded media data and additional data are analyzed. It can be converted into a data payload. The format of the modified bitstream is not limited, but the modified bitstream may be an MPEG-D USAC-based bitstream (i.e., a bitstream compliant with the MPEG-D USAC standard). Referring to the example in Figure 4, in one embodiment the modified bitstream may include a USAC-type bitstream extension mechanism, and the analysis in step S201 may further be based on identifying the USAC-type bitstream extension mechanism in the modified bitstream. In one embodiment the USAC-type bitstream extension mechanism may be a new USAC-type bitstream extension element that includes a unique identifier.

[0099] Referring to the example in Figure 3 or 4, in step S202, the encoded media data can be core-decoded by the core decoder.

[0100] In step S203, the core-decoded media data is mixed with additional media data by a mixer to obtain an output signal.

[0101] Next, in step 204, the acquired output signal can be output by the control unit.

[0102] Here, referring to the example in Figure 5, a further example of how a modified bitstream is processed on a sink device is illustrated. In the example in Figure 5, in one embodiment, the modified bitstream may further include information about additional media data, and processing the modified bitstream by the sink device may be based on such information. In one embodiment, such information may include one or more of the following: information about the configuration of the additional media data, information about the presence of the additional media data in the modified bitstream, information about the default loudness value, information about the value for delay alignment, information about delay adaptation, information about panning, and information about the type of additional media data.

[0103] In the example in Figure 5, the method may further include step S202a, in which the additional media data is processed by a processing unit before mixing the core-decoded media data and the additional media data. Next, in step S203, the core-decoded media data is mixed with the processed additional media data. In one embodiment, processing the additional media data may include one or more of resampling, delay adaptation, and loudness processing. Processing the additional media data may be guided (based on) generated information contained in the modified bitstream and given by the bitstream parser in step S201 (indicated by dashed lines). For example, resampling may be guided by information about the sampling rate of the additional media data, which may be contained in the information about the configuration of the additional media data. Alternatively, or additionally, processing the additional media data may include downmixing / upmixing of the additional media data, which may be guided by information about the channel mode, which may be contained in the information about the configuration of the additional media data. The woodness treatment was measured according to one of the methods described in ITU-R BS.1770-4. This can be guided by information about default loudness values, which may include noise levels.

[0104] As an alternative or addition, in one embodiment, the modified bitstream may further include information regarding the relative gain between the encoded media data and the additional media data, and may be processed before mixing as needed, and the processed additional media data may be obtained (indicated by a dotted line). Mixing the core-decoded media data and the additional media data in step S203 may be based on the information regarding the relative gain. Here, the sink device may use the relative gain value generated by the source device. In other words, both the source device and the sink device may use the same relative gain value. The relative gain value may also be further modified via user input in the mixing step S203. The relative gain value may include the mixing gain and / or attenuation gain value. The information regarding the relative gain may be provided by the bitstream parser in step S201, as indicated by a dashed line. The information regarding the additional media data and, as an alternative or addition, the information regarding the relative gain, is included in the modified bitstream as described above in system_sound_config It can be embedded within top-level syntax elements.

[0105] In one embodiment, the additional media data may be compressed media data or uncompressed media data.

[0106] Refer to the example in Figure 6. In one embodiment, the supplemental media data may be compressed media data, and the method may further include step S205 of decoding the compressed media data with a decoder to obtain decoded supplemental media data, in step S203 of which the decoded supplemental media data may be mixed with the core decoded media data. If the method further includes step S202a of processing the supplemental media data when the supplemental media data is compressed media data, the decoded supplemental media data may, if necessary, undergo the processing in step S202a, indicated by the dotted line, before being mixed with the core decoded media data. The format of the compressed media data may, but is not limited, belong to the Bluetooth ecosystem. In one embodiment, the compressed media data may be data in SBC format or aptX-based format. Also, if it is compressed media data, the processing in step S202a may be guided by (based on) information about the generated supplemental media data, which is included in the modified bitstream and given in step S201, as indicated by the dashed line.

[0107] Alternatively, or as an addition, the mixing of the core-decoded media data and the decoded additional media data in step S203, which may be processed before mixing as needed, may also be based on the relative gain information described above. The relative gain information may also be given in step S201, as indicated by the dashed line.

[0108] Alternatively, in one embodiment, the additional media data may be uncompressed media data, and in step S203, the uncompressed additional media data may be mixed with the core-decoded media data according to the example illustrated in Figure 3 or Figure 4. According to the example in Figure 5, the uncompressed media data may also undergo the processing in step S202a as necessary, as described above, before being mixed with the core-decoded media data. In one embodiment, the uncompressed media data may be PCM data generated in the source device. In one embodiment, the PCM data may include one or more of earphone data and system sound data.

[0109] Source device Referring here to the example in Figure 7, an example of a source device that generates a modified bitstream is illustrated. Source device 100 may include a receiver 102 configured to receive a bitstream containing encoded media data 101. The bitstream may be received from, for example, one or more media servers. The bitstream may deliver audio content to be played back by, for example, a sink device connected to source device 100. Source device 100 is not limited. For example, source device 100 may be a mobile device including one or more of the following: a mobile phone, a smartwatch, a laptop including a convertible, and a tablet PC.

[0110] In one embodiment, encoded media data can be replayed by a source device, a sink device, or both a source device and a sink device. In one embodiment, supplemental media data can be replayed by a source device, a sink device, or both a source device and a sink device. The source device 100 may further include an embedder 104 configured to generate a payload for supplemental media data 103 and embed the generated payload into a bitstream to obtain a modified bitstream containing the payloads for encoded media data 101 and supplemental media data 103. Here, the modified bitstream can also be generated even when the source device is operating in pass-through mode. The source device 100 may be configured to pass through the encoded media data to the sink device via the modified bitstream. The encoded media data received from the source device is embedded in the modified bitstream and output to the sink device.

[0111] Advantageously, generating a modified bitstream (at the source device) and outputting it to the sink device streamlines the mixing of media data with additional media data at the sink device. A single bitstream is used by the source device to embed the media data and additional media data and transport them to the sink device. This allows for the use of a single channel / connection between the source and sink devices. Furthermore, there are no further decoders and / or encoders (e.g., for decoding and re-encoding encoded media data) at the source device. No further mixers (e.g., for mixing the decoded encoded media data with additional media data) are used at the source device.

[0112] The format of the modified bitstream is not limited, but in one embodiment, the modified bitstream may be an MPEG-D USAC-based bitstream (i.e., a bitstream compliant with the MPEG-D USAC standard). In one embodiment, the embedder 104 may be configured to carry the generated payload in the modified bitstream via a USAC-type bitstream expansion mechanism, as described above.

[0113] In one embodiment, the additional media data 103 may be compressed or uncompressed media data. Uncompressed media data may, in one embodiment, be PCM data generated in the source device 100. The generation of PCM data may, in one embodiment, be based on user input. User input may, for example, be keystrokes. In this case, the source device 100 may include a user interface. The user interface may, for example, be a touch display. Alternatively, or additionally, the source device 100 may further include at least a keyboard. However, the generation of PCM data may also be facilitated in the source device 100 independently of user input, for example, based on email notifications. In one embodiment... The PCM data may include one or more of the earphone data and system sound data. The PCM data may be stored in the source device 100.

[0114] Compressed media data, and its format, may belong to the Bluetooth ecosystem, although this is not limited to the compressed media data itself. In one embodiment, the compressed media data may be data in SBC format or aptX-based format. The compressed media data may be generated in source device 100. The compressed media data may be stored within source device 100. Alternatively, or additionally, the compressed media data may be received via receiver 102.

[0115] Referring here to the example in Figure 8, in one embodiment, the embedder 104 may be further configured to generate information 106 about the additional media data 103 and embed the generated information 106 into the bitstream (indicated by a dashed line). Alternatively, or additionally, in one embodiment, the embedder 104 may be further configured to generate information 106 about the relative gain between the encoded media data 101 and the additional media data 103 and embed the relative gain information 106 into the bitstream (also indicated by a dashed line). In one embodiment, the embedder 104 may be further configured to carry the generated information 106 and / or the generated relative gain information 106 in the modified bitstream via a USAC-type bitstream expansion mechanism, as described above.

[0116] The source device 100 may further include a control unit 105 configured to output the modified bitstream to a sink device.

[0117] sink device Referring here to the example in Figure 9, an example of a sink device for processing a modified bitstream is illustrated. The sink device 200 may include a receiver 201 configured to receive the modified bitstream, which includes a payload of encoded media data and additional media data. The sink device 200 is not limited. The sink device 200 may be a peripheral device of the source device, such as an earphone. The sink device 200 and the source device may be connected via Bluetooth. The sink device 200 may be configured to play audio content contained in the modified bitstream. For example, the sink device 200 may be configured to play encoded media data or additional media data, or both encoded media data and additional media data.

[0118] The sink device 200 may further include a bitstream parser 202 configured to analyze the modified bitstream to obtain a payload of encoded media data and additional media data. The format of the modified bitstream is not limited, but the modified bitstream may be an MPEG-D USAC-based bitstream (i.e., a bitstream compliant with the MPEG-D USAC standard). As already described above, in one embodiment, the bitstream parser 202 may be configured to analyze the bitstream based on identifying USAC-type bitstream extension mechanisms in the modified bitstream.

[0119] The sink device 200 may further include a core decoder 203 configured to core-decode encoded media data and obtain core-decoded media data. In one embodiment, the core decoder 203 may be an MPEG-D USAC based decoder (compliant with the MPEG-D USAC standard described above).

[0120] The sink device 200 may further include a mixer 204 configured to mix core-decoded media data and additional media data to obtain an output signal. In one embodiment, the additional media data may be compressed or uncompressed media data. In one embodiment, the additional media data may be uncompressed media data, and the mixer 204 may be configured to mix core-decoded media data and uncompressed additional media data.

[0121] Referring here to the example in Figure 10, the modified bitstream may further include information about additional media data, and the sink device 200 may be configured to process the modified bitstream based on this information. In one embodiment, the sink device 200 may further include a processing unit 206 configured to process the additional media data before mixing the core-decoded media data and the additional media data in the mixer 204. In one embodiment, processing the additional media data by the processing unit 206 may include one or more of the following: resampling, delay adaptation, and loudness processing. Processing the additional media data by the processing unit 206 may be guided by (based on) the information about the additional media data described above. The information about the additional media data may be provided to the processing unit 206 by the bitstream parser 202, as indicated by the dashed line. If the additional media data is uncompressed media data, the uncompressed media data may be processed directly after the modified bitstream has been parsed in the bitstream parser 202 and before it enters the mixer 204.

[0122] Alternatively, or as an addition, in one embodiment, the modified bitstream may further include information regarding the relative gain between the encoded media data and the additional media data, and the mixer 204 may be further configured to mix the core-decoded media data and the additional media data, which may be processed before mixing as needed (indicated by a dotted line), based on the above relative gain information. The relative gain information may be provided to the mixer 204 by the bitstream parser 202, as indicated by a dashed line.

[0123] In one embodiment, the additional media data may be compressed media data. Referring to the example in Figure 11, the sink device 200 may further include a decoder 207 configured to decode the compressed media data to obtain decoded additional media data, and the mixer 204 may be configured to mix the core-decoded media data and the decoded additional media data. Alternatively, or additionally, the modified bitstream may further include information regarding the relative gain between the encoded media data and the additional media data, and the mixer 204 may be configured to mix the core-decoded media data and the decoded additional media data based on the above relative gain information. The relative gain information may be provided to the mixer 204 by a bitstream parser 202, as indicated by the dashed line.

[0124] Referring further to the example in Figure 12, the decoded additional media data may be further processed by the processing unit 206 (indicated by the dotted line) as needed before being mixed with the core-decoded media data in the mixer 204. Processing of the decoded additional media data by the processing unit 206 may be guided by (based on) information about the additional media data that may be included in the modified bitstream described above. Information about the additional media data may be provided to the processing unit 206 by the bitstream parser 202, as indicated by the dashed line.

[0125] Referring to the examples in Figures 9-12, the sink device 200 is configured to output an output signal. The system may further include a control unit 205. For example, the output signal may be output to the user of the sink device 200 via earphones.

[0126] The above methods may be implemented individually on the above devices. The above devices capable of performing the above methods may also form their own systems. The device 300 referenced herein may include one or more processors 301, 302, as illustrated in Figure 13. Alternatively, or additionally, the above methods may also be implemented on device 300 as a computer program product comprising a computer-readable storage medium having instructions configured to perform the method when executed on device 300 having processing capabilities 301, 302.

[0127] interpretation Unless otherwise specified, as is evident from the following statements, the use of terms such as “process,” “calculate,” “calculate,” “determine,” and “analyze” throughout the disclosed statements is understood to refer to the operation and / or process of a computer or computing system or similar electronic computing device that manipulates and / or converts data expressed as physical quantities, such as electron quantities, into other data similarly expressed as physical quantities.

[0128] Similarly, the term “processor” can refer to any device or any part of a device that processes electronic data, for example, from registers and / or memory, in order to convert such electronic data into other electronic data that can be stored, for example, in registers and / or memory. A “computer,” “computing machine,” or “computing platform” may include one or more processors.

[0129] The methods described herein are executable in one embodiment by one or more processors that receive computer-readable (or machine-readable) code containing a set of instructions that, when executed by one or more of the processors, perform at least one of the methods described herein. This includes any processor capable of executing a set of instructions (sequential or otherwise) that specify the operation to be performed. Thus, an example is a typical processing system comprising one or more processors. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system may further include a memory subsystem including main RAM and / or static RAM, and / or ROM. A bus subsystem may be included for communication between these components. Furthermore, the processing system may be a distributed processing system having processors connected by a network. If the processing system requires a display, such a display may be included, for example, a liquid crystal display (LCD) or a cathode ray tube (CRT). If manual data entry is required, the processing system also includes input devices such as one or more of an alphanumeric input unit, such as a keyboard, and a pointing control device, such as a mouse. The processing system may also include a storage system, such as a disk drive unit. In some configurations, the processing system may include a sound output device and a network interface device. Therefore, the memory subsystem includes a computer-readable carrier medium that carries computer-readable code (e.g., software) containing a set of instructions that, when executed by one or more processors, cause one or more of the methods described herein to be performed. Where the method includes several elements, e.g., several steps, the ordering of such elements is not implied unless otherwise specified. The software may reside in the hard disk, or, during execution by the computer system, may reside entirely or at least partially in RAM and / or within the processor. Thus, the memory and processor also include computer-readable code. It constitutes a computer-readable carrier medium that carries data-readable codes. Furthermore, the computer-readable carrier medium may form a computer program product or be included in a computer program product.

[0130] In alternative embodiments, one or more processors may operate as standalone devices or, in a network configuration, be connected to other processors, for example, via a network connection. One or more processors may operate as a server or user machine in a server-user network environment, or as a peer machine in a peer-to-peer or distributed network environment. One or more processors may form a personal computer (PC), tablet PC, personal digital assistant (PDA), mobile phone, web appliance, network router, switch or bridge, or any machine capable of executing a set of instructions (sequentially or otherwise) that specify the operations to be performed by that machine.

[0131] Furthermore, the term “machine” is also interpreted to include any group of machines that individually or collectively execute one or more sets of instructions for performing any one or more of the methods described herein.

[0132] Accordingly, each embodiment of the method described herein is in the form of a computer-readable carrier medium that carries a set of instructions, for example, a computer program to run on one or more processors, for example, one or more processors that are part of a web server configuration. Accordingly, as will be understood by those skilled in the art, embodiments of the present disclosure may be implemented as a method, an apparatus such as a dedicated device, an apparatus such as a data processing system, or a computer-readable carrier medium, for example, a computer program product. The computer-readable carrier medium carries computer-readable code that includes a set of instructions that cause the processors to implement the method when executed on one or more processors. Accordingly, embodiments of the present disclosure may take the form of a method, an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware embodiments. Furthermore, the present disclosure may take the form of a carrier medium (for example, a computer program product on a computer-readable storage medium) that carries computer-readable program code implemented within the medium.

[0133] The software may also be transmitted and received over a network via a network interface device. In the embodiment, the carrier medium is a single medium, but the term “carrier medium” should be interpreted to include a single or multiple mediums (e.g., a centralized or distributed database, and / or associated caches and servers) that store one or more sets of instructions. The term “carrier medium” should also be interpreted to include any medium that can store, encode, or carry a set of instructions for execution by one or more processors, causing the one or more processors to perform any one or more of the methods of the Disclosure. The carrier medium may take many forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical disks, magnetic disks, and magneto-optical disks. Volatile media include dynamic memory such as main memory. Transmission media include coaxial cables, copper wires, and optical fibers, including wires that constitute a bus subsystem. Transmission media may also take the form of sound waves or light waves, such as those generated during radio and infrared data communications. Therefore, for example, the term "carrier medium" is a computer product implemented in solid-state memory, optical media and magnetic media, a medium detectable by at least one processor or more processors that carries propagating signals representing a set of instructions that implement a method when executed, and a medium detectable by at least one of the one or more processors that carries a set of instructions This is interpreted to include, but is not limited to, transmission media within a network that carry propagation signals representing commands.

[0134] It is understood that, in one embodiment, the steps of the above method are performed by a suitable processor of a processing system (e.g., a computer) that executes instructions (e.g., computer-readable code) stored in storage. It is also understood that this disclosure is not limited to any particular implementation or programming technique, and that this disclosure may be implemented using any suitable technique for implementing the functions described herein. This disclosure is not limited to any particular programming language or operating system.

[0135] Throughout this disclosure, any reference to “one embodiment,” “several embodiments,” or “embodiment” means that any particular feature, structure, or property described in relation to such embodiment is included in at least one embodiment of this disclosure. Therefore, the phrases “in one embodiment,” “several embodiments,” or “in an embodiment” appearing in various places throughout this disclosure do not necessarily all refer to the same embodiment. Furthermore, any particular feature, structure, or property may be combined in one or more embodiments in any suitable manner, as will be apparent to those skilled in the art from this disclosure.

[0136] As used herein, unless otherwise specified, the use of ordinal numbers such as “first,” “second,” and “third” to describe common objects merely indicates that different examples of similar objects are being referred to, and is not intended to imply that the objects described in this manner must be in a given order, whether temporal, spatial, ranking, or any other way.

[0137] In the following claims and in this specification, both the terms “equipment” and “inclusion” are open terms meaning that include at least the elements / features that follow the term, but do not exclude other elements / features. Therefore, the term “equipment,” when used in the claims, should not be interpreted as being limited to the means, elements, or steps (steps) that are subsequently enumerated. For example, the expression “a device comprising A and B” should not be limited to a device consisting only of elements A and B. The term “inclusion,” as used herein, is also an open term meaning that includes at least the elements / features that follow the term, but does not exclude other elements / features. Therefore, “inclusion” is synonymous with and means the same as “equipment.”

[0138] In the above description of the embodiments of this disclosure, please understand that various features of this disclosure may be summarized in a single embodiment, figure, or description thereof in order to simplify the disclosure and to aid in understanding one or more of the various embodiments of the invention. However, the method of this disclosure should not be interpreted as reflecting an intention that the claims require more features than are expressly described in each claim. Rather, as reflected in the following claims, the embodiments of the invention consist of fewer features than all the features of the single embodiment disclosed above. Accordingly, the claims following this specification are expressly incorporated herein, and each claim exists in itself as a separate embodiment of this disclosure.

[0139] Furthermore, some embodiments described herein include some features included in other embodiments and do not include other features, but combinations of features of different embodiments are, as will be understood by those skilled in the art, within the scope of the disclosure and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0140] Numerous specific details are provided in this specification. However, it should be understood that embodiments of this disclosure may be carried out without these specific details. In other examples, well-known methods, structures, and techniques are not described in detail so as not to obscure the understanding of this description.

[0141] Therefore, while we have described what we believe to be the best form of this disclosure, those skilled in the art will recognize that other and further modifications may be made thereto without departing from the spirit of this disclosure, and it is intended that all such variations and modifications will be within the scope of this disclosure. For example, any of the above formulas merely represent the procedures that may be used. Functions may be added to or removed from the block diagrams, and operations may be interchangeable between function blocks. Steps may be added to or removed from the methods described within the scope of this disclosure.

[0142] The following are enumerated example embodiments (EEEs) of this disclosure. do.

[0143] A-EEE1. A method for generating a modified bitstream on a source device, a) The receiver receives a bitstream containing encoded media data, b) A step of obtaining a modified bitstream as output from the embedder, which includes the encoded media data and the payload of the additional media data, by generating a payload of additional media data using an embedder and embedding the payload into the bitstream, c) The step of outputting the modified bitstream to a sink device, A method that includes this.

[0144] A-EEE2. The method according to A-EEE1, further comprising step b) generating information about the additional media data and embedding the information in the bitstream.

[0145] A-EEE3. The method according to A-EEE2, wherein the generated information includes one or more of the following: information about the configuration of the additional media data, information about the presence of the additional media data in the modified bitstream, information about a default loudness value, information about a value for delay alignment, information about delay adaptation, information about panning, and information about the type of the additional media data.

[0146] A-EEE4. The method according to any one of A-EEE1 to A-3, further comprising step b) generating information regarding the relative gain between the encoded media data and the additional media data, and embedding the information regarding the relative gain in the bitstream.

[0147] A-EEE5. The modified bitstream is an MPEG-D USAC-based bitstream, as described in any of A-EEE1-4.

[0148] A-EEE6. The method according to A-EEE5, wherein the generated payload is embedded in the bitstream by transporting the payload in the bitstream via a USAC-type bitstream expansion mechanism.

[0149] A-EEE7. Regarding the generated information and / or the generated relative gain The method according to A-EEE5 or 6, wherein the information is embedded in the bitstream by transporting the generated information and / or the information regarding the generated relative gain in the bitstream via the USAC-type bitstream expansion mechanism.

[0150] A-EEE8. The method according to A-EEE6 or 7, wherein the USAC-type bitstream extension mechanism is a new USAC-type bitstream extension element containing a unique identifier.

[0151] A-EEE9. In step c), the modified bitstream is output to the sink device via a Bluetooth connection, according to any of A-EEE1 to A-8.

[0152] A-EEE10. The method according to any one of A-EEE1 to A-9, wherein the additional media data is compressed media data or uncompressed media data.

[0153] A-EEE11. The method according to A-EEE10, wherein the uncompressed media data is PCM data generated in the source device.

[0154] A-EEE12. The generation of the PCM data is based on user input, as described in A-EEE11.

[0155] A-EEE13. The method according to A-EEE11 or 12, wherein the PCM data includes one or more of the earphone data and system sound data.

[0156] A-EEE14. The method according to A-EEE10, wherein the compressed data is data in SBC format or aptX-based format.

[0157] A-EEE15. A method for processing a modified bitstream on a sink device, a) The receiver receives a modified bitstream, which includes a payload of encoded media data and additional media data. b) A bitstream parser analyzes the modified bitstream to obtain the payload of the encoded media data and the additional media data, c) The step of obtaining the core-decoded media data by core decoding the encoded media data using a core decoder, d) A step of mixing the core-decoded media data and the additional media data with a mixer to obtain an output signal, e) A step of outputting the output signal, A method that includes this.

[0158] A-EEE16. The method according to A-EEE15, wherein the modified bitstream further includes information about the additional media data, and the method further includes the step of processing the modified bitstream based on the information after step a) and before step b).

[0159] A-EEE17. The information includes information about the configuration of the additional media data, information about the presence of the additional media data in the modified bitstream, information about the default loudness value, information about the value for delay alignment, information about delay adaptation, information about panning, and the additional media data The method described in A-EEE16, which includes one or more pieces of information about the type of data.

[0160] A-EEE18. The modified bitstream further includes information regarding the relative gain between the encoded media data and the additional media data, and the mixing of the core-decoded media data and the additional media data in step d) is performed according to any of A-EEE15 to 17, based on the information regarding the relative gain.

[0161] A-EEE19. The method according to any one of A-EEE15 to 18, further comprising the step of processing the additional media data by a processing unit before mixing the core-decoded media data and the additional media data.

[0162] A-EEE20. The method according to A-EEE19, wherein the step of processing the additional media data includes one or more of resampling, delay adaptation, and loudness processing.

[0163] A-EEE21. The method according to any one of A-EEE15 to 20, wherein the additional media data is compressed media data, and the method further includes the step of decoding the compressed media data with a decoder to obtain decoded additional media data, and in step d), the decoded additional media data is mixed with the core decoded media data.

[0164] A-EEE22. The method according to A-EEE21, wherein the compressed media data is data in SBC format or aptX-based format.

[0165] A-EEE23. The method according to any one of A-EEE15 to 20, wherein the additional media data is uncompressed media data, and in step d), the uncompressed additional media data is mixed with the core-decoded media data.

[0166] A-EEE24. The modified bitstream is an MPEG-D USAC-based bitstream, as described in any of A-EEE15-23.

[0167] A-EEE25. The method according to A-EEE24, wherein the modified bitstream includes a USAC-type bitstream extension mechanism, and the analysis in step b) is further based on identifying the USAC-type bitstream extension mechanism in the modified bitstream.

[0168] A-EEE26. The method according to A-EEE25, wherein the USAC-type bitstream extension mechanism is a new USAC-type bitstream extension element containing a unique identifier.

[0169] A-EEE27. A source device that generates a modified bitstream, a) A receiver configured to receive a bitstream containing encoded media data, b) An embedder configured to generate a payload of additional media data, embed the generated payload into the bitstream, and obtain a modified bitstream containing the encoded media data and the payload of the additional media data, c) A control unit that outputs the modified bitstream to a sink device, A device that includes this.

[0170] A-EEE28. The device according to A-EEE27, wherein the embedder is further configured to generate information about the additional media data and embed the information in the bitstream.

[0171] A-EEE29. The device according to A-EEE27 or 28, wherein the embedder is further configured to generate information regarding the relative gain between the encoded media data and the added media data, and to embed the information regarding the relative gain in the bitstream.

[0172] A-EEE30. The modified bitstream is an MPEG-D USAC-based bitstream, as described in any of the devices listed in A-EEE27-29.

[0173] A-EEE31. The device according to any one of A-EEE27 to 30, wherein the additional media data is compressed media data or uncompressed media data.

[0174] A-EEE32. The device described in A-EEE31, wherein the uncompressed media data is PCM data generated in the source device.

[0175] A-EEE33. The device described in A-EEE32 generates the PCM data based on user input.

[0176] A-EEE34. The device according to A-EEE32 or 33, wherein the PCM data includes one or more of earphone data and system sound data.

[0177] A-EEE35. The device described in A-EEE31, wherein the compressed media data is data in SBC format or aptX-based format.

[0178] A-EEE36. A sink device for processing a modified bitstream, a) A receiver configured to receive a modified bitstream, which includes a payload of encoded media data and additional media data, b) A bitstream parser configured to analyze the modified bitstream to obtain the payload of the encoded media data and the additional media data, c) A core decoder configured to core-decode the encoded media data and obtain the core-decoded media data, d) A mixer configured to mix the core-decoded media data and the additional media data to obtain an output signal, e) A control unit configured to output the output signal, A device that includes this.

[0179] A-EEE37. The device according to A-EEE36, further comprising a processing unit configured to process the additional media data before mixing the core-decoded media data and the additional media data.

[0180] A-EEE38. The device according to A-EEE36 or 37, wherein the additional media data is uncompressed media data, and the mixer is configured to mix the core-decoded media data and the uncompressed additional media data.

[0181] A-EEE39. The additional media data is compressed media data, and the device decodes the compressed media data and takes the decoded additional media data. The device according to A-EEE36 or 37, further comprising a decoder, wherein the mixer is configured to mix the core-decoded media data and the decoded additional media data.

[0182] A-EEE40. The device according to any one of A-EEE36 to 39, wherein the modified bitstream further includes information about the additional media data, and the sink device is configured to process the modified bitstream based on the information.

[0183] A-EEE41. The device according to any one of A-EEE36-40, wherein the modified bitstream further includes information regarding the relative gain between the encoded media data and the additional media data, and the mixer is further configured to mix the core-decoded media data and the additional media data based on the information regarding the relative gain.

[0184] A-EEE42. The core decoder is an MPEG-D USAC-based decoder, as described in any of A-EEE36-41.

[0185] A-EEE43. A system comprising a source device that generates a modified bitstream and a sink device that processes the modified bitstream, wherein the source device includes one or more processors configured to perform the methods described in A-EEE1 to A-14, and the sink device includes one or more processors configured to perform the methods described in A-EEE15 to A-26.

[0186] A-EEE44. A computer program product comprising a computer-readable storage medium having instructions configured to cause a device having processing capabilities to perform any one of the methods described in A-EEE1 to A-EEE14 when executed by the device.

[0187] A-EEE45. A computer program product comprising a computer-readable storage medium having instructions configured to cause a device having processing capabilities to perform any one of the methods described in A-EEE15 to A-EEE26 when executed by the device.

[0188] B-EEE1. A method for generating a modified bitstream on a source device, a) The receiver receives a bitstream containing encoded media data, b) A step of obtaining a modified bitstream as output from the embedder, which includes the encoded media data and the payload of the additional media data, by generating a payload of additional media data using an embedder and embedding the payload into the bitstream, c) The step of outputting the modified bitstream to a sink device, A method that includes this.

[0189] B-EEE2. Step b) further includes generating information about the additional media data and embedding the information in the bitstream, and if necessary, The method according to B-EEE1, wherein the generated information includes one or more of the following: information about the configuration of the additional media data, information about the presence of the additional media data in the modified bitstream, information about a default loudness value, information about a value for delay alignment, information about delay adaptation, information about panning, and information about the type of the additional media data.

[0190] B-EEE3. The method according to B-EEE1, further comprising step b) generating information regarding the relative gain between the encoded media data and the additional media data, and embedding the information regarding the relative gain in the bitstream.

[0191] B-EEE4. The modified bitstream is an MPEG-D USAC-based bitstream, and The method according to B-EEE1, wherein the generated payload is embedded in the bitstream by transporting the payload in the bitstream via a USAC-type bitstream expansion mechanism.

[0192] B-EEE5. The modified bitstream is an MPEG-D USAC-based bitstream, and The method of B-EEE3, wherein the generated information and / or the information regarding the generated relative gain is embedded in the bitstream by transporting the generated information and / or the information regarding the generated relative gain in the bitstream via the USAC-type bitstream expansion mechanism.

[0193] B-EEE6. The method according to B-EEE4, wherein the USAC-type bitstream extension mechanism is a new USAC-type bitstream extension element containing a unique identifier.

[0194] B-EEE7. The method according to B-EEE1, wherein in step c), the modified bitstream is output to the sink device via a Bluetooth connection.

[0195] B-EEE8. The method according to B-EEE1, wherein the additional media data is PCM data generated in the source device.

[0196] B-EEE9. The method according to B-EEE8, wherein the PCM data includes one or more of the earphone data and system sound data.

[0197] B-EEE10. The method according to B-EEE1, wherein the additional media data is compressed data in SBC format or aptX-based format.

[0198] B-EEE11. A method for processing a modified bitstream on a sink device, a) The receiver receives a modified bitstream, which includes a payload of encoded media data and additional media data. b) A bitstream parser analyzes the modified bitstream to obtain the payload of the encoded media data and the additional media data, c) The step of obtaining the core-decoded media data by core decoding the encoded media data using a core decoder, d) A step of mixing the core-decoded media data and the additional media data with a mixer to obtain an output signal, e) A step of outputting the output signal, A method that includes this.

[0199] B-EEE12. The modified bitstream further includes information regarding the relative gain between the encoded media data and the additional media data, and step The mixing of the core-decoded media data and the additional media data in d) is the method according to B-EEE11, based on the information regarding the relative gain.

[0200] B-EEE13. The method further includes the step of processing the additional media data by a processing unit before mixing the core-decoded media data and the additional media data, and if necessary, The method according to B-EEE11, wherein the step of processing the additional media data includes one or more of resampling, delay adaptation, and loudness processing.

[0201] B-EEE14. The modified bitstream is an MPEG-D USAC-based bitstream, and The method according to B-EEE11, wherein the modified bitstream includes a USAC-type bitstream extension mechanism, and the analysis in step b) is further based on identifying the USAC-type bitstream extension mechanism in the modified bitstream.

[0202] B-EEE15. The method according to B-EEE11, wherein the USAC-type bitstream extension mechanism is a new USAC-type bitstream extension element that includes a unique identifier.

[0203] B-EEE16. A source device that generates a modified bitstream, a) A receiver configured to receive a bitstream containing encoded media data, b) An embedder configured to generate a payload of additional media data, embed the generated payload into the bitstream, and obtain a modified bitstream containing the encoded media data and the payload of the additional media data, c) A control unit that outputs the modified bitstream to a sink device, A device that includes this.

[0204] B-EEE17. A sink device for processing a modified bitstream, a) A receiver configured to receive a modified bitstream, which includes a payload of encoded media data and additional media data, b) A bitstream parser configured to analyze the modified bitstream to obtain the payload of the encoded media data and the additional media data, c) A core decoder configured to core-decode the encoded media data and obtain the core-decoded media data, d) A mixer configured to mix the core-decoded media data and the additional media data to obtain an output signal, e) A control unit configured to output the output signal, A device that includes this.

[0205] B-EEE18. A system comprising a source device that generates a modified bitstream and a sink device that processes the modified bitstream, wherein the source device includes one or more processors configured to perform the method described in B-EEE1, and the sink device includes one or more processors configured to perform the method described in B-EEE11.

[0206] B-EEE19. A computer having instructions configured to cause a device having processing power to perform the method described in B-EEE1 when executed by said device. A computer program product equipped with a readable storage medium.

[0207] B-EEE20. A computer program product comprising a computer-readable storage medium having instructions configured to cause a device having processing capabilities to perform the method described in B-EEE11 when executed by the device.

Claims

1. A method for generating a modified MPEG-D USAC-based bitstream by a source device comprising a receiver and an embedder, a) The receiver receives a bitstream containing encoded media data, b) A step of obtaining a modified bitstream as output from the embedder, which includes the encoded media data and the payload of the additional media data, by generating a payload of additional media data using the embedder and embedding the payload in the bitstream, wherein the generated payload is embedded in the bitstream by transporting the payload in the bitstream via a USAC-type bitstream expansion mechanism, the USAC-type bitstream expansion mechanism is a USAC-type bitstream expansion element or array expansion element including a unique identifier, and the additional media data is uncompressed audio data. c) The step of outputting the modified bitstream to a sink device, A method that includes this.

2. The method according to claim 1, further comprising step b) generating information about the additional media data and embedding the information in the bitstream.

3. The method according to claim 2, wherein the generated information includes one or more of the following: information about the configuration of the additional media data; information about the presence of the additional media data in the modified bitstream; information about a default loudness value; information about a value for delay alignment; information about delay adaptation; information about panning; and information about the type of the additional media data.

4. The method according to claim 1, further comprising step b) generating information relating to the relative gain between the encoded media data and the added media data, and embedding the information relating to the relative gain in the bitstream.

5. The method according to claim 4, wherein the generated information and / or the generated relative gain information are embedded in the bitstream by transporting the generated information and / or the generated relative gain information in the bitstream via the USAC type bitstream expansion mechanism.

6. The method according to claim 1, wherein in step c), the modified bitstream is output to the sink device via a Bluetooth connection.

7. A method for processing a modified MPEG-D USAC-based bitstream using a sink device comprising a receiver, a bitstream parser, a core decoder, and a mixer, a) The receiver receives a modified bitstream comprising encoded media data, an additional media data payload, and a USAC-type bitstream extension mechanism, wherein the USAC-type bitstream extension mechanism is a new USAC-type bitstream extension element or array extension element containing a unique identifier, and the additional media data is uncompressed audio data. b) A step of analyzing the modified bitstream with the bitstream parser to obtain the payload of the encoded media data and the additional media data, wherein the analysis is based on identifying the USAC type bitstream extension mechanism in the modified bitstream. c) The step of obtaining the core-decoded media data by core-decoding the encoded media data using the core decoder, d) The step of mixing the core-decoded media data and the additional media data by the mixer to obtain an output signal, e) The step of outputting the output signal, A method that includes this.

8. The method according to claim 7, wherein the modified bitstream further includes information about the additional media data, and the method further includes the step of processing the modified bitstream based on the information after step a) and before step b).

9. The method according to claim 8, wherein the information includes one or more of the following: information about the configuration of the additional media data; information about the presence of the additional media data in the modified bitstream; information about a default loudness value; information about a value for delay alignment; information about delay adaptation; information about panning; and information about the type of the additional media data.

10. The method according to claim 7, wherein the modified bitstream further includes information regarding the relative gain between the encoded media data and the additional media data, and the mixing of the core-decoded media data and the additional media data in step d) is based on the information regarding the relative gain.

11. The method according to claim 7, further comprising the step of processing the additional media data by a processing unit before mixing the core-decoded media data and the additional media data.

12. The method according to claim 11, wherein the step of processing the additional media data includes one or more of resampling, delay adaptation, and loudness processing.

13. A computer program having instructions configured to cause a device having processing capabilities to perform the method according to any one of claims 1 to 12 when executed by the device.

Images