Method and apparatus for processing audio data - Patents.com
The method and system for dynamic metadata-based audio processing address real-time encoding challenges by adjusting loudness and range, ensuring consistent and optimized playback experiences.
Patent Information
- Application Number
- JP2024534300
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-12-16
- Filing Date
- 2022-08-24
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-08-24
AI Technical Summary
Existing audio playback systems struggle to dynamically adjust loudness and dynamic range in real-time encoding scenarios, leading to inconsistent playback experiences and potential deviations from intended audio content characteristics.
A method and system for dynamic metadata-based processing of audio data, involving encoders and decoders that utilize metadata to determine processing parameters for loudness adjustment and dynamic range compression, allowing for real-time adjustments based on playback conditions.
Enables consistent and optimized audio playback across various environments by dynamically adjusting loudness and range, preserving the intended audio characteristics and ensuring compliance with regulations.
Smart Images

Figure 2024543726000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure generally relates to a method for performing metadata-based dynamic processing on audio data to be reproduced, and in particular to a method for determining and applying one or more processing parameters to audio data to be subjected to dynamic loudness adjustment and / or dynamic range compression. The present disclosure further relates to a method for encoding audio data to be subjected to dynamic loudness adjustment and / or dynamic range compression and metadata into a bitstream. The present disclosure further relates to respective decoders and encoders and respective systems and computer program products. The present disclosure further relates to a method for processing audio data to be reproduced, a decoder for processing audio data to be reproduced, and respective computer program products.
[0002] Although certain embodiments are described herein with particular reference to the above disclosure, it should be understood that the present disclosure is not limited to such fields of use, but is applicable in a broader context. [Background technology]
[0003] Although background art is described throughout this disclosure, no description thereof should be construed as an admission that such art is well known or forms part of the common general knowledge in the art.
[0004] When reproducing audio content, loudness is the sound pressure felt by an individual human being. In film or television content, it has been found that the most important parameter determining what a listener perceives as the loudness of a program is the loudness of the dialogue within the program.
[0005] To determine the average loudness of a program, the entire program or only the dialogue within the program needs to be analyzed. Average loudness is typically required to comply with loudness regulations (e.g., CALM Act in the United States) and is also used to align Dynamic Range Control (DRC) parameters. The dynamic range of a program is the difference between the quietest and loudest sounds. The dynamic range of a program varies depending on its content, for example, an action movie may have a different, wider dynamic range than a documentary. The dynamic range of a program reflects the creator's intention. However, the capabilities of devices to reproduce audio content in its original dynamic range vary. Therefore, in addition to loudness management, dynamic range control becomes an additional key factor to provide an optimal listening experience.
[0006] For loudness management and dynamic range control, the entire audio program or a segment of the audio program must be analyzed, and the resulting loudness and DRC parameters can be supplied together with the audio data or encoded audio data to be applied to a decoder or playback device.
[0007] When the entire audio program or a segment of an audio program cannot be analyzed prior to encoding, e.g., in the case of real-time (dynamic) encoding, loudness processing or leveling is used to ensure compliance with loudness regulations and, where applicable, constraints on the potential dynamic range, depending on the playback conditions. This approach delivers processed audio that is "optimized" for a single playback environment. Summary of the Invention [Problem to be solved by the invention]
[0008] Therefore, there is a need for metadata-based processing that provides the "original" unprocessed audio along with metadata, allowing the playback device to use the metadata to dynamically modify the audio depending on device constraints or user requirements. [Means for solving the problem]
[0009] According to a first aspect of the present disclosure, there is provided a method for performing metadata-based dynamic processing on audio data to be reproduced. The method may include receiving, by a decoder, a bitstream including audio data to be subjected to dynamic loudness adjustment and metadata. The method may further include obtaining decoded audio data and metadata by decoding, by the decoder, the audio data and the metadata. The method may further include determining, by the decoder, one or more processing parameters for dynamic loudness adjustment from the metadata based on a reproduction condition. The method may further include applying, to the decoded audio data, the determined one or more processing parameters, to obtain processed audio data. And the method may further include outputting the processed audio data for reproduction.
[0010] The metadata for dynamic loudness adjustment may include multiple sets of metadata, each set corresponding to a respective (e.g., different) playback condition. Then, determining from the metadata one or more processing parameters for dynamic loudness adjustment based on a (particular) playback condition may include selecting a set of metadata corresponding to the (particular) playback condition in response to playback condition information provided to the decoder, and extracting one or more processing parameters for dynamic loudness adjustment from the selected set of metadata. In this case, the playback condition information may indicate the (particular) playback condition, or information derived therefrom.
[0011] In some embodiments, the metadata may indicate processing parameters for multiple playback conditions for dynamic loudness adjustment.
[0012] In some embodiments, determining the one or more processing parameters may further include determining one or more processing parameters for dynamic range compression (DRC) based on playback conditions.
[0013] In some embodiments, the playback condition information may indicate a particular loudspeaker setting. In general, the playback conditions may include one or more of the following: a device type of the decoder, playback device characteristics, loudspeaker characteristics, loudspeaker settings, background noise characteristics, ambient noise characteristics, acoustic environment characteristics.
[0014] In some embodiments, the selected set of metadata may include a set of DRC sequences, DRCSet. Furthermore, each set of metadata may include a respective set of DRC sequences, DRCSet. In general, it can be said that determining the one or more processing parameters further includes selecting, by the decoder, at least one of a set of DRC sequences, DRCSet, a set of equalizer parameters, EQSet, and a downmix, which corresponds to the playback condition.
[0015] In some embodiments, determining the one or more processing parameters may further include determining the one or more processing parameters from the metadata by identifying a metadata identifier indicative of the selected at least one of a DRCSet, an EQSet, and a downmix. Specifically, selecting the set of metadata may include identifying a set of metadata corresponding to a particular downmix. The particular downmix may be determined based on a configuration of the loudspeakers.
[0016] In some embodiments, the metadata may include one or more processing parameters related to an average loudness value and, optionally, one or more processing parameters related to dynamic range compression characteristics. Specifically, each set of metadata may include one or more processing parameters related to the average loudness value and, optionally, one or more processing parameters related to dynamic range compression characteristics.
[0017] In some embodiments, the bitstream may further include additional metadata for static loudness adjustments to be applied to the decoded audio data.
[0018] In some embodiments, the bitstream may be an MPEG-D DRC bitstream, and the presence of metadata may be signaled based on MPEG-D DRC bitstream syntax.
[0019] In some embodiments, a loudnessInfoSetExtension()-element may be used to carry the metadata as a payload.
[0020] In some embodiments, the metadata may include one or more metadata payloads, each of which may include a plurality of sets of parameters and identifiers, each set including at least one of a DRCSet identifier drcSetId, an EQSet identifier eqSetId, and a downmix identifier downmixId in combination with one or more processing parameters associated with the identifiers in the set.
[0021] In some embodiments, determining the one or more processing parameters may include selecting a set from the multiple sets in the payload based on the at least one of a DRCSet, an EQSet, and a downmix selected by the decoder, and the one or more processing parameters determined by the decoder may be the one or more processing parameters associated with the identifier in the selected set.
[0022] According to a second aspect of the present disclosure, there is provided a decoder for performing metadata-based dynamic processing on audio data to be reproduced, which may include one or more processors and non-transitory memory configured to perform a method including the steps of receiving, by a decoder, a bitstream including audio data and metadata to be subjected to dynamic loudness adjustment, decoding, by the decoder, the audio data and the metadata to obtain decoded audio data and metadata, determining, by the decoder, one or more processing parameters for dynamic loudness adjustment from the metadata based on a reproduction condition, applying the determined one or more processing parameters to the decoded audio data to obtain processed audio data, and outputting the processed audio data for reproduction.
[0023] The metadata for dynamic loudness adjustment may include multiple sets of metadata, each set corresponding to a respective (e.g., different) playback condition. Then, determining from the metadata one or more processing parameters for dynamic loudness adjustment based on a (particular) playback condition may include selecting a set of metadata corresponding to the (particular) playback condition in response to playback condition information provided to the decoder, and extracting one or more processing parameters for dynamic loudness adjustment from the selected set of metadata. In this case, the playback condition information may indicate the (particular) playback condition, or information derived therefrom.
[0024] According to a third aspect of the present disclosure, there is provided a method for encoding audio data to be subjected to dynamic loudness adjustment and metadata into a bitstream. The method may include inputting original audio data into a loudness leveller for loudness processing, thereby obtaining loudness-processed audio data as an output from the loudness leveller. The method may further include generating metadata for dynamic loudness adjustment based on the loudness-processed audio data and the original audio data. And the method may include encoding the original audio data and the metadata into the bitstream.
[0025] In some embodiments, the metadata may include multiple sets of metadata, each set of metadata may correspond to a respective (e.g., different) playback condition.
[0026] In some embodiments, the method may further include generating additional metadata for static loudness adjustment to be used by the decoder.
[0027] In some embodiments, generating the metadata may include comparing the loudness processed audio data and the original audio data, and the metadata may be generated based on a result of the comparison.
[0028] In some embodiments, generating the metadata may further include measuring loudness over one or more predetermined periods, and the metadata may be generated further based on the measured loudness.
[0029] In some embodiments, the measuring may include measuring the loudness of the entire audio data.
[0030] In some embodiments, the measuring may include measuring the loudness of speech in the audio data.
[0031] In some embodiments, the bitstream may be an MPEG-D DRC bitstream, and the presence of metadata may be signaled based on MPEG-D DRC bitstream syntax.
[0032] In some embodiments, a loudnessInfoSetExtension()-element may be used to carry the metadata as a payload.
[0033] In some embodiments, the metadata may include one or more metadata payloads, each of which may include a plurality of sets of parameters and identifiers, each set including at least one of a DRCSet identifier drcSetId, an EQSet identifier eqSetId, and a downmix identifier downmixId in combination with the one or more processing parameters associated with the identifier in the set, and the one or more processing parameters may be parameters for dynamic loudness adjustment by a decoder.
[0034] In some embodiments, the at least one of the drcSetId, the eqSetId, and the downmixId may relate to at least one of a DRCSet, which is a set of DRC sequences, an EQSet, which is a set of equalizer parameters, and a downmix to be selected by the decoder.
[0035] According to a fourth aspect of the present disclosure, there is provided an encoder for encoding original audio data to be subjected to dynamic loudness adjustment and metadata in a bitstream, which may include one or more processors and a non-transitory memory configured to execute a method including: inputting original audio data to a loudness leveller for loudness processing to obtain loudness processed audio data as output from the loudness leveller, generating metadata for the dynamic loudness adjustment based on the loudness processed audio data and the original audio data, and encoding the original audio data and the metadata into the bitstream.
[0036] According to a fifth aspect of the present disclosure, there is provided a system including an encoder for encoding original audio data and metadata to be subjected to dynamic loudness adjustment in a bitstream, and a decoder for applying metadata-based dynamic processing to the audio data to be reproduced.
[0037] According to a sixth aspect of the present disclosure, there is provided a computer program product including a computer-readable storage medium having instructions configured, when executed by a device having processing capabilities, to cause the device to perform a method of applying metadata-based dynamic processing to audio data to be reproduced or a method of encoding audio data and metadata to be subjected to dynamic loudness adjustment into a bitstream.
[0038] According to a seventh aspect of the present disclosure, there is provided a computer readable storage medium storing a computer program product as described herein.
[0039] According to an eighth aspect of the present disclosure, there is provided a method of processing audio data to be reproduced. The method may include receiving, by a decoder, a bitstream including encoded audio data and metadata. The metadata includes one or more Dynamic Range Control (DRC) sets, and for each of the DRC sets includes an indication of whether the DRC set is configured to provide a dynamic loudness compensation effect. The method may further include identifying a DRC set configured to provide the dynamic loudness compensation effect by analyzing the metadata by the decoder. The method may further include obtaining decoded audio data by decoding the encoded audio data by the decoder. The method may further include selecting, by the decoder, one of the identified DRC sets configured to provide the dynamic loudness compensation effect. The method may further include applying, by the decoder, the one or more DRC gains corresponding to the selected DRC set to the decoded audio data to obtain dynamic loudness compensated audio data. And the method may further include outputting the dynamic loudness compensated audio data for reproduction.
[0040] In some embodiments, the metadata may include a plurality of DRC sets configured to provide dynamic loudness adjustment, each of the plurality of DRC sets may be further associated with one or more playback conditions, and the selection may be made in response to an indication of a playback condition provided to the decoder.
[0041] In some embodiments, the one or more DRC sets may be configured to provide dynamic range control in addition to providing dynamic loudness compensation effects.
[0042] In some embodiments, the playback conditions may include one or more of the device type of the decoder, playback device characteristics, loudspeaker characteristics, loudspeaker settings, background noise characteristics, ambient noise characteristics, and acoustic environment characteristics.
[0043] In some embodiments, an indication of whether the DRC set is configured to provide the dynamic loudness compensation effect may be provided in parameters indicating one or more effects that the DRC set provides.
[0044] In some embodiments, the parameter indicating one or more effects provided by the DRC set may be a drcSetEffect bitfield in an MEPG-D DRC bitstream, with each bit of the drcSetEffect bitfield corresponding to a different effect, and one of the bits of the drcSetEffect bitfield corresponding to the dynamic loudness compensation effect.
[0045] In some embodiments, the indication of whether the DRC set is configured to provide the dynamic loudness compensation effect may be whether the DRC set is specified within a dynamic loudness compensation bitstream payload.
[0046] In some embodiments, the dynamic loudness compensation bitstream payload may be included in an extension field of an already defined bitstream syntax.
[0047] In some embodiments, the extension field may be a uniDrcConfigExtension field of a MEPG-D DRC bitstream, and the dynamic loudness compensation bitstream payload may only be included for certain values of the uniDrcConfigExtType parameter.
[0048] In some embodiments, multiple dynamic loudness compensation payloads specifying multiple DRC sets configured to provide the dynamic loudness compensation effect may be included in an extension field of the already defined bitstream syntax.
[0049] In some embodiments, the indication of whether the DRC set is configured to provide the dynamic loudness compensation effect may be a field of a pre-existing configuration element of an already defined bitstream syntax.
[0050] In some embodiments, the field may be a dynamicLoudCompDRCSet parameter and the already-existing configuration element may be a downmixInstructions element, a drcInstructionsBasic element or a drcInstructionsUniDRC element of an MPEG-D DRC bitstream.
[0051] In some embodiments, the field may be a pre-existing field that is reserved for future use.
[0052] In some embodiments, the indication of whether the DRC set is configured to provide the dynamic loudness compensation effect may be a field in an updated version of a pre-existing configuration element of an already defined bitstream syntax.
[0053] In some embodiments, the field may be a dynamicLoudCompDRCSet parameter and the updated version of the pre-existing configuration element may be a downmixInstructionsV2 element or a drcInstructionsUniDrcV2 element.
[0054] In some embodiments, an indication that a dynamic loudness compensation effect is desired may be provided to the decoder via an interface, and the DRC set may be selected in response to the indication provided to the decoder via the interface.
[0055] In some embodiments, an indication regarding additional desired effects may be provided to the decoder via the interface, and the metadata may include a plurality of DRC sets configured to provide the dynamic loudness compensation effect, and the selection may depend on the additional desired effect.
[0056] In some embodiments, the indication that a dynamic loudness compensation effect is desired may be provided via a drcEffectTypeRequest parameter in the dynamicRangeControllerInterface payload.
[0057] In some embodiments, the metadata may include one or more static loudness values configured to provide a static loudness adjustment to the decoded audio data.
[0058] In some embodiments, in response to one or more of the static loudness values, the static loudness adjustment may be applied to the decoded audio data or the dynamic loudness-compensated audio data.
[0059] In some embodiments, a first one of the DRC sets may be configured to provide dynamic range control, and the first DRC set may include an indication that the selected DRC set, the DRC set configured to provide the dynamic loudness compensation effect, may be configured to be applied in combination with the first DRC set.
[0060] In some embodiments, the selected DRC set may include an indication of whether the one or more DRC gains corresponding to the selected DRC set may be applied only in combination with a DRC gain corresponding to the first DRC set.
[0061] In some embodiments, the DRC gains corresponding to the first DRC set may be extracted from the bitstream and applied to the decoded audio data.
[0062] According to a ninth aspect of the present disclosure, there is provided a decoder for processing audio data to be reproduced, the decoder may include one or more processors and a non-transitory memory configured to perform the method for processing audio data to be reproduced as described above.
[0063] According to a tenth aspect of the present disclosure, there is provided a computer program product including a computer-readable storage medium having instructions configured to cause a device to perform the method for processing audio data to be played as described above.
[0064] According to an eleventh aspect of the present disclosure, there is provided a computer-readable storage medium storing a computer program product as described herein.
[0065] Exemplary embodiments of the present disclosure will now be described, by way of example only, with reference to the accompanying drawings, in which: [Brief description of the drawings]
[0066] [Figure 1] FIG. 2 illustrates an example of a decoder for performing metadata-based dynamic processing on audio data to be reproduced; [Diagram 2] FIG. 2 illustrates an example of a method for performing metadata-based dynamic processing on audio data to be reproduced. [Diagram 3] FIG. 2 illustrates an example of an encoder for encoding original audio data and metadata to be subjected to dynamic loudness adjustment in a bitstream. [Figure 4] FIG. 2 illustrates an example of a method for encoding audio data and metadata to be subjected to dynamic loudness adjustment into a bitstream. [Diagram 5] FIG. 1 illustrates an example of a device including one or more processors and non-transitory memory configured to perform the methods described herein. [Figure 6] FIG. 2 illustrates an example of a method for processing audio data to be played back. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0067] (overview) The average loudness of a program or dialogue is the main parameter or value used for loudness regulatory compliance of broadcast or streaming programs. The average loudness is typically set to a maximum of 24 or 23 LKFS. For audio codecs that support loudness metadata, this single loudness value, which represents the loudness of the entire program, is carried in the bitstream. This value is used in the decoding process to allow gain adjustments, resulting in a predictable playback level, so that the program is played back at a known and consistent level. Therefore, it is important that this loudness value is set properly and accurately. The average loudness relies on measuring the entire program before encoding. However, this is not possible in real-time situations such as dynamic encoding with unknown loudness and changing dynamic range.
[0068] When it is not possible to measure the loudness of the entire file before encoding, dynamic loudness levelers are often used to obtain the required loudness by modifying or contouring the audio data before encoding. This type of loudness management is often seen as an inferior method for regulatory compliance because it often changes the dynamic range relationships within the audio content and therefore may change the creative intent. This is especially true when it is desired to deliver one audio asset to all playback devices. Such delivery is one of the advantages of metadata-driven codecs and delivery systems.
[0069] In some approaches, the audio content is mixed with a required target loudness and the corresponding loudness metadata is set to that value. A loudness leveller may still be used in such situations, as it helps to move the audio content towards the target loudness. However, the loudness leveller is less "active" and is only used when the audio content starts to deviate from the required target loudness.
[0070] In view of the above, the method and apparatus described herein aims to create a real-time processing situation, also called a dynamic processing situation, which is further metadata-driven. The metadata enables dynamic loudness adjustment and dynamic range compression in the real-time situation. The above method and apparatus have the advantage of enabling: Real-time loudness adjustment and use of DRC in MPEG-D DRC syntax - Real-time loudness adjustment and use of DRC in combination with downmixId - Real-time loudness adjustment and using DRC in combination with drcSetId - Using DRC in combination with real-time loudness adjustment and eqSetId
[0071] That is, depending on the decoder settings (e.g., DRCSet, EQSet, and downmix), the decoder can search a given payload for an appropriate set of parameters and identifiers based on the syntax by matching the identifiers with the settings. The parameters included in the set with the identifier that best matches the settings are then selected as processing parameters for dynamic loudness adjustment and applied to the received original audio data to perform the correction.
[0072] Additionally, multiple sets of parameters for dynamic processing (multiple instances of dynLoudCompValue) can be sent.
[0073] In addition to correcting the overall loudness, metadata-driven dynamic loudness compensation can also be used to "center" the calculation and application of DRC gains. Centering can be a result of correcting the loudness of content via dynamic loudness compensation, and can be how DRC is typically calculated and applied. In this sense, metadata for dynamic loudness compensation can be said to be used to align DRC parameters.
[0074] (Metadata-based dynamic processing of audio data) A decoder 100 for performing metadata-based dynamic processing on audio data to be reproduced will be described with reference to the example of Fig. 1. The decoder 100 may include one or more processors and non-transitory memory configured to perform a method including a process according to steps S101 to S105 shown in the example of Fig. 2.
[0075] The decoder 100 may receive a bitstream containing audio data and metadata and may be able to output unprocessed (original) audio data, processed audio data after application of dynamic processing parameters determined from the metadata, and / or the metadata itself, depending on the requirements.
[0076] See the example of Fig. 2. In step S101, the decoder 100 may receive a bitstream including audio data and metadata to be subjected to dynamic loudness adjustment and, if necessary, dynamic range compression (DRC). The audio data may be encoded audio data or may be unprocessed. That is, the audio data may be said to be original audio data. The metadata may include multiple sets of metadata. For example, each metadata payload may include multiple such sets of metadata. These different sets of metadata may be associated with respective playback conditions (e.g., different playback conditions).
[0077] Although there is no limitation on the format of the bitstream, in one embodiment the bitstream may be an MPEG-D DRC bitstream. In this case, the presence of metadata for dynamic processing on the audio data may be indicated based on the MPEG-D DRC bitstream syntax. In one embodiment, the loudnessInfoSetExtension()-element may be used to carry the metadata as a payload, as described in more detail below.
[0078] The audio data and metadata may then be decoded by a decoder in step S102 to obtain decoded audio data and metadata. In an embodiment, the metadata may include one or more processing parameters related to an average loudness value and, optionally, one or more processing parameters related to dynamic range compression characteristics. It is understood that each set of metadata may include respective processing parameters. The metadata allows for the application of dynamic or real-time corrections. For example, when encoding and decoding for live real-time playback, application of "real-time" or dynamic loudness metadata may be desired to ensure that the loudness of the live playback audio is properly managed.
[0079] Then, in step S103, the decoder determines from the metadata one or more processing parameters for dynamic loudness adjustment based on the playback conditions, which may be done by using the playback conditions or information derived from the playback conditions (e.g., playback condition information) to identify an appropriate metadata set from multiple sets of metadata.
[0080] In an embodiment, the playback conditions may include one or more of the following: the device type of the decoder, the characteristics of the playback device, the characteristics of the loudspeakers, the loudspeaker settings, the characteristics of the background noise, the characteristics of the ambient noise, and the characteristics of the acoustic environment. Preferably, the playback condition information may indicate a specific loudspeaker setting. Taking into account the playback conditions allows the decoder to target the selection of processing parameters for dynamic loudness adjustment with respect to device and environmental constraints.
[0081] In an embodiment, the process of determining one or more processing parameters in step S103 may further include selecting, by the decoder, at least one of a set of DRC sequences DRCSet, a set of equalizer parameters EQSet, and a downmix depending on the playback conditions, such that the at least one of DRCSet, EQSet, and downmix correlates with or is indicative of environmental constraints due to individual devices and playback conditions.
[0082] Preferably, step S103 includes selecting a DRCSet, which is a set of DRC sequences, in other words, the selected set of metadata may include such a set of DRC sequences.
[0083] In an embodiment, the determining process of step S103 may further include determining the one or more processing parameters from metadata by identifying a metadata identifier indicative of the selected at least one of DRCSet, EQSet and DownmixSet, such that the metadata identifier can associate metadata with the selected DRCSet, EQSet and / or downmix corresponding to the metadata and thus associated with the respective playback conditions.
[0084] In some embodiments, a particular loudspeaker setting may be used to determine a downmix, and the downmix may be used to identify and select an appropriate one of multiple sets of metadata. In such cases, the particular loudspeaker setting and / or downmix may be indicated by the playback condition information.
[0085] In an embodiment, the metadata may include one or more metadata payloads (e.g., dynLoudComp() payload, e.g., as shown in Table 5 below). In this case, each metadata payload may include multiple sets of parameters (e.g., parameter dynLoudCompValue) and identifiers, each set including at least one of a DRCSet identifier drcSetId, an EQSet identifier EqSetId, and a downmix identifier downmixId in combination with one or more processing parameters associated with the identifiers in the set. That is, each payload may include an array of entries, each entry including a processing parameter and an identifier (e.g., drcSetId, eqSetId, downmixId). The array of entries may correspond to multiple sets of metadata as described above. Preferably, each entry includes a downmix identifier.
[0086] Therefore, in a further embodiment, the process of determining step S103 may include selecting one of the sets in the payload based on the downmix selected by the decoder (or based on at least one of the DRCSet, EQSet and downmix). In this case, the one or more processing parameters determined in step S103 may be one or more processing parameters associated with an identifier in the selected set. That is, depending on the settings (e.g. DRCSet, EQSet and downmix) present in the decoder, the decoder may search a given payload for a suitable set of parameters and identifiers. This is done by matching the settings with the identifiers. The parameters included in the set with the identifier that best matches the settings may then be selected as processing parameters for the dynamic loudness adjustment.
[0087] Then, in step S104, the determined one or more processing parameters may be applied by the decoder to the decoded audio data to obtain processed audio data, so that the loudness of the processed audio data, e.g., live real-time audio data, is appropriately managed.
[0088] Then, in step S105, the processed audio data is output for playback.
[0089] In some embodiments, the bitstream may further include additional metadata for static loudness adjustment to be applied to the decoded audio data, where static loudness adjustment refers to processing done for general purpose loudness normalization as opposed to dynamic processing for real-time situations.
[0090] By conveying metadata for dynamic processing separately from the additional metadata for general loudness normalization, it is possible to not apply "real-time" corrections.
[0091] For example, when encoding and decoding for live real-time playback, it is desirable to apply dynamic processing to ensure that the loudness of the live playback audio is properly managed, but for non-real-time playback, or transcoding where dynamic correction is not desired or required, there is no need to apply dynamic processing parameters determined from metadata.
[0092] Additionally, dynamic processing (dynamic / real-time) metadata is kept separate from additional metadata, allowing the original unprocessed content to be preserved if desired. The original audio is encoded along with the metadata, allowing playback devices to selectively apply dynamic processing, and enabling playback of the original audio content on high-end devices capable of playing the original audio.
[0093] Keeping dynamic loudness metadata separate from long term loudness measurements / information, such as the contentLoudness (ISO / IEC 23003-4) mentioned above, has several advantages. When they are combined, the content loudness (or anything after the dynamic loudness metadata has been applied) does not indicate the actual loudness of the content, since the only metadata available is the decoded value. Besides removing any ambiguity about what the content loudness (or program or anchor loudness) is, there are several cases where the above is particularly useful:
[0094] Keeping the metadata for dynamic processing separate as described above allows a decoder or playback device to turn off the application of dynamic processing and avoid cascade-type leveling by applying an implemented real-time loudness leveler instead. This situation can arise, for example, if the device's own real-time leveling solution is superior to the one used with the audio codec, or if the device's own real-time leveling solution cannot be disabled and therefore becomes the always active solution in further processing, which leads to a defective playback experience.
[0095] Additionally, keeping the metadata for dynamic processing separate as described above allows transcoding to codecs that do not support dynamic loudness processing, in which case one might want to apply their own loudness processing before re-encoding.
[0096] A further example is a live broadcast with a single encoding for the live feed. For archive or on-demand services dynamic processing metadata can be used or stored. Thus for archive or on-demand services more accurate or more adapted loudness measurements based on the entire program can be performed and appropriate metadata can be reset.
[0097] The above is also advantageous for use cases where a fixed target loudness is used throughout the workflow, for example an R128 compliant situation where a maximum of 23 LKFS is recommended. In this scenario, adding dynamic processing metadata is a "safety" measure. In this case, the content is close to the estimated and required target, and adding dynamic processing metadata is a secondary check, so having the ability to turn it off is desirable.
[0098] (Encodes audio data and metadata to be subjected to dynamic loudness adjustment) With reference to the examples of Figures 3 and 4, an encoder for encoding original audio data and metadata to be subjected to dynamic loudness adjustment and, if necessary, dynamic range compression (DRC) in a bitstream is described. The encoder may include one or more processors and non-transitory memory configured to perform a method including the process shown in the example steps of Figure 4.
[0099] In step S201, original audio data is input to a loudness leveller 201 for loudness processing, so that loudness-processed audio data can be obtained as an output from the loudness leveller 201.
[0100] Then, in step S202, metadata for dynamic loudness adjustment may be generated based on the loudness processed audio data and the original audio data. Appropriate smoothing and time frames may be used to reduce artifacts.
[0101] In an embodiment, step S202 may include comparing the loudness processed audio data with the original audio data by the analyzer 202. In this case, metadata may be generated based on the result of said comparison. The metadata thus generated may emulate the effect of a leveller at the decoder site. The metadata may include: Gain (wideband and / or multiband) processing parameters that, when applied to the original audio data, produce audio that is intended to be reproduced and complies with loudness regulations. Processing parameters that describe the dynamics of the audio, e.g. Peaks - Sample and true peaks Short-term loudness value Changes in short-term loudness values
[0102] In some embodiments, step S202 may further include measuring loudness by the analyzer 202 for one or more predetermined periods. In this case, metadata may be generated further based on the measured loudness. In some embodiments, the measuring may include measuring the loudness of the entire audio data. In some embodiments, the measuring may alternatively or additionally include measuring the loudness of dialogue in the audio data.
[0103] Then, in step S203, the original audio data and metadata may be encoded into a bitstream. Although there is no limitation on the format of the bitstream, in one embodiment, the bitstream may be an MPEG-D DRC bitstream, and the presence of metadata may be signaled based on the MPEG-D DRC bitstream syntax. In this case, in one embodiment, the loudnessInfoSetExtension()-element may be used to carry the metadata as a payload, which is described in more detail below.
[0104] In an embodiment, the metadata may include one or more metadata payloads. In this case, each metadata payload may include a plurality of sets of parameters and identifiers, each set including at least one of a DRCSet identifier drcSetId, an EQSet identifier eqSetId, and a downmix identifier downmixId in combination with one or more processing parameters associated with the identifier in the set. In this case, the one or more processing parameters may be parameters for dynamic loudness adjustment by the decoder. In this case, in an embodiment, the at least one of drcSetId, EQSetId, and downmixId may be associated with at least one of a set of DRC sequences DRCSet, a set of equalizer parameters EQSet, and a downmix to be selected by the decoder. In general, the metadata may be said to include a plurality of sets of metadata, each set corresponding to a respective playback condition (e.g., a different playback condition from each other).
[0105] In some embodiments, the method may further include generating additional metadata for static loudness adjustment to be used by the decoder. Keeping the metadata for dynamic loudness processing and the additional metadata separate in the bitstream and further encoding the original audio data into the bitstream has several advantages as discussed above.
[0106] The methods described herein may be implemented on a decoder or an encoder, respectively. In this case, the decoder and the encoder may include one or more processors and non-transitory memory configured to execute the methods. An example of a device having such processing capabilities is shown in the example of Figure 5. Figure 5 shows a device 300 including two processors 301 and a non-transitory memory 302.
[0107] It should be noted that the method described herein can also be implemented on an encoder and decoder system, where the encoder encodes original audio data and metadata to be subjected to dynamic loudness adjustment and, if necessary, dynamic range compression (DRC) in a bitstream, and the decoder applies metadata-based dynamic processing to the audio data to be reproduced as described herein.
[0108] The method may further be implemented as a computer program product including a computer readable storage medium having instructions configured to perform the method when executed by a device having processing capabilities. The computer program product may be stored on the computer readable storage medium.
[0109] (MPEG-D DRC Modified Bitstream Syntax) In the following, it is described how the MPEG-D DRC bitstream syntax as described in ISO / IEC 23003-4 may be modified according to the embodiments described herein.
[0110] In order to further carry dynamic processing metadata as frame-based dynLoudComp updates, the MPEG-D DRC syntax can be extended, for example with the loudnessInfoSetExtension()-element shown in Table 2 below.
[0111] For example, another switch case UNIDRCLOUDEXT_DYNLOUDCOMP may be added to the loudnessInfoSetExtension()-element as shown in Table 1. A new element dynLoudComp() may be identified using the switch case UNIDRCLOUDEXT_DYNLOUDCOMP as shown in Table 5. The loudnessInfoSetExtension()-element may be an extension of the loudnessInfoSet()-element as shown in Table 2. Furthermore, the loudnessInfoSet()-element may be part of uniDRC() as shown in Table 3.
[0112] [Table 1]
[0113] [Table 2]
[0114] [Table 3]
[0115] [Table 4]
[0116] New dynLoudComp() [Table 5]
[0117] · drcSetId enables dynLoudComp (metadata related) to be applied per DRC-set. ·eqSetId enables dynLoudComp to be applied in combination with various settings for the equalization tool. downmixId enables dynLoudComp to be applied per DownmixId.
[0118] In some cases, in addition to the above parameters, it may be advantageous for the dynLoudComp() element to further include a methodDefinition parameter (e.g., specified by 4 bits) and / or a measurementSystem parameter (e.g., specified by 4 bits). The methodDefinition parameter specifies the loudness measurement method used to derive the dynamic program loudness metadata (e.g., anchor loudness, program loudness, short-term loudness, instantaneous loudness, etc.). The measurementSystem parameter specifies the loudness measurement method used to measure the dynamic program loudness metadata (e.g., EBU R.128, ITU-R BS-1770 with or without preprocessing, ITU-R BS-1771, etc.). Such parameters may be included in the dynLoudComp() element, for example, between the downmixId parameter and the dynLoudCompValue parameter.
[0119] Alternative Syntax 1 [Table 6]
[0120] [Table 7]
[0121] [Table 8]
[0122] In some cases, it may be advantageous to modify the syntax shown in Table 8 above such that the dynLoudCompPresent parameter and (if dynLoudCompPresent=1) the dynLoudCompValue parameter follow the reliability parameter within the measurementCount loop of loudnessInfoV2() instead of outside the measurementCount loop. Additionally, if dynLoudCompPresent is 0, it may also be advantageous to set dynLoudCompValue equal to 0.
[0123] Alternative Syntax 2 Alternatively, the dynLoudComp()-element can be incorporated into a uniDrcGainExtension()-element.
[0124] [Table 9]
[0125] [Table 10]
[0126] [Table 11]
[0127] Semantics dynLoudCompValue This field contains the value for dynLoudCompDb. The value is encoded according to the table below. The default value is 0dB.
[0128] [Table 12]
[0129] Updated MEPG-D DRC loudness normalization process [Table 13]
[0130] Pseudocode for dynLoudComp Selection and Processing
number
[0131] In some cases, in addition to the selection process shown in the pseudocode above (e.g., considering drcSetID, eqSetID and downmixId when selecting the dynLoudCompValue parameter), it may also be advantageous for the selection process to consider the methodDefinition parameter and / or the measurementSystem parameter when selecting the dynLoudCompValue parameter.
[0132] Another updated MEPG-D DRC loudness normalization process [Table 14]
[0133] When using the alternative loudness normalization process shown in Table 14 above, the loudness normalization process pseudo code above may be replaced with the alternative loudness normalization process pseudo code below: Note that a default value for dynLoudCompDb, e.g., 0 dB, may be assumed to ensure that the value of dynLoudCompDb is specified even if no dynamic loudness processing metadata is present in the bitstream.
[0134]
number
[0135] Alternative Syntax 3 In some cases, to increase flexibility in transmitting dynamic loudness processing values, it may be advantageous to combine the syntax shown in Tables 1 to 5 above with another syntax 1 shown in Tables 6 to 8 above, as shown in the following table.
[0136] [Table 15]
[0137] [Table 16]
[0138] [Table 17]
[0139] Alternative syntax for dynLoudComp() 3 [Table 18]
[0140] Interface Extension Syntax In some cases, even when dynamic loudness processing information is present in the received bitstream, it may be advantageous to allow control, e.g., by an end user, of whether dynamic loudness processing is performed. Such control may be provided by updating the MPEG-D DRC interface syntax to include an additional interface extension (e.g., UNIDRCINTERFACEEXT_DYNLOUD) that contains a revised loudness normalization control interface payload (e.g., loudnessNormalizationControlInterfaceV1()). This is shown in the table below.
[0141] [Table 19]
[0142] [Table 20]
[0143] [Table 21]
[0144] Interface Extension Semantics loudnessNormalizationOn This flag indicates whether loudness normalization should be turned on / off. The default value is 0. If loudnessNormalizationOn=0, loudnessNormalizationGainDb is set to 0dB. targetLoudness This field contains the desired output loudness. The value is coded according to the following table:
[0145] [Table 22]
[0146] dynLoudnessNormalizationOn This flag indicates whether the dynamic loudness normalization process should be switched on / off. The default value is 0. If dynLoudnessNormalizationOn=0, then dynloudnessNormalizationGainDb is set to 0dB.
[0147] Further methods for enabling dynamic loudness compensation In addition to the above, further ways of enabling dynamic loudness compensation are also possible. For example, instead of including a parameter indicating a specific dynamic loudness compensation value to be applied to the signal, it is possible to specify a specific set of DRC gains that already include dynamic loudness compensation, e.g., such that dynamic loudness compensation is applied when these specific DRC gains are applied by a decoder. A set of DRC gains suitable for applying such dynamic loudness compensation may be identified via a specific bitstream element. This is further described below. The advantage of such an approach is that there is no need to transmit explicit dynamic loudness compensation gains in addition to other loudness information.
[0148] Additionally, the user can specify via the interface that DRC gains, including dynamic loudness compensation, should be selected and applied, as described further below.
[0149] Finally, additional bitstream elements may be provided to allow for more precise creative control over whether and how dynamic loudness compensation is performed. For example, a flag may indicate whether dynamic loudness compensation can be turned off by the user. In such a case, if the content creator allows dynamic loudness compensation to be disabled, it is up to the user to specify whether dynamic loudness compensation is applied or not.
[0150] Specify the DRC set for Dynamic Loudness Compensation via the DRC Set Effect field In the MPEG-D DRC standard, it is possible to indicate one or more DRC effects provided by a particular DRC set. For example, a DRC set may be appropriate for "late night" viewing, viewing in a "noisy environment", viewing at "low playback levels", etc. A DRC set effect parameter may indicate which particular effects are provided by the DRC set.
[0151] Additionally, the MPEG-D DRC standard allows a user to specify one or more desired DRC effects and one or more optional reserve DRC effects. Such information from the user may be used to select the most suitable DRC set from among the available DRC sets. For example, if there is a DRC set that matches the desired DRC effects, that set may be selected. If there is no such set but there is a set that matches the reserve DRC effects, that set may be selected.
[0152] The list of defined DRC effects may be expanded to include DRC effects that provide dynamic loudness compensation. Additionally, the list of DRC effects that can be specified by the user via the interface may also be expanded to include DRC set effects that provide dynamic loudness compensation. In this manner, dynamic loudness compensation may be provided by including a DRC set that has been indicated as providing a dynamic loudness compensation effect and indicating via the interface that a DRC set that provides dynamic loudness compensation is desired. In the above case, the decoder selects and applies a DRC set that corresponds to the dynamic loudness compensation DRC effect.
[0153] As noted above, a previous version of a table identifying DRC set effects that may be signaled in a bitstream (e.g., Table A.45 of ISO / IEC 23003-4:2020) may be updated to include an entry for a "Dynamic Loudness Compensation" effect, as shown below. Specifically, a row may be added with a particular bit position (e.g., 13) corresponding to the Dynamic Loudness Compensation effect, as shown in the table below.
[0154] [Table 23]
[0155] In the above example, a bit within a bit field associated with a particular DRC set (e.g., the drcSetEffect bit field of a drcInstructionsBasic(), or drcInstructionsUniDrc(), or drcInstructionsUniDrcV1() payload) may be set (e.g., bit position 13) to indicate that the DRC set provides a dynamic loudness compensation effect.
[0156] Similarly, a previous version of a table specifying DRC set effects that may be specified to a decoder via an interface (e.g., Table 12 of ISO / IEC 23003-4:2020) may be updated to include an entry called Dynamic Loudness Compensation Effect, as shown below. Specifically, a row may be added with a particular index value (e.g., 9) corresponding to the Dynamic Loudness Compensation Effect, as shown in the table below.
[0157] [Table 24]
[0158] In the above example, to instruct the decoder that a DRC set providing a dynamic loudness compensation effect should be selected and applied if available, an interface parameter (e.g., the drcEffectTypeRequest parameter of the dynamicRangeControllerInterface() payload) may be set to a particular value (e.g., 9) corresponding to a dynamic loudness compensation effect. However, because dynamic loudness compensation may be intended to be used in combination with loudness normalization, in some cases a decoder that also performs loudness normalization may be required to select and apply a DRC set providing a dynamic loudness compensation effect. Such a requirement may only be achieved by allowing the decoder to select a DRC set providing a dynamic loudness compensation effect when loudness normalization is also enabled in the decoder. Alternatively or additionally, such a requirement may be met by modifying the interface to require that loudness normalization is also enabled when requesting a DRC set providing a dynamic loudness compensation effect (e.g., by setting both loudness normalization and a target loudness value on the flags).
[0159] Specifying DRC sets for dynamic loudness compensation using new extension payloads Alternatively, a new payload (e.g., dynLoudInstructions()) containing instructions for dynamic loudness compensation may be defined. One or more of such payloads may be included, for example, in an extension field (e.g., UNIDRCCONFEXT_V2) of an existing bitstream. Including such payloads in an extension field may maintain compatibility with existing decoders that are not designed to parse the new payload, since such decoders will simply ignore the additional payload. Each such payload is assigned a unique identifier (e.g., drcSetId) corresponding to the DRC set that provides dynamic loudness compensation. To aid in easy identification of such sets, a flag (e.g., dynamicLoudCompDRCSet flag) for each DRC set signaled via the new payload may be set to 1.
[0160] Furthermore, the above-mentioned user interface for selecting a DRC set that provides a dynamic loudness compensation effect may also be used for the DRC set notified via this new type of payload (e.g., dynLoudInstructions() payload). Specifically, the user may indicate to the decoder via the interface (e.g., using the drcEffectTypeRequest field) that a DRC set that provides dynamic loudness compensation is desired. The decoder then selects the DRC set notified via the new payload (e.g., a DRC set with a DRC Set ID identified as corresponding to dynamic loudness compensation and with the dynamicLoudCompDRCSet flag set to, e.g., 1) if any. However, as mentioned above, since dynamic loudness compensation may be intended to be used in combination with loudness normalization, in some cases a decoder that also performs loudness normalization may be required to select and apply a DRC set that provides a dynamic loudness compensation effect. Such a requirement may only be achieved by allowing the decoder to select a DRC set that provides a dynamic loudness compensation effect when loudness normalization is also enabled in the decoder. Alternatively or additionally, such requirements may be met by modifying the interface to require that loudness normalization also be enabled (e.g., by setting both loudness normalization and a target loudness value on a flag) when requesting a DRC set that provides a dynamic loudness compensation effect.
[0161] An advantage of this type of notification (e.g., rather than reporting the DRC sets that provide dynamic loudness compensation via existing parameters such as the drcSetEffect parameter) is that additional parameters can be included in such a new payload indicating whether and how the user can enable and / or disable dynamic loudness compensation, as described above. For example, the new payload may also include a parameter (e.g., a dynamicLoudCompSwOffAllowed parameter) indicating whether turning off dynamic loudness compensation via a user interface is allowed for each DRC set that applies dynamic loudness compensation.
[0162] An example of the syntax of a new payload for dynamic loudness compensation (eg, dynLoudInstructions() payload) is shown in the table below.
[0163] [Table 25]
[0164] As described above, a unique DRC Set ID may be associated with the payload along with a flag (e.g., dynamicLoudCompDRCSet) that specifically indicates the DRC Set as a DRC set that provides dynamic loudness compensation. Additionally, the payload may be associated with one or more downmix identification parameters (e.g., downmixId parameters) that indicate that the DRC Set is intended for use with one or more particular downmixes / downmix structures of the audio program. Such a DRC Set identifier (e.g., drcSetId parameters), along with a flag (e.g., dynamicLoudCompDRCSet flag), enables selection of the DRC Set associated with the payload in response to a user indication, provided via an interface, that the user desires a dynamic loudness compensation DRC effect. Additionally, the inclusion of one or more downmix identifiers (e.g., downmixID parameters) enables selection of a DRC Set that provides dynamic loudness compensation that is intended for use with a particular downmix of the audio program. Finally, as also mentioned above, the inclusion of a parameter (e.g., dynamicLoudCompSWOffAllowed) indicating whether dynamic loudness compensation is allowed to be turned off enables additional control for the content creator as to whether dynamic loudness compensation can be applied by the decoder.
[0165] Finally, as noted above, such additional payloads (e.g., a dynLoudInstructions() payload) may be included within an extension payload of a configuration payload (e.g., a uniDrcConfigExtension() payload). A particular extension type (e.g., a uniDrcConfigExtType of UNIDRCCONFEXT_V2 with a value of 0x2) may be used to indicate the presence of such additional payloads within the extension field of a configuration payload.
[0166] An example of the syntax for the extension field of a configuration payload (eg, uniDrcConfigExtension() payload) is shown in the table below.
[0167] [Table 26] JPEG2024543726000033.jpg192132JPEG2024543726000034.jpg109133
[0168] The data associated with a particular extension type in the extension field of the configuration payload includes a parameter indicating whether dynamic loudness compensation instructions are present (e.g., a dynLoudPresent parameter). If present, the data includes a parameter indicating the number of sets of dynamic loudness compensation instructions in the extension payload (e.g., a dynLoudInstructionsCount parameter) and each set of dynamic loudness compensation instructions (e.g., each dynLoudInstructions() payload). In that case, the syntax for the dynamic loudness compensation instructions may be as shown in the table above.
[0169] A DRC Set for dynamic loudness compensation is signaled via an existing configuration element. Alternatively, a new field may be added to an existing configuration element to indicate that a particular DRC set provides dynamic loudness compensation. For example, a new parameter (e.g., dynamicLoudCompDRCSet parameter) indicating whether a DRC set provides dynamic loudness compensation may be included in an existing configuration element (e.g., downmixInstructions() element, drcInstructionsBasic() element, or drcInstructionsUniDrc() element). Ideally, to avoid compatibility issues with legacy decoders, the new element may be included in a field that legacy decoders already ignore (e.g., a reserved field). Alternatively, the new parameter may be included in an updated version of an existing configuration element (e.g., downmixInstructionsV2() element or drcInstructionsUniDrcV2() element). This version is ignored by legacy decoders.
[0170] Finally, similar to the above selection process, the DRC set thus notified as providing dynamic loudness compensation may be selected by the decoder via the interface by indicating that a DRC set providing a dynamic loudness compensation effect is desired. When such an indication is provided to the decoder via the interface, the decoder identifies and selects the DRC set indicated as providing dynamic loudness compensation (e.g., having a dynamicLoudCompDRCSet parameter equal to 1). If multiple DRC sets are indicated as providing dynamic loudness compensation, the decoder may use other rules (e.g., predefined rules specified in ISO / IEC 23003-4:2020) to select the most suitable DRC among the multiple DRC sets that provide dynamic loudness compensation.
[0171] However, because dynamic loudness compensation may be intended to be used together with loudness normalization, in some cases a decoder that also performs loudness normalization may be required to select and apply a DRC set that provides a dynamic loudness compensation effect. Such a requirement may only be achieved by allowing a decoder to select a DRC set that provides a dynamic loudness compensation effect when loudness normalization is also enabled in the decoder. Alternatively or additionally, such a requirement may be met by modifying the interface to require that loudness normalization is also enabled (e.g., by setting both loudness normalization and a target loudness value on a flag) when requesting a DRC set that provides a dynamic loudness compensation effect.
[0172] Fig. 6 shows an example of a method for processing the audio data to be played back. See the example of Fig. 6. In step S301, the decoder 100 may receive a bitstream including encoded audio data and metadata. In this case, the metadata includes one or more Dynamic Range Control (DRC) sets, and for each DRC set, an indication of whether the DRC set is configured to provide a dynamic loudness compensation effect. This may be done, for example, as described above.
[0173] Although there is no limitation on the format of the bitstream, in one embodiment the bitstream may be an MPEG-D DRC bitstream. Then, based on the MPEG-D DRC bitstream syntax, the presence of metadata providing dynamic loudness compensation effects may be signaled, for example as described above.
[0174] Then, in step S302, the decoder may analyze the metadata to identify a DRC set configured to provide a dynamic loudness compensation effect, for example as described above.
[0175] Then, in step S303, the decoder may obtain decoded audio data by decoding the audio data, for example as described above.
[0176] Then, in step S304, the decoder may select one of the identified DRC sets configured to provide a dynamic loudness compensation effect, for example as described above.
[0177] Then, in step S305, the decoder may extract from the bitstream one or more DRC gains corresponding to the selected DRC set, for example as described above.
[0178] Then, in step S306, the decoder may obtain dynamic loudness compensated audio data by applying one or more DRC gains corresponding to the selected DRC set to the decoded audio data, for example as described above.
[0179] Finally, in step S307, the dynamic loudness compensated audio data is output for playback, for example as described above.
[0180] Efficient coding / transmission of dynamic loudness compensation data As described above, dynamic loudness compensation combined with dynamic range control may be achieved by transmitting a DRC set that includes gains that are a combination of dynamic range gains and dynamic loudness compensation gains. However, this may require redundant transmission of dynamic range control gain data. For example, a first DRC set may include gains that apply dynamic range control to a signal, and it may be important to allow that DRC gain set to be applied independently of other gains (e.g., dynamic loudness compensation gains). Therefore, if a DRC set that includes gains that apply a combination of dynamic range control and dynamic loudness compensation is desired, it is necessary to identify a second DRC set that represents a combination of the first set of dynamic range control gains and the desired dynamic loudness compensation gains. Unfortunately, this requires that both the first and second DRC sets include information about the first set of dynamic range control gains, which is not efficient.
[0181] A more efficient way of transmitting data to achieve the same goal is to provide only the dynamic range control in the first DRC set and only the dynamic loudness compensation gains in the second DRC set, along with some additional metadata indicating the relationship between the two DRC sets. For example, the first DRC set may include a parameter indicating that there is another DRC set that depends on the first DRC set. The decoder may then understand that the gains of the first set can be applied independently, for example if only dynamic range control is desired. The decoder may also understand that the gains of the first and second DRC sets can be combined to obtain a combination of dynamic range control and dynamic loudness compensation. Finally, if the gains of the second set are not intended to be applied alone, metadata may be included with the second DRC set to indicate whether the gains of the DRC set can be used only in combination with the first set or independently of the gains of the first set.
[0182] For example, consider two DRC sets. A first DRC set is identified as DRC Set 1 (e.g., has drcSetId equal to 1) and provides gains for dynamic range control. A second DRC set is identified as DRC Set 2 (e.g., has drcSetId equal to 2) and provides gains for dynamic loudness compensation. Because DRC Set 2 is intended for use in combination with DRC Set 1, DRC Set 1 may include a parameter indicating that DRC Set 2 is dependent on DRC Set 1 (e.g., the dependsOnDrcSet parameter of DRC Set 1 may be equal to 2). Furthermore, if dynamic loudness compensation is intended for use only in combination with dynamic range control, DRC Set 2 may include a parameter indicating that the gains of DRC Set 2 may not be used independently (e.g., the noIndependentUse flag of DRC Set 2 may be set to a value of 1). Conversely, if DRC Set 2 is not intended to be used solely in combination with dynamic range control, DRC Set 2 may include a parameter indicating that the gains of DRC Set 2 may be used independently (e.g., the noIndependentUse flag of DRC Set 2 may be set to a value of 0).
[0183] The above efficient coding / transmission of dynamic loudness compensation data may be used in conjunction with any of the further methods for enabling dynamic loudness compensation described above.
[0184] For example, in the first case, the presence of a DRC Set providing a dynamic loudness compensation effect is signaled via an existing effect flag (e.g., a little drcSetEffect bit field), but in this case, DRC Set 1 may have a drcSetEffect parameter with the bit corresponding to the dynamic loudness compensation effect set to 0, DRC Set 2 may have the bit corresponding to the dynamic loudness compensation effect set to 1, and all other bits of the bit field may be the same for both DRC Sets. According to such signaling, when a user requests a dynamic loudness compensation effect via the interface, the decoder extracts and applies the gains of DRC Set 1 and DRC Set 2 to apply a combination of dynamic range control and dynamic loudness compensation.
[0185] The following table shows example values of parameters that achieve the above signaling for efficient transmission of dynamic loudness compensation data.
[0186] [Table 27]
[0187] In the second case, the presence of a DRC Set providing a dynamic loudness compensation effect is signaled using a new payload (e.g., dynLoudInstructions payload is set to 2), but in this case DRC Set 2 is signaled via a new payload (e.g., dynLoudInstructions payload is set to 2). DRC Set 1 is signaled as above (e.g., drcSetId parameter is set to 1 and dependsDrcSet parameter is set to 2, which indicates that DRC Set 2 depends on DRC Set 1). Finally, it is not possible to indicate via the above new payload (e.g., dynLoudInstructions) that DRC Set 2 should not be used independently (e.g., should always be used together with DRC Set 1). Therefore, an additional parameter may be added to the new payload (e.g., a noIndependentUse parameter may be added to dynLoudInstructions) such that when the new parameter is set to 1, DRC Set 2 is used only together with DRC Set 1, and when the new parameter is set to 0, DRC Set 2 may be used independently of DRC Set 1. As in the first case, when the user requests a dynamic loudness compensation effect via the interface, the decoder extracts and applies the gains of both DRC Set 1 and DRC Set 2 to apply a combination of dynamic range control and dynamic loudness compensation.
[0188] Finally, in the third case, existing configuration elements are used that already contain all of the necessary parameters to effectively enable dynamic loudness compensation as described above. Specifically, DRC Set 1 may contain an indication that DRC Set 2 is dependent on DRC Set 1 as described above, and DRC Set 2 contains an indication of whether it may be used independently or only in combination with DRC Set 2. Again, as in the previous two cases, when the user requests a dynamic loudness compensation effect via the interface, the decoder extracts and applies the gains of DRC Set 1 and DRC Set 2 to apply a combination of dynamic range control and dynamic loudness compensation.
[0189] As described above, an advantage of using a dependent DRC Set to enable dynamic loudness compensation is that the data required for dynamic loudness compensation can be transmitted more efficiently, since it is no longer necessary to redundantly transmit the dynamic range control gains in the DRC Set that contain the dynamic loudness compensation gains.
[0190] (interpretation) Unless otherwise indicated, and as will be apparent from the description that follows, use of terms such as "processing," "calculating," "determining," "analyzing," and the like throughout the disclosed description will be understood to refer to operations and / or processes of a computer or computing system, or similar electronic computing device, that manipulates and / or converts data expressed as physical quantities, such as quantities of electrons, into other data similarly expressed as physical quantities.
[0191] Similarly, the term "processor" may refer to any device or any part of a device that processes electronic data, e.g., from a register and / or memory, to convert that data into other electronic data that may be stored in a register and / or memory. A "computer" or a "computing machine" or a "computing platform" may include one or more processors.
[0192] The methods described herein can be executed, in an example embodiment, by one or more processors receiving computer-readable (also referred to as machine-readable) code that includes a set of instructions that, when executed by one or more of the processors, performs at least one of the methods described herein. Any processor capable of executing a set of instructions (sequential or otherwise) that specify operations of interest is included. Thus, an example is a typical processing system that includes one or more processors. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system may further include a memory subsystem that includes a main RAM and / or a static RAM, and / or a ROM. A bus subsystem may be included for communication between these components. Furthermore, the processing system may be a distributed processing system with processors coupled by a network. If the processing system requires a display, a display such as, for example, a liquid crystal display (LCD) or a cathode ray tube display (CRT) may be included. If manual data entry is required, the processing system also includes one or more input devices, such as an alphanumeric input unit such as a keyboard, a pointing control device such as a mouse, and the like. The processing system may also include a storage system, such as a disk drive unit. In some configurations, the processing system may include a sound output device and a network interface device. Thus, the memory subsystem includes a computer-readable carrier medium carrying computer-readable code (e.g., software) that includes a set of instructions that, when executed by one or more processors, causes one or more of the methods described herein to be performed. It should be noted that, when a method includes several elements, e.g., several steps, no ordering of such elements is implied unless specifically stated. The software may reside in a hard disk, or may reside, completely or at least partially, in a RAM and / or in a processor during execution by the computer system.Thus, the memory and the processor also constitute a computer-readable carrier medium carrying computer-readable code. Moreover, the computer-readable carrier medium may form or be included in a computer program product.
[0193] In another example embodiment, one or more processors may operate as stand-alone devices or may be connected, e.g., networked, to other processors in a network configuration, one or more processors may operate in the capacity of a server or user machine in a server-user network environment, or as a peer machine in a peer-to-peer or distributed network environment. One or more processors may form a personal computer (PC), tablet PC, personal digital assistant (PDA), mobile phone, web appliance, network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify operations to be performed by the machine.
[0194] It should be noted that the term "machine" is also intended to include any collection of machines that individually or collectively execute a set (or sets) of instructions for performing any one or more of the methods described herein.
[0195] Thus, an example embodiment of each of the methods described herein is in the form of a computer-readable carrier medium carrying a set of instructions, e.g., a computer program for execution on one or more processors (e.g., one or more processors that are part of a web server configuration). Thus, as will be appreciated by those skilled in the art, example embodiments of the present disclosure may be embodied as a method, an apparatus, such as a dedicated device, an apparatus, such as a data processing system, or a computer-readable carrier medium, e.g., a computer program product. The computer-readable carrier medium carries computer-readable code, which when executed on one or more processors, causes the processor to implement the method. Thus, aspects of the present disclosure may take the form of a method, an example entirely hardware embodiment, an example entirely software embodiment, or an example embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a carrier medium (e.g., a computer program product on a computer-readable storage medium) carrying computer-readable program code embodied in the medium.
[0196] The software may further be transmitted and received across a network via a network interface device. Although the carrier medium is a single medium in the example embodiment, the term "carrier medium" should be interpreted to include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) that store one or more sets of instructions. The term "carrier medium" should also be interpreted to include any medium that can store, encode or carry a set of instructions for execution by one or more of the processors, causing the one or more processors to perform any one or more of the methods of the present disclosure. Carrier media may take many forms, including, but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical disks, magnetic disks, and magneto-optical disks. Volatile media include dynamic memory, such as main memory. Transmission media include coaxial cables, copper wire, and fiber optics, including the wires that make up a bus subsystem. Transmission media may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications. Thus, for example, the term "carrier medium" is intended to include, but is not limited to, computer products embodied in solid-state memories, optical and magnetic media, media carrying propagated signals detectable by at least one processor or one or more processors and representing a set of instructions that when executed implement a method, and transmission media within a network carrying propagated signals detectable by at least one processor of one or more processors and representing a set of instructions.
[0197] It is understood that the steps of the above method are performed, in an example embodiment, by a suitable processor of a processing (e.g., computer) system executing instructions (e.g., computer readable code) stored in storage. It is also understood that the present disclosure is not limited to any particular implementation or programming technique, and that the present disclosure may be implemented using any suitable technique for implementing the functionality described herein. The present disclosure is not limited to any particular programming language or operating system.
[0198] References throughout this disclosure to "one example embodiment," "example embodiments," or "example embodiments" mean that a particular feature, structure, or characteristic described in connection with that example embodiment is included in at least one example embodiment of the disclosure. Thus, the appearances of the phrases "in one example embodiment," "in example embodiments," or "in example embodiments" in various places throughout this disclosure do not necessarily all refer to the same example embodiment. Furthermore, particular features, structures, or characteristics may be combined in any suitable manner, as would be apparent to one of ordinary skill in the art from this disclosure, in one or more example embodiments.
[0199] As used herein, unless otherwise specified, the use of ordinal numbers such as "first," "second," "third," etc. to describe a common object merely indicates that different instances of a similar object are being referenced, and is not intended to imply that the objects so described must be in a given sequence, whether temporally, spatially, in ranking order, or in any other manner.
[0200] In the following claims and in the description of this specification, the terms "comprise", "comprises" or "comprises" are all open terms meaning at least the element / feature following the term, but not excluding other elements / features. Thus, the term "comprises", when used in the claims, should not be interpreted as being limited to the means or elements or steps listed thereafter. For example, the scope of an expression of a device comprising A and B should not be limited to a device consisting of only elements A and B. As used herein, the terms "containing" or "comprises" are also open terms meaning at least the element / feature following the term, but not excluding other elements / features. Thus, "comprises" is synonymous with "comprises" and means "comprises".
[0201] In the above description of example embodiments of the present disclosure, it should be understood that various features of the present disclosure may be grouped together in a single example embodiment, figure, or description thereof in order to simplify the disclosure and aid in understanding one or more of the various inventive aspects. However, this method of disclosure should not be interpreted as reflecting an intention that the claims require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in fewer than all features of a single disclosed example embodiment described above. Thus, the claims following the "Description of the Invention" are expressly incorporated herein, with each claim standing on its own as a separate example embodiment of the present disclosure.
[0202] Moreover, although some example embodiments described herein include some features and not others that are included in other example embodiments, combinations of features of different example embodiments are meant to be within the scope of the disclosure and form different example embodiments, as would be understood by one of ordinary skill in the art. For example, in the following claims, any of the claimed embodiments can be used in any combination.
[0203] In the description provided herein, numerous specific details are set forth. However, it will be understood that example embodiments of the present disclosure may be practiced without these specific details. In other instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
[0204] Thus, while what is believed to be the best mode of the disclosure has been described, those skilled in the art will recognize that other and further modifications may be made thereto without departing from the spirit of the disclosure, and it is intended to claim all such variations and modifications as being within the scope of the disclosure. For example, any formulas above are merely representative of procedures that may be used. Functions may be added or deleted from the block diagrams, and operations may be interchanged between functional blocks. Steps may be added or deleted to methods within the scope of the disclosure.
[0205] The two sets of example embodiments listed below (EEE-A and EEE-B) are not claims, but describe the structure, features, and functionality of some aspects of the example embodiments disclosed herein.
[0206] EEE-A1. A method for performing metadata-based dynamic processing of audio data to be reproduced, comprising: (a) receiving, by a decoder, a bitstream comprising audio data and metadata to be subjected to dynamic loudness adjustment; (b) decoding said audio data and said metadata by said decoder to obtain decoded audio data and metadata; (c) determining by the decoder from the metadata one or more processing parameters for dynamic loudness adjustment based on playback conditions; (d) applying the determined one or more processing parameters to the decoded audio data to obtain processed audio data; and (e) outputting the processed audio data for playback; The method includes:
[0207] EEE-A2. The method of EEE-A1, wherein the metadata indicates processing parameters for a plurality of playback conditions for dynamic loudness adjustment.
[0208] EEE-A3. The method of EEE-A1 or EEE-A2, wherein said process of determining said one or more processing parameters further comprises determining one or more processing parameters for dynamic range compression (DRC) based on playback conditions.
[0209] EEE-A4. A method according to any one of EEE-A1 to EEE-A3, wherein the playback conditions include one or more of the device type of the decoder, playback device characteristics, loudspeaker characteristics, loudspeaker settings, background noise characteristics, ambient noise characteristics, and acoustic environment characteristics.
[0210] EEE-A5. A method according to any one of EEE-A1 to EEE-A4, wherein process (c) further comprises selecting by the decoder at least one of a set of DRC sequences DRCSet, a set of equalizer parameters EQSet, and a downmix, which corresponds to the playback conditions.
[0211] EEE-A6. The method described in EEE-A5, wherein process (c) further includes determining the one or more processing parameters from the metadata by identifying a metadata identifier indicating the selected at least one of DRCSet, EQSet, and downmix.
[0212] EEE-A7. The method of any one of EEE-A1 to EEE-A6, wherein the metadata includes one or more processing parameters related to average loudness values and, optionally, one or more processing parameters related to dynamic range compression characteristics.
[0213] EEE-A8. The method of any one of EEE-A1 to EEE-A7, wherein the bitstream further contains additional metadata for static loudness adjustment to be applied to the decoded audio data.
[0214] EEE-A9. The method of any one of EEE-A1 to EEE-A8, wherein the bitstream is an MPEG-D DRC bitstream and the presence of metadata is signaled based on MPEG-D DRC bitstream syntax.
[0215] EEE-A10. The method according to EEE-A9, wherein the loudnessInfoSetExtension()-element is used to carry the metadata as payload.
[0216] EEE-A11. A method according to any one of EEE-A1 to EEE-A10, wherein the metadata includes one or more metadata payloads, each metadata payload including a plurality of sets of parameters and identifiers, each set including at least one of a DRCSet identifier drcSetId, an EQSet identifier eqSetId, and a downmix identifier downmixId in combination with one or more processing parameters associated with the identifiers in the set.
[0217] EEE-A12. A method according to EEE-A11 which depends on EEE-A5, wherein process (c) includes selecting one set from the plurality of sets in the payload based on the at least one of DRCSet, EQSet and downmix selected by the decoder, and the one or more processing parameters determined in process (c) are the one or more processing parameters associated with the identifier in the selected set.
[0218] EEE-A13. A decoder for performing metadata-based dynamic processing of audio data to be reproduced, comprising: (a) receiving, by a decoder, a bitstream comprising audio data and metadata to be subjected to dynamic loudness adjustment; (b) decoding said audio data and said metadata by said decoder to obtain decoded audio data and metadata; (c) determining by the decoder from the metadata one or more processing parameters for dynamic loudness adjustment based on playback conditions; (d) applying the determined one or more processing parameters to the decoded audio data to obtain processed audio data; and (e) outputting the processed audio data for playback; 11. A decoder comprising: one or more processors and a non-transitory memory configured to perform a method comprising:
[0219] EEE-A14. A method for encoding audio data and metadata to be subjected to dynamic loudness adjustment into a bitstream, comprising: (a) a process of inputting original audio data to a loudness leveller for loudness processing, thereby obtaining loudness-processed audio data as an output from said loudness leveller; (b) generating metadata for dynamic loudness adjustment based on the loudness processed audio data and the original audio data; (c) encoding said original audio data and said metadata into said bitstream; The method includes:
[0220] EEE-A15. The method according to EEE-A14, further comprising generating additional metadata for static loudness adjustment, the metadata to be used by the decoder.
[0221] EEE-A16. The method according to EEE-A14 or EEE-A15, wherein process (b) includes comparing the loudness processed audio data with the original audio data, and the metadata is generated based on a result of the comparison.
[0222] EEE-A17. The method of EEE-A16, wherein process (b) further comprises measuring loudness over one or more predetermined periods, and wherein the metadata is generated further based on the measured loudness.
[0223] EEE-A18. The method of EEE-A17, wherein said measuring includes measuring the loudness of the entire audio data.
[0224] EEE-A19. The method of EEE-A17, wherein said measuring includes measuring loudness of speech in said audio data.
[0225] EEE-A20. The method of any one of EEE-A14 to EEE-A19, wherein the bitstream is an MPEG-D DRC bitstream and the presence of metadata is signaled based on MPEG-D DRC bitstream syntax.
[0226] EEE-A21. The method according to EEE-A20, wherein a loudnessInfoSetExtension()-element is used to carry said metadata as payload.
[0227] EEE-A22. A method according to any one of EEE-A14 to EEE-A21, wherein the metadata includes one or more metadata payloads, each metadata payload including a plurality of sets of parameters and identifiers, each set including at least one of a DRCSet identifier drcSetId, an EQSet identifier eqSetId, and a downmix identifier downmixId in combination with the one or more processing parameters associated with the identifier in the set, the one or more processing parameters being parameters for dynamic loudness adjustment by a decoder.
[0228] EEE-A23. The method according to EEE-A22, wherein the at least one of the drcSetId, the eqSetId and the downmixId relates to at least one of a DRCSet, a set of DRC sequences, an EQSet, a set of equalizer parameters, and a downmix to be selected by the decoder.
[0229] EEE-A24. An encoder for encoding original audio data and metadata to be subjected to dynamic loudness adjustment in a bitstream, comprising: (a) a process of inputting original audio data to a loudness leveller for loudness processing, thereby obtaining loudness-processed audio data as an output from said loudness leveller; (b) generating metadata for the dynamic loudness adjustment based on the loudness processed audio data and the original audio data; (c) encoding said original audio data and said metadata into said bitstream; 16. An encoder comprising: one or more processors and a non-transitory memory configured to perform a method comprising:
[0230] EEE-A25. A system comprising an encoder as described in EEE-A24 for encoding original audio data and metadata to be subjected to dynamic loudness adjustment and / or dynamic range compression (DRC) in a bitstream, and a decoder as described in EEE-A13 for performing metadata-based dynamic processing on the audio data to be reproduced.
[0231] EEE-A26. A computer program product including a computer readable storage medium having instructions configured, when executed by a device having processing capability, to cause the device to perform the method according to any one of EEE-A1 to EEE-A12 and EEE-A14 to EEE-A23.
[0232] A computer readable storage medium storing a computer program product according to EEE-A27.EEE-A26.
[0233] EEE-A28. The method of any one of EEE-A1 to EEE-A12, further comprising receiving by the decoder via an interface an indication of whether or not the metadata-based dynamic processing should be applied to the audio data to be reproduced, and skipping at least the step of applying the determined one or more processing parameters to the decoded audio data if the decoder receives an indication that the metadata-based dynamic processing should not be applied to the audio data to be reproduced.
[0234] EEE-A29. The method of EEE-A28, wherein the decoder skips at least the step of applying the determined one or more processing parameters to the decoded audio data until it receives an indication via an interface as to whether or not metadata-based dynamic processing should be applied to the audio data to be reproduced.
[0235] EEE-A30. The method of any one of EEE-A1 to EEE-A12, EEE-A28 and EEE-A29, wherein the metadata indicates processing parameters for a plurality of playback conditions for dynamic loudness adjustment, and the metadata further includes parameters identifying a loudness measurement method used to derive a processing parameter from the plurality of processing parameters.
[0236] EEE-A31. The method of any one of EEE-A1 to EEE-A12 and EEE-A28 to EEE-A30, wherein the metadata indicates processing parameters for a plurality of playback conditions for dynamic loudness adjustment, and the metadata further includes a parameter identifying a loudness measurement system used to measure one of the plurality of processing parameters.
[0237] EEE-B1. A method for performing metadata-based dynamic processing of audio data to be reproduced, comprising: receiving, by a decoder, a bitstream comprising audio data to be subjected to dynamic loudness adjustment and metadata, the metadata for dynamic loudness adjustment comprising a plurality of sets of metadata, each set of metadata corresponding to a respective playback condition; decoding said audio data and said metadata with said decoder to obtain decoded audio data and metadata; in response to playback condition information provided to the decoder, selecting a set of metadata corresponding to a particular playback condition, and extracting from the selected set of metadata one or more processing parameters for dynamic loudness adjustment; applying the extracted one or more processing parameters to the decoded audio data to obtain processed audio data; outputting the processed audio data for playback; The method includes:
[0238] EEE-B2. The method of EEE-B1, wherein extracting one or more processing parameters further comprises extracting one or more processing parameters for dynamic range compression (DRC).
[0239] EEE-B3. The method according to EEE-B1 or EEE-B2, wherein said playback condition information indicates a particular loudspeaker setting.
[0240] EEE-B4. The method of any one of EEE-B1 to EEE-B3, wherein the selected set of metadata includes a DRSet, which is a set of DRC sequences.
[0241] EEE-B5. The method of any one of EEE-B1 to EEE-B4, wherein selecting the set of metadata includes identifying a set of metadata that corresponds to a particular downmix.
[0242] EEE-B6. The method of any one of EEE-B1 to EEE-B5, wherein each of the sets of metadata includes one or more processing parameters related to average loudness values and, optionally, one or more processing parameters related to dynamic range compression characteristics.
[0243] EEE-B7. The method of any one of EEE-B1 to EEE-B6, wherein the bitstream further comprises additional metadata for static loudness adjustment to be applied to the decoded audio data.
[0244] EEE-B8. The method of any one of EEE-B1 to EEE-B7, wherein the bitstream is an MPEG-D DRC bitstream and the presence of metadata is signaled based on MPEG-D DRC bitstream syntax.
[0245] EEE-B9. The method according to EEE-B8, wherein a loudnessInfoSetExtension()-element is used to carry said metadata as payload.
[0246] EEE-B10. A method according to any one of EEE-B1 to EEE-B9, wherein the metadata comprises one or more metadata payloads, each metadata payload comprising a plurality of sets of parameters and identifiers, each set comprising a respective downmix identifier, downmixId, in combination with one or more processing parameters associated with the downmix identifier in the set.
[0247] EEE-B11. A decoder for performing metadata-based dynamic processing on audio data to be reproduced, comprising: receiving, by a decoder, a bitstream comprising audio data to be subjected to dynamic loudness adjustment and metadata, the metadata for dynamic loudness adjustment comprising a plurality of sets of metadata, each set of metadata corresponding to a respective playback condition; decoding said audio data and said metadata with said decoder to obtain decoded audio data and metadata; in response to playback condition information provided to the decoder, selecting a set of metadata corresponding to a particular playback condition, and extracting from the selected set of metadata one or more processing parameters for dynamic loudness adjustment; applying the extracted one or more processing parameters to the decoded audio data to obtain processed audio data; outputting the processed audio data for playback; 11. A decoder comprising: one or more processors and a non-transitory memory configured to perform a method comprising:
[0248] EEE-B12. A method for encoding audio data and metadata to be subjected to dynamic loudness adjustment into a bitstream, comprising: inputting original audio data into a loudness leveller for loudness processing, thereby obtaining loudness-processed audio data as an output from said loudness leveller; generating metadata for the dynamic loudness adjustment based on the loudness processed audio data and the original audio data; encoding the original audio data and the metadata into the bitstream; The method includes:
[0249] EEE-B13. The method of EEE-B12, further comprising generating additional metadata for static loudness adjustment, the metadata to be used by the decoder.
[0250] EEE-B14. The method of EEE-B12 or EEE-B13, wherein generating metadata includes comparing the loudness processed audio data with the original audio data, and the metadata is generated based on a result of the comparison.
[0251] EEE-B15. The method of EEE-B14, wherein generating the metadata further comprises measuring loudness over one or more predetermined periods, and wherein the metadata is generated further based on the measured loudness.
[0252] EEE-B16. The method of EEE-B15, wherein said measuring includes measuring the loudness of the entire audio data.
[0253] EEE-B17. The method of EEE-B15, wherein said measuring includes measuring loudness of speech in said audio data.
[0254] EEE-B18. The method of any one of EEE-B12 to EEE-B17, wherein the bitstream is an MPEG-D DRC bitstream and the presence of the metadata is signaled based on MPEG-D DRC bitstream syntax.
[0255] EEE-B19. The method according to EEE-B18, wherein the loudnessInfoSetExtension()-element is used to carry said metadata as payload.
[0256] EEE-B20. The method according to any one of EEE-B12 to EEE-B19, wherein the metadata comprises a plurality of sets of metadata, each set of metadata corresponding to a respective playback condition.
[0257] EEE-B21. A method according to any one of EEE-B12 to EEE-B20, wherein the metadata comprises one or more metadata payloads, each metadata payload comprising a plurality of sets of parameters and identifiers, each set comprising a respective downmix identifier, downmixId, in combination with one or more processing parameters associated with the downmix identifier in the set, the one or more processing parameters being parameters for dynamic loudness adjustment by a decoder.
[0258] EEE-B22. An encoder for encoding original audio data and metadata to be subjected to dynamic loudness adjustment in a bitstream, comprising: inputting original audio data into a loudness leveller for loudness processing, thereby obtaining loudness-processed audio data as an output from said loudness leveller; generating metadata for the dynamic loudness adjustment based on the loudness processed audio data and the original audio data; encoding the original audio data and the metadata into the bitstream; 16. An encoder comprising: one or more processors and a non-transitory memory configured to perform a method comprising:
[0259] EEE-B23. A system comprising an encoder as described in EEE-B22 for encoding original audio data and metadata to be subjected to dynamic loudness adjustment in a bitstream, and a decoder as described in EEE-B11 for performing metadata-based dynamic processing on the audio data to be reproduced.
[0260] EEE-B24. A computer program product including a computer readable storage medium having instructions configured, when executed by a device having processing capability, to cause the device to perform a method according to any one of EEE-B1 to EEE-B10 and EEE-B12 to EEE-B21.
[0261] A computer readable storage medium storing a computer program product according to EEE-B25.EEE-B24.
Claims
1. 1. A method for processing audio data to be played back, comprising the steps of: receiving, by a decoder, a bitstream comprising encoded audio data and metadata, the metadata comprising one or more Dynamic Range Control (DRC) sets, the DRC set including, for each of the DRC sets, an indication of whether the DRC set is configured to provide a dynamic loudness compensation effect; analyzing the metadata by the decoder to identify a DRC set configured to provide the dynamic loudness compensation effect; decoding the encoded audio data by the decoder to obtain decoded audio data; selecting, by the decoder, one of the identified DRC sets configured to provide the dynamic loudness compensation effect; extracting, by the decoder, one or more DRC gains from the bitstream corresponding to the selected DRC set; applying, by the decoder, the one or more DRC gains corresponding to the selected DRC set to the decoded audio data to obtain dynamic loudness compensated audio data; outputting the dynamic loudness compensated audio data for playback; The method includes:
2. 2. The method of claim 1, wherein the metadata includes a plurality of DRC sets configured to provide dynamic loudness adjustment, each of the plurality of DRC sets further associated with one or more playback conditions, and the selection is made in response to an indication of a playback condition provided to the decoder.
3. The method of claim 1 or 2, wherein the one or more DRC sets, in addition to providing a dynamic loudness compensation effect, are also configured to provide a dynamic range control.
4. The method according to claim 2 or 3, wherein the playback conditions include one or more of the following: a device type of the decoder, playback device characteristics, loudspeaker characteristics, loudspeaker settings, background noise characteristics, ambient noise characteristics, and acoustic environment characteristics.
5. 5. The method of claim 1, wherein the indication of whether the DRC set is configured to provide the dynamic loudness compensation effect is provided in parameters indicative of one or more effects that the DRC set provides.
6. 6. The method of claim 5, wherein the parameter indicating one or more effects provided by the DRC set is a drcSetEffect bitfield of an MPEG-D DRC bitstream, each bit of the drcSetEffect bitfield corresponding to a different effect, and one of the bits of the drcSetEffect bitfield corresponding to the dynamic loudness compensation effect.
7. 5. The method of claim 1, wherein the indication of whether the DRC set is configured to provide the dynamic loudness compensation effect is whether the DRC set is specified in a dynamic loudness compensation bitstream payload.
8. The method of claim 7 , wherein the dynamic loudness compensation bitstream payload is included in an extension field of a predefined bitstream syntax.
9. 9. The method of claim 8, wherein the extension field is a uniDrcConfigExtension field of an MPEG-D DRC bitstream, and the dynamic loudness compensation bitstream payload is included only for certain values of a uniDrcConfigExtType parameter.
10. 10. The method of claim 8 or 9, wherein a plurality of dynamic loudness compensation payloads specifying a plurality of DRC sets configured to provide the dynamic loudness compensation effect are included in an extension field of the already defined bitstream syntax.
11. 5. The method of claim 1, wherein the indication of whether the DRC set is configured to provide the dynamic loudness compensation effect is a field of a pre-existing configuration element of an already defined bitstream syntax.
12. The method of claim 11, wherein the field is a dynamicLoudCompDRCSet parameter and the pre-existing configuration element is a downmixInstructions element, a drcInstructionsBasic element or a drcInstructionsUniDRC element of an MPEG-D DRC bitstream.
13. 13. The method of claim 12, wherein the field is a pre-existing field reserved for future use.
14. 5. The method of claim 1, wherein the indication of whether the DRC set is configured to provide the dynamic loudness compensation effect is a field of an updated version of a pre-existing configuration element of an already defined bitstream syntax.
15. 15. The method of claim 14, wherein the field is a dynamicLoudCompDRCSet parameter and the updated version of the pre-existing configuration element is a downmixInstructionsV2 element or a drcInstructionsUniDrcV2 element.
16. 16. The method of claim 1, wherein an indication that a dynamic loudness compensation effect is desired is provided to the decoder via an interface, and the DRC set is selected in response to the indication provided to the decoder via the interface.
17. 17. The method of claim 16, wherein an indication regarding an additional desired effect is provided to the decoder via the interface, and the metadata includes a plurality of DRC sets configured to provide the dynamic loudness compensation effect, the selection being dependent on the additional desired effect.
18. 17. The method of claim 15 or 16, wherein the indication that a dynamic loudness compensation effect is desired is provided via a drcEffectTypeRequest parameter of a dynamicRangeControllerInterface payload.
19. 19. The method of any one of claims 1 to 18, wherein the metadata comprises one or more static loudness values configured to provide a static loudness adjustment to the decoded audio data.
20. 20. The method of claim 19, comprising applying the static loudness adjustment to the decoded audio data or the dynamic loudness compensated audio data in response to one or more of the static loudness values.
21. 2. The method of claim 1, wherein a first one of the DRC sets is configured to provide dynamic range control, and the first DRC set includes an indication that the selected DRC set, the DRC set configured to provide the dynamic loudness compensation effect, is configured to be applied in combination with the first DRC set.
22. 22. The method of claim 21, wherein the selected DRC set includes an indication of whether the one or more DRC gains corresponding to the selected DRC set may be applied only in combination with a DRC gain corresponding to the first DRC set.
23. 23. The method of claim 21 or 22, further comprising: extracting the DRC gains corresponding to the first DRC set from the bitstream; and applying the DRC gains corresponding to the first DRC set to the decoded audio data.
24. 24. A decoder for processing audio data to be reproduced, comprising one or more processors and non-transitory memory configured to carry out a method according to any one of claims 1 to 23.
25. A computer program product comprising a computer readable storage medium having instructions configured to cause a device to perform the method of any one of claims 1 to 23.
26. 26. A computer readable storage medium storing the computer program product of claim 25.
Citation Information
Patent Citations
Loudness adjustment for downmixed audio content
JP2016534669A
Dynamic range control of encoded audio extension metadatabase
JP2019148807A
Efficient DRC profile transmission
JP2021073814A
