Method and apparatus for processing audio data

By processing the loudness leveling and dynamic range compression of audio data in real-time encoding according to playback conditions in real-time encoding, the inconsistency problem of loudness leveling and dynamic range compression in the prior art under real-time encoding is solved, and the audio content is realized on different devices and Personalized playback experience in the environment.

CN119156665APending Publication Date: 2024-12-17DOLBY LABORATORIES LICENSING CORP +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202280095833.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-05-09
Filing Date
2022-08-24
Publication Date
2024-12-17

AI Technical Summary

Technical Problem

The prior art cannot effectively perform loudness leveling and dynamic range compression under real-time encoding, resulting in inconsistent playback experiences of audio content in different devices and environments, and existing standards need further revision to process metadata to achieve personalized settings.

Method used

Receive a bitstream containing audio data and loudness leveling metadata through the decoder, decode and determine processing parameters according to the playback conditions, apply these parameters to loudness leveling and dynamic range compression, output audio data suitable for playback, and encode the metadata to In bitstream to support personalized settings.

Benefits of technology

It realizes consistent loudness management and dynamic range control in different devices and environments, supports personalized experiences, ensuring loudness compliance and optimal playback of audio content.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119156665A_ABST
    Figure CN119156665A_ABST
Patent Text Reader

Abstract

Decoder devices, computer programs and methods of processing audio data for playback are described. The method includes receiving a bitstream containing encoded audio data and metadata, the metadata including DRC set (s), and an indication of whether the DRC set is configured to provide a loudness leveling effect for each DRC set. The metadata further includes personalized experience information. The method further includes identifying a DRC set configured to provide a dynamic range compensation effect; decoding the encoded audio data to obtain decoded audio data; selecting one of the identified DRC sets configured to provide a loudness leveling effect; extracting, from the bitstream, one or more DRC gains corresponding to the selected DRC set; applying one or more DRC gains corresponding to the selected DRC set to the decoded audio data to obtain dynamic loudness compensated audio data; and outputting the dynamic loudness compensated audio data for playback.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the following priority applications: U.S. Provisional Application No. 63 / 328,035 filed on April 6, 2022 and European Application No. 22172243.2 filed on May 9, 2022. Technical Field

[0003] The present disclosure generally relates to a method for metadata-based dynamic processing of audio data for playback, and in particular to determining and applying one or more processing parameters to audio data for loudness leveling and / or dynamic range compression in combination with personalized settings (dialogue enhancement, home or away commentary, etc.). The present disclosure further relates to a method for encoding audio data and metadata for loudness leveling and / or dynamic range compression into a bitstream. The present disclosure further relates to a corresponding decoder and encoder and a corresponding system and computer program product. The present disclosure further relates to a method for processing audio data for playback, a decoder for processing audio data for playback and a corresponding computer program product.

[0004] Although some embodiments will be described herein with particular reference to this disclosure, it will be appreciated that the disclosure is not limited to this field of use and may be applicable in a wider context. Background Art

[0005] Any discussion of the background art throughout the disclosure should in no way be considered as an admission that such art is widely known or forms part of the common general knowledge in the art.

[0006] Loudness is the individual's experience of sound pressure when playing back audio content. In film or television content, the loudness of the dialogue in the program has been found to be the most critical parameter in determining the listener's perception of program loudness.

[0007] In order to determine the average loudness of a program (whether the complete program or just the dialogue), an analysis must be performed on the entire program. Average loudness is often required for loudness compliance (e.g., the CALM Act in the United States) and is also used to calibrate dynamic range control (DRC) parameters. The dynamic range of a program is the difference between its quietest and loudest sounds. The dynamic range of a program depends on its content (e.g., an action movie may have a different and wider dynamic range than a documentary) and reflects the creator's intent. However, the ability of devices to play back audio content within the original dynamic range varies greatly. Therefore, in addition to loudness management, dynamic range control is another key factor in providing the best listening experience.

[0008] To perform loudness management and dynamic range control, the entire audio program or audio program segment must be analyzed, and the resulting loudness and DRC parameters may be passed along with the audio data or encoded audio data for application in a decoder or playback device.

[0009] When it is not possible to analyze an entire audio program or audio program segment prior to encoding, such as in real-time (dynamic) encoding, loudness processing or leveling is used to ensure loudness compliance and, where applicable, underlying dynamic range constraints based on playback requirements. This approach provides processed audio that is "optimized" for a single playback environment.

[0010] Therefore, there is an existing need for a metadata-based process that provides "raw" unprocessed audio along with accompanying metadata to allow a playback device to use the metadata to dynamically modify the audio based on device constraints, user requirements, and user settings (e.g., audio personalization settings).

[0011] In addition, industry audio standards include descriptions and syntax for implementing loudness control and / or loudness management. For example, the Moving Picture Experts Group (MPEG), a consortium of working groups jointly established by the International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC), sets standards for media coding, including audio coding. MPEG is organized under ISO / IEC SC 29, and the audio group is currently identified as Working Group (WG) 6. This WG-6 helped establish the MPEG-H 3D Audio standard, which includes compatibility with loudness control and / or loudness management (DRC) technology, however further revisions to the existing standard are needed to handle the above-mentioned metadata. Summary of the invention

[0012] According to a first aspect of the present disclosure, a method for performing metadata-based dynamic processing of audio data for playback is provided. The method may include receiving a bitstream by a decoder, the bitstream including audio data and metadata for loudness leveling. The method may further include decoding the audio data and the metadata by the decoder to obtain decoded audio data and metadata. The method may further include determining, by the decoder, one or more processing parameters for loudness leveling from the metadata based on playback conditions. The method may further include applying the determined one or more processing parameters to the decoded audio data to obtain processed audio data. And the method may include outputting the processed audio data for playback.

[0013] In some embodiments, the metadata may indicate processing parameters for loudness leveling for a plurality of playback conditions.

[0014] In some embodiments, said determining said one or more processing parameters may further include determining one or more processing parameters for dynamic range compression (DRC) based on said playback condition.

[0015] In some embodiments, the playback conditions may include one or more of: a device type of the decoder, characteristics of a playback device, characteristics of a speaker, speaker settings, characteristics of background noise, characteristics of ambient noise, a personalized experience selected on the device, and characteristics of the acoustic environment.

[0016] The personalized experience can be based on the version of the audio (such as language) or the user experience (such as enhanced dialogue). It can also include the ability to select different experiences or angles, such as choosing a home team commentary versus an away team commentary, or choosing a home crowd or away crowd as the background.

[0017] The personalized experience may depend on previous listening experiences and / or the capabilities of the listening device. Alternatively, the personalized experience may be selected by the device based on previous listening preferences (including by including external data via the cloud).

[0018] Personalized experiences can be encoded in real-time, such as sports with home and away commentary, where loudness leveling will be used to ensure that the audio meets loudness compliance (e.g., CALM Act in the United States). For metadata-based solutions, loudness leveling metadata will be generated for each of the various personalized experiences, device capabilities, and may also include DRC metadata.

[0019] In some embodiments, the determining the one or more processing parameters may further include selecting, by the decoder, at least one of a DRC sequence set DRCSet, an equalizer parameter set EQSet, and a downmix corresponding to the playback condition.

[0020] In some embodiments, said determining said one or more processing parameters may further comprise identifying a metadata identifier indicative of said at least one selected DRCSet, EQSet and downmix to determine said one or more processing parameters from said metadata.

[0021] In some embodiments, the metadata may include one or more processing parameters related to average loudness values ​​and, optionally, one or more processing parameters related to dynamic range compression characteristics.

[0022] In some embodiments, the bitstream may further include additional metadata for static loudness adjustment to be applied to the decoded audio data.

[0023] In some embodiments, the bitstream may be an MPEG-D DRC bitstream, and the presence of metadata may be signaled based on an MPEG-D DRC bitstream syntax.

[0024] In some embodiments, a uniDrcConfigExtension() element may be used to carry metadata as a payload.

[0025] In some embodiments, the metadata may include one or more metadata payloads, wherein each metadata payload may include multiple parameter and identifier sets, each set including at least one of a DRCSet identifier drcSetId, an EQSet identifier eqSetId, and a downmix identifier downmixId and a combination of one or more processing parameters associated with the identifiers in the set.

[0026] In some embodiments, determining the one or more processing parameters may involve selecting a set from the multiple sets in the payload based on the at least one DRCSet, EQSet and downmix selected by the decoder, wherein the one or more processing parameters determined by the decoder may be the one or more processing parameters associated with an identifier in the selected set.

[0027] According to a second aspect of the present disclosure, a decoder for performing metadata-based dynamic processing of audio data for playback is provided. The decoder may include one or more processors and a non-transitory memory configured to perform a method, the method comprising receiving, by the decoder, a bitstream including audio data and metadata for loudness leveling; decoding, by the decoder, the audio data and the metadata to obtain decoded audio data and metadata; determining, by the decoder, one or more processing parameters for loudness leveling from the metadata based on playback conditions; applying the determined one or more processing parameters to the decoded audio data to obtain processed audio data; and outputting the processed audio data for playback.

[0028] According to a third aspect of the present disclosure, a method for encoding audio data and metadata for loudness leveling into a bitstream is provided. The method may include inputting original audio data into a loudness leveler for loudness processing to obtain loudness-processed audio data as an output of the loudness leveler. The method may further include generating the metadata for loudness leveling based on the loudness-processed audio data and the original audio data. And the method may include encoding the original audio data and the metadata into the bitstream.

[0029] In some embodiments, the method may further comprise generating additional metadata for static loudness adjustment to be used by a decoder.

[0030] In some embodiments, generating metadata may include comparing the loudness processed audio data with the original audio data, wherein the metadata may be generated based on a result of the comparison.

[0031] In some embodiments, generating metadata may further include measuring loudness over one or more predefined time periods, wherein the metadata may be generated further based on the measured loudness.

[0032] In some embodiments, the measuring may include measuring the overall loudness of the audio data.

[0033] In some embodiments, the measuring may include measuring the loudness of dialogue in the audio data.

[0034] In some embodiments, the bitstream may be an MPEG-D DRC bitstream, and the presence of the metadata may be signaled based on an MPEG-D DRC bitstream syntax.

[0035] In some embodiments, a uniDrcConfigExtension() element may be used to carry metadata as a payload.

[0036] In some embodiments, the metadata may include one or more metadata payloads, wherein each metadata payload may include multiple parameter and identifier sets, each set including a combination of at least one of a DRCSet identifier drcSetId, an EQSet identifier eqSetId, and a downmix identifier downmixId and one or more processing parameters associated with the identifiers in the set, and wherein the one or more processing parameters may be parameters for loudness leveling by a decoder.

[0037] In some embodiments, the at least one of the drcSetId, the eqSetId and the downmixId may be related to at least one of a DRC sequence set DRCSet, an equalizer parameter set EQSet and a downmix to be selected by the decoder.

[0038] According to a fourth aspect of the present disclosure, an encoder for encoding original audio data and metadata for loudness leveling in a bitstream is provided. The encoder may include one or more processors and a non-transitory memory configured to perform a method, the method comprising inputting original audio data into a loudness leveler for loudness processing to obtain loudness-processed audio data as output of the loudness leveler; generating the metadata for loudness leveling based on the loudness-processed audio data and the original audio data; and encoding the original audio data and the metadata into the bitstream.

[0039] According to a fifth aspect of the present disclosure, a system of an encoder for encoding original audio data and metadata for loudness leveling in a bitstream and a decoder for performing metadata-based dynamic processing on audio data for playback is provided.

[0040] According to a sixth aspect of the present disclosure, a computer program product is provided, the computer program product comprising a computer-readable storage medium having instructions, the instructions being suitable for causing a device having processing capabilities to perform, when executed by the device, a method for performing metadata-based dynamic processing of audio data for playback or a method for encoding audio data and metadata for loudness leveling into a bitstream.

[0041] According to a seventh aspect of the present disclosure, a computer-readable storage medium storing the computer program product described herein is provided.

[0042] According to an eighth aspect of the present disclosure, a method for processing audio data for playback is provided. The method may include receiving, by a decoder, a bitstream including encoded audio data and metadata, wherein the metadata includes one or more dynamic range control (DRC) sets and an indication of whether the DRC set for each DRC set is configured to provide a loudness leveling effect. The method may further include parsing the metadata by the decoder to identify the DRC set configured to provide the loudness leveling effect. The method may further include decoding, by the decoder, the encoded audio data to obtain decoded audio data. The method may further include selecting, by the decoder, one of the identified DRC sets configured to provide the loudness leveling effect. The method may further include applying, by the decoder, the one or more DRC gains corresponding to the selected DRC set to the decoded audio data to obtain dynamically loudness compensated audio data. And the method may include outputting the dynamically loudness compensated audio data for playback.

[0043] In some embodiments, the metadata may include a plurality of DRC sets configured to provide the loudness leveling, wherein each of the plurality of DRC sets may also be associated with one or more playback conditions, and wherein the selecting may be performed in response to an indication of a playback condition provided to the decoder.

[0044] In some embodiments, in addition to providing a loudness leveling effect, the one or more DRC sets may also be configured to provide dynamic range control.

[0045] In some embodiments, the playback conditions may include one or more of: a device type of the decoder, characteristics of a playback device, characteristics of a speaker, speaker settings, characteristics of background noise, characteristics of ambient noise, and characteristics of an acoustic environment.

[0046] In some embodiments, an indication of whether the DRC set is configured to provide the loudness leveling effect may be provided in a parameter indicative of one or more effects provided by the DRC set.

[0047] In some embodiments, the parameter indicating one or more effects provided by the DRC set may be a drcSetEffect bit field of an MPEG-D DRC bitstream, wherein respective bits of the drcSetEffect bit field correspond to different effects, and one of the bits of the drcSetEffect bit field corresponds to the loudness leveling effect.

[0048] In some embodiments, the indication of whether the DRC set is configured to provide the loudness leveling effect may be whether the DRC set is specified in a loudness leveling bitstream payload.

[0049] In some embodiments, the loudness leveling bitstream payload may be included in an extension field of a previously defined bitstream syntax.

[0050] In some embodiments, the extension field may be a uniDrcConfigExtension field of an MPEG-D DRC bitstream, and the loudness leveling bitstream payload may be included only for specific values ​​of the uniDrcConfigExtType parameter.

[0051] In some embodiments, a plurality of loudness leveling payloads specifying a plurality of DRC sets configured to provide the loudness leveling effect may be included in an extension field of the previously defined bitstream syntax.

[0052] In some embodiments, the indication of whether the DRC set is configured to provide the loudness leveling effect may be a field of a pre-existing configuration element of a previously defined bitstream syntax.

[0053] In some embodiments, the field may be a levelingPresent parameter and the pre-existing configuration element may be a downmixInstructions element, a drcInstructionsBasic element, or a drcInstructionsUniDRC element of an MPEG-D DRC bitstream.

[0054] In some embodiments, the field may be a pre-existing field reserved for future use.

[0055] In some embodiments, the indication of whether the DRC set is configured to provide the loudness leveling effect may be a field of an updated version of a pre-existing configuration element of a previously defined bitstream syntax.

[0056] In some embodiments, the field may be a levelingPresent parameter and the updated version of the pre-existing configuration element may be a downmixInstructionsV2 element or a drcInstructionsUniDrcV2 element.

[0057] In some embodiments, an indication of a desired loudness leveling effect may be provided to the decoder via an interface, and the DRC set may be selected in response to the indication provided to the decoder via the interface.

[0058] In some embodiments, an indication of an additional desired effect may be provided to the decoder via the interface, the metadata may include a plurality of DRC sets configured to provide the loudness leveling effect, and the selection may be dependent on the additional desired effect.

[0059] In some embodiments, an indication of a desired loudness leveling effect may be provided via the drcEffectTypeRequest parameter of the dynamicRangeControllerInterface payload.

[0060] In some embodiments, the metadata may include one or more static loudness values ​​configured to provide a static loudness adjustment to the decoded audio data.

[0061] In some embodiments, a static loudness adjustment may be applied to the decoded audio data or the dynamic loudness compensated audio data in response to one or more of the static loudness values.

[0062] In some embodiments, a first one of the DRC sets may be configured to provide dynamic range control, and the first DRC set may include an indication that the selected DRC set configured to provide the loudness leveling effect may be configured to be applied in conjunction with the first DRC set.

[0063] In some embodiments, the selected DRC set may include an indication of whether the one or more DRC gains corresponding to the selected DRC set may be applied only in combination with DRC gains corresponding to the first DRC set.

[0064] In some embodiments, DRC gains corresponding to the first DRC set may be extracted from the bitstream and applied to the decoded audio data.

[0065] According to a ninth aspect of the present disclosure, a decoder for processing audio data for playback is provided. The decoder may include one or more processors and a non-transitory memory configured to execute the method for processing audio data for playback as described above.

[0066] According to a tenth aspect of the present disclosure, a computer program product is provided, the computer program product comprising a computer-readable storage medium having instructions, the instructions being suitable for causing a device to execute the method of processing audio data for playback as described above.

[0067] According to an eleventh aspect of the present disclosure, a computer-readable storage medium storing the computer program product described herein is provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Example embodiments of the present disclosure will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0069] Figure 1 An example of a decoder for metadata-based dynamic processing of audio data for playback is illustrated.

[0070] Figure 2 An example of a method for metadata-based dynamic processing of audio data for playback is illustrated.

[0071] Figure 3A An example of an encoder for encoding raw audio data and metadata for loudness leveling in a bitstream is illustrated.

[0072] Figure 3B An example of an encoder for encoding raw audio data and metadata for loudness leveling and personalization in a bitstream is illustrated.

[0073] Figure 4A An example of a method of encoding audio data and metadata for loudness leveling into a bitstream is illustrated.

[0074] Figure 4B An example of a method of encoding audio data and metadata for personalized loudness leveling into a bitstream is illustrated.

[0075] Figure 5 An example of a device comprising one or more processors and non-transitory memory configured to perform the methods described herein is illustrated.

[0076] Figure 6 An example of a method of processing audio data for playback is illustrated. DETAILED DESCRIPTION

[0077] Overview

[0078] The average loudness of a program or dialog is the primary parameter or value used for loudness compliance of a broadcast or streamed program. Average loudness is typically set to -24 or -23 LKFS. In the case of an audio codec that supports loudness metadata, this single loudness value representing the loudness of the entire program is carried in the bitstream. Using this value during the decoding process allows gain adjustments to be made, resulting in predictable playback levels so that the program plays back at a known, consistent level. Therefore, it is very important that this loudness value is set correctly and accurately. However, since average loudness depends on measuring the entire program prior to encoding, this is not possible for real-time situations such as dynamic encoding with unknown loudness and dynamic range variations.

[0079] When it is not possible to measure the loudness of an entire file prior to encoding, a dynamic loudness leveler is often used to modify or contour the audio data prior to encoding so that it meets the required loudness. This type of loudness management is often viewed as a poor method of meeting compliance because it often changes the dynamic range interrelationships in the audio content and, therefore, can change the creative intent. This is especially true when it is desirable to distribute one audio asset to all playback devices, which is one of the benefits of metadata-driven codecs and delivery systems.

[0080] In some approaches, the audio content is mixed to a desired target loudness and the corresponding loudness metadata is set to that value. In these cases, the loudness leveler can still be used, as it will be used to help guide the audio content to the target loudness, but it will not be as "active" and will only be used when the audio content starts to deviate from the desired target loudness.

[0081] In view of the above, the method and apparatus described herein are intended to enable real-time processing situations (also referred to as dynamic processing situations) to be driven by metadata as well. Metadata allows loudness leveling and dynamic range compression to be performed in real-time situations. The described method and apparatus advantageously achieve the following:

[0082] Use real-time loudness adjustment and DRC in MPEG-D DRC and MPEG-H 3D Audio syntax;

[0083] Use real-time loudness adjustment and DRC in conjunction with downmixId;

[0084] Use real-time loudness adjustment and DRC in conjunction with drcSetId;

[0085] Use real-time loudness adjustment and DRC in conjunction with eqSetId;

[0086] Use real-time loudness adjustment and DRC in conjunction with personalized settings identified using mae_groupID and mae_groupPresetID.

[0087] That is, according to the decoder settings (e.g., DRCSet, EQSet, downmix and personalization settings), the decoder can search for appropriate parameter and identifier sets in a given payload based on the syntax by matching the above settings with identifiers. Then, the parameters included in the set whose identifier best matches the setting can be selected as processing parameters for loudness leveling to be applied to the received original audio data for correction.

[0088] Furthermore, multiple parameter sets (multiple dynLoudCompValue instances) may be transmitted for dynamics processing.

[0089] In addition to correcting overall loudness, metadata-driven loudness leveling can also be used to "center" the DRC gain calculation and application. This centering can be a result of correcting the loudness of the content through loudness leveling and the typical calculation and application of DRC. In this sense, the metadata used for loudness leveling can be said to be used to calibrate the DRC parameters.

[0090] Metadata-based dynamic processing of audio data

[0091] refer to Figure 1 The decoder 100 for performing metadata-based dynamic processing of audio data for playback is described in an example. The decoder 100 may include one or more processors and non-transitory memory configured to perform a method including Figure 2 The process is illustrated by steps S101 to S105 in the example of FIG.

[0092] The decoder 100 may receive a bitstream including audio data and metadata, and may be capable of outputting unprocessed (raw) audio data, processed audio data after applying dynamic processing parameters determined according to the metadata, and / or the metadata itself, as required. The metadata includes personalized metadata (including interactivity metadata) as described herein. The personalized metadata may include information for identifying a personalized experience.

[0093] The personalized experience can be based on the version of the audio (such as language) or the user experience (such as enhanced dialogue). It can also include the ability to select different experiences or angles, such as choosing a home team commentary versus an away team commentary, or choosing a home crowd or away crowd as the background.

[0094] The personalized experience may depend on previous listening experiences and / or the capabilities of the listening device. Alternatively, the personalized experience may be selected by the device based on previous listening preferences (including by including external data via the cloud).

[0095] Personalized experiences can be encoded in real-time, such as sports with home and away commentary, where loudness leveling will be used to ensure that the audio meets loudness compliance (e.g., CALM Act in the United States). For metadata-based solutions, loudness leveling metadata will be generated for each of the various personalized experiences, device capabilities, and may also include DRC metadata.

[0096] refer to Figure 2 In the example of, in step S101, the decoder 100 may receive a bitstream including audio data and metadata for loudness leveling and optional dynamic range compression (DRC). The audio data may be encoded audio data, and the audio data may further be unprocessed. That is, the audio data may be raw audio data.

[0097] Although the format of the bitstream is not limited, in an embodiment, the bitstream may be an MPEG-D DRC bitstream. The presence of metadata for dynamically processing the audio data may then be signaled based on the MPEG-D DRC bitstream syntax. In an embodiment, a uniDrcConfigExtension() element may be used to carry the metadata as a payload, as further detailed below. Alternatively, the bitstream may be an MPEG-H 3D Audio compatible bitstream, and the metadata may be signaled using MPEG-H 3D Audio bitstream syntax (e.g., using MPEG-H Audio Stream (MHAS) packets).

[0098] In step S102, the audio data and metadata may then be decoded by a decoder to obtain decoded audio data and metadata. In an embodiment, the metadata may include one or more processing parameters related to an average loudness value, and optionally one or more processing parameters related to a dynamic range compression characteristic. The metadata allows dynamic or real-time corrections to be applied. For example, when encoding and decoding a live real-time playback, it is desirable to apply "real-time" or dynamic loudness metadata to ensure that the live playback audio is properly loudness managed. The metadata further includes personalized metadata referenced herein, which may be general metadata for allowing processing by various devices and implementing various device capabilities.

[0099] In step S103, the decoder then determines one or more processing parameters for loudness leveling from the metadata based on the playback conditions. In an embodiment, the playback conditions may include one or more of the following: the device type of the decoder, the characteristics of the playback device, the characteristics of the speakers, the speaker settings, the characteristics of the background noise, the characteristics of the ambient noise, and the characteristics of the acoustic environment. In addition, the playback conditions may include information about the personalized experience as discussed herein. Considering the playback conditions allows the decoder to select processing parameters for loudness leveling in a targeted manner for device and environmental constraints.

[0100] In an embodiment, the process of determining the one or more processing parameters in step S103 may further include selecting, by the decoder, at least one of a DRC sequence set DRCSet, an equalizer parameter set EQSet, and a downmix corresponding to the playback condition. Therefore, the at least one of DRCSet, EQSet, and downmix is ​​related to or indicates the individual device and environment constraints caused by the playback condition.

[0101] In an embodiment, the determination process in step S103 may further include identifying a metadata identifier indicating the at least one selected DRCSet, EQSet and DownmixSet to determine the one or more processing parameters from the metadata. Thus, the metadata identifier enables the metadata to be connected to the corresponding selected DRCSet, EQSet and / or downmix, and therefore to the corresponding playback condition.

[0102] In an embodiment, the metadata may include one or more metadata payloads (e.g., dynLoudComp() payloads, as shown in Table 5 below), wherein each metadata payload may include a plurality of parameters (e.g., parameter dynLoudCompValue) and a set of identifiers, each set including a combination of at least one of a DRCSet identifier drcSetId, an EQSet identifier eqSetId, a downmix identifier downmixId, and a personalization identifier mae_groupID / mae_groupPresetID and one or more processing parameters associated with the identifier in the set. That is, each payload may include an array of entries, each entry including a processing parameter and an identifier (e.g., drcSetId, eqSetId, downmixId, mae_groupID / mae_groupPresetID). In a further embodiment, the determination in step S103 may therefore involve selecting a set from the plurality of sets in the payload based on the at least one DRCSet, EQSet, and downmix selected by the decoder, wherein the one or more processing parameters determined in step S103 may be the one or more processing parameters associated with the identifier in the selected set. That is, according to the settings present in the decoder (e.g., DRCSet, EQSet, downmix, personalization settings), the decoder can search for the appropriate set of parameters and identifiers in a given payload by matching the above settings with the identifiers. Then, the parameters included in the set whose identifier best matches the setting can be selected as the processing parameters for loudness leveling.

[0103] In step S104, the determined one or more processing parameters may then be applied by the decoder to the decoded audio data to obtain processed audio data. Thus, the processed audio data (eg, live real-time audio data) is appropriately loudness managed.

[0104] In step S105 , the processed audio data may then be output for playback.

[0105] In an embodiment, the bitstream may further include additional metadata for static loudness adjustment to be applied to the decoded audio data.Static loudness adjustment refers to processing performed for general loudness normalization, compared to dynamic processing in real-time situations.

[0106] Carrying metadata for dynamics processing separately from additional metadata for general loudness normalization allows for "real-time" corrections not to be applied.

[0107] For example, when encoding and decoding live real-time playback, it is desirable to apply dynamics processing to ensure that the live playback audio is properly loudness managed. However, for non-real-time playback or transcoding where dynamics correction is not desired or required, it is not necessary to apply dynamics processing parameters determined from metadata.

[0108] By further keeping the (dynamic / real-time) metadata for dynamics processing separate from the additional metadata, the originally unprocessed content can be retained if desired. The original audio is encoded along with the metadata. This allows playback devices to selectively apply dynamics processing and further enables playback of original audio content on high-end devices capable of playing back original audio.

[0109] As mentioned above, keeping dynamic loudness metadata distinct from long-term loudness measurements / information such as contentLoudness (in ISO / IEC 23003-4) has some advantages. If combined, the loudness of the content (or what it should be after applying the dynamic loudness metadata) will not indicate the actual loudness of the content, as the available metadata will be a synthesized value. In addition to removing this ambiguity of what the content loudness (or program or anchor loudness) is, this will be particularly beneficial in some cases:

[0110] Keeping metadata for dynamics processing separate allows a decoder or playback device to turn off application of dynamics processing and apply an implemented real-time loudness leveler instead, to avoid cascading leveling. This may happen, for example, if the device's own real-time leveling solution is superior to the solution used with the audio codec, or if, for example, the device's own real-time leveling solution cannot be disabled and therefore will always be active, and the resolution in further processing results in a compromised playback experience.

[0111] Keeping the metadata for dynamics processing separate further allows transcoding to codecs that do not support loudness leveling and wish to apply their own loudness processing before re-encoding.

[0112] Live to broadcast with a single encoding of a live broadcast would be another example. Dynamic processing metadata can be used or stored for archiving or on-demand services. Thus, for archiving or on-demand services, more accurate or compliant loudness measurements can be performed based on the entire program and the appropriate metadata reset.

[0113] This is also beneficial for use cases where a fixed target loudness is used throughout the workflow, such as in R128-compatible situations where -23LKFS (or LUFS) is recommended. In this case, adding the dynamics processing metadata is a "safety" measure, where the content is assumed to be close to the desired target, and adding the dynamics processing metadata is a secondary check. Therefore, having the ability to turn it off is desirable.

[0114] Encodes audio data and metadata for loudness leveling

[0115] refer to Figure 3A , Figure 3B 4, describing an encoder for encoding raw audio data and metadata for loudness leveling and optional dynamic range compression DRC in a bitstream, the encoder may include a processor configured to perform the following steps: Figure 4A and Figure 4B The steps of the examples illustrate the method of the process of one or more processors and non-transitory memory.

[0116] refer to Figure 3A and Figure 4A For example, in step S201, original audio data may be input into the loudness equalizer 201 for loudness processing to obtain loudness-processed audio data as the output of the loudness equalizer 201.

[0117] In step S202, metadata for loudness leveling may then be generated based on the loudness processed audio data and the original audio data. Appropriate smoothing and time framing may be used to reduce artifacts.

[0118] In an embodiment, step S202 may include comparing, by the analyzer 202, the loudness processed audio data with the original audio data, wherein metadata may be generated based on the result of the comparison. The metadata thus generated may simulate the effect of a leveler at the decoder site. The metadata may include:

[0119] - Gain (wideband and / or multi-band) processing parameters such that when applied to the original audio, loudness compliant audio will be produced for playback;

[0120] -Processing parameters that describe audio dynamics, such as the following:

[0121] oPeak – Sample and true peak

[0122] o Short-term loudness value

[0123] o Changes in short-term loudness values.

[0124] In an embodiment, step S202 may further include measuring loudness within one or more predefined time periods by the analyzer 202, wherein the metadata may be further generated based on the measured loudness. In an embodiment, the measuring may include measuring the overall loudness of the audio data. Alternatively or additionally, in an embodiment, the measuring may include measuring the loudness of dialogue in the audio data.

[0125] In step S203, the original audio data and metadata may then be encoded into a bitstream. Although the format of the bitstream is not limited, in an embodiment, the bitstream may be an MPEG-D DRC bitstream, and the presence of metadata may be signaled based on the MPEG-D DRC bitstream syntax. In this case, in an embodiment, a uniDrcConfigExtension() element may be used to carry metadata as a payload, as further described below.

[0126] In an embodiment, the metadata may include one or more metadata payloads, wherein each metadata payload may include a plurality of parameter and identifier sets, each set including at least one of a DRCSet identifier drcSetId, an EQSet identifier eqSetId, a downmix identifier downmixId, and a personalization identifier mae_groupID / mae_groupPresetID and a combination of one or more processing parameters associated with the identifiers in the set, and wherein the one or more processing parameters may be parameters for loudness leveling by a decoder. In this case, in an embodiment, the at least one of drcSetId, eqSetId, and downmixId may be associated with at least one of a DRC sequence set DRCSet, an equalizer parameter set EQSet, and a downmix to be selected by a decoder.

[0127] In an embodiment, the method may further comprise generating additional metadata for static loudness adjustment to be used by the decoder.Keeping metadata for loudness leveling and additional metadata separate in the bitstream and further encoding the original audio data into the bitstream has several advantages as detailed above.

[0128] refer to Figure 3B and Figure 4B In the example of , in step S200B, the original audio data can be input into the personalization block 200B. The personalization block 200B can process the original audio (and / or additional information or metadata) to determine and / or generate personalized audio and personalized metadata.

[0129] In step S201B, the personalized audio data from 200B is input into the loudness equalizer 201B for loudness processing to obtain the loudness processed personalized audio data as the output of the loudness equalizer 201B. The loudness equalizer 201B can be used with respect to Figure 3A and Figure 3B The loudness leveler 201 described is similar or identical.

[0130] In step S202B, metadata for loudness leveling may then be generated based on the loudness processed personalized audio data output from S201B / analyzer 201B and the personalized audio data output from S200B / personalization block 200B. In an embodiment, step S202 may include comparing, by the analyzer 202B, the loudness processed audio data with the original audio data, wherein metadata may be generated based on a result of the comparison. Figure 3B Step 202B can be based on Figure 3A The process is performed as described in step S202.

[0131] In step S203B, personalized audio data from S200B / 200B and metadata from S200B and / or S202B may be encoded into a bitstream. Step 203B outputs the encoded audio data and metadata, including the personalized metadata described herein. Although the format of the bitstream is not limited, in an embodiment, the bitstream may be an MPEG-D DRC bitstream, and the presence of metadata may be signaled based on the MPEG-D DRC bitstream syntax. Alternatively, the bitstream may be an MPEG-H 3D Audio compatible bitstream, and metadata may be signaled based on the MPEG-H 3D Audio bitstream syntax. In an embodiment, the metadata may include one or more metadata payloads, such as information about Figure 3A The step S203 described herein. Figure 3A and Figure 3B ) The method described in the present invention may be implemented on a decoder or an encoder, respectively, wherein the decoder and the encoder may include one or more processors and non-volatile memory configured to perform the method. Figure 5 An example of a device having such processing capabilities is illustrated in the example of , which shows the device 300 comprising two processors 301 and a non-transitory memory 302 .

[0132] It should be noted that this article is about Figure 3A and Figure 3BThe method can be further implemented on a system of an encoder and a decoder as described herein, wherein the encoder is used to encode the original audio data and metadata for loudness leveling and optional dynamic range compression DRC in a bitstream, and the decoder is used to perform metadata-based dynamic processing on the audio data for playback.

[0133] The method may further be implemented as a computer program product comprising a computer readable storage medium having instructions adapted to, when executed by a device having processing capabilities, cause the device to perform the method. The computer program product may be stored on a computer readable storage medium.

[0134] MPEG-D DRC modified bitstream syntax

[0135] In the following, it will be described how the MPEG-D DRC bitstream syntax as described in ISO / IEC 23003-4 may be modified according to the embodiments described herein.

[0136] The MPEG-D DRC syntax (eg, the loudnessInfoSetExtension() element shown in Table 1 below) may be extended to also carry dynamics processing metadata as frame-based dynLoudComp updates.

[0137] For example, another switch-case UNIDRCLOUDEXT_DYNLOUDCOMP may be added in the loudnessInfoSetExtension() element, as shown in Table 1. The switch-case UNIDRCLOUDEXT_DYNLOUDCOMP may be used to identify a new element dynLoudComp(), as shown in Table 5. The loudnessInfoSetExtension() element may be an extension of the loudnessInfoSet() element, as shown in Table 2.

[0138] Further, the loudnessInfoSet() element may be a part of the uniDRC() element, as shown in Table 3.

[0139]

[0140]

[0141] Table 1: Syntax of the loudnessInfoSetExtension() element

[0142]

[0143] Table 2: Syntax of the loudnessInfoSet() element

[0144]

[0145] Table 3: Syntax of the uniDRC() element

[0146] symbol Values ​​of loudnessInfoSetExtType Purpose UNIDRCLOUDEXT_TERM 0×0 Termination Tag UNIDRCLOUDEXT_EQ 0×1 Expanding for equalization UNIDRCLOUDEXT_DYNLOUDCOMP 0×2 Expanded for dynamic processing (reserve) (all remaining values) For future use

[0147] Table 4: loudnessInfoSet extension type

[0148] New dynLoudComp():

[0149]

[0150] Table 5: Syntax of the dynLoudComp() element

[0151] drcSetId enables the application of dynLoudComp per DRC set (associated with metadata).

[0152] eqSetId enables dynLoudComp to be used with different settings of the equalization tool.

[0153] downmixId enables the application of dynLoudComp per DownmixID.

[0154] • mae_groupID enables application of dynLoudComp according to personalized settings set at the device. mae_groupPresetID enables application of dynLoudComp according to personalized preset settings set at the device.

[0155] In some cases, in addition to the above parameters, it may be beneficial for the dynLoudComp() element to include a methodDefinition parameter (e.g., specified by 4 bits) that specifies a loudness measurement method used to derive dynamic program loudness metadata (e.g., anchor loudness, program loudness, short-term loudness, transient loudness, etc.) and / or a measurementSystem parameter (e.g., specified by 4 bits) that specifies a loudness measurement system used to measure dynamic program loudness metadata (e.g., EBU R.128, ITU-R BS-1770 with or without preprocessing, ITU-R BS-1771, etc.). For example, such parameters may be included between the downmixId and dynLoudCompValue parameters in the dynLoudComp() element.

[0156] Alternative grammar 1

[0157]

[0158]

[0159] Table 6: Syntax of the loudnessInfoSetExtension() element

[0160] symbol Values ​​of loudnessInfoSetExtType Purpose UNIDRCLOUDEXT_TERM 0×0 Termination Tag UNIDRCLOUDEXT_EQ 0×1 Expanding for equalization UNIDRCLOUDEXT_DYNLOUDCOMP 0×2 Expanded for dynamic processing (reserve) (all remaining values) For future use

[0161] Table 7: loudnessInfoSet extension type

[0162]

[0163]

[0164] Table 8: Syntax of loudnessInfoV2() payload

[0165] In some cases, it may be beneficial to modify the syntax shown in Table 8 so that the levelingPresent and (if levelingPresent == 1) dynLoudCompValue parameters are within the measurementCount loop of the loudnessInfoV2() payload, rather than following the reliability parameter outside the measurementCount loop. Additionally, it may be beneficial to set dynLoudCompValue to 0 in the case where levelingPresent is 0.

[0166] Alternative Grammar 2

[0167] Alternatively, the dynLoudComp() element may be placed within a uniDrcGainExtension() element.

[0168]

[0169]

[0170] Table 9: Syntax of the uniDrcGain() element

[0171]

[0172] Table 10: Syntax of the uniDrcGainExtension() element

[0173] symbol Value of uniDrcGainExtType Purpose UNIDRCGAINEXT_TERM 0×0 Termination Tag UNIDRCGAINEXT_DYNLOUDCOMP 0×1 Expanded for dynamic processing (reserve) (all remaining values) For future use

[0174] Table 11: UniDrc gain extension types

[0175] Semantics

[0176] dynLoudCompValue This field contains the value of dynLoudCompDb. The value is encoded according to Table 12. The default value is 0dB.

[0177]

[0178] Table 12: Encoding of the dynLoudCompValue field

[0179] In one example, dynLoudCompValue may be transmitted in the bitstream using the syntax used to transmit the DRC gains associated with a DRC set.

[0180] Updated MPEG-D DRC Loudness Normalization Processing

[0181]

[0182] Table 13: Loudness Normalization Processing

[0183] Pseudocode for selection and processing of dynLoudComp

[0184] Additional parameters like mae_groupID or mae_PresetID identifying different combinations of audio elements personalized for the user may be used in the selection process.

[0185]

[0186]

[0187]

[0188] In some cases, in addition to the selection process shown in the pseudocode above (e.g., considering drcSetId, eqSetID, downmixId, mae_groupID, and mae_PresetID when selecting the dynLoudCompValue parameter), it may be beneficial for the selection process to also consider the methodDefinition parameter and / or the measurementSystem parameter when selecting the dynLoudCompValue parameter.

[0189] Alternative updated MPEG-D DRC loudness normalization processing

[0190]

[0191] Table 14: Alternative Loudness Normalization Processes

[0192] In the case of using the alternative loudness normalization process of Table 14 above, the above loudness normalization process pseudo code can be replaced by the following alternative loudness normalization process pseudo code. It should be noted that a default value of dynLoudCompDb (e.g., 0 dB) can be assumed to ensure that the value of dynLoudCompDb is defined even in the absence of loudness leveling metadata in the bitstream.

[0193] Alternative grammar 3

[0194] In some cases, it may be beneficial to combine the syntax described above in Tables 1 to 5 with Alternative Syntax 1 described above in Tables 6 to 8, as shown in the following table, to increase the flexibility of loudness leveling value transmission.

[0195]

[0196] Table 15: Alternative syntax 3 for the loudnessInfoSetExtension() element

[0197] symbol Values ​​of loudnessInfoSetExtType Purpose UNIDRCLOUDEXT_TERM 0×0 Termination Tag UNIDRCLOUDEXT_EQ 0×1 Expanding for equalization UNIDRCLOUDEXT_DYNLOUDCOMP 0×2 Expanded for dynamic processing UNIDRCLOUDEXT_DYNLOUDCOMP2 0×3 Expanded for dynamic processing (reserve) (all remaining values) For future use

[0198] Table 16: Alternative syntax for the loudnessInfoSet extension type3

[0199]

[0200] Table 17: Alternative syntax for loudnessInfoV2() payload3

[0201] Alternative syntax for dynLoudComp()3:

[0202]

[0203] Table 18: Alternative syntax for the dynLoudComp() element 3 Interface extension syntax

[0204] In some cases, it may be beneficial to allow, for example, an end-user to control whether loudness leveling is performed even when loudness leveling information is present in the received bitstream. Such control may be provided by updating the MPEG-D DRC interface syntax to include an additional interface extension (e.g., UNIDRCINTERFACEEXT_LEVELING) containing a loudness leveling control interface payload (e.g., levelingControlInterface()), as shown in the following table.

[0205]

[0206]

[0207] Table 19: Syntax of uniDRCInterfaceExtension() payload

[0208]

[0209] Table 20: Syntax of levelingControlInterface() payload

[0210]

[0211] Table 21: UniDRC interface extension types

[0212] Interface extension semantics

[0213] loudnessLevelingOn This flag signals whether loudness leveling should be turned on or off. If loudness leveling is turned on, the flag should be equal to 1. If loudness leveling is turned off, the flag should be equal to 0. The default value is 1.

[0214] Additional methods for implementing loudness leveling

[0215] In addition to the above, additional methods for implementing loudness leveling are possible. For example, as an alternative to including a parameter indicating a specific loudness leveling value to be applied to the signal, a specific set of DRC gains that already include loudness leveling can be defined, so that when the decoder applies these specific DRC gains, loudness leveling is applied. The DRC gain sets that are suitable for applying such loudness leveling can be identified by specific bitstream elements, as explained further below. The benefit of this approach is that no explicit loudness leveling gains need to be transmitted in addition to other loudness information.

[0216] Additionally, the user may specify through the interface that DRC gains including loudness leveling should be selected and applied, as explained further below.

[0217] Finally, additional bitstream elements may be provided to allow tighter creative control over whether and how loudness leveling is performed. For example, a flag may indicate whether loudness leveling may be turned off by the user. In this case, if the content creator allows loudness leveling to be disabled, loudness leveling will be applied or not applied based on the user's specification.

[0218] Specify a DRC set with loudness leveling via the DRC Set Effect field

[0219] In the MPEG-D DRC standard, one or more DRC effects provided by a particular DRC set may be indicated. For example, a DRC set may be suitable for "late night" viewing, viewing in a "noisy environment", viewing at a "low playback level", etc. A DRC set effect parameter may indicate which specific effects a DRC set provides.

[0220] In addition, the MPEG-D DRC standard allows the user to specify one or more desired DRC effects, and one or more optional fallback DRC effects. This information from the user can be used to select the most appropriate DRC set from the available DRC sets. For example, if there is a DRC set that matches the desired DRC effect, then that set is selected. If such a set does not exist, but there is a set that matches the fallback DRC effect, then that set can be selected.

[0221] The list of defined DRC effects may be extended or updated to include a DRC effect that provides loudness leveling. In addition, the list of DRC effects that a user may specify through an interface may also be extended or updated to include a DRC set effect that provides loudness leveling. Thus, loudness leveling may be provided by including a DRC set that is indicated as providing a loudness leveling effect and indicating a desire for a DRC set that provides loudness leveling through an interface, in which case the decoder will select and apply the DRC set that corresponds to the loudness leveling DRC effect.

[0222] As described above, a previous version of a table that specifies DRC set effects that can be signaled in a bitstream (e.g., Table A.45 of ISO / IEC 23003-4:2020) may be updated to include an entry for a "loudness leveling" effect as shown below. Specifically, a row having a specific bit position (e.g., 13) corresponding to the loudness leveling effect may be added as shown in the following table. Alternatively, an existing row having a specific bit position (e.g., 12) corresponding to the "self-ducking / leveling" effect may be updated as shown in the following table.

[0223]

[0224] Table 22: Example of DRC effect

[0225] In the above example, a bit (e.g., bit position 13 or bit position 12) in a bit field associated with a particular DRC set (e.g., the drcSetEffect bit field of the drcInstructionsBasic() or drcInstructionsUniDrc() or drcInstructionsUniDrcV1() payload) may be set to indicate that the DRC set provides a loudness leveling effect.

[0226] Similarly, a previous version of a table specifying DRC set effects (e.g., Table 12 of ISO / IEC 23003-4:2020) may be updated to include an entry for a loudness leveling effect as shown below, which may be specified to a decoder through an interface. Specifically, a row having a specific index value (e.g., 9) corresponding to a loudness leveling effect may be added as shown in the following table.

[0227]

[0228] Table 23: Examples of DRC effects that can be specified via the interface

[0229] In the above example, an interface parameter (e.g., the drcEffectTypeRequest parameter of the dynamicRangeControllerInterface() payload) may be set to a specific value (e.g., 9) corresponding to the loudness leveling effect to indicate that the decoder should select and apply a DRC set that provides a loudness leveling effect set, if available. However, since loudness leveling may be intended to be used in conjunction with loudness normalization, in some cases the decoder may require that in order to select and apply a DRC set that provides a loudness leveling effect, the decoder must also perform loudness normalization. Such a requirement may be achieved by only allowing the decoder to select a DRC set that provides a loudness leveling effect when loudness normalization is also enabled in the decoder. Alternatively or in addition, such a requirement may be met by modifying the interface to require that when a DRC set that provides a loudness leveling effect is requested, loudness normalization must also be enabled (e.g., by setting loudness normalization and target loudness values ​​on a flag).

[0230] Specifying DRC sets with loudness leveling using the new extended payload

[0231] Alternatively, a new payload (e.g., dynLoudInstructions()) may be defined that includes instructions for loudness leveling. For example, one or more such payloads may be included in an extension field (e.g., UNIDRCCONFEXT_V2) of an existing bitstream. By including such a payload in an extension field, compatibility with existing decoders that are not designed to parse the new payload may be maintained, as such decoders simply ignore the additional payload(s). Each of such payloads would be assigned a unique identifier (e.g., drcSetId) corresponding to a DRC set that provides loudness leveling, and to help easily identify such a set, a flag (e.g., a levelingPresent flag) may be set to 1 for each such DRC set signaled via the new payload.

[0232] Additionally, the user interface described above for selecting a DRC set that provides a loudness leveling effect may also be used for DRC sets signaled via this new type of payload (e.g., the dynLoudInstructions() payload). Specifically, a user may indicate to a decoder via the interface (e.g., using the drcEffectTypeRequest field) that a DRC set that provides loudness leveling is desired. The decoder will then select the DRC set signaled via the new payload (if present) (e.g., a DRC set with a DRC set ID that is identified as corresponding to loudness leveling and, for example, has a levelingPresent flag set to 1). However, as described above, since loudness leveling may be intended to be used in conjunction with loudness normalization, in some cases a decoder may require that in order to select and apply a DRC set that provides a loudness leveling effect, the decoder must also perform loudness normalization. Such a requirement may be achieved by only allowing the decoder to select a DRC set that provides a loudness leveling effect when loudness normalization is also enabled in the decoder. Alternatively or additionally, such a requirement may be met by modifying the interface to require that when a DRC set providing a loudness leveling effect is requested, loudness normalization must also be enabled (eg, by setting loudness normalization and target loudness values ​​on flags).

[0233] An advantage of using this type of signaling (e.g., instead of signaling the DRC sets that provide loudness leveling via existing parameters such as the drcSetEffect parameter) is that, as described above, additional parameters that indicate whether and how a user should enable and / or disable loudness leveling can be included in this new payload. For example, the new payload may also include a parameter (e.g., a dynamicLoudCompSwOffAllowed parameter) that indicates, for each DRC set to which loudness leveling is applied, whether loudness leveling is allowed to be turned off via a user interface.

[0234] An example syntax for a new payload (eg, dynLoudInstructions() payload) for loudness leveling is shown in Table 24.

[0235]

[0236]

[0237] Table 24: Example syntax for dynamic loudness command payload

[0238] As described above, a unique DRC set ID and a flag (e.g., levelingPresent) that specifically indicates the DRC set as a DRC set that provides loudness leveling can be associated with the payload. In addition, the payload can be associated with one or more downmix identification parameters (e.g., downmixId parameters) that indicate that the DRC set is intended for use with one or more specific downmixes / downmix configurations of the audio program. Such a DRC set identifier (e.g., drcSetId parameter) and a flag (e.g., levelingPresent flag) allow the DRC set associated with the payload to be selected in response to a user indication provided through an interface that the user desires a loudness leveling DRC effect. Additionally, the inclusion of one or more downmix identifiers (e.g., downmixID parameters) enables the selection of a DRC set that provides loudness leveling, the DRC set being intended for use with one or more specific downmixes of the audio program. In addition, the payload can be associated with one or more personalization identification parameters (e.g., mae_groupID, mae_groupPresetID) that indicate that the DRC set is intended for use with one or more specific personalization configurations of the audio program. Finally, as also described above, including a parameter indicating whether loudness leveling is allowed to be turned off (eg, dynamicLoudComSwOffAllowed) enables content creators to have additional control over whether the decoder can apply loudness leveling.

[0239] Finally, as indicated above, such additional payloads (e.g., dynLoudInstructions() payloads) may be contained within an extension payload (e.g., uniDrcConfigExtension() payload) of a configuration payload. A specific extension type (e.g., uniDrcConfigExtType of UNIDRCCONFEXT_V2 with a value of 0x2) may be used to indicate the presence of such additional payloads within the extension field of the configuration payload.

[0240] An example of the syntax of the extension field of a configuration payload (eg, uniDrcConfigExtension() payload) is shown in Table 25.

[0241]

[0242]

[0243]

[0244] Table 25: Example of update syntax for a configuration payload including extension fields

[0245] The data associated with a particular extension type of the extension field of the configuration payload includes a parameter indicating whether loudness leveling instructions are present (e.g., a dynLoudPresent parameter), and a parameter indicating the number of loudness leveling instruction sets in the extension payload (e.g., a dynLoudInstructionsCount parameter), if present, and each loudness leveling instruction set (e.g., each dynLoudInstructions() payload), where the syntax of the loudness leveling instructions may be as shown in Table 25.

[0246] Signaling DRC sets for loudness leveling via existing configuration elements

[0247] Still alternatively, a particular DRC set may be indicated to provide loudness leveling by adding a new field to an existing configuration element. For example, a new parameter (e.g., a levelingPresent parameter) indicating whether a DRC set provides loudness leveling may be included in an existing configuration element (e.g., a downmixInstructions() element, a drcInstructionsBasic() element, or a drcInstructionsUniDrc() element). Ideally, to avoid compatibility issues with legacy decoders, the new element may be included in a field that is already ignored by legacy decoders (e.g., a reserved field). Alternatively, the new parameter may be included in an updated version of an existing configuration element (e.g., a downmixInstructionsV2() element or a drcInstructionsUniDrcV2() element) that will be ignored by legacy decoders.

[0248] In one example, signaling a DRC set for loudness leveling through an existing configuration element may use an extension of the uniDrcConfigExtension() element shown in Table 26 to carry instructions to control and apply loudness leveling metadata received as dynamics processing metadata based on the DRC set as a frame-based update.

[0249] As shown in Table 26, a switch case UNIDRCCONFEXT_LEVELING may be added to the uniDrcConfigExtension() element. The switch case UNIDRCCONFEXT_LEVELING may be used to identify new elements including levelingPresent and duckingOnlyDrcSetPresent. The uniDrcConfigExtension() element may be an extension of the uniDrcConfig() element as described in ISO / IEC 23003-4. New parameters including levelingPresent and duckingOnlyDrcSetPresent may be used to extend the drcInstructionsBasic(), drcInstructionsUniDrc(), and drcInstructionsUniDrcV1() elements as defined in ISO / IEC 23003-4.

[0250] New elements including levelingPresent and duckingOnlyDrcSetPresent may be used to select a DRC set (as described in ISO / IEC 23003-4, 6.3.7) as a standalone parameter or as a parameter included in the drcInstructionsBasic(), drcInstructionsUniDrc() and drcInstructionsUniDrcV1() elements.

[0251] The leveling gain that needs to be applied as part of the loudness leveling process (such as the parameter dynLoudCompValue in other syntax examples) can be included in the bitstream using the corresponding uniDrcGain() element.

[0252]

[0253]

[0254]

[0255] Table 26: Alternative syntax for the uniDrcConfigExtension() element

[0256] The semantics of the new elements including levelingPresent and duckingOnlyDrcSetPresent can be defined as shown in Table 27.

[0257]

[0258] Table 27: Semantics of the levelingPresent and duckingOnlyDrcSetPresent elements

[0259] Finally, similar to the selection process described above, a DRC set that is signaled in this manner as providing loudness leveling may be selected to a decoder via an interface by indicating that the DRC set is desired to provide a loudness leveling effect. When such an indication is provided to the decoder via the interface, the decoder will identify and select the DRC set that is indicated as providing loudness leveling (e.g., its levelingPresent parameter is equal to 1). If multiple DRC sets are indicated as providing loudness leveling, the decoder may use other rules (e.g., predefined rules, such as those defined in ISO / IEC 23003-4:2020) to select the most appropriate DRC set among the multiple DRC sets that provide loudness leveling.

[0260] However, since loudness leveling may be intended to be used in conjunction with loudness normalization, in some cases a decoder may require that in order to select and apply a DRC set that provides a loudness leveling effect, the decoder must also perform loudness normalization. Such a requirement may be achieved by only allowing the decoder to select a DRC set that provides a loudness leveling effect when loudness normalization is also enabled in the decoder. Alternatively or additionally, such a requirement may be met by modifying the interface to require that when a DRC set that provides a loudness leveling effect is requested, loudness normalization must also be enabled (e.g., by setting loudness normalization and target loudness values ​​on flags).

[0261] Figure 6 An example method for playing back audio data as described above is illustrated. Figure 6 In step S301, for example, as described above, the decoder 100 may receive a bitstream including encoded audio data and metadata, wherein the metadata includes one or more dynamic range control (DRC) sets, and an indication of whether the DRC set for each DRC set is configured to provide a loudness leveling effect that matches the selected personalized experience. The metadata may include personalized information and / or indications as described herein.

[0262] Although the format of the bitstream is not limited, in an embodiment, the bitstream may be an MPEG-D DRC bitstream. Alternatively, the bitstream may be an MPEG-H 3D audio compatible bitstream. Then, for example, as described above, the presence of metadata for providing a loudness leveling effect may be signaled based on the MPEG-D DRC bitstream syntax.

[0263] In step S302, the decoder may then parse the metadata to identify a DRC set configured to provide a loudness leveling effect that matches the selected personalized experience, for example, as described above. In addition, in step S302, the metadata may be parsed to identify personalized metadata, thereby allowing a loudness leveling effect that matches the selected personalized experience to be provided.

[0264] In step S303 , the decoder may then decode the audio data to obtain decoded audio data, for example, as described above.

[0265] In step S304, the decoder may then select one of the identified DRC sets configured to provide a loudness leveling effect matching the selected personalized experience, for example, as described above. The selection may be based on the personalized metadata identified at step S302.

[0266] In step S305 , the decoder may then extract one or more DRC gains corresponding to the selected DRC set from the bitstream, for example, as described above.

[0267] In step S306 , the decoder may then apply the one or more DRC gains corresponding to the selected DRC set to the decoded audio data to obtain loudness-leveled audio data, for example, as described above.

[0268] Finally, in step S307 , the loudness-leveled audio data may then be output for playback, for example, as described above.

[0269] Efficient encoding / transmission of loudness leveling data

[0270] As described above, loudness leveling in conjunction with dynamic range control may be achieved by transmitting a DRC set containing gains that are a combination of dynamic range control gains and loudness leveling gains. However, doing so may require redundant transmission of dynamic range control gain data. For example, a first DRC set may contain gains for applying dynamic range control to a signal, and it may be important to allow this DRC gain set to be applied independently of any other gains (e.g., loudness leveling gains). Therefore, if a DRC set containing gains for applying a combination of dynamic range control and loudness leveling is desired, then a second DRC set needs to be specified that represents a combination of the dynamic range control gains of the first set and the desired loudness leveling gains. Unfortunately, doing so requires that both the first DRC set and the second DRC set contain information about the dynamic range control gains of the first set, which is inefficient.

[0271] A more efficient way to transmit data to achieve the same goal is to provide only dynamic range control gains in the first DRC set and only loudness leveling gains in the second DRC set, as well as some additional metadata indicating the relationship between the two DRC sets. For example, the first DRC set may include parameters indicating that there is another DRC set that is dependent on the first DRC set. The decoder may then understand that, for example, the gains of the first set may be applied independently in cases where only dynamic range control is desired. Additionally, the decoder will understand that the gains of the first DRC set and the second DRC set may also be combined to obtain a combination of dynamic range control and loudness leveling. Finally, in cases where the gains of the second set are never intended to be applied alone, metadata may be included with the second DRC set to indicate whether the gains of that DRC set may only be used in conjunction with the first set, or whether they may be used independently of the gains of the first set.

[0272] For example, consider 2 DRC sets, where a first DRC set identified as DRC set 1 (e.g., its drcSetId equals 1) provides gains for dynamic range control, and a second DRC set identified as DRC set 2 (e.g., its drcSetId equals 2) provides gains for loudness leveling. Since DRC set 2 is intended to be used in conjunction with DRC set 1, DRC set 1 may include a parameter indicating that DRC set 2 depends on DRC set 1 (e.g., the dependsOnDrcSet parameter of DRC set 1 may be equal to 2). Additionally, if loudness leveling is intended to be used only in conjunction with dynamic range control, DRC set 2 may include a parameter indicating that the gains of DRC set 2 may not be used independently (e.g., the noIndependentUse flag of DRC set 2 may be set to a value of 1). Conversely, if DRC set 2 is not intended to be used only in conjunction with dynamic range control, DRC set 2 may include a parameter indicating that the gains of DRC set 2 may be used independently (e.g., the noIndependentUse flag of DRC set 2 may be set to a value of 0).

[0273] The efficient encoding / transmission of loudness leveling data described above may be used with any of the additional methods of implementing loudness leveling described above.

[0274] For example, in the first case where the presence of a DRC set providing a loudness leveling effect is signaled through an existing effect flag (e.g., a bit of the drcSetEffect bit field), DRC set 1 may have a drcSetEffect parameter in which the bit corresponding to the loudness leveling effect is set to 0, while DRC set 2 may have the bit corresponding to the loudness leveling effect set to 1, and all other bits of the bit field may be the same for the two DRC sets. With this signaling, when the user requests the loudness leveling effect through the interface, the decoder will extract and apply the gains of DRC set 1 and DRC set 2 to apply a combination of dynamic range control and loudness leveling.

[0275] Table 28 shows exemplary values ​​of the parameters of the above-mentioned signaling to achieve efficient transmission of loudness leveling data.

[0276]

[0277] Table 28: Exemplary values ​​for parameters for efficient transmission of signaling loudness leveling data

[0278] In the second case, where the presence of a DRC set providing a loudness leveling effect is signaled using a new payload, DRC set 2 will be signaled via the new payload (e.g., the drcSetId parameter of the dynLoudInstructions payload will be set to 2). DRC set 1 will be signaled as described above (e.g., the drcSetId parameter will be set to 1 and the dependsonDrcSet parameter will be set to 2, indicating that DRC set 2 depends on DRC set 1). Finally, it is not possible to indicate via the new payload (e.g., dynLoudInstructions) described above that DRC set 2 should not be used independently (e.g., it should always be used with DRC set 1). Therefore, additional parameters may be added to the new payload (e.g., a noIndependentUse parameter may be added to the dynLoudInstructions payload) such that when the new parameter is set to 1, DRC set 2 is only used with DRC set 1, and when the new parameter is set to 0, DRC set 2 may be used independently of DRC set 1. Similar to the first case, when the user requests a loudness leveling effect through the interface, the decoder will extract and apply the gains of both DRC Set 1 and DRC Set 2 to apply a combination of dynamic range control and loudness leveling.

[0279] Finally, in the third case, existing configuration elements are used, which already include all the parameters needed to efficiently implement loudness leveling as described above. Specifically, as described above, DRC Set 1 will include an indication that DRC Set 2 is dependent on DRC Set 1, and DRC Set 2 will include an indication of whether it can be used independently or only in conjunction with DRC Set 2. Again, similar to the first two cases, when the user requests the loudness leveling effect through the interface, the decoder will extract and apply the gains of DRC Set 1 and DRC Set 2 to apply a combination of dynamic range control and loudness leveling.

[0280] As indicated above, a benefit of using dependent DRC sets to implement loudness leveling is that doing so allows for more efficient transmission of the data required for loudness leveling because it eliminates the need to transmit redundant dynamic range control gains in the DRC sets containing the loudness leveling gains.

[0281] Differences in DRC implementation between MPEG-D DRC and MPEG-H 3D Audio

[0282] As mentioned above, references herein to MPEG-D and MPEG-D DRC refer to the standard in ISO / IEC 23003-4, also identified as Information Technology - MPEG Audio Techniques - Part 4: Dynamic Range Control. This standard is further incorporated into the MPEG-H 3D Audio standard specified in ISO / IEC 23008-3 (MPEG-H Part 3). The MPEG-H 3D Audio standard is an audio coding standard developed by MPEG to support the encoding of audio as audio channels, audio objects, or higher-order Ambisonics (HOA).

[0283] Although the MPEG-H 3D Audio standard references the MPEG-D standard, it includes several modifications to the syntax and techniques of the MPEG-D standard in order to provide appropriate technical compatibility. In particular, the main differences from MPEG-DDRC in MPEG-H Audio that are relevant to the present invention include:

[0284] In MPEG-D DRC (and hence in MPEG-D USAC as specified by ISO / IEC 23003-3), DRC instructions are defined in the Dynamic Loudness Control profile using the drcInstructionsUniDrcV1() syntax element. In the Low Complexity and Baseline profiles of MPEG-H 3D Audio, DRC instructions are defined using the drcInstructionsUniDrc() syntax element.

[0285] ·MPEG-H 3D audio provides the possibility of personalizing the user experience. Therefore, the DRC set selection process has additional steps, as defined in 6.4.4.2 and 6.4.4.3 of ISO / IEC 23008-3 (MPEG-H 3D audio).

[0286] ·The MPEG-D DRC decoder interface in MPEG-H 3D audio is implemented through the bitstream interface based on MPEG-H audio stream (MHAS) packets with the packet type of PACTYP_LOUDNESS_DRC. Creative syntax updates for MPEG-H 3D audio

[0287] Since MPEG-H 3D audio uses the drcInstructionsUniDrc() syntax element to define the instructions for the DRC set selection process, the uniDrcConfigExtension() syntax shown in Table 26 needs to be updated, as shown in Table 29, especially the creative elements illustrated in the for loop "for(i = 0; i < drcInstructionsUniDrcCount; i++)". This syntax can be part of the playback conditions considered in figures such as Figure 2 and Figure 6 and so on.

[0288] This additional syntax enables mapping the DRC instruction parameters required for loudness leveling (e.g., levelingPresent and duckingOnlyDrcSetPresent) to the DRC instruction parameters required for selecting an appropriate DRC set that matches the playback conditions (including the selected personalized experience, e.g., mae_groupID and mae_groupPresetID). Further, this mapping enables the decoder to select an appropriate DRC set based on the playback conditions (including the selected personalized experience) and turning the loudness leveling process on or off (e.g., using levelingControlInterface()).

[0289]

[0290]

[0291]

[0292] Table 29: Syntax update of uniDrcConfigExtension()

[0293] Alternatively, the syntax between “case UNIDRCCONFEXT_DYNLOUD:” and “break;” may be referred to as a separate syntax element, such as levelingInstructions(), as shown in Table 30.

[0294]

[0295]

[0296] Table 30: Syntax of levelingInstructions() payload

[0297] MPEG-H 3D Audio differences during DRC set selection

[0298] In the functionality and syntax specified by MPEG-H 3D Audio, there may be more than one DRC set with ducking / leveling effects, with different maeGroupID or maePresetID values.

[0299] The inventive aspects include taking into account the newly added parameters related to loudness leveling in any syntax element (e.g. levelingPresent, duckingOnlyDrcSetPresent in uniDrcConfigExtension() element) in the DRC set selection process described in section 6.4.4 of MPEG-H 3D Audio, so as to allow the selection of the correct DRC set according to the current playback conditions (including the selected personalized experience; e.g. mae_groupID and mae_groupPresetID). The selection of the correct DRC set may further depend on the state of loudness leveling (e.g. on or off), which may be controlled using the MPEG-D DRC decoder interface. The MPEG-D DRC decoder interface in MPEG-H 3D Audio may be implemented over a bitstream interface based on MHAS using a packetization type of PACTYP_LOUDNESS_DRC, enabling the transmission of mpegh3daLoudnessDrcInterface() elements, including uniDrcInterface() elements and uniDrcInterfaceExtension() elements. The uniDrcInterfaceExtension() element may be updated as shown in Tables 19 to 21.

[0300] explain

[0301] Unless otherwise specifically stated, it will be apparent from the following discussion that it should be understood that throughout the disclosed discussion, terms such as "processing," "computing," "determining," "analyzing," etc. are utilized to refer to the actions and / or processes of a computer or computing system or similar electronic computing device to manipulate and / or transform data represented as physical (e.g., electronic) quantities into other data similarly represented as physical quantities.

[0302] In a similar manner, the term "processor" may refer to any device or portion of a device that processes electronic data, such as from registers and / or memory, to transform the electronic data into other electronic data, such as may be stored in registers and / or memory. A "computer" or "computing machine" or "computing platform" may include one or more processors.

[0303] In an example embodiment, the methods described herein may be performed by one or more processors that accept computer-readable (also referred to as machine-readable) code containing a set of instructions that, when executed by one or more processors, perform at least one of the methods described herein. Any processor that is capable of executing a set of instructions (sequential or otherwise) that specifies an action to be taken is included. Thus, an example is a typical processing system that includes one or more processors. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system may further include a memory subsystem that includes a main RAM and / or a static RAM and / or a ROM. A bus subsystem may be included for communication between components. The processing system may further be a distributed processing system in which processors are coupled together via a network. If the processing system requires a display, such a display may be included, for example, a liquid crystal display (LCD) or a cathode ray tube (CRT) display. If manual data entry is required, the processing system also includes an input device, such as one or more of an alphanumeric input unit (such as a keyboard), a pointing control device (such as a mouse), etc. The processing system may also cover storage systems such as disk drive units. The processing system in some configurations may include a sound output device and a network interface device. The memory subsystem thus includes a computer-readable carrier medium carrying computer-readable code (e.g., software), the computer-readable code including a set of instructions that, when executed by one or more processors, causes one or more of the methods described herein to be performed. It should be noted that when the method includes multiple elements (e.g., several steps), any order of these elements is not implied unless otherwise stated. During the execution of the software by the computer system, the software may reside in a hard disk, or may also reside completely or at least partially in RAM and / or a processor. Therefore, the memory and the processor also constitute a computer-readable carrier medium carrying computer-readable code. In addition, the computer-readable carrier medium may form or be included in a computer program product.

[0304] In alternative example embodiments, one or more processors may operate as standalone devices, or may be connected (e.g., networked) to other processors in a networked deployment, the one or more processors may operate as a server or user machine in a server-user network environment, or as a peer machine in a peer-to-peer or distributed network environment. The one or more processors may form a personal computer (PC), a tablet PC, a personal digital assistant (PDA), a cellular phone, a web appliance, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by the machine.

[0305] It should be noted that the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

[0306] Therefore, an example embodiment of each method described herein is in the form of a computer-readable carrier medium carrying a set of instructions, such as a computer program for executing on one or more processors (e.g., one or more processors as part of a web server arrangement). Therefore, as will be appreciated by those skilled in the art, the example embodiments of the present disclosure may be embodied as a method, a device such as a special-purpose device, a device such as a data processing system, or a computer-readable carrier medium (e.g., a computer program product). The computer-readable carrier medium carries a computer-readable code including a set of instructions, which, when executed on one or more processors, causes one or more processors to implement the method. Therefore, aspects of the present disclosure may take the form of a method, a complete hardware example embodiment, a complete software example embodiment, or an example embodiment combining software and hardware aspects. In addition, the present disclosure may take the form of a carrier medium (e.g., a computer program product on a computer-readable storage medium), which carries a computer-readable program code embodied in the medium.

[0307] The software may be further sent or received over a network via a network interface device. Although in the example embodiments the carrier medium is a single medium, the term "carrier medium" should be considered to include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) storing one or more sets of instructions. The term "carrier medium" should also be considered to include any medium capable of storing, encoding, or carrying a set of instructions for execution by one or more of the processors and causing one or more processors to perform any one or more of the methods of the present disclosure. The carrier medium may take a variety of forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical disks, magnetic disks, and magneto-optical disks. Volatile media include dynamic memory, such as main memory. Transmission media include coaxial cables, copper wires, and optical fibers, including wires containing bus subsystems. Transmission media may also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communications. For example, the term "carrier medium" should therefore be taken to include, but is not limited to, solid-state memories, computer products embodied in optical and magnetic media; media carrying propagated signals detectable by at least one processor or one or more processors and representing a set of instructions that, when executed, implement a method; and transmission media in a network carrying propagated signals detectable by at least one of one or more processors and representing the set of instructions.

[0308] It will be appreciated that, in an example embodiment, the steps of the method discussed are performed by an appropriate processor (or processors) in a processing (e.g., computer) system executing instructions (computer readable code) stored in a storage device. It will also be appreciated that the present disclosure is not limited to any particular implementation or programming technique, and that the present disclosure may be implemented using any suitable technique for implementing the functionality described herein. The present disclosure is not limited to any particular programming language or operating system.

[0309] References throughout this disclosure to "one embodiment," "some embodiments," or "example embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosure. Thus, the phrases "in one embodiment," "in some embodiments," or "in an example embodiment" appearing in various places throughout this disclosure do not necessarily all refer to the same example embodiment. Furthermore, in one or more example embodiments, the particular features, structures, or characteristics may be combined in any suitable manner, as will be apparent to one of ordinary skill in the art in light of this disclosure.

[0310] As used herein, unless otherwise specified, the use of ordinal adjectives "first," "second," "third," etc. to describe common objects merely indicates that different instances of similar objects are being referred to and is not intended to imply that the objects described must be in a given order in time, space, rank, or in any other manner.

[0311] In the claims below and the description herein, any of the terms comprising, including, or which comprises is an open term, which means at least including the elements / features that follow, but not excluding other elements / features. Therefore, when the term comprising is used in a claim, the term should not be interpreted as being limited to the devices or elements or steps listed thereafter. For example, the scope of the expression "a device comprising A and B" should not be limited to a device comprising only elements A and B. As used herein, the term including, or any of which includes, or that includes, is also an open term, which also means at least including the elements / features that follow the term, but not excluding other elements / features. Therefore, including is synonymous with comprising and means comprising.

[0312] It should be understood that in the above description of the example embodiments of the present disclosure, various features of the present disclosure are sometimes combined together in a single example embodiment / figure or its description in order to simplify the present disclosure and help understand one or more of the various creative aspects. However, the method of the present disclosure should not be interpreted as reflecting the intention that the claims require more features than the features explicitly recited in each claim. On the contrary, as reflected in the following claims, each creative aspect lies in less than all the features of a single previously disclosed example embodiment. Therefore, the claims following the specification are hereby expressly incorporated into this specification, wherein each claim independently serves as a separate example embodiment of the present disclosure.

[0313] In addition, although some example embodiments described herein include some features included in other example embodiments and do not include other features included in other example embodiments, as will be understood by those skilled in the art, the combination of features of different example embodiments is intended to be within the scope of the present disclosure and form different example embodiments. For example, in the appended claims, any of the example embodiments claimed for protection may be used in any combination.

[0314] In the description provided herein, many specific details are set forth. However, it should be understood that the exemplary embodiments of the present disclosure can be practiced without these specific details. In other instances, well-known methods, structures, and techniques are not shown in detail in order to avoid obscuring the understanding of this specification.

[0315] Therefore, although the mode that is considered to be the best mode of the present disclosure has been described, it will be recognized by those skilled in the art that other and further modifications may be made thereto without departing from the spirit of the present disclosure, and it is intended to claim all such changes and modifications that fall within the scope of the present disclosure. For example, any formula given above merely represents a program that may be used. Functions may be added or deleted from the block diagram, and operations may be interchanged among functional blocks. Steps may be added or deleted to the methods described within the scope of the present disclosure.

[0316] In the following, two sets of enumerated example embodiments (EEEA and EEEB) describe some structures, features, and functions of some aspects of the example embodiments disclosed herein.

[0317] EEEA 1. A method for performing metadata-based dynamic processing on audio data for playback, the method comprising the following process:

[0318] (a) receiving a bitstream by a decoder, the bitstream comprising audio data and metadata for loudness leveling;

[0319] (b) decoding the audio data and the metadata by the decoder to obtain decoded audio data and metadata;

[0320] (c) determining, by the decoder, from the metadata one or more processing parameters for loudness leveling based on playback conditions;

[0321] (d) applying the determined one or more processing parameters to the decoded audio data to obtain processed audio data; and

[0322] (e) outputting the processed audio data for playback.

[0323] EEEA 2. The method according to EEEA 1, wherein the metadata indicates processing parameters for loudness leveling for multiple playback conditions.

[0324] EEEA 3. The method according to EEEA 1 or EEEA 2, wherein the determining the one or more processing parameters further comprises determining one or more processing parameters for dynamic range compression (DRC) based on the playback condition.

[0325] EEEA 4. A method according to any one of EEEA 1 to EEEA 3, wherein the playback conditions include one or more of the following: a device type of the decoder, characteristics of a playback device, characteristics of a speaker, speaker settings, characteristics of background noise, characteristics of ambient noise, and characteristics of an acoustic environment.

[0326] EEEA 5. A method according to any one of EEEA 1 to EEEA 4, wherein process (c) further comprises selecting, by the decoder, at least one of a DRC sequence set DRCSet, an equalizer parameter set EQSet, and a downmix corresponding to the playback condition.

[0327] EEEA 6. The method according to EEEA 5, wherein process (c) further comprises identifying a metadata identifier indicating the at least one selected DRCSet, EQSet and downmix to determine the one or more processing parameters from the metadata.

[0328] EEEA 7. The method according to any one of EEEA 1 to EEEA 6, wherein the metadata comprises one or more processing parameters related to an average loudness value and optionally one or more processing parameters related to a dynamic range compression characteristic.

[0329] EEEA 8. The method according to any one of EEEA 1 to EEEA 7, wherein the bitstream further comprises additional metadata for static loudness adjustment to be applied to the decoded audio data.

[0330] EEEA 9. The method according to any one of EEEA 1 to EEEA 8, wherein the bitstream is an MPEG-D DRC bitstream and the presence of metadata is signaled based on an MPEG-D DRC bitstream syntax.

[0331] EEEA 10. The method according to EEEA 9, wherein the uniDrcConfigExtension() element is used to carry the metadata as a payload.

[0332] EEEA 11. A method according to any one of EEEA 1 to EEEA 10, wherein the metadata includes one or more metadata payloads, wherein each metadata payload includes multiple parameter and identifier sets, each set including at least one of a DRCSet identifier drcSetId, an EQSet identifier eqSetId, and a downmix identifier downmixId and a combination of one or more processing parameters associated with the identifier in the set.

[0333] EEEA 12. A method according to EEEA 11 when subordinate to EEEA 5, wherein process (c) involves selecting a set from the multiple sets in the payload based on the at least one DRCSet, EQSet and downmix selected by the decoder, and wherein the one or more processing parameters determined at process (c) are the one or more processing parameters associated with the identifier in the selected set.

[0334] EEEA 13. A decoder for metadata-based dynamic processing of audio data for playback, wherein the decoder comprises one or more processors and non-transitory memory configured to perform a method comprising:

[0335] (a) receiving, by a decoder, a bitstream, the bitstream comprising audio data and metadata for loudness leveling;

[0336] (b) decoding the audio data and the metadata by the decoder to obtain decoded audio data and metadata;

[0337] (c) determining, by the decoder, from the metadata one or more processing parameters for loudness leveling based on playback conditions;

[0338] (d) applying the determined one or more processing parameters to the decoded audio data to obtain processed audio data; and

[0339] (e) outputting the processed audio data for playback.

[0340] EEEA 14. A method of encoding audio data and metadata for loudness leveling into a bitstream, the method comprising the following process:

[0341] (a) inputting original audio data into a loudness equalizer for loudness processing to obtain loudness-processed audio data as output of the loudness equalizer;

[0342] (b) generating metadata for loudness leveling based on the loudness-processed audio data and the original audio data; and

[0343] (c) encoding the original audio data and the metadata into the bitstream.

[0344] EEEA 15. The method according to EEEA 14, wherein the method further comprises generating additional metadata for static loudness adjustment to be used by a decoder.

[0345] EEEA 16. The method according to EEEA 14 or EEEA 15, wherein process (b) comprises comparing the loudness processed audio data with the original audio data, and wherein the metadata is generated based on a result of the comparison.

[0346] EEEA 17. The method according to EEEA 16, wherein process (b) further comprises measuring loudness within one or more predefined time periods, and wherein the metadata is further generated based on the measured loudness.

[0347] EEEA 18. The method according to EEEA 17, wherein the measuring comprises measuring the overall loudness of the audio data.

[0348] EEEA 19. The method according to EEEA 17, wherein the measuring comprises measuring the loudness of dialogue in the audio data.

[0349] EEEA 20. The method according to any one of EEEA 14 to EEEA 19, wherein the bitstream is an MPEG-D DRC bitstream and the presence of the metadata is signaled based on an MPEG-D DRC bitstream syntax.

[0350] EEEA 21. The method according to EEEA 20, wherein a uniDrcConfigExtension() element is used to carry the metadata as a payload.

[0351] EEEA 22. A method according to any one of EEEA 14 to EEEA 21, wherein the metadata comprises one or more metadata payloads, wherein each metadata payload comprises multiple parameter and identifier sets, each set comprises a combination of at least one of a DRCSet identifier drcSetId, an EQSet identifier eqSetId, and a downmix identifier downmixId and one or more processing parameters associated with the identifiers in the set, and wherein the one or more processing parameters are parameters for loudness leveling by a decoder.

[0352] EEEA 23. The method according to EEEA 22, wherein the at least one of the drcSetId, the eqSetId and the downmixId is related to at least one of a DRC sequence set DRCSet, an equalizer parameter set EQSet and a downmix to be selected by the decoder.

[0353] EEEA 24. An encoder for encoding raw audio data and metadata for loudness leveling in a bitstream, wherein the encoder comprises one or more processors and non-transitory memory configured to perform a method comprising:

[0354] (a) inputting original audio data into a loudness equalizer for loudness processing to obtain loudness-processed audio data as output of the loudness equalizer;

[0355] (b) generating the metadata for loudness leveling based on the loudness-processed audio data and the original audio data; and

[0356] (c) encoding the original audio data and the metadata into the bitstream.

[0357] EEEA 25. A system of an encoder according to EEEA 24 and a decoder according to EEEA 13, wherein the encoder is used to encode original audio data and metadata for loudness leveling and / or dynamic range compression DRC in a bitstream, and the decoder is used to perform metadata-based dynamic processing on the audio data for playback.

[0358] EEEA 26. A computer program product comprising a computer readable storage medium having instructions, the instructions being adapted to, when executed by a device having processing capabilities, cause the device to perform a method according to any one of EEEA 1 to EEEA 12 or EEEA 14 to EEEA 23.

[0359] EEEA 27. A computer-readable storage medium storing the computer program product according to EEEA 26.

[0360] EEEA 28. The method according to any one of EEEA 1 to EEEA 12, further comprising receiving, by the decoder through an interface, an indication of whether to perform the metadata-based dynamic processing on the audio data for playback, and when the decoder receives an indication indicating that the metadata-based dynamic processing on the audio data for playback is not to be performed, at least bypassing the step of applying the determined one or more processing parameters to the decoded audio data.

[0361] EEEA 29. A method according to EEEA 28, wherein before the decoder receives the indication of whether to perform the metadata-based dynamic processing on the audio data for playback through the interface, the decoder at least bypasses the step of applying the determined one or more processing parameters to the decoded audio data.

[0362] EEEA 30. A method according to any one of EEEA 1 to EEEA 12, EEEA 28 or EEEA 29, wherein the metadata indicates processing parameters for loudness leveling for multiple playback conditions, and the metadata further includes parameters specifying a loudness measurement method used to derive a processing parameter in the multiple processing parameters.

[0363] EEEA 31. A method according to any one of EEEA 1 to EEEA 12 or EEEA 28 to EEEA 30, wherein the metadata indicates processing parameters for loudness leveling for multiple playback conditions, and the metadata further includes parameters specifying a loudness measurement system for measuring processing parameters in the multiple processing parameters.

[0364] EEEB 1. A method for performing metadata-based dynamic processing on audio data for playback, the method comprising the following operations:

[0365] receiving, by a decoder, a bitstream including audio data and metadata for dynamic loudness adjustment;

[0366] decoding, by the decoder, the audio data and the metadata to obtain decoded audio data and metadata;

[0367] selecting, by the decoder, at least one set of decoder settings corresponding to a playback condition, and determining, from the metadata, one or more processing parameters for dynamic loudness adjustment based on the at least one selected set;

[0368] applying the determined one or more processing parameters to the decoded audio data to obtain processed audio data; and

[0369] The processed audio data is output for playback.

[0370] EEEB 2. The method of EEEB 1, wherein the metadata indicates processing parameters for dynamic loudness adjustment for a plurality of playback conditions.

[0371] EEEB 3. The method according to EEEB 1 or EEEB 2, wherein the determining the one or more processing parameters further comprises determining one or more processing parameters for dynamic range compression (DRC) based on the playback condition.

[0372] EEEB 4. A method according to any one of EEEB 1 to 3, wherein the playback conditions include one or more of the following: a device type of the decoder, characteristics of a playback device, characteristics of a speaker, speaker settings, characteristics of background noise, characteristics of ambient noise, and characteristics of an acoustic environment.

[0373] EEEB 5. The method according to any one of EEEB 1 to 4, wherein the at least one decoder setting set comprises a DRC sequence set DRCSet, an equalizer parameter set EQSet, and a downmix.

[0374] EEEB 6. The method according to EEEB 5, wherein the determining the one or more processing parameters further comprises identifying a metadata identifier indicating the at least one selected DRCSet, EQSet and downmix to determine the one or more processing parameters from the metadata.

[0375] EEEB 7. The method according to any one of EEEB 1 to 6, wherein the metadata comprises one or more processing parameters related to average loudness values ​​and optionally one or more processing parameters related to dynamic range compression characteristics.

[0376] EEEB 8. The method according to any one of EEEB 1 to 7, wherein the bitstream further comprises additional metadata for static loudness adjustment to be applied to the decoded audio data.

[0377] EEEB 9. The method according to any one of EEEB 1 to 8, wherein the bitstream is an MPEG-D DRC bitstream and the presence of metadata is signaled based on an MPEG-D DRC bitstream syntax.

[0378] EEEB 10. The method according to EEEB 9, wherein a loudnessInfoSetExtension() element is used to carry the metadata as a payload.

[0379] EEEB 11. A method according to any one of EEEB 1 to 10, wherein the metadata includes one or more metadata payloads, wherein each metadata payload includes multiple parameter and identifier sets, each set includes at least one of a DRCSet identifier drcSetId, an EQSet identifier eqSetId, and a downmix identifier downmixId and a combination of one or more processing parameters associated with the identifier in the set.

[0380] EEEB 12. A method according to EEEB 11 when subordinate to EEEB 5, wherein the determining of the one or more processing parameters involves selecting a set from the multiple sets in the payload based on the at least one DRCSet, EQSet and downmix selected by the decoder, and wherein the one or more processing parameters determined by the decoder are the one or more processing parameters associated with an identifier in the selected set.

[0381] EEEB 13. A decoder for metadata-based dynamic processing of audio data for playback, wherein the decoder comprises one or more processors and non-transitory memory configured to perform a method comprising:

[0382] receiving, by the decoder, a bitstream comprising audio data and metadata for dynamic loudness adjustment;

[0383] decoding, by the decoder, the audio data and the metadata to obtain decoded audio data and metadata;

[0384] selecting, by the decoder, at least one set of decoder settings corresponding to a playback condition, and determining, from the metadata, one or more processing parameters for dynamic loudness adjustment based on the at least one selected set;

[0385] applying the determined one or more processing parameters to the decoded audio data to obtain processed audio data; and

[0386] The processed audio data is output for playback.

Claims

1. A method of processing audio data for playback, the method comprising: receiving, by a decoder, a bitstream comprising encoded audio data and metadata, wherein the metadata comprises one or more dynamic range control (DRC) sets and, for each DRC set, an indication of whether the DRC set is configured to provide a loudness leveling effect, wherein the metadata further comprises personalized metadata; parsing, by the decoder, the metadata to identify a DRC set configured to provide the loudness leveling effect, wherein the DRC set is identified based at least in part on the personalized metadata; Decoding the encoded audio data by the decoder to obtain decoded audio data; selecting, by the decoder, one of the identified DRC sets configured to provide the loudness leveling effect; extracting, by the decoder, from the bitstream one or more DRC gains corresponding to the selected DRC set; applying, by the decoder, the one or more DRC gains corresponding to the selected DRC set to the decoded audio data to obtain dynamic loudness compensated audio data; and The dynamically loudness compensated audio data is output for playback.

2. The method of claim 1, wherein: The bitstream is a bitstream compatible with the MPEG-H 3D Audio standard.

3. The method of claim 2, wherein: The personalized metadata includes mae_groupID and maegroupPresetID syntax and semantics as described in the MPEG-H 3D Audio standard.

4. The method according to any one of claims 1 to 3, wherein: The metadata includes a plurality of DRC sets configured for providing the loudness leveling effect, wherein each of the plurality of DRC sets is further associated with one or more playback conditions, and wherein the selecting is performed in response to an indication of a playback condition provided to the decoder.

5. The method according to any one of claims 1 to 4, wherein: In addition to providing a loudness leveling effect, the one or more DRC sets are also configured to provide dynamic range control.

6. The method according to any one of claims 1 to 4, wherein: The playback conditions include one or more of: a device type of the decoder, characteristics of a playback device, characteristics of a speaker, speaker settings, characteristics of background noise, characteristics of ambient noise, capabilities of the device type, a personalized experience selected on the device, and characteristics of the acoustic environment.

7. The method of claim 6, wherein: The personalized experience selected by the user or device is based on user preferences such as language, user experience, previous listening selections, and / or device capabilities.

8. The method according to any one of claims 1 to 7, wherein: An indication of whether the DRC set is configured to provide the loudness leveling effect is provided in a parameter indicative of one or more effects provided by the DRC set.

9. The method of claim 8, wherein: The parameter indicating one or more effects provided by the DRC set is a drcSetEffect bit field of an MPEG-D DRC bitstream, wherein individual bits of the drcSetEffect bit field correspond to different effects, and one of the bits of the drcSetEffect bit field corresponds to the loudness leveling effect.

10. The method according to any one of claims 1 to 9, wherein: The indication of whether the DRC set is configured to provide the loudness leveling effect is whether the DRC set is specified in a loudness leveling bitstream payload.

11. The method of claim 10, wherein: The loudness leveling bitstream payload is included in an extension field of the previously defined bitstream syntax.

12. The method of claim 11, wherein: The extension field is the uniDrcConfigExtension field of the MPEG-H 3D Audio bitstream, and wherein the loudness leveling bitstream payload is included only for a specific value of the uniDrcConfigExtType parameter.

13. The method of claim 11 or claim 12, wherein: A plurality of loudness leveling payloads specifying a plurality of DRC sets configured to provide the loudness leveling effect are included in the extension field of the previously defined bitstream syntax.

14. The method according to any one of claims 1 to 13, wherein: The indication of whether the DRC set is configured for providing the loudness leveling effect is a field of a pre-existing configuration element of a previously defined bitstream syntax.

15. The method according to any one of claims 1 to 14, wherein: The indication of whether the DRC set is configured for providing the loudness leveling effect is a field of an updated version of a previously existing configuration element of a previously defined bitstream syntax.

16. The method according to any one of claims 1 to 15, wherein: An indication of a desired loudness leveling effect is provided to the decoder via an interface, and wherein the DRC set is selected in response to the indication provided to the decoder via the interface.

17. The method of claim 16, wherein: The interface receives the indication from an MHAS-compatible syntax.

18. The method of claim 17, wherein: An indication of an additional desired effect is provided to the decoder via the interface, wherein the metadata includes a plurality of DRC sets configured to provide the loudness leveling effect, and wherein the selection is dependent on the additional desired effect.

19. The method according to any one of claims 16 to 18, wherein: Said indication of the desired loudness leveling effect is provided via the loudnessLevelingOn parameter of the levelingControlInterface payload.

20. The method of any one of claims 1 to 19, wherein: The metadata includes one or more static loudness values ​​configured for providing a static loudness adjustment to the decoded audio data.

21. The method of claim 20, comprising applying a static loudness adjustment to the decoded audio data or the dynamic loudness compensated audio data in response to one or more of the static loudness values.

22. The method of claim 1, wherein: A first one of the DRC sets is configured for providing dynamic range control, wherein the first DRC set includes an indication that a selected DRC set configured for providing the loudness leveling effect is configured for application in conjunction with the first DRC set.

23. The method of claim 22, wherein: The selected DRC set includes an indication of whether the one or more DRC gains corresponding to the selected DRC set can only be applied in combination with the DRC gains corresponding to the first DRC set.

24. The method of claim 22 or claim 23, further comprising extracting the DRC gains corresponding to the first DRC set from the bitstream and applying the DRC gains corresponding to the first DRC set to the decoded audio data.

25. A decoder for processing audio data for playback, wherein: The decoder comprises one or more processors and non-transitory memory configured to perform the method of any one of claims 1 to 24.

26. A computer program product comprising a computer readable storage medium having instructions adapted to cause an apparatus to perform the method of any one of claims 1 to 24.

27. A computer-readable storage medium storing the computer program product according to claim 26.