Optimizing loudness and dynamic range across different playback devices

Abstract

To provide optimization of the loudness and the dynamic range across different playback devices.SOLUTION: Embodiments are directed to a method and system for receiving, in a bitstream, metadata associated with the audio data, and analyzing the metadata to determine whether a loudness parameter for a first group of audio playback devices are available in the bitstream. In response to determining that the parameters are present for the first group, the system uses the parameters and audio data to render audio. In response to determining that the loudness parameters are not present for the first group, the system analyzes one or more characteristics of the first group, and determines the parameter on the basis of the one or more characteristics.SELECTED DRAWING: Figure 16

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 61 / 754,882, filed January 21, 2013, U.S. Provisional Patent Application No. 61 / 809,250, filed April 5, 2013, and U.S. Provisional Patent Application No. 61 / 824,010, filed May 16, 2013.

[0002] FIELD OF THEINVENTION One or more embodiments relate generally to audio signal processing, and more particularly to processing audio data bitstreams with metadata indicative of loudness and dynamic range characteristics of the audio content based on playback environment and device. [Background technology]

[0003] It should not be assumed that the subject matter discussed in the Background section is prior art merely as a result of being mentioned in the Background section. Similarly, it should not be assumed that a problem mentioned in the Background section or related to the subject matter of the Background section has been previously recognized in the prior art. The subject matter in the Background section merely represents various approaches, and such approaches may themselves be inventive.

[0004] The dynamic range of an audio signal is generally the ratio between the maximum and minimum possible values ​​of sounds embodied in the signal, and is usually measured as a decibel (base 10) value. In many audio processing systems, dynamic range control (or dynamic range compression (DRC)) is used to reduce the level of loud sounds and / or amplify the level of quiet sounds to make wide dynamic range source content fit into a narrower recorded dynamic range that can be more easily stored and played back using electronic equipment. For audio / visual (AV) content, a dialogue reference level may be used to define the "null" point for compression via the DRC mechanism. DRC serves to boost content below the dialogue reference level and cut content above the reference level.

[0005] Known audio encoding systems use metadata associated with the audio signal to set a DRC level based on the type of content and intended use. The DRC mode sets the amount of compression applied to the audio signal and defines the decoder's output reference level. Such systems may be limited to two DRC levels that are programmed into the encoder and selected by the user. For example, a dialnorm (dialog normalization) value of -31 dB (line) is traditionally used for content played on an AVR or full dynamic range capable device, and a dialnorm value of -20 dB is used for content played on a television set or similar device. This type of system allows a single audio bitstream to be used in two common but very different playback scenarios through the use of two different sets of DRC metadata. However, such systems are limited to pre-set dialnorm values ​​and are not optimized for playback on the wide variety of different playback devices and listening environments that are possible today through the advent of digital media and Internet-based streaming technologies.

[0006] In current metadata-based audio encoding systems, a stream of audio data may contain both audio content (e.g., one or more channels of audio content) and metadata that describes at least one characteristic of the audio content. For example, in an AC-3 bitstream, there are several audio metadata parameters that are specifically intended for use in altering the sound of a program delivered to a listening environment. One of the metadata parameters is the dialnorm parameter, which indicates the average loudness level of the dialogue (or the average loudness of the content) appearing in an audio program and is used to determine the audio playback signal level.

[0007] During playback of a bitstream containing a sequence of different audio program segments (each with a different dialnorm parameter), an AC-3 decoder uses the dialnorm parameter of each segment to perform a type of loudness processing that modifies the playback level or loudness of the segments so that the perceived loudness of the dialogue of the segments is at a consistent level. Each encoded audio segment (item) in a sequence of encoded audio items will (in general) have a different dialnorm parameter, and the decoder will scale the level of each item so that the playback level or loudness of the dialogue for each item is the same or very similar. However, this may require applying different amounts of gain to different ones of the items during playback.

[0008] In some embodiments, the dialnorm parameters are set by the user and are not automatically generated, although there is a default dialnorm value if no value is set by the user. For example, a content creator may make loudness measurements using equipment external to the AC-3 encoder and then forward the results (indicative of the loudness of the spoken dialogue of an audio program) to the encoder to set the dialnorm value. Thus, we rely on the content creator to set the dialnorm parameters correctly.

[0009] There are several different reasons why the dialnorm parameter in an AC-3 bitstream may be incorrect. First, each AC-3 encoder has a default dialnorm value that is used during the generation of the bitstream if no dialnorm value is set by the content creator. This default value may differ substantially from the actual dialogue loudness level of the audio. Second, even if the content creator measures loudness and sets the dialnorm value accordingly, a loudness measurement algorithm or meter that does not follow the recommended loudness measurement method may have been used, leading to an incorrect dialnorm value. Third, even if an AC-3 bitstream was generated with a dialnorm value that was correctly measured and set by the content creator, it may have been changed to an incorrect value by an intermediate module during transmission and / or storage of the bitstream. For example, in television broadcast applications, it is not uncommon for an AC-3 bitstream to be decoded, modified, and then re-encoded using incorrect dialnorm metadata information. Thus, the dialnorm values ​​included in the AC-3 bitstream may be incorrect or inaccurate, and therefore may have a negative impact on the quality of the listening experience.

[0010] Furthermore, the dialnorm parameter does not indicate the loudness processing state of the corresponding audio data (e.g., what type(s) of loudness processing have been performed on that audio data). Furthermore, currently deployed loudness and DRC systems, such as those in the Dolby Digital (DD) and Dolby Digital Plus (DD+) systems, are designed to render AV content in consumers' living rooms or movie theaters. To adapt such content for playback in other environments and listening equipment (e.g., mobile devices), post-processing needs to be applied "blindly" in the playback device to adapt the AV content to the listening environment. In other words, the post-processor (or decoder) assumes that the loudness level of the received content is at a certain level (e.g., -31 or -20 dB) and the post-processor sets the level to a predetermined fixed target level suitable for the particular device. If the assumed loudness level or the predetermined target level is incorrect, the post-processing may have an effect opposite to its intended effect. That is, post-processing can make the output audio less desirable to the user.

[0011] The disclosed embodiments are not limited to use with AC-3, E-AC-3, or Dolby E bitstreams, but for convenience are discussed in the context of systems in which such bitstreams include loudness processing state metadata. Dolby, Dolby Digital, Dolby Digital Plus, and Dolby E are trademarks of Dolby Laboratories Licensing Corporation. Dolby Laboratories offers its own implementations of AC-3 and E-AC-3 known as Dolby Digital and Dolby Digital Plus, respectively. Summary of the Invention [Means for solving the problem]

[0012] Embodiments are directed to a method of decoding audio data by receiving a bitstream including metadata associated with audio data and analyzing the metadata in the bitstream to determine whether loudness parameters for a first group of audio playback devices are available in the bitstream. In response to determining that the parameters are present for the first group, a processing component uses the parameters and audio data to render the audio. In response to determining that the loudness parameters are not present for the first group, a processing component analyzes one or more characteristics of the first group and determines the parameters based on the one or more characteristics. The method may further use the parameters and audio data to render audio by sending the parameters and audio data to a downstream module that renders the audio for playback. The parameters and audio data may be used to render audio by rendering the audio data based on the parameters and audio data.

[0013] In one embodiment, the method also includes determining an output device that renders the received audio stream and determining whether the output device belongs to the first group of audio playback devices, where the step of analyzing metadata in the stream to determine whether loudness parameters for the first group of audio playback devices are available is performed after the step of determining that the output device belongs to the first group of audio playback devices. In one embodiment, the step of determining that the output device belongs to the first group of audio playback devices includes receiving an indication from a module connected to the output device indicating an identity of the output device or indicating an identity of a group of devices that includes the output device, and determining that the output device belongs to the first group of audio playback devices based on the received indication.

[0014] The embodiments are further directed to apparatus or systems that include processing components that perform the steps described in the encoding method embodiments above.

[0015] Embodiments are further directed to a method of decoding audio data by receiving audio data and metadata associated with the audio data, analyzing metadata in a bitstream to determine whether loudness information associated with loudness parameters for a first group of audio devices is available in the stream, and in response to determining that the loudness information is present for the first group, determining loudness information from the stream and transmitting the audio data and loudness information for use in rendering the audio, or if the loudness information is not present for the first group, determining loudness information associated with an output profile and transmitting the determined loudness information for the output profile for use in rendering the audio. In an embodiment, determining loudness information associated with an output profile may include analyzing characteristics of the output profile and determining the parameters based on the characteristics, and transmitting the determined loudness information may include transmitting the determined parameters. The loudness information may include loudness parameters for the output profile or characteristics of the output profile. In some embodiments, the method further includes determining a low bit rate encoded stream to be transmitted, the loudness information including characteristics for one or more output profiles.

[0016] The embodiments are further directed to apparatus or systems that include processing components that perform the steps described in the above decoding method embodiments. [Brief description of the drawings]

[0017] In the following drawings, like reference numerals are used to refer to like elements. The following drawings illustrate various examples, however, implementations described herein are not limited to the examples illustrated in the drawings. [Figure 1] FIG. 1 is a block diagram of an embodiment of an audio processing system configured to perform loudness and dynamic range optimization, under some embodiments. [Diagram 2] 2 is a block diagram of an encoder for use in the system of FIG. 1, under some embodiments. [Diagram 3] 2 is a block diagram of a decoder for use in the system of FIG. 1, under some embodiments. [Figure 4] FIG. 2 shows an AC-3 frame, including the segments into which it is divided. [Diagram 5] FIG. 2 illustrates the Synchronization Information (SI) segment of an AC-3 frame, including the segments into which it is divided. [Figure 6] FIG. 2 illustrates a Bitstream Information (BSI) segment of an AC-3 frame, including the segments into which it is divided. [Figure 7] A diagram showing an E-AC-3 frame, including the segments into which it is divided. [Figure 8] 1 is a table illustrating the format of certain frames and metadata of an encoded bitstream under some embodiments. [Figure 9] 1 is a table illustrating the format of loudness processing state metadata under some embodiments. [Figure 10] FIG. 2 is a more detailed block diagram of the audio processing system of FIG. 1 that may be configured to perform loudness and dynamic range optimization, under some embodiments. [Figure 11]1 is a table illustrating different dynamic range requirements for various playback devices and background listening environments in an exemplary use case. [Figure 12] FIG. 1 is a block diagram of a dynamic range optimization system, under an embodiment. [Figure 13] FIG. 2 is a block diagram illustrating an interface between different profiles for various different playback device classes, under one embodiment. [Figure 14] 1 is a table illustrating the correlation between long term loudness and short term dynamic range for several defined profiles, under an embodiment. [Figure 15] 1 illustrates example loudness profiles for various types of audio content, under one embodiment. [Figure 16] 1 is a flow chart illustrating a method for optimizing loudness and dynamic range across playback devices and applications, under an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Definitions and Nomenclature Throughout this disclosure, including the claims, the expression performing an operation "on" a signal or data (e.g., filtering, scaling, transforming, or applying a gain to the signal or data) is used broadly to refer to performing the operation directly on the signal or data or on a processed version of the signal or data (e.g., on a version of the signal that has undergone preliminary filtering or preprocessing prior to performing the operation). The expression "system" is used broadly to refer to a device, system, or subsystem. For example, a subsystem that implements a decoder may be referred to as a decoder system, and a system that includes such a subsystem (e.g., a system that generates X output signals in response to a number of inputs, where the subsystem generates M of the inputs and the other XM inputs are received from external sources) may also be referred to as a decoder system. The term "processor" is used broadly to refer to a system or device that is programmable (e.g., with software or firmware) or otherwise configurable to perform operations on data (e.g., audio or video or other image data). Examples of processors include field programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors programmed and / or otherwise configured to perform pipelined processing on audio or other sound data, programmable general-purpose processors or computers, and programmable microprocessor chips or chipsets.

[0019] The terms "audio processor" and "audio processing unit" are used interchangeably and broadly to refer to a system configured to process audio data. Examples of audio processing units include, but are not limited to, encoders (e.g., transcoders), decoders, codecs, pre-processing systems, post-processing systems, and bitstream processing systems (sometimes referred to as bitstream processing tools). The expression "processing state metadata" (e.g., as in the expression "loudness processing state metadata") refers to data that is separate and distinct from the corresponding audio data (the audio content of an audio data stream that also includes the processing state metadata). The processing state metadata is associated with the audio data and indicates the loudness processing state of the corresponding audio data (e.g., what type(s) of processing have already been performed on the audio data), and optionally at least one feature or characteristic of the audio data. In some embodiments, the association of the processing state metadata with the audio data is time-synchronous. Thus, the current (most recently received or updated) processing state metadata indicates that the corresponding audio data currently comprises the results of audio data processing of the indicated type(s). In some cases, the processing state metadata may include some or all of the processing history and / or parameters used in and / or derived from processing of the indicated type. Additionally, the processing state metadata may include at least one feature or characteristic of the corresponding audio data that has been calculated or extracted from the audio data. The processing state metadata may also include other metadata that is not related to or derived from any processing of the corresponding audio data. For example, third party data, tracking information, identifiers, proprietary or standard information, user annotation data, user preference data, etc. may be added by a particular audio processing unit and passed to other audio processing units.

[0020] The expression "loudness processing state metadata" (or "LPSM") refers to processing state metadata that indicates the loudness processing state of corresponding audio data (e.g. what type(s) of loudness processing have already been performed on the audio data), and optionally also at least one feature or characteristic (e.g. loudness) of the corresponding audio data. The loudness processing state metadata may include data (e.g. other metadata) that is not (considered in isolation) loudness processing state metadata. The terms "couple" or "coupled" are used to mean a direct or indirect connection.

[0021] Systems and methods are described for an audio encoder / decoder that non-destructively normalize audio loudness and dynamic range across a variety of devices that require or use different target loudness values ​​and have different dynamic range capabilities. Methods and functional components according to some embodiments send information about audio content from an encoder to a decoder for one or more device profiles. A device profile specifies the desired target loudness and dynamic range for one or more devices. The system is extensible so that new device profiles with different "nominal" loudness targets can be supported.

[0022] In one embodiment, the system generates appropriate gains based on loudness control and dynamic range requirements at the encoder, and generates the gains at the decoder under control from the encoder through parameterization of the original gains to reduce data rate. The dynamic range system includes two mechanisms for implementing loudness control: artistic dynamic range profiles that provide content creators control over how the audio is played back, and a separate protection mechanism that ensures that overloads do not occur for various playback profiles. The system is also configured to allow other metadata (internal or external) parameters to be used to appropriately control loudness and dynamic range gains and / or profiles. The decoder is configured to support n-channel auxiliary inputs that utilize decoder-side loudness and dynamic range settings / processing.

[0023] In some embodiments, loudness processing state metadata (LPSM) is embedded in one or more reserved fields (or slots) of a metadata segment of an audio bitstream. The audio bitstream also contains audio data in other segments (audio data segments). For example, at least one segment of each frame of the bitstream contains an LPSM and at least one other segment of the frame contains corresponding audio data (i.e., audio data whose loudness processing state and loudness are indicated by said LPSM). In some embodiments, the data volume of the LPSM may be small enough to be carried without affecting the bitrate allocated to carry the audio data.

[0024] Communicating loudness processing state metadata in an audio data processing chain is particularly useful when two or more audio processing units need to function in cascade with each other throughout the processing chain (or content lifecycle). Without including loudness processing state metadata in the audio bitstream, media processing problems such as quality, level and spatial degradation can occur when two or more audio codecs are utilized in the chain and single-ended volume leveling is applied more than once during the bitstream's journey to the media consumption device (or rendering point of the bitstream's audio content).

[0025] Loudness and Dynamic Range Metadata Processing System FIG. 1 is a block diagram of an embodiment of an audio processing system that may be configured to perform loudness and dynamic range optimization under some embodiments using certain media processing (e.g., pre-processing and post-processing) components. FIG. 1 illustrates an exemplary audio processing chain (audio data processing system), one or more of the system's elements may be configured in accordance with an embodiment of the present invention. The system 10 of FIG. 1 includes the following elements coupled together as shown: a pre-processing unit 12, an encoder 14, a signal analysis and metadata correction unit 16, a transcoder 18, a decoder 20, and a post-processing unit 24. Variations on the illustrated system omit one or more of the elements or include additional audio data processing units. For example, in some embodiments, the post-processing unit 22 is part of the decoder 20 rather than a separate unit.

[0026] In some implementations, the pre-processing unit of FIG. 1 is configured to accept as input 11 PCM (time domain) samples comprising audio content and to output processed PCM samples. The encoder 14 may be configured to accept as input the PCM samples and to output an encoded (e.g., compressed) audio bitstream indicative of the audio content. The data in the bitstream indicative of the audio content is sometimes referred to herein as "audio data." In one embodiment, the audio bitstream output from the encoder includes loudness processing state metadata (and optionally other metadata) in addition to the audio data.

[0027] The signal analysis and metadata correction unit 16 may accept as input one or more encoded audio bitstreams and determine whether the processing state metadata in each encoded audio bitstream is correct (e.g., validity checking) by performing signal analysis. In some embodiments, the validity checking may be performed by a state validity checker component such as element 102 shown in FIG. 2, one such validity checking technique is described below in the context of state validity checker 102. In some embodiments, unit 16 is included in an encoder and the validity checking is done by either unit 16 or validity checker 102. If the signal analysis and metadata correction unit finds that the included metadata is invalid, the metadata correction unit 16 performs signal analysis to determine the correct value(s) and replaces the incorrect value(s) with the determined correct value(s). In this way, each encoded audio bitstream output from the signal analysis and metadata correction unit may include corrected processing state metadata in addition to the encoded audio data. The signal analysis and metadata correction unit 16 may be part of the pre-processing unit 12, the encoder 14, the transcoder 18, the decoder 20 or the post-processing unit 22. Alternatively, the signal analysis and metadata correction unit 16 may be a separate unit or part of another unit in the audio processing chain.

[0028] The transcoder 18 may accept an encoded audio bitstream as input and output a modified (e.g., differently encoded) audio bitstream in response (e.g., by decoding the input stream and re-encoding the decoded stream in a different encoding format). The audio bitstream output from the transcoder contains loudness processing state metadata (and optionally other metadata) in addition to the encoded audio data. The metadata may be included in the bitstream.

[0029] The decoder 20 of FIG. 1 may accept as input an encoded (e.g., compressed) audio bitstream and (in response) output a stream of decoded PCM audio samples. In an embodiment, the output of the decoder is or includes any of the following: a stream of audio samples and a corresponding stream of loudness processing state metadata (and optionally other metadata) extracted from the input encoded bitstream; a stream of audio samples and a corresponding stream of control bits determined from the loudness processing state metadata (and optionally other metadata) extracted from the input encoded bitstream; or a stream of audio samples without the corresponding stream of processing state metadata or control bits determined from the processing state metadata. In this last case, the decoder may extract the loudness processing state metadata (and / or other metadata) from the input encoded bitstream and perform at least one operation on the extracted metadata (e.g., validity checking) without outputting the extracted metadata or the control bits determined therefrom.

[0030] By configuring the post-processing unit of FIG. 1 according to one embodiment of the present invention, the post-processing unit 22 is configured to accept a stream of decoded PCM audio samples and perform post-processing thereon (e.g., volume leveling of the audio content) using the loudness processing state metadata (and optionally other metadata) received with the samples or control bits (determined by the decoder from the loudness processing state metadata and optionally other metadata) received with the samples. The post-processing unit 22 is also optionally configured to render the post-processed audio content for playback through one or more speakers. These speakers may be embodied in any of a variety of different listening devices or items of playback equipment, such as computers, televisions, stereo systems (home or cinema), mobile phones and other portable playback devices. The speakers may be of any suitable size and power rating and may be provided in the form of free-standing drivers, speaker enclosures, surround sound systems, sound bars, headphones, earbuds, etc.

[0031] Some embodiments provide an improved audio processing chain in which audio processing units (e.g., encoder, decoder, transcoder, and pre-processing and post-processing units) adapt their respective processing applied to audio data according to a contemporaneous state of media data as indicated by loudness processing state metadata received by each audio processing unit. The audio data input 11 to any audio processing unit (e.g., encoder or transcoder in FIG. 1 ) of the system 100 may include loudness processing state metadata (and optionally other metadata) in addition to audio data (e.g., encoded audio data). The metadata may, according to some embodiments, have been included in the input audio by another element or source. A processing unit receiving the input audio (with metadata) may be configured to perform at least one operation on the metadata (e.g., validity checking) or in response to the metadata (e.g., adaptive processing of the input audio) and to include in its output audio the metadata, a processed version of the metadata, or control bits determined from the metadata.

[0032] An embodiment of the audio processing unit (or audio processor) is configured to perform adaptive processing of the audio data based on a state of the audio data indicated by loudness processing state metadata corresponding to the audio data. In some embodiments, the adaptive processing is (or includes) a loudness processing (if the metadata indicates that no loudness processing or similar processing has already been performed on the audio data) but is not (or does not include) a loudness processing (if the metadata indicates that such loudness processing or similar processing has already been performed on the audio data). In some embodiments, the adaptive processing is (or includes) a metadata validity check (e.g., performed in a metadata validity check subunit) to ensure that the audio processing unit performs other adaptive processing of the audio data based on a state of the audio data indicated by the loudness processing state metadata. In some embodiments, the validity check determines the reliability of loudness processing state metadata associated with the audio data (e.g., included in a bitstream together with the audio data). For example, if the metadata is validated as reliable, results from already performed audio processing of a certain type may be reused and additional performance of the same type of audio processing may be avoided, whereas if the metadata is found to be tampered with (or otherwise unreliable), the media processing of the type claimed to have been previously performed (as indicated by the unreliable metadata) may be repeated by the audio processing unit and / or other processing may be performed by the audio processing unit on the metadata and / or audio data.The audio processing unit may be configured to signal to other downstream audio processing units in the enhanced media processing chain that the loudness processing state metadata (e.g. present in the media bitstream) is valid if the audio processing unit determines that the processing state metadata is valid (e.g. based on a match of the extracted cryptographic value and the reference cryptographic value).

[0033] 1 , pre-processing component 12 may be part of encoder 14, and post-processing component 22 may be part of decoder 22. Alternatively, pre-processing component 12 may be embodied in a functional component that is separate from encoder 14. Similarly, post-processing component 22 may be embodied in a functional component that is separate from decoder 20.

[0034] Figure 2 is a block diagram of an encoder 100 that may be used in conjunction with the system 10 of Figure 1. Any of the components or elements of the encoder 100 may be implemented as one or more processes and / or one or more circuits (e.g., ASIC, FPGA or other integrated circuits) in hardware, software or a combination of hardware and software. The encoder 100 includes a frame buffer 110, a parser 111, a decoder 101, an audio state validator 102, a loudness processing stage 103, an audio stream selection stage 104, an encoder 105, a stuffer / formatter stage 107, a metadata generation stage 106, a dialogue loudness measurement subsystem 108 and a frame buffer 109, connected as shown. Optionally, the encoder 100 also includes other processing elements (not shown). Encoder 100 (which is a transcoder) is configured to convert an input audio bitstream (which may be, for example, one of an AC-3 bitstream, an E-AC-3 bitstream, or a Dolby E bitstream) into an encoded output audio bitstream (which may be, for example, another of an AC-3 bitstream, an E-AC-3 bitstream, or a Dolby E bitstream), including by performing adaptive and automated loudness processing using loudness processing state metadata included in the input bitstream. For example, encoder 100 may be configured to convert an input Dolby E bitstream (a format typically used in production and broadcast facilities, but not typically used in consumer devices that receive broadcasted audio programs) into an encoded output audio bitstream in AC-3 or E-AC-3 format (suitable for broadcast to consumer devices).

[0035] 2 also includes an encoded audio delivery subsystem 150 (which stores and / or delivers the encoded bitstream output from encoder 100) and a decoder 152. The encoded audio bitstream output from encoder 100 may be stored by subsystem 150 (e.g., in the form of a DVD or Blu-ray disc), or may be transmitted by subsystem 150 (which may implement a transmission link or network), or may be both stored and transmitted by subsystem 150. Decoder 152 is configured to decode the encoded audio bitstream (generated by encoder 100) received via subsystem 150, including by extracting loudness processing state metadata (LPSM) from each frame of the bitstream to generate decoded audio data. In one embodiment, decoder 152 is configured to perform adaptive loudness processing on the decoded audio data using LPSM and / or forward the decoded audio data and the LPSM to a post-processor configured to perform adaptive loudness processing on the decoded audio data using LPSM. Optionally, decoder 152 includes a buffer that stores (e.g., in a non-temporary manner) the encoded audio bitstream received from subsystem 150.

[0036] Various implementations of the encoder 100 and the decoder 152 are configured to perform the various embodiments described herein. The frame buffer 110 is a buffer memory coupled to receive the encoded input audio bitstream. In operation, the buffer 110 stores (e.g., in a non-transient manner) at least one frame of the encoded audio bitstream, and a sequence of frames of the encoded audio bitstream are presented from the buffer 110 to the parser 111. The parser 111 is coupled and configured to extract loudness processing metadata (LPSM) and other metadata from each frame of the encoded input audio, present at least the LPSM to the audio state validator 102, the loudness processing stage 103, the stage 106 and the subsystem 108, extract audio data from the encoded input audio, and present the audio data to the decoder 101. The decoder 101 of the encoder 100 is configured to decode audio data to generate decoded audio data and to present the decoded audio data to a loudness processing stage 103, an audio stream selection stage 104, a subsystem 108 and optionally also to a status validity checker 102.

[0037] The state validator 102 is configured to authenticate and validate the LPSM (and optionally other metadata) presented to it. In some embodiments, the LPSM is (or is included in) a data block that has been included in the input bitstream (e.g., in accordance with an embodiment of the present invention). The block may include a cryptographic hash (a hash-based message authentication code or "HMAC") for processing the LPSM (and optionally other metadata) and / or the underlying audio data (provided to the validator 102 from the decoder 101). The data block may, in these embodiments, be digitally signed, so that downstream audio processing units may relatively easily authenticate and validate the processing state metadata.

[0038] For example, HMAC may be used to generate a digest, which the protection value(s) included in the bitstream of the invention may include. For an AC-3 frame, this digest may be generated as follows: (1) After the AC-3 data and the LPSM are encoded, the frame data bytes (concatenated frame data #1 and frame data #2) and the LPSM data bytes are used as input for the hash function HMAC. Other data that may be present in the auxiliary data field are not taken into account for calculating this digest. Such other data may be bytes that do not belong to the AC-3 data or to the LPSM data. The protection bits included in the LPSM may not be taken into account for calculating the HMAC digest. (2) After the digest is calculated, it is written to the bitstream in the field reserved for the protection bits. (3) The last step in the generation of a complete AC-3 frame is the calculation of the CRC check. This is written at the very end of the frame and takes into account all the data that belongs to this frame, including the LPSM bits.

[0039] Other cryptographic methods, including but not limited to any one or more non-HMAC cryptographic methods, may be used for validation of the LPSM (e.g., in validator 102) to ensure secure transmission and receipt of the LPSM and / or underlying audio data. For example, validation (using such cryptographic methods) may be performed at each audio processing unit that receives an embodiment of an audio bitstream to determine whether loudness processing state metadata and corresponding audio data included in the bitstream have been subjected to (and / or result from) a particular loudness processing (as indicated by the metadata) and have not been modified since performance of such particular loudness processing.

[0040] The status validator 102 presents control data to the audio stream selection stage 104, the metadata generator 106 and the dialogue loudness measurement subsystem 108 to indicate the result of the validity checking operation. In response to the control data, stage 104 can select (and communicate to the encoder 105) either: (1) the adaptively processed output of the loudness processing stage 103 (e.g., when LPSM indicates that the audio data output from the decoder 101 has not undergone a particular type of loudness processing and the control bit from the validator 102 indicates that LPSM is enabled); or (2) said audio data output from the decoder 101 (e.g., when LPSM indicates that the audio data output from the decoder 101 has already undergone a particular type of loudness processing to be performed by stage 103 and the control bit from the validator 102 indicates that LPSM is enabled).

[0041] Stage 103 of encoder 100 is configured to perform adaptive loudness processing on the decoded audio data output from decoder 101 based on one or more audio data characteristics indicated by the LPSM extracted by decoder 101. Stage 103 may be an adaptive transform-domain real-time loudness and dynamic range control processor. Stage 103 may receive user input (e.g., user target loudness / dynamic range values ​​or dialnorm values) or other metadata input (e.g., one or more types of third party data, tracking information, identifiers, proprietary or standard information, user annotation data, user preference data, etc.) and / or other information (e.g., from a fingerprinting process) and use such input to process the decoded audio data output from decoder 101.

[0042] The dialogue loudness measurement subsystem 108 may operate to determine the loudness of segments of decoded audio (from decoder 101) that indicate dialogue (or other speech) using, for example, the LPSM (and / or other metadata) extracted by decoder 101 if the control bit from the validity checker 102 indicates that LPSM is disabled. If the control bit from the validity checker 102 indicates that LPSM is enabled, operation of the dialogue loudness measurement subsystem 108 may be disabled when the LPSM indicates a previously determined loudness of a dialogue (or other speech) segment of the decoded audio (from decoder 101).

[0043] There are useful tools (e.g., Dolby LM100 loudness meters) for conveniently and easily measuring the level of dialogue in audio content. Some embodiments of the APU (e.g., stage 108 of encoder 100) are implemented to include such tools (or to perform the functions of such tools) to measure the average dialogue loudness of the audio content of an audio bitstream (e.g., a decoded AC-3 bitstream presented to stage 108 from decoder 101 of encoder 100). When stage 108 is implemented to measure the true average dialogue loudness of the audio data, the measurement may include isolating segments of the audio content that contain primarily speech. The primarily speech audio segments are then processed according to a loudness measurement algorithm. For audio data decoded from an AC-3 bitstream, this algorithm may be a standard K-weighted loudness measure (in accordance with international standard ITU-R BS.1770). Alternatively, other loudness measures (eg, based on psychoacoustic models of loudness) may be used.

[0044] Isolating speech segments is not essential for measuring the average dialogue loudness of audio data, but it improves the accuracy of the metric and gives more satisfying results from the listener's point of view. Since not all audio content contains dialogue, a loudness metric for the entire audio content may provide a sufficient approximation of the dialogue level of that audio if speech was present.

[0045] The metadata generator 106 generates metadata to be included by stage 107 in the encoded bitstream output from the encoder 100. The metadata generator 106 may pass the LPSM (and / or other metadata) extracted by the encoder 101 to the stage 107 (e.g., if the control bit from the validity checker 102 indicates that the LPSM and / or other metadata are valid), or may generate new LPSM (and / or other metadata) and present the new metadata to the stage 107 (e.g., if the control bit from the validity checker 102 indicates that the LPSM and / or other metadata extracted by the decoder 101 is invalid). Alternatively, the metadata generator 106 may present to the stage 107 a combination of the metadata extracted by the decoder 101 and the newly generated metadata. The metadata generator 106 may include loudness data generated by the subsystem 108 and at least one value indicative of the type of loudness processing performed by the subsystem 108 in the LPSM presented to the stage 107 for inclusion in the encoded bitstream output from the encoder 100. Metadata generator 106 may generate protection bits (which may consist of or include a hash-based message authentication code or "HMAC") useful for at least one of decrypting, authenticating or validating the LPSM (and optionally other metadata) to be included in the encoded bitstream and / or the underlying audio data to be included in the encoded bitstream. Metadata generator 106 may provide such protection bits to stage 107 for inclusion in the encoded bitstream.

[0046] In one embodiment, the dialogue loudness measurement subsystem 108 processes the audio data output from the decoder 101 and responsively generates loudness values ​​(e.g., gated and ungated dialogue loudness values) and dynamic range values. In response to these values, the metadata generator 106 may generate loudness processing state metadata (LPSM) for inclusion (by the stuffer / formatter 107) in the encoded bitstream output from the encoder 100. In one embodiment, the loudness may be calculated based on techniques specified by the ITU-R BS.1770-1 and ITU-R BS.1770-2 standards or other similar loudness measurement standards. The gated loudness can be dialog-gated loudness or relative-gated loudness or a combination of these gated loudness types, and the system can use an appropriate gating block depending on the application requirements and system constraints.

[0047] Additionally, optionally or alternatively, subsystems 106 and / or 108 of encoder 100 may perform additional analysis of the audio data to generate metadata indicative of at least one characteristic of the audio data for inclusion in the encoded bitstream output by stage 107. Encoder 105 encodes the audio data output by selection stage 104 (e.g., by performing compression thereon) and presents the encoded audio to stage 107 for inclusion in the encoded bitstream output by stage 107.

[0048] Stage 107 multiplexes the encoded audio from encoder 105 and metadata (including LPSM) from generator 106 to generate an encoded bitstream that is output from stage 107, such that the encoded bitstream has a format specified by an embodiment. Frame buffer 109 is a buffer memory that stores (e.g., in a non-transient manner) at least one frame of the encoded audio bitstream that is output from stage 107. A sequence of those frames of the encoded audio bitstream are then presented from buffer 109 to delivery system 150 as output from encoder 100.

[0049] The LPSM generated by metadata generator 106 and included in the encoded bitstream by stage 107 indicates the loudness processing state of the corresponding audio data (e.g. what type(s) of loudness processing have been performed on the audio data) and the loudness of the corresponding audio data (e.g. measured dialogue loudness, gated and / or ungated loudness and / or dynamic range). Here, "gating" the loudness and / or level measurements performed on the audio data refers to a particular level or loudness threshold such that calculated value(s) above the threshold are included in the final measurement (e.g. ignoring short-term loudness values ​​below -60 dBFS in the final measured value). Gating on an absolute value refers to a fixed level or loudness, while gating on a relative value refers to a value that depends on the current "ungated" measurement.

[0050] In some implementations of the encoder 100, the encoded bitstream buffered in the memory 109 (and output to the delivery system 150) is an AC-3 or E-AC-3 bitstream and includes audio data segments (e.g., the AB0-AB5 segments of the frame shown in FIG. 4) and metadata segments, where the audio data segments represent audio data and where each of at least some of the metadata segments includes loudness processing state metadata (LPSM). Stage 107 inserts the LPSM into the bitstream in the following format: Each metadata segment that includes the LPSM is included in an "addbsi" field of a bitstream information ("BSI") segment of a frame of the bitstream or in an auxiliary data field at the end of the frame of the bitstream (e.g., the AUX segment shown in FIG. 4).

[0051] A frame of the bitstream may contain one or two metadata segments, each containing an LPSM; if a frame contains two metadata segments, one may be present in the addbsi field of the frame and the other in the AUX field of the frame. Each metadata segment that contains an LPSM includes an LPSM payload (or container) segment having the following format: a header (e.g., including a synchronization word identifying the beginning of the LPSM payload, followed by at least one identification value, e.g., the LPSM format version, length, period, count and sub-stream association values ​​shown in Table 2 below); after the header: at least one dialogue indication value indicating whether the corresponding audio data indicates dialogue or not (e.g., which channels of the corresponding audio data indicate dialogue) (e.g., the parameter "Dialogue Channel" in Table 2); at least one loudness regulation compliance value indicating whether the corresponding audio data complies with an indicated set of loudness regulations (e.g., the parameter "Loudness Regulation Type" in Table 2); at least one loudness processing value indicating at least one type of loudness processing that has been performed on the corresponding audio data (e.g., one or more of the parameters "Dialogue Gated Loudness Compensation Flag" and "Loudness Compensation Type" in Table 2); and at least one loudness value indicating at least one loudness (e.g. peak or average loudness) characteristic of the corresponding audio data (e.g. one or more of the parameters "ITU relative gated loudness", "ITU speech gated loudness", "ITU (EBU3341) short-term 3s loudness" and "true peak");

[0052] In some implementations, each of the metadata segments inserted by stage 107 into the “addbsi” or auxiliary data fields of a frame of the bitstream has the following format: a core header (e.g., including a sync word identifying the start of the metadata segment, followed by identification values, such as the Core Element Version, Length and Period, Extension Element Count and Sub-Stream Association values ​​shown in Table 1 below); and after the Core Header, at least one protection value useful for at least one of decryption, authentication and / or validation of the loudness processing state metadata or the corresponding audio data (e.g., the HMAC digest and Audio Fingerprint values ​​of Table 1); and also after the Core Header, if the metadata segment contains an LPSM, an LPSM payload identification (“ID”) and LPSM payload size values, which identify the following metadata as an LPSM payload and indicate the size of the LPSM payload.

[0053] An LPSM payload (or container) segment (eg, having the format shown above) follows the LPSM payload ID and LPSM payload size values.

[0054] In some embodiments, each metadata segment in the ancillary data field (or "addbsi" field) of a frame has a three-level structure: a high-level structure, which includes a flag indicating whether the ancillary data (or addbsi) field contains metadata, at least one ID value indicating what type(s) of metadata are present, and optionally also a value indicating how many bits of metadata (e.g., of each type) are present (if metadata is present); one type of metadata that may be present is LPSM, and another type of metadata that may be present is media research metadata (e.g., Nielsen Media Research metadata); a mid-level structure, which includes core elements for each identified type of metadata (e.g., the core header, protection value, and LPSM payload ID and LPSM payload size values ​​as described above for each identified type of metadata); a low-level structure. This includes each payload for a core element (e.g., an LPSM payload if an LPSM payload is identified as present by the core element and / or another type of metadata payload if another type of metadata payload is identified as present by the core element).

[0055] The data values ​​in such a three-level structure may be nested. For example, a protection value(s) for the LPSM payload and / or another metadata payload identified by the core element may be included after each payload identified by the core element (and thus after the core header of the core element). In one example, the core header may identify the LPSM payload and another metadata payload, a payload ID and payload size value for a first payload (e.g., an LPSM payload) may follow the core header, the first payload itself may follow the ID and size values, a payload ID and payload size value for a second payload may follow the first payload, the second payload itself may follow these ID and size values, and protection values ​​for both payloads (or for the core element value and both payloads) may follow the last payload.

[0056] In some embodiments, when the decoder 101 receives an audio bitstream generated according to an embodiment of the present invention having a cryptographic hash, the decoder is configured to parse and extract the cryptographic hash from a determined data block from the bitstream, the block including loudness processing state metadata (LPSM). The validator 102 may use the cryptographic hash to validate the received bitstream and / or associated metadata. For example, if the validator 102 finds that the LPSM is valid based on a match between a reference cryptographic hash and the cryptographic hash extracted from the data block, the validator 102 may disable the operation of the processor 103 on the corresponding audio data and may pass the audio data through (without modification) to the selection stage 104. Additionally, optionally or alternatively, other types of cryptographic techniques may be used instead of cryptographic hash-based methods.

[0057] 2 may determine that a post / pre-processing unit (in response to the LPSM extracted by decoder 101) has performed some type of loudness processing on the audio data to be encoded (in elements 105, 106 and 107), and may thus generate (in generator 106) loudness processing state metadata that includes certain parameters used in and / or derived from the previously performed loudness processing. In some implementations, encoder 100 may generate (and include in the output encoded bitstream from) processing state metadata that indicates the processing history on the audio content, so long as the encoder is aware of the type of processing that has been performed on the audio content.

[0058] FIG. 3 is a block diagram of a decoder that may be used in conjunction with the system 10 of FIG. 1. Any of the components or elements of the decoder 200 and the post-processor 300 may be implemented as one or more processes and / or one or more circuits (e.g., ASIC, FPGA or other integrated circuits) in hardware, software or a combination of hardware and software. The decoder 200 includes a frame buffer 201, a parser 205, an audio decoder 202, an audio state validity check stage (validator) 203 and a control bit generation stage 204, connected as shown. The decoder 200 may include other processing elements (not shown). The frame buffer 201 (buffer memory) stores (e.g., in a non-transient manner) at least one frame of the encoded audio bitstream received by the decoder 200. A sequence of frames of the encoded audio bitstream is presented from the buffer 201 to the parser 205. Parser 205 is coupled and configured to extract loudness processing metadata (LPSM) and other metadata from each frame of the encoded input audio, provide at least the LPSM to audio state validator 203 and stage 204, provide the LPSM as an output (e.g., to post-processor 300), extract audio data from the encoded input audio, and provide the extracted audio data to decoder 202. The encoded audio bitstream input to decoder 200 may be one of an AC-3 bitstream, an E-AC-3 bitstream, or a Dolby E bitstream.

[0059] The system of FIG. 3 also includes a post-processor 300. The post-processor 300 has a frame buffer 301 and other processing elements (not shown) including at least one processing element coupled to the buffer 301. The frame buffer 301 stores (e.g., in a non-temporary manner) at least one frame of the decoded audio bitstream received by the post-processor 300 from the decoder 200. The processing elements of the post-processor 300 are coupled and configured to receive a sequence of frames of the decoded audio bitstream output from the buffer 301 and adaptively process using metadata (including LPSM values) output from the decoder 202 and / or control bits output from stage 204 of the decoder 200. In one embodiment, the post-processor 300 is configured to perform adaptive loudness processing on the decoded audio data using the LPSM values ​​(e.g., based on a loudness processing state and / or one or more audio data characteristics indicated by the LPSM). Various implementations of the decoder 200 and post-processor 300 are configured to perform various embodiments of the methods according to the embodiments described herein.

[0060] The audio decoder 202 of the decoder 200 is configured to decode the audio data extracted by the parser 205 to generate decoded audio data and to present the decoded audio data as output (e.g., to the post-processor 300). The state validator 203 is configured to authenticate and validate the LPSM (and optionally other metadata) presented thereto. In some embodiments, the LPSM is (or is included in) a data block that has been included in the input bitstream (e.g., in accordance with an embodiment of the present invention). The block may include a cryptographic hash (hash-based message authentication code or "HMAC") for processing the LPSM (and optionally other metadata) and / or the underlying audio data (provided to the validator 203 from the parser 205 and / or the decoder 202). The data block may, in these embodiments, be digitally signed, so that downstream audio processing units can relatively easily authenticate and validate the processing state metadata.

[0061] Other cryptographic methods, including but not limited to any one or more non-HMAC cryptographic methods, may be used for validation of the LPSM (e.g., in validator 203) to ensure secure transmission and reception of the LPSM and / or underlying audio data. For example, validation (using such cryptographic methods) may be performed at each audio processing unit receiving an embodiment of an audio bitstream of the present invention to determine whether loudness processing state metadata and corresponding audio data included in the bitstream have been subjected to (and / or result from) a particular loudness processing (as indicated by the metadata) and have not been modified since such particular loudness processing was performed.

[0062] State validator 203 provides control data to control bit generator 204 and / or provides the control data as output (e.g. to post-processor 300) to indicate the result of the validation operation. In response to the control data (and optionally other metadata extracted from the input bitstream), stage 204 may generate (and provide to post-processor 300) any of the following: a control bit indicating that the decoded audio data output from decoder 202 has undergone a particular type of loudness processing (e.g., when LPSM indicates that the audio data output from decoder 202 has undergone the particular type of loudness processing and the control bit from validity checker 203 indicates that LPSM is enabled); or a control bit indicating that the decoded audio data output from decoder 202 should undergo the particular type of loudness processing (e.g., when LPSM indicates that the audio data output from decoder 202 has not undergone the particular type of loudness processing, or when LPSM indicates that the audio data output from decoder 202 has undergone the particular type of loudness processing but the control bit from validity checker 203 indicates that LPSM is not enabled).

[0063] Alternatively, the decoder 200 submits the metadata (and any other metadata) extracted by the decoder 202 from the input bitstream to a post-processor 300 which performs loudness processing on the decoded audio data using the LPSM, performs a validation check of the LPSM, and then, if the validation check indicates that the LPSM is valid, performs loudness processing on the decoded audio data using the LPSM.

[0064] In some embodiments, when the decoder 201 receives an audio bitstream generated according to an embodiment of the present invention having a cryptographic hash, the decoder is configured to parse and extract the cryptographic hash from a determined data block from the bitstream, the block including loudness processing state metadata (LPSM). The validator 203 may use the cryptographic hash to validate the received bitstream and / or associated metadata. For example, if the validator 203 finds that the LPSM is valid based on a match between a reference cryptographic hash and the cryptographic hash extracted from the data block, the validator 203 may signal a downstream audio processing unit (e.g., a post-processor 300 that is or may include a volume leveling unit) to pass the audio data of the bitstream through (without modification). Additionally, optionally or alternatively, other types of cryptographic techniques may be used instead of a cryptographic hash-based method.

[0065] In some implementations of the decoder 100, the encoded bitstream received (and buffered in the memory 201) is an AC-3 or E-AC-3 bitstream and includes audio data segments (e.g., the AB0-AB5 segments of the frames shown in FIG. 4) and metadata segments, where the audio data segments represent audio data and where each of at least some of the metadata segments includes loudness processing state metadata (LPSM). The decoder stage 202 is configured to extract LPSMs from the bitstream, having the following format: Each of the metadata segments including the LPSM is included in an "addbsi" field of a bitstream information ("BSI") segment of a frame of the bitstream or in an auxiliary data field at the end of the frame of the bitstream (e.g., the AUX segment shown in FIG. 4). A frame of the bitstream may include one or two metadata segments each including an LPSM, and if a frame includes two metadata segments, one may be present in the addbsi field of the frame and the other may be present in the AUX field of the frame.Each metadata segment that contains an LPSM includes an LPSM payload (or container) segment having the following format: a header (e.g., including a sync word identifying the beginning of the LPSM payload, followed by identification information values, e.g., including the LPSM format version, length, period, count and sub-stream association values ​​shown in Table 2 below); after the header: at least one dialogue indication value indicating whether the corresponding audio data indicates dialogue or not (e.g., which channels of the corresponding audio data indicate dialogue) (e.g., which channels of the corresponding audio data indicate dialogue); at least one loudness regulation compliance value indicating whether the corresponding audio data complies with an indicated set of loudness regulations (e.g., the parameter "loudness regulation type" in Table 2); at least one loudness processing value indicating at least one type of loudness processing that has been performed on the corresponding audio data (e.g., one or more of the parameters "dialogue gated loudness compensation flag" and "loudness compensation type" in Table 2); and At least one loudness value indicating at least one loudness (e.g. peak or average loudness) characteristic of the corresponding audio data (e.g. one or more of the parameters "ITU relative gated loudness", "ITU speech gated loudness", "ITU (EBU3341) short-term 3s loudness" and "true peak" in Table 2).

[0066] In some implementations, the decoder stage 202 is configured to extract from the “addbsi” field or the auxiliary data field of a frame of the bitstream, a metadata segment having the following format, respectively: a header (e.g. including a sync word identifying the start of the metadata segment followed by at least one identification value, e.g. including the core element version, length and period, extension element count and sub-stream association values ​​shown in Table 1 below); and after the core header, at least one protection value useful for at least one of decryption, authentication and / or validation of the loudness processing state metadata or the corresponding audio data (e.g. the HMAC digest and audio fingerprint values ​​of Table 1); and also after the core header, if the metadata segment contains an LPSM, an LPSM payload identification (“ID”) and LPSM payload size values ​​identifying the following metadata as an LPSM payload and indicating the size of the LPSM payload. An LPSM payload (or container) segment (e.g., having the format specified above) follows the LPSM payload ID and LPSM payload size values.

[0067] More generally, encoded audio bitstreams produced by some embodiments have a structure that provides a mechanism for labeling metadata elements and sub-elements as core (mandatory) or extension (optional elements). This allows the data rate of the bitstream (including metadata) to be scaled across a multitude of applications. Core (mandatory) elements of the bitstream syntax should also be able to signal the presence (in-band) and / or remote (out of band) locations of extension (optional) elements associated with the audio content.

[0068] In some embodiments, a core element(s) is required to be present in all frames of the bitstream. Some sub-elements of a core element are optional and may be present in any combination. Extension elements are not required to be present in all frames (to limit bitrate overhead). Thus, an extension element may be present in some frames and absent in other frames. Some sub-elements of an extension element are optional and may be present in any combination, while some sub-elements of an extension element may be mandatory (i.e., mandatory if the extension element is present in a frame of the bitstream).

[0069] In some embodiments, an encoded audio bitstream is generated (e.g., by an audio processing unit embodying the invention) that includes a sequence of audio data segments and metadata segments. The audio data segments indicate audio data, and each of at least some of the metadata segments includes loudness processing state metadata (LPSM), and the audio data segments are time division multiplexed with the metadata segments. In some embodiments of this class, each of the metadata segments has a format described herein. In one format, the encoded bitstream is an AC-3 bitstream or an E-AC-3 bitstream, and each of the metadata segments that includes LPSM is included as additional bitstream information in an "addbsi" field of a bitstream information ("BSI") segment of a frame of the bitstream (as shown in FIG. 6) or in an auxiliary data field of a frame of the bitstream (e.g., by stage 107 of encoder 100). Each frame includes a core element having the format shown in Table 1 of FIG. 8 in the addbsi field of the frame.

[0070] In one format, each addbsi (or ancillary data) field that contains an LPSM contains a Core Header (and optionally additional Core Elements) and, after the Core Header (or after the Core Header and other Core Elements), the following LPSM values ​​(parameters): Payload ID (which identifies the metadata as an LPSM), which is followed by a Core Element value (e.g., as specified in Table 1); Payload Size (which indicates the size of the LPSM payload), which is followed by the Payload ID; LPSM Data (which is followed by the Payload ID and Payload Size values). This has the format shown in Table 2 of Figure 9.

[0071] In a second format of the encoded bitstream, the bitstream is an AC-3 bitstream or an E-AC-3 bitstream, and each of the metadata segments that contains an LPSM is included (e.g., by stage 107 of encoder 100) either: in an "addbsi" field of a bitstream information ("BSI") segment of a frame of the bitstream (as shown in FIG. 6); or in an auxiliary data field at the end of a frame of the bitstream (e.g., the AUX field shown in FIG. 4). A frame may contain one or two metadata segments, each containing an LPSM, and if a frame contains two metadata segments, one may be present in the addbsi field of the frame and the other in the AUX field of the frame. Each metadata segment that contains an LPSM has the format specified above with reference to Tables 1 and 2 above (i.e., it contains the core elements specified in Table 1, followed by a payload ID (which identifies the metadata as an LPSM) and a payload size value as specified above, followed by the payload (the LPSM data having the format shown in Table 2).

[0072] In another format, the encoded bitstream is a Dolby E bitstream, and each of the metadata segments containing LPSM is the first N sample positions of the Dolby E guard band interval. A Dolby E bitstream containing such a metadata segment containing LPSM includes a value indicating the LPSM payload length signaled, for example, in the Pd word of the SMPTE 337M preamble (the SMPTE 337M Pa word repetition rate may remain the same as the associated video frame rate).

[0073] In a format in which the encoded bitstream is an E-AC-3 bitstream, each of the metadata segments containing LPSMs is included (e.g., by stage 107 of encoder 100) as additional bitstream information in the "addbsi" field of the bitstream information ("BSI") segment of a frame of the bitstream. Further aspects of encoding an E-AC-3 bitstream with LPSMs in this format are as follows: (1) During the generation of the E-AC-3 bitstream, while the E-AC-3 encoder (inserting LPSM values ​​into the bitstream) is "active", the bitstream should include a metadata block (containing LPSMs) carried in the addbsi field of the frame for every frame (sync frame) that is generated. The bits required to carry the metadata block should not increase the encoder bitrate (frame length). (2) All metadata blocks (including LPSM) should contain the following information: loudness_correction_type_flag: where "1" indicates that the loudness of the corresponding audio data was corrected upstream from the encoder, and "0" indicates that the loudness was corrected by a loudness corrector built into the encoder (e.g., loudness processor 103 of encoder 100 in FIG. 2); speech_channel: indicates which source channel(s) contain speech (during the previous 0.5 seconds). If no speech is detected, this is indicated; speech_loudness: indicates the integrated speech loudness (over the previous 0.5 seconds) of each corresponding audio channel that contains speech; ITU_loudness: indicates the integrated ITU BS.1770-2 loudness of each corresponding audio channel; gain: the loudness composite gain(s) to invert at the decoder (to demonstrate reversibility).

[0074] While an E-AC-3 encoder (which inserts LPSM values ​​into the bitstream) is "active" and receives an AC-3 frame with the "trusted" flag, the loudness controller in the encoder (e.g. loudness processor 103 of encoder 100 in FIG. 2) is bypassed. The "trusted" source dialnorm and DRC values ​​should be passed (e.g. by generator 106 of encoder 100) to the E-AC-3 encoder component (e.g. stage 107 of encoder 100). LPSM block generation continues and loudness_correction_type_flag is set to "1". The loudness controller bypass sequence should be synchronized to the beginning of a decoded AC-3 frame in which the "trusted" flag appears. The loudness controller bypass sequence is implemented as follows: the leveler_amount control is decremented from value 9 to value 0 over 10 audio block periods (i.e., 53.3 msec), and the leveler_back_end_meter control is put into bypass mode (this action should give a seamless transition). The term "trusted" bypass of the leveler implies that the dialnorm value of the source bitstream is also reused at the encoder's output (e.g., if a "trusted" source bitstream has a dialnorm value of -30, the encoder's output should utilize -30 for the outgoing dialnorm value).

[0075] While the E-AC-3 encoder (which inserts LPSM values ​​into the bitstream) is "active" and receives AC-3 frames without the "confidence" flag, the loudness controller built into the encoder (e.g. loudness processor 103 of encoder 100 in Fig. 2) is active. LPSM block generation continues and loudness_correction_type_flag is set to "0". The loudness controller activation sequence is synchronized to the beginning of the decoded AC-3 frame where the "confidence" flag disappears. The loudness controller activation sequence is implemented as follows: the leveler_amount control is incremented from value 0 to value 9 over one audio block period (i.e. 5.3 msec) and the leveler_back_end_meter control is put into "active" mode (this action gives a seamless transition and includes a back_end_meter integration reset). During encoding, the Graphical User Interface (GUI) presented the user with the following parameters: "Input Audio Program [Trusted / Not Trusted]" - the state of this parameter is based on the presence of the "Trusted" flag in the input signal; and "Real-time Loudness Correction: [Enabled / Disabled]" - the state of this parameter is based on whether the loudness controller built into the encoder is active or not.

[0076] When decoding an AC-3 or E-AC-3 bitstream that has LPSM (in the described format) included in the "addbsi" field of the Bitstream Information ("BSI") segment of each frame of the bitstream, the decoder parses the LPSM block data (in the addbsi field) and passes the extracted LPSM values ​​to a Graphical User Interface (GUI). The set of extracted LPSM values ​​is refreshed every frame.

[0077] In yet another format, the encoded bitstream is an AC-3 bitstream or an E-AC-3 bitstream, and each of the metadata segments that contains an LPSM is included (e.g., by stage 107 of encoder 100) as additional bitstream information in the "addbsi" field (shown in FIG. 6) of the bitstream information ("BSI") segment of a frame of the bitstream (or in an Aux segment). In this format (which is a variation on the formats described above with reference to Tables 1 and 2), each of the addbsi (or Aux) fields that contains an LPSM contains the following LPSM values: the core elements as defined in Table 1, followed by a payload ID (which identifies the metadata as an LPSM) and a payload size value, followed by a payload (LPSM data) with the following format (similar to the elements shown in Table 2 above): LPSM payload version: a 2-bit field indicating the version of the LPSM payload; dialchan: a 3-bit field indicating whether the left, right and / or center channels of the corresponding audio data contain spoken dialogue. The bit assignment of the dialchan field may be as follows: bit 0 indicating the presence of dialogue in the left channel is stored in the most significant bit of the dialchan field, bit 2 indicating the presence of dialogue in the center channel is stored in the least significant bit of the dialchan field. Each bit of the dialchan field is set to "1" if the corresponding channel contains dialogue spoken during the preceding 0.5 seconds of the program; loudregtyp: a 3-bit field indicating to which regulatory standard the program loudness complies. Setting the "loudregtyp" field to "000" indicates that the LPSM does not indicate loudness regulatory compliance.For example, one value of this field (e.g., 000) may indicate that no compliance with a loudness regulatory standard is indicated, another value of this field (e.g., 001) may indicate that the audio data of the program conforms to the ATSC A / 85 standard, and another value of this field (e.g., 010) may indicate that the audio data of the program conforms to the EBU R128 standard. In this example, if this field is set to any value other than "000", the payload should be followed by the loudcorrdialgat and loudcorrtyp fields; loudcorrdialgat: a 1-bit field indicating whether dialogue-gated loudness correction was applied. If the loudness of the program has been corrected using dialogue gating, the value of the loudcorrdialgat field is set to "1". Otherwise, its value is set to "0"; loudcorrtyp: a 1-bit field indicating the type of loudness correction applied to the program. The value of the loudcorrtyp field is set to "0" if the program loudness has been corrected with an infinite look-ahead (file-based) loudness correction process. The value of this field is set to "1" if the program loudness has been corrected using a combination of real-time loudness measurement and dynamic range control; loudrelgate: a 1-bit field indicating whether relative gated loudness data (ITU) is present. If the loudrelgate field is set to "1", it should be followed in the payload by a 7-bit ituloudrelgat field; loudrelgat: a 7-bit field indicating the relative gated program loudness (ITU). This field indicates the integrated loudness of the audio program, measured according to ITU-R BS.1770-2, without the application of any gain adjustments due to dialnorm and dynamic range compression.Values ​​0 to 127 are interpreted as -58 LKFS to +5.5 LKFS in 0.5 LKFS increments; loudspchgate: a 1-bit field indicating whether speech-gated loudness data (ITU) is present. If the loudspchgate field is set to "1", it should be followed in the payload by a 7-bit loudspchgat field; loudspchgat: a 7-bit field indicating speech-gated program loudness. This field indicates the integrated loudness of the entire corresponding audio program, measured according to formula (2) of ITU-R BS.1770-3, without the application of any gain adjustments due to dialnorm and dynamic range compression. Values ​​0 to 127 are interpreted as -58 to +5.5 LKFS in 0.5 LKFS increments; loudstrm3se: a 1-bit field indicating whether short-term (3 second) loudness data is present. If this field is set to "1", it shall be followed in the payload by a 7-bit loudstrm3s field; loudstrm3s: a 7-bit field indicating the ungated loudness of the preceding 3 seconds of the corresponding audio program, measured in accordance with ITU-R BS.1771-1, without any gain adjustments due to dialnorm and dynamic range compression being applied. Values ​​0 to 256 are interpreted as -116LKFS to +11.5LKFS, in increments of 0.5LKFS; truepke: a 1-bit field indicating whether true peak loudness data is present. If the truepke field is set to "1", it shall be followed in the payload by an 8-bit truepk field; truepk: an 8-bit field indicating the true peak sample value of the program, measured in accordance with Annex 2 of ITU-R BS.1770-3, without any gain adjustments due to dialnorm and dynamic range compression being applied. Values ​​from 0 to 256 are interpreted as -116LKFS to +11.5LKFS in increments of 0.5LKFS.

[0078] In some embodiments, a Core element of a metadata segment in an ancillary data field (or "addbsi" field) of a frame of an AC-3 or E-AC-3 bitstream includes a Core Header (optionally including an identification value, e.g., Core element version), followed by: a value indicating whether fingerprint data (or other protection value) is included for the metadata of the metadata segment, a value indicating whether external data (related to the audio data corresponding to the metadata of the metadata segment) is present, a payload ID and payload size value for each type of metadata identified by the Core element (e.g., LPSM and / or non-LPSM types of metadata), and a protection value for at least one type of metadata identified by the Core element. The metadata payload(s) of the metadata segment follow the Core Header and are (possibly) nested within the value of the Core element.

[0079] Optimized loudness and dynamic range system The above secure metadata encoding and transmission scheme is used in the context of a scalable and extensible system for optimizing loudness and dynamic range across different playback devices, applications and listening environments, as shown in FIG. 1. In one embodiment, the system 10 is configured to normalize the loudness level and dynamic range of the input audio 11 across various devices that require different target loudness values ​​and have different dynamic range capabilities. To normalize the loudness level and dynamic range, the system 10 includes various device profiles along with the audio content, and normalization is based on those profiles. Those profiles may be included by one of the audio processing units in the audio processing chain, and the included profiles may be used by downstream processing units in the audio processing chain to determine the desired target loudness and dynamic range for the target device. Additional processing components may provide or process information for device profile management, gain control and wideband and / or multi-band gain generation functions (including (but not limited to) the following parameters: null band range, true peak threshold, loudness range, fast / slow time constants (coefficients) and maximum boost).

[0080] FIG. 10 shows a more detailed diagram of the system of FIG. 1 for a system providing optimized loudness and dynamic range control, under some embodiments. For the system 321 of FIG. 10, the encoder stage includes a core encoder component 304 that encodes the audio input 303 in a suitable digital format for transmission to a decoder 312. The audio is processed so that it can be reproduced in a variety of different listening environments, each of which may require different loudness and / or dynamic range target settings. Thus, as shown in FIG. 10, the decoder outputs a digital signal that is converted to an analog format by a digital-to-analog converter 316 for reproduction through a variety of different driver types, including a full-range speaker 320, a miniature speaker 322, and a headphone 324. These drivers represent just a few examples of possible reproduction drivers, and any transducer or driver of any suitable size may be used. Furthermore, the drivers / transducers 320-324 of FIG. 10 may be embodied in any suitable reproduction device for use in any corresponding listening environment. Device types may include, for example, AVRs, televisions, stereo equipment, computers, mobile phones, tablet computers, MP3 players, etc., and listening environments may include, for example, auditoriums, homes, automobiles, listening booths, etc.

[0081] Since the range of playback environments and driver types can vary from very small private contexts to very large public venues, the span of possible and optimal playback loudness and dynamic range configurations can vary greatly depending on the content type, background noise levels, etc. For example, in a home theater environment, wide dynamic range content can be played through surround sound equipment, narrower dynamic range content can be played through regular television systems (flat panel LED / LCD types, etc.), while very narrow dynamic range modes may be used for certain listening conditions where large level variations are not desired (e.g. at night, or for devices with severe acoustic output power limitations, e.g. built-in speakers or headphone outputs of mobile phones / tablets). In portable or mobile listening contexts, such as using small computer or dock speakers or headphones / earbuds, the optimal dynamic range of playback can vary depending on the environment. For example, in a quiet environment, the optimal dynamic range can be larger than in a noisy environment. The adaptive audio processing system embodiments of FIG. 10 vary the dynamic range to make audio content more intelligible depending on parameters such as the listening environment and playback device type.

[0082] FIG. 11 is a table showing various dynamic range requirements for various playback devices and background listening environments in one exemplary use case. Similar requirements can be derived for loudness. These various dynamic range and loudness requirements generate various profiles used by the optimization system 321. The system 321 includes a loudness and dynamic range measurement component 302 that analyzes and measures the loudness and dynamic range of the input audio. In one embodiment, the system analyzes the entire program content to determine the overall loudness parameters. In this context, loudness refers to the long-term program loudness or the average loudness of a program, where a program is a single unit of audio content such as a movie, television program, commercial, or similar program content. Loudness is used to provide an indication of the artistic dynamic range profile used by content creators to control how the audio is reproduced. Loudness is related to the dialnorm metadata value in that dialnorm represents the average dialogue loudness of a single program (e.g., movie, program, commercial, etc.). Short-term dynamic range quantifies the variation of a signal over a much shorter period of time than program loudness. For example, short-term dynamic range may be measured on the order of a few seconds, while program loudness may be measured over a span of minutes or even hours. Short-term dynamic range provides a protection mechanism independent of program loudness to ensure that overload does not occur for various playback profiles and device types. In one embodiment, the loudness (long-term program loudness) target is based on dialogue loudness, and the short-term dynamic range is based on relative gated and / or ungated loudness. In this case, certain DRC and loudness components in the system are context-aware with respect to content type and / or target device type and characteristics.As part of this context-awareness functionality, the system is configured to analyze one or more characteristics of the output device to determine whether the device is a member of a particular group of devices that are optimized for certain DRC and loudness reproduction conditions, such as AVR-type devices, televisions, computers, portable devices, etc.

[0083] The pre-processing component analyzes the program content to determine loudness, peaks, true peaks and quiet periods to generate unique metadata for each profile of a number of different profiles. In one embodiment, loudness may be dialogue-gated loudness and / or relative-gated loudness. The different profiles define various DRC (Dynamic Range Control) and target loudness modes, in which different gain values ​​are generated at the encoder depending on the source audio content, the desired target loudness and the characteristics of the playback device type and / or environment. The decoder may offer various DRC and target loudness modes (enabled by the profiles mentioned above), which may include: DRC and target loudness off / disable allows full dynamic range listing without compression and loudness normalization of the audio signal. Loudness normalization with a target of -31 LKFS Line mode for playback on a home theater system provides moderate dynamic range compression through gain values ​​generated (specifically for this playback mode and / or device profile) in the encoder with loudness normalization with a target of -31 LKFS. RF mode for playback through TV speakers provides severe dynamic range compression with loudness normalization with a target of -24, -23 or -20 LKFS. Intermediate mode for playback on a computer or similar device provides compression with loudness normalization with a target of -14 LKFS. Portable mode provides very severe dynamic range compression with a loudness normalization target of -11 LKFS. The target loudness values ​​of -31, -23 / -20, -14 and -11 LKFS are intended to be examples of different playback / device profiles that may be defined for the system under some embodiments; any other suitable target loudness values ​​may be used, and the system will generate appropriate gain values ​​for those particular playback modes and / or device profiles.Furthermore, the system is extensible and adaptable to accommodate different playback devices and listening environments by defining new profiles in the encoder or elsewhere that are loaded into the encoder. In this way, new, unique playback / device profiles can be created to support improved or different playback devices for future applications.

[0084] In an embodiment, the gain values ​​may be calculated in any suitable processing component of the system 321, such as the encoder 304, the decoder 312, or the transcoder 308, or any associated pre-processing component associated with the encoder, or any post-processing component associated with the decoder.

[0085] FIG. 13 is a block diagram illustrating the interface between different profiles for various different playback device classes, under one embodiment. As shown in FIG. 13, an encoder 502 receives an audio input 501 and one of several different profiles 506. The encoder combines the audio data with the selected profile to generate an output bitstream file. The output bitstream file is to be processed in a decoder component within or associated with the target playback device. For the example of FIG. 13, the different playback devices may be a computer 510, a mobile phone 512, an AVR 514, and a television 516, although many other output devices are possible. Each of the devices 510-516 includes or is coupled to a speaker (including a driver and / or transducer), such as drivers 320-324. The combination of processing, power rating, and size of the playback device and associated speakers generally dictates which profile is most optimal for that particular target. Thus, a profile 506 may be specifically defined for playback through an AVR, a TV, a mobile speaker, a mobile headphone, etc. Profiles may also be defined for particular operating modes or conditions, such as quiet mode, night mode, outdoors, indoors, etc. The profiles shown in Figure 13 are merely example modes and any suitable profiles may be defined, including custom profiles for particular goals and environments.

[0086] It should be noted that although Figure 13 illustrates an embodiment in which the encoder 502 receives the profile 506 and generates appropriate parameters for loudness and DRC processing, the parameters generated based on the profile and audio content can be executed on any appropriate audio processing unit, such as an encoder, decoder, transcoder, pre-processor, post-processor, etc. For example, each output device 510-516 of Figure 13 has or is coupled to a decoder component that processes metadata in the bitstream in the file 504 sent from the encoder 502 to enable adaptation of loudness and dynamic range to match the device or device type of the target output device.

[0087] In one embodiment, the dynamic range and loudness of the audio content are optimized for each possible playback device. This is achieved by keeping the long-term loudness targeted and controlling the short-term dynamic range to optimize the audio experience for each of the target playback modes (by controlling signal dynamics, sample peaks and / or true peaks). Different metadata elements are defined for the long-term loudness and short-term dynamic range. As shown in FIG. 10, component 302 analyzes the entire input audio signal (or a part of it, e.g., the speech component, if applicable) to derive meaningful characteristics for both of these separate DR components. This allows different gain values ​​to be defined for artistic gain and clip (overload protection) gain values.

[0088] These gain values ​​for long term loudness and short term dynamic range are then mapped to a profile 305 to provide parameters describing the loudness and dynamic range control gain values. These parameters are combined with the encoded audio signal from the encoder 304 in a multiplexer 306 or similar component to generate a bitstream that is transmitted to a decoder stage via a transcoder 308. The bitstream input to the decoder stage is demultiplexed in a demultiplexer 310 and then decoded in a decoder 312. A gain component 314 applies a gain corresponding to the appropriate profile to generate digital audio data. The digital audio data is then processed through a DACS unit 416 for playback through an appropriate playback device and driver or transducer 320-324.

[0089] FIG. 14 is a table illustrating the correlation between long-term loudness and short-term dynamic range for multiple defined profiles, under one embodiment. As shown in Table 4 of FIG. 14, each profile includes a set of gain values ​​that specify the amount of dynamic range compression (DRC) to be applied at the decoder or each target device of the system. Each of the N profiles, denoted Profiles 1 through N, sets a particular long-term loudness parameter (e.g., dialnorm) and overload compression parameter by specifying a corresponding gain value to be applied at the decoder stage. The DRC gain values ​​for these profiles may be defined by an external source accepted by the encoder, or may be generated internally within the encoder as default gain values ​​if no external values ​​are provided.

[0090] In one embodiment, the gain values ​​for each profile are embodied in DRC gain terms that are calculated based on an analysis of certain characteristics of the audio signal, such as peaks, true peaks, dialogue short term loudness or overall short term loudness or a combination of both (hybrid). The static gain (i.e. transfer characteristic or curve) based on the chosen profile and the time constants required to implement the fast / slow attack and fast / slow release of the final DRC gain for each possible device profile and / or target loudness are calculated. As mentioned above, these profiles may be pre-configured in the encoder, decoder or may be generated externally and conveyed to the encoder via external metadata from the content creator.

[0091] In some embodiments, the gain values ​​may be broadband gains that apply the same gain to all frequencies of the audio content. Alternatively, the gains may consist of multi-band gain values, with different gain values ​​applied to different frequencies or frequency bands of the audio content. In the multi-channel case, each profile may comprise a matrix of gain values ​​indicating the gains for various frequency bands instead of a single gain value.

[0092] Referring to FIG. 10, in one embodiment, information about attributes or characteristics of the listening environment and / or the capabilities and configuration of the playback device is provided by the decoder stage to the encoder stage by a feedback link 330. Profile information 332 is also input to the encoder 304. In one embodiment, the decoder analyzes metadata in the bitstream to determine whether loudness parameters for a first group of audio playback devices are available in the bitstream. If so, the decoder transmits those parameters downstream for use in rendering the audio. If not, the encoder analyzes certain characteristics of the device to derive those parameters. These parameters are then sent to a downstream rendering component for playback. The encoder also determines the output device (or group of output devices including the output device) that will render the received audio stream. For example, the output device may be determined to be a mobile phone or belong to a group such as a portable device. In one embodiment, the decoder indicates the determined output device or group of output devices to the encoder using the feedback link 330. For this feedback, a module connected to an output device (e.g. a module in a sound card connected to a headset or to speakers in a laptop) may indicate to the decoder the identity of the output device or the group of devices that includes the output device. The decoder sends this information to the encoder through the feedback link 330. In one embodiment, the decoder executes and the decoder determines the loudness and DRC parameters. In one embodiment, the decoder determines the loudness and DRC parameters. In this embodiment, instead of sending the information through the feedback link 330, the decoder uses information about the determined device or group of output devices to determine the loudness and DRC parameters. In another embodiment, a separate audio processing unit determines the loudness and DRC parameters and the decoder sends the information to that audio processing unit instead of the decoder.

[0093] FIG. 12 is a block diagram of a dynamic range optimization system, under an embodiment. As shown in FIG. 12, an encoder 402 receives input audio 401. The encoded audio is combined with parameters 404 generated from a selected compression curve 422 and a dialnorm value 424 in a multiplexer 409. The resulting bitstream is transmitted to a demultiplexer 411, which generates an audio signal that is decoded by a decoder 406. The parameters and the dialnorm value are used by a gain calculation unit 408 to generate a gain level that drives an amplifier 410 for amplifying the decoder output. FIG. 12 shows how the dynamic range control is parameterized and inserted into the bitstream. Loudness can also be parameterized and inserted into the bitstream using similar components. In an embodiment, an output reference level control (not shown) can also be provided to the decoder. Although the diagram shows the loudness and dynamic range parameters as being determined and inserted in the encoder, similar determinations can be performed in other audio processing units, such as a pre-processor, decoder and post-processor.

[0094] FIG. 15 illustrates examples of loudness profiles for various types of audio content, under one embodiment. As shown in FIG. 15, exemplary curves 600 and 602 plot input loudness (at LKFS) versus gain centered at 0 LKFS. Different types of content exhibit different curves as shown in FIG. 15. In the figure, curve 600 may represent speech and curve 602 may represent standard film content. As shown in FIG. 15, speech content receives a larger amount of gain compared to film content. FIG. 15 is intended to be an example of a representative profile curve for certain types of audio content, and other profile curves may be used. Certain aspects of the profile characteristics as shown in FIG. 15 are used to derive meaningful parameters for an optimization system. In one embodiment, these parameters include: null bandwidth, cut ratio, boost ratio, maximum boost, FS attack, FS decay, holdoff, peak limiting, and target level loudness. Other parameters may be used in addition to or instead of at least some of these parameters depending on the application requirements and system constraints.

[0095] FIG. 16 illustrates a method for optimizing loudness and dynamic range across playback devices and applications, under an embodiment. Although the figure illustrates loudness and dynamic range optimization as being performed in an encoder, similar optimization can be performed in other audio processing units, such as pre-processors, decoders, and post-processors. As shown in process 620, the method begins with an encoder stage receiving an input signal from a source (603). The encoder or pre-processing component then determines whether the source signal has undergone a process that achieves a target loudness and / or dynamic range (604). The target loudness corresponds to a long-term loudness and can be defined externally or internally. If the source signal has not undergone processing to achieve a target loudness and / or dynamic range, the system performs the appropriate loudness and / or dynamic range control operation (608); otherwise, if the source signal has undergone this loudness and / or dynamic range control operation, the system enters a bypass mode to skip the loudness control and / or dynamic range operation and allow the original process to specify the appropriate long-term loudness and / or dynamic range (606). The appropriate gain value for either the bypass mode 606 or the executed mode 608 (which may be a single wideband gain value or may be frequency-dependent multi-band gain values) is then applied in the decoder (612).

[0096] Bitstream Format As previously described, the system for optimizing loudness and dynamic range uses a secure extensible metadata format to ensure that metadata and audio content transmitted in a bitstream between an encoder and a decoder or between a source and a rendering / playback device are not separated from each other or otherwise corrupted during transmission over a network or other proprietary facility such as a service provider interface. The bitstream provides a mechanism for signaling for an encoder and / or decoder component to adapt the loudness and dynamic range of an audio signal to match the audio content and output device characteristics through appropriate profile information. In an embodiment, the system is configured to determine a low bitrate encoded bitstream to be transmitted between the encoder and the decoder, and the loudness information encoded through the metadata includes characteristics for one or more output profiles. A bitstream format for use with the loudness and dynamic range optimization system under an embodiment is described below.

[0097] An AC-3 encoded bitstream contains metadata and one to six channels of audio content. The audio content is audio data compressed using perceptual audio coding. The metadata includes several audio metadata parameters that are intended for use in altering the sound of the program delivered to the listening environment. Each frame of an AC-3 encoded audio bitstream contains audio content and metadata for 1536 samples of digital audio. For a sampling rate of 48 kHz, this represents 32 milliseconds of digital audio or a rate of 31.25 frames of audio per second.

[0098] Each frame in an E-AC-3 encoded audio bitstream contains audio content and metadata for 256, 512, 768 or 1536 samples of digital audio, respectively, depending on whether the frame contains one, two, three or six blocks of audio data. For a sampling rate of 48 kHz, this represents 5.333, 10.667, 16 or 32 milliseconds of digital audio, respectively, or a rate of 189.9, 93.75, 62.5 or 31.25 frames per second of audio, respectively.

[0099] As shown in Figure 4, each AC-3 frame is divided into sections (segments), which include (as shown in Figure 5) a synchronization information (SI) section containing the synchronization word (SW) and the first of two error correction words (CRC1); a bitstream information (BSI) section containing most of the metadata; six audio blocks (AB0 through AB5) containing the data-compressed audio content (and which may also contain metadata); a waste bits segment (W) containing any unused bits left after the audio content is compressed; an auxiliary (AUX) information section which may contain further metadata; and the second of two error correction words (CRC2).

[0100] As shown in Figure 7, each E-AC-3 frame is divided into sections (segments) that contain the synchronization information (SI) section, which contains the synchronization word (SW) (as shown in Figure 5); the bitstream information (BSI) section, which contains most of the metadata; between one and six audio blocks (AB0 to AB5) that contain the data-compressed audio content (and may also contain metadata); a waste bit segment (W) that contains any unused bits left after the audio content has been compressed; an auxiliary (AUX) information section, which may contain further metadata; and an error correction word (CRC).

[0101] In an AC-3 (or E-AC-3) bitstream, there are several audio metadata parameters that are specifically intended for use in altering the sound of the program delivered to the listening environment. One such metadata parameter is the dialnorm parameter, which is contained in the BSI segment.

[0102] As shown in Figure 6, the BSI segment of an AC-3 frame includes a five-bit parameter ("dialnorm") that indicates the dialnorm value for the program, and a five-bit parameter ("dialnorm2") that indicates the dialnorm value for a second audio program carried in the same AC-3 frame if the audio coding mode ("acmod") for the AC-3 frame is "0", indicating that a dual mono or "1+1" channel configuration is being used.

[0103] The BSI segment includes a flag ("addbsie") indicating the presence (or absence) of additional bitstream information following the "addbsie" bit, a parameter ("addbsil") indicating the length of the additional bitstream information, if any, following the "addbsil" value, and up to 64 bits of additional bitstream information ("addbsi") following the "addbsil" value.

[0104] The BSI segment may include other metadata values ​​not specifically shown in FIG.

[0105] Aspects of one or more of the embodiments described herein may be implemented in an audio system that processes audio signals for transmission over a network, including one or more computers or processing devices executing software instructions. Any of the described embodiments may be used alone or with each other in any combination. Various embodiments may be motivated by various shortcomings in the prior art that may be discussed or alluded to in one or more places herein, but the embodiments do not necessarily address any of these shortcomings. That is, various embodiments may address various shortcomings that may be discussed herein. Some embodiments may only partially address some or only one of the shortcomings that may be discussed herein, and some embodiments may not address any of these shortcomings.

[0106] Aspects of the system described herein may be implemented in a computer-based sound processing network environment suitable for processing digital or digitized audio files. Portions of the adaptive audio system may include one or more networks including any desired number of individual machines, including one or more routers (not shown) that serve to buffer and route data transmitted between computers. Such networks may be built on a variety of network protocols and may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0107] One or more of the above components, blocks, processes, or other functional elements may be implemented through a computer program that controls the execution of a processor-based computing device of the system. It should also be noted that the various functions disclosed herein may be described with behavioral, register transfer, logical components, and / or other characteristics using any number of combinations of hardware, firmware, and / or as data and / or instructions embodied in various machine-readable or computer-readable media. The computer-readable media on which such formatted data and / or instructions may be embodied include, but are not limited to, various forms of physical (non-transitory), non-volatile storage media, such as optical, magnetic, or semiconductor storage media.

[0108] Unless the context clearly requires otherwise, throughout this description and the claims, the words "having," "including," and the like are to be interpreted in an inclusive sense rather than an exclusive or exhaustive sense; i.e., "including but not limited to." Words using the singular or plural also include the plural or singular, respectively. Furthermore, the words "herein," "herein," "above," "below," and words of similar meaning refer to this application as a whole and not to any particular portions of this application. When the word "or" is used in reference to a list of two or more items, the word covers all of the following interpretations of that word: any of the items in the list, all of the items in the list, and any combination of the items in the list.

[0109] While one or more implementations have been described using specific embodiments as examples, it is to be understood that the one or more implementations are not limited to the disclosed embodiments. On the contrary, it is intended to cover various modifications and similar arrangements that would be apparent to those skilled in the art. Therefore, the scope of the appended claims should be accorded the broadest interpretation so as to encompass all such modifications and similar arrangements.

[0110] Several aspects will be described. [Aspect 1] receiving metadata associated with the audio data in a bitstream; analyzing the metadata in the bitstream to determine whether loudness parameters for a first group of audio playback devices are available in the bitstream; in response to determining that the parameters are present for the first group, using the parameters and audio data to render audio; and in response to determining that the loudness parameter is not present for the first group, analyzing one or more characteristics of the first group and determining the parameter based on the one or more characteristics. method. [Aspect 2] 2. The method of aspect 1, wherein the one or more characteristics include gain levels for different profiles of the audio data. [Aspect 3] The method of claim 2, wherein the gain levels define at least one of: an artistic dynamic range profile that controls how the audio data is reproduced for a defined program; and a short-term dynamic range profile that provides overload protection for portions of the defined program. Aspect 4 2. The method of embodiment 1, wherein the step of using the parameters and audio data to render audio includes sending the parameters and audio data to a downstream module that renders the audio for playback. Aspect 5 2. The method of embodiment 1, wherein using the parameters and audio data to render audio includes rendering the audio data based on the parameters and audio data. Aspect 6 determining an output device for rendering the received audio stream; and determining whether the output device belongs to the first group of audio playback devices, wherein the step of analyzing metadata in the stream to determine whether loudness parameters are available for the first group of audio playback devices is performed after the step of determining that the output device belongs to the first group of audio playback devices. The method of embodiment 1. Aspect 7 The step of determining that the output device belongs to the first group of audio reproduction devices comprises: receiving an indication from a module connected to the output device indicating an identity of the output device or indicating an identity of a group of devices that includes the output device; determining that the output device belongs to the first group of audio reproduction devices based on the received indication; The method of embodiment 6. Aspect 8 an interface configured to receive a bitstream including metadata associated with the audio data; an analyzer coupled to the interface configured to analyze the metadata in the bitstream to determine whether loudness parameters for a first group of audio playback devices are available in the bitstream; a rendering component configured to use the parameters and audio data to render audio in response to the analyzer determining that the parameters are present for the first group, and further configured to analyze one or more characteristics of the first group and determine the parameters based on the one or more characteristics in response to the analyzer determining that the loudness parameter is not present for the first group. Device. Aspect 9 The apparatus of embodiment 8, wherein the rendering component using the parameters and audio data to render audio includes the rendering component sending the parameters and audio data to a downstream module that renders the audio for playback. Aspect 10 10. The apparatus of claim 9, wherein the rendering component using the parameters and audio data to render audio comprises the rendering component rendering the audio data based on the parameters and audio data. Aspect 11 a second component configured to identify an output device for rendering the received audio stream and determine whether the output device belongs to the first group of audio playback devices; the analyzer analyzes metadata in the stream to determine whether loudness parameters for the first group of audio playback devices are available after the second component determines whether the output device belongs to the first group of audio playback devices; 11. The apparatus of embodiment 10. Aspect 12 The device of embodiment 11, further comprising an interface configured to receive an indication from a module connected to the output device indicating an identity of the output device or an identity of a group of devices that includes the output device, and based on the received indication, the output device belongs to the first group of audio playback devices. Aspect 13 receiving audio data and metadata associated with the audio data; analyzing the metadata in the bitstream to determine whether loudness information relating to loudness parameters for a first group of audio devices is available in the bitstream; in response to determining that the loudness information is present for the first group, determining loudness information from the stream and sending the audio data and loudness information for use in rendering audio; in response to determining that the loudness information is not present for the first group, determining loudness information associated with an output profile and sending the determined loudness information for the output profile for use in rendering the audio. method. Aspect 14 The step of determining loudness information associated with the output profile further comprises: Analyzing a characteristic of the output profile; determining the parameter based on the characteristic; Transmitting the determined loudness information includes transmitting the determined parameters. The method of embodiment 13. Aspect 15 15. The method of embodiment 14, wherein the characteristics include gain levels for different profiles of the audio data. Aspect 16 The method of embodiment 15, wherein the gain levels define at least one of: an artistic dynamic range profile that controls how the audio data is reproduced for a defined program; and a short-term dynamic range profile that provides overload protection for portions of the defined program. Aspect 17 14. The method of embodiment 13, wherein the loudness information comprises loudness parameters for an output profile. Aspect 18 14. The method of claim 13, wherein the loudness information comprises the characteristics of the output profile. Aspect 19 14. The method of embodiment 13, further comprising determining a low bit rate encoded stream to be transmitted, wherein the loudness information includes characteristics for one or more output profiles. Aspect 20 20. The method of embodiment 17, wherein the one or more output profiles do not include a premium content profile. Aspect 21 13. An apparatus for decoding audio data comprising: an interface for receiving the audio data and metadata associated with the audio data; a first component that analyzes metadata in a bitstream to determine whether loudness information associated with loudness parameters for a first group of audio devices is available in the stream, the first component, in response to determining that the loudness information is present for the first group, determining loudness information from the stream and sending the audio data and loudness information for use in rendering the audio, and in response to determining that the loudness information is not present for the first group, determining loudness information associated with an output profile and sending the determined loudness information for the output profile for use in rendering the audio. Device. Aspect 22 wherein said first component analyzing the metadata includes said first component analyzing characteristics of said output profile and determining said parameters based on said characteristics, and wherein said first component transmitting said determined loudness information includes transmitting the determined parameters. 22. The device of embodiment 21. Aspect 23 23. The apparatus of embodiment 22, wherein the loudness information includes loudness parameters for an output profile. Aspect 24 24. The apparatus of embodiment 23, wherein the loudness information includes the characteristics of the output profile. Aspect 25 25. The apparatus of embodiment 24, further comprising a second component that determines a low bit rate encoded stream to be transmitted, the loudness information including characteristics for one or more output profiles.

Claims

1. 1. An audio processing device for decoding one or more frames of an encoded audio bitstream, the encoded audio bitstream including audio data and metadata for a plurality of dynamic range control (DRC) profiles, the audio processing device comprising: a bitstream parser configured to parse the encoded audio bitstream to extract the encoded audio data and metadata for one or more of the DRC profiles; an audio decoder configured to decode the encoded audio data and apply a DRC gain to the decoded audio data; Each DRC profile is suitable for at least one device type or listening environment; the audio decoder selecting one or more of the DRC profiles in response to information about the audio processing device or the listening environment; the DRC gains applied to the decoded audio data are different for different frequency bands and correspond to the one or more selected DRC profiles. Audio processing device.

2. 1. A method performed by an audio processing device for decoding one or more frames of an encoded audio bitstream, the encoded audio bitstream including audio data and metadata for a plurality of dynamic range control (DRC) profiles, the method comprising: parsing the encoded audio bitstream to extract the encoded audio data and metadata for one or more of the DRC profiles; decoding the encoded audio data and applying a DRC gain to the decoded audio data; Each DRC profile is suitable for at least one device type or listening environment; one or more of the DRC profiles are selected in response to information about the audio processing device or the listening environment; the DRC gains applied to the decoded audio data are different for different frequency bands and correspond to the one or more selected DRC profiles. method.

3. A non-transitory computer readable storage medium having sequences of instructions which, when executed by an audio decoding device, cause the audio decoding device to perform the method of claim 2.

4. A computer program product having executable instructions for performing the method of claim 2 when executed on a computer.

Images