Optimization of loudness and dynamic range across different playback devices.

Abstract

It provides optimization of loudness and dynamic range across different playback devices. [Solution] Embodiments are directed to a method and system for receiving metadata associated with audio data in a bitstream and analyzing the metadata to determine whether loudness parameters for a first group of audio playback devices are available in the bitstream. In response to determining that the parameters exist for the first group, the system uses the parameters and audio data to render the audio. In response to determining that the loudness parameters do not exist for the first group, the system analyzes one or more characteristics of the first group and determines the parameters based on the one or more characteristics.

Description

Technical Field

[0004] , ,

[0001] Cross - Reference to Related Applications This application claims priority to U.S. Provisional Patent Application No. 61 / 754,882, filed on January 21, 2013; U.S. Provisional Patent Application No. 61 / 809,250, filed on April 5, 2013; and U.S. Provisional Patent Application No. 61 / 824,010, filed on May 16, 2013.

[0002] Field of the Invention One or more embodiments generally relate to audio signal processing, and more particularly to processing an audio data bitstream having metadata indicative of loudness and dynamic range characteristics of audio content based on a playback environment and device.

Background Art

[0005] In known audio encoding systems, metadata associated with the audio signal is used to set the DRC level based on the type of content and its intended use. The DRC mode sets the amount of compression applied to the audio signal and defines the decoder's output reference level. Such systems may be limited to two DRC levels programmed in the encoder and selected by the user. For example, a -31dB (line) dialnorm (dialog normalization) value is traditionally used for content played on AVR or full dynamic range equipment, while a -20dB dialnorm value is used for content played on television sets or similar equipment. This type of system allows a single audio bitstream to be used in two common but very different playback scenarios by using two different sets of DRC metadata. However, such systems are limited to pre-set dialnorm values ​​and are not optimized for playback in the wide variety of different playback devices and listening environments made possible today through the advent of digital media and internet-based streaming technologies.

[0006] In current metadata-based audio encoding systems, a stream of audio data can contain both audio content (e.g., one or more channels of the audio content) and metadata that describes at least one characteristic of the audio content. For example, in an AC-3 bitstream, there are several audio metadata parameters that are specifically intended for use in altering the sound of a program delivered to a listening environment. One such metadata parameter is the `dialnorm` parameter, which indicates the average loudness level of dialogue appearing in an audio program (or the average loudness of the content) and is used to determine the audio playback signal level.

[0007] During playback of a bitstream containing a sequence of different audio program segments (each with a different dialnorm parameter), the AC-3 decoder performs a type of loudness processing that modifies the playback level or loudness of the segments using the dialnorm parameter of each segment so that the perceived loudness of the dialogue in the segments is at a consistent level. Each encoded audio segment (item) in a sequence of encoded audio items (generally) has a different dialnorm parameter, and the decoder scales the level of each item so that the playback level or loudness of the dialogue for each item is the same or very similar. However, this may require applying different amounts of gain to different items during playback.

[0008] In some embodiments, the dialnorm parameter is set by the user and not automatically generated. However, there is a default dialnorm value if the value is not set by the user. For example, a content creator may use an external device to the AC-3 encoder to measure loudness and then transfer the results (showing the loudness of the spoken dialogue in the audio program) to the encoder to set the dialnorm value. Thus, the content creator is relied upon to correctly set the dialnorm parameter.

[0009] There are several different reasons why the dialnorm parameter in an AC-3 bitstream might be incorrect. Firstly, each AC-3 encoder has a default dialnorm value that is used during bitstream generation if the content creator does not set a dialnorm value. This default value may be substantially different from the actual dialogue loudness level of the audio. Secondly, even if the content creator measures loudness and sets the dialnorm value appropriately, a loudness measurement algorithm or meter that does not follow the recommended loudness measurement method may be used, leading to an incorrect dialnorm value. Thirdly, even if the AC-3 bitstream is generated with a correctly measured and set dialnorm value by the content creator, it may be changed to an incorrect value by an intermediate module during bitstream transmission and / or storage. For example, in television broadcast applications, it is not uncommon for AC-3 bitstreams to be decoded, modified, and then re-encoded with incorrect dialnorm metadata information. Thus, the dialnorm values ​​included in the AC-3 bitstream may be incorrect or inaccurate, and therefore may have a negative impact on the quality of the listening experience.

[0010] Furthermore, the `dialnorm` parameter does not indicate the loudness processing status of the corresponding audio data (e.g., what type (singular or plural) of loudness processing was performed on that audio data). Moreover, currently deployed loudness and DRC systems, such as those in Dolby Digital (DD) and Dolby Digital Plus (DD+) systems, are designed to render AV content in a consumer's living room or cinema. To adapt such content for playback in other environments and listening equipment (e.g., mobile devices), post-processing must be applied "blindly" in the playback device to adapt the AV content to that listening environment. In other words, the post-processor (or decoder) assumes that the loudness level of the received content is at a specific level (e.g., -31 or -20 dB) and sets the level to a predetermined fixed target level suitable for the particular device. If the assumed loudness level or predetermined target level is incorrect, the post-processing may have the opposite effect of its intended effect. In other words, post-processing can make the output audio less desirable for the user.

[0011] The disclosed embodiments are not limited to use with AC-3 bitstreams, E-AC-3 bitstreams, or Dolby E bitstreams, but for convenience, such bitstreams are discussed in relation to a system that includes loudness processing state metadata. Dolby, Dolby Digital, Dolby Digital Plus, and Dolby E are trademarks of Dolby Laboratories Licensing Corporation. Dolby Laboratories provides its own implementations of AC-3 and E-AC-3, known as Dolby Digital and Dolby Digital Plus, respectively. [Overview of the project] [Means for solving the problem]

[0012] Embodiments are directed toward a method for decoding audio data by receiving a bitstream containing metadata associated with audio data and analyzing the metadata in the bitstream to determine whether loudness parameters for a first group of audio playback devices are available in the bitstream. In response to determining that those parameters are present for the first group, the processing component uses those parameters and audio data to render the audio. In response to determining that those loudness parameters are not present for the first group, the processing component analyzes one or more characteristics of the first group and determines the parameters based on the one or more characteristics. The method may further use the parameters and audio data to render the audio by transmitting the parameters and audio data to a downstream module that renders the audio for playback. The parameters and audio data may also be used to render the audio by rendering the audio data based on the parameters and audio data.

[0013] In one embodiment, the method also includes determining an output device that renders an received audio stream and determining whether the output device belongs to the first group of audio playback devices. Herein, the step of analyzing metadata in the stream to determine whether loudness parameters for the first group of audio playback devices are available is performed after the step of determining whether the output device belongs to the first group of audio playback devices. In one embodiment, the step of determining whether the output device belongs to the first group of audio playback devices includes receiving an index from a module connected to the output device that indicates the character of the output device or the character of a group of devices including the output device, and determining whether the output device belongs to the first group of audio playback devices based on the received index.

[0014] The embodiments are further directed to an apparatus or system that includes a processing component that performs the steps described in the above-described encoding method embodiments.

[0015] Embodiments further relate to a method for decoding audio data by receiving audio data and metadata associated with the audio data, analyzing the metadata in the bitstream to determine whether loudness information related to loudness parameters for a first group of audio devices is available in the stream, determining loudness information from the stream in response to determining that the loudness information exists for the first group, and transmitting the audio data and loudness information for use in audio rendering, or, if the loudness information does not exist for the first group, determining loudness information associated with an output profile and transmitting the determined loudness information for the output profile for use in audio rendering. In some embodiments, the step of determining loudness information associated with an output profile may include analyzing the characteristics of the output profile and determining the parameters based on the characteristics, and transmitting the determined loudness information may include transmitting the determined parameters. The loudness information may include loudness parameters for the output profile or characteristics of the output profile. In one embodiment, the method further includes determining a low-bitrate encoded stream to be transmitted, and the loudness information includes characteristics for one or more output profiles.

[0016] The embodiments are further directed towards an apparatus or system that includes a processing component that performs the steps described in the above-described decoding method embodiment. [Brief explanation of the drawing]

[0017] In the following diagrams, similar reference numerals are used to refer to the same elements. The following diagrams illustrate various examples, but the implementations described in this paper are not limited to those shown in the diagrams. [Figure 1] This is a block diagram of an embodiment of an audio processing system configured to perform loudness and dynamic range optimization under several embodiments. [Figure 2] This is a block diagram of an encoder for use in the system shown in Figure 1, under several embodiments. [Figure 3] This is a block diagram of a decoder for use in the system shown in Figure 1, under several embodiments. [Figure 4] This diagram shows the AC-3 frame, including its divided segments. [Figure 5] This diagram shows the Synchronization Information (SI) segment of an AC-3 frame, including the various segments into which it is divided. [Figure 6] This diagram shows the bitstream information (BSI) segment of an AC-3 frame, including the segments into which it is divided. [Figure 7] This diagram shows the E-AC-3 frame, including its divided segments. [Figure 8] This is a table showing the format of certain frames and metadata of an encoded bitstream under several embodiments. [Figure 9] This table shows the format of loudness processing status metadata under several embodiments. [Figure 10] Figure 1 is a more detailed block diagram of the audio processing system, which may be configured to perform loudness and dynamic range optimization under several embodiments. [Figure 11]A table showing various dynamic range requirements for various playback devices and background listening environments in exemplary use cases. [Figure 12] A block diagram of a dynamic range optimization system under an embodiment. [Figure 13] A block diagram showing an interface between different profiles for various different playback device classes under an embodiment. [Figure 14] A table showing the correlation between long-term loudness and short-term dynamic range for a plurality of defined profiles under an embodiment. [Figure 15] A diagram showing examples of loudness profiles for various types of audio content under an embodiment. [Figure 16] A flowchart showing a method for optimizing loudness and dynamic range across playback devices and applications under an embodiment.

Mode for Carrying Out the Invention

[0018] <Definitions and Nomenclature> Throughout this disclosure, including the claims, the expression "performing an operation on a signal or data" (e.g., filtering, scaling, transforming, or applying gain to a signal or data) is used broadly to mean performing the operation directly on the signal or data, or on a processed version of the signal or data (e.g., on a version of the signal that has undergone preliminary filtering or preprocessing prior to the performance of the operation). The expression "system" is used broadly to mean a device, system, or subsystem. For example, a subsystem implementing a decoder may be called a decoder system, and a system containing such a subsystem (e.g., a system that generates X output signals in response to a number of inputs, where the subsystem generates M of the inputs and the other XM inputs are received from an external source) may also be called a decoder system. The term "processor" is used broadly to mean a system or device that is programmable or otherwise configurable (e.g., using software or firmware) to perform an operation on data (e.g., audio or video or other image data). Examples of processors include field-programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors programmed and / or otherwise configured to perform pipelining processing on audio or other sound data, programmable general-purpose processors or computers, and programmable microprocessor chips or chipsets.

[0019] The terms “audio processor” and “audio processing unit” are used interchangeably and in a broad sense to describe a system configured to process audio data. Examples of audio processing units include, but are not limited to, encoders (e.g., transcoders), decoders, codecs, pre-processing systems, post-processing systems, and bitstream processing systems (sometimes referred to as bitstream processing tools). The expression “processing state metadata” (for example, in the expression “loudness processing state metadata”) refers to data separate from the corresponding audio data (the audio content of the audio data stream, including the processing state metadata). The processing state metadata is associated with audio data and indicates the loudness processing state of the corresponding audio data (e.g., what type(s) of processing has already been performed on that audio data), and optionally also indicates at least one feature or characteristic of that audio data. In some embodiments, the association of the processing state metadata with the audio data is time-synchronous. Thus, the current (most recently received or updated) processing status metadata indicates that the corresponding audio data simultaneously contains the result of audio data processing of the indicated type (one or more). In some cases, the processing status metadata may include some or all of the parameters used in and / or derived from processing of the indicated type in the processing history and / or processing of the indicated type. Furthermore, the processing status metadata may include at least one feature or characteristic of the corresponding audio data that has been calculated or extracted from the audio data. The processing status metadata may also include other metadata that is not related to any processing of the corresponding audio data and is not derived from any processing of the corresponding audio data. For example, third-party data, tracking information, identifiers, ownership or standard information, user annotation data, user preference data, etc., may be added by a particular audio processing unit and passed to other audio processing units.

[0020] The term "loudness processing state metadata" (or "LPSM") refers to processing state metadata that indicates the loudness processing state of the corresponding audio data (e.g., what type(s) of loudness processing has already been performed on that audio data), and optionally also indicates at least one feature or characteristic (e.g., loudness) of the corresponding audio data. Loudness processing state metadata may include data that is not loudness processing state metadata (e.g., other metadata) when considered in isolation. The terms "combined" or "combined" are used to mean a direct or indirect connection.

[0021] A system and method for an audio encoder / decoder that non-destructively normalizes the loudness and dynamic range of audio across various devices with different dynamic range capabilities, requesting or using various target loudness values, is described. Based on several embodiments, the method and functional components send information about the audio content from the encoder to the decoder for one or more device profiles. The device profile specifies the desired target loudness and dynamic range for one or more devices. The system is extensible to support new device profiles with different "nominal" loudness targets.

[0022] In one embodiment, the system generates appropriate gain in the decoder, under control from the encoder through parameterization of the original gain, in order to reduce the data rate and generate appropriate gain based on loudness control and dynamic range requirements in the encoder. The dynamic range system includes two mechanisms for implementing loudness control: an artistic dynamic range profile that provides content creators with control over how audio is reproduced, and a separate protection mechanism that ensures that overload does not occur for various playback profiles. The system is also configured to allow other metadata (internal or external) parameters to be used to properly control loudness and dynamic range gain and / or profile. The decoder is configured to support n-channel auxiliary inputs that utilize decoder-side loudness and dynamic range settings / processing.

[0023] In some embodiments, loudness processing state metadata (LPSM) is embedded in one or more reserved fields (or slots) of the metadata segment of the audio bitstream. The audio bitstream also includes audio data in other segments (audio data segments). For example, at least one segment of each frame of the bitstream contains an LPSM, and at least one other segment of the frame contains the corresponding audio data (i.e., the audio data whose loudness processing state and loudness are indicated by the LPSM). In some embodiments, the data volume of the LPSM may be small enough to be carried without affecting the bitrate allocated to carry the audio data.

[0024] Communicating loudness processing status metadata in an audio data processing chain is particularly useful when two or more audio processing units need to function sequentially with each other throughout the processing chain (or content lifecycle). Without including loudness processing status metadata in the audio bitstream, media processing problems such as degradation of quality, level, and spatial characteristics can occur when two or more audio codecs are used in the chain and single-ended volume leveling is applied two or more times during the bitstream's journey to the media consumer (or the rendering point of the audio content in the bitstream).

[0025] <Loudness and Dynamic Range Metadata Processing System> Figure 1 is a block diagram of an embodiment of an audio processing system which may be configured to perform loudness and dynamic range optimization under several embodiments that use certain media processing (e.g., pre-processing and post-processing) components. Figure 1 shows an exemplary audio processing chain (audio data processing system) in which one or more elements of the system may be configured based on certain embodiments of the present invention. System 10 in Figure 1 includes the following elements coupled together as shown: a pre-processing unit 12, an encoder 14, a signal analysis and metadata correction unit 16, a transcoder 18, a decoder 20, and a post-processing unit 24. Variations of the illustrated system may omit one or more elements or include additional audio data processing units. For example, in one embodiment, the post-processing unit 22 is part of the decoder 20 rather than a separate unit.

[0026] In some implementations, the preprocessing unit in Figure 1 is configured to accept a PCM (time-domain) sample containing audio content as input 11 and to output a processed PCM sample. The encoder 14 may be configured to accept a PCM sample as input and to output an encoded (e.g., compressed) audio bitstream representing the audio content. The data in the bitstream representing the audio content is sometimes referred to in this paper as "audio data". In some embodiments, the audio bitstream output from the encoder includes loudness processing status metadata (and optionally other metadata) in addition to the audio data.

[0027] The signal analysis and metadata correction unit 16 may accept one or more encoded audio bitstreams as input and perform signal analysis to determine (e.g., validate) whether the processing state metadata in each encoded audio bitstream is correct. In some embodiments, validate may be performed by a state validater component such as element 102 shown in Figure 2, one such validater technique will be described later in the context of state validater 102. In some embodiments, unit 16 is included in the encoder, and validate is performed by either unit 16 or validater 102. If the signal analysis and metadata correction unit finds that the metadata it contains is invalid, the metadata correction unit 16 performs signal analysis to determine the correct value(s) and replaces the incorrect value(s) with the determined correct value(s). Thus, each encoded audio bitstream output from the signal analysis and metadata correction unit may contain the validated processing state metadata in addition to the encoded audio data. The signal analysis and metadata correction unit 16 may be part of the pre-processing unit 12, encoder 14, transcoder 18, decoder 20, or post-processing unit 22. Alternatively, the signal analysis and metadata correction unit 16 may be a separate unit or part of another unit in the audio processing chain.

[0028] The transcoder 18 may accept an encoded audio bitstream as input and output a modified (e.g., encoded in a different way) audio bitstream as response (e.g., by decoding the input stream and re-encoding the decoded stream in a different encoding format). The audio bitstream output from the transcoder includes loudness processing status metadata (and optionally other metadata) in addition to the encoded audio data. The metadata may be included in the bitstream.

[0029] The decoder 20 in Figure 1 may accept an encoded (e.g., compressed) audio bitstream as input and output a decoded stream of PCM audio samples (in response). In one embodiment, the decoder's output may be any of the following, or include: a stream of audio samples and a corresponding stream of loudness processing state metadata (and optionally other metadata) extracted from the input encoded bitstream; a corresponding stream of control bits determined from the stream of audio samples and the loudness processing state metadata (and optionally other metadata) extracted from the input encoded bitstream; or a stream of audio samples without processing state metadata or a corresponding stream of control bits determined from the processing state metadata. In this last case, the decoder may extract the loudness processing state metadata (and / or other metadata) from the input encoded bitstream and perform at least one action (e.g., validity check) on the extracted metadata without outputting the extracted metadata or the control bits determined therefrom.

[0030] By configuring the post-processing unit in Figure 1 according to one embodiment of the present invention, the post-processing unit 22 is configured to receive a stream of decoded PCM audio samples and to perform post-processing thereon (e.g., volume leveling of the audio content) using loudness processing state metadata (and optionally other metadata) or control bits (determined by the decoder from the loudness processing state metadata and optionally other metadata) received with the samples. The post-processing unit 22 is also optionally configured to render the post-processed audio content for playback by one or more speakers. These speakers can be embodied in any of a variety of different listening devices or playback equipment items such as computers, televisions, stereo systems (home or cinema), mobile phones, and other portable playback devices. The speakers may be of any appropriate size and power rating and may be provided in the form of a standalone driver, speaker enclosure, surround sound system, soundbar, headphones, earbuds, etc.

[0031] Some embodiments provide an improved audio processing chain in which audio processing units (e.g., encoders, decoders, transcoders, and pre- and post-processing units) adapt each processing applied to audio data according to the simultaneous state of media data indicated by loudness processing state metadata received by each audio processing unit. The audio data input 11 to any audio processing unit of System 100 (e.g., the encoder or transcoder in Figure 1) may include loudness processing state metadata (and optionally other metadata) in addition to the audio data (e.g., encoded audio data). The metadata may, according to some embodiments, be included in the input audio by another element or another source. A processing unit receiving input audio (with metadata) may perform at least one operation on (e.g., validity check) or in response to (e.g., adaptive processing of the input audio) and may be configured to include the metadata, a processed version of the metadata, or control bits determined from the metadata in its output audio.

[0032] In some embodiments of an audio processing unit (or audio processor), the system is configured to perform adaptive processing on audio data based on the state of the audio data indicated by loudness processing state metadata corresponding to the audio data. In some embodiments, the adaptive processing is (or includes) loudness processing (if the metadata indicates that loudness processing or similar processing has not already been performed on the audio data), but is not (or does not include) loudness processing (if the metadata indicates that such loudness processing or similar processing has already been performed on the audio data). In some embodiments, the adaptive processing is (or includes) metadata validity verification (for example, performed in a metadata validity verification subunit) to ensure that the audio processing unit performs other adaptive processing on the audio data based on the state of the audio data indicated by the loudness processing state metadata. In some embodiments, the validity verification determines the reliability of the loudness processing state metadata associated with the audio data (for example, included in the bitstream with the audio data). For example, if metadata is verified as trustworthy, the results from an already performed audio processing of a certain type may be reused, and additional performance of the same type of audio processing may be avoided. On the other hand, if metadata is found to be tampered with (or otherwise untrustworthy), media processing of a type that is claimed to have been previously performed (as indicated by the untrustworthy metadata) may be repeated by the audio processing unit, and / or other processing may be performed on the metadata and / or audio data by the audio processing unit.An audio processing unit may be configured to signal to other downstream audio processing units in the improved media processing chain that loudness processing state metadata (e.g., present in the media bitstream) is valid, if the unit determines that the processing state metadata is valid (e.g., based on a match between the extracted cryptographic value and the reference cryptographic value).

[0033] In the embodiment shown in Figure 1, the pre-processing component 12 may be part of the encoder 14, and the post-processing component 22 may be part of the decoder 22. Alternatively, the pre-processing component 12 may be implemented in a functional component separate from the encoder 14. Similarly, the post-processing component 22 may be implemented in a functional component separate from the decoder 20.

[0034] Figure 2 is a block diagram of an encoder 100 that may be used in conjunction with system 10 of Figure 1. Any component or element of the encoder 100 may be implemented as one or more processes and / or one or more circuits (e.g., ASIC, FPGA or other integrated circuit) in hardware, software, or a combination of hardware and software. The encoder 100 has a frame buffer 110, a parser 111, a decoder 101, an audio state enable verifier 102, a loudness processing stage 103, an audio stream selection stage 104, an encoder 105, a stuffer / formatter stage 107, a metadata generation stage 106, a dialogue loudness measurement subsystem 108, and a frame buffer 109, connected as shown in the figure. Optionally, the encoder 100 may also include other processing elements (not shown). The encoder 100 (which is a transcoder) is configured to convert an input audio bitstream (which may be, for example, one of AC-3 bitstream, E-AC-3 bitstream, or Dolby E bitstream) into an encoded output audio bitstream (which may be, for example, another of AC-3 bitstream, E-AC-3 bitstream, or Dolby E bitstream). This includes performing adaptive and automated loudness processing using loudness processing state metadata contained in the input bitstream. For example, the encoder 100 may be configured to convert an input Dolby E bitstream (a format typically used in production and broadcast facilities, but not typically used in consumer devices receiving broadcasted audio programs) into an encoded output audio bitstream in the form of AC-3 or E-AC-3 format (suitable for broadcasting to consumer devices).

[0035] The system in Figure 2 also includes an encoded audio delivery subsystem 150 (which stores and / or delivers the encoded bitstream output from encoder 100) and a decoder 152. The encoded audio bitstream output from encoder 100 may be stored by subsystem 150 (for example, in the form of a DVD or Blu-ray disc), or transmitted by subsystem 150 (which may implement a transmission link or network), or both stored and transmitted by subsystem 150. Decoder 152 is configured to decode the encoded audio bitstream (generated by encoder 100) received via subsystem 150. This includes extracting loudness processing state metadata (LPSM) from each frame of the bitstream and generating decoded audio data. In one embodiment, the decoder 152 is configured to perform adaptive loudness processing on the decoded audio data using an LPSM and / or transfer the decoded audio data and the LPSM to a post-processor configured to perform adaptive loudness processing on the decoded audio data using an LPSM. Optionally, the decoder 152 includes a buffer for storing (for example, non-temporarily) the encoded audio bitstream received from subsystem 150.

[0036] Various implementations of the encoder 100 and decoder 152 are configured to perform the various embodiments described in this paper. The frame buffer 110 is a buffer memory coupled to receive an encoded input audio bitstream. In operation, the buffer 110 stores (for example, non-temporarily) at least one frame of the encoded audio bitstream, and a sequence of frames of the encoded audio bitstream is presented from the buffer 110 to the parser 111. The parser 111 is coupled and configured to extract loudness processing metadata (LPSM) and other metadata from each frame of the encoded input audio, present at least the LPSM to the audio state enable verifier 102, loudness processing stages 103, 106 and subsystem 108, extract audio data from the encoded input audio, and present the audio data to the decoder 101. The decoder 101 of the encoder 100 is configured to decode audio data to generate decoded audio data, and to present the decoded audio data to the loudness processing stage 103, the audio stream selection stage 104, the subsystem 108, and optionally the status validity checker 102.

[0037] The state validity verifier 102 is configured to authenticate and validate the LPSM (and optionally other metadata) presented to it. In some embodiments, the LPSM is (or is contained within) a data block included in the input bitstream (for example, according to one embodiment of the present invention). The block may include a cryptographic hash (hash-based message authentication code or "HMAC") for processing the LPSM (and optionally other metadata) and / or the underlying audio data (provided from the decoder 101 to the validity verifier 102). In these embodiments, the data block may be digitally signed. This allows downstream audio processing units to authenticate and validate the processing state metadata relatively easily.

[0038] For example, HMAC may be used to generate a digest, and the protection values ​​(one or more) contained in the bitstream of the present invention may include the digest. The digest may be generated for an AC-3 frame as follows: (1) After the AC-3 data and LPSM are encoded, the frame data bytes (concatenated frame data #1 and frame data #2) and the LPSM data bytes are used as input for the hash function HMAC. Any other data that may be present in the auxiliary data field is not taken into consideration for calculating this digest. Such other data may be bytes that do not belong to either the AC-3 data or the LPSM data. Protection bits contained in the LPSM may not be taken into consideration for calculating the HMAC digest. (2) After the digest is calculated, it is written to the bitstream in the field reserved for the protection bits. (3) The final stage in generating a complete AC-3 frame is the calculation of the CRC check, which is written at the very end of the frame and takes into consideration all data belonging to this frame, including the LPSM bits.

[0039] Other cryptographic methods, including but not limited to one or more non-HMAC cryptographic methods, may be used for validating the LPSM (e.g., in validator 102) to ensure the secure transmission and reception of the LPSM and / or the underlying audio data. For example, validating (using such cryptographic methods) may be performed in each audio processing unit receiving an embodiment of the audio bitstream to determine whether the loudness processing status metadata and corresponding audio data contained in the bitstream have undergone (and / or result from) a particular loudness processing (as indicated by the metadata) and have not been modified after the execution of such particular loudness processing.

[0040] The state validity checker 102 presents control data to the audio stream selection stage 104, the metadata generator 106, and the dialogue loudness measurement subsystem 108 to indicate the result of the validity check operation. In response to the control data, stage 104 may select (and communicate to encoder 105) either (1) the adaptively processed output of the loudness processing stage 103 (for example, when the LPSM indicates that the audio data output from decoder 101 has not undergone a particular type of loudness processing, and the control bit from the validity checker 102 indicates that the LPSM is valid); or (2) the audio data output from decoder 101 (for example, when the LPSM indicates that the audio data output from decoder 101 has already undergone a particular type of loudness processing that would be performed by stage 103, and the control bit from the validity checker 102 indicates that the LPSM is valid).

[0041] Stage 103 of the encoder 100 is configured to perform adaptive loudness processing on the decoded audio data output from the decoder 101, based on one or more audio data characteristics indicated by the LPSM extracted by the decoder 101. Stage 103 may also be a real-time loudness and dynamic range control processor for the adaptive conversion domain. Stage 103 may receive user input (e.g., user target loudness / dynamic range value or dialnorm value) or other metadata input (e.g., one or more types of third-party data, tracking information, identifiers, ownership or standard information, user annotation data, user preference data, etc.) and / or other information (e.g., from a fingerprinting process) and use such input to process the decoded audio data output from the decoder 101.

[0042] The dialogue loudness measurement subsystem 108 may, if the control bits from the enable certifier 102 indicate that the LPSM is disabled, operate to determine the loudness of the segments of the decoded audio (from decoder 101) that represent the dialogue (or other utterances), for example, using the LPSM (and / or other metadata) extracted by decoder 101. If the control bits from the enable certifier 102 indicate that the LPSM is enabled, the operation of the dialogue loudness measurement subsystem 108 may be disabled when the LPSM indicates the previously determined loudness of the dialogue (or other utterances) segments of the decoded audio (from decoder 101).

[0043] Useful tools exist for conveniently and easily measuring the level of dialogue in audio content (e.g., the Dolby LM100 loudness meter). Some embodiments of the APU (e.g., stage 108 of encoder 100) are implemented to include (or perform the function of) such a tool for measuring the average dialogue loudness of the audio content of an audio bitstream (e.g., the decoded AC-3 bitstream presented to stage 108 from decoder 101 of encoder 100). If stage 108 is implemented to measure the true average dialogue loudness of the audio data, the measurement may include a step of isolating segments of the audio content that primarily contain utterances. The audio segments that primarily contain utterances are then processed according to a loudness measurement algorithm. For audio data decoded from an AC-3 bitstream, this algorithm may be a standard K-weighted loudness measure (according to the international standard ITU-R BS.1770). Alternatively, other loudness metrics (for example, those based on psychoacoustic models of loudness) may be used.

[0044] While isolating speech segments is not essential for measuring the average dialogue loudness of audio data, it improves the accuracy of the metric and provides more satisfactory results from the listener's perspective. Since not all audio content contains dialogue (utterances), the loudness metric for the entire audio content can provide a sufficient approximation of the dialogue level of that audio if utterances were present.

[0045] The metadata generator 106 generates metadata to be included by stage 107 in the encoded bitstream output from encoder 100. The metadata generator 106 may pass to stage 107 the LPSM (and / or other metadata) extracted by encoder 101 (for example, if the control bits from validity checker 102 indicate that the LPSM and / or other metadata are valid), or it may generate a new LPSM (and / or other metadata) and present the new metadata to stage 107 (for example, if the control bits from validity checker 102 indicate that the LPSM and / or other metadata extracted by decoder 101 are invalid). Alternatively, it may present stage 107 with a combination of the metadata extracted by decoder 101 and the newly generated metadata. The metadata generator 106 may include in the LPSM presented to stage 107 the loudness data generated by subsystem 108 and at least one value indicating the type of loudness processing performed by subsystem 108, in order to include them in the encoded bitstream output from encoder 100. The metadata generator 106 may generate useful protection bits (which may consist of, or include, a hash-based message authentication code or "HMAC") for decoding, authenticating, or validating at least one of the LPSM (and optionally other metadata) and / or the underlying audio data to be included in the encoded bitstream. The metadata generator 106 may provide such protection bits to stage 107 for inclusion in the encoded bitstream.

[0046] In one embodiment, the dialogue loudness measurement subsystem 108 processes the audio data output from the decoder 101 and, in response, generates loudness values ​​(e.g., gated and ungated dialogue loudness values) and dynamic range values. In response to these values, the metadata generator 106 may generate loudness processing state metadata (LPSM) to be included (by the packer / formatter 107) in the encoded bitstream output from the encoder 100. In one embodiment, the loudness may be calculated based on techniques specified by the ITU-R BS.1770-1 and ITU-R BS.1770-2 standards or other similar loudness measurement standards. Gated loudness can be dialogue-gated loudness, relative-gated loudness, or a combination of these gated loudness types, and the system may use appropriate gating blocks depending on the application requirements and system constraints.

[0047] Additionally, optionally, or alternatively, subsystems 106 and / or 108 of encoder 100 may perform additional analysis of the audio data to generate metadata indicating at least one characteristic of the audio data for inclusion in the encoded bitstream output from stage 107. Encoder 105 encodes the audio data output from selection stage 104 (for example, by performing compression thereon) and presents the encoded audio to stage 107 for inclusion in the encoded bitstream output from stage 107.

[0048] Stage 107 multiplexes the encoded audio from encoder 105 and metadata (including LPSM) from generator 106 to produce an encoded bitstream that is output from stage 107. The encoded bitstream is then given a format specified by one embodiment. Frame buffer 109 is a buffer memory that stores (for example, non-temporarily) at least one frame of the encoded audio bitstream output from stage 107. The sequence of those frames of the encoded audio bitstream is then presented from buffer 109 to the delivery system 150 as output from encoder 100.

[0049] The LPSM generated by the metadata generator 106 and included in the encoded bitstream by stage 107 indicates the loudness processing status of the corresponding audio data (e.g., what type(s) of loudness processing was performed on the audio data) and the loudness of the corresponding audio data (e.g., measured dialogue loudness, gated and / or ungated loudness, and / or dynamic range). Here, “gating” of loudness and / or level measurements performed on audio data refers to a certain level or loudness threshold such that calculated values(s) above the threshold are included in the final measurement (e.g., short-term loudness values ​​below -60 dBFS are ignored in the final measured value). Gating for absolute values ​​refers to a fixed level or loudness, while gating for relative values ​​refers to a value that depends on the current “ungated” measurement.

[0050] In some implementations of encoder 100, the encoded bitstream buffered in memory 109 (and output to delivery system 150) is an AC-3 bitstream or E-AC-3 bitstream, and includes audio data segments (e.g., AB0-AB5 segments of the frame shown in Figure 4) and metadata segments. Here, the audio data segments represent audio data, and each of at least some segments of the metadata segments contains loudness processing state metadata (LPSM). Stage 107 inserts the LPSM into the bitstream in the following format: Each metadata segment containing the LPSM is included in the "addbsi" field of the bitstream information ("BSI") segment of the frame of the bitstream or in an auxiliary data field at the end of the frame of the bitstream (e.g., the AUX segment shown in Figure 4).

[0051] Each bitstream frame may contain one or two metadata segments, each containing an LPSM. If a frame contains two metadata segments, one may be in the frame's addbsi field and the other in the frame's AUX field. Each metadata segment containing an LPSM includes an LPSM payload (or container) segment having the following format: a header (for example, including a synchronization word that identifies the beginning of the LPSM payload, followed by at least one identifier value, for example, the LPSM format version, length, period, count, and substream association values ​​shown in Table 2 below); following the header, at least one dialog indicator value indicating whether the corresponding audio data is dialog-gated or not (for example, which channels of the corresponding audio data are dialog-gated) (for example, the parameter "Dialogue Channel" in Table 2); at least one loudness regulation compliance value indicating whether the corresponding audio data conforms to the indicated set of loudness regulations (for example, the parameter "Loudness Regulation Type" in Table 2); at least one loudness processing value indicating at least one type of loudness processing performed on the corresponding audio data (for example, one or more of the parameters "Dialogue-Gated Loudness Correction Flag" and "Loudness Correction Type" in Table 2); and At least one loudness value that represents at least one loudness characteristic of the corresponding audio data (e.g., peak or average loudness) (e.g., one or more of the parameters "ITU relative gated loudness", "ITU speech gated loudness", "ITU (EBU3341) short-time 3s loudness", and "true peak").

[0052] In some implementations, each metadata segment inserted by step 107 into the "addbsi" field or auxiliary data field of the bitstream frame has the following format: core header (e.g., a synchronization word identifying the start of the metadata segment, followed by identification values, e.g., core element version, length and period, extended element count and substream association values ​​shown in Table 1 below); and after the core header, at least one protection value useful for at least one of decoding, authentication or validation of loudness processing state metadata or corresponding audio data (e.g., HMAC digest and audio fingerprint values ​​in Table 1); and also after the core header, if the metadata segment contains an LPSM, LPSM payload identification information ("ID") and LPSM payload size values ​​that identify the subsequent metadata as an LPSM payload and indicate the size of the LPSM payload.

[0053] An LPSM payload (or container) segment (for example, one with the format shown above) is followed by the LPSM payload ID and LPSM payload size values.

[0054] In some embodiments, each metadata segment in the frame's auxiliary data field (or "addbsi" field) has three levels of structure: High-level structure. This includes a flag indicating whether the auxiliary data (or addbsi) field contains metadata, at least one ID value indicating what type(s) of metadata is present, and optionally also a value indicating how many bits of metadata (e.g., for each type) are present (if metadata exists). One type of metadata that may exist is LPSM, and another type of metadata that may exist is media research metadata (e.g., Nielsen Media Research metadata); Middle-level structure. This includes core elements for each identified type of metadata (e.g., the core header, protection value, and LPSM payload ID and LPSM payload size values ​​for each identified type of metadata as described above); Low-level structure. This includes each payload for a given core element (for example, the LPSM payload if the core element identifies that an LPSM payload exists, and / or the metadata payload of another type if the core element identifies that another type of metadata payload exists).

[0055] Data values ​​in such a three-level structure can be nested. For example, protection values ​​(one or more) for the LPSM payload and / or another metadata payload identified by the core element can be included after each payload identified by the core element (and thus after the core header of the core element). In one example, the core header can identify the LPSM payload and another metadata payload, the payload ID and payload size values ​​for the first payload (e.g., the LPSM payload) can follow the core header, the first payload itself can follow the ID and size values, the payload ID and payload size values ​​for the second payload can follow the first payload, the second payload itself can follow these ID and size values, and protection values ​​for both payloads (or the core element value and for both payloads) can follow the last payload.

[0056] In some embodiments, when decoder 101 receives an audio bitstream generated according to a certain embodiment of the present invention having a cryptographic hash, the decoder is configured to parse and extract the cryptographic hash from a data block determined from the bitstream. The block includes loudness processing state metadata (LPSM). A validity verifier 102 may use the cryptographic hash to validate the received bitstream and / or associated metadata. For example, if validity verifier 102 finds the LPSM to be valid based on a match between a reference cryptographic hash and the cryptographic hash extracted from the data block, validity verifier 102 may disable the operation of processor 103 on the corresponding audio data and allow the audio data to pass through selection stage 104 (unmodified). Additionally, optionally, or alternatively, other types of cryptographic techniques may be used instead of the cryptographic hash-based method.

[0057] The encoder 100 in Figure 2 may determine (in response to the LPSM extracted by the decoder 101) that a post / preprocessing unit has performed a certain type of loudness processing on the audio data to be encoded (in elements 105, 106, and 107), and thus may generate loudness processing state metadata (in generator 106) that includes specific parameters used in and / or derived from the previously performed loudness processing. In some implementations, the encoder 100 may generate (and include in the encoded bitstream output therefrom) processing state metadata indicating the processing history of the audio content, as long as the encoder is aware of the type of processing performed on the audio content.

[0058] Figure 3 is a block diagram of a decoder that may be used in conjunction with system 10 of Figure 1. Any component or element of the decoder 200 and post-processor 300 may be implemented as one or more processes and / or one or more circuits (e.g., ASIC, FPGA or other integrated circuit) in hardware, software, or a combination of hardware and software. The decoder 200 has a frame buffer 201, a parser 205, an audio decoder 202, an audio state validity check stage (validity checker) 203, and a control bit generation stage 204, connected as shown in the figure. The decoder 200 may include other processing elements (not shown). The frame buffer 201 (buffer memory) stores (e.g., non-temporarily) at least one frame of the encoded audio bitstream received by the decoder 200. A sequence of frames of the encoded audio bitstream is presented from the buffer 201 to the parser 205. The parser 205 is configured to extract loudness processing metadata (LPSM) and other metadata from each frame of the encoded input audio, present at least the LPSM to the audio state validator 203 and stage 204, present the LPSM as an output (for example to the post-processor 300), extract audio data from the encoded input audio, and present the extracted audio data to the decoder 202. The encoded audio bitstream input to the decoder 200 may be one of AC-3 bitstream, E-AC-3 bitstream, or Dolby E bitstream.

[0059] The system in Figure 3 also includes a post-processor 300. The post-processor 300 has a frame buffer 301 and other processing elements (not shown) which include at least one processing element coupled to the buffer 301. The frame buffer 301 stores (e.g., non-temporarily) at least one frame of the decoded audio bitstream received by the post-processor 300 from the decoder 200. The processing elements of the post-processor 300 are coupled and configured to receive a sequence of frames of the decoded audio bitstream output from the buffer 301 and to process it adaptively using metadata (including LPSM values) output from the decoder 202 and / or control bits output from stage 204 of the decoder 200. In one embodiment, the post-processor 300 is configured to perform adaptive loudness processing on the decoded audio data using LPSM values ​​(e.g., based on loudness processing status and / or one or more audio data characteristics indicated by the LPSM). Various implementations of the decoder 200 and post-processor 300 are configured to perform various embodiments of the method based on the embodiments described in this paper.

[0060] The audio decoder 202 of decoder 200 is configured to decode the audio data extracted by parser 205 to produce decoded audio data and present the decoded audio data as output (for example, to post-processor 300). The state validity verifier 203 is configured to authenticate and validate the LPSM (and optionally other metadata) presented thereto. In some embodiments, the LPSM is (or is contained within) a data block included in the input bitstream (for example, according to one embodiment of the present invention). The block may include a cryptographic hash (hash-based message authentication code or "HMAC") for processing the LPSM (and optionally other metadata) and / or the underlying audio data (provided from parser 205 and / or decoder 202 to validity verifier 203). In these embodiments, the data block may be digitally signed. This allows downstream audio processing units to authenticate and validate the processing state metadata relatively easily.

[0061] Other cryptographic methods, including but not limited to one or more non-HMAC cryptographic methods, may be used for validating the LPSM (e.g., in validating verifier 203) to ensure the secure transmission and reception of the LPSM and / or the underlying audio data. For example, validating (using such cryptographic methods) may be performed in each audio processing unit receiving an embodiment of the audio bitstream of the present invention to determine whether the loudness processing status metadata and corresponding audio data contained in the bitstream have undergone (and / or result from) a particular loudness processing (as indicated by the metadata) and have not been modified after the execution of such particular loudness processing.

[0062] The state validity checker 203 presents control data that controls the bit generator 204 and / or presents said control data as an output (for example, to the post-processor 300) to indicate the result of the validity check operation. In response to said control data (and optionally other metadata extracted from the input bitstream), stage 204 may generate (and present to the post-processor 300) any of the following: (For example, when the LPSM indicates that the audio data output from the decoder 202 has undergone a particular type of loudness processing, and the control bits from the validity checker 203 indicate that the LPSM is valid) A control bit indicating that the decoded audio data output from the decoder 202 has undergone the particular type of loudness processing; or (for example, when the LPSM indicates that the audio data output from the decoder 202 has not undergone a particular type of loudness processing, or when the LPSM indicates that the audio data output from the decoder 202 has undergone a particular type of loudness processing, but the control bits from the validity checker 203 indicate that the LPSM is not valid) A control bit indicating that the decoded audio data output from the decoder 202 should undergo the particular type of loudness processing.

[0063] Alternatively, decoder 200 may present metadata extracted from the input bitstream by decoder 202 (and any other arbitrary metadata) to post-processor 300, which may use LPSM to perform loudness processing on the decoded audio data, perform an LPSM validity check, and if the validity check indicates that LPSM is valid, then use LPSM to perform loudness processing on the decoded audio data.

[0064] In some embodiments, when decoder 201 receives an audio bitstream generated according to a certain embodiment of the present invention having a cryptographic hash, the decoder is configured to parse and extract the cryptographic hash from a data block determined from the bitstream. The block includes loudness processing state metadata (LPSM). Validity verifier 203 may use the cryptographic hash to validate the received bitstream and / or associated metadata. For example, if validity verifier 203 finds the LPSM valid based on a match between a reference cryptographic hash and the cryptographic hash extracted from the data block, validity verifier 203 may signal a downstream audio processing unit (e.g., a volume leveling unit or a post-processor 300 which may include a volume leveling unit) to pass the audio data of the bitstream through (unmodified). Additionally, optionally, or alternatively, other types of cryptographic techniques may be used instead of the cryptographic hash-based method.

[0065] In some implementations of decoder 100, the encoded bitstream received (and buffered in memory 201) is an AC-3 bitstream or an E-AC-3 bitstream, which includes audio data segments (e.g., AB0-AB5 segments of the frame shown in Figure 4) and metadata segments. Here, the audio data segments represent audio data, and at least some of each segment of the metadata segments contains loudness processing state metadata (LPSM). Decoder stage 202 is configured to extract LPSM from the bitstream in the following format: Each metadata segment containing an LPSM is contained in the "addbsi" field of the bitstream information ("BSI") segment of the bitstream frame, or in the auxiliary data field at the end of the bitstream frame (e.g., the AUX segment shown in Figure 4). Each frame of the bitstream may contain one or two metadata segments containing an LPSM, and if a frame contains two metadata segments, one may be in the addbsi field of the frame and the other in the AUX field of the frame.Each metadata segment containing an LPSM includes an LPSM payload (or container) segment having the following format: a header (for example, including a synchronization word that identifies the beginning of the LPSM payload, followed by identification values, for example, the LPSM format version, length, period, count, and substream association values ​​shown in Table 2 below); following the header, at least one dialog indication value (for example, the parameter "Dialogue Channel" in Table 2) indicating whether the corresponding audio data exhibits dialogue or not (for example, which channels of the corresponding audio data exhibit dialogue); at least one loudness regulation compliance value (for example, the parameter "Loudness Regulation Type" in Table 2) indicating whether the corresponding audio data conforms to the indicated set of loudness regulations; at least one loudness processing value (for example, one or more of the parameters "Dialogue-Gated Loudness Correction Flag" and "Loudness Correction Type" in Table 2) indicating at least one type of loudness processing performed on the corresponding audio data; and At least one loudness value that represents at least one loudness characteristic of the corresponding audio data (e.g., peak or average loudness) (e.g., one or more of the parameters "ITU Relative Gated Loudness," "ITU Speech Gated Loudness," "ITU (EBU3341) Short-Time 3s Loudness," and "True Peak" in Table 2).

[0066] In some implementations, the decoder stage 202 is configured to extract metadata segments from the "addbsi" field or auxiliary data field of the bitstream frame, each having the following format: a header (e.g., a synchronization word identifying the beginning of the metadata segment, followed by at least one identifier value, e.g., the core element version, length and period, extended element count and substream association value shown in Table 1 below); and, after the core header, at least one protection value useful for at least one of decoding, authentication or validation of loudness processing state metadata or corresponding audio data (e.g., the HMAC digest and audio fingerprint values ​​in Table 1); and also after the core header, if the metadata segment contains an LPSM, the LPSM payload identifier ("ID") and LPSM payload size value, which identify the subsequent metadata as an LPSM payload and indicate the size of the LPSM payload. The LPSM payload (or container) segment (for example, having the format specified above) is followed by the LPSM payload ID and LPSM payload size values.

[0067] More generally, an encoded audio bitstream produced by a particular embodiment has a structure that provides a mechanism for labeling metadata elements and sub-elements as core (required) or extensions (optional elements). This allows for scaling the data rate of the bitstream (including metadata) across numerous applications. Core (required) elements of the bitstream syntax should also signal that extensions (optional) elements associated with the audio content are present (in-band) and / or at remote locations (out-of-band).

[0068] In some embodiments, core elements (one or more) are required to be present in all frames of the bitstream. Some sub-elements of the core elements are optional and may be present in any combination. Extension elements are not required to be present in all frames (to limit bitrate overhead). Thus, extension elements may be present in some frames and not in others. Some sub-elements of the extension elements are optional and may be present in any combination, but some sub-elements of the extension elements may be required (i.e., required if the extension element is present in a frame of the bitstream).

[0069] In some embodiments, an encoded audio bitstream is generated (for example, by an audio processing unit embodying the present invention) that includes a sequence of audio data segments and metadata segments. The audio data segments represent audio data, and each of at least some segments of the metadata segments includes loudness processing state metadata (LPSM), and the audio data segments are time-division multiplexed with the metadata segments. In some embodiments of this class, each of the metadata segments has a format described herein. In some formats, the encoded bitstream is an AC-3 bitstream or an E-AC-3 bitstream, and each of the metadata segments containing LPSM is included as additional bitstream information in the “addbsi” field of the bitstream information (“BSI”) segment of the bitstream frame of the bitstream (shown in Figure 6), or in the auxiliary data field of the bitstream frame (for example, by stage 107 of encoder 100). Each frame includes a core element having the format shown in Table 1 of Figure 8 in the addbsi field of the frame.

[0070] In one format, each addbsi (or auxiliary data) field containing an LPSM includes a core header (and optionally additional core elements) and the following LPSM values ​​(parameters) following the core header (or the core header and other core elements): Payload ID (identifies the metadata as an LPSM). This follows the core element value (as specified, for example, in Table 1); Payload Size (indicates the size of the LPSM payload). This follows the Payload ID; LPSM Data (following the Payload ID and Payload Size values). This has the format shown in Table 2 of Figure 9.

[0071] In the second format of the encoded bitstream, the bitstream is an AC-3 bitstream or an E-AC-3 bitstream, and each metadata segment containing an LPSM is included (for example, by stage 107 of encoder 100) in either the "addbsi" field of the bitstream information ("BSI") segment of the bitstream frame (shown in Figure 6) or in the auxiliary data field at the end of the bitstream frame (for example, the AUX field shown in Figure 4). Each frame may contain one or two metadata segments containing an LPSM, and if a frame contains two metadata segments, one may reside in the addbsi field of the frame and the other in the AUX field of the frame. Each metadata segment containing an LPSM has the format defined above with reference to Tables 1 and 2 above (i.e., it includes the core elements specified in Table 1, followed by the payload ID (identifying the metadata as an LPSM) and payload size value defined above, followed by the payload (LPSM data in the format shown in Table 2)).

[0072] In the other format, the encoded bitstream is a Dolby E bitstream, and each metadata segment containing LPSMs is the first N sample positions of the Dolby E protected bandwidth section. A Dolby E bitstream containing such metadata segments containing LPSMs includes, for example, a value indicating the LPSM payload length transmitted in the Pd word of the SMPTE 337M preamble (the SMPTE 337M Pa word repetition rate may remain the same as the associated video frame rate).

[0073] In a format where the encoded bitstream is an E-AC-3 bitstream, each metadata segment containing an LPSM is included as additional bitstream information (for example, by stage 107 of encoder 100) in the "addbsi" field of the bitstream information ("BSI") segment of the bitstream frame. Further aspects of encoding an E-AC-3 bitstream with an LPSM in this format are described below: (1) While the E-AC-3 encoder (inserting the LPSM value into the bitstream) is "active" during the generation of the E-AC-3 bitstream, the bitstream should include a metadata block (containing the LPSM) carried in the addbsi field of the frame for all frames being generated (synchronous frames). The bits required to carry the metadata block should not increase the encoder bitrate (frame length). (2) All metadata blocks (including LPSMs) should contain the following information: loudness_correction_type_flag: where "1" indicates that the loudness of the corresponding audio data was corrected upstream of the encoder, and "0" indicates that the loudness was corrected by a loudness corrector built into the encoder (for example, the loudness processor 103 of encoder 100 in Figure 2); speech_channel: indicates which source channel(s) contains the utterance (in the preceding 0.5 seconds). If no speech is detected, this will be indicated; speech_loudness: Indicates the combined speech loudness (for the preceding 0.5 seconds) for each corresponding audio channel containing speech; ITU_loudness: Indicates the combined ITU BS.1770-2 loudness for each corresponding audio channel; Gain: The combined loudness gain (single or plural) for inversion in the decoder (to demonstrate reversibility).

[0074] While the E-AC-3 encoder (which inserts LPSM values ​​into the bitstream) is "active" and receiving an AC-3 frame with the "trust" flag, the loudness controller in that encoder (e.g., the loudness processor 103 of encoder 100 in Figure 2) is bypassed. The "trusted" source dialnorm and DRC values ​​should be passed to the E-AC-3 encoder component (e.g., stage 107 of encoder 100) (e.g., by the generator 106 of encoder 100). LPSM block generation continues, and loudness_correction_type_flag is set to "1". The loudness controller bypass sequence should be synchronized to the beginning of the decoded AC-3 frame in which the "trust" flag appears. The loudness controller bypass sequence is implemented as follows: the leveler_amount control is decremented from value 9 to value 0 over 10 audio block durations (i.e., 53.3 msec), and the leveler_back_end_meter control is put into bypass mode (this operation should provide a seamless transition). The term "trusted" bypass for the leveler implies that the dialnorm value of the source bitstream is also reused at the encoder output (for example, if the "trusted" source bitstream has a dialnorm value of -30, the encoder output should utilize -30 for the outgoing dialnorm value).

[0075] While the E-AC-3 encoder (which inserts LPSM values ​​into the bitstream) is "active" and receiving AC-3 frames without the "trust" flag, the loudness controller built into that encoder (e.g., loudness processor 103 of encoder 100 in Figure 2) is active. LPSM block generation continues, and loudness_correction_type_flag is set to "0". The loudness controller activation sequence is synchronized to the beginning of the decoded AC-3 frame where the "trust" flag disappears. The loudness controller activation sequence is implemented as follows: the leveler_amount control is incremented from value 0 to value 9 over one audio block duration (i.e., 5.3 msec), and the leveler_back_end_meter control is set to "active" mode (this operation provides a seamless transition, including a back_end_meter integrated reset). During encoding, the graphical user interface (GUI) displayed the following parameters to the user: "Input audio program [trusted / untrusted]" - the state of this parameter is based on the presence of a "trusted" flag in the input signal; and "Real-time loudness correction: [enable / disabled]" - the state of this parameter is based on whether this loudness controller built into the encoder is active.

[0076] When decoding an AC-3 or E-AC-3 bitstream that has LPSMs (in the format described) contained in the "addbsi" field of the bitstream information ("BSI") segment of each frame of the bitstream, the decoder parses the LPSM block data (in the addbsi field) and passes the extracted LPSM values ​​to the graphical user interface (GUI). The set of extracted LPSM values ​​is refreshed for each frame.

[0077] In yet another format, the encoded bitstream is an AC-3 bitstream or an E-AC-3 bitstream, and each metadata segment containing an LPSM is included as additional bitstream information in the "addbsi" field (shown in Figure 6) of the bitstream information ("BSI") segment of the bitstream frame (or in the Aux segment) (for example, by stage 107 of encoder 100). In this format (a variation of the above format with reference to Tables 1 and 2), each addbsi (or Aux) field containing an LPSM contains the following LPSM values: core elements as defined in Table 1; followed by a payload ID (identifying the metadata as an LPSM) and a payload size value; followed by a payload (LPSM data) in the following format (similar to the elements shown in Table 2 above): LPSM payload version: a 2-bit field indicating the version of the LPSM payload; dialchan: a 3-bit field indicating whether the left, right, and / or center channels of the corresponding audio data contain spoken dialogue. The bit assignment for the dialchan field may be as follows: Bit 0, indicating the presence of a dialogue in the left channel, is stored in the most significant bit of the dialchan field, and Bit 2, indicating the presence of a dialogue in the middle channel, is stored in the least significant bit of the dialchan field. If the corresponding channel contains a dialogue spoken during the preceding 0.5 seconds of the program, each bit of the dialchan field is set to "1"; loudregtyp: A 3-bit field indicating which regulatory standards the program loudness complies with. Setting the "loudregtyp" field to "000" indicates that the LPSM does not comply with loudness regulations.For example, a value of this field (e.g., 000) may indicate that compliance with loudness regulation standards is not indicated, another value of this field (e.g., 001) may indicate that the program's audio data complies with the ATSC A / 85 standard, and another value of this field (e.g., 010) may indicate that the program's audio data complies with the EBU R128 standard. In this example, if this field is set to any value other than "000", the loudcorrdialgat and loudcorrtyp fields should follow the payload; loudcorrdialgat: A 1-bit field indicating whether dialog-gated loudness correction has been applied. If the program's loudness is corrected using dialog gating, the value of the loudcorrdialgat field is set to "1". Otherwise, the value is set to "0"; loudcorrtyp: A 1-bit field indicating the type of loudness correction applied to the program. The value of the loudcorrtyp field is set to "0" if the program's loudness is compensated using an infinite look-ahead (file-based) loudness compensation process. The value of this field is set to "1" if the program's loudness is compensated using a combination of real-time loudness measurement and dynamic range control; loudrelgate: A 1-bit field indicating whether relative-gated loudness data (ITU) exists. If the loudrelgate field is set to "1", the payload should be followed by a 7-bit ituloudrelgat field; loudrelgat: A 7-bit field indicating relative-gated program loudness (ITU). This field shows the integrated loudness of the audio program, measured according to ITU-R BS.1770-2, without any gain adjustments due to dialnorm and dynamic range compression.Values ​​from 0 to 127 are interpreted as -58 LKFS to +5.5 LKFS in increments of 0.5 LKFS; loudspchgate: A 1-bit field indicating whether speech-gated loudness data (ITU) is present. If the loudspchgate field is set to "1", the payload should be followed by a 7-bit loudspchgat field; loudspchgat: A 7-bit field indicating speech-gated program loudness. This field shows the integrated loudness of the corresponding audio program as measured according to formula (2) of ITU-R BS.1770-3, without any gain adjustments due to dialnorm and dynamic range compression. Values ​​from 0 to 127 are interpreted as -58 to +5.5 LKFS in increments of 0.5 LKFS; loudstrm3se: A 1-bit field indicating whether short-time (3-second) loudness data is present. If this field is set to "1", the payload should be followed by a 7-bit loudstrm3s field. loudstrm3s: A 7-bit field indicating the ungated loudness of the preceding 3 seconds of the corresponding audio program, measured according to ITU-R BS.1771-1, without any gain adjustments due to dialnorm and dynamic range compression. Values ​​from 0 to 256 are interpreted as -116 LKFS to +11.5 LKFS in increments of 0.5 LKFS; truepke: A 1-bit field indicating whether true peak loudness data exists. If the truepke field is set to "1", the payload should be followed by an 8-bit truepk field; truepk: An 8-bit field indicating the true peak sample value of the program, measured according to Annex 2 of ITU-R BS.1770-3, without any gain adjustments due to dialnorm and dynamic range compression. Values ​​from 0 to 256 are interpreted in increments of 0.5 LKFS, ranging from -116 LKFS to +11.5 LKFS.

[0078] In some embodiments, the core element of a metadata segment in an auxiliary data field (or "addbsi" field) of a frame of an AC-3 bitstream or E-AC-3 bitstream includes a core header (optionally including an identification value, e.g., core element version), followed by: a value indicating whether fingerprint data (or other protection values) are included for the metadata of the metadata segment; a value indicating whether external data exists (related to the audio data corresponding to the metadata of the metadata segment); payload ID and payload size values ​​for each type of metadata identified by the core element (e.g., LPSM and / or non-LPSM type metadata); and protection values ​​for at least one type of metadata identified by the core element. The metadata payload(s) of the metadata segment follow the core header and are (optionally) nested within the core element's value.

[0079] <Optimized loudness and dynamic range system> The secure metadata encoding and transfer scheme described above is used in conjunction with a scalable and expandable system for optimizing loudness and dynamic range across different playback devices, applications, and listening environments, as shown in Figure 1. In one embodiment, system 10 is configured to normalize the loudness level and dynamic range of input audio 11 across various devices that require different target loudness values ​​and have different dynamic range capabilities. To normalize the loudness level and dynamic range, system 10 includes various device profiles along with the audio content, and the normalization is based on these profiles. These profiles may be included by one of the audio processing units in the audio processing chain, and the included profiles may be used by downstream processing units in the audio processing chain to determine the desired target loudness and dynamic range for the target device. Additional processing components may provide or process information for instrument profile management, gain control, and broadband and / or multiband gain generation functions (including, but not limited to, the following parameters: null bandwidth range, true peak threshold, loudness range, fast / slow time constant (coefficient), and maximum boost).

[0080] Figure 10 shows a more detailed diagram of the system of Figure 1 for a system that provides optimized loudness and dynamic range control under several embodiments. In the system 321 of Figure 10, the encoder stage has a core encoder component 304 that encodes the audio input 303 in a suitable digital format for transmission to the decoder 312. The audio is processed so that it can be reproduced in a variety of different listening environments, each which may require different loudness and / or dynamic range target settings. Thus, as shown in Figure 10, the decoder outputs a digital signal, which is converted to an analog format by a digital-to-analog converter 316 for reproduction through a variety of different driver types, including full-range speakers 320, miniature speakers 322, and headphones 324. These drivers represent only a few examples of possible playback drivers, and any transducer or driver of any appropriate size may be used. Furthermore, the drivers / transducers 320-324 of Figure 10 may be embodied in any suitable playback device for use in any corresponding listening environment. The device type may include, for example, AVRs, televisions, stereo systems, computers, mobile phones, tablet computers, MP3 players, etc., and the listening environment may include, for example, auditoriums, homes, automobiles, listening booths, etc.

[0081] Because playback environments and driver types can range from very small personal contexts to very large public venues, the span of possible and optimal playback loudness and dynamic range configurations can vary greatly depending on the content type, background noise level, etc. For example, in a home theater environment, content with a wide dynamic range can be played back through surround sound equipment, while content with a narrower dynamic range can be played back through a regular television system (flat panel LED / LCD type, etc.). On the other hand, for certain listening conditions where large level fluctuations are undesirable (e.g., at night, or with devices with strict acoustic output power limitations, such as the built-in speakers or headphone outputs of mobile phones / tablets), a very narrow dynamic range mode may be used. In portable or mobile listening contexts, such as using a small computer or dock speaker or headphones / earbuds, the optimal dynamic range for playback can vary depending on the environment. For example, in a quiet environment, the optimal dynamic range may be greater than in a noisy environment. The embodiments of the adaptive audio processing system shown in Figure 10 change the dynamic range to make the audio content more understandable, depending on parameters such as the listening environment and playback device type.

[0082] Figure 11 is a table showing various dynamic range requirements for diverse playback devices and background listening environments in an exemplary use case. Similar requirements can be derived for loudness. These various dynamic range and loudness requirements generate various profiles used by the optimization system 321. System 321 includes a loudness and dynamic range measurement component 302 that analyzes and measures the loudness and dynamic range of the input audio. In one embodiment, the system analyzes the overall program content to determine the overall loudness parameter. In this context, loudness refers to long-term program loudness or the average loudness of the program, where program is a single unit of audio content such as a movie, television program, commercial, or similar program content. Loudness is used to provide an index of the artistic dynamic range profile used by content creators to control how audio is played back. Loudness is related to the dialnorm metadata value in that dialnorm represents the average dialogue loudness of a single program (e.g., movie, program, commercial, etc.). Short-term dynamic range quantifies signal fluctuations over much shorter time periods than program loudness. For example, short-term dynamic range may be measured on the order of seconds, while program loudness may be measured over spans of minutes or even hours. Short-term dynamic range provides an independent protection mechanism from program loudness to ensure that overload does not occur for various playback profiles and device types. In one embodiment, the loudness (long-term program loudness) target is based on dialogue loudness, and the short-term dynamic range is based on relative gated and / or ungated loudness. In this case, certain DRC and loudness components in the system are context-aware with respect to content type and / or target device type and characteristics.As part of this context-aware functionality, the system is configured to analyze one or more characteristics of an output device to determine whether the device belongs to a specific group of devices optimized for certain DRC and loudness reproduction conditions, such as AVR devices, televisions, computers, and portable devices.

[0083] The preprocessing component analyzes the program content to determine loudness, peak, true peak, and quiet periods in order to generate unique metadata for each of several different profiles. In one embodiment, loudness may be dialogue-gated loudness and / or relative-gated loudness. The different profiles define various DRC (Dynamic Range Control) and target loudness modes. In these modes, different gain values ​​are generated in the encoder depending on the source audio content, the desired target loudness, and the characteristics of the playback device and / or environment. The decoder may offer various DRC and target loudness modes (enabled by the profiles mentioned above), which may include: DRC and target loudness off / disabled, allowing full dynamic range listing without compression and loudness normalization of the audio signal. Loudness normalization with a target of -31 LKFS for DRC off / disabled and playback on home theater systems provides moderate dynamic range compression through the gain value generated (specifically for this playback mode and / or device profile) in an encoder with loudness normalization targeting -31 LKFS. RF mode for playback through TV speakers provides heavy dynamic range compression with loudness normalization targeting -24, -23, or -20 LKFS. Intermediate mode for playback on a computer or similar device provides compression with loudness normalization targeting -14 LKFS. Portable mode provides very heavy dynamic range compression with loudness normalization targeting -11 LKFS. The LKFS target loudness values ​​of -31, -23 / -20, -14, and -11 are intended to be examples of different playback / device profiles that may be defined for this system under several embodiments, and any other suitable target loudness values ​​may be used, and the system will generate appropriate gain values ​​in particular for these playback modes and / or device profiles.Furthermore, this system is extensible and adaptable to accommodate various playback devices and listening environments by defining new profiles in the encoder or defining new profiles to be loaded into the encoder elsewhere. In this way, new, unique playback / device profiles can be generated to support improved or different playback devices for future applications.

[0084] In one embodiment, the gain value can be calculated in any suitable processing component of the system 321, for example, in the encoder 304, decoder 312, or transcoder 308, or in any related pre-processing component associated with the encoder or any post-processing component associated with the decoder.

[0085] Figure 13 is a block diagram showing the interface between different profiles for various different playback device classes under one embodiment. As shown in Figure 13, encoder 502 receives an audio input 501 and one of several different profiles 506. The encoder combines the audio data with the selected profile to generate an output bitstream file. The output bitstream file is processed in a decoder component within or associated with the target playback device. In the example of Figure 13, the different playback devices may be a computer 510, a mobile phone 512, an AVR 514, and a television 516, but many other output devices are also possible. Each of the devices 510-516 includes or is coupled to a speaker (including a driver and / or transducer), such as drivers 320-324. The combination of processing, power rating, and size of the playback device and associated speaker generally determines which profile is best suited for its particular target. Thus, profile 506 may be specifically defined for playback through an AVR, TV, mobile speaker, mobile headphones, etc. Profiles may also be defined for specific operating modes or conditions, such as quiet mode, night mode, outdoor, indoor, etc. The profiles shown in Figure 13 are merely illustrative modes, and any appropriate profile may be defined, including custom profiles for specific targets and environments.

[0086] Figure 13 shows an embodiment in which encoder 502 receives profile 506 and generates appropriate parameters for loudness and DRC processing. It should be noted that the parameters generated based on the profile and audio content can be executed on any suitable audio processing unit, such as encoders, decoders, transcoders, preprocessors, and postprocessors. For example, each output device 510-516 in Figure 13 has or is coupled to a decoder component that processes metadata in the bitstream in file 504 sent from encoder 502 to enable loudness and dynamic range adaptation to match the device or device type of the target output device.

[0087] In one embodiment, the dynamic range and loudness of the audio content are optimized for each possible playback device. This is achieved by maintaining a target long-term loudness and controlling the short-term dynamic range to optimize the audio experience for each target playback mode (by controlling signal dynamics, sample peaks, and / or true peaks). Various metadata elements are defined for long-term loudness and short-term dynamic range. As shown in Figure 10, component 302 analyzes the entire input audio signal (or a portion of it, e.g., the speech component, if applicable) to derive significant characteristics for both of these distinct DR components. This allows for different gain values ​​to be defined for artistic gain and clipping (overload protection) gain values.

[0088] These gain values ​​for long-term loudness and short-term dynamic range are then mapped to profile 305 to provide parameters describing the loudness and dynamic range control gain values. These parameters are combined with the encoded audio signal from encoder 304 in multiplexer 306 or a similar component for generating a bitstream transmitted to the decoder stage via transcoder 308. The bitstream input to the decoder stage is multiplexed and separated in demultiplexer 310 and then decoded in decoder 312. Gain component 314 applies gains corresponding to the appropriate profile to generate digital audio data. This digital audio data is then processed through DACS unit 416 for playback through appropriate playback equipment and drivers or transducers 320-324.

[0089] Figure 14 is a table showing the correlation between long-term loudness and short-term dynamic range for several defined profiles under one embodiment. As shown in Table 4 of Figure 14, each profile includes a set of gain values ​​that specify the amount of dynamic range compression (DRC) applied in the system's decoder or each target device. Each of the N profiles, denoted as Profile 1 to N, sets specific long-term loudness parameters (e.g., dialnorm) and overload compression parameters by specifying the corresponding gain values ​​applied in the decoder stage. The DRC gain values ​​for these profiles may be defined by an external source accepted by the encoder, or they may be generated internally within the encoder as default gain values ​​if no external values ​​are provided.

[0090] In one embodiment, the gain value for each profile is embodied in DRC gain terminology, which is calculated based on an analysis of certain characteristics of the audio signal, such as peak, true peak, short-term loudness of dialogue, or overall short-term loudness, or a combination of both (hybrid). A static gain (i.e., transfer characteristics or curve) based on the selected profile and the time constants required to implement the fast / slow attack and fast / slow release of the final DRC gain for each possible device profile and / or target loudness are calculated. As described above, these profiles may be pre-configured in the encoder, decoder, or they may be generated externally and transported from the content creator to the encoder via external metadata.

[0091] In one embodiment, the gain value may be a broadband gain that applies the same gain to all frequencies of the audio content. Alternatively, the gain may consist of multiband gain values, with different gain values ​​applied to different frequencies or frequency bands of the audio content. In the case of multichannel, each profile may constitute a matrix of gain values ​​that show the gain for various frequency bands instead of a single gain value.

[0092] Referring to Figure 10, in one embodiment, information regarding the attributes or characteristics of the listening environment and / or the function and configuration of the playback device is provided by the decoder stage to the encoder stage via the feedback link 330. Profile information 332 is also input to the encoder 304. In one embodiment, the decoder analyzes metadata in the bitstream to determine whether loudness parameters for a first group of audio playback devices are available in the bitstream. If available, the decoder transmits those parameters downstream for use in audio rendering. Otherwise, the encoder derives those parameters by analyzing certain characteristics of the device. These parameters are then sent to downstream rendering components for playback. The encoder also determines the output device (or a group of output devices including said output device) that renders the received audio stream. For example, the output device may be determined to belong to a group such as a mobile phone or a portable device. In one embodiment, the decoder uses the feedback link 330 to indicate the determined output device or group of output devices to the encoder. For this feedback, a module connected to an output device (for example, a module in a sound card connected to a headset or speakers in a laptop) may indicate to the decoder the characteristics of the output device or the group of devices including the output device. The decoder transmits this information to the encoder via the feedback link 330. In one embodiment, the decoder performs a test and determines the loudness and DRC parameters. In another embodiment, instead of transmitting the information via the feedback link 330, the decoder uses the information about the identified device or group of output devices to determine the loudness and DRC parameters. In yet another embodiment, a separate audio processing unit determines the loudness and DRC parameters, and the decoder transmits the information to that audio processing unit on behalf of the decoder.

[0093] Figure 12 is a block diagram of a dynamic range optimization system under one embodiment. As shown in Figure 12, encoder 402 receives input audio 401. The encoded audio is combined in multiplexer 409 with parameters 404 generated from a selected compression curve 422 and dialnorm value 424. The resulting bitstream is transmitted to demultiplexer 411, which generates an audio signal to be decoded by decoder 406. The parameters and dialnorm value are used by gain calculation unit 408 to generate a gain level that drives amplifier 410 for amplification of the decoder output. Figure 12 illustrates how dynamic range control is parameterized and inserted into the bitstream. Loudness can also be parameterized and inserted into the bitstream using similar components. In one embodiment, output reference level control (not shown) may also be provided to the decoder. The diagram shows that loudness and dynamic range parameters are determined and inserted in the encoder, but similar determinations can be performed in other audio processing units such as pre-processors, decoders, and post-processors.

[0094] Figure 15 shows examples of loudness profiles for various types of audio content under one embodiment. As shown in Figure 15, exemplary curves 600 and 602 plot the input loudness (at LKFS) against the gain centered around 0 LKFS. Different types of content result in different curves, as shown in Figure 15. In the figure, curve 600 may represent speech, and curve 602 may represent standard film content. As shown in Figure 15, speech content receives a larger amount of gain than film content. Figure 15 is intended to be an example of representative profile curves for certain types of audio content, and other profile curves may also be used. Certain aspects of the profile characteristics, as shown in Figure 15, are used to derive significant parameters for an optimization system. In one embodiment, these parameters include: null bandwidth, cutoff ratio, boost ratio, maximum boost, FS attack, FS attenuation, holdoff, peak limit, and target level loudness. Depending on the application requirements and system constraints, other parameters may be used in addition to or as substitutes for at least some of these parameters.

[0095] Figure 16 illustrates a method for optimizing loudness and dynamic range across a playback device and application under one embodiment. While the figure shows the loudness and dynamic range optimization performed in the encoder, similar optimizations can be performed in other audio processing units such as pre-processors, decoders, and post-processors. As shown in process 620, the method begins with an encoder stage (603) that receives an input signal from the source. The encoder or pre-processing component then determines whether the source signal has undergone a process to achieve the target loudness and / or dynamic range (604). The target loudness corresponds to the long-term loudness and may be defined externally or internally. If the source signal has not undergone a process to achieve the target loudness and / or dynamic range, the system performs an appropriate loudness and / or dynamic range control operation (608); otherwise, if the source signal has undergone this loudness and / or dynamic range control operation, the system enters bypass mode, skips the loudness and / or dynamic range operation, and allows the original process to specify the appropriate long-term loudness and / or dynamic range (606). An appropriate gain value for either bypass mode 606 or mode 608 being performed (which may be a single broadband gain value or a frequency-dependent multiband gain value) is then applied in the decoder (612).

[0096] <Bitstream Format> As previously mentioned, systems for optimizing loudness and dynamic range employ a secure, extensible metadata format to ensure that metadata and audio content transmitted in the bitstream between the encoder and decoder or between the source and the rendering / playback device are not isolated from each other or otherwise damaged during transmission through other proprietary equipment such as a network or service provider interface. This bitstream provides a mechanism for signal transmission to allow the encoder and / or decoder components to adapt the loudness and dynamic range of the audio signal to match the audio content and output device characteristics through appropriate profile information. In one embodiment, the system is configured to determine a low-bitrate encoded bitstream to be transmitted between the encoder and decoder, and the loudness information encoded through the metadata includes characteristics for one or more output profiles. A bitstream format for use with a loudness and dynamic range optimization system under one embodiment is described below.

[0097] An AC-3 encoded bitstream contains metadata and audio content for one to six channels. The audio content is audio data compressed using perceptual audio coding. The metadata includes several audio metadata parameters intended for use in altering the sound of the program delivered to the listening environment. Each frame of an AC-3 encoded audio bitstream contains audio content and metadata for 1536 samples of digital audio. For a sampling rate of 48 kHz, this represents a rate of 32 milliseconds of digital audio or 31.25 frames per second of audio.

[0098] Each frame of an E-AC-3 encoded audio bitstream contains audio content and metadata for 256, 512, 768, or 1536 samples of digital audio, depending on whether the audio data contained in the frame is in one, two, three, or six blocks, respectively. For a sampling rate of 48 kHz, this represents 5.333, 10.667, 16, or 32 milliseconds of digital audio, or 189.9, 93.75, 62.5, or 31.25 frames per second of audio, respectively.

[0099] As shown in Figure 4, each AC-3 frame is divided into sections (segments). The sections include: a synchronization information (SI) section containing the synchronization word (SW) and the first of two error correction words (CRC1) (as shown in Figure 5); a bitstream information (BSI) section containing most of the metadata; six audio blocks (AB0 to AB5) containing the data-compressed audio content (and may also contain metadata); a waste bit segment (W) containing any unused bits remaining after the audio content has been compressed; an auxiliary (AUX) information section which may contain further metadata; and the second of two error correction words (CRC2).

[0100] As shown in Figure 7, each E-AC-3 frame is divided into sections (segments). The sections include a synchronization information (SI) section containing the synchronization word (SW) (as shown in Figure 5); a bitstream information (BSI) section containing most of the metadata; one to six audio blocks (AB0 to AB5) containing the data-compressed audio content (and may also contain metadata); a waste bit segment (W) containing any unused bits remaining after the audio content has been compressed; an auxiliary (AUX) information section which may contain further metadata; and a error correction word (CRC).

[0101] AC-3 (or E-AC-3) bitstreams have several audio metadata parameters specifically intended for use in altering the sound of a program delivered to the listening environment. One such metadata parameter is the dialnorm parameter, which is included in the BSI segment.

[0102] As shown in Figure 6, the BSI segment of an AC-3 frame includes a five-bit parameter ("dialnorm") indicating the dialnorm value for the program in question. If the audio encoding mode ("acmod") of the AC-3 frame is "0", indicating that a dual-mono or "1+1" channel configuration is being used, then a five-bit parameter ("dialnorm2") indicating the dialnorm value for a second audio program carried in the same AC-3 frame is included.

[0103] A BSI segment includes a flag ("addbsie") indicating the presence (or absence) of additional bitstream information following the "addbsie" bit, a parameter ("addbsil") indicating the length of any additional bitstream information following the "addbsil" value, and up to 64 bits of additional bitstream information ("addbsi") following the "addbsil" value.

[0104] The BSI segment may include other metadata values ​​not specifically shown in Figure 6.

[0105] Aspects of one or more embodiments described herein may be implemented in an audio system for processing audio signals for transmission over a network, including one or more computers or processing units that execute software instructions. Any of the embodiments described may be used alone or in any combination with one another. Various embodiments may be motivated by various shortcomings of the prior art that may be discussed or implied in one or more places herein, but such embodiments do not necessarily address any of these shortcomings. That is, various embodiments may address various shortcomings that may be discussed herein. Some embodiments may only partially address some or just one of the shortcomings that may be discussed herein, and some embodiments may not address any of these shortcomings.

[0106] The aspects of the systems described in this paper may be implemented in a suitable computer-based audio processing network environment for processing digital or digitized audio files. The components of the adaptive audio system may include one or more networks, each containing any desired number of individual machines, including one or more routers (not shown) that buffer and route data transmitted between computers. Such networks may be built on various network protocols and may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0107] One or more of the above-described components, blocks, processes, or other functional elements may be implemented through a computer program that controls the execution of the system's processor-based computing device. It should also be noted that the various functions disclosed in this paper may be described using behavioral, register transfer, logical components, and / or other characteristics as data and / or instructions embodied using several combinations of hardware and firmware and / or in various machine-readable or computer-readable media. Computer-readable media in which such formatted data and / or instructions may be embodied include, but are not limited to, various forms of physical (non-temporary), non-volatile storage media such as optical, magnetic, or semiconductor storage media.

[0108] Unless the context explicitly requires otherwise, throughout this description and claims, words such as “have,” “include,” etc., shall be interpreted in an inclusive rather than exclusive or exhaustive sense; that is, “include, but not limited to….” Words used with the singular or plural also include the plural or singular, respectively. Furthermore, “in this paper,” “below,” “above,” “below,” and similar words refer to the Application as a whole, and not to any particular part of the Application. When the word “or” is used in reference to a list of two or more items, the word covers all of the following interpretations of the word: any item in the list, all items in the list, and any combination of items in the list.

[0109] While one or more implementations are described, for example, using individual embodiments, it should be understood that the one or more implementations are not limited to the disclosed embodiments. Conversely, it is intended to cover a variety of modifications and similar configurations that would be obvious to those skilled in the art. Therefore, the scope of the accompanying claims should be given the broadest interpretation to encompass all such modifications and similar configurations.

[0110] Several aspects are described below. [Aspect 1] The stage of receiving metadata associated with audio data in a bitstream; The steps include: analyzing the metadata in the bitstream to determine whether loudness parameters for a first group of audio playback devices are available in the bitstream; In response to determining that the parameters exist for the first group, the step is to use the parameters and audio data to render the audio; The process includes the step of analyzing one or more characteristics of the first group in response to determining that the loudness parameter does not exist for the first group, and determining the parameter based on the one or more characteristics, method. [Aspect 2] The method according to embodiment 1, wherein the one or more characteristics include gain levels for different profiles of the audio data. [Aspect 3] The method according to aspect 2, wherein the gain level defines at least one of an artistic dynamic range profile that controls how the audio data is reproduced for a defined program, and a short-term dynamic range profile that provides overload protection for parts of the defined program. [Aspect 4] The method according to embodiment 1, wherein the step of using the parameters and audio data to render audio includes transmitting the parameters and audio data to a downstream module that renders the audio for playback. [Aspect 5] The method according to embodiment 1, wherein the step of using the parameters and audio data to render audio includes rendering the audio data based on the parameters and audio data. [Aspect 6] The steps include determining the output device that renders the received audio stream; The process further includes the step of determining whether the output device belongs to the first group of audio playback devices, wherein the step of analyzing metadata in the stream to determine whether loudness parameters for the first group of audio playback devices are available is performed after the step of determining whether the output device belongs to the first group of audio playback devices. The method described in Embodiment 1. [Aspect 7] The step of determining that the output device belongs to the first group of audio playback devices is: Receiving an indicator from a module connected to the output device that indicates the characteristics of the output device or the characteristics of a group of devices including the output device; This includes determining, based on the received indicators, that the output device belongs to the first group of audio playback devices. The method described in aspect 6. [Aspect 8] An interface configured to receive a bitstream containing metadata associated with audio data; An analyzer coupled to the interface, configured to analyze the metadata in the bitstream to determine whether loudness parameters for a first group of audio playback devices are available in the bitstream; A rendering component configured to use the parameter and audio data to render audio in response to the analyzer determining that the parameter exists for the first group, and further configured to analyze one or more characteristics of the first group and determine the parameter based on the one or more characteristics in response to the analyzer determining that the loudness parameter does not exist for the first group, Device. [Aspect 9] The apparatus according to embodiment 8, wherein the rendering component uses the parameters and audio data to render audio, and the rendering component transmits the parameters and audio data to a downstream module that renders the audio for playback. [Aspect 10] The apparatus according to embodiment 9, wherein the rendering component uses the parameters and audio data to render audio, and the rendering component renders the audio data based on the parameters and audio data. [Aspect 11] The system further comprises a second component configured to determine an output device that renders the received audio stream and to determine whether the output device belongs to the first group of audio playback devices. The analyzer analyzes metadata in the stream to determine whether loudness parameters for the first group of audio playback devices are available, after the second component has determined whether the output device belongs to the first group of audio playback devices. The apparatus according to embodiment 10. [Aspect 12] The apparatus according to embodiment 11, further comprising an interface that receives an index from a module connected to the output device that indicates the characteristics of the output device or the characteristics of a group of devices including the output device, and which is configured such that the output device belongs to the first group of audio playback devices based on the received index. [Aspect 13] The steps include receiving audio data and metadata associated with said audio data; The steps include: analyzing the metadata in the bitstream to determine whether loudness information related to loudness parameters for a first group of audio devices is available in the stream; In response to determining that the loudness information exists for the first group, the steps include determining the loudness information from the stream and transmitting the audio data and loudness information for use in audio rendering; The process includes determining loudness information associated with an output profile in response to determining that the loudness information does not exist for the first group, and transmitting the determined loudness information for the output profile for use in audio rendering. method. [Aspect 14] The next step in determining the loudness information associated with the output profile is: The characteristics of the output profile are analyzed; Further includes determining the parameters based on the aforementioned characteristics, Sending determined loudness information includes sending determined parameters. The method described in Embodiment 13. [Aspect 15] The method according to embodiment 14, wherein the characteristics include gain levels for different profiles of the audio data. [Aspect 16] The method according to aspect 15, wherein the gain level defines at least one of an artistic dynamic range profile that controls how the audio data is reproduced for a defined program, and a short-term dynamic range profile that provides overload protection for parts of the defined program. [Aspect 17] The method according to embodiment 13, wherein the loudness information includes loudness parameters for an output profile. [Aspect 18] The method according to embodiment 13, wherein the loudness information includes the characteristics of the output profile. [Aspect 19] The method according to embodiment 13, further comprising the step of determining a low-bitrate encoded stream to be transmitted, wherein the loudness information includes characteristics for one or more output profiles. [Aspect 20] The method according to aspect 17, wherein one or more output profiles do not include a premium content profile. [Aspect 21] A device for decoding audio data: An interface for receiving the aforementioned audio data and metadata associated with the aforementioned audio data; The system includes a first component that analyzes metadata in a bitstream to determine whether loudness information related to loudness parameters for a first group of audio devices is available in the stream, the first component, in response to determining that the loudness information is present for the first group, determines the loudness information from the stream and transmits the audio data and loudness information for use in audio rendering, and in response to determining that the loudness information is not present for the first group, determines the loudness information associated with an output profile and transmits the determined loudness information for the output profile for use in audio rendering. Device. [Aspect 22] The first component analyzing the metadata includes the first component analyzing the characteristics of the output profile and determining the parameters based on those characteristics, and the first component transmitting the determined loudness information includes transmitting the determined parameters. The apparatus according to embodiment 21. [Aspect 23] The apparatus according to embodiment 22, wherein the loudness information includes loudness parameters for an output profile. [Aspect 24] The apparatus according to embodiment 23, wherein the loudness information includes the characteristics of the output profile. [Aspect 25] The apparatus according to embodiment 24, further comprising a second component that determines a low-bitrate encoded stream to be transmitted, wherein the loudness information includes characteristics for one or more output profiles.

Claims

1. An audio processing device for decoding one or more frames of an encoded audio bitstream, wherein the encoded audio bitstream includes audio data and metadata for a plurality of dynamic range control (DRC) profiles, and the audio processing device: A bitstream parser configured to parse the encoded audio bitstream and extract the encoded audio data and metadata for one or more of the DRC profiles; The system includes an audio decoder configured to decode the encoded audio data and apply DRC gain to the decoded audio data. Each DRC profile is suitable for at least one device type or listening environment. The audio decoder selects one or more of the DRC profiles in response to information about the audio processing device or the listening environment; The DRC gain applied to the decoded audio data differs for different frequency bands and corresponds to one or more selected DRC profiles. The DRC gain corresponding to one or more selected DRC profiles is included in the metadata of the encoded audio bitstream. Audio processing unit.

2. A method performed by an audio processing unit for decoding one or more frames of an encoded audio bitstream, wherein the encoded audio bitstream includes audio data and metadata for a plurality of dynamic range control (DRC) profiles, and the method: The steps include: parsing the encoded audio bitstream to extract the encoded audio data and metadata for one or more of the DRC profiles; The process includes the steps of decoding the encoded audio data and applying DRC gain to the decoded audio data, Each DRC profile is suitable for at least one device type or listening environment. One or more of the aforementioned DRC profiles are selected in response to information about the audio processing device or the listening environment. The DRC gain applied to the decoded audio data differs for different frequency bands and corresponds to one or more selected DRC profiles. The DRC gain corresponding to one or more selected DRC profiles is included in the metadata of the encoded audio bitstream. method.

3. A non-temporary computer-readable storage medium having a sequence of instructions that, when executed by an audio decoding device, cause the audio decoding device to perform the method described in claim 2.

4. A computer program for causing a computer to perform the method described in claim 2.

Images