System and method for optimizing loudness and dynamic range between different playback devices

Abstract

The disclosure relates to systems and methods for optimizing loudness and dynamic range between different playback devices. Embodiments relate to methods and systems for receiving metadata associated with audio data in a bitstream and analyzing the metadata to determine whether loudness parameters for a first set of audio playback devices are available in the bitstream. In response to determining that there is a parameter for the first set, the system renders audio using the parameter and the audio data. In response to determining that there is no loudness parameter for the first set, the system analyzes one or more characteristics of the first set and determines a parameter based on the one or more characteristics.

Description

[0001] This application is a divisional application of the invention patent application with application number 201810859612.4, application date January 15, 2014, and invention name “System and method for optimizing loudness and dynamic range between different playback devices”. The invention patent application with application number 201810859612.4 is a divisional application of the invention patent application with application number 201480005314.9, application date January 15, 2014, and invention name “System and method for optimizing loudness and dynamic range between different playback devices”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of U.S. Provisional Application No. 61 / 754,882, filed on January 21, 2013, U.S. Provisional Application No. 61 / 809,250, filed on April 5, 2013, and U.S. Provisional Patent Application No. 61 / 824,010, filed on May 16, 2013, all of which are incorporated herein by reference in their entirety. Technical Field

[0004] One or more embodiments relate generally to audio signal processing, and more particularly to processing an audio data bitstream having metadata indicating loudness and dynamic range characteristics of the audio content based on a playback environment and device. Background Art

[0005] The subject matter discussed in the Background section should not be assumed to be prior art simply because it is mentioned in this section. Similarly, the issues mentioned in the Background section or related to the subject matter of the Background section should not be assumed to have been recognized in any prior art. The subject matter in the Background section merely represents different approaches, which themselves may also be inventions.

[0006] The dynamic range of an audio signal is generally the ratio between the maximum possible value and the minimum possible value of the sound embodied in the signal, and is usually measured as a decibel value (based on 10). In many audio processing systems, dynamic range control (or dynamic range compression) is used to reduce the level of loud sounds, and / or amplify the level of quiet sounds to adapt wide dynamic range source content to a narrower recorded dynamic range that can be more easily stored and reproduced using electronic devices. For audio-visual (AV) content, a dialogue reference level can be used to define a "zero" point for compression through a DRC mechanism. DRC is used to increase content below the dialogue reference level and cut content above the reference level.

[0007] In known audio coding systems, metadata associated with audio signals is used to set DRC levels based on the type and intended use of content. The DRC mode sets the amount of compression to be applied to the audio signal, and defines the output reference level of the decoder. Such a system can be limited to two DRC level settings, which are programmed into the encoder and selected by the user. For example, conventionally, a dialogue normalization (dialnorm) value of -31dB is used for content played back on a device supporting an AVR or full dynamic range, and a dialogue normalization (dialogue normalization) value of -20dB is used for content played back on a television or similar device. This type of system allows a single audio bitstream to be used for two common but very different playback scenarios by using two different sets of DRC metadata. However, such a system is limited to a pre-set dialogue normalization value, and is not optimized for playback in a variety of different playback devices and listening environments that become possible with the emergence of digital media and Internet-based streaming technology.

[0008] In current metadata-based audio coding systems, an audio data stream may include audio content (e.g., one or more channels of audio content) and metadata indicating at least one characteristic of the audio content. For example, in an AC-3 bitstream, there are several audio metadata parameters that are specifically intended to be used to change the sound of a program delivered to a listening environment. One of the metadata parameters is a dialogue normalization parameter that indicates the average loudness level of dialogue present in an audio program (or the average loudness of the content), and is used to determine the audio playback signal level.

[0009] During playback of a bitstream containing a sequence of different audio program segments, each with different dialog normalization parameters, an AC-3 decoder uses the dialog normalization parameters for each segment to perform a loudness process that modifies the playback level or loudness of the segment so that the perceived loudness of the dialogue for the segment is at a consistent level. Each encoded audio item in a sequence of encoded audio segments (items) will typically have different dialog normalization parameters, and the decoder will scale the level of each item so that the playback level or loudness of the dialogue for each item is the same or very similar, but this may require adding different amounts of gain to different items during playback.

[0010] In some embodiments, the dialogue normalization parameter is set by the user, rather than being automatically generated, but there is a default dialogue normalization value in the event that the user does not set a value. For example, the content creator may perform loudness measurements via a device external to the AC-3 encoder and then pass the results (indicative of the loudness of the spoken dialogue of the audio program) to the encoder to set the dialogue normalization value. Thus, the content creator is relied upon to correctly set the dialogue normalization parameter.

[0011] There are several different reasons why the dialogue normalization parameters in the AC-3 bitstream may be incorrect. First, each AC-3 encoder has a default dialogue normalization value that is used during bitstream generation if the dialogue normalization value is not set by the content creator. This default value may be significantly different from the actual dialogue loudness level of the audio. Second, even if the content creator measures the loudness and sets the dialogue normalization value accordingly, a loudness measurement algorithm or meter that does not conform to the recommended loudness measurement method may have been used, which results in incorrect dialogue normalization values. Third, even if the AC-3 bitstream has been created with the dialogue normalization value correctly measured and set by the content creator, the dialogue normalization value may have been changed to an incorrect value during the transmission and / or storage of the bitstream due to an intermediate module. For example, in television broadcast applications, it is common for AC-3 bitstreams to be decoded, corrected, and then re-encoded using incorrect dialogue normalization metadata information. Therefore, the dialogue normalization value included in the AC-3 bitstream may be incorrect or inaccurate, and thus may adversely affect the quality of the listening experience.

[0012] Furthermore, the dialog normalization parameters do not indicate the loudness processing status of the corresponding audio data (e.g., the type of loudness processing that has been performed on the audio data). In addition, currently employed loudness and DRC systems (such as those in Dolby Digital (DD) and Dolby Digital Plus (DD+) systems) are designed to present AV content in a consumer living room or theater. In order to make such content suitable for playback in other environments and listening devices (e.g., mobile devices), post-processing must be applied "blindly" in the playback device to make the AV content suitable for the listening environment. In other words, the post-processor (or decoder) assumes that the loudness level of the received content is at a certain level (e.g., -31 dB or -20 dB), and the post-processor sets the level to a predetermined fixed target level suitable for the specific device. If the assumed loudness level or the predetermined target level is incorrect, the post-processing may have the opposite effect of its intended effect, i.e., the post-processing may cause the output audio to be lower than the user's expectations.

[0013] The disclosed embodiments are not limited to use with AC-3 bitstreams, E-AC-3 bitstreams, or Dolby E bitstreams, but for convenience, such bitstreams will be discussed in conjunction with a system that includes loudness processing state metadata. Dolby, Dolby Digital, Dolby Digital Plus, and Dolby E are trademarks of Dolby Laboratories, Inc., which provides proprietary implementations of AC-3 and E-AC-3 known as Dolby and Dolby Digital, respectively. Summary of the invention

[0014] Embodiments relate to a method for decoding audio data, the method receiving a bitstream containing metadata associated with the audio data, and analyzing the metadata in the bitstream to determine whether a loudness parameter for a first group of audio playback devices is available in the bitstream. In response to determining that the parameter exists for the first group, a processing component uses the parameter and audio data to render audio. In response to determining that the parameter does not exist for the first group, the processing component analyzes one or more characteristics of the first group and determines the parameter based on the one or more characteristics. The method may further use the parameter and audio data to render audio by transmitting the parameter and audio data to a downstream module that renders the audio for playback. The parameter and audio data may also be used to render audio by rendering the audio data based on the parameter and audio data.

[0015] In one embodiment, the method further comprises determining an output device that will present the received audio stream, and determining whether the output device belongs to a first group of audio playback devices; wherein the step of analyzing metadata in the stream to determine whether loudness parameters of the first group of audio playback devices are available is performed after the step of determining that the output device belongs to the first group of audio playback devices. In one embodiment, the step of determining that the output device belongs to the first group of audio playback devices comprises: receiving an indication indicating an identity of the output device or an identity of a group of devices including the output device from a module connected to the output device, and determining whether the output device belongs to the first group of audio playback devices based on the received indication.

[0016] Embodiments further relate to an apparatus or system comprising a processing component that performs the actions described in the encoding method embodiments above.

[0017] Embodiments further relate to an audio data decoding method that receives audio data and metadata associated with the audio data, analyzes the metadata in a bitstream to determine whether loudness information associated with loudness parameters of a first group of audio devices is available in the bitstream, and in response to determining that the loudness information is present for the first group, determines the loudness information from the bitstream and transmits the audio data and the loudness information for rendering the audio, or if the loudness information is not present for the first group, determines the loudness information associated with an output profile, and transmits the determined loudness information for the output profile for rendering the audio. In one embodiment, the step of determining the loudness information associated with the output profile may further include analyzing characteristics of the output profile, determining parameters based on the characteristics, and transmitting the determined loudness information includes transmitting the determined parameters. The loudness information may include loudness parameters of the output profile or characteristics of the output profile. In one embodiment, the method may further include determining a low bitrate encoded stream to be transmitted, wherein the loudness information includes one or more characteristics of the output profile.

[0018] Embodiments further relate to an apparatus or system comprising a processing component that performs the actions described in the above decoding method embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Like reference numerals are used to refer to like elements in the following figures.Although the following figures depict various examples, the implementations described herein are not limited to the examples depicted in the figures.

[0020] Figure 1 is a block diagram of an embodiment of an audio processing system configured to perform optimization of loudness and dynamic range according to some embodiments;

[0021] Figure 2 According to some embodiments Figure 1 Block diagram of the encoder used in the system.

[0022] Figure 3 According to some embodiments Figure 1 Block diagram of the decoder used in the system.

[0023] Figure 4 is a diagram of an AC-3 frame, including segmentation of the AC-3 frame into multiple segments.

[0024] Figure 5 is a diagram of a synchronization information (SI) segment of an AC-3 frame, including segmentation of an AC-3 frame into multiple segments.

[0025] Figure 6 is a diagram of the bit stream information (BSI) segmentation of an AC-3 frame, including the segmentation of an AC-3 frame into multiple segments.

[0026] Figure 7 is a diagram of an E-AC-3 frame, including segmentation of the E-AC-3 frame into multiple segments.

[0027] Figure 8 is a table showing the format of certain frames and metadata of an encoded bitstream according to some embodiments.

[0028] Fig. 9 is a table illustrating the format of loudness processing state metadata according to some embodiments.

[0029] Fig.10 According to some embodiments, it can be configured to perform optimization of loudness and dynamic range. Figure 1 A more detailed block diagram of the audio processing system.

[0030] Fig.11 is a table illustrating different dynamic range requirements for various playback devices and background listening environments in exemplary use cases.

[0031] Fig.12 is a block diagram of a dynamic range optimization system according to an embodiment.

[0032] Fig.13 is a block diagram of the interface between different profiles for various different playback device classes, according to some embodiments.

[0033] Fig.14 is a table showing the correlation between long term loudness and short term dynamic range for various defined profiles according to an embodiment.

[0034] Fig.15 Examples of loudness profiles for different types of audio content are shown according to an embodiment.

[0035] Fig.16 is a flow chart illustrating a method of optimizing loudness and dynamic range between a playback device and an application according to an embodiment. DETAILED DESCRIPTION

[0036] Definition and nomenclature

[0037] In the context of the present disclosure, including in the claims, the expression "performing an operation on a signal or data (e.g., filtering, scaling, transforming, or applying a gain to the signal or data)" is used in a broad sense to indicate that the operation is performed directly on the signal or data, or on a processed version of the signal or data (e.g., a version of the signal that has undergone preliminary filtering or preprocessing before performing the operation). The expression "system" is used in a broad sense to indicate a device, system, or subsystem. For example, a subsystem that implements a decoder may be referred to as a decoder system, and a system including such a subsystem (e.g., a system that generates X output signals in response to a plurality of data, where the subsystem generates M of the outputs and another XM of the inputs are received from external sources) may also be referred to as a decoder system. The term "processor" is used in a broad sense to indicate a system or device that is programmable or configurable (e.g., by software or firmware) to perform an operation on data (e.g., audio, video, or other image data). Examples of processors include field programmable gate arrays (or other programmable integrated circuits or chipsets), digital signal processors programmed and / or otherwise configured to perform pipeline processing of audio or other sound data, programmable general purpose processors or computers, and programmable microprocessor chips or chipsets.

[0038] The expressions "audio processor" and "audio processing unit" are used interchangeably and in a broad sense refer to a system configured to process audio data. Examples of audio processing units include, but are not limited to, encoders (e.g., transcoders), decoders, codecs, pre-processing systems, post-processing systems, and bitstream processing systems (sometimes referred to as bitstream processing tools). The expression "processing state metadata" (e.g., in the expression "loudness processing state metadata") refers to separate and distinct data from corresponding audio data (the audio content of the audio data stream that also includes the processing state metadata). The processing state metadata is associated with the audio data, indicating the loudness processing state of the corresponding audio data (e.g., what type of processing has been performed on the audio data), and optionally also indicating at least one feature or characteristic of the audio data. In some embodiments, the association of the processing state metadata with the audio data is time-synchronized. Thus, the current (most recently received or updated) processing state metadata indicates that the corresponding audio data simultaneously includes the results of the indicated type of audio data processing. In some cases, the processing state metadata may include some or all of the processing history and / or parameters used in and / or derived from the indicated type of processing. In addition, the processing state metadata may include at least one feature or characteristic of the corresponding audio data that has been calculated or extracted from the audio data. The processing state metadata may also include other metadata that is not related to or derived from any processing of the corresponding audio data. For example, third-party data, tracking information, identifiers, proprietary or standard information, user annotation data, user preference data, etc. can be added by a specific audio processing unit to be passed to other audio processing units.

[0039] The expression "loudness processing state metadata" (or "LPSM") indicates processing state metadata that indicates a loudness processing state of corresponding audio data (e.g., what type of processing has been performed on the audio data), and optionally also indicates at least one feature or characteristic of the corresponding audio data (e.g., loudness). The loudness processing state metadata may include data that is not loudness processing state metadata (e.g., when considered alone). The terms "couple" or "coupled" are used to indicate a direct or indirect connection.

[0040] Systems and methods are described for an audio encoder / decoder that non-destructively normalizes the loudness and dynamic range of audio between various devices that require or use different target loudness values ​​and have different dynamic range capabilities. Methods and functional components according to some embodiments send information about audio content from an encoder to a decoder for one or more device profiles. A device profile specifies a desired target loudness and dynamic range for one or more devices. The system is extensible so that new device profiles with different "nominal" loudness targets can be supported.

[0041] In one embodiment, the system generates appropriate gains in the encoder based on loudness control and dynamic range requirements, or generates gains in the decoder under control from the encoder through parameterization of the original gains to reduce the data rate. The dynamic range system includes two mechanisms for implementing loudness control: an artistic dynamic range profile that provides content creator control over how the audio will be played back; and a separate protection mechanism that ensures that overload does not occur for various playback profiles. The system is also configured to allow the use of other metadata (internal or external) parameters to properly control loudness and dynamic range gains and / or profiles. The decoder is configured to support n-channel auxiliary inputs that will affect the loudness and dynamic range on the decoder side.

[0042] In some embodiments, loudness processing state metadata (LPSM) is embedded in one or more reserved fields (or empty bits) of a metadata segment of an audio bitstream that also includes audio data in other segments (audio data segments). For example, at least one segment of each frame of the bitstream includes the LPSM, and at least one other segment of the frame includes corresponding audio data (i.e., audio data whose loudness processing state and loudness are indicated by the LPSM). In some embodiments, the amount of data of the LPSM may be small enough to be transported without affecting the bit rate allocated for transporting the audio data.

[0043] Passing loudness processing state metadata in an audio data processing chain is particularly useful in situations where two or more audio processing units need to work in tandem with each other in a processing chain (or in a content lifecycle). In the case where loudness processing state metadata is not included in the audio bitstream, media processing issues such as quality, level and spatial degradation may occur, for example, when two or more audio codecs are applied in the chain and single-ended volume adjustment is applied more than once during the transmission of the bitstream to a media consumption device (or a rendering point of the audio content of the bitstream).

[0044] Loudness and dynamic range metadata processing system

[0045] Figure 1 is a block diagram of one embodiment of an audio processing system that may be configured to perform optimization of loudness and dynamic range in accordance with some embodiments using certain metadata processing (eg, pre-processing and post-processing) components. Figure 1 An exemplary audio processing chain (audio data processing system) is shown, wherein one or more of the elements of the system may be configured according to embodiments of the invention. Figure 1The system 10 includes the following elements coupled together as shown: a pre-processing unit 12, an encoder 14, a signal analysis and metadata correction unit 16, a transcoder 18, a decoder 20, and a post-processing unit 24. In variations of the system shown, one or more of the elements are omitted, or additional audio data processing units are included. For example, in one embodiment, the post-processing unit 22 is part of the decoder 20 rather than a separate unit.

[0046] In some implementations, Figure 1 The pre-processing unit of the encoder is configured to accept as input PCM (time domain) samples containing audio content and output processed PCM samples. The encoder 14 may be configured to accept as input PCM samples and output an encoded (e.g., compressed) audio bitstream indicative of the audio content. The data indicative of the bitstream of audio content is sometimes referred to herein as "audio data." In some embodiments, the audio bitstream output from the encoder includes loudness processing state metadata (and optionally other metadata) and the audio data.

[0047] The signal analysis and metadata correction unit 16 may accept one or more encoded audio bitstreams as input and determine (e.g., verify) whether the processing state metadata in each encoded audio bitstream is correct by performing signal analysis. In some embodiments, the verification may be performed by a state verifier component (such as Figure 2 102), and one such verification technique is described below in the context of state verifier 102. In some embodiments, unit 16 is included in an encoder and verification is performed by unit 16 or verifier 102. If the signal analysis and metadata correction unit finds that the included metadata is invalid, the metadata correction unit 16 performs signal analysis to determine the correct value and replaces the incorrect value with the determined correct value. Therefore, each encoded audio bitstream output from the signal analysis and metadata correction unit may include corrected processing state metadata as well as encoded audio data. The signal analysis and metadata correction unit 16 may be part of a pre-processing unit 12, an encoder 14, a transcoder 18, a decoder 20, or a post-processing unit 22. Alternatively, the signal analysis and metadata correction unit 16 may be a separate unit or part of another unit in the audio processing chain.

[0048] The transcoder 18 may accept an encoded audio bitstream as data and in response output a modified (or differently encoded) audio bitstream (e.g., by decoding the input stream and re-encoding the decoded bitstream in a different encoding format). The audio bitstream output from the transcoder includes the loudness processing state metadata (and optionally other metadata) and the encoded audio data. The metadata is already included in the bitstream.

[0049] Figure 1 The decoder 20 may accept as input an encoded (e.g., compressed) audio bitstream and output (in response) a stream of decoded PCM audio samples. In one embodiment, the output of the decoder is or includes any of: a stream of audio samples and a corresponding stream of loudness processing state metadata (and optionally other metadata) extracted from the input encoded bitstream; a stream of audio samples and a corresponding stream of control bits determined by the loudness processing state metadata (and optionally other metadata) extracted from the input encoded bitstream; or a stream of audio samples without the processing state metadata or a corresponding stream of control bits determined by the processing state metadata. In this last case, the decoder may extract the loudness processing state metadata (and / or other metadata) from the input encoded bitstream and perform at least one operation (e.g., validation) on the extracted metadata, but it does not output the extracted metadata or the control bits determined therefrom.

[0050] By configuring according to an embodiment of the present invention Figure 1 The post-processing unit 22 is configured to accept a stream of decoded PCM audio samples and perform post-processing (e.g., volume adjustment of the audio content) on the stream of decoded PCM audio samples using loudness processing state metadata (and optionally other metadata) received with the samples or control bits received with the samples (determined by the decoder from the loudness processing state metadata and optionally other metadata). The post-processing unit 22 may optionally also be configured to present the post-processed audio content for playback by one or more speakers. These speakers may be embodied in a variety of different listening devices or playback devices (such as computers, televisions, stereo systems (for home or theater use), mobile phones, or other portable playback devices). The speakers may be of any suitable size and power rating, and may be provided in the form of stand-alone drivers, speaker cabinets, surround sound systems, sound bars, headphones, earbuds, etc.

[0051] Some embodiments provide an enhanced audio processing chain in which audio processing units (e.g., encoders, decoders, transcoders, and pre-processing and post-processing units) change their respective processing to be applied to audio data according to a contemporaneous state of media data indicated by loudness processing state metadata respectively received by the audio processing units. Any audio processing unit (e.g., Figure 1The audio data input 11 of the encoder or transcoder (e.g., an encoder or transcoder) may include loudness processing state metadata (and optionally other metadata) and audio data (e.g., encoded audio data). According to some embodiments, this metadata may be included in the input audio by other elements or other sources. The processing unit receiving the input audio (with metadata) may be configured to perform at least one operation on the metadata (e.g., validation) or respond to the metadata (e.g., adaptive processing of the input audio), and optionally also be configured to include the metadata, a processed version of the metadata, or control bits determined from the metadata in its output audio.

[0052] Embodiments of an audio processing unit (or audio processor) are configured to perform adaptive processing of audio data based on a state of the audio data indicated by loudness processing state metadata corresponding to the audio data. In some embodiments, the adaptive processing is (or includes) loudness processing (if the metadata indicates that loudness processing or similar processing has not been performed on the audio data) but is not (or does not include) loudness processing (if the metadata indicates that loudness processing or similar processing has been performed on the audio data). In some embodiments, the adaptive processing is or includes metadata verification (e.g., performed in a metadata verification subunit) to ensure that the audio processing unit performs other adaptive processing of the audio data based on the state of the audio data indicated by the loudness processing state metadata. In some embodiments, the verification determines the reliability of the loudness processing state metadata associated with the audio data (e.g., included in the bitstream). For example, if the metadata is verified to be reliable, the results of a previously performed audio processing can be reused, and additional execution of the same type of audio processing can be avoided. On the other hand, if the metadata is found to have been tampered with (or is unreliable), the type of media processing purportedly previously performed (as indicated by the unreliable metadata) may be repeated by the audio processing unit, and / or the audio processing unit may perform other processing on the metadata and / or audio data. The audio processing unit may also be configured to signal the loudness processing state metadata (e.g., present in the media bitstream) to other audio processing units downstream in the enhanced media processing chain that the loudness processing state metadata is valid if the audio processing unit determines that the loudness processing state metadata is valid (e.g., based on the extracted cryptographic value matching the reference cryptographic value).

[0053] for Figure 1 In an embodiment of the present invention, the pre-processing component 12 may be part of the encoder 14, and the post-processing component 22 may be part of the decoder 22. Alternatively, the pre-processing component 12 may be represented as a functional component separate from the encoder 14. Similarly, the post-processing component 22 may be represented as a functional component separate from the decoder 22.

[0054] Figure 2 It is available with Figure 11 is a block diagram of an encoder 100 for use in conjunction with the system 10 of FIG. Any component of the encoder 100 may be implemented in hardware, software, or a combination of hardware and software as one or more processes and / or one or more circuits (e.g., an ASIC, FPGA, or other integrated circuit). The encoder 100 includes a frame buffer 110, a parser 111, a decoder 101, an audio state validator 102, a loudness processing stage 103, an audio stream selection stage 104, an encoder 105, a filler / formatter stage 107, a metadata generation stage 106, a dialog loudness measurement subsystem 108, and a frame buffer 109, which are connected as shown. Optionally, the encoder 100 also includes other processing elements (not shown). The encoder 100 (which is a transcoder) is configured to convert an input audio bitstream (which may be, for example, one of an AC-3 bitstream, an E-AC-3 bitstream, or a Dolby E bitstream) into an encoded output audio bitstream (which may be, for example, another of an AC-3 bitstream, an E-AC-3 bitstream, or a Dolby E bitstream), including performing adaptive and automatic loudness processing using loudness processing state metadata contained in the input bitstream. For example, the encoder 100 may be configured to convert an input Dolby E bitstream (a format typically used in production and broadcast facilities, but not used in consumer devices that receive audio programs broadcast thereto) into an encoded output audio bitstream in an AC-3 or E-AC-3 format (suitable for broadcasting to consumer devices).

[0055] Figure 2 The system also includes an encoded audio delivery system 150 (which stores and / or delivers the encoded bitstream output from the encoder 100) and a decoder 152. The encoded audio bitstream output from the encoder 100 may be stored by the subsystem 150 (e.g., in the form of a DVD or BluRay disc), or transmitted by the subsystem 150 (which may implement a transmission link or network), or may be stored and transmitted by the subsystem 150. The decoder 152 is configured to decode the encoded bitstream (generated by the encoder 100) received via the subsystem 150, including extracting loudness processing state metadata (LPSM) from each frame of the bitstream, and generating decoded audio data. In one embodiment, the decoder 152 is configured to perform adaptive loudness processing on the decoded audio data using the LPSM, and / or forward the decoded audio data and the LPSM to a post-processor, which is configured to perform adaptive loudness processing on the decoded audio data using the LPSM. Optionally, the decoder 152 includes a buffer that stores (e.g., in a non-transitory manner) the encoded audio bitstream received from the subsystem 150.

[0056] Various implementations of the encoder 100 and decoder 152 are configured to perform various embodiments described herein. The frame buffer 110 is a buffer memory coupled to receive an encoded input audio bitstream. In operation, the buffer 110 stores (e.g., in a non-transitory manner) at least one frame in the encoded audio bitstream, and a sequence of frames of the encoded audio bitstream is asserted from the buffer 110 to the parser 111. The parser 111 is coupled and configured to extract loudness processing state metadata (LPSM) and other metadata from each frame in the encoded input audio, to assert at least the LPSM to the audio state validator 102, the loudness processing stage 103, the stage 106, and the subsystem 108, to extract audio data from the encoded input audio, and to assert the audio data to the decoder 101. The decoder 101 of the encoder 100 is configured to decode audio data to generate decoded audio data and assert the decoded audio data to the loudness processing stage 103 , the audio stream selection stage 104 , the subsystem 108 and optionally also to the state validator 102 .

[0057] The state verifier 102 is configured to authenticate and verify the LPSM (and optionally other metadata) asserted to the state verifier 102. In some embodiments, the LPSM is (or is included in) a data block that is already included in the input bitstream (e.g., according to embodiments of the present invention). The block may include a cryptographic hash (hash message authentication code or "HMAC") for processing the LPSM (and optionally other metadata) and / or the base layer audio data (provided to the verifier 102 from the decoder 101). In these embodiments, the data block may be digitally signed so that downstream audio processing units can relatively easily authenticate and verify the processing state metadata.

[0058] For example, HMAC is used to generate a digest, and the protection value included in the bitstream of the present invention may include the digest. The digest may be generated for an AC-3 frame as follows: (1) After the AC-3 data and LPSM are encoded, the frame data bytes (linking frame data #1 and frame data #2) and the LPSM data bytes are used as inputs to the hash function HMAC. Other data that may be present in the auxiliary data (auxdata) field is not considered for the digest calculation. Such other data may be bytes that are neither AC-3 data nor LPSM data. The protection bits contained in the LPSM may not be considered for the HMAC digest calculation. (2) After the digest is calculated, the digest is written to the field reserved for the protection bits in the bitstream. (3) The last step in generating a complete AC-3 frame is to calculate the CRC check bits. This is written to the very end of the frame, and all data belonging to this frame (including the LPSM bits) is taken into account.

[0059] Other cryptographic methods, including but not limited to any of one or more non-HMAC cryptographic methods, may be used to authenticate the LPSM (e.g., in the authenticator 102) to ensure secure transmission and reception of the LPSM and / or base layer audio data. For example, authentication (using such cryptographic methods) may be performed in each audio processing unit that receives an embodiment of the inventive audio bitstream to determine whether the loudness processing state metadata and corresponding audio data contained in the bitstream have been subjected to (and / or derived from) specific loudness processing (as indicated by the metadata) and have not been modified after performing such specific loudness processing.

[0060] The state validator 102 asserts control data to the audio stream selection stage 104, the metadata generator 106, and the dialog relative measurement subsystem 108 to indicate the result of the validation operation. In response to the control data, the stage 104 may select (and pass to the encoder 105) either: (1) the adaptively processed output of the loudness processing stage 103 (e.g., when the LPSM indicates that the audio data output by the decoder 101 has not been subjected to a particular type of loudness processing, and the control bit from the validator 102 indicates that the LPSM is valid); or (2) the audio data output from the decoder 101 (e.g., when the LPSM indicates that the audio data output by the decoder 101 has been subjected to a particular type of loudness processing performed by the stage 103, and the control bit from the validator 102 indicates that the LPSM is valid). In one embodiment, the loudness processing stage 103 corrects the loudness to a specified target and loudness range.

[0061] Stage 103 of encoder 100 is configured to perform adaptive loudness processing on decoded audio data output from decoder 101 based on one or more audio data characteristics indicated by the LPSM extracted by decoder 101. Stage 103 may be an adaptive transform domain real-time loudness and dynamic range control processor. Stage 103 may receive user input (e.g., user target loudness / dynamic range value or dialog normalization value), or other metadata input (e.g., one or more third-party data, tracking information, identifiers, proprietary or standard information, usage annotation data, user preference data, etc.), and / or other input (e.g., from a fingerprinting process), and use such input to process the decoded audio data output from decoder 101.

[0062] The dialogue loudness measurement subsystem 108 may be operable to determine the loudness of a segment of the decoded audio (from the decoder 101) indicating dialogue (or other speech) when the control bit from the validator 102 indicates that the LPSM is invalid, for example, using the LPSM (and / or other metadata) extracted by the decoder 101. When the LPSM indicates a previously determined loudness of a segment of the decoded audio (from the decoder 101), the operation of the dialogue loudness measurement subsystem 108 may be disabled when the control bit from the validator 102 indicates that the LPSM is valid.

[0063] There are useful tools (e.g., DolbyLM100 loudness meter) for conveniently and easily measuring the level of dialogue in audio content. Some embodiments of the APU (e.g., stage 108 of encoder 100) are implemented to include such a tool (or perform its functions) to measure the average dialogue loudness of the audio content of an audio bitstream (e.g., a decoded AC-3 bitstream asserted from decoder 101 of encoder 100 to stage 108). If stage 108 is implemented to measure the true average dialogue loudness of the audio data, the measurement may include the step of isolating the segment of the audio content that mainly contains speech. The audio segment that is mainly speech is then processed according to a loudness measurement algorithm. For audio data decoded from an AC-3 bitstream, this algorithm may be a standard K-weighted loudness measurement (according to international standard ITU-R BS.1770). As an alternative, other loudness measurements may be used (e.g., those based on psychoacoustic models of loudness).

[0064] Isolation of speech segments is not necessary to measure the average dialog loudness of audio data. However, it improves the accuracy of the measurement and provides more pleasing results from the listener's perspective. Since not all audio content contains dialogue (speech), loudness measurement of the entire audio content can provide a sufficient approximation of the dialog level of the audio (in the presence of speech).

[0065] The metadata generator 106 generates metadata to be included in the encoded bitstream to be output from the encoder 100 by the stage 107. The metadata generator 106 may pass the LPSM (and / or other metadata) extracted by the encoder 101 to the stage 107 (e.g., when the control bit from the verifier 102 indicates that the LPSM and / or other metadata are valid), or generate a new LPSM (and / or other metadata) and assert the new LPSM to the stage 107 (e.g., when the control bit from the verifier 102 indicates that the LPSM and / or other metadata extracted by the decoder 101 are invalid), or it may assert a combination of the metadata extracted by the decoder 101 and the newly generated metadata to the stage 107. The metadata generator 106 may include loudness data generated by the subsystem 108 and at least one value indicating the type of loudness processing performed by the subsystem 108 in the LPSM asserted by the metadata generator 106 to the stage 107 for inclusion in the encoded bitstream to be output from the encoder 100. The metadata generator 106 may generate protection bits (which may consist of or include a hashed message authentication code or "HMAC") that are useful for at least one of decryption, authentication, or verification of the LPSM (and / or other metadata) to be included in the encoded bitstream and / or base layer audio data to be included in the encoded bitstream. The metadata generator 106 may provide such protection bits to the stage 107 for inclusion in the encoded bitstream.

[0066] In one embodiment, the dialog loudness measurement subsystem 108 processes the audio data output from the decoder 101 to generate loudness values ​​(e.g., gated or ungated dialog loudness values) and dynamic range values ​​in response thereto. In response to these values, the metadata generator 106 may generate loudness processing state metadata (LPSM) for inclusion (via the stuffer / formatter 107) in the encoded bitstream to be output from the encoder 100. In one embodiment, the loudness may be calculated based on the techniques specified in the ITU-R BS.1770-1 and ITU-R BS.1770-2 standards, or other similar loudness measurement standards. The gated loudness may be dialog gated loudness or relative gated loudness, or a combination of these gated loudness types, and the system may employ the appropriate gating block based on application requirements and system constraints.

[0067] Additionally, optionally, or alternatively, subsystems 106 and / or 108 of encoder 100 may perform additional analysis of the audio data to generate metadata indicative of at least one characteristic of the audio data for inclusion in an encoded bitstream to be output from stage 107. Encoder 105 encodes (e.g., by compressing) the audio data output from selection stage 104 and asserts the encoded audio to stage 107 for inclusion in an encoded bitstream to be output from stage 107.

[0068] Stage 107 multiplexes the encoded audio from encoder 105 and metadata (including LPSM) from generator 106 to generate an encoded bitstream to be output from stage 107, so that the encoded bitstream has a format as specified by the embodiment. Frame buffer 109 is a buffer memory that stores (e.g., in a non-transitory manner) at least one frame of the encoded audio bitstream output from stage 107, and then a sequence of frames of the encoded audio bitstream is asserted from buffer 109 as an output of encoder 100 to delivery system 150.

[0069] The LPSM generated by metadata generator 106 and included in the encoded bitstream by stage 107 indicates the state of loudness processing of the corresponding audio data (e.g., the type of loudness processing that has been performed on the audio data), and the loudness of the corresponding audio data (e.g., measured dialog loudness, gated and / or ungated loudness, and / or dynamic range). Here, "gating" of loudness and / or level measurements performed on the audio data refers to a certain level or loudness threshold, and calculated values ​​exceeding the threshold are included in the final measurement (e.g., short-term loudness values ​​below -60 dBFS are ignored in the final measurement). Gating of an absolute value refers to a fixed level or loudness, while gating of a relative value refers to a value that is dependent on the current "ungated" measurement value.

[0070] In some implementations of encoder 100, the encoded bitstream buffered in memory 109 (and output to delivery system 150) is an AC-3 bitstream or an E-AC-3 bitstream and includes audio data segments (e.g., Figure 4 The stage 107 inserts the LPSM into the bitstream in the following format. Each of the metadata segments including the LPSM is included in an "addbsi" field of a bitstream information ("BSI") segment of a frame of the bitstream, or in an auxdata field at the end of a frame of the bitstream (e.g., Figure 4 AUX segment shown in ).

[0071] A frame of the bitstream may include one or two metadata segments, each of which includes an LPSM, and if the frame includes two metadata segments, one is present in the addbsi field of the frame and the other is present in the AUX field of the frame. Each metadata segment including the LPSM includes an LPSM payload (or container) segment having the following format: a header (e.g., including a synchronization word identifying the start of the LPSM payload, followed by at least one identification value, such as the LPSM format version, length, period, count, and substream association value indicated in Table 2 below); and after the header, at least one dialog indication value indicating whether the corresponding audio data indicates dialog or does not indicate dialog (e.g., which channel of the corresponding audio data indicates dialog) (e.g., parameter “dialog channel” of Table 2); at least one loudness rule compliance value indicating whether the corresponding audio data complies with the indicated set of loudness rules (e.g., parameter “loudness rule type” of Table 2); at least one loudness processing value indicating at least one loudness processing performed on the corresponding audio data (e.g., one or more of parameters “dialog gated loudness correction flag”, “loudness correction type” of Table 2); and at least one loudness value indicating at least one loudness (e.g., peak loudness or average loudness) characteristic of the corresponding audio data (e.g., parameters “ITU relative gated loudness”, “ITU speech gated loudness”, “ITU (EBU) loudness” of Table 2). 3341) One or more of "Short-term 3s loudness" and "True Peak".

[0072] In some implementations, each of the metadata segments inserted into the “addbsi” field or the auxdata field of a frame of the bitstream by stage 107 has the following format: a core header (e.g., including a synchronization word indicating the start of the metadata segment, followed by identification values, such as the core element version, length, period, extended element count, and substream association values ​​indicated in Table 1 below); and, following the core header, at least one protection value useful for at least one of decryption, authentication, or verification of at least one of the loudness processing state metadata or the corresponding audio data (e.g., the HMAC digest and audio fingerprint value of Table 1); and, also following the core header, in the event that the metadata segment includes an LPSM, an LPSM payload identification (“ID”) and an LPSM payload size value, which identifies the following metadata as an LPSM payload and indicates the size of the LPSM payload.

[0073] An LPSM payload (or container) segment (eg, having the format specified above) follows the LPSM payload ID and LPSM payload size value.

[0074] In some embodiments, each of the metadata segments in the auxdata field (or "addbsi" field) of the frame has a three-level structure: a high-level structure, including a flag indicating whether the auxdata (or addbsi) field includes metadata, at least one ID value indicating the type of metadata present, and optionally also including a value indicating how many bits of metadata (e.g., of each type) are present (if metadata is present). One type of metadata that may be present is LPSM, and another type of metadata that may be present is media survey metadata (e.g., Nielsen Media Survey metadata); a middle-level structure, including core elements of metadata of each identified type (e.g., a core header, protection value, and LPSM payload ID and LPSM payload size values ​​as mentioned above for each identified type of metadata); and a low-level structure, including each payload of a core element (e.g., an LPSM payload if identified as present by the core element, or another type of metadata payload if identified as present by the core element).

[0075] Data values ​​in such a three-level structure may be nested. For example, a protection value for an LPSM payload and / or another metadata payload identified by a core element may be included after each payload identified by the core element (thus, after a core header of the core element). In one example, a core header may identify an LPSM payload and another metadata payload, a payload ID and a payload size value for a first payload (e.g., an LPSM payload) may be after the core header, the first payload itself may be after the ID and size values, a payload ID and a payload size for a second payload may be after the first payload, the second payload itself may be after these ID and size values, and the protection values ​​for both payloads (or the core element value and both payloads) may be after the last payload.

[0076] In some embodiments, if the decoder 101 receives an audio bitstream generated according to an embodiment of the present invention having a cryptographic hash, the decoder is configured to parse and retrieve the cryptographic hash from a data block determined by the bitstream, the block comprising loudness processing state metadata (LPSM). The verifier 102 may use the cryptographic hash to verify the received bitstream and / or associated metadata. For example, the verifier 102 finds that the LPSM is valid based on a match between a reference cryptographic hash and a cryptographic hash retrieved from the data block, and then may inhibit the processor 103 from operating on the corresponding audio data and cause the selection stage 104 to pass the (unchanged) video data. Additionally, alternatively or alternatively, other types of cryptographic techniques may be used instead of cryptographic hash-based methods.

[0077] Figure 2The encoder 100 may determine (in response to the LPSM extracted by the decoder 101) that a post-processing / pre-processing unit has already performed loudness processing on the audio data to be encoded (in elements 105, 106, and 107), and may therefore create (in the generator 106) loudness processing state metadata including specific parameters used in and / or derived from previously performed loudness processing. In some embodiments, the encoder 100 may create (and include in the encoded bitstream it outputs) processing state metadata indicating the processing history of the audio content, at least so long as the encoder knows the types of processing that have been performed on the audio content.

[0078] Figure 3 It is available with Figure 1 1 is a block diagram of a decoder used in combination with the system 10 of the present invention. Any of the components or elements of the decoder 200 and the post-processor 300 can be implemented as one or more processes and / or one or more circuits (e.g., ASICs, FPGAs, or other integrated circuits) in hardware, software, or a combination of hardware and software. The decoder 200 includes a frame buffer 201, a parser 205, an audio decoder 202, an audio state verification stage (verifier) ​​203, and a control bit generation stage 204, which are connected as shown. The decoder 200 includes other processing elements (not shown). The frame buffer 201 (buffer memory) stores (e.g., in a non-transient manner) at least one frame in the encoded audio bit stream received by the decoder 200. A sequence of frames of the encoded audio bit stream is asserted from the buffer 201 to the parser 205. The parser 205 is coupled and configured to extract loudness processing state metadata (LPSM) and other metadata from each frame in the encoded input audio, to assert at least the LPSM to the audio state validator 203 and the stage 204, to assert the LPSM as an output (e.g., to the post-processor 300), to extract audio data from the encoded input audio, and to assert the extracted audio data to the decoder 202. The encoded audio bitstream input to the decoder 200 may be one of an AC-3 bitstream, an E-AC-3 bitstream, or a DolbyE bitstream.

[0079] Figure 3The system further includes a post-processor 300. The post-processor 300 includes a frame buffer 301 and other processing elements (not shown) including at least one processing element coupled to the buffer 301. The frame buffer 301 stores (e.g., in a non-transitory manner) at least one frame of a decoded audio bitstream received by the post-processor 300 from the decoder 200. The processing elements of the post-processor 300 are coupled and configured to receive and adaptively process a sequence of frames of the decoded audio bitstream output from the buffer 301 using metadata (including LPSM values) output from the decoder 202 and / or control bits output from the stage 204 of the decoder 200. In one embodiment, the post-processor 300 is configured to perform adaptive loudness processing on the decoded audio data using the LPSM value (e.g., based on a loudness processing state indicated by the LPSM and / or one or more audio data characteristics). Various implementations of the decoder 200 and the post-processor 300 are configured to perform different embodiments of the methods according to the embodiments described herein.

[0080] The audio decoder 202 of the decoder 200 is configured to decode the audio data extracted by the parser 205 to generate decoded audio data, and assert the decoded audio data as output (e.g., to the post-processor 300). The state verifier 203 is configured to authenticate and verify the LPSM (and optionally other metadata) asserted to the state verifier 203. In some embodiments, the LPSM is (or is included in) a data block that is already included in the input bitstream (e.g., according to an embodiment of the present invention). The block may include a cryptographic hash (hash message authentication code or "HMAC") for processing the LPSM (and optionally other metadata) and / or the base layer audio data (provided to the verifier 203 from the parser 205 and / or the decoder 202). In these embodiments, the data block may be digitally signed so that the downstream audio processing unit can relatively easily authenticate and verify the processing state metadata.

[0081] Other cryptographic methods, including but not limited to any of one or more non-HMAC cryptographic methods, may be used to authenticate the LPSM (e.g., in the authenticator 203) to ensure secure transmission and reception of the LPSM and / or base layer audio data. For example, authentication (using such cryptographic methods) may be performed in each audio processing unit of an embodiment of the invention that receives an audio bitstream to determine whether the loudness processing state metadata and corresponding audio data contained in the bitstream have been subjected to (and / or derived from) specific loudness processing (as indicated by the metadata) and have not been modified after performing such specific loudness processing.

[0082] The state validator 203 asserts the control data to the control bit generator 204 and asserts the control data as an output (e.g., to the post-processor 300) indicating the result of the validation operation. In response to the control data (and optionally other metadata extracted from the input bitstream), the stage 204 may generate (and assert to the post-processor 300) either of the following: (1) a control bit indicating that the decoded audio data output from the decoder 202 has been subjected to a particular type of loudness processing (e.g., when the LPSM indicates that the audio data output from the decoder 202 has been subjected to a particular type of loudness processing and the control bit from the validator 203 indicates that the LPSM is valid); or (2) a control bit indicating that the decoded audio data output from the decoder 202 should be subjected to a particular type of loudness processing (e.g., when the LPSM indicates that the audio data output from the decoder 202 has not been subjected to a particular type of loudness processing, or when the LPSM indicates that the audio data output from the decoder 202 has been subjected to a particular type of loudness processing but the control bit from the validator 203 indicates that the LPSM is invalid).

[0083] Alternatively, the decoder 200 asserts the LPSM (and any other metadata) extracted by the decoder 200 from the input bitstream to the post-processor 300, and the post-processor 300 performs loudness processing on the decoded audio data using the LPSM, or performs verification of the LPSM and then performs loudness processing on the decoded audio data using the LPSM if the verification indicates that the LPSM is valid.

[0084] In some embodiments, if the decoder 201 receives an audio bitstream generated according to an embodiment of the present invention having a cryptographic hash, the decoder is configured to parse and retrieve the cryptographic hash from a data block determined by the bitstream, the block comprising loudness processing state metadata (LPSM). The verifier 203 may use the cryptographic hash to verify the received bitstream and / or associated metadata. For example, if the verifier 203 finds that the LPSM is valid based on a match between a reference cryptographic hash and a cryptographic hash retrieved from the data block, it may signal a downstream audio processing unit (e.g., a post-processor 300, which may be or include a volume adjustment unit) to pass the (unchanged) audio data of the bitstream. Additionally, optionally, or alternatively, other types of cryptographic techniques may be used to replace the cryptographic hash-based approach.

[0085] In some implementations of decoder 100, the encoded bitstream received (and buffered in memory 201) is an AC-3 bitstream or an E-AC-3 bitstream and includes audio data segments (e.g., Figure 4AB0-AB5 segments of the frame shown in FIG. 1 ) and metadata segments, wherein the audio data segments indicate audio data, and each of at least some of the metadata segments includes loudness processing state metadata (LPSM). The decoder stage 202 is configured to extract the LPSM from the bitstream having the following format. Each of the metadata segments including the LPSM is included in an "addbsi" field of a bitstream information ("BSI") segment of a frame of the bitstream, or in an auxdata field at the end of a frame of the bitstream (e.g., Figure 4 A frame of the bitstream may include one or two metadata segments, each of which includes the LPSM, and if the frame includes two metadata segments, one is present in the addbsi field of the frame and the other is present in the AUX field of the frame. Each metadata segment including the LPSM includes an LPSM payload (or container) segment having the following format: a header (e.g., including a synchronization word identifying the start of the LPSM payload, followed by at least one identification value, such as the LPSM format version, length, period, count, and substream association value indicated in Table 2 below); and after the header, at least one dialog indication value indicating whether the corresponding audio data indicates dialog or does not indicate dialog (e.g., which channel of the corresponding audio data indicates dialog) (e.g., parameter “dialog channel” of Table 2); at least one loudness rule compliance value indicating whether the corresponding audio data complies with the indicated set of loudness rules (e.g., parameter “loudness rule type” of Table 2); at least one loudness processing value indicating at least one loudness processing performed on the corresponding audio data (e.g., one or more of parameters “dialog gated loudness correction flag”, “loudness correction type” of Table 2); and at least one loudness value indicating at least one loudness (e.g., peak loudness or average loudness) characteristic of the corresponding audio data (e.g., parameters “ITU relative gated loudness”, “ITU speech gated loudness”, “ITU (EBU) loudness” of Table 2). 3341) One or more of "Short-term 3s loudness" and "True Peak".

[0086] In some implementations, the decoder stage 202 is configured to extract each metadata segment having the following format from an "addbsi" field or an auxdata field of a frame of the bitstream: a core header (e.g., including a synchronization word identifying the start of the metadata segment, followed by at least one identification value, such as a core element version, length, period, extended element count, and substream association value as indicated in Table 1 below); and, following the core header, at least one protection value useful for at least one of decryption, authentication, or verification of at least one of the loudness processing state metadata or the corresponding audio data (e.g., an HMAC digest and an audio fingerprint value of Table 1); and, also following the core header, in the case where the metadata segment includes an LPSM, an LPSM payload identification ("ID") and an LPSM payload size value, which identifies the following metadata as an LPSM payload and indicates the size of the LPSM payload. An LPSM payload (or container) segment (e.g., having the format specified above) follows the LPSM payload ID and the LPSM payload size value.

[0087] More generally, the coded audio bitstream generated by the embodiments has a structure that provides a mechanism to mark metadata elements and sub-elements as core (mandatory elements) or extensions (optional elements). This allows the data rate of the bitstream (including its metadata) to be scaled between various applications. The core (mandatory) elements in the bitstream syntax should also be able to signal that extension (optional) elements associated with the audio content are present (in-band) and / or located at a remote location (out-of-band).

[0088] In some embodiments, the core element needs to be present in each frame of the bitstream. Some sub-elements of the core element are optional and can be present in any combination. The extended element does not need to be present in each frame of the bitstream (to limit the bit rate overhead). Therefore, the extended element can be present in some frames and not in other frames. Some sub-elements of the extended element are optional and can be present in any combination, and some sub-elements of the extended element can be mandatory (that is, when the extended element is present in the frame of the bitstream).

[0089] In some embodiments, an encoded audio bitstream is generated (e.g., by an audio processing unit embodying the invention) comprising a sequence of audio data segments and metadata segments. The audio data segments indicate audio data, at least some of the metadata segments each comprise loudness processing state metadata (LPSM), and the audio data segments are time-division multiplexed with the metadata segments. In some embodiments, in this type, each metadata segment has a format to be described herein. In one format, the encoded bitstream is an AC-3 bitstream or an E-AC-3 bitstream, and each metadata segment comprising the LPSM is included as additional bitstream information in an "addbsi" field ( Figure 6 ) or in the auxdata field of a frame of a bit stream. Each frame includes Figure 8 The core elements of the addbsi field of the frame in the format shown in Table 1.

[0090] In one format, each of the addbsi (or auxdata) fields containing an LPSM contains a core header (and optionally additional core elements); after the core header (or core header and other core elements), the following LPSM values ​​(parameters): a payload ID (identifying the metadata as an LPSM) after the core element value (e.g., as indicated in Table 1); a payload size (indicating the size of the LPSM payload) after the payload ID; and a LPSM payload having Fig. 9 The LPSM data in the format indicated by Table 2 (after the payload ID and payload size).

[0091] In a second format of the encoded bitstream, the bitstream is an AC-3 bitstream or an E-AC-3 bitstream, and each metadata segment containing the LPSM is contained (e.g., by stage 107 of encoder 100) in a field such as an "addbsi" field of a bitstream information ("BSI") segment of a frame of the bitstream, or in an auxdata field at the end of a frame of the bitstream (e.g., Figure 4 A frame may include one or two metadata segments, each of which includes an LPSM, and if a frame includes two metadata segments, one is present in the addbsi field of the frame and the other is present in the AUX field of the frame. Each metadata segment including an LPSM has the format specified above with reference to Table 1 or Table 2 (i.e., it includes the core elements specified in Table 1, followed by the payload ID specified above (identifying the metadata as an LPSM) and the payload size value, followed by the payload (LPSM data having the format indicated in Table 2)).

[0092] In another implementation, the encoded bitstream is a Dolby E bitstream, and each of the metadata segments including the LPSM is the first N sample positions of the Dolby E guard band interval. The Dolby E bitstream including such metadata segments includes the LPSM, for example including a value indicating the LPSM payload length signaled in the Pd word of the SMPTE 337M preamble (the SMPTE 337M Pa word repetition rate may remain the same as the associated video frame rate).

[0093] In a format where the encoded bitstream is an E-AC-3 bitstream, each metadata segment containing the LPSM (e.g., by stage 107 of encoder 100) is contained in the "addbsi" field of the bitstream information ("BSI") segment of a frame of the bitstream. Additional aspects of encoding an E-AC-3 bitstream with LPSM in this format are described as follows: (1) During E-AC-3 bitstream generation, while the E-AC-3 encoder (which inserts the LPSM value into the bitstream) is "on", for each frame generated (a sync frame), the bitstream shall contain a metadata block (including the LPSM) conveyed in the addbsi field of the frame. The bits conveying the metadata block shall not increase the encoder bit rate (frame length); (2) Each metadata block (including the LPSM) shall contain the following information:

[0094] loudness_correction_type_flag: where "1" indicates that the loudness of the corresponding audio data is corrected upstream of the encoder, and "0" indicates that the loudness is corrected by a loudness corrector embedded in the encoder (e.g., Figure 2 speed_channel indicates which source channel(s) contain speech (in the previous 0.5 seconds). If no speech is detected, it should be indicated as such; speech_loudness indicates the integrated speech loudness (in the previous 0.5 seconds) of each corresponding audio channel containing speech; ITU_loudness indicates the integrated ITU BS.1770-2 loudness of each corresponding audio channel; gain: loudness composite gain for inversion in the decoder (indicating reversibility).

[0095] When an E-AC-3 encoder (which inserts LPSM values ​​into the bitstream) is "on" and is receiving AC-3 frames with a "true" flag, a loudness controller (e.g. Figure 2The loudness controller bypass sequence is synchronized with the start of the decoded AC-3 frame in which the "confidence" flag appears. The loudness controller bypass sequence is implemented as follows: the leveler_amount control is reduced from a value of 9 to a value of 0 over 10 audio block periods (i.e., 53.3 milliseconds), and the leveler_back_end_meter control is placed in bypass mode (this operation will result in a seamless transition). The term "trusted" bypass of the leveler implies that the source bitstream's dialogue normalization value is also reused at the output of the encoder (e.g., if the "trusted" source bitstream has a dialogue normalization value of -30, then the output of the encoder should utilize -30 for the outbound dialogue normalization value).

[0096] When an E-AC-3 encoder (which inserts LPSM values ​​into the bitstream) is "working" and is receiving AC-3 frames without a "true" flag, a loudness controller (e.g. Figure 2 The loudness processor 103 of the encoder 100 is activated. LPSM block generation continues and loudness_correction_type_flag is set to "0". The loudness controller activation sequence is synchronized with the start of the decoded AC-3 frame where the "confidence" mark disappears. The loudness controller activation sequence is implemented as follows: the leveler_amount control is increased from a value of 0 to a value of 9 over 1 audio block period (i.e., 5.3 milliseconds), and the leveler_back_end_meter control is placed in an active mode (this operation will result in a seamless transition and includes a back_end_meter integral reset); and during encoding, a graphical user interface (GUI) indicates the following parameters to the user: "Input audio program: [Confidence / Unconfidence]" - the state of this parameter is based on the presence of the "Confidence" mark in the input signal; and "Real-time loudness correction: [Enabled / Disabled]" - the state of this parameter is based on whether this loudness controller embedded in the encoder is activated.

[0097] When decoding an AC-3 or E-AC-3 bitstream that contains an LPSM (in the described format) in the "addbsi" field of the Bitstream Information ("BSI") segment of each frame of the bitstream, the decoder parses the LPSM block data (in the addbsi field) and passes the extracted LPSM values ​​to a graphical user interface (GUI). The set of extracted LPSM values ​​is refreshed every frame.

[0098] In yet another format, the encoded bitstream is an AC-3 bitstream or an E-AC-3 bitstream, and each of the metadata segments including the LPSM is included (e.g., by stage 107 of encoder 100) as additional bitstream information in an "addbsi" field ( Figure 6) (or in the Aux segment). In this format (which is a variation of the format described above with reference to Tables 1 and 2), each of the addbsi (or Aux) fields containing the LPSM contains the following LPSM value: the core elements specified in Table 1, followed by a payload ID (identifying the metadata as LPSM) and a payload size value, followed by a payload (LPSM data) having the following format (similar to the elements indicated in Table 2 above): Version of the LPSM payload: a 2-bit field indicating the version of the LPSM payload; dialchan: a 3-bit field indicating whether the left, right, and / or center channel of the corresponding audio data contains spoken dialogue. The bit allocation of the dialchan field can be as follows: bit 0, indicating that dialogue is present in the left channel, is stored in the most significant bit of the dialchan field; and bit 2, indicating that dialogue is present in the center channel, is stored in the least significant bit of the dialchan field. Each bit of the dialchan field is set to "1" if the corresponding channel contains spoken dialogue during the first 0.5 seconds of the program;loudregtyp: A 3-bit field indicating the loudness rule standard to which the program loudness complies. Setting "loudregtyp" to "000" indicates that the LPSM does not indicate loudness rule compliance. For example, one value of this field (e.g., 000) may indicate that no loudness rule standard compliance is indicated, another value of this field (e.g., 001) may indicate that the program's audio data complies with the ATSC A / 85 standard, and another value of this field (e.g., 010) may indicate that the program's audio data complies with the EBU R128 standard. In the example, if the field is set to any value other than "000", the loudcorrdialgat and loudcorrtyp fields should follow in the payload;loudcorrdialgat: A 1-bit field indicating whether dialog gated loudness correction has been applied. If the loudness of the program has been corrected using dialog gating, the value of the loudcorrdialgat field is set to "1". Otherwise, it is set to "0";loudcorrtyp: A 1-bit field indicating the type of loudness correction applied to the program. If the loudness of the program has been corrected by an infinite look-ahead (file-based) loudness correction process, the value of the loudcorrtyp field is set to "0". If the loudness of the program has been corrected using a combination of real-time loudness measurement and dynamic range control, the value of this field is set to "1";loudrelgate: A 1-bit field indicating whether relative gated loudness data (ITU) is present. If the loudrelgate field is set to "1", a 7-bit ituludrelgat field shall follow in the payload;loudrelgat: A 7-bit field indicating the relative gated program loudness (ITU).This field indicates the integrated loudness of the audio program measured according to ITU-R BS.1770-2, without any gain adjustment due to the application of dialogue normalization and dynamic range compression. Values ​​0 to 127 are interpreted as -58LKFS to +5.5LKFS, with a step size of 0.5LKFS;loudspchgate: A 1-bit field indicating whether speech gated loudness data (ITU) is present. If the loudspchgate field is set to "1", a 7-bit loudspchgat field shall follow in the payload;loudspchgat: A 7-bit field indicating the loudness of the speech gated program. This field indicates the integrated loudness of the entire corresponding audio program measured according to equation (2) of ITU-R BS.1770-3, without any gain adjustment due to the application of dialogue normalization and dynamic range compression. Values ​​0 to 127 are interpreted as -58LKFS to +5.5LKFS, with a step size of 0.5LKFS;loudstrm3se: A 1-bit field indicating whether short-term (3-second) loudness data is present. If this field is set to "1", a 7-bit loudstrm3s field shall follow in the payload;loudstrm3s: A 7-bit field indicating the ungated loudness of the first 3 seconds of the corresponding audio program measured according to ITU-R BS.1771-1, without any gain adjustment due to the application of dialogue normalization and dynamic range compression. Values ​​0 to 256 are interpreted as -116LKFS to +11.5LKFS, with a step size of 0.5LKFS;truepke: A 1-bit field indicating whether true peak loudness data is present. If the truepke field is set to "1", then the 8-bit truepk field shall follow in the payload; and truepk: An 8-bit field indicating the true peak sample value of the program, measured according to Annex 2 of ITU-R BS.1770-3 and without any gain adjustment due to the application of dialogue normalization and dynamic range compression. Values ​​0 to 256 are interpreted as -116LKFS to +11.5LKFS, with a step size of 0.5LKFS.

[0099] A core element of a metadata segment in an auxdata field (or "addbsi" field) of a frame of an AC-3 bitstream or an E-AC-3 bitstream includes a core header (optionally including an identification value, such as a core element version), and includes, after the core header: a value indicating whether the metadata for the metadata segment contains fingerprint data (or other protection values), an indication of whether external data (related to the audio data corresponding to the metadata of the metadata segment) is present, a payload ID and payload size for each type of metadata identified by the core element (e.g., LPSM and / or metadata other than LPSM), and a protection value for at least one type of metadata identified by the core element. The metadata payload of the metadata segment follows the core header and (in some cases) is nested within the value of the core element.

[0100] Optimizing loudness and dynamic range systems

[0101] The secure metadata encoding and transmission scheme described above is similar to Figure 1 The scalable and extensible system shown for optimizing loudness and dynamic range between different playback devices, applications, and listening environments is used in combination. In one embodiment, the system 10 is configured to normalize the loudness level and dynamic range of the input audio 11 between various devices that require different loudness values ​​and have different dynamic range capabilities. In order to normalize the loudness level and dynamic range, the system 10 includes different device profiles about the audio content, and the normalization is performed based on these profiles. The profile can be included by one of the audio processing units in the audio processing chain, and the included profile can be used by the downstream processing unit in the audio processing chain to determine the desired target loudness and dynamic range of the target device. Additional processing components can provide or process information for device profile management (including but not limited to the following parameters: empty band range, true peak threshold, loudness range, fast / slow time constant (coefficient) and maximum boost), gain control and broadband and / or multi-band gain generation functions.

[0102] Fig.10 A system for providing optimized loudness and dynamic range control according to some embodiments is shown. Figure 1 A more detailed diagram of the system. Fig.10 The encoder stage of the system 321 includes a core encoder component 304, which encodes the audio input 303 into a suitable digital format for transmission to the decoder 312. The audio is processed so that it can be played back in a variety of different listening environments, each of which may require different loudness and / or dynamic range target settings. Fig.10 As shown, the decoder outputs a digital signal that is converted to an analog format by a digital-to-analog converter 316 for playback through a variety of different driver types including a full-range speaker 320, a small speaker 322, and headphones 324. These drivers are merely some examples of possible playback drivers, and any transducer or driver of any suitable size and type may be used. In addition, Fig.10 The drivers / transducers 320-324 may represent any suitable playback device for use in any corresponding listening environment. Device types may include, for example, AVR, television, stereo, computer, mobile phone, tablet, MP3 player, etc.; and listening environments may include, for example, an auditorium, a home, a car, a listening room, etc.

[0103] Since playback environments and driver types can range from very small private areas to very large public venues, the span of possible and optimal playback loudness and dynamic range configurations can vary significantly depending on the type of content, type of background noise, etc. For example, in a home theater environment, wide dynamic range content can be played through surround sound equipment, while narrower dynamic range content can be played through conventional TV systems (such as flat-panel LED / LCD types), and an extremely narrow dynamic range mode can be used for certain listening conditions when large level changes are not desired (e.g., at night or on devices with severe acoustic output power limitations (such as mobile phone / tablet internal speakers or headphone outputs)). In portable or mobile listening scenarios such as using small computer or docking speakers or headphones / earbuds, the optimal playback dynamic range can change depending on the environment. For example, the optimal dynamic range may be larger in a quiet environment than in a noisy environment. Fig.10 Embodiments of the adaptive audio processing system will change the dynamic range based on parameters (such as the listening device environment and the playback device type) to more clearly present the audio content.

[0104] Fig.11321. The system 321 includes a loudness and dynamic range measurement component 302 that analyzes and measures the loudness and dynamic range of the input audio. In one embodiment, the system analyzes the overall program content to determine the overall loudness parameter. In this context, loudness refers to the long-term program loudness or average loudness of a program, where a program is a single unit of audio content such as a movie, television program, commercial, or similar program content. Loudness is used to provide an indication of an artistic dynamic range profile used by a content creator to control how the audio will be played back. Loudness is related to a dialogue normalization metadata value, which represents the average dialogue loudness of a single program (e.g., movie, television, commercial, etc.). Compared to program loudness, short-term dynamic range quantifies signal changes over a much shorter period of time. For example, short-term dynamic range can be measured on the order of seconds, while program loudness can be measured on the span of minutes or even hours. Short term dynamic range provides a protection mechanism independent of program loudness to ensure that overload does not occur for a variety of playback profiles and device types. In one embodiment, the loudness (long term program loudness) target is based on dialog loudness, while the short term dynamic range is based on relative gated and / or ungated loudness. In this case, certain DRC and loudness components in the system are context-aware with respect to content type and / or target device type and characteristics. As part of this context-aware capability, the system is configured to analyze one or more characteristics of an output device to determine whether the device is a member of a particular group of devices (such as AVR type devices, televisions, computers, portable devices, etc.) that are optimized for certain DRC and loudness playback conditions.

[0105] The pre-processing component analyzes the program content to determine loudness, peak, true peak, and quiet periods to create unique metadata for each of a plurality of different profiles. In one embodiment, the loudness may be dialogue gated loudness and / or relative gated loudness. The different profiles define various DRC (dynamic range control) and target loudness modes in which different gain values ​​are generated in the encoder based on the characteristics of the source audio content, the desired target loudness, and the playback device type and / or environment. The decoder may provide different DRC and target loudness modes (enabled by the profiles mentioned above), and may include DRC and target loudness off / disabled, which allows full dynamic range presentation, no audio signal compression and no relative normalization; a Line mode with DRC off / disabled and loudness normalization targeting -31 LKFS, for playback on a home theater system, providing moderate dynamic range compression and loudness normalization targeting -31 LKFS via gain values ​​generated in the encoder (particularly for this playback mode and / or device profile); an RF mode, for playback through TV speakers, providing heavy dynamic range compression and loudness normalization targeting -24, -23 or -20 LKFS, an Intermediate mode, for playback through a computer or similar device, providing compression and loudness normalization targeting -14 LKFS, and a Portable mode, providing very heavy dynamic range compression and loudness normalization targeting -11 LKFS. The target loudness values ​​of -31, -23 / -20, -14, and -11 LKFS are contemplated to be examples of different playback / device profiles that may be defined for a system according to some embodiments, and any other suitable target loudness values ​​may be employed, and the system may generate suitable gain values, particularly for these playback modes and / or device profiles. Furthermore, the system may be expanded and modified so that different playback devices and listening environments may be accommodated by defining new profiles in the encoder or elsewhere and loaded into the encoder. In this way, new and unique playback / device profiles may be generated to support improved or different playback devices for future applications.

[0106] In one embodiment, the gain value may be calculated at any suitable processing component of system 321, such as at encoder 304, decoder 312, or transcoder 308, or any associated pre-processing component associated with an encoder or any post-processing component associated with a decoder.

[0107] Fig.13 is a block diagram illustrating the interface between different profiles for various different playback device classes according to one embodiment. Fig.13As shown, encoder 502 receives audio input 501 and one of several different possible profiles 506. The encoder combines the audio data with the selected profile to generate an output bitstream file, which is processed in a decoder component in or associated with a target playback device. Fig.13 514, and TV 516, but many other output devices are possible. Each of devices 510 to 516 includes or is coupled to a speaker (including a driver and / or a transducer), such as drivers 320 to 324. The combination of the size, power rating, and processing of the playback device and the associated speakers generally dictates which profile is optimal for a particular goal. Thus, profiles 506 may be specifically defined for playback through an AVR, TV, mobile speakers, mobile headphones, etc. They may also be defined for specific operating modes or conditions (such as quiet mode, night mode, outdoor, indoor, etc.). Fig.13 The profiles shown in are exemplary modes only, and any suitable profiles may be defined, including customized profiles for specific goals and circumstances.

[0108] although Fig.13 The following embodiment is shown, in which the encoder 502 receives the profile 506 and generates appropriate parameters for loudness and DRC processing, it should be noted that, however, the parameters generated based on the profile and the audio content can be performed on any suitable audio processing unit (such as an encoder, decoder, transcoder, pre-processor, post-processor, etc.). For example, Fig.13 Each output device 510 to 516 has or is coupled to a decoder component that processes metadata in the bitstream of file 504 sent from encoder 502 so that the loudness and dynamic range can be adapted to match the device or device type of the target output device.

[0109] In one embodiment, the dynamic range and loudness of the audio content is optimized for each possible playback device. This is achieved by maintaining long-term loudness as a target for each target playback mode and controlling short-term dynamic range to optimize the audio experience (by controlling signal dynamics, sample peaks, and / or true peaks). Different metadata elements are defined for long-term loudness and short-term dynamic range. Fig.10 As shown, component 302 analyses the entire input audio signal (or a portion thereof, such as a speech component, if applicable) to derive relevant characteristics of these individual DR components. This allows different gain values ​​to be defined for artistic gain versus clipping (overload protection) gain values.

[0110] These gain values ​​for long term loudness and short term dynamic range are then mapped to profiles 305 to generate parameters describing loudness and dynamic range control gain values. These parameters are combined with the encoded audio signal from encoder 304 in a multiplexer 306 or similar component for creating a bit stream, which is transmitted to the decoder stage through a transcoder 308. The bit stream input to the decoder stage is demultiplexed in a demultiplexer 310. It is then decoded in a decoder 312. Gain component 314 applies the gain corresponding to the appropriate profile to generate digital audio data, which is then processed by DACS unit 416 for playback through an appropriate playback device and driver or transducer 320-324.

[0111] Fig.14 is a table showing the correlation between long-term loudness and short-term dynamic range for multiple defined profiles according to one embodiment. Fig.14 As shown in Table 4 of , each profile includes a set of gain values ​​that indicate the amount of dynamic range compression (DRC) applied in the decoder of the system or each target device. Each of the N profiles, indicated as Profiles 1-N, sets specific long-term loudness parameters (e.g., dialogue normalization) and overload compression parameters by indicating corresponding gain values ​​applied in the decoder stage. The DRC gain values ​​for a profile may be defined by an external source accepted by the encoder, or they may be generated internally in the encoder as default gain values ​​if no external values ​​are provided.

[0112] In one embodiment, the gain values ​​for each profile are embodied in a DRC gain word that is calculated based on analysis of certain characteristics of the audio signal, such as peak, true peak, short term loudness of dialog, or overall long term loudness, or a combination (mix) thereof, to calculate static gain based on the time constants required to achieve the fast / slow rise and fast / slow release of the final DRC gain for each possible device profile and / or target loudness, and the selected profile (e.g., transfer characteristic or curve). As described above, these profiles may be pre-set in the encoder, decoder, or generated externally and sent to the encoder via external metadata from the content creator.

[0113] In one embodiment, the gain value can be a broadband gain that applies the same gain across all frequencies of the audio content. Alternatively, the gain can include a multi-band gain value, thereby applying different gain values ​​to different frequencies or frequency bands of the audio content. In a multi-channel situation, each profile can constitute a matrix of gain values ​​indicating gains for different frequency bands, rather than a single gain value.

[0114] Reference Fig.10In one embodiment, information about the nature or characteristics of the listening environment and / or the capabilities and configuration of the playback device is provided to the encoder stage by the feedback link 330 decoder stage. Profile information 332 is also input to the encoder 304. In one embodiment, the decoder analyzes the metadata in the bitstream to determine whether loudness parameters for the first group of playback devices are available in the bitstream. If available, the parameters are sent downstream for rendering the audio. Otherwise, the encoder analyzes certain characteristics of the device to derive the parameters. These parameters are then sent to the downstream rendering component for playback. The encoder also determines the output device (or a group of output devices including the output device) that will render the received audio stream. For example, the output device may be determined to be a mobile phone or to belong to a group of similar portable devices. In one embodiment, the decoder uses the feedback link 330 to indicate the determined output device or the group of output devices to the encoder. For this feedback, a module connected to the output device (e.g., a component in a sound card in a laptop computer connected to headphones or connected to speakers) can indicate the identity of the output device or the identity of a group of devices including the output device to the decoder. The decoder transmits this information to the encoder via the feedback link 330. In one embodiment, the encoder performs the decoder determination of loudness and DRC parameters. In one embodiment, the decoder determines the loudness and DRC parameters. In this embodiment, rather than transmitting information via feedback link 330, the decoder uses information about the determined device or group of output devices to determine the loudness and DRC parameters. In another embodiment, another audio processing unit determines the loudness and DRC parameters, and the decoder transmits this information to the audio processing unit instead of the decoder.

[0115] Fig.12 is a block diagram of a dynamic range optimization system according to one embodiment. Fig.12 As shown, encoder 402 receives input audio 401. The encoded audio is combined in multiplexer 409 with parameters 404 generated from a selected compression curve 422 and a dialogue normalization value 424. The resulting bitstream is transmitted to demultiplexer 411, which generates an audio signal, which is decoded by decoder 406. The parameters and dialogue normalization value are used by gain calculation unit 408 to generate a gain level, which drives amplifier 410 to amplify the decoder output. Fig.12 1 shows how dynamic range control can be parameterized and inserted into the bitstream. Loudness can also be parameterized and inserted into the bitstream using similar components. In one embodiment, an output reference level control (not shown) can be provided to the decoder. Although the figures show loudness and dynamic range parameters being determined and inserted at the encoder, similar determinations can be performed in other audio processing units (e.g., pre-processors, decoders, and post-processors).

[0116] Fig.152 shows examples of loudness profiles for different types of audio content according to one embodiment. Fig.15 As shown, exemplary curves 600 and 602 plot input loudness (in LFKS) against gain centered around 0 LKFS. Different types of content exhibit different curves, such as Fig.15 As shown, curve 600 may represent speech and curve 602 may represent standard capacitance content. Fig.15 As shown, speech content experiences a greater amount of gain than movie content. Fig.15 is an example of a representative profile curve for certain types of audio content, and other profiles may also be used. Fig.15 Certain aspects of the profile characteristics shown are used to derive relevant parameters for optimizing the system. In one embodiment, these parameters include: null bandwidth, clip ratio, boost ratio, maximum boost amount, FS rise, FS decay, hysteresis, peak limit, and target horizontal loudness. Other parameters may be used in addition to or instead of at least some of these parameters, depending on application requirements and system constraints.

[0117] Fig.16 6 is a flow chart illustrating a method for optimizing loudness and dynamic range between playback devices and applications according to one embodiment. Although the figures illustrate that loudness and dynamic range optimization is performed at an encoder, similar optimization may be performed at other audio processing units (e.g., preprocessors, decoders, and postprocessors). As shown in process 620, the method begins with the encoder stage receiving an input signal from a source (603). The encoder or preprocessing component then determines whether the original signal has been processed to achieve a target loudness and / or dynamic range (604). The target loudness corresponds to long-term loudness and may be defined externally or internally. If the source signal has not been processed to achieve the target loudness and / or dynamic range, the system performs an appropriate loudness and / or dynamic range control operation (608); otherwise, if the source signal has been subjected to such a loudness and / or dynamic range control operation, the system enters a bypass mode to skip such a loudness and / or dynamic range control operation and allow the original processing to indicate an appropriate long-term loudness and / or dynamic range (606). The appropriate gain value for either bypass mode 606 or executed mode 608 (which may be a single wideband gain value or a frequency-dependent multi-band gain value) is then applied in the decoder (612).

[0118] Bitstream format

[0119] As previously mentioned, the system for optimizing loudness and dynamic range employs a secure, extensible metadata format to ensure that metadata and audio content transmitted in a bitstream between an encoder and decoder or between a source and a rendering / playback device are not separated from each other or interrupted during transmission over a network or other proprietary device, such as a service provider interface, etc. The bitstream provides a mechanism for signaling the encoder and / or decoder components to change the loudness and dynamic range of the audio signal to suit the audio content and output device characteristics through appropriate profile information. In one embodiment, the system is configured to determine a low bitrate encoded bitstream to be transmitted between an encoder and a decoder, and the loudness information encoded through the metadata includes characteristics of one or more output profiles. A description of the bitstream format for the loudness and dynamic range optimization system according to one embodiment is as follows.

[0120] The AC-3 encoded bitstream includes metadata, as well as 1 to 6 channels of audio content. The audio content is audio data that has been compressed using perceptual audio coding. The metadata includes several audio metadata parameters that are intended to be used to change the sound of the program delivered to the listening environment. Each frame of the AC-3 encoded audio bitstream contains audio content and metadata for 1536 frames of digital audio. For a sampling rate of 48kHz, this represents 32 milliseconds of digital audio or 31.25 frames per second of audio.

[0121] Each frame of the E-AC-3 encoded audio bitstream contains audio content and metadata for 256, 512, 768, or 1536 frames of digital audio, depending on whether the frame contains 1, 2, 3, or 6 audio data blocks. For a sampling rate of 48kHz, this represents 5.333, 10.667, 16, or 32 milliseconds of digital audio, or 189.9, 93.75, 62.5, or 31.25 frames per second of audio, respectively.

[0122] like Figure 4 As indicated, each AC-3 frame is divided into segments (segments), including a synchronization information (SI) segment, which contains (eg Figure 5 ) synchronization word (SW) and the first of two error correction words (CRC1); a bit stream information (BSI) segment, which contains most of the metadata; six audio blocks (AB0 to AB5), which contain data compressed audio content (and may also include metadata); discarded bits (WI), which contain any unused bits left over after the audio content is compressed; an auxiliary (AUX) information segment, which contains more metadata; and the second of two error correction words (CRC2).

[0123] like Figure 7 As shown, each E-AC-3 frame is divided into segments (segments), including a synchronization information (SI) segment, which contains (eg Figure 5 The audio data frame includes a synchronization word (SW) as shown in the figure; a bit stream information (BSI) segment, which contains most of the metadata; between 1 and 6 audio blocks (AB0 to AB5), which contain data compressed audio content (and may also include metadata); discarded bits (WI), which contain any unused bits left over after the audio content is compressed; an auxiliary (AUX) information segment, which contains more metadata; and an error correction word (CRC).

[0124] In an AC-3 (or E-AC-3) bitstream, there are several audio metadata parameters that are specifically intended to change the sound of a program delivered to a listening environment. One of the metadata parameters is the Dialog Normalization parameter, which is contained in the BSI segment.

[0125] like Figure 6 As shown, the BSI segment of an AC-3 frame includes a 5-bit parameter ("dialnorm") indicating the dialogue normalization value of the program. If the audio coding mode ("acmod") of the AC-3 frame is "0", indicating dual mono or a "1+1" channel configuration, a 5-bit parameter ("dialnorm2") indicating the dialogue normalization value of the second audio program carried in the same AC-3 frame is included.

[0126] The BSI segment also includes a flag ("addbsie") indicating the presence (or absence) of additional bitstream information following the "addbsie" bits, a parameter ("addbsil") indicating the length of any additional bitstream information following the "addbsil" value, and up to 64 bits of additional bitstream information following the "addbsil" value ("addbsi"). The BSI segment may include Figure 6 Other metadata values ​​not specifically shown.

[0127] Aspects of one or more embodiments described herein may be implemented in an audio system that processes audio signals for transmission over a network including one or more computers or processing devices that execute software instructions. Any of the embodiments described may be used alone or in any combination in conjunction with one another. Although various embodiments have been conceived due to various defects in the prior art (which have been discussed or mentioned in one or more places in the specification), the embodiments do not necessarily solve any of these defects. In other words, different embodiments may solve different defects that may be discussed in the specification. Some embodiments may only partially solve some defects or only one defect that may be discussed in the specification, while some embodiments may not solve any of these defects.

[0128] Various aspects of the system described herein can be implemented in a suitable computer-based sound processing network environment for processing digital or digitized audio files. Portions of the adaptive audio system may include one or more networks containing any desired number of individual machines, including one or more routers (not shown) for buffering and routing data transmitted between computers. Such networks may be constructed based on a variety of different computer protocols and may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.

[0129] One or more of the components, blocks, processes or other functional components may be implemented by a computer program executed by a processor-based computing device that controls the system. It should also be noted that the various functions disclosed herein may be implemented in terms of their behavior, register transfers, logical components and / or other characteristics using any number of combinations of hardware, firmware and / or data and / or instructions embodied in various machine-readable or computer-readable media. Such computer-readable media in which formatted data and / or instructions are embodied include, but are not limited to, various forms of physical (non-transitory), non-volatile storage media, such as optical, magnetic or semiconductor storage media.

[0130] Unless the context clearly requires otherwise, in the context of the specification and claims, words such as "comprises," "including," and the like are to be interpreted in an inclusive sense as opposed to an exclusive or exhaustive sense; that is, in the sense of "including, but not limited to." Words using the singular or plural number also include the plural or singular number, respectively. In addition, "herein," "herein below," "above," "below," and similar words refer to the application as a whole and not to any particular part of the application. When the word "or" is used in reference to a list of two or more items, the word covers all of the following interpretations of the word: any one of the items in the list, all of the items in the list, or any combination of the items in the list.

[0131] Although one or more implementations have been described as examples and according to specific embodiments, it should be understood that one or more implementations are not limited to the disclosed embodiments. On the contrary, it is intended to cover various modifications and similar arrangements that will be apparent to those skilled in the art. Therefore, the scope of the appended claims should be given the broadest interpretation so as to cover all such modifications and similar arrangements.

Claims

1. An audio processing device for decoding one or more frames of an encoded audio bitstream, the encoded audio bitstream comprising metadata and audio data for a plurality of dynamic range control (DRC) profiles, the audio processing device comprising: a bitstream parser configured to parse the encoded audio bitstream and extract metadata for one or more of the DRC profiles and the encoded audio data; as well as an audio decoder configured to decode the encoded audio data and apply the DRC gain to the decoded audio data, wherein: Each DRC profile is applicable to at least one device type or listening environment; The audio decoder selects one or more of the DRC profiles in response to information about the audio processing device or the listening environment; and The DRC gains applied to the decoded audio data correspond to the one or more selected DRC profiles.

2. The audio processing device according to claim 1, wherein: The DRC gains corresponding to the one or more selected DRC profiles are included in metadata of the encoded audio bitstream.

3. The audio processing device according to claim 1, wherein: The DRC gains corresponding to the one or more selected DRC profiles are determined based on DRC parameters of the one or more selected DRC profiles contained in metadata of the encoded audio bitstream.

4. The audio processing device according to claim 3, wherein: The DRC parameters indicate the static gain transfer characteristic as well as the gain smoothing time constant.

5. The audio processing device according to claim 4, wherein: The time constants include slow and fast attack time constants, and slow and fast release time constants.

6. The audio processing device according to claim 4, wherein: The static gain transfer characteristic includes the null-band range and the maximum boost.

7. The audio processing device according to claim 1, wherein: One or more of the DRC profiles corresponds to playback of the decoded audio data by the portable device.

8. The audio processing device according to claim 1, wherein: One or more of the DRC profiles corresponds to playback of decoded audio data in a noisy listening environment.

9. The audio processing device according to claim 1, wherein: One or more of the DRC profiles corresponds to playback of decoded audio data in a quiet listening environment.

10. The audio processing device according to claim 1, wherein: One or more of the DRC profiles corresponds to playback of decoded audio data through a pair of headphones.

11. The audio processing device according to claim 1, wherein: One or more of the DRC profiles corresponds to playback of the decoded audio data through a speaker.

12. A method performed by an audio processing device for decoding one or more frames of an encoded audio bitstream, the encoded audio bitstream comprising metadata and audio data for a plurality of dynamic range control (DRC) profiles, the method comprising: parsing the encoded audio bitstream and extracting metadata for one or more of the DRC profiles and the encoded audio data; The encoded audio data is decoded and a DRC gain is applied to the decoded audio data, wherein: Each DRC profile is applicable to at least one device type or listening environment; One or more of the DRC profiles are selected in response to information about the audio processing device or the listening environment; and The DRC gains applied to the decoded audio data correspond to the one or more selected DRC profiles.

13. A non-transitory computer-readable storage medium comprising a sequence of instructions which, when executed by an audio decoding device, causes the audio decoding device to perform the method according to claim 12.

14. A method comprising: receiving metadata associated with the audio data in a bitstream; analyzing the metadata in the bitstream to determine whether loudness parameters for a first set of audio playback devices are available in the bitstream; In response to determining that parameters exist for the first group, rendering audio using the parameters and the audio data; as well as In response to determining that no loudness parameters exist for the first group, one or more characteristics of the first group are analyzed and parameters are determined based on the one or more characteristics.

15. An apparatus comprising: an interface configured to receive a bitstream including metadata associated with audio data; an analyzer coupled to the interface and configured to analyze metadata in the bitstream to determine whether loudness parameters for a first set of audio playback devices are available in the bitstream; as well as A rendering component is configured to, in response to the analyzer determining that parameters exist for the first group, render the audio using the parameters and the audio data, and the rendering component is further configured to, in response to the analyzer determining that loudness parameters do not exist for the first group, analyze one or more characteristics of the first group and determine parameters based on the one or more characteristics.

16. A method comprising: receiving audio data and metadata associated with the audio data; analyzing metadata in a bitstream to determine whether loudness information associated with loudness parameters for a first set of audio devices is available in the bitstream; responsive to determining that the loudness information exists for the first group, determining loudness information from the stream, and transmitting the audio data and the loudness information for rendering audio; as well as In response to determining that the loudness information is not present for the first group, loudness information associated with an output profile is determined, and the determined loudness information for the output profile is transmitted for rendering audio.

17. An audio data decoding device, comprising: An interface for receiving audio data and metadata associated with the audio data; a first component for analyzing metadata in a bitstream to determine whether loudness information associated with a loudness parameter for a first set of audio devices is available in the bitstream, the first component, in response to determining that the loudness information exists for the first group, determining the loudness information from the stream, and transmitting the audio data and the loudness information for rendering audio; And in response to determining that the loudness information is not present for the first group, determining loudness information associated with an output profile, and transmitting the determined loudness information for the output profile for rendering audio.

Images